12 Jails: Confining the omnipotent root.
14 This paper was presented at the 2nd International System Administration and Networking Conference "SANE 2000" May 22-25, 2000 in Maastricht, The Netherlands and is published in the proceedings.
17 Poul-Henning Kamp <phk@FreeBSD.org>
19 Robert N. M. Watson <rwatson@FreeBSD.org>
23 This work was sponsored by \fChttp://www.servetheweb.com/\fP and
24 donated to the FreeBSD Project for inclusion in the FreeBSD
25 OS. FreeBSD 4.0-RELEASE was the first release including this
27 Follow-on work was sponsored by Safeport Network Services,
28 \fChttp://www.safeport.com/\fP
31 The traditional UNIX security model is simple but inexpressive.
32 Adding fine-grained access control improves the expressiveness,
33 but often dramatically increases both the cost of system management
34 and implementation complexity.
35 In environments with a more complex management model, with delegation
36 of some management functions to parties under varying degrees of trust,
37 the base UNIX model and most natural
38 extensions are inappropriate at best.
39 Where multiple mutually un-trusting parties are introduced,
40 ``inappropriate'' rapidly transitions to ``nightmarish'', especially
41 with regards to data integrity and privacy protection.
43 The FreeBSD ``Jail'' facility provides the ability to partition
44 the operating system environment, while maintaining the simplicity
45 of the UNIX ``root'' model.
46 In Jail, users with privilege find that the scope of their requests
47 is limited to the jail, allowing system administrators to delegate
48 management capabilities for each virtual machine
50 Creating virtual machines in this manner has many potential uses; the
51 most popular thus far has been for providing virtual machine services
52 in Internet Service Provider environments.
57 The UNIX access control mechanism is designed for an environment with two
58 types of users: those with, and without administrative privilege.
59 Within this framework, every attempt is made to provide an open
60 system, allowing easy sharing of files and inter-process communication.
61 As a member of the UNIX family, FreeBSD inherits these
63 Users of FreeBSD in non-traditional UNIX environments must balance
64 their need for strong application support, high network performance
65 and functionality, and low total cost of ownership with the need
66 for alternative security models that are difficult or impossible to
67 implement with the UNIX security mechanisms.
69 One such consideration is the desire to delegate some (but not all)
70 administrative functions to untrusted or less trusted parties, and
71 simultaneously impose system-wide mandatory policies on process
72 interaction and sharing.
73 Attempting to create such an environment in the current-day FreeBSD
74 security environment is both difficult and costly: in many cases,
75 the burden of implementing these policies falls on user
76 applications, which means an increase in the size and complexity
77 of the code base, in turn translating to higher development
78 and maintenance cost, as well as less overall flexibility.
80 This abstract risk becomes more clear when applied to a practical,
82 many web service providers turn to the FreeBSD
83 operating system to host customer web sites, as it provides a
84 high-performance, network-centric server environment.
85 However, these providers have a number of concerns on their plate, both in
86 terms of protecting the integrity and confidentiality of their own
87 files and services from their customers, as well as protecting the files
88 and services of one customer from (accidental or
89 intentional) access by any other customer.
90 At the same time, a provider would like to provide
91 substantial autonomy to customers, allowing them to install and
92 maintain their own software, and to manage their own services,
93 such as web servers and other content-related daemon programs.
95 This problem space points strongly in the direction of a partitioning
96 solution, in which customer processes and storage are isolated from those of
97 other customers, both in terms of accidental disclosure of data or process
98 information, but also in terms of the ability to modify files or processes
99 outside of a compartment.
100 Delegation of management functions within the system must
101 be possible, but not at the cost of system-wide requirements, including
102 integrity and privacy protection between partitions.
104 However, UNIX-style access control makes it notoriously difficult to
105 compartmentalise functionality.
106 While mechanisms such as chroot(2) provide a modest
107 level compartmentalisation, it is well known
108 that these mechanisms have serious shortcomings, both in terms of the
109 scope of their functionality, and effectiveness at what they provide \s-2[CHROOT]\s+2.
111 In the case of the chroot(2) call, a process's visibility of
112 the file system name-space is limited to a single subtree.
113 However, the compartmentalisation does not extend to the process
114 or networking spaces and therefore both observation of and interference
115 with processes outside their compartment is possible.
117 To this end, we describe the new FreeBSD ``Jail'' facility, which
118 provides a strong partitioning solution, leveraging existing
119 mechanisms, such as chroot(2), to what effectively amounts to a
120 virtual machine environment. Processes in a jail are provided
121 full access to the files that they may manipulate, processes they
122 may influence, and network services they can make use of, and neither
123 access nor visibility of files, processes or network services outside
126 Unlike other fine-grained security solutions, Jail does not
127 substantially increase the policy management requirements for the
128 system administrator, as each Jail is a virtual FreeBSD environment
129 permitting local policy to be independently managed, with much the
130 same properties as the main system itself, making Jail easy to use
131 for the administrator, and far more compatible with applications.
133 Traditional UNIX Security, or, ``God, root, what difference?" \s-2[UF]\s+2.
135 The traditional UNIX access model assigns numeric uids to each user of the
136 system. In turn, each process ``owned'' by a user will be tagged with that
137 user's uid in an unforgeable manner. The uids serve two purposes: first,
138 they determine how discretionary access control mechanisms will be applied, and
139 second, they are used to determine whether special privileges are accorded.
141 In the case of discretionary access controls, the primary object protected is
142 a file. The uid (and related gids indicating group membership) are mapped to
143 a set of rights for each object, courtesy the UNIX file mode, in effect acting
144 as a limited form of access control list. Jail is, in general, not concerned
145 with modifying the semantics of discretionary access control mechanisms,
146 although there are important implications from a management perspective.
148 For the purposes of determining whether special privileges are accorded to a
149 process, the check is simple: ``is the numeric uid equal to 0 ?''.
151 process is acting with ``super-user privileges'', and all access checks are
152 granted, in effect allowing the process the ability to do whatever it wants
155 \&... no matter how patently stupid it may be.
158 For the purposes of human convenience, uid 0 is canonically allocated
159 to the ``root'' user \s-2[ROOT]\s+2.
160 For the purposes of jail, this behaviour is extremely relevant: many of
161 these privileged operations can be used to manage system hardware and
162 configuration, file system name-space, and special network operations.
164 Many limitations to this model are immediately clear: the root user is a
165 single, concentrated source of privilege that is exposed to many pieces of
166 software, and as such an immediate target for attacks. In the event of a
167 compromise of the root capability set, the attacker has complete control over
168 the system. Even without an attacker, the risks of a single administrative
169 account are serious: delegating a narrow scope of capability to an
170 inexperienced administrator is difficult, as the granularity of delegation is
171 that of all system management abilities. These features make the omnipotent
172 root account a sharp, efficient and extremely dangerous tool.
174 The BSD family of operating systems have implemented the ``securelevel''
175 mechanism which allows the administrator to block certain configuration
176 and management functions from being performed by root,
177 until the system is restarted and brought up into single-user mode.
178 While this does provide some amount of protection in the case of a root
179 compromise of the machine, it does nothing to address the need for
180 delegation of certain root abilities.
182 Other Solutions to the Root Problem
184 Many operating systems attempt to address these limitations by providing
185 fine-grained access controls for system resources \s-2[BIBA]\s+2.
186 These efforts vary in
187 degrees of success, but almost all suffer from at least three serious
190 First, increasing the granularity of security controls increases the
191 complexity of the administration process, in turn increasing both the
192 opportunity for incorrect configuration, as well as the demand on
193 administrator time and resources. In many cases, the increased complexity
194 results in significant frustration for the administrator, which may result
196 disastrous types of policy: ``all doors open as it's too much trouble'', and
197 ``trust that the system is secure, when in fact it isn't''.
199 The extent of the trouble is best illustrated by the fact that an entire
200 niche industry has emerged providing tools to manage fine grained security
201 controls \s-2[UAS]\s+2.
203 Second, usefully segregating capabilities and assigning them to running code
204 and users is very difficult. Many privileged operations in UNIX seem
205 independent, but are in fact closely related, and the handing out of one
206 privilege may, in effect, be transitive to the many others. For example, in
207 some trusted operating systems, a system capability may be assigned to a
208 running process to allow it to read any file, for the purposes of backup.
209 However, this capability is, in effect, equivalent to the ability to switch to
210 any other account, as the ability to access any file provides access to system
211 keying material, which in turn provides the ability to authenticate as any
212 user. Similarly, many operating systems attempt to segregate management
213 capabilities from auditing capabilities. In a number of these operating
214 systems, however, ``management capabilities'' permit the administrator to
215 assign ``auditing capabilities'' to itself, or another account, circumventing
216 the segregation of capability.
218 Finally, introducing new security features often involves introducing new
219 security management APIs. When fine-grained capabilities are introduced to
220 replace the setuid mechanism in UNIX-like operating systems, applications that
221 previously did an ``appropriateness check'' to see if they were running as
222 root before executing must now be changed to know that they need not run as
223 root. In the case of applications running with privilege and executing other
224 programs, there is now a new set of privileges that must be voluntarily given
225 up before executing another program. These change can introduce significant
226 incompatibility for existing applications, and make life more difficult for
227 application developers who may not be aware of differing security semantics on
228 different systems \s-2[POSIX1e]\s+2.
230 The Jail Partitioning Solution
232 Jail neatly side-steps the majority of these problems through partitioning.
234 than introduce additional fine-grained access control mechanism, we partition
235 a FreeBSD environment (processes, file system, network resources) into a
236 management environment, and optionally subset Jail environments. In doing so,
237 we simultaneously maintain the existing UNIX security model, allowing
238 multiple users and a privileged root user in each jail, while
239 limiting the scope of root's activities to his jail.
240 Consequently the administrator of a
241 FreeBSD machine can partition the machine into separate jails, and provide
242 access to the super-user account in each of these without losing control of
243 the over-all environment.
245 A process in a partition is referred to as ``in jail''. When a FreeBSD
246 system is booted up after a fresh install, no processes will be in jail.
248 a process is placed in a jail, it, and any descendents of the process created
249 after the jail creation, will be in that jail. A process may be in only one
250 jail, and after creation, it can not leave the jail.
251 Jails are created when a
252 privileged process calls the jail(2) syscall, with a description of the jail as an
253 argument to the call. Each call to jail(2) creates a new jail; the only way
254 for a new process to enter the jail is by inheriting access to the jail from
255 another process already in that jail.
257 leave the jail they created, or were created in.
259 .if t .PSPIC jail01.eps 4i
261 Fig. 1 \(em Schematic diagram of machine with two configured jails
265 Membership in a jail involves a number of restrictions: access to the file
266 name-space is restricted in the style of chroot(2), the ability to bind network
267 resources is limited to a specific IP address, the ability to manipulate
268 system resources and perform privileged operations is sharply curtailed, and
269 the ability to interact with other processes is limited to only processes
270 inside the same jail.
272 Jail takes advantage of the existing chroot(2) behaviour to limit access to the
273 file system name-space for jailed processes. When a jail is created, it is
274 bound to a particular file system root.
275 Processes are unable to manipulate files that they cannot address,
276 and as such the integrity and confidentiality of files outside of the jail
277 file system root are protected. Traditional mechanisms for breaking out of
278 chroot(2) have been blocked.
279 In the expected and documented configuration, each jail is provided
280 with its exclusive file system root, and standard FreeBSD directory layout,
281 but this is not mandated by the implementation.
283 Each jail is bound to a single IP address: processes within the jail may not
284 make use of any other IP address for outgoing or incoming connections; this
285 includes the ability to restrict what network services a particular jail may
286 offer. As FreeBSD distinguishes attempts to bind all IP addresses from
287 attempts to bind a particular address, bind requests for all IP addresses are
288 redirected to the individual Jail address. Some network functionality
289 associated with privileged calls are wholesale disabled due to the nature of the
290 functionality offered, in particular facilities which would allow ``spoofing''
291 of IP numbers or disruptive traffic to be generated have been disabled.
293 Processes running without root privileges will notice few, if any differences
294 between a jailed environment or un-jailed environment. Processes running with
295 root privileges will find that many restrictions apply to the privileged calls
296 they may make. Some calls will now return an access error \(em for example, an
297 attempt to create a device node will now fail. Others will have a more
298 limited scope than normal \(em attempts to bind a reserved port number on all
299 available addresses will result in binding only the address associated with
300 the jail. Other calls will succeed as normal: root may read a file owned by
301 any uid, as long as it is accessible through the jail file system name-space.
303 Processes within the jail will find that they are unable to interact or
304 even verify the existence of
305 processes outside the jail \(em processes within the jail are
306 prevented from delivering signals to processes outside the jail, as well as
307 connecting to those processes with debuggers, or even see them in the
308 sysctl or process file system monitoring mechanisms. Jail does not prevent,
309 nor is it intended to prevent, the use of covert channels or communications
310 mechanisms via accepted interfaces \(em for example, two processes may communicate
311 via sockets over the IP network interface. Nor does it attempt to provide
312 scheduling services based on the partition; however, it does prevent calls
313 that interfere with normal process operation.
315 As a result of these attempts to retain the standard FreeBSD API and
316 framework, almost all applications will run unaffected. Standard system
317 services such as Telnet, FTP, and SSH all behave normally, as do most third
318 party applications, including the popular Apache web server.
322 Processes running with root privileges in the jail find that there are serious
323 restrictions on what it is capable of doing \(em in particular, activities that
324 would extend outside of the jail:
326 \(bu Modifying the running kernel by direct access and loading kernel
327 modules is prohibited.
329 \(bu Modifying any of the network configuration, interfaces, addresses, and
330 routing table is prohibited.
332 \(bu Mounting and unmounting file systems is prohibited.
334 \(bu Creating device nodes is prohibited.
336 \(bu Accessing raw, divert, or routing sockets is prohibited.
338 \(bu Modifying kernel runtime parameters, such as most sysctl settings, is
341 \(bu Changing securelevel-related file flags is prohibited.
343 \(bu Accessing network resources not associated with the jail is prohibited.
345 Other privileged activities are permitted as long as they are limited to the
348 \(bu Signalling any process within the jail is permitted.
350 \(bu Changing the ownership and mode of any file within the jail is permitted, as
351 long as the file flags permit this.
353 \(bu Deleting any file within the jail is permitted, as long as the file flags
356 \(bu Binding reserved TCP and UDP port numbers on the jails IP address is
357 permitted. (Attempts to bind TCP and UDP ports using INADDR_ANY will be
358 redirected to the jails IP address.)
360 \(bu Functions which operate on the uid/gid space are all permitted since they
361 act as labels for filesystem objects of proceses
362 which are partitioned off by other mechanisms.
364 These restrictions on root access limit the scope of root processes, enabling
365 most applications to run un-hindered, but preventing calls that might allow an
366 application to reach beyond the jail and influence other processes or
367 system-wide configuration.
369 .so implementation.ms
375 The jail facility provides FreeBSD with a conceptually simple security
376 partitioning mechanism, allowing the delegation of administrative rights
377 within virtual machine partitions.
379 The implementation relies on
380 restricting access within the jail environment to a well-defined subset
381 of the overall host environment. This includes limiting interaction
382 between processes, and to files, network resources, and privileged
383 operations. Administrative overhead is reduced through avoiding
384 fine-grained access control mechanisms, and maintaining a consistent
385 administrative interface across partitions and the host environment.
387 The jail facility has already seen widespread deployment in particular as
388 a vehicle for delivering "virtual private server" services.
390 The jail code is included in the base system as part of FreeBSD 4.0-RELEASE,
391 and fully documented in the jail(2) and jail(8) man-pages.
395 .IP \s-2[BIBA]\s+2 .5i
396 K. J. Biba, Integrity Considerations for Secure
397 Computer Systems, USAF Electronic Systems Division, 1977
398 .IP \s-2[CHROOT]\s+2 .5i
399 Dr. Marshall Kirk Mckusick, private communication:
400 ``According to the SCCS logs, the chroot call was added by Bill Joy
401 on March 18, 1982 approximately 1.5 years before 4.2BSD was released.
402 That was well before we had ftp servers of any sort (ftp did not
403 show up in the source tree until January 1983). My best guess as
404 to its purpose was to allow Bill to chroot into the /4.2BSD build
405 directory and build a system using only the files, include files,
406 etc contained in that tree. That was the only use of chroot that
407 I remember from the early days.''
408 .IP \s-2[LOTTERY1]\s+2 .5i
409 David Petrou and John Milford. Proportional-Share Scheduling:
410 Implementation and Evaluation in a Widely-Deployed Operating System,
413 \s-2\fChttp://www.cs.cmu.edu/~dpetrou/papers/freebsd_lottery_writeup98.ps\fP\s+2
414 \s-2\fChttp://www.cs.cmu.edu/~dpetrou/code/freebsd_lottery_code.tar.gz\fP\s+2
415 .IP \s-2[LOTTERY2]\s+2 .5i
416 Carl A. Waldspurger and William E. Weihl. Lottery Scheduling: Flexible Proportional-Share Resource Management, Proceedings of the First Symposium on Operating Systems Design and Implementation (OSDI '94), pages 1-11, Monterey, California, November 1994.
418 \s-2\fChttp://www.research.digital.com/SRC/personal/caw/papers.html\fP\s+2
419 .IP \s-2[POSIX1e]\s+2 .5i
420 Draft Standard for Information Technology \(em
421 Portable Operating System Interface (POSIX) \(em
422 Part 1: System Application Program Interface (API) \(em Amendment:
423 Protection, Audit and Control Interfaces [C Language]
424 IEEE Std 1003.1e Draft 17 Editor Casey Schaufler
425 .IP \s-2[ROOT]\s+2 .5i
426 Historically other names have been used at times, Zilog for instance
427 called the super-user account ``zeus''.
428 .IP \s-2[UAS]\s+2 .5i
429 One such niche product is the ``UAS'' system to maintain and audit
430 RACF configurations on MVS systems.
432 \s-2\fChttp://www.entactinfo.com/products/uas/\fP\s+2
434 Quote from the User-Friendly cartoon by Illiad.
436 \s-2\fChttp://www.userfriendly.org/cartoons/archives/98nov/19981111.html\fP\s+2
projects / FreeBSD / FreeBSD.git / blob