1 .\" Copyright (C) 2001 Matthew Dillon. All rights reserved.
2 .\" Copyright (C) 2012 Eitan Adler.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
14 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16 .\" ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
17 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
32 .Nd performance tuning under FreeBSD
33 .Sh SYSTEM SETUP - DISKLABEL, NEWFS, TUNEFS, SWAP
34 The swap partition should typically be approximately 2x the size of
36 for systems with less than 4GB of RAM, or approximately equal to
37 the size of main memory
39 Keep in mind future memory
40 expansion when sizing the swap partition.
41 Configuring too little swap can lead
42 to inefficiencies in the VM page scanning code as well as create issues
43 later on if you add more memory to your machine.
45 with multiple disks, configure swap on each drive.
46 The swap partitions on the drives should be approximately the same size.
47 The kernel can handle arbitrary sizes but
48 internal data structures scale to 4 times the largest swap partition.
50 the swap partitions near the same size will allow the kernel to optimally
51 stripe swap space across the N disks.
52 Do not worry about overdoing it a
53 little, swap space is the saving grace of
55 and even if you do not normally use much swap, it can give you more time to
56 recover from a runaway program before being forced to reboot.
58 It is not a good idea to make one large partition.
60 each partition has different operational characteristics and separating them
61 allows the file system to tune itself to those characteristics.
65 partitions are read-mostly, with very little writing, while
66 a lot of reading and writing could occur in
69 partitioning your system fragmentation introduced in the smaller more
70 heavily write-loaded partitions will not bleed over into the mostly-read
73 Properly partitioning your system also allows you to tune
80 option worthwhile turning on is
83 .Dq Li "tunefs -n enable /filesystem" .
84 Softupdates drastically improves meta-data performance, mainly file
85 creation and deletion.
86 We recommend enabling softupdates on most file systems; however, there
87 are two limitations to softupdates that you should be aware of when
88 determining whether to use it on a file system.
89 First, softupdates guarantees file system consistency in the
90 case of a crash but could very easily be several seconds (even a minute!\&)
91 behind on pending write to the physical disk.
92 If you crash you may lose more work
94 Secondly, softupdates delays the freeing of file system
96 If you have a file system (such as the root file system) which is
97 close to full, doing a major update of it, e.g.,\&
98 .Dq Li "make installworld" ,
99 can run it out of space and cause the update to fail.
100 For this reason, softupdates will not be enabled on the root file system
101 during a typical install.
102 There is no loss of performance since the root
103 file system is rarely written to.
107 options exist that can help you tune the system.
108 The most obvious and most dangerous one is
110 Only use this option in conjunction with
112 as it is far too dangerous on a normal file system.
113 A less dangerous and more
119 file systems normally update the last-accessed time of a file or
120 directory whenever it is accessed.
121 This operation is handled in
123 with a delayed write and normally does not create a burden on the system.
124 However, if your system is accessing a huge number of files on a continuing
125 basis the buffer cache can wind up getting polluted with atime updates,
126 creating a burden on the system.
127 For example, if you are running a heavily
128 loaded web site, or a news server with lots of readers, you might want to
129 consider turning off atime updates on your larger partitions with this
132 However, you should not gratuitously turn off atime
136 file system customarily
137 holds mailboxes, and atime (in combination with mtime) is used to
138 determine whether a mailbox has new mail.
139 You might as well leave
140 atime turned on for mostly read-only partitions such as
145 This is especially useful for
147 since some system utilities
148 use the atime field for reporting.
150 In larger systems you can stripe partitions from several drives together
151 to create a much larger overall partition.
152 Striping can also improve
153 the performance of a file system by splitting I/O operations across two
160 utilities may be used to create simple striped file systems.
162 speaking, striping smaller partitions such as the root and
164 or essentially read-only partitions such as
166 is a complete waste of time.
167 You should only stripe partitions that require serious I/O performance,
170 or custom partitions used to hold databases and web pages.
171 Choosing the proper stripe size is also
173 File systems tend to store meta-data on power-of-2 boundaries
174 and you usually want to reduce seeking rather than increase seeking.
176 means you want to use a large off-center stripe size such as 1152 sectors
177 so sequential I/O does not seek both disks and so meta-data is distributed
178 across both disks rather than concentrated on a single disk.
181 variables permit system behavior to be monitored and controlled at
183 Some sysctls simply report on the behavior of the system; others allow
184 the system behavior to be modified;
185 some may be set at boot time using
187 but most will be set via
189 There are several hundred sysctls in the system, including many that appear
190 to be candidates for tuning but actually are not.
191 In this document we will only cover the ones that have the greatest effect
196 sysctl defines the overcommit behaviour of the vm subsystem.
197 The virtual memory system always does accounting of the swap space
198 reservation, both total for system and per-user.
200 are available through sysctl
202 that gives the total bytes available for swapping, and
203 .Va vm.swap_reserved ,
204 that gives number of bytes that may be needed to back all currently
205 allocated anonymous memory.
209 sysctl causes the virtual memory system to return failure
210 to the process when allocation of memory causes
214 Bit 1 of the sysctl enforces
219 Root is exempt from this limit.
220 Bit 2 allows to count most of the physical
221 memory as allocatable, except wired and free reserved pages
223 .Va vm.stats.vm.v_free_target
225 .Va vm.stats.vm.v_wire_count
226 sysctls, respectively).
229 .Va kern.ipc.maxpipekva
230 loader tunable is used to set a hard limit on the
231 amount of kernel address space allocated to mapping of pipe buffers.
232 Use of the mapping allows the kernel to eliminate a copy of the
233 data from writer address space into the kernel, directly copying
234 the content of mapped buffer to the reader.
235 Increasing this value to a higher setting, such as `25165824' might
236 improve performance on systems where space for mapping pipe buffers
237 is quickly exhausted.
238 This exhaustion is not fatal; however, and it will only cause pipes
239 to fall back to using double-copy.
242 .Va kern.ipc.shm_use_phys
243 sysctl defaults to 0 (off) and may be set to 0 (off) or 1 (on).
245 this parameter to 1 will cause all System V shared memory segments to be
246 mapped to unpageable physical RAM.
247 This feature only has an effect if you
248 are either (A) mapping small amounts of shared memory across many (hundreds)
249 of processes, or (B) mapping large amounts of shared memory across any
251 This feature allows the kernel to remove a great deal
252 of internal memory management page-tracking overhead at the cost of wiring
253 the shared memory into core, making it unswappable.
256 .Va vfs.vmiodirenable
257 sysctl defaults to 1 (on).
258 This parameter controls how directories are cached
260 Most directories are small and use but a single fragment
261 (typically 2K) in the file system and even less (typically 512 bytes) in
263 However, when operating in the default mode the buffer
264 cache will only cache a fixed number of directories even if you have a huge
266 Turning on this sysctl allows the buffer cache to use
267 the VM Page Cache to cache the directories.
268 The advantage is that all of
269 memory is now available for caching directories.
270 The disadvantage is that
271 the minimum in-core memory used to cache a directory is the physical page
272 size (typically 4K) rather than 512 bytes.
273 We recommend turning this option off in memory-constrained environments;
274 however, when on, it will substantially improve the performance of services
275 that manipulate a large number of files.
276 Such services can include web caches, large mail systems, and news systems.
277 Turning on this option will generally not reduce performance even with the
278 wasted memory but you should experiment to find out.
282 sysctl defaults to 1 (on).
283 This tells the file system to issue media
284 writes as full clusters are collected, which typically occurs when writing
285 large sequential files.
286 The idea is to avoid saturating the buffer
287 cache with dirty buffers when it would not benefit I/O performance.
289 this may stall processes and under certain circumstances you may wish to turn
293 .Va vfs.hirunningspace
294 sysctl determines how much outstanding write I/O may be queued to
295 disk controllers system-wide at any given time.
296 It is used by the UFS file system.
297 The default is self-tuned and
298 usually sufficient but on machines with advanced controllers and lots
299 of disks this may be tuned up to match what the controllers buffer.
300 Configuring this setting to match tagged queuing capabilities of
301 controllers or drives with average IO size used in production works
302 best (for example: 16 MiB will use 128 tags with IO requests of 128 KiB).
303 Note that setting too high a value
304 (exceeding the buffer cache's write threshold) can lead to extremely
305 bad clustering performance.
306 Do not set this value arbitrarily high!
307 Higher write queuing values may also add latency to reads occurring at
312 sysctl governs VFS read-ahead and is expressed as the number of blocks
313 to pre-read if the heuristics algorithm decides that the reads are
315 It is used by the UFS, ext2fs and msdosfs file systems.
316 With the default UFS block size of 32 KiB, a setting of 64 will allow
317 speculatively reading up to 2 MiB.
318 This setting may be increased to get around disk I/O latencies, especially
319 where these latencies are large such as in virtual machine emulated
321 It may be tuned down in specific cases where the I/O load is such that
322 read-ahead adversely affects performance or where system memory is really
327 sysctl defines how large VFS namecache may grow.
328 The number of currently allocated entries in namecache is provided by
330 sysctl and the condition
331 debug.numcache < kern.maxvnodes * vfs.ncsizefactor
336 sysctl defines how many negative entries VFS namecache is allowed to create.
337 The number of currently allocated negative entries is provided by
339 sysctl and the condition
340 vfs.ncnegfactor * debug.numneg < debug.numcache
343 There are various other buffer-cache and VM page cache related sysctls.
344 We do not recommend modifying these values.
347 .Va net.inet.tcp.sendspace
349 .Va net.inet.tcp.recvspace
350 sysctls are of particular interest if you are running network intensive
352 They control the amount of send and receive buffer space
353 allowed for any given TCP connection.
354 The default sending buffer is 32K; the default receiving buffer
357 improve bandwidth utilization by increasing the default at the cost of
358 eating up more kernel memory for each connection.
360 increasing the defaults if you are serving hundreds or thousands of
361 simultaneous connections because it is possible to quickly run the system
362 out of memory due to stalled connections building up.
364 high bandwidth over a fewer number of connections, especially if you have
365 gigabit Ethernet, increasing these defaults can make a huge difference.
366 You can adjust the buffer size for incoming and outgoing data separately.
367 For example, if your machine is primarily doing web serving you may want
368 to decrease the recvspace in order to be able to increase the
369 sendspace without eating too much kernel memory.
370 Note that the routing table (see
372 can be used to introduce route-specific send and receive buffer size
375 As an additional management tool you can use pipes in your
378 to limit the bandwidth going to or from particular IP blocks or ports.
379 For example, if you have a T1 you might want to limit your web traffic
380 to 70% of the T1's bandwidth in order to leave the remainder available
381 for mail and interactive use.
382 Normally a heavily loaded web server
383 will not introduce significant latencies into other services even if
384 the network link is maxed out, but enforcing a limit can smooth things
385 out and lead to longer term stability.
386 Many people also enforce artificial
387 bandwidth limitations in order to ensure that they are not charged for
388 using too much bandwidth.
390 Setting the send or receive TCP buffer to values larger than 65535 will result
391 in a marginal performance improvement unless both hosts support the window
392 scaling extension of the TCP protocol, which is controlled by the
393 .Va net.inet.tcp.rfc1323
395 These extensions should be enabled and the TCP buffer size should be set
396 to a value larger than 65536 in order to obtain good performance from
397 certain types of network links; specifically, gigabit WAN links and
398 high-latency satellite links.
399 RFC1323 support is enabled by default.
402 .Va net.inet.tcp.always_keepalive
403 sysctl determines whether or not the TCP implementation should attempt
404 to detect dead TCP connections by intermittently delivering
407 By default, this is enabled for all applications; by setting this
408 sysctl to 0, only applications that specifically request keepalives
410 In most environments, TCP keepalives will improve the management of
411 system state by expiring dead TCP connections, particularly for
412 systems serving dialup users who may not always terminate individual
413 TCP connections before disconnecting from the network.
414 However, in some environments, temporary network outages may be
415 incorrectly identified as dead sessions, resulting in unexpectedly
416 terminated TCP connections.
417 In such environments, setting the sysctl to 0 may reduce the occurrence of
418 TCP session disconnections.
421 .Va net.inet.tcp.delayed_ack
422 TCP feature is largely misunderstood.
423 Historically speaking, this feature
424 was designed to allow the acknowledgement to transmitted data to be returned
425 along with the response.
426 For example, when you type over a remote shell,
427 the acknowledgement to the character you send can be returned along with the
428 data representing the echo of the character.
429 With delayed acks turned off,
430 the acknowledgement may be sent in its own packet, before the remote service
431 has a chance to echo the data it just received.
432 This same concept also
433 applies to any interactive protocol (e.g.,\& SMTP, WWW, POP3), and can cut the
434 number of tiny packets flowing across the network in half.
437 delayed ACK implementation also follows the TCP protocol rule that
438 at least every other packet be acknowledged even if the standard 100ms
439 timeout has not yet passed.
440 Normally the worst a delayed ACK can do is
441 slightly delay the teardown of a connection, or slightly delay the ramp-up
442 of a slow-start TCP connection.
443 While we are not sure we believe that
444 the several FAQs related to packages such as SAMBA and SQUID which advise
445 turning off delayed acks may be referring to the slow-start issue.
448 .Va net.inet.ip.portrange.*
449 sysctls control the port number ranges automatically bound to TCP and UDP
451 There are three ranges: a low range, a default range, and a
452 high range, selectable via the
457 network programs use the default range which is controlled by
458 .Va net.inet.ip.portrange.first
460 .Va net.inet.ip.portrange.last ,
461 which default to 49152 and 65535, respectively.
462 Bound port ranges are
463 used for outgoing connections, and it is possible to run the system out
464 of ports under certain circumstances.
465 This most commonly occurs when you are
466 running a heavily loaded web proxy.
467 The port range is not an issue
468 when running a server which handles mainly incoming connections, such as a
469 normal web server, or has a limited number of outgoing connections, such
471 For situations where you may run out of ports,
472 we recommend decreasing
473 .Va net.inet.ip.portrange.first
475 A range of 10000 to 30000 ports may be reasonable.
476 You should also consider firewall effects when changing the port range.
478 may block large ranges of ports (usually low-numbered ports) and expect systems
479 to use higher ranges of ports for outgoing connections.
481 .Va net.inet.ip.portrange.last
482 is set at the maximum allowable port number.
485 .Va kern.ipc.somaxconn
486 sysctl limits the size of the listen queue for accepting new TCP connections.
487 The default value of 128 is typically too low for robust handling of new
488 connections in a heavily loaded web server environment.
489 For such environments,
490 we recommend increasing this value to 1024 or higher.
492 may itself limit the listen queue size (e.g.,\&
495 often have a directive in its configuration file to adjust the queue size up.
496 Larger listen queues also do a better job of fending off denial of service
501 sysctl determines how many open files the system supports.
503 typically a few thousand but you may need to bump this up to ten or twenty
504 thousand if you are running databases or large descriptor-heavy daemons.
507 sysctl may be interrogated to determine the current number of open files
511 .Va vm.swap_idle_enabled
512 sysctl is useful in large multi-user systems where you have lots of users
513 entering and leaving the system and lots of idle processes.
515 tend to generate a great deal of continuous pressure on free memory reserves.
516 Turning this feature on and adjusting the swapout hysteresis (in idle
518 .Va vm.swap_idle_threshold1
520 .Va vm.swap_idle_threshold2
521 allows you to depress the priority of pages associated with idle processes
522 more quickly then the normal pageout algorithm.
523 This gives a helping hand
524 to the pageout daemon.
525 Do not turn this option on unless you need it,
526 because the tradeoff you are making is to essentially pre-page memory sooner
527 rather than later, eating more swap and disk bandwidth.
529 this option will have a detrimental effect but in a large system that is
530 already doing moderate paging this option allows the VM system to stage
531 whole processes into and out of memory more easily.
533 Some aspects of the system behavior may not be tunable at runtime because
534 memory allocations they perform must occur early in the boot process.
535 To change loader tunables, you must set their values in
537 and reboot the system.
540 controls the scaling of a number of static system tables, including defaults
541 for the maximum number of open files, sizing of network memory resources, etc.
543 is automatically sized at boot based on the amount of memory available in
544 the system, and may be determined at run-time by inspecting the value of the
553 tunables set the default soft limits for process data and stack size
555 Processes may increase these up to the hard limits by calling
562 tunables set the hard limits for process data, stack, and text size
563 respectively; processes may not exceed these limits.
566 tunable controls how much the stack segment will grow when a process
567 needs to allocate more stack.
569 .Va kern.ipc.nmbclusters
570 may be adjusted to increase the number of network mbufs the system is
572 Each cluster represents approximately 2K of memory,
573 so a value of 1024 represents 2M of kernel memory reserved for network
575 You can do a simple calculation to figure out how many you need.
576 If you have a web server which maxes out at 1000 simultaneous connections,
577 and each connection eats a 16K receive and 16K send buffer, you need
578 approximately 32MB worth of network buffers to deal with it.
580 thumb is to multiply by 2, so 32MBx2 = 64MB/2K = 32768.
582 you would want to set
583 .Va kern.ipc.nmbclusters
585 We recommend values between
586 1024 and 4096 for machines with moderates amount of memory, and between 4096
587 and 32768 for machines with greater amounts of memory.
588 Under no circumstances
589 should you specify an arbitrarily high value for this parameter, it could
590 lead to a boot-time crash.
595 may be used to observe network cluster use.
597 More and more programs are using the
599 system call to transmit files over the network.
602 sysctl controls the number of file system buffers
604 is allowed to use to perform its work.
605 This parameter nominally scales
608 so you should not need to modify this parameter except under extreme
614 manual page for details.
615 .Sh KERNEL CONFIG TUNING
616 There are a number of kernel options that you may have to fiddle with in
617 a large-scale system.
618 In order to change these options you need to be
619 able to compile a new kernel from source.
622 manual page and the handbook are good starting points for learning how to
624 Generally the first thing you do when creating your own custom
625 kernel is to strip out all the drivers and services you do not use.
628 and drivers you do not have will reduce the size of your kernel, sometimes
629 by a megabyte or more, leaving more memory available for applications.
632 may be used to reduce system boot times.
633 The defaults are fairly high and
634 can be responsible for 5+ seconds of delay in the boot process.
637 to something below 5 seconds could work (especially with modern drives).
639 There are a number of
641 options that can be commented out.
642 If you only want the kernel to run
643 on a Pentium class CPU, you can easily remove
647 if you are sure your CPU is being recognized as a Pentium II or better.
648 Some clones may be recognized as a Pentium or even a 486 and not be able
649 to boot without those options.
652 will be able to better use higher-end CPU features for MMU, task switching,
653 timebase, and even device operations.
654 Additionally, higher-end CPUs support
655 4MB MMU pages, which the kernel uses to map the kernel itself into memory,
656 increasing its efficiency under heavy syscall loads.
657 .Sh CPU, MEMORY, DISK, NETWORK
658 The type of tuning you do depends heavily on where your system begins to
659 bottleneck as load increases.
660 If your system runs out of CPU (idle times
661 are perpetually 0%) then you need to consider upgrading the CPU
662 or perhaps you need to revisit the
663 programs that are causing the load and try to optimize them.
665 is paging to swap a lot you need to consider adding more memory.
667 system is saturating the disk you typically see high CPU idle times and
668 total disk saturation.
670 can be used to monitor this.
671 There are many solutions to saturated disks:
672 increasing memory for caching, mirroring disks, distributing operations across
673 several machines, and so forth.
675 Finally, you might run out of network suds.
676 Optimize the network path
680 we describe a firewall protecting internal hosts with a topology where
681 the externally visible hosts are not routed through it.
682 Most bottlenecks occur at the WAN link.
683 If expanding the link is not an option it may be possible to use the
685 feature to implement peak shaving or other forms of traffic shaping to
686 prevent the overloaded service (such as web services) from affecting other
687 services (such as email), or vice versa.
688 In home installations this could
689 be used to give interactive traffic (your browser,
692 over services you export from your box (web services, email).
726 manual page was originally written by
732 The manual page was greatly modified by
733 .An Eitan Adler Aq Mt eadler@FreeBSD.org .