1 .\" Copyright (C) 2001 Matthew Dillon. All rights reserved.
2 .\" Copyright (C) 2012 Eitan Adler.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
14 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16 .\" ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
17 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
30 .Nd performance tuning under FreeBSD
31 .Sh SYSTEM SETUP - DISKLABEL, NEWFS, TUNEFS, SWAP
32 The swap partition should typically be approximately 2x the size of
34 for systems with less than 4GB of RAM, or approximately equal to
35 the size of main memory
37 Keep in mind future memory
38 expansion when sizing the swap partition.
39 Configuring too little swap can lead
40 to inefficiencies in the VM page scanning code as well as create issues
41 later on if you add more memory to your machine.
43 with multiple disks, configure swap on each drive.
44 The swap partitions on the drives should be approximately the same size.
45 The kernel can handle arbitrary sizes but
46 internal data structures scale to 4 times the largest swap partition.
48 the swap partitions near the same size will allow the kernel to optimally
49 stripe swap space across the N disks.
50 Do not worry about overdoing it a
51 little, swap space is the saving grace of
53 and even if you do not normally use much swap, it can give you more time to
54 recover from a runaway program before being forced to reboot.
56 It is not a good idea to make one large partition.
58 each partition has different operational characteristics and separating them
59 allows the file system to tune itself to those characteristics.
63 partitions are read-mostly, with very little writing, while
64 a lot of reading and writing could occur in
67 partitioning your system fragmentation introduced in the smaller more
68 heavily write-loaded partitions will not bleed over into the mostly-read
71 Properly partitioning your system also allows you to tune
78 option worthwhile turning on is
81 .Dq Li "tunefs -n enable /filesystem" .
82 Softupdates drastically improves meta-data performance, mainly file
83 creation and deletion.
84 We recommend enabling softupdates on most file systems; however, there
85 are two limitations to softupdates that you should be aware of when
86 determining whether to use it on a file system.
87 First, softupdates guarantees file system consistency in the
88 case of a crash but could very easily be several seconds (even a minute!\&)
89 behind on pending write to the physical disk.
90 If you crash you may lose more work
92 Secondly, softupdates delays the freeing of file system
94 If you have a file system (such as the root file system) which is
95 close to full, doing a major update of it, e.g.,\&
96 .Dq Li "make installworld" ,
97 can run it out of space and cause the update to fail.
98 For this reason, softupdates will not be enabled on the root file system
99 during a typical install.
100 There is no loss of performance since the root
101 file system is rarely written to.
105 options exist that can help you tune the system.
106 The most obvious and most dangerous one is
108 Only use this option in conjunction with
110 as it is far too dangerous on a normal file system.
111 A less dangerous and more
117 file systems normally update the last-accessed time of a file or
118 directory whenever it is accessed.
119 This operation is handled in
121 with a delayed write and normally does not create a burden on the system.
122 However, if your system is accessing a huge number of files on a continuing
123 basis the buffer cache can wind up getting polluted with atime updates,
124 creating a burden on the system.
125 For example, if you are running a heavily
126 loaded web site, or a news server with lots of readers, you might want to
127 consider turning off atime updates on your larger partitions with this
130 However, you should not gratuitously turn off atime
134 file system customarily
135 holds mailboxes, and atime (in combination with mtime) is used to
136 determine whether a mailbox has new mail.
137 You might as well leave
138 atime turned on for mostly read-only partitions such as
143 This is especially useful for
145 since some system utilities
146 use the atime field for reporting.
148 In larger systems you can stripe partitions from several drives together
149 to create a much larger overall partition.
150 Striping can also improve
151 the performance of a file system by splitting I/O operations across two
158 utilities may be used to create simple striped file systems.
160 speaking, striping smaller partitions such as the root and
162 or essentially read-only partitions such as
164 is a complete waste of time.
165 You should only stripe partitions that require serious I/O performance,
168 or custom partitions used to hold databases and web pages.
169 Choosing the proper stripe size is also
171 File systems tend to store meta-data on power-of-2 boundaries
172 and you usually want to reduce seeking rather than increase seeking.
174 means you want to use a large off-center stripe size such as 1152 sectors
175 so sequential I/O does not seek both disks and so meta-data is distributed
176 across both disks rather than concentrated on a single disk.
179 variables permit system behavior to be monitored and controlled at
181 Some sysctls simply report on the behavior of the system; others allow
182 the system behavior to be modified;
183 some may be set at boot time using
185 but most will be set via
187 There are several hundred sysctls in the system, including many that appear
188 to be candidates for tuning but actually are not.
189 In this document we will only cover the ones that have the greatest effect
194 sysctl defines the overcommit behaviour of the vm subsystem.
195 The virtual memory system always does accounting of the swap space
196 reservation, both total for system and per-user.
198 are available through sysctl
200 that gives the total bytes available for swapping, and
201 .Va vm.swap_reserved ,
202 that gives number of bytes that may be needed to back all currently
203 allocated anonymous memory.
207 sysctl causes the virtual memory system to return failure
208 to the process when allocation of memory causes
212 Bit 1 of the sysctl enforces
217 Root is exempt from this limit.
218 Bit 2 allows to count most of the physical
219 memory as allocatable, except wired and free reserved pages
221 .Va vm.stats.vm.v_free_target
223 .Va vm.stats.vm.v_wire_count
224 sysctls, respectively).
227 .Va kern.ipc.maxpipekva
228 loader tunable is used to set a hard limit on the
229 amount of kernel address space allocated to mapping of pipe buffers.
230 Use of the mapping allows the kernel to eliminate a copy of the
231 data from writer address space into the kernel, directly copying
232 the content of mapped buffer to the reader.
233 Increasing this value to a higher setting, such as `25165824' might
234 improve performance on systems where space for mapping pipe buffers
235 is quickly exhausted.
236 This exhaustion is not fatal; however, and it will only cause pipes
237 to fall back to using double-copy.
240 .Va kern.ipc.shm_use_phys
241 sysctl defaults to 0 (off) and may be set to 0 (off) or 1 (on).
243 this parameter to 1 will cause all System V shared memory segments to be
244 mapped to unpageable physical RAM.
245 This feature only has an effect if you
246 are either (A) mapping small amounts of shared memory across many (hundreds)
247 of processes, or (B) mapping large amounts of shared memory across any
249 This feature allows the kernel to remove a great deal
250 of internal memory management page-tracking overhead at the cost of wiring
251 the shared memory into core, making it unswappable.
254 .Va vfs.vmiodirenable
255 sysctl defaults to 1 (on).
256 This parameter controls how directories are cached
258 Most directories are small and use but a single fragment
259 (typically 2K) in the file system and even less (typically 512 bytes) in
261 However, when operating in the default mode the buffer
262 cache will only cache a fixed number of directories even if you have a huge
264 Turning on this sysctl allows the buffer cache to use
265 the VM Page Cache to cache the directories.
266 The advantage is that all of
267 memory is now available for caching directories.
268 The disadvantage is that
269 the minimum in-core memory used to cache a directory is the physical page
270 size (typically 4K) rather than 512 bytes.
271 We recommend turning this option off in memory-constrained environments;
272 however, when on, it will substantially improve the performance of services
273 that manipulate a large number of files.
274 Such services can include web caches, large mail systems, and news systems.
275 Turning on this option will generally not reduce performance even with the
276 wasted memory but you should experiment to find out.
280 sysctl defaults to 1 (on).
281 This tells the file system to issue media
282 writes as full clusters are collected, which typically occurs when writing
283 large sequential files.
284 The idea is to avoid saturating the buffer
285 cache with dirty buffers when it would not benefit I/O performance.
287 this may stall processes and under certain circumstances you may wish to turn
291 .Va vfs.hirunningspace
292 sysctl determines how much outstanding write I/O may be queued to
293 disk controllers system-wide at any given time.
294 It is used by the UFS file system.
295 The default is self-tuned and
296 usually sufficient but on machines with advanced controllers and lots
297 of disks this may be tuned up to match what the controllers buffer.
298 Configuring this setting to match tagged queuing capabilities of
299 controllers or drives with average IO size used in production works
300 best (for example: 16 MiB will use 128 tags with IO requests of 128 KiB).
301 Note that setting too high a value
302 (exceeding the buffer cache's write threshold) can lead to extremely
303 bad clustering performance.
304 Do not set this value arbitrarily high!
305 Higher write queuing values may also add latency to reads occurring at
310 sysctl governs VFS read-ahead and is expressed as the number of blocks
311 to pre-read if the heuristics algorithm decides that the reads are
313 It is used by the UFS, ext2fs and msdosfs file systems.
314 With the default UFS block size of 32 KiB, a setting of 64 will allow
315 speculatively reading up to 2 MiB.
316 This setting may be increased to get around disk I/O latencies, especially
317 where these latencies are large such as in virtual machine emulated
319 It may be tuned down in specific cases where the I/O load is such that
320 read-ahead adversely affects performance or where system memory is really
325 sysctl defines how large VFS namecache may grow.
326 The number of currently allocated entries in namecache is provided by
328 sysctl and the condition
329 debug.numcache < kern.maxvnodes * vfs.ncsizefactor
334 sysctl defines how many negative entries VFS namecache is allowed to create.
335 The number of currently allocated negative entries is provided by
337 sysctl and the condition
338 vfs.ncnegfactor * debug.numneg < debug.numcache
341 There are various other buffer-cache and VM page cache related sysctls.
342 We do not recommend modifying these values.
345 .Va net.inet.tcp.sendspace
347 .Va net.inet.tcp.recvspace
348 sysctls are of particular interest if you are running network intensive
350 They control the amount of send and receive buffer space
351 allowed for any given TCP connection.
352 The default sending buffer is 32K; the default receiving buffer
355 improve bandwidth utilization by increasing the default at the cost of
356 eating up more kernel memory for each connection.
358 increasing the defaults if you are serving hundreds or thousands of
359 simultaneous connections because it is possible to quickly run the system
360 out of memory due to stalled connections building up.
362 high bandwidth over a fewer number of connections, especially if you have
363 gigabit Ethernet, increasing these defaults can make a huge difference.
364 You can adjust the buffer size for incoming and outgoing data separately.
365 For example, if your machine is primarily doing web serving you may want
366 to decrease the recvspace in order to be able to increase the
367 sendspace without eating too much kernel memory.
368 Note that the routing table (see
370 can be used to introduce route-specific send and receive buffer size
373 As an additional management tool you can use pipes in your
376 to limit the bandwidth going to or from particular IP blocks or ports.
377 For example, if you have a T1 you might want to limit your web traffic
378 to 70% of the T1's bandwidth in order to leave the remainder available
379 for mail and interactive use.
380 Normally a heavily loaded web server
381 will not introduce significant latencies into other services even if
382 the network link is maxed out, but enforcing a limit can smooth things
383 out and lead to longer term stability.
384 Many people also enforce artificial
385 bandwidth limitations in order to ensure that they are not charged for
386 using too much bandwidth.
388 Setting the send or receive TCP buffer to values larger than 65535 will result
389 in a marginal performance improvement unless both hosts support the window
390 scaling extension of the TCP protocol, which is controlled by the
391 .Va net.inet.tcp.rfc1323
393 These extensions should be enabled and the TCP buffer size should be set
394 to a value larger than 65536 in order to obtain good performance from
395 certain types of network links; specifically, gigabit WAN links and
396 high-latency satellite links.
397 RFC1323 support is enabled by default.
400 .Va net.inet.tcp.always_keepalive
401 sysctl determines whether or not the TCP implementation should attempt
402 to detect dead TCP connections by intermittently delivering
405 By default, this is enabled for all applications; by setting this
406 sysctl to 0, only applications that specifically request keepalives
408 In most environments, TCP keepalives will improve the management of
409 system state by expiring dead TCP connections, particularly for
410 systems serving dialup users who may not always terminate individual
411 TCP connections before disconnecting from the network.
412 However, in some environments, temporary network outages may be
413 incorrectly identified as dead sessions, resulting in unexpectedly
414 terminated TCP connections.
415 In such environments, setting the sysctl to 0 may reduce the occurrence of
416 TCP session disconnections.
419 .Va net.inet.tcp.delayed_ack
420 TCP feature is largely misunderstood.
421 Historically speaking, this feature
422 was designed to allow the acknowledgement to transmitted data to be returned
423 along with the response.
424 For example, when you type over a remote shell,
425 the acknowledgement to the character you send can be returned along with the
426 data representing the echo of the character.
427 With delayed acks turned off,
428 the acknowledgement may be sent in its own packet, before the remote service
429 has a chance to echo the data it just received.
430 This same concept also
431 applies to any interactive protocol (e.g.,\& SMTP, WWW, POP3), and can cut the
432 number of tiny packets flowing across the network in half.
435 delayed ACK implementation also follows the TCP protocol rule that
436 at least every other packet be acknowledged even if the standard 40ms
437 timeout has not yet passed.
438 Normally the worst a delayed ACK can do is
439 slightly delay the teardown of a connection, or slightly delay the ramp-up
440 of a slow-start TCP connection.
441 While we are not sure we believe that
442 the several FAQs related to packages such as SAMBA and SQUID which advise
443 turning off delayed acks may be referring to the slow-start issue.
446 .Va net.inet.ip.portrange.*
447 sysctls control the port number ranges automatically bound to TCP and UDP
449 There are three ranges: a low range, a default range, and a
450 high range, selectable via the
455 network programs use the default range which is controlled by
456 .Va net.inet.ip.portrange.first
458 .Va net.inet.ip.portrange.last ,
459 which default to 49152 and 65535, respectively.
460 Bound port ranges are
461 used for outgoing connections, and it is possible to run the system out
462 of ports under certain circumstances.
463 This most commonly occurs when you are
464 running a heavily loaded web proxy.
465 The port range is not an issue
466 when running a server which handles mainly incoming connections, such as a
467 normal web server, or has a limited number of outgoing connections, such
469 For situations where you may run out of ports,
470 we recommend decreasing
471 .Va net.inet.ip.portrange.first
473 A range of 10000 to 30000 ports may be reasonable.
474 You should also consider firewall effects when changing the port range.
476 may block large ranges of ports (usually low-numbered ports) and expect systems
477 to use higher ranges of ports for outgoing connections.
479 .Va net.inet.ip.portrange.last
480 is set at the maximum allowable port number.
483 .Va kern.ipc.soacceptqueue
484 sysctl limits the size of the listen queue for accepting new TCP connections.
485 The default value of 128 is typically too low for robust handling of new
486 connections in a heavily loaded web server environment.
487 For such environments,
488 we recommend increasing this value to 1024 or higher.
490 may itself limit the listen queue size (e.g.,\&
493 often have a directive in its configuration file to adjust the queue size up.
494 Larger listen queues also do a better job of fending off denial of service
499 sysctl determines how many open files the system supports.
501 typically a few thousand but you may need to bump this up to ten or twenty
502 thousand if you are running databases or large descriptor-heavy daemons.
505 sysctl may be interrogated to determine the current number of open files
509 .Va vm.swap_idle_enabled
510 sysctl is useful in large multi-user systems where you have lots of users
511 entering and leaving the system and lots of idle processes.
513 tend to generate a great deal of continuous pressure on free memory reserves.
514 Turning this feature on and adjusting the swapout hysteresis (in idle
516 .Va vm.swap_idle_threshold1
518 .Va vm.swap_idle_threshold2
519 allows you to depress the priority of pages associated with idle processes
520 more quickly then the normal pageout algorithm.
521 This gives a helping hand
522 to the pageout daemon.
523 Do not turn this option on unless you need it,
524 because the tradeoff you are making is to essentially pre-page memory sooner
525 rather than later, eating more swap and disk bandwidth.
527 this option will have a detrimental effect but in a large system that is
528 already doing moderate paging this option allows the VM system to stage
529 whole processes into and out of memory more easily.
531 Some aspects of the system behavior may not be tunable at runtime because
532 memory allocations they perform must occur early in the boot process.
533 To change loader tunables, you must set their values in
535 and reboot the system.
538 controls the scaling of a number of static system tables, including defaults
539 for the maximum number of open files, sizing of network memory resources, etc.
541 is automatically sized at boot based on the amount of memory available in
542 the system, and may be determined at run-time by inspecting the value of the
551 tunables set the default soft limits for process data and stack size
553 Processes may increase these up to the hard limits by calling
560 tunables set the hard limits for process data, stack, and text size
561 respectively; processes may not exceed these limits.
564 tunable controls how much the stack segment will grow when a process
565 needs to allocate more stack.
567 .Va kern.ipc.nmbclusters
568 may be adjusted to increase the number of network mbufs the system is
570 Each cluster represents approximately 2K of memory,
571 so a value of 1024 represents 2M of kernel memory reserved for network
573 You can do a simple calculation to figure out how many you need.
574 If you have a web server which maxes out at 1000 simultaneous connections,
575 and each connection eats a 16K receive and 16K send buffer, you need
576 approximately 32MB worth of network buffers to deal with it.
578 thumb is to multiply by 2, so 32MBx2 = 64MB/2K = 32768.
580 you would want to set
581 .Va kern.ipc.nmbclusters
583 We recommend values between
584 1024 and 4096 for machines with moderates amount of memory, and between 4096
585 and 32768 for machines with greater amounts of memory.
586 Under no circumstances
587 should you specify an arbitrarily high value for this parameter, it could
588 lead to a boot-time crash.
593 may be used to observe network cluster use.
595 More and more programs are using the
597 system call to transmit files over the network.
600 sysctl controls the number of file system buffers
602 is allowed to use to perform its work.
603 This parameter nominally scales
606 so you should not need to modify this parameter except under extreme
612 manual page for details.
613 .Sh KERNEL CONFIG TUNING
614 There are a number of kernel options that you may have to fiddle with in
615 a large-scale system.
616 In order to change these options you need to be
617 able to compile a new kernel from source.
620 manual page and the handbook are good starting points for learning how to
622 Generally the first thing you do when creating your own custom
623 kernel is to strip out all the drivers and services you do not use.
626 and drivers you do not have will reduce the size of your kernel, sometimes
627 by a megabyte or more, leaving more memory available for applications.
630 may be used to reduce system boot times.
631 The defaults are fairly high and
632 can be responsible for 5+ seconds of delay in the boot process.
635 to something below 5 seconds could work (especially with modern drives).
637 There are a number of
639 options that can be commented out.
640 If you only want the kernel to run
641 on a Pentium class CPU, you can easily remove
645 if you are sure your CPU is being recognized as a Pentium II or better.
646 Some clones may be recognized as a Pentium or even a 486 and not be able
647 to boot without those options.
650 will be able to better use higher-end CPU features for MMU, task switching,
651 timebase, and even device operations.
652 Additionally, higher-end CPUs support
653 4MB MMU pages, which the kernel uses to map the kernel itself into memory,
654 increasing its efficiency under heavy syscall loads.
655 .Sh CPU, MEMORY, DISK, NETWORK
656 The type of tuning you do depends heavily on where your system begins to
657 bottleneck as load increases.
658 If your system runs out of CPU (idle times
659 are perpetually 0%) then you need to consider upgrading the CPU
660 or perhaps you need to revisit the
661 programs that are causing the load and try to optimize them.
663 is paging to swap a lot you need to consider adding more memory.
665 system is saturating the disk you typically see high CPU idle times and
666 total disk saturation.
668 can be used to monitor this.
669 There are many solutions to saturated disks:
670 increasing memory for caching, mirroring disks, distributing operations across
671 several machines, and so forth.
673 Finally, you might run out of network suds.
674 Optimize the network path
678 we describe a firewall protecting internal hosts with a topology where
679 the externally visible hosts are not routed through it.
680 Most bottlenecks occur at the WAN link.
681 If expanding the link is not an option it may be possible to use the
683 feature to implement peak shaving or other forms of traffic shaping to
684 prevent the overloaded service (such as web services) from affecting other
685 services (such as email), or vice versa.
686 In home installations this could
687 be used to give interactive traffic (your browser,
690 over services you export from your box (web services, email).
724 manual page was originally written by
730 The manual page was greatly modified by
731 .An Eitan Adler Aq Mt eadler@FreeBSD.org .