2 ''_Readme for FreeBSD X3100 Series 10GbE PCIe I/O Virtualized Server Adapter Drivers_'''
5 FreeBSD Driver for X3100 10GbE Server/Storage adapters
6 * Drivers support all X3100 10GbE adapters with FreeBSD version 7.x, 8.x and 9.x
7 * Supports both i386 and amd64 architectures
8 * Features: Jumbo frames (up to 9600),
9 LRO (Large Receive Offload),
10 TSO (TCP segmentation offload),
11 RTH (Receive Traffic Hash).
12 Also, Extended Message Signaled Interrupts (MSI-X).
18 X3110 and X3120 supports MTU up to 9600 bytes, modifiable using ifconfig command.
20 b. LRO (Large Receive Offload):
21 LRO can be enabled/disabled before loading driver.
22 Set lro_enable in vxge.conf to 1 before loading driver.
24 c. TSO (TCP Segmentation Offload)
25 TSO can be enabled/disabled before loading driver.
26 Set tso_enable in vxge.conf to 1 before loading driver.
28 d. RTH (Receive Traffic Hash)
29 Receive side steering for better scaling.
30 Set rth_enable in vxge.conf to 1 before loading driver.
33 Can be enabled on platforms which support it, resulting in noticeable
34 performance improvement.
37 Up to 17 hardware based transmit and receive data channels, with
38 multiple steering options.
41 ''X3100 & Driver configuration: vxge.conf''
43 The vxge.conf contains following attributes.
48 Enable MSI (Message Signaled Interrupts) feature in driver.
56 Enables Receive side steering for better scaling (RTH - Receive Traffic Hash)
63 Enables LRO (Large Receive Offload) feature in driver.
70 Enables TSO (TCP Segmentaton Offload) feature in driver.
77 Specifies maximum VPATH(s) configured for each device function.
79 Default: Optimized by driver
84 Change PCI function mode
85 0 - SF1_VP17 (1 function with 17 VPATHs)
86 1 - MF8_VP2 (8 functions with 2 VPATHs per function)
87 8 - MF2_VP8 (2 functions, 8 Paths/Function)
88 9 - MF4_VP4 (4 Functions, 4 Paths/Function)
89 11 - MF8P_VP2 (8 functions with 2 VPATHS per function required for DirectIO)
95 Change the default dual port mode
103 Turn on/off the inter function traffic through l2 switch
104 0 - Disallow inter function traffic
105 1 - Allow inter function traffic
109 ''bandwidth_0 - bandwidth_7''
111 Desired max receive/transmit bandwidth,in Mbps for function 0 to function 7
112 Minimum value is 100 Mbps, for 1 Gbps specify a value of 1024.
115 ''priority_0 - priority_7''
117 Desired receive/transmit priority for function 0 to function 7
122 Adaptive interrupt coalescing
132 === Installation Instructions ===
134 ''Identifying the Adapter''
136 The X3100 adapter is identified by the board ID number on the adapter.
138 Look for a label that has a barcode and a number, for example,
139 SXT0425072. The factory-burned MAC address (hardware address)
140 shows up on the board above the serial number,
141 (similar to 000CFC000449 -- 00:0C:FC:00:04:49).
144 ''Kernel Driver Source Package''
146 This package contains kernel_update.sh script which is to be used to copy driver sources to kernel path.
147 It creates vxge folder with source code in /usr/src/sys/dev and Makefile in /usr/src/sys/modules.
149 Loadable parameters can be changed by putting below lines in /boot/device.hints and set values as desired.
150 hint.vxge.0.msix_enable="1"
151 hint.vxge.0.rth_enable="1"
152 hint.vxge.0.lro_enable="1"
153 hint.vxge.0.tso_enable="1"
154 hint.vxge.0.tx_steering="1"
155 hint.vxge.0.no_of_vpath="-1"
156 hint.vxge.0.func_mode="-1"
157 hint.vxge.0.port_mode="-1"
158 hint.vxge.0.fw_upgrade="1"
159 hint.vxge.0.bandwidth_0="-1"
160 hint.vxge.0.bandwidth_1="-1"
161 hint.vxge.0.bandwidth_2="-1"
162 hint.vxge.0.bandwidth_3="-1"
163 hint.vxge.0.bandwidth_4="-1"
164 hint.vxge.0.bandwidth_5="-1"
165 hint.vxge.0.bandwidth_6="-1"
166 hint.vxge.0.bandwidth_7="-1"
167 hint.vxge.0.priority_0="-1"
168 hint.vxge.0.priority_1="-1"
169 hint.vxge.0.priority_2="-1"
170 hint.vxge.0.priority_3="-1"
171 hint.vxge.0.priority_4="-1"
172 hint.vxge.0.priority_5="-1"
173 hint.vxge.0.priority_6="-1"
174 hint.vxge.0.priority_7="-1"
175 hint.vxge.0.intr_coalesce="0"
176 hint.vxge.0.low_latency="0"
178 e.g., set hint.vxge.0.msix_enable to 0 to load driver in INTA mode.
179 Then reboot the system to add loadable parameters to kenv.
182 ''Standalone Driver Source Package''
184 vxge (FreeBSD package)
185 This directory contains FreeBSD driver sources for X3100 device(s),
186 Makefile, and X3100 Hardware Abstraction headers and sources
187 (vxgehal and include folders)
190 This directory contains the X3100 HAL sources.
191 The driver uses HAL to perform operations on the X3100 hardware.
194 The include subdirectory contains HAL header files.
196 Source code for vxge-manage tool (used to get statistics, pciconfig and
197 register dump) are included in the freebsd directory.
200 ''Building the driver''
202 The driver is distributed in the source form. Driver and installation
203 utility executables need to be built for the target platform.
205 In the directory containing Makefile for building Exar driver for FreeBSD,
209 Please unload previously installed Exar drivers before proceeding with following steps.
213 ''Loading the driver''
215 Use "kldload" to load driver module vxge.ko.
218 Run "kldstat" and find an entry for vxge kernel module to ensure driver installation
221 3 1 0xc22cc000 26000 vxge.ko
224 ''Enabling interface and assigning IP address''
226 #ifconfig <INTERFACE> <IP_ADDRESS> up
227 <INTERFACE> will be similar to vxge0, vxge1 etc. and can be
228 found by executing "ifconfig -a".
229 Neterion adapters typically have MAC addresses starting with
230 "00:0C:FC" or "00:11:25".
233 #ifconfig vxge0 10.2.2.40 up
234 Enables vxge0 interface and assigns to it the IP address 10.2.2.40.
236 vxge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
237 options=53b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4,LRO>
238 ether 00:0c:fc:00:da:47
239 inet6 fe80::20c:fcff:fe00:da47%vxge0 prefixlen 64 scopeid 0x3
240 inet 10.2.2.40 netmask 0xff000000 broadcast 17.255.255.255
241 media: Ethernet autoselect (10Gbase-SR <full-duplex>)
245 ''Disabling the interface''
247 #ifconfig <INTERFACE> down
252 ''Unloading the Driver''
256 === Performance Suggestions ===
258 Sysctl Tuning Parameters
259 #sysctl net.inet.tcp.sendspace=786432
260 #sysctl net.inet.tcp.recvspace=786432
261 #sysctl net.inet.tcp.recvbuf_max=16777216
262 #sysctl net.inet.tcp.sendbuf_max=16777216
263 #sysctl net.inet.tcp.blackhole=1
264 #sysctl net.inet.tcp.rfc1323=1
265 #sysctl net.inet.tcp.path_mtu_discovery=1
266 #sysctl net.inet.tcp.inflight.enable=0
267 #sysctl net.inet.ip.maxfragsperpacket=2147483647
268 #sysctl kern.ipc.maxsockbuf=8388608
269 #sysctl kern.ipc.nmbclusters=2147483647
270 #sysctl kern.ipc.nmbjumbop=262144
271 #sysctl kern.ipc.maxsockets=81920
272 #sysctl hw.intr_storm_threshold=9000
275 ''Usage & Troubleshooting''
277 For general information and support, please visit Neterion support website at
278 http://www.neterion.com/support/support.html
280 Make sure that the operating system identifies the X3100 adapter. Note that
281 Neterion vendor ID is 0x17D5 and X3110 and X3120 adapters can be fixed to both PCIe slots.
283 The rest of this section details troubleshooting tips and information. Some of
284 them are general and some are more specific. For online Troubleshooting tips
285 and faqs, please visit
286 http://trac.neterion.com/cgi-bin/trac.cgi/wiki/TitleIndex?anonymous
289 ''Loading the driver and initializing the device''
291 The first time FreeBSD identifies the device it stores the corresponding
292 device/vendor IDs in an enumerated tree of PCI devices. Note that Neterion
295 After cold reboot FreeBSD finds the device and tries to load the corresponding
296 driver. If it fails, try to switch the card and/or cable. And, in parallel, send
297 us the "tail" of the "/var/log/messages". We also need traces, register dump and
298 statistics(Use vxge-manage tool).
301 ''Collect information''
303 If you're reporting a problem to Neterion, please describe:
304 host(s) and adapter(s), switch (if used), software version.
309 If this is a basic connectivity issue (e.g., cannot connect, cannot ping),
310 make sure first that ARP works
311 Do you see ARPs coming through a switch (in case switch is used)?
312 Do you see frame drops at the switch when pinging?
313 Do you see frame counts increasing via statistics?
318 a. A previous driver release;
319 b. A different adapter in the same PCI slot;
320 c. A different PCI slot;
321 d. Back-to-back setup so that the switch is excluded from the equation.
322 e. To replace the cables?
323 f. To use a different PCI slot?
328 a. Have you tried to reboot the switch? Cold-reboot the host?
329 b. Make sure that the latest released driver gets loaded after the host
330 reboot, and that the 10GE interface shows up via ifconfig.
335 Do you see LED going green after everything is connected and drivers loaded?
336 How/when does the color changes?
341 Run "ifconfig -a" at the command prompt and check whether the output Looks as
342 expected. Include the output in your problem report.
344 Note for instance that "all-foxes" i.e, FF:FF:FF:FF:FF:FF MAC address could
345 explain a general connectivity issue, if that's what you see.
350 Please attach the generated log, with traces enabled.
352 Note that the driver's logging facility is configurable at compile-time. Errors
353 and traces can be compiled out on a per-component basis. The components are: HAL
354 fifos and rings, device, etc., see VXGE_COMPONENT_HAL_??? in the Makefile.
356 Here's how you enable all except data path traces:
357 CFLAGS_VXGE = -DVXGE_DEBUG_MODULE_MASK=0xffffffbf \
358 -DVXGE_DEBUG_ERR_MASK=0xffffffbf
359 Recompile with traces and include the log in the report.
366 To print hardware and software statistics for interface instance 0
368 #vxge-manage vxgeX stats common
369 #vxge-manage vxgeX stats mrpcim
370 #vxge-manage vxgeX stats driver
371 #vxge-manage vxgeX pciconfig
372 #vxge-manage vxgeX hwinfo
373 #vxge-manage vxgeX bw_pri_get
374 #vxge-manage vxgeX bw_pri_get vf_id
375 #vxge-manage vxgeX port_mode_get
377 The vxge-manage tool generates log file in the working directory. Once done, ping a
378 few times, and collect the statistics again (Ping both from this and the remote
381 Many counters could be of interest. For example, "rx_vld_frms" counts all
382 valid incoming Ethernet frames seen by the adapter. Information could be derived
383 from the fact that (for instance) counter stay constant during ping, if that is
384 what happening. For detailed description of the X3100 counters, please refer
385 to the "X3100 User Guide".
387 Please include the statistics into your problem report.
392 Use vxge-manage to dump all X3100 BAR0 space registers. Include this register dump
393 into your problem report.
394 #vxge-manage vxgeX regs
397 ''PCI configuration space''
399 Use vxge-manage to retrieve PCI vendor, device, etc. Include the PCI configutation
400 space in your problem report.
401 #vxge-manage vxgeX pciconfig
406 To retrieve hardware info of device, e.g, serial / part number and function mode etc.
408 #vxge-manage vxgeX hwinfo
411 ''Bandwidth and Priority''
413 Use vxge-manage to display Bandwidth and Priority information.
414 #vxge-manage vxgeX bw_pri_get
416 #vxge-manage vxgeX bw_pri_get vf_id
418 vxge-manage can also be used to set bandwidth and priority for individual VF.
419 #vxge-manage vxgeX bw_pri_set vf_id bandwidth
421 #vxge-manage vxgeX bw_pri_set vf_id bandwidth priority
424 #vxge-manage vxge0 bw_pri_set 0 1500
426 #vxge-manage vxge0 bw_pri_set 0 1500 1
430 Use vxge-manage to display Port mode setting
431 #vxge-manage vxgeX port_mode_get
433 vxge-manage can also be used to set Port mode.
434 #vxge-manage vxgeX port_mode_set port_mode_value
437 #vxge-manage vxge0 port_mode_set 2
442 === Available Downloads ===
444 For latest available drivers or further support please contact your network
445 adapter provider or neterionsupport@exar.com.
448 ===================================================================
449 Exar Corp., Proprietary
450 COPYRIGHT (c) 2002-2011 Exar corp., ALL RIGHTS RESERVED