Diagnostic Tools shaharf@voltaire.com, halr@voltaire.com General: Model of operation: All utilities use direct MAD access to perform their operations. Operations that require QP 0 mads only, may use direct routed mads, and therefore may work even in unconfigured subnets. Almost all utilities can operate without accessing the SM, unless GUID to lid translation is required. Dependencies: Most utilities depend on libibmad and libibumad. All utilities depend on the ib_umad kernel module. Multiple port/Multiple CA support: when no IB device or port is specified (see the "local umad parameters" below), the libibumad library selects the port to use by the following criteria: 1. the first port that is ACTIVE. 2. if not found, the first port that is UP (physical link up). If a port and/or CA name is specified, the libibumad library attempts to fulfill the user request, and will fail if it is not possible. For example: ibaddr # use the 'best port' ibaddr -C mthca1 # pick the best port from mthca1 only. ibaddr -P 2 # use the second (active/up) port from the first available IB device. ibaddr -C mthca0 -P 2 # use the specified port only. Common options & flags: Most diagnostics take the following flags. The exact list of supported flags per utility can be found in the usage message and can be shown using util_name -h syntax. # Debugging flags -d raise the IB debugging level. May be used several times (-ddd or -d -d -d). -e show umad send receive errors (timeouts and others) -h show the usage message -v increase the application verbosity level. May be used several times (-vv or -v -v -v) -V show the internal version info. # Addressing flags -D use directed path address arguments. The path is a comma separated list of out ports. Examples: "0" # self port "0,1,2,1,4" # out via port 1, then 2, ... -G use GUID address arguments. In most cases, it is the Port GUID. Examples: "0x08f1040023" -s use 'smlid' as the target lid for SA queries. # Local umad parameters: -C use the specified ca_name. -P use the specified ca_port. -t override the default timeout for the solicited mads. CLI notation: all utilities use the POSIX style notation, meaning that all options (flags) must precede all arguments (parameters). Utilities descriptions: 1. ibstatus Description: ibstatus is a script which displays basic information obtained from the local IB driver. Output includes LID, SMLID, port state, link width active, and port physical state. Syntax: ibstatus [-h] [devname[:port]]... Examples: ibstatus # display status of all IB ports ibstatus mthca1 # status of mthca1 ports ibstatus mthca1:1 mthca0:2 # show status of specified ports See also: ibstat 2. ibstat Description: Similar to the ibstatus utility but implemented as a binary and not a script. It has options to list CAs and/or ports. Syntax: ibstat [-d(ebug) -l(ist_of_cas) -p(ort_list) -s(hort)] [portnum] Examples: ibstat # display status of all IB ports ibstat mthca1 # status of mthca1 ports ibstat mthca1 2 # show status of specified ports ibstat -p mthca0 # list the port guids of mthca0 ibstat -l # list all CA names See also: ibstatus 3. ibroute Description: ibroute uses SMPs to display the forwarding tables (unicast (LinearForwardingTable or LFT) or multicast (MulticastForwardingTable or MFT)) for the specified switch LID and the optional lid (mlid) range. The default range is all valid entries in the range 1...FDBTop. Syntax: ibroute [options] [ []]] Non standard flags: -a show all lids in range, even invalid entries. -n do not try to resolve destinations. -M show multicast forwarding tables. In this case the range parameters are specifying mlid range. Examples: ibroute 2 # dump all valid entries of switch lid 2 ibroute 2 15 # dump entries in the range 15...FDBTop. ibroute -a 2 10 20 # dump all entries in the range 10..20 ibroute -n 2 # simple format ibroute -M 2 # show multicast tables See also: ibtracert 4. ibtracert Description: ibtracert uses SMPs to trace the path from a source GID/LID to a destination GID/LID. Each hop along the path is displayed until the destination is reached or a hop does not respond. By using the -m option, multicast path tracing can be performed between source and destination nodes. Syntax: ibtracert [options] Non standard flags: -n simple format; don't show additional information. -m show the multicast trace of the specified mlid. Examples: ibtracert 2 23 # show trace between lid 2 and 23 ibtracert -m 0xc000 3 5 # show multicast trace between lid 3 and 5 for mcast lid 0xc000. 5. smpquery Description: smpquery allows a basic subset of standard SMP queries including the following: node info, node description, switch info, port info. Fields are displayed in human readable format. Syntax: smpquery [options] [op_params] Current supported operations and their parameters: nodeinfo nodedesc portinfo [] # default port is zero switchinfo pkeys [] sl2vl [] vlarb [] Examples: smpquery nodeinfo 2 # show nodeinfo for lid 2 smpquery portinfo 2 5 # show portinfo for lid 2 port 5 6. smpdump Description: smpdump is a general purpose SMP utility which gets SM attributes from a specified SMA. The result is dumped in hex by default. Syntax: smpdump [options] [mod] Non standard flags: -s show output as string Examples: smpdump -D 0,1,2 0x15 2 # port info, port 2 smpdump 3 0x15 2 # port info, lid 3 port 2 7. ibaddr Description: ibaddr can be used to show the lid and GID addresses of the specified port, or the local port by default. Note: this utility can be used as simple address resolver. Syntax: ibaddr [options] [] Examples: ibaddr # show local address ibaddr 2 # show address of the specified port lid ibaddr -G 0x8f1040023 # show address of the specified port guid 8. sminfo Description: sminfo issue and dumps the output of a sminfo query in human readable format. The target SM is the one listed in the local port info, or the SM specified by the optional SM lid or by the SM direct routed path. Note: using sminfo for any purposes other then simple query may be very dangerous, and may result in a malfunction of the target SM. Syntax: sminfo [options] [sminfo_modifier] Non standard flags: -s # use the specified state in sminfo mad -p # use the specified priority in sminfo mad -a # use the specified activity in sminfo mad Examples: sminfo # show sminfo of SM listed in local portinfo sminfo 2 # query SM on port lid 2 9. perfquery Description: perfquery uses PerfMgt GMPs to obtain the PortCounters (basic performance and error counters) from the PMA at the node specified. Optionally show aggregated counters for all ports of node. Also, optionally, reset after read, or only reset counters. Syntax: perfquery [options] [ [[port] [reset_mask]]] Non standard flags: -a show aggregated counters for all ports of the destination lid. -r reset counters after read. -R only reset counters. Examples: perfquery # read local port's performance counters perfquery 32 1 # read performance counters from lid 32, port 1 perfquery -a 32 # read node aggregated performance counters perfquery -r 32 1 # read performance counters and reset perfquery -R 32 1 # reset performance counters of port 1 only perfquery -R -a 32 # reset performance counters of all ports perfquery -R 32 2 0xf000 # reset only non-error counters of port 2 10. ibping Description: ibping uses vendor mads to validate connectivity between IB nodes. On exit, (IP) ping like output is show. ibping is run as client/server. Default is to run as client. Note also that a default ping server is implemented within the kernel. Syntax: ibping [options] Non standard flags: -c stop after count packets -f flood destination: send packets back to back w/o delay -o use specified OUI number to multiplex vendor mads -S start in server mode (do not return) 11. ibnetdiscover Description: ibnetdiscover performs IB subnet discovery and outputs a human readable topology file. GUIDs, node types, and port numbers are displayed as well as port LIDs and NodeDescriptions. All nodes (and links) are displayed (full topology). Optionally this utility can be used to list the current connected nodes. The output is printed to the standard output unless a topology file is specified. Syntax: ibnetdiscover [options] [] Non standard flags: -l List of connected nodes -H List of connected HCAs -S List of connected switches -g Grouping 12. ibhosts Description: ibhosts either walks the IB subnet topology or uses an already saved topology file and extracts the CA nodes. Syntax: ibhosts [-h] [] Dependencies: ibnetdiscover, ibnetdiscover format 13. ibswitches Description: ibswitches either walks the IB subnet topology or uses an already saved topology file and extracts the IB switches. Syntax: ibswitches [-h] [] Dependencies: ibnetdiscover, ibnetdiscover format 14. ibchecknet Description: ibchecknet uses a full topology file that was created by ibnetdiscover, scans the network to validate the connectivity and reports errors (from port counters). Syntax: ibchecknet [-h] [] Dependencies: ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs 15. ibcheckport Description: Check connectivity and do some simple sanity checks for the specified port. Port address is lid unless -G option is used to specify a GUID address. Syntax: ibcheckport [-h] [-G] Example: ibcheckport 2 3 # check lid 2 port 3 Dependencies: smpquery, smpquery output format, ibaddr 16. ibchecknode Description: Check connectivity and do some simple sanity checks for the specified node. Port address is lid unless -G option is used to specify a GUID address. Syntax: ibchecknode [-h] [-G] Example: ibchecknode 2 # check node via lid 2 Dependencies: smpquery, smpquery output format, ibaddr Usage: 17. ibcheckerrs Description: Check specified port (or node) and report errors that surpassed their predefined threshold. Port address is lid unless -G option is used to specify a GUID address. The predefined thresholds can be dumped using the -s option, and a user defined threshold_file (using the same format as the dump) can be specified using the -t option. Syntax: ibcheckerrs [-h] [-G] [-t ] [-s(how_thresholds)] [] Examples: ibcheckerrs 2 # check aggregated node counter for lid 2 ibcheckerrs 2 4 # check port counters for lid 2 port 4 ibcheckerrs -t xxx 2 # check node using xxx threshold file Dependencies: perfquery, perfquery output format, ibaddr 18. ibportstate Description: ibportstate allows the port state and port physical state of an IB port to be queried or a switch port to be disabled or enabled. Syntax: ibportstate [-d(ebug) -e(rr_show) -v(erbose) -D(irect) -G(uid) -s smlid -V(ersion) -C ca_name -P ca_port -t timeout_ms] [] supported ops: enable, disable, query Examples: ibportstate 3 1 disable # by lid ibportstate -G 0x2C9000100D051 1 enable # by guid ibportstate -D 0 1 # by direct route 19. ibcheckwidth Description: ibcheckwidth uses a full topology file that was created by ibnetdiscover, scans the network to validate the active link widths and reports any 1x links. Syntax: ibcheckwidth [-h] [] Dependencies: ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportwidth 20. ibcheckportwidth Description: Check connectivity and check the specified port for 1x link width. Port address is lid unless -G option is used to specify a GUID address. Syntax: ibcheckportwidth [-h] [-G] Example: ibcheckportwidth 2 3 # check lid 2 port 3 Dependencies: smpquery, smpquery output format, ibaddr 21. ibcheckstate Description: ibcheckstate uses a full topology file that was created by ibnetdiscover, scans the network to validate the port state and port physical state, and reports any ports which have a port state other than Active or a port physical state other than LinkUp. Syntax: ibcheckstate [-h] [] Dependencies: ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportstate 22. ibcheckportstate Description: Check connectivity and check the specified port for proper port state (Active) and port physical state (LinkUp). Port address is lid unless -G option is used to specify a GUID address. yntax: ibcheckportstate [-h] [-G] Example: ibcheckportstate 2 3 # check lid 2 port 3 Dependencies: smpquery, smpquery output format, ibaddr 23. ibcheckerrors ibcheckerrors uses a full topology file that was created by ibnetdiscover, scans the network to validate the connectivity and reports errors (from port counters). Syntax: ibnetcheckerrors [-h] [] Dependencies: ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs 24. ibdiscover.pl ibdiscover.pl uses a topology file create by ibnetdiscover and a discover.map file which the network administrator creates which indicates the nodes to be expected and a ibdiscover.topo file which is the expected connectivity and produces a new connectivity file (discover.topo.new) and outputs the changes to stdout. The network administrator can choose to replace the "old" topo file with the new one or certain changes in. The syntax of the ibdiscover.map file is: |port|"Text for node"| e.g. 8f10400410015|8|"ISR 6000"|# SW-6IB4 Voltaire port 0 lid 5 8f10403960558|2|"HCA 1"|# MT23108 InfiniHost Mellanox Technologies The syntax of the old and new topo files (ibdiscover.topo and ibdiscover.topo.new) are: ||| e.g. 10|5442ba00003080|1|8f10400410015 These topo files are produced by the ibdiscover.pl tool. Syntax: ibnetdiscover | ibdiscover.pl Dependencies: ibnetdiscover, ibnetdiscover format 25. ibnodes Description: ibnodes either walks the IB subnet topology or uses an already saved topology file and extracts the IB nodes (CAs and switches). Syntax: ibnodes [] Dependencies: ibnetdiscover, ibnetdiscover format 26. ibclearerrors Description: ibclearerrors clears the PMA error counters in PortCounters by either walking the IB subnet topology or using an already saved topology file. Syntax: ibclearerrors [-h] [] Dependencies: ibnetdiscover, ibnetdiscover format, perfquery 27. ibclearcounters Description: ibclearcounters clears the PMA port counters by either walking the IB subnet topology or using an already saved topology file. Syntax: ibclearcounters [-h] [] Dependencies: ibnetdiscover, ibnetdiscover format, perfquery 28. saquery Description: Issue some SA queries. Syntax: Usage: saquery [-h -d -P -N -L -G -s -g][] Queries node records by default -d enable debugging -P get PathRecord info -N get NodeRecord info -L Return just the Lid of the name specified -G Return just the Guid of the name specified -s Return the PortInfoRecords with isSM capability mask bit on -g get multicast group info Dependencies: OpenSM libvendor, OpenSM libopensm, libibumad 29. ibsysstat Description: ibsysstat uses vendor mads to validate connectivity between IB nodes and obtain other information about the IB node. ibsysstat is run as client/server. Default is to run as client. Syntax: ibsysstat [options] [] Non standard flags: Current supported operations: ping - verify connectivity to server (default) host - obtain host information from server cpu - obtain cpu information from server -o use specified OUI number to multiplex vendor mads -S start in server mode (do not return)