ibtracer 1/11/05 Description: ibtracer is used to build a source route into a UD packet and validate the path taken to a destination. It is based on a client/server architecture and relies on a special ibtracer IB agent in each node along the way. It can deal with switches which do not currently run this agent but validation for that part of the path is impossible. Syntax: ibtracer [-I mthca0] [-p port] [-r <# retries>] [-t ] \ [-l LID] [-g DGID] [-v] Architecture: IBA 1.2 defines a new set of vendor specific MADs which include OUI. OpenIB will use one of these classes (0x30) to implement ibtracer (and the vendor MAD option of ibping). Note that these are general service MADs and rely on the network being up. If the network is not up, then DR SMPs must be used. There are a number of separate SMP tools for this. The OpenIB vendor specific MAD agent will support the following attributes: ClassPortInfo (0x0001) and SourceRoute (0x0010). There may be an additional attribute (TBD) to support ibping but this can be done out of the same agent. Only the VendorGet method needs to be supported by this agent. No traps are currently defined for this class. Although from the ibtracer client perspective, these vendor MADs are sent on outgoing ports, it is the server (agent) which needs to validate the incoming port. As a result of this, it is the expected incoming port at the next hop which needs to be added to the SourceRoute attribute. SourceRoute requests (Gets) and responses (GetResps) are exchanged directly between the source node where the ibtracer command is initiated and each hop along the way to the destination until the destination is reached. As the hops to the destination are walked, the incoming ports are added to the SourceRoute attribute and checked when the packet is received by that hop that it did arrive on that port. If it did not arrive on that port, an error is indicated in the status field (status 7). In either case, the port it did arrive on is put in the SourceRoute attribute in the GetResp. One of DLID or DGID must be specified in the ibtracer invocation. If DGID is specified, a PathRecord request is made to the SA to obtain the DLID. Other than that, no SM or SA is involved with ibtracer although the SM is needed to set up the forwarding tables. Once the DLID is obtained from either the command invoication or the SA, a DR SMP packet is sent to the next hop to obtain the PortInfo attribute to obtain the base LID for the next hop. A VendorGet(ClassPortInfo) is then attempted to see if this management class is supported on that node. If it is (a VendorGetResp is received), a VendorGet(SourceRoute) to the next hop LID is attempted after updating the source route attribute with the local port number from the returned PortInfo attribute. Upon receipt of the VendorGet(SourceRoute), the receiving agent validates the port number it is received on with the port number in the SourceRoute attribute. It indicates failure when they do not match and in either case the port is was received on is put back into the VendorGetResp(SourceRoute). If the next hop does not support this management class, this is indicated (if -v is enabled) and the algorithm proceeds with the next hop. The algorithm is terminated when the next hop LID is the DLID (factoring in the LMC). Note that rather than doing much of this with DR SMPs directly, these could be SA requests (using PortInfoRecords and LinkRecords, or TraceRecords). Investigation would need to be done to validate whether these SA attributes are supported by the various SMs (although OpenSM is most important in terms of OpenIB). TraceRecords are optional and are not believed to be currently supported. It can be done with just PortInfoRecords and LinkRecords. Since vendor MADs are UD, there is a retransmission strategy (timeout/retry) which have defaults but are settable on the command line. -v option displays entire path. Note that the incoming (rather than outgoing) ports are displayed. Without -v specified, just success or failure is displayed. Reversible paths are used for the responses. Note that the path from A to B might not be the same from B to A so ibtracer needs to be initiated at both ends if this is of interest. This tool cannot currently be used for multicast tracing. There are a couple of reasons for this. Base switch port 0 does not support multicast and it is not a requirement of enhanced switch port 0 to support this so there would be more hop skipping. Also, the attribute format would need to be enhanced for this as well as the client needing to handle multiple responses to a single request. SourceRoute attribute format Actual Incoming Port Number (valid on response) - 1 byte Current Hop Count - 1 byte Vector of Incoming Port Numbers (0-63) - 64 bytes Outstanding Questions Should SL be supported rather than assume SL 0 ? Should GRH be supported (and tied to GID specification in command invocation) ? Is multicast tracing important ?