5 Repo: https://openib.org/svn/trunk/contrib/mellanox/gen2/src/userspace/management/osm
11 This document describes the contents of the OpenSM IBG2 release.
12 OpenSM is an InfiniBand compliant Subnet Manager and Administrator,
13 and runs on top of OpenIB.
15 This document includes the following sections:
16 1 This Overview section (describing new features and software
18 2 Known Issues And Limitations
19 3 Unsupported IB compliancy statements
21 5 Main Verification Flows
22 6 Qualified software stacks and devices
26 * New libs created during installation: libopensm - contains interface
27 to the logging and mads pool machanism. libosmcomp - contains
28 interface to the complib utilities. libosmvendor - contains
29 interface to sending/receiving MADs through the SMI or GSI over the
32 * Change building mechanism to use autotools.
34 * Change directory stucturing of the OpenSM code according to libs:
35 osm/libvendor - for vendor specific files. osm/complib - for complib
36 specific files. osm/opensm - for opensm core files. osm/include
38 * Semi-static LID assignment: OpenSM uses a cache file for storing all
39 LID assignments such that, even after a reboot, the LIDs do not
40 change. The static LID assignment is built on top of a new
41 "persistancy" layer that abstracts that actual database from its
42 usage. The implemented database is based on files stored under
43 /var/cache/osm (this location can be overriden via the environment
44 variable OSM_CACHE_DIR). Other implementations can use LDAP for
45 example. Note that a standby SM ignores its previously assigned LIDs
46 when it becomes the master, and the previous master LID settings are
49 * Irresponsive Port Handling: A port that does not respond to SM
50 queries will be queried upon future light or heavy sweeps, and if
51 then it responds, it will be setup immediately. Previously such a
52 port was queried only upon a heavy sweep.
54 * Leaf Switch Port HOQ: A different maximal head of queue life time is
55 assigned to switch ports connected to HCAs such that a bad chipset
56 or defective hardware will not cause back presure on the fabric.
58 * OSM_TMP_DIR: This is a new environment variable controlling the
59 directory where subnet.lst, osm.fdbs and osm.mcfdbs files are
60 created. The deafult is still /tmp.
62 * Configuration Options cache file: OpenSM was enhanced to provide a
63 means to modify all its internal configuration options, including
64 the ones that oreviously were only available under osmsh. The new
65 file is located under the cache directory and is named
66 opensm.opts. To automatically create this file OpenSM supports a new
67 flag: `-c'. The file is generated with the current set of options
70 * Previously, under extreme load conditions, when OpenSM got
71 overloaded with SA queries during which the incoming messages queue
72 also grew, delays were incurred in message response-time beyond the
73 expected. This new version of OpenSM has been enhanced such that,
74 under such a case, incoming new SA queries are returned with a
75 RESOURCE_BUSY status (per the InfiniBand Architecture
78 * Kill -HUP: If the OpenSM process (ps -efww |grep opensm.bin) gets a
79 SIGHUP (sent by kill -HUP), it will start a heavy sweep as if a trap
80 was received or a change in topology was observed by the SM.
82 1.2 Software Dependencies
84 OpenSM depends on the installation of either OpenIB gen2 (e.g. IBG2
85 distribution), OpenIB gen1 (r.g. IBGD distribution) or Mellanox VAPI
86 stacks. The qualified driver versions are provided in Table 2,
87 "Qualified IB Stacks".
89 1.4 Supported Devices Firmware
91 The main task of OpenSM is to initialize InfiniBand devices. The
92 qualified devices and their corresponding firmware versions
95 2 Known Issues And Limitations
96 ------------------------------
98 * No Partition/Pkey policy support:
99 OpenSM does not provide means to set poartitions.
101 * IB "trusted" concept is unsupported:
102 Queries that should be classified according to the trustworthiness of
103 their sources will not be handled correctly.
105 * No Service / Key associations:
106 There is no way to manage Service access by Keys.
108 * No SM to SM SMDB synchronization:
109 Puts the burden of re-registering services, multicast groups, and
110 inform-info on the client application (or IB access layer core).
112 * NPTL problem under Red Hat 9.0, Red Hat AS 3.0:
113 There are some bugs (pthread conditional wait missing events)
114 with thread handling when using the dynamic Native POSIX Thread
115 Library (/lib/tls) of Red Hat 9.0 & Red Hat AS 3.0 OSs. To overcome
116 that, OpenSM installation places wrapper scripts named opensm and
117 osmtest in the /usr/bin directory, which preload the standard libc
118 and libptherad before invoking the executables. If using the osm
119 package, a similar workaround is possible by putting the LD_PRELOAD
120 setting in .tclshrc file, for example: set env (LD_PRELOAD)
121 "/lib/libc.so.6:/lib/libpthread.so.0"
123 * InformInfo failure over IBMGT:
124 OpenSM might not respect a valid InformInfo unsubscribe request when
125 running over Mellanox's IBMGT user level MAD interface (not on
126 IBGD). This will be fixed in the next release.
128 * No "port down" event handling:
129 Changing the switch port through which OpenSM connects to the IB
130 fabric may cause wrong operation. Please restart OpenSM whenever
131 such a connectivity change is made.
133 3 Unsupported IB Compliancy Statements
134 --------------------------------------
135 The following section lists all the IB compliancy statements which
136 OpenSM does not support. Please refer to IB specification for detailed
137 information on each compliancy statement.
139 * C14-22 (Authentication):
140 M_Key M_KeyProtectBits and M_KeyLeasePeriod shall be set in one
141 SubnSet method. As a work-around, an OpenSM option is provided for
142 defining the protect bits.
144 * C14-67 (Authentication):
145 On SubnGet(SMInfo) and SubnSet(SMInfo) - if M_Key is not zero then
146 the SM shall generate a SubnGetResp if the M_Key is matching or
147 silently drop the packet if M_Key is not matching.
149 * C15-0.1.23.1 (Authentication):
150 PortInfoRecords shall always be provided with the M_Key component
151 set to 0, except in the case of a trusted request, in which case the
152 actual M_Key component contents shall be provided.
154 * C15-0.1.23.2 (Authentication):
155 P_KeyTableRecords and ServiceAssociationRecords shall only be
156 provided in responses to trusted requests.
158 * C15-0.1.23.4 (Authentication):
159 InformInfoRecords shall always be provided with the QPN set to
160 0, except for the case of a trusted request, in which case the actual
161 subscriber QPN shall be returned.
163 * o13-17.1.2 (Event-FWD):
164 If no permission to forward, the subscription should be removed and
165 no further forwarding should occur.
167 * C14-37.1.2 (Handover):
168 Priority should be kept in non-volatile memory.
170 * C14-38.1.1 (Handover):
171 Support AttributeModifier values in SubnSet(SMInfo). If the state
172 transition requested is invalid - return with status code 7.
174 * C14-24.1.1.5 and C14-62.1.1.22 (Initialization):
175 GUIDInfo - SM should enable assigning Port GUIDInfo.
177 * C14-44 (Initialization):
178 If the SM discovers that it is missing an M_Key to update CA/RT/SW,
179 it should notify the higher level.
181 * C14-62.1.1.11 (Initialization):
182 PortInfo:VLHighLimit should match the configured VLArb on the port.
184 * C14-62.1.1.12 (Initialization):
185 PortInfo:M_Key - Set the M_Key to a node based random value.
187 * C14-62.1.1.13 (Initialization):
188 PortInfo:P_KeyProtectBits - set according to an optional policy.
190 * C14-62.1.1.24 (Initialization):
191 SwitchInfo:DefaultPort - should be configured for random FDB.
193 * C14-62.1.1.32 (Initialization):
194 RandomForwardingTable should be configured.
196 * o15-0.1.12 (Multicast):
197 If the JoinState is SendOnlyNonMember = 1 (only), then the endport
198 should join as sender only.
200 * o15-0.1.13 (Multicast):
201 If a Join request using unrealistic parameters is received, return
204 * o15-0.1.8 (Multicast):
205 If a request for creating an MCG with fields that cannot be met,
206 return ERR_REQ_INVALID (currently ignoring SL and FlowLabelTclass).
208 * C15-0.1.11 (SA-Query):
209 Query response should use only base LIDs (as the feature has not
212 * C15-0.1.19 (SA-Query):
213 Respond to SubnGetMulti(MultiPathRec)
215 * C15-0.1.8.6 (SA-Query):
216 Respond to SubnAdmGetTraceTable - this is an optional attribute.
218 * C15-0.1.8.7 (SA-Query):
219 SubnAdmGetMulti SubnAdmGetMultiResp - Only in case of a MultiPath.
221 * C15-X.Y.Z.W (SA-Query):
222 SubAdmGet/GetTable GUIDInfo - support GUIDInfo setting/retrieval.
224 * C15-0.1.13 Services:
225 Reject ServiceRecord create, modify or delete if the given
226 ServiceP_Key does not match the one included in the ServiceGID port
227 and the port that sent the request.
229 * C15-0.1.14 (Services):
230 Provide means to associate service name and ServiceKeys.
235 The following list of bugs were fixed. Note that other less critical
236 or visible bugs were also fixed.
238 * PortInfo query was not matching on several fields. These fields
239 were added to teh comparison function.
241 * OpenSM would crash during exit flow if run with "-o" flag A fix to
242 the complib global timer destruction sequence solves this problem.
244 * OpenSM does not complete the sweep if the driver fails to send a MAD
245 Counting the number of outstanding MADs the SM waits for response
246 for was enhanced to take this acse into acount
248 * OpenSM was not compliant to the spec statement: C14.62.1.1 Table 183
249 p870 l34: ".., the SM shall ensure that one of the P_KeyTable
250 entries in every node contains either the value 0xFFFF (the default
251 P_Key, full membership) or the value 0x7FFF (the default P_Key,
252 partial membership)." OpenSM sets the PKey table with an entry of
253 0xffff in case there is no such entry or 0x7fff entries on that
254 port. Switch ports are ignored.
256 * If the SA is queried with IB_PIR_COMPMASK_BASELID and base_lid of 0,
257 the SA was incorrectly returning all the ports. Fix: do not ignore base
258 lid of 0 as a query criteria.
260 * When provided a PathRecord query with num_paths = 0 the SM should
261 assuem num_paths = 1. Fix: in the PathRecord query code.
263 * PathRecord query returned a deleted multicast groups info. Fix:
264 Added a check for multicast group state to avoid such cases.
266 * LinkRecord query provided wrong results. Fix: in query code.
268 * PathRecord did not honor PacketLifeTime component. Fix: Added the
269 check for packet lifetime matching.
271 * Multicast and other registration hapenning all the time on the
272 cluster. Fix: OpenSM was sending false "client-re-registration"
273 messages (in PortInfo).
275 * On some heavy load cases OpenSM would consume 100% CPU time. Fix: an
276 endless loop in timer implementation that would happen under rare
277 heavy CPU load cases.
279 * OpenSM hangs during LID assignment phase. Fix: Some condition that
280 cause that was fixed.
282 * OpenSM core dump in the middle of sweep. Fix: A memory range
283 overflow write was found by valgrind and fix.
285 * OpenSM core dump as result fo PathRecord query with no results. Fix:
286 A memory free on non allocated memory was fixed.
288 * OpenSM sweep algorithm confused by a timing race. Fix: A significant
289 race conditionin the SM sweep algorithm was found and fixed.
291 * OpenSM deadlock due to out of order SMINfo and NodeInfo MAD
292 received. Fix: A fix in lock ordering resolves this issue.
294 * TrapRepress sent even if not a master. Fix: in trap receiver.
296 5 Main Verification Flows
297 -------------------------
299 OpenSM verification is run using the following activities:
300 * osmtest - a standalone program
301 * ibmgtsim (IB management simulator) based - a set of flows that
302 simulate clusters, inject errors and verify OpenSM capability to
303 respond and bring up the network correctly.
304 * small cluster regression testing - where the SM is used on back to
305 back or single switch configuration. The regression includes
306 multiple OpenSM dedicated tests
307 * cluster testing - when we run OpenSM to setup large cluster, perform
308 handoff, reboots and reconnects, verify routing correctness and SA
309 responsiveness at teh ULP level (IPoIB and SDP)
313 OsmTest is the main automated verification tool used for OpenSM
314 testing. Its verification flows are described by list below.
316 * Inventory File: Obtain and verify all port info, node info, and path
320 - Register new service
321 - Register another service (with a lease period)
322 - Register another service (with service p_key set to zero)
323 - Get all services by name
324 - Delete the first service
325 - Delete the third service.
326 - Added bad flows of get/delete non valid service
327 - Add / Get same service with different data
328 - Add / Get / Delete by different component mask values (services
329 by Name & Key / Name & Data / Name & Id / Id only )
331 * Multicast Member Record:
332 - Query of existing Groups (IPoIB)
333 - BAD Join with insufficient comp mask (o15.0.1.3)
334 - Create given MGID=0 (o15.0.1.4)
335 - Create given MGID=0xFF12A01C,FE800000,00000000,12345678 (o15.0.1.4)
336 - Create BAD MGID=0xFA. (o15.0.1.6)
337 - Create BAD MGID=0xFF12A01B w/ link-local not set (o15.0.1.6)
338 - New MGID with invalid join state (o15.0.1.9)
339 - Retry of existing MGID - See JoinState update (o15.0.1.11)
340 - BAD RATE when connecting to existing MGID (o15.0.1.13)
341 - Partial JoinState delete request - removing FullMember (o15.0.1.14)
342 - Full Delete of a group (o15.0.1.14)
343 - Verify Delete by trying to Join deleted group (o15.0.1.14)
344 - BAD Delete of IPoIB membership (no prev join) (o15.0.1.15)
346 * Event Forwarding: Register for trap forwarding using reports
347 - Send a trap and wait for report
348 - Unregister non-existing
350 * Trap 64/65 Flow: Register to Trap 64-65, create traps (by
351 disconnect/connect ports) and wait for report, then unregister.
353 * Stress Test: send PortInfoRecord queries both single and RMPP and
354 check for the rate of responses as well as their validity.
356 5.2 IB Management Simulator OpenSM Test Flows:
358 The simulator provides ability to simulate the SM handling of virtual
359 topologies that are not limitted to actual lab equipment availability.
360 OpenSM was simulated to bring up clusters of up to 10,000 nodes. Daily
361 regressions use smaller (16 and 128 nodes clusters).
363 The following test flows are running on the IB management simulator:
366 Up to 12 links from the fabric are randomly selected to drop packets
367 at drop rates up to 90%. The SM is required to succeed bringing the
368 fabric up. The reulting routing is verified to be correct too.
371 Using LMC = 2 the fabric is being initialized with LIDs. Faults like
372 zero LID, Duplicated LID, non-aligned (to LMC) LIDs are being
373 randomly assigned to various nodes and other errors are randomly
374 output to the guid2lid cache file. The SM sweep is run 5 times and
375 after each iteration a complete verification is made to ensure all
376 LIDs that could possibly be maintained are kept, as well as all nodes
377 were assigned a legal LID range.
380 Nodes are randomly joining the 0xc000 group and eventually the
381 resulting routing is verified for completness and adherance to
382 Up/Down routing rules.
385 The complete osmtest flow as desribed in previous table is run on
386 the simulated fabrics.
388 5.3 OpenSM Regression
390 Using a back to back or single switch connection the following set of
391 tests are run nightly on the stacks described in table 2. The included
394 * Stress Testing: Flood the SA with queries from multiple channel
395 adapters to check the robustness of the entire stack up to the SA.
397 * Dynamic Changes: Dynamic Topology changes, through randomlly
398 droping SMP packets used to test OpenSM adaptation to unstable
399 network & verify DB correctness.
401 * Trap Injection: This flow injects traps to the SM and verify it does
402 handle them gracefully.
404 * SA Query Test: This test exhoustivly checks the SA responses to all
405 possible single component mask. To do that the test examine the
406 entire set of records the SA can provide, classify them by their
407 field values and then select every field (using component mask and a
408 value) and verify the response matches the expected set of records.
409 A random selection using multiple component mask bits is also performed.
413 Cluster testing is usually run before a distribution release. It
414 involves real hardware setup of 16 to 32 nodes (or more if beta site
415 is available). Each test is validated by running all-to-all ping through IB
416 interface. The test procedure includes:
420 * Handoff between 2 or 3 SM's while performing
422 - Switch power cycles (disconneting the SM's)
424 * Irresponsive port detection and recovery
426 * osmtest from multiple nodes
428 * Trap injection and recovery
434 Table 2 - Qualified IB Stacks
435 =============================
438 ----------------------------------------|--------------------------
439 VAPI (Mellanox Infininband HCA Driver) | 3.2 and later
440 OpenIB Gen1 (IBGD distribution) | 1.8.0
441 OpenIB Gen2 (IBG2 distribution) | 1.0
443 Table 3 - Qualified Devices and Corresponding Firmware
444 ======================================================
447 --------|-----------------------------------------------------------
448 MT43132 | InfiniScale - fw-43132 5.2.0 (and later)
449 MT47396 | InfiniScale III - fw-47396 0.5.0 (and later)
450 MT23108 | InfiniHost - fw-23108 3.3.2
451 MT25204 | InfiniHost III Lx - fw-25204 1.0.1
452 MT25208 | InfiniHost III Ex (InfiniHost Mode) - fw-25208 4.6.2 (and later)
453 MT25208 | InfiniHost III Ex (MemFree Mode) - fw-25218 5.0.1 (and later)
455 Other vendors HCAs not yet verified but eHCA is known to be discovered and configured