1 OpenSM Release Notes 2.0.5
2 ============================
4 Version: OpenFabrics Enterprise Distribution (OFED) 1.1
5 Repo: https://openib.org/svn/gen2/branches/1.1/src/userspace/management/osm
6 Version: 9535 (openib-2.0.5)
11 This document describes the contents of the OpenSM OFED 1.1 release.
12 OpenSM is an InfiniBand compliant Subnet Manager and Administration,
13 and runs on top of OpenIB. The OpenSM version for this release
16 This document includes the following sections:
17 1 This Overview section (describing new features and software
19 2 Known Issues And Limitations
20 3 Unsupported IB compliance statements
22 5 Main Verification Flows
23 6 Qualified software stacks and devices
25 1.1 Major New Features
28 The partition manager provides a means to setup multiple partitions
29 by providing a partition policy file. For details please read the
30 doc/partition-config.txt or the opensm man page.
33 Provides a uniform configuration of the entire fabric with values defined
34 in the OpenSM options file. The options support different settings for
35 CAs, Switches, and Routers. Note that this is disabled by default and
36 using -Q enables QoS fabric setup.
38 * Loading pre-routes from a file:
39 A new routing module enables loading pre-routes from a file.
40 To use this option you should use the command line options:
41 "-R file --U <your routing file>" or
42 "--routing_engine file --ucast_file <your routing file>"
43 For more information refer to the file doc/modular-routing.txt
44 or the opensm man page.
46 * SA MultiPathRecord support:
47 The SA can now handle requests for multiple PathRecords in one query.
48 This includes methods SA GetMulti/GetMultiResp and dual sided RMPP.
50 * PPC64 is now QAed and supported
52 * Support LMC > 0 for Switch Enhanced Port 0:
53 Allows enhanced switch port 0 (ESP0) to have a non zero
54 LMC. Use the configured subnet wide LMC for this. Modifications were
55 necessary to the LID assignment and routing to support this.
56 Also, added an option to the configuration to use LMC configured for
57 subnet for enhanced switch port 0 or set it to 0 even if a non zero
58 LMC is configured for the subnet. The default is currently the
59 latter option. The new configuration option is: lmc_esp0
61 1.2 Minor New Features:
63 * IPoIB broadcast group configuration:
64 It is now possible to control the IPoIB broadcast group parameters
65 (MTU, rate, SL) through the partitions configuration file.
67 * Limiting OpenSM log file size:
68 By providing the command line option: "-L <size in MB>" or
69 "--log_limit <size in MB>" the user can limit the generated log
70 file size. When specified, the log file will be truncated upon reaching
73 * Favor 1K MTU for Tavor (MT23108) HCA
74 In cases where a PathRecord or MultiPathRecord is queried and the
75 requestor does not specify the MTU or does specify it in a way
76 that allows for MTU to be 1K and one of the path ends in a Tavor,
77 limit the MTU to 1K max.
80 Added opensm.8 and osmtest.8
82 * Leaf VL stall count control:
83 A new parameter (leaf_vl_stall_count) for controlling the number of
84 sequential packets dropped on a switch port driving a HCA/TCA/Router
85 that cause the port to enter the VLStalled state was added to the
88 * SM Polling/Handover defaults changed
89 The default SMInfo polling retries was decreased from 18 to 4
90 which reduces the default handover time from 3 min to 40 seconds.
92 1.3 Library API Changes
94 * cl_mem* APIs deprecated in complib:
95 These functions are now considered as deprecated and should be
96 replaced by direct calls to malloc, free, memset, etc.
98 * osm_log_init_v2 API added in libopensm:
99 Supports providing the new option for log file truncation.
101 1.4 Software Dependencies
103 OpenSM depends on the installation of either OFED 1.1, OFED 1.0,
104 OpenIB gen2 (e.g. IBG2 distribution), OpenIB gen1 (e.g. IBGD
105 distribution), or Mellanox VAPI stacks. The qualified driver versions
106 are provided in Table 2, "Qualified IB Stacks".
108 1.5 Supported Devices Firmware
110 The main task of OpenSM is to initialize InfiniBand devices. The
111 qualified devices and their corresponding firmware versions
112 are listed in Table 3.
114 2 Known Issues And Limitations
115 ------------------------------
117 * No Service / Key associations:
118 There is no way to manage Service access by Keys.
120 * No SM to SM SMDB synchronization:
121 Puts the burden of re-registering services, multicast groups, and
122 inform-info on the client application (or IB access layer core).
124 * No "port down" event handling:
125 Changing the switch port through which OpenSM connects to the IB
126 fabric may cause incorrect operation. Please restart OpenSM whenever
127 such a connectivity change is made.
129 * Changing connections during SM operation:
130 Under some conditions the SM can get confused by a change in
131 cabling (moving a cable from one switch port to the other) and
132 momentarily see this as having the same GUID appear connected
133 to two different IB ports. Under some conditions, when the SM fails to
134 get the corresponding change event it might mistakenly report this case
135 as a "duplicated GUID" case and abort. It is advisable to double-check
136 the syslog after each such change in connectivity and restart
137 OpenSM if it has exited.
139 3 Unsupported IB Compliance Statements
140 --------------------------------------
141 The following section lists all the IB compliance statements which
142 OpenSM does not support. Please refer to the IB specification for detailed
143 information regarding each compliance statement.
145 * C14-22 (Authentication):
146 M_Key M_KeyProtectBits and M_KeyLeasePeriod shall be set in one
147 SubnSet method. As a work-around, an OpenSM option is provided for
148 defining the protect bits.
150 * C14-67 (Authentication):
151 On SubnGet(SMInfo) and SubnSet(SMInfo) - if M_Key is not zero then
152 the SM shall generate a SubnGetResp if the M_Key matches, or
153 silently drop the packet if M_Key does not match.
155 * C15-0.1.23.4 (Authentication):
156 InformInfoRecords shall always be provided with the QPN set to 0,
157 except for the case of a trusted request, in which case the actual
158 subscriber QPN shall be returned.
160 * o13-17.1.2 (Event-FWD):
161 If no permission to forward, the subscription should be removed and
162 no further forwarding should occur.
164 * C14-24.1.1.5 and C14-62.1.1.22 (Initialization):
165 GUIDInfo - SM should enable assigning Port GUIDInfo.
167 * C14-44 (Initialization):
168 If the SM discovers that it is missing an M_Key to update CA/RT/SW,
169 it should notify the higher level.
171 * C14-62.1.1.12 (Initialization):
172 PortInfo:M_Key - Set the M_Key to a node based random value.
174 * C14-62.1.1.13 (Initialization):
175 PortInfo:P_KeyProtectBits - set according to an optional policy.
177 * C14-62.1.1.24 (Initialization):
178 SwitchInfo:DefaultPort - should be configured for random FDB.
180 * C14-62.1.1.32 (Initialization):
181 RandomForwardingTable should be configured.
183 * o15-0.1.12 (Multicast):
184 If the JoinState is SendOnlyNonMember = 1 (only), then the endport
185 should join as sender only.
187 * o15-0.1.8 (Multicast):
188 If a request for creating an MCG with fields that cannot be met,
189 return ERR_REQ_INVALID (currently ignores SL and FlowLabelTClass).
191 * C15-0.1.8.6 (SA-Query):
192 Respond to SubnAdmGetTraceTable - this is an optional attribute.
194 * C15-0.1.13 Services:
195 Reject ServiceRecord create, modify or delete if the given
196 ServiceP_Key does not match the one included in the ServiceGID port
197 and the port that sent the request.
199 * C15-0.1.14 (Services):
200 Provide means to associate service name and ServiceKeys.
205 The following is a list of bugs that were fixed. Note that other less critical
206 or visible bugs were also fixed.
208 * "Broken" fabric (duplicated port GUIDs) handling improved
209 Replace assert with a real check to handle invalid physical port
210 in osm_node_info_rcv.c which could occur on a broken fabric
212 * SA client synchronous request failed but status returned was IB_SUCCESS
213 even if there was no response.
214 There was a missing setting of the status in the synchronous case.
217 1. In libvendor/osm_vendor_ibumad.c:osm_vendor_get_all_port_attr
218 2. In libvendor/osm_vendor_ibumad_sa.c:__osmv_sa_mad_rcv_cb
219 3. On receiving SMInfo SA request from a node that does not share a
220 partition, the response mad was allocated but never free'd
221 as it was never sent.
223 * Set(InformInfo) OpenSM Deadlock:
224 When receiving a request with unknown LID
226 * PathRecord to inconsistent multicast destination:
227 Fix the return error when multicast destination is not consistently
230 * Remove double calculation of reversible path
231 In osm_sa_path_record.c:__osm_pr_rcv_get_lid_pair_path a PathRecord
232 query used to double check if the path is reversible
234 * Some PathRecord log messages use "net order":
235 Fix GUID net to host conversion in some osm_log messages
237 * DR/LID routed SMPs direction bit handling:
238 osm_resp.c:osm_resp_make_resp_smp, set direction bit only if direct
239 routed class. This bug caused two issues:
240 1. Get/Set responses always had direction bit set.
241 2. Trap represses never had direction bit set.
242 The direction bit needs setting in direct routed responses and it
243 doesn't exist in LID routed responses.
244 osm_sm_mad_ctrl.c: did not detect the "direction bit" correctly.
246 * OpenSM crash due to transaction lookup (interop with Cisco stack)
247 When a wire TID that maps to internal TID of zero (after applying
248 mask) was received the lookup of the transaction was successful.
249 The stale transaction pointed to "free'd" memory.
251 * Better handling for Path/MultiPath requests for raw traffic
253 * Wrong ProducerType provided in Notice Reports:
254 When formating an SM generated report, the ProducerType was using
255 CL_NTOH32 which can not be used to format a 24bit network order number.
257 * OpenSM break on PPC64
258 complib: Fixed memory corruption in cl_pool.c:cl_qcpool_init. This
259 affected big endian 64-bit architectures only.
261 * Illegal Set(InformInfo) was wrongly successful in updating the SMDB
262 osm_sa_informinfo.c: In osm_infr_rcv_process_set_method, if sending
263 error, don't call osm_infr_rcv_process_set_method
265 * RMPP queries of InformInfoRecord fail
266 ib_types.h: Pad ib_inform_info_record_t to be modulo 8 in size so
267 that attribute offset is calculated properly
269 * Returning "invalid request" rather than "unsupported method/attribute"
270 In these cases, a noncompliant response was being provided.
272 * Noncompliant response for SubnAdmGet(PortInfoRecord) with no match
273 osm_pir_rcv_process, now returns "SA no records error" for SubnAdmGet
276 * Noncompliant non base LID returned by some queries:
277 The following attributes used to return the request LID rather than
278 its base LID in responses: PKeyTableRecord, GUIDInfoRecord,
279 SLtoVLMappingTableRecord, VLArbitrationTableRecord, LinkRecord
281 * Noncompliant SubnAdmGet and SubnAdmGetTable:
282 Mixing of error codes in case of no records or multiple records
283 fixed for the attributes:
284 LinearForwardingTableRecord, GUIDInfoRecord,
285 VLArbitrationTableRecord, LinkRecord, PathRecord
287 * segfault in InformInfo flows
288 Under stress concurrent Set/Delete/Get flows. Fixed by adding
291 * SA queries containing LID out if range did not return ERR_REQ_INVALID
293 5 Main Verification Flows
294 -------------------------
296 OpenSM verification is run using the following activities:
297 * osmtest - a stand-alone program
298 * ibmgtsim (IB management simulator) based - a set of flows that
299 simulate clusters, inject errors and verify OpenSM capability to
300 respond and bring up the network correctly.
301 * small cluster regression testing - where the SM is used on back to
302 back or single switch configurations. The regression includes
303 multiple OpenSM dedicated tests.
304 * cluster testing - when we run OpenSM to setup a large cluster, perform
305 hand-off, reboots and reconnects, verify routing correctness and SA
306 responsiveness at the ULP level (IPoIB and SDP).
310 osmtest is an automated verification tool used for OpenSM
311 testing. Its verification flows are described by list below.
313 * Inventory File: Obtain and verify all port info, node info, link and path
317 - Register new service
318 - Register another service (with a lease period)
319 - Register another service (with service p_key set to zero)
320 - Get all services by name
321 - Delete the first service
322 - Delete the third service
323 - Added bad flows of get/delete non valid service
324 - Add / Get same service with different data
325 - Add / Get / Delete by different component mask values (services
326 by Name & Key / Name & Data / Name & Id / Id only )
328 * Multicast Member Record:
329 - Query of existing Groups (IPoIB)
330 - BAD Join with insufficient comp mask (o15.0.1.3)
331 - Create given MGID=0 (o15.0.1.4)
332 - Create given MGID=0xFF12A01C,FE800000,00000000,12345678 (o15.0.1.4)
333 - Create BAD MGID=0xFA. (o15.0.1.6)
334 - Create BAD MGID=0xFF12A01B w/ link-local not set (o15.0.1.6)
335 - New MGID with invalid join state (o15.0.1.9)
336 - Retry of existing MGID - See JoinState update (o15.0.1.11)
337 - BAD RATE when connecting to existing MGID (o15.0.1.13)
338 - Partial JoinState delete request - removing FullMember (o15.0.1.14)
339 - Full Delete of a group (o15.0.1.14)
340 - Verify Delete by trying to Join deleted group (o15.0.1.14)
341 - BAD Delete of IPoIB membership (no prev join) (o15.0.1.15)
344 - All GUIDInfoRecords in subnet are obtained
347 - Perform some compliant and noncompliant MultiPathRecord requests
348 - Validation is via status in responses and IB analyzer
351 - Perform some compliant and noncompliant PKeyTableRecord queries
352 - Validation is via status in responses and IB analyzer
354 * LinearForwardingTableRecord:
355 - Perform some compliant and noncompliant LinearForwardingTableRecord queries
356 - Validation is via status in responses and IB analyzer
358 * Event Forwarding: Register for trap forwarding using reports
359 - Send a trap and wait for report
360 - Unregister non-existing
362 * Trap 64/65 Flow: Register to Trap 64-65, create traps (by
363 disconnecting/connecting ports) and wait for report, then unregister.
365 * Stress Test: send PortInfoRecord queries, both single and RMPP and
366 check for the rate of responses as well as their validity.
369 5.2 IB Management Simulator OpenSM Test Flows:
371 The simulator provides ability to simulate the SM handling of virtual
372 topologies that are not limited to actual lab equipment availability.
373 OpenSM was simulated to bring up clusters of up to 10,000 nodes. Daily
374 regressions use smaller (16 and 128 nodes clusters).
376 The following test flows are run on the IB management simulator:
379 Up to 12 links from the fabric are randomly selected to drop packets
380 at drop rates up to 90%. The SM is required to succeed in bringing the
381 fabric up. The resulting routing is verified to be correct as well.
384 Using LMC = 2 the fabric is initialized with LIDs. Faults such as
385 zero LID, Duplicated LID, non-aligned (to LMC) LIDs are
386 randomly assigned to various nodes and other errors are randomly
387 output to the guid2lid cache file. The SM sweep is run 5 times and
388 after each iteration a complete verification is made to ensure that all
389 LIDs that could possibly be maintained are kept, as well as that all nodes
390 were assigned a legal LID range.
393 Nodes randomly join the 0xc000 group and eventually the
394 resulting routing is verified for completeness and adherence to
395 Up/Down routing rules.
398 The complete osmtest flow as described in the previous table is run on
399 the simulated fabrics.
402 This flow merges fabric, LID and stability issues with continuous
403 PathRecord, ServiceRecord and Multicast Join/Leave activity to
404 stress the SM/SA during continuous sweeps. InformInfo Set/Delete/Get
405 were added to the test such both existing and non existing nodes
406 perform them in random order.
408 5.3 OpenSM Regression
410 Using a back-to-back or single switch connection, the following set of
411 tests is run nightly on the stacks described in table 2. The included
414 * Stress Testing: Flood the SA with queries from multiple channel
415 adapters to check the robustness of the entire stack up to the SA.
417 * Dynamic Changes: Dynamic Topology changes, through randomly
418 dropping SMP packets, used to test OpenSM adaptation to an unstable
419 network & verify DB correctness.
421 * Trap Injection: This flow injects traps to the SM and verifies that it
422 handles them gracefully.
424 * SA Query Test: This test exhaustively checks the SA responses to all
425 possible single component mask. To do that the test examines the
426 entire set of records the SA can provide, classifies them by their
427 field values and then selects every field (using component mask and a
428 value) and verifies that the response matches the expected set of records.
429 A random selection using multiple component mask bits is also performed.
433 Cluster testing is usually run before a distribution release. It
434 involves real hardware setups of 16 to 32 nodes (or more if a beta site
435 is available). Each test is validated by running all-to-all ping through the IB
436 interface. The test procedure includes:
440 * Hand-off between 2 or 3 SM's while performing:
442 - Switch power cycles (disconnecting the SM's)
444 * Unresponsive port detection and recovery
446 * osmtest from multiple nodes
448 * Trap injection and recovery
454 Table 2 - Qualified IB Stacks
455 =============================
458 -----------------------------------------|--------------------------
461 OpenIB Gen2 (IBG2 distribution) | 1.0
462 OpenIB Gen1 (IBGD distribution) | 1.8.0
463 VAPI (Mellanox InfiniBand HCA Driver) | 3.2 and later
465 Table 3 - Qualified Devices and Corresponding Firmware
466 ======================================================
470 --------|-----------------------------------------------------------
471 MT43132 | InfiniScale - fw-43132 5.2.0 (and later)
472 MT47396 | InfiniScale III - fw-47396 0.5.0 (and later)
473 MT23108 | InfiniHost - fw-23108 3.3.2 (and later)
474 MT25204 | InfiniHost III Lx - fw-25204 1.0.1i (and later)
475 MT25208 | InfiniHost III Ex (InfiniHost Mode) - fw-25208 4.6.2 (and later)
476 MT25208 | InfiniHost III Ex (MemFree Mode) - fw-25218 5.0.1 (and later)
480 --------|-----------------------------------------------------------
481 iPath | QHT6040 (PathScale InfiniPath HT-460)
482 iPath | QHT6140 (PathScale InfiniPath HT-465)
483 iPath | QLE6140 (PathScale InfiniPath PE-880)
485 Note: OpenSM does not run on an IBM Galaxy (eHCA) as it does not expose
486 QP0 and QP1. However, it does support it as a device on the subnet.