3 CTL - CAM Target Layer Description
5 Revision 1.4 (December 29th, 2011)
6 Ken Merry <ken@FreeBSD.org>
13 Configuring and Running CTL
22 CTL is a disk and processor device emulation subsystem originally written
23 for Copan Systems under Linux starting in 2003. It has been shipping in
24 Copan (now SGI) products since 2005.
26 It was ported to FreeBSD in 2008, and thanks to an agreement between SGI
27 (who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is
28 available under a BSD-style license. The intent behind the agreement was
29 that Spectra would work to get CTL into the FreeBSD tree.
34 - Disk and processor device emulation.
36 - SCSI task attribute support (ordered, head of queue, simple tags)
37 - SCSI implicit command ordering support. (e.g. if a read follows a mode
38 select, the read will be blocked until the mode select completes.)
39 - Full task management support (abort, LUN reset, target reset, etc.)
40 - Support for multiple ports
41 - Support for multiple simultaneous initiators
42 - Support for multiple simultaneous backing stores
43 - Persistent reservation support
44 - Mode sense/select support
45 - Error injection support
46 - High Availability support (1)
47 - All I/O handled in-kernel, no userland context switch overhead.
49 (1) HA Support is just an API stub, and needs much more to be fully
50 functional. See the to-do list below.
52 Configuring and Running CTL:
53 ===========================
55 - After applying the CTL patchset to your tree, build world and install it
56 on your target system.
58 - Add 'device ctl' to your kernel configuration file.
60 - If you're running with a 8Gb or 4Gb Qlogic FC board, add
61 'options ISP_TARGET_MODE' to your kernel config file. Keep in mind that
62 the isp(4) driver can run in target or initiator mode, but not both on
63 the same machine. 'device ispfw' or loading the ispfw module is also
66 - Rebuild and install a new kernel.
68 - Reboot with the new kernel.
70 - To add a LUN with the RAM disk backend:
72 ctladm create -b ramdisk -s 10485760000000000000
75 - You should now see the CTL disk LUN through camcontrol devlist:
77 scbus6 on ctl2cam0 bus 0:
78 <FREEBSD CTLDISK 0001> at scbus6 target 1 lun 0 (da24,pass32)
79 <> at scbus6 target -1 lun -1 ()
81 This is visible through the CTL CAM SIM. This allows using CTL without
82 any physical hardware. You should be able to issue any normal SCSI
83 commands to the device via the pass(4)/da(4) devices.
85 If any target-capable HBAs are in the system (e.g. isp(4)), and have
86 target mode enabled, you should now also be able to see the CTL LUNs via
87 that target interface.
89 Note that all CTL LUNs are presented to all frontends. There is no
90 LUN masking, or separate, per-port configuration.
92 - Note that the ramdisk backend is a "fake" ramdisk. That is, it is
93 backed by a small amount of RAM that is used for all I/O requests. This
94 is useful for performance testing, but not for any data integrity tests.
96 - To add a LUN with the block/file backend:
98 truncate -s +1T myfile
99 ctladm create -b block -o file=myfile
102 - You can also see a list of LUNs and their backends like this:
105 LUN Backend Size (Blocks) BS Serial Number Device ID
106 0 block 2147483648 512 MYSERIAL 0 MYDEVID 0
107 1 block 2147483648 512 MYSERIAL 1 MYDEVID 1
108 2 block 2147483648 512 MYSERIAL 2 MYDEVID 2
109 3 block 2147483648 512 MYSERIAL 3 MYDEVID 3
110 4 block 2147483648 512 MYSERIAL 4 MYDEVID 4
111 5 block 2147483648 512 MYSERIAL 5 MYDEVID 5
112 6 block 2147483648 512 MYSERIAL 6 MYDEVID 6
113 7 block 2147483648 512 MYSERIAL 7 MYDEVID 7
114 8 block 2147483648 512 MYSERIAL 8 MYDEVID 8
115 9 block 2147483648 512 MYSERIAL 9 MYDEVID 9
116 10 block 2147483648 512 MYSERIAL 10 MYDEVID 10
117 11 block 2147483648 512 MYSERIAL 11 MYDEVID 11
119 - You can see the LUN type and backing store for block/file backend LUNs
123 LUN Backend Size (Blocks) BS Serial Number Device ID
124 0 block 2147483648 512 MYSERIAL 0 MYDEVID 0
128 1 block 2147483648 512 MYSERIAL 1 MYDEVID 1
132 2 block 2147483648 512 MYSERIAL 2 MYDEVID 2
136 3 block 2147483648 512 MYSERIAL 3 MYDEVID 3
140 4 block 2147483648 512 MYSERIAL 4 MYDEVID 4
144 5 block 2147483648 512 MYSERIAL 5 MYDEVID 5
148 6 block 2147483648 512 MYSERIAL 6 MYDEVID 6
152 7 block 2147483648 512 MYSERIAL 7 MYDEVID 7
156 8 block 2147483648 512 MYSERIAL 8 MYDEVID 8
160 9 block 2147483648 512 MYSERIAL 9 MYDEVID 9
164 10 ramdisk 0 0 MYSERIAL 0 MYDEVID 0
166 11 ramdisk 204800000000000 512 MYSERIAL 1 MYDEVID 1
172 - Added in the second HA mode (where CTL does the data transfers instead
173 of having data transfers done below CTL), and abstracted out the Copan
176 - Fixed the phantom device problem in the CTL CAM SIM and improved the
177 CAM SIM to automatically trigger a rescan when the port is enabled and
180 - Made the number of threads in the block backend configurable via sysctl,
181 loader tunable and the ctladm command line. (You can now specify
182 -o num_threads=4 when creating a LUN with ctladm create.)
184 - Fixed some LUN selection issues in ctlstat(8) and allowed for selection
185 of LUN numbers up to 1023.
189 - This version intended for public release.
193 - Added descriptor sense support to CTL. It can be enabled through the
194 control mode page (10), but is disabled by default.
196 - Improved error injection support. The number of errors that can be
197 injected with 'ctladm inject' has been increased, and any arbitrary
198 sense data may now be injected as well.
200 - The port infrastructure has been revamped. Individual ports and types
201 of ports may now be enabled and disabled from the command line. ctladm
202 now has the ability to set the WWNN and WWPN for each port.
204 - The block backend can now send multiple I/Os to backing files. Multiple
205 writes are only allowed for ZFS, but multiple readers are allowed for
208 - The block and ramdisk backends now support setting the LUN blocksize.
209 There are some restrictions when the backing device is a block device,
210 but otherwise the blocksize may be set to anything.
215 - CTL initialization process has been revamped. Instead of using an
216 ad-hoc method, it is now sequenced through SYSINIT() calls.
218 - A block/file backend has been added. This allows using arbitrary files
219 or block devices as a backing store.
221 - The userland LUN configuration interface has been completely rewritten.
222 Configuration is now done out of band.
224 - The ctladm(8) command line interface has been revamped, and is now
225 similar to camcontrol(8).
230 - Use devstat(9) for CTL's statistics collection. CTL uses a home-grown
231 statistics collection system that is similar to devstat(9). ctlstat
232 should be retired in favor of iostat, etc., once aggregation modes are
233 available in iostat to match the behavior of ctlstat -t and dump modes
234 are available to match the behavior of ctlstat -d/ctlstat -J.
236 - ZFS ARC backend for CTL. Since ZFS copies all I/O into the ARC
237 (Adaptive Replacement Cache), running the block/file backend on top of a
238 ZFS-backed zdev or file will involve an extra set of copies. The
239 optimal solution for backing targets served by CTL with ZFS would be to
240 allocate buffers out of the ARC directly, and DMA to/from them directly.
241 That would eliminate an extra data buffer allocation and copy.
243 - Switch CTL over to using CAM CCBs instead of its own union ctl_io. This
244 will likely require a significant amount of work, but will eliminate
245 another data structure in the stack, more memory allocations, etc. This
246 will also require changes to the CAM CCB structure to support CTL.
248 - Full-featured High Availability support. The HA API that is in ctl_ha.h
249 is essentially a renamed version of Copan's HA API. There is no
250 substance to it, but it remains in CTL to show what needs to be done to
251 implement active/active HA from a CTL standpoint. The things that would
252 need to be done include:
253 - A kernel level software API for message passing as well as DMA
254 between at least two nodes.
255 - Hardware support and drivers for inter-node communication. This
256 could be as simples as ethernet hardware and drivers.
257 - A "supervisor", or startup framework to control and coordinate
258 HA startup, failover (going from active/active to single mode),
259 and failback (going from single mode to active/active).
260 - HA support in other components of the stack. The goal behind HA
261 is that one node can fail and another node can seamlessly take
262 over handling I/O requests. This requires support from pretty
263 much every component in the storage stack, from top to bottom.
264 CTL is one piece of it, but you also need support in the RAID
265 stack/filesystem/backing store. You also need full configuration
266 mirroring, and all peer nodes need to be able to talk to the
267 underlying storage hardware.
272 CTL has the concept of pluggable frontend ports and backends. All
273 frontends and backends can be active at the same time. You can have a
274 ramdisk-backed LUN present along side a file backed LUN.
279 This is the core of CTL, where all of the command handlers and a lot of
280 other things live. Yes, it is large. It started off small and grew to its
281 current size over time. Perhaps it can be split into more files at some
284 Here is a roadmap of some of the primary functions in ctl.c. Starting here
285 and following the various leaf functions will show the command flow.
287 ctl_queue() This is where commands from the frontend ports come
290 ctl_queue_sense() This is only used for non-packetized SCSI. i.e.
291 parallel SCSI prior to U320 and perhaps U160.
293 ctl_work_thread() This is the primary work thread, and everything gets
296 ctl_scsiio_precheck() This where all of the initial checks are done, and I/O
297 is either queued for execution or blocked.
299 ctl_scsiio() This is where the command handler is actually
300 executed. (See ctl_cmd_table.c for the mapping of
301 SCSI opcode to command handler function.)
303 ctl_done() This is the routine called (or ctl_done_lock()) to
304 initiate the command completion process.
306 ctl_process_done() This is where command completion actually happens.
311 Basic function declarations and data structures.
317 These files define the basic CTL backend API. The comments in the header
324 The block and file backend. This allows for using a disk or a file as the
325 backing store for a LUN. Multiple threads are started to do I/O to the
326 backing device, primarily because the VFS API requires that to get any
329 ctl_backend_ramdisk.c:
330 ---------------------
332 A "fake" ramdisk backend. It only allocates a small amount of memory to
333 act as a source and sink for reads and writes from an initiator. Therefore
334 it cannot be used for any real data, but it can be used to test for
335 throughput. It can also be used to test initiators' support for extremely
341 This is a table with all 256 possible SCSI opcodes, and command handler
342 functions defined for supported opcodes. It is included in ctl.c.
347 Simplistic debugging support.
353 CTL-specific wrappers around the CAM sense building functions.
359 These files define the basic CTL frontend port API. The comments in the
360 header explain the API.
362 ctl_frontend_cam_sim.c:
363 ----------------------
365 This is a CTL frontend port that is also a CAM SIM. The idea is that this
366 frontend allows for using CTL without any target-capable hardware. So any
367 LUNs you create in CTL are visible via this port.
370 ctl_frontend_internal.c
371 ctl_frontend_internal.h:
372 -----------------------
374 This is a frontend port written for Copan to do some system-specific tasks
375 that required sending commands into CTL from inside the kernel. This isn't
376 entirely relevant to FreeBSD in general, but can perhaps be repurposed or
382 This is a stubbed-out High Availability API. See the comments in the
383 header and the description of what is needed as far as HA support above.
388 This defines most of the core CTL I/O structures. union ctl_io is
389 conceptually very similar to CAM's union ccb.
394 This defines all ioctls available through the CTL character device, and
395 the data structures needed for those ioctls.
400 Private data structres (e.g. CTL softc) and function prototypes. This also
401 includes the SCSI vendor and product names used by CTL.
407 CTL wrappers around CAM sense printing functions.
412 Command serialization table. This defines what happens when one type of
413 command is followed by another type of command. e.g., what do you do when
414 you have a mode select followed by a write? You block the write until the
415 mode select is complete. That is defined in this table.
421 CTL utility functions, primarily designed to be used from userland. See
422 ctladm for the primary consumer of these functions. These include CDB
428 CAM target peripheral driver and CTL frontend port. This is the path into
429 CTL for commands from target-capable hardware/SIMs.
434 ctladm(8) fills a role similar to camcontrol(8). It allow configuring LUNs,
435 issuing commands, injecting errors and various other control functions.
437 ctlstat(8) fills a role similar to iostat(8). It reports I/O statistics