Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
fork

Configure Feed

Select the types of activity you want to include in your feed.

[SCSI] scsi: Error handler description document

This patch adds Documentation/scsi/scs_eh.txt. I've chosen plain
text over DocBook as most other scsi docs are in plain text and it's
more accessible.

Signed-off-by: Tejun Heo <htejun@gmail.com>
Signed-off-by: James Bottomley <James.Bottomley@SteelEye.com>

authored by

Tejun Heo and committed by
James Bottomley
70c83e11 6f16b535

+481
+2
Documentation/scsi/00-INDEX
··· 60 60 - short blurb on using SCSI support as a module. 61 61 scsi_mid_low_api.txt 62 62 - info on API between SCSI layer and low level drivers 63 + scsi_eh.txt 64 + - info on SCSI midlayer error handling infrastructure 63 65 st.txt 64 66 - info on scsi tape driver 65 67 sym53c500_cs.txt
+479
Documentation/scsi/scsi_eh.txt
··· 1 + 2 + SCSI EH 3 + ====================================== 4 + 5 + This document describes SCSI midlayer error handling infrastructure. 6 + Please refer to Documentation/scsi/scsi_mid_low_api.txt for more 7 + information regarding SCSI midlayer. 8 + 9 + TABLE OF CONTENTS 10 + 11 + [1] How SCSI commands travel through the midlayer and to EH 12 + [1-1] struct scsi_cmnd 13 + [1-2] How do scmd's get completed? 14 + [1-2-1] Completing a scmd w/ scsi_done 15 + [1-2-2] Completing a scmd w/ timeout 16 + [1-3] How EH takes over 17 + [2] How SCSI EH works 18 + [2-1] EH through fine-grained callbacks 19 + [2-1-1] Overview 20 + [2-1-2] Flow of scmds through EH 21 + [2-1-3] Flow of control 22 + [2-2] EH through hostt->eh_strategy_handler() 23 + [2-2-1] Pre hostt->eh_strategy_handler() SCSI midlayer conditions 24 + [2-2-2] Post hostt->eh_strategy_handler() SCSI midlayer conditions 25 + [2-2-3] Things to consider 26 + 27 + 28 + [1] How SCSI commands travel through the midlayer and to EH 29 + 30 + [1-1] struct scsi_cmnd 31 + 32 + Each SCSI command is represented with struct scsi_cmnd (== scmd). A 33 + scmd has two list_head's to link itself into lists. The two are 34 + scmd->list and scmd->eh_entry. The former is used for free list or 35 + per-device allocated scmd list and not of much interest to this EH 36 + discussion. The latter is used for completion and EH lists and unless 37 + otherwise stated scmds are always linked using scmd->eh_entry in this 38 + discussion. 39 + 40 + 41 + [1-2] How do scmd's get completed? 42 + 43 + Once LLDD gets hold of a scmd, either the LLDD will complete the 44 + command by calling scsi_done callback passed from midlayer when 45 + invoking hostt->queuecommand() or SCSI midlayer will time it out. 46 + 47 + 48 + [1-2-1] Completing a scmd w/ scsi_done 49 + 50 + For all non-EH commands, scsi_done() is the completion callback. It 51 + does the following. 52 + 53 + 1. Delete timeout timer. If it fails, it means that timeout timer 54 + has expired and is going to finish the command. Just return. 55 + 56 + 2. Link scmd to per-cpu scsi_done_q using scmd->en_entry 57 + 58 + 3. Raise SCSI_SOFTIRQ 59 + 60 + SCSI_SOFTIRQ handler scsi_softirq calls scsi_decide_disposition() to 61 + determine what to do with the command. scsi_decide_disposition() 62 + looks at the scmd->result value and sense data to determine what to do 63 + with the command. 64 + 65 + - SUCCESS 66 + scsi_finish_command() is invoked for the command. The 67 + function does some maintenance choirs and notify completion by 68 + calling scmd->done() callback, which, for fs requests, would 69 + be HLD completion callback - sd:sd_rw_intr, sr:rw_intr, 70 + st:st_intr. 71 + 72 + - NEEDS_RETRY 73 + - ADD_TO_MLQUEUE 74 + scmd is requeued to blk queue. 75 + 76 + - otherwise 77 + scsi_eh_scmd_add(scmd, 0) is invoked for the command. See 78 + [1-3] for details of this funciton. 79 + 80 + 81 + [1-2-2] Completing a scmd w/ timeout 82 + 83 + The timeout handler is scsi_times_out(). When a timeout occurs, this 84 + function 85 + 86 + 1. invokes optional hostt->eh_timedout() callback. Return value can 87 + be one of 88 + 89 + - EH_HANDLED 90 + This indicates that eh_timedout() dealt with the timeout. The 91 + scmd is passed to __scsi_done() and thus linked into per-cpu 92 + scsi_done_q. Normal command completion described in [1-2-1] 93 + follows. 94 + 95 + - EH_RESET_TIMER 96 + This indicates that more time is required to finish the 97 + command. Timer is restarted. This action is counted as a 98 + retry and only allowed scmd->allowed + 1(!) times. Once the 99 + limit is reached, action for EH_NOT_HANDLED is taken instead. 100 + 101 + *NOTE* This action is racy as the LLDD could finish the scmd 102 + after the timeout has expired but before it's added back. In 103 + such cases, scsi_done() would think that timeout has occurred 104 + and return without doing anything. We lose completion and the 105 + command will time out again. 106 + 107 + - EH_NOT_HANDLED 108 + This is the same as when eh_timedout() callback doesn't exist. 109 + Step #2 is taken. 110 + 111 + 2. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the 112 + command. See [1-3] for more information. 113 + 114 + 115 + [1-3] How EH takes over 116 + 117 + scmds enter EH via scsi_eh_scmd_add(), which does the following. 118 + 119 + 1. Turns on scmd->eh_eflags as requested. It's 0 for error 120 + completions and SCSI_EH_CANCEL_CMD for timeouts. 121 + 122 + 2. Links scmd->eh_entry to shost->eh_cmd_q 123 + 124 + 3. Sets SHOST_RECOVERY bit in shost->shost_state 125 + 126 + 4. Increments shost->host_failed 127 + 128 + 5. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed 129 + 130 + As can be seen above, once any scmd is added to shost->eh_cmd_q, 131 + SHOST_RECOVERY shost_state bit is turned on. This prevents any new 132 + scmd to be issued from blk queue to the host; eventually, all scmds on 133 + the host either complete normally, fail and get added to eh_cmd_q, or 134 + time out and get added to shost->eh_cmd_q. 135 + 136 + If all scmds either complete or fail, the number of in-flight scmds 137 + becomes equal to the number of failed scmds - i.e. shost->host_busy == 138 + shost->host_failed. This wakes up SCSI EH thread. So, once woken up, 139 + SCSI EH thread can expect that all in-flight commands have failed and 140 + are linked on shost->eh_cmd_q. 141 + 142 + Note that this does not mean lower layers are quiescent. If a LLDD 143 + completed a scmd with error status, the LLDD and lower layers are 144 + assumed to forget about the scmd at that point. However, if a scmd 145 + has timed out, unless hostt->eh_timedout() made lower layers forget 146 + about the scmd, which currently no LLDD does, the command is still 147 + active as long as lower layers are concerned and completion could 148 + occur at any time. Of course, all such completions are ignored as the 149 + timer has already expired. 150 + 151 + We'll talk about how SCSI EH takes actions to abort - make LLDD 152 + forget about - timed out scmds later. 153 + 154 + 155 + [2] How SCSI EH works 156 + 157 + LLDD's can implement SCSI EH actions in one of the following two 158 + ways. 159 + 160 + - Fine-grained EH callbacks 161 + LLDD can implement fine-grained EH callbacks and let SCSI 162 + midlayer drive error handling and call appropriate callbacks. 163 + This will be dicussed further in [2-1]. 164 + 165 + - eh_strategy_handler() callback 166 + This is one big callback which should perform whole error 167 + handling. As such, it should do all choirs SCSI midlayer 168 + performs during recovery. This will be discussed in [2-2]. 169 + 170 + Once recovery is complete, SCSI EH resumes normal operation by 171 + calling scsi_restart_operations(), which 172 + 173 + 1. Checks if door locking is needed and locks door. 174 + 175 + 2. Clears SHOST_RECOVERY shost_state bit 176 + 177 + 3. Wakes up waiters on shost->host_wait. This occurs if someone 178 + calls scsi_block_when_processing_errors() on the host. 179 + (*QUESTION* why is it needed? All operations will be blocked 180 + anyway after it reaches blk queue.) 181 + 182 + 4. Kicks queues in all devices on the host in the asses 183 + 184 + 185 + [2-1] EH through fine-grained callbacks 186 + 187 + [2-1-1] Overview 188 + 189 + If eh_strategy_handler() is not present, SCSI midlayer takes charge 190 + of driving error handling. EH's goals are two - make LLDD, host and 191 + device forget about timed out scmds and make them ready for new 192 + commands. A scmd is said to be recovered if the scmd is forgotten by 193 + lower layers and lower layers are ready to process or fail the scmd 194 + again. 195 + 196 + To achieve these goals, EH performs recovery actions with increasing 197 + severity. Some actions are performed by issueing SCSI commands and 198 + others are performed by invoking one of the following fine-grained 199 + hostt EH callbacks. Callbacks may be omitted and omitted ones are 200 + considered to fail always. 201 + 202 + int (* eh_abort_handler)(struct scsi_cmnd *); 203 + int (* eh_device_reset_handler)(struct scsi_cmnd *); 204 + int (* eh_bus_reset_handler)(struct scsi_cmnd *); 205 + int (* eh_host_reset_handler)(struct scsi_cmnd *); 206 + 207 + Higher-severity actions are taken only when lower-severity actions 208 + cannot recover some of failed scmds. Also, note that failure of the 209 + highest-severity action means EH failure and results in offlining of 210 + all unrecovered devices. 211 + 212 + During recovery, the following rules are followed 213 + 214 + - Recovery actions are performed on failed scmds on the to do list, 215 + eh_work_q. If a recovery action succeeds for a scmd, recovered 216 + scmds are removed from eh_work_q. 217 + 218 + Note that single recovery action on a scmd can recover multiple 219 + scmds. e.g. resetting a device recovers all failed scmds on the 220 + device. 221 + 222 + - Higher severity actions are taken iff eh_work_q is not empty after 223 + lower severity actions are complete. 224 + 225 + - EH reuses failed scmds to issue commands for recovery. For 226 + timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd 227 + before reusing it for EH commands. 228 + 229 + When a scmd is recovered, the scmd is moved from eh_work_q to EH 230 + local eh_done_q using scsi_eh_finish_cmd(). After all scmds are 231 + recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to 232 + either retry or error-finish (notify upper layer of failure) recovered 233 + scmds. 234 + 235 + scmds are retried iff its sdev is still online (not offlined during 236 + EH), REQ_FAILFAST is not set and ++scmd->retries is less than 237 + scmd->allowed. 238 + 239 + 240 + [2-1-2] Flow of scmds through EH 241 + 242 + 1. Error completion / time out 243 + ACTION: scsi_eh_scmd_add() is invoked for scmd 244 + - set scmd->eh_eflags 245 + - add scmd to shost->eh_cmd_q 246 + - set SHOST_RECOVERY 247 + - shost->host_failed++ 248 + LOCKING: shost->host_lock 249 + 250 + 2. EH starts 251 + ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q 252 + is cleared. 253 + LOCKING: shost->host_lock (not strictly necessary, just for 254 + consistency) 255 + 256 + 3. scmd recovered 257 + ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd 258 + - shost->host_failed-- 259 + - clear scmd->eh_eflags 260 + - scsi_setup_cmd_retry() 261 + - move from local eh_work_q to local eh_done_q 262 + LOCKING: none 263 + 264 + 4. EH completes 265 + ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper 266 + layer of failure. 267 + - scmd is removed from eh_done_q and scmd->eh_entry is cleared 268 + - if retry is necessary, scmd is requeued using 269 + scsi_queue_insert() 270 + - otherwise, scsi_finish_command() is invoked for scmd 271 + LOCKING: queue or finish function performs appropriate locking 272 + 273 + 274 + [2-1-3] Flow of control 275 + 276 + EH through fine-grained callbacks start from scsi_unjam_host(). 277 + 278 + <<scsi_unjam_host>> 279 + 280 + 1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local 281 + eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is 282 + cleared by this action. 283 + 284 + 2. Invoke scsi_eh_get_sense. 285 + 286 + <<scsi_eh_get_sense>> 287 + 288 + This action is taken for each error-completed 289 + (!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most 290 + SCSI transports/LLDDs automatically acquire sense data on 291 + command failures (autosense). Autosense is recommended for 292 + performance reasons and as sense information could get out of 293 + sync inbetween occurrence of CHECK CONDITION and this action. 294 + 295 + Note that if autosense is not supported, scmd->sense_buffer 296 + contains invalid sense data when error-completing the scmd 297 + with scsi_done(). scsi_decide_disposition() always returns 298 + FAILED in such cases thus invoking SCSI EH. When the scmd 299 + reaches here, sense data is acquired and 300 + scsi_decide_disposition() is called again. 301 + 302 + 1. Invoke scsi_request_sense() which issues REQUEST_SENSE 303 + command. If fails, no action. Note that taking no action 304 + causes higher-severity recovery to be taken for the scmd. 305 + 306 + 2. Invoke scsi_decide_disposition() on the scmd 307 + 308 + - SUCCESS 309 + scmd->retries is set to scmd->allowed preventing 310 + scsi_eh_flush_done_q() from retrying the scmd and 311 + scsi_eh_finish_cmd() is invoked. 312 + 313 + - NEEDS_RETRY 314 + scsi_eh_finish_cmd() invoked 315 + 316 + - otherwise 317 + No action. 318 + 319 + 3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds(). 320 + 321 + <<scsi_eh_abort_cmds>> 322 + 323 + This action is taken for each timed out command. 324 + hostt->eh_abort_handler() is invoked for each scmd. The 325 + handler returns SUCCESS if it has succeeded to make LLDD and 326 + all related hardware forget about the scmd. 327 + 328 + If a timedout scmd is successfully aborted and the sdev is 329 + either offline or ready, scsi_eh_finish_cmd() is invoked for 330 + the scmd. Otherwise, the scmd is left in eh_work_q for 331 + higher-severity actions. 332 + 333 + Note that both offline and ready status mean that the sdev is 334 + ready to process new scmds, where processing also implies 335 + immediate failing; thus, if a sdev is in one of the two 336 + states, no further recovery action is needed. 337 + 338 + Device readiness is tested using scsi_eh_tur() which issues 339 + TEST_UNIT_READY command. Note that the scmd must have been 340 + aborted successfully before reusing it for TEST_UNIT_READY. 341 + 342 + 4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs() 343 + 344 + <<scsi_eh_ready_devs>> 345 + 346 + This function takes four increasingly more severe measures to 347 + make failed sdevs ready for new commands. 348 + 349 + 1. Invoke scsi_eh_stu() 350 + 351 + <<scsi_eh_stu>> 352 + 353 + For each sdev which has failed scmds with valid sense data 354 + of which scsi_check_sense()'s verdict is FAILED, 355 + START_STOP_UNIT command is issued w/ start=1. Note that 356 + as we explicitly choose error-completed scmds, it is known 357 + that lower layers have forgotten about the scmd and we can 358 + reuse it for STU. 359 + 360 + If STU succeeds and the sdev is either offline or ready, 361 + all failed scmds on the sdev are EH-finished with 362 + scsi_eh_finish_cmd(). 363 + 364 + *NOTE* If hostt->eh_abort_handler() isn't implemented or 365 + failed, we may still have timed out scmds at this point 366 + and STU doesn't make lower layers forget about those 367 + scmds. Yet, this function EH-finish all scmds on the sdev 368 + if STU succeeds leaving lower layers in an inconsistent 369 + state. It seems that STU action should be taken only when 370 + a sdev has no timed out scmd. 371 + 372 + 2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset(). 373 + 374 + <<scsi_eh_bus_device_reset>> 375 + 376 + This action is very similar to scsi_eh_stu() except that, 377 + instead of issuing STU, hostt->eh_device_reset_handler() 378 + is used. Also, as we're not issuing SCSI commands and 379 + resetting clears all scmds on the sdev, there is no need 380 + to choose error-completed scmds. 381 + 382 + 3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset() 383 + 384 + <<scsi_eh_bus_reset>> 385 + 386 + hostt->eh_bus_reset_handler() is invoked for each channel 387 + with failed scmds. If bus reset succeeds, all failed 388 + scmds on all ready or offline sdevs on the channel are 389 + EH-finished. 390 + 391 + 4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset() 392 + 393 + <<scsi_eh_host_reset>> 394 + 395 + This is the last resort. hostt->eh_host_reset_handler() 396 + is invoked. If host reset succeeds, all failed scmds on 397 + all ready or offline sdevs on the host are EH-finished. 398 + 399 + 5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs() 400 + 401 + <<scsi_eh_offline_sdevs>> 402 + 403 + Take all sdevs which still have unrecovered scmds offline 404 + and EH-finish the scmds. 405 + 406 + 5. Invoke scsi_eh_flush_done_q(). 407 + 408 + <<scsi_eh_flush_done_q>> 409 + 410 + At this point all scmds are recovered (or given up) and 411 + put on eh_done_q by scsi_eh_finish_cmd(). This function 412 + flushes eh_done_q by either retrying or notifying upper 413 + layer of failure of the scmds. 414 + 415 + 416 + [2-2] EH through hostt->eh_strategy_handler() 417 + 418 + hostt->eh_strategy_handler() is invoked in the place of 419 + scsi_unjam_host() and it is responsible for whole recovery process. 420 + On completion, the handler should have made lower layers forget about 421 + all failed scmds and either ready for new commands or offline. Also, 422 + it should perform SCSI EH maintenance choirs to maintain integrity of 423 + SCSI midlayer. IOW, of the steps described in [2-1-2], all steps 424 + except for #1 must be implemented by eh_strategy_handler(). 425 + 426 + 427 + [2-2-1] Pre hostt->eh_strategy_handler() SCSI midlayer conditions 428 + 429 + The following conditions are true on entry to the handler. 430 + 431 + - Each failed scmd's eh_flags field is set appropriately. 432 + 433 + - Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry. 434 + 435 + - SHOST_RECOVERY is set. 436 + 437 + - shost->host_failed == shost->host_busy 438 + 439 + 440 + [2-2-2] Post hostt->eh_strategy_handler() SCSI midlayer conditions 441 + 442 + The following conditions must be true on exit from the handler. 443 + 444 + - shost->host_failed is zero. 445 + 446 + - Each scmd's eh_eflags field is cleared. 447 + 448 + - Each scmd is in such a state that scsi_setup_cmd_retry() on the 449 + scmd doesn't make any difference. 450 + 451 + - shost->eh_cmd_q is cleared. 452 + 453 + - Each scmd->eh_entry is cleared. 454 + 455 + - Either scsi_queue_insert() or scsi_finish_command() is called on 456 + each scmd. Note that the handler is free to use scmd->retries and 457 + ->allowed to limit the number of retries. 458 + 459 + 460 + [2-2-3] Things to consider 461 + 462 + - Know that timed out scmds are still active on lower layers. Make 463 + lower layers forget about them before doing anything else with 464 + those scmds. 465 + 466 + - For consistency, when accessing/modifying shost data structure, 467 + grab shost->host_lock. 468 + 469 + - On completion, each failed sdev must have forgotten about all 470 + active scmds. 471 + 472 + - On completion, each failed sdev must be ready for new commands or 473 + offline. 474 + 475 + 476 + -- 477 + Tejun Heo 478 + htejun@gmail.com 479 + 11th September 2005