Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux

docs: fault-injection: convert docs to ReST and rename to *.rst

The conversion is actually:
- add blank lines and identation in order to identify paragraphs;
- fix tables markups;
- add some lists markups;
- mark literal blocks;
- adjust title markups.

At its new index.rst, let's add a :orphan: while this is not linked to
the main index.rst file, in order to avoid build warnings.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
Acked-by: Federico Vaga <federico.vaga@vaga.pv.it>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>

authored by

Mauro Carvalho Chehab and committed by
Jonathan Corbet
10ffebbe f0ba4377

+344 -295
+138 -127
Documentation/fault-injection/fault-injection.txt Documentation/fault-injection/fault-injection.rst
··· 1 + =========================================== 1 2 Fault injection capabilities infrastructure 2 3 =========================================== 3 4 ··· 8 7 Available fault injection capabilities 9 8 -------------------------------------- 10 9 11 - o failslab 10 + - failslab 12 11 13 12 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...) 14 13 15 - o fail_page_alloc 14 + - fail_page_alloc 16 15 17 16 injects page allocation failures. (alloc_pages(), get_free_pages(), ...) 18 17 19 - o fail_futex 18 + - fail_futex 20 19 21 20 injects futex deadlock and uaddr fault errors. 22 21 23 - o fail_make_request 22 + - fail_make_request 24 23 25 24 injects disk IO errors on devices permitted by setting 26 25 /sys/block/<device>/make-it-fail or 27 26 /sys/block/<device>/<partition>/make-it-fail. (generic_make_request()) 28 27 29 - o fail_mmc_request 28 + - fail_mmc_request 30 29 31 30 injects MMC data errors on devices permitted by setting 32 31 debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request 33 32 34 - o fail_function 33 + - fail_function 35 34 36 35 injects error return on specific functions, which are marked by 37 36 ALLOW_ERROR_INJECTION() macro, by setting debugfs entries 38 37 under /sys/kernel/debug/fail_function. No boot option supported. 39 38 40 - o NVMe fault injection 39 + - NVMe fault injection 41 40 42 41 inject NVMe status code and retry flag on devices permitted by setting 43 42 debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default ··· 48 47 Configure fault-injection capabilities behavior 49 48 ----------------------------------------------- 50 49 51 - o debugfs entries 50 + debugfs entries 51 + ^^^^^^^^^^^^^^^ 52 52 53 53 fault-inject-debugfs kernel module provides some debugfs entries for runtime 54 54 configuration of fault-injection capabilities. ··· 57 55 - /sys/kernel/debug/fail*/probability: 58 56 59 57 likelihood of failure injection, in percent. 58 + 60 59 Format: <percent> 61 60 62 61 Note that one-failure-per-hundred is a very high error rate ··· 86 83 - /sys/kernel/debug/fail*/verbose 87 84 88 85 Format: { 0 | 1 | 2 } 86 + 89 87 specifies the verbosity of the messages when failure is 90 88 injected. '0' means no messages; '1' will print only a single 91 89 log line per failure; '2' will print a call trace too -- useful ··· 95 91 - /sys/kernel/debug/fail*/task-filter: 96 92 97 93 Format: { 'Y' | 'N' } 94 + 98 95 A value of 'N' disables filtering by process (default). 99 96 Any positive value limits failures to only processes indicated by 100 97 /proc/<pid>/make-it-fail==1. 101 98 102 - - /sys/kernel/debug/fail*/require-start: 103 - - /sys/kernel/debug/fail*/require-end: 104 - - /sys/kernel/debug/fail*/reject-start: 105 - - /sys/kernel/debug/fail*/reject-end: 99 + - /sys/kernel/debug/fail*/require-start, 100 + /sys/kernel/debug/fail*/require-end, 101 + /sys/kernel/debug/fail*/reject-start, 102 + /sys/kernel/debug/fail*/reject-end: 106 103 107 104 specifies the range of virtual addresses tested during 108 105 stacktrace walking. Failure is injected only if some caller ··· 121 116 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: 122 117 123 118 Format: { 'Y' | 'N' } 119 + 124 120 default is 'N', setting it to 'Y' won't inject failures into 125 121 highmem/user allocations. 126 122 ··· 129 123 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: 130 124 131 125 Format: { 'Y' | 'N' } 126 + 132 127 default is 'N', setting it to 'Y' will inject failures 133 128 only into non-sleep allocations (GFP_ATOMIC allocations). 134 129 ··· 141 134 - /sys/kernel/debug/fail_futex/ignore-private: 142 135 143 136 Format: { 'Y' | 'N' } 137 + 144 138 default is 'N', setting it to 'Y' will disable failure injections 145 139 when dealing with private (address space) futexes. 146 140 147 141 - /sys/kernel/debug/fail_function/inject: 148 142 149 143 Format: { 'function-name' | '!function-name' | '' } 144 + 150 145 specifies the target function of error injection by name. 151 146 If the function name leads '!' prefix, given function is 152 147 removed from injection list. If nothing specified ('') ··· 169 160 function for given function. This will be created when 170 161 user specifies new injection entry. 171 162 172 - o Boot option 163 + Boot option 164 + ^^^^^^^^^^^ 173 165 174 166 In order to inject faults while debugfs is not available (early boot time), 175 - use the boot option: 167 + use the boot option:: 176 168 177 169 failslab= 178 170 fail_page_alloc= ··· 181 171 fail_futex= 182 172 mmc_core.fail_request=<interval>,<probability>,<space>,<times> 183 173 184 - o proc entries 174 + proc entries 175 + ^^^^^^^^^^^^ 185 176 186 - - /proc/<pid>/fail-nth: 187 - - /proc/self/task/<tid>/fail-nth: 177 + - /proc/<pid>/fail-nth, 178 + /proc/self/task/<tid>/fail-nth: 188 179 189 180 Write to this file of integer N makes N-th call in the task fail. 190 181 Read from this file returns a integer value. A value of '0' indicates ··· 202 191 How to add new fault injection capability 203 192 ----------------------------------------- 204 193 205 - o #include <linux/fault-inject.h> 194 + - #include <linux/fault-inject.h> 206 195 207 - o define the fault attributes 196 + - define the fault attributes 208 197 209 198 DECLARE_FAULT_ATTR(name); 210 199 211 200 Please see the definition of struct fault_attr in fault-inject.h 212 201 for details. 213 202 214 - o provide a way to configure fault attributes 203 + - provide a way to configure fault attributes 215 204 216 205 - boot option 217 206 ··· 233 222 single kernel module, it is better to provide module parameters to 234 223 configure the fault attributes. 235 224 236 - o add a hook to insert failures 225 + - add a hook to insert failures 237 226 238 - Upon should_fail() returning true, client code should inject a failure. 227 + Upon should_fail() returning true, client code should inject a failure: 239 228 240 229 should_fail(attr, size); 241 230 242 231 Application Examples 243 232 -------------------- 244 233 245 - o Inject slab allocation failures into module init/exit code 234 + - Inject slab allocation failures into module init/exit code:: 246 235 247 - #!/bin/bash 236 + #!/bin/bash 248 237 249 - FAILTYPE=failslab 250 - echo Y > /sys/kernel/debug/$FAILTYPE/task-filter 251 - echo 10 > /sys/kernel/debug/$FAILTYPE/probability 252 - echo 100 > /sys/kernel/debug/$FAILTYPE/interval 253 - echo -1 > /sys/kernel/debug/$FAILTYPE/times 254 - echo 0 > /sys/kernel/debug/$FAILTYPE/space 255 - echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 256 - echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 238 + FAILTYPE=failslab 239 + echo Y > /sys/kernel/debug/$FAILTYPE/task-filter 240 + echo 10 > /sys/kernel/debug/$FAILTYPE/probability 241 + echo 100 > /sys/kernel/debug/$FAILTYPE/interval 242 + echo -1 > /sys/kernel/debug/$FAILTYPE/times 243 + echo 0 > /sys/kernel/debug/$FAILTYPE/space 244 + echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 245 + echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 257 246 258 - faulty_system() 259 - { 247 + faulty_system() 248 + { 260 249 bash -c "echo 1 > /proc/self/make-it-fail && exec $*" 261 - } 250 + } 262 251 263 - if [ $# -eq 0 ] 264 - then 252 + if [ $# -eq 0 ] 253 + then 265 254 echo "Usage: $0 modulename [ modulename ... ]" 266 255 exit 1 267 - fi 256 + fi 268 257 269 - for m in $* 270 - do 258 + for m in $* 259 + do 271 260 echo inserting $m... 272 261 faulty_system modprobe $m 273 262 274 263 echo removing $m... 275 264 faulty_system modprobe -r $m 276 - done 265 + done 277 266 278 267 ------------------------------------------------------------------------------ 279 268 280 - o Inject page allocation failures only for a specific module 269 + - Inject page allocation failures only for a specific module:: 281 270 282 - #!/bin/bash 271 + #!/bin/bash 283 272 284 - FAILTYPE=fail_page_alloc 285 - module=$1 273 + FAILTYPE=fail_page_alloc 274 + module=$1 286 275 287 - if [ -z $module ] 288 - then 276 + if [ -z $module ] 277 + then 289 278 echo "Usage: $0 <modulename>" 290 279 exit 1 291 - fi 280 + fi 292 281 293 - modprobe $module 282 + modprobe $module 294 283 295 - if [ ! -d /sys/module/$module/sections ] 296 - then 284 + if [ ! -d /sys/module/$module/sections ] 285 + then 297 286 echo Module $module is not loaded 298 287 exit 1 299 - fi 288 + fi 300 289 301 - cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start 302 - cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end 290 + cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start 291 + cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end 303 292 304 - echo N > /sys/kernel/debug/$FAILTYPE/task-filter 305 - echo 10 > /sys/kernel/debug/$FAILTYPE/probability 306 - echo 100 > /sys/kernel/debug/$FAILTYPE/interval 307 - echo -1 > /sys/kernel/debug/$FAILTYPE/times 308 - echo 0 > /sys/kernel/debug/$FAILTYPE/space 309 - echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 310 - echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 311 - echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem 312 - echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth 293 + echo N > /sys/kernel/debug/$FAILTYPE/task-filter 294 + echo 10 > /sys/kernel/debug/$FAILTYPE/probability 295 + echo 100 > /sys/kernel/debug/$FAILTYPE/interval 296 + echo -1 > /sys/kernel/debug/$FAILTYPE/times 297 + echo 0 > /sys/kernel/debug/$FAILTYPE/space 298 + echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 299 + echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 300 + echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem 301 + echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth 313 302 314 - trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT 303 + trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT 315 304 316 - echo "Injecting errors into the module $module... (interrupt to stop)" 317 - sleep 1000000 305 + echo "Injecting errors into the module $module... (interrupt to stop)" 306 + sleep 1000000 318 307 319 308 ------------------------------------------------------------------------------ 320 309 321 - o Inject open_ctree error while btrfs mount 310 + - Inject open_ctree error while btrfs mount:: 322 311 323 - #!/bin/bash 312 + #!/bin/bash 324 313 325 - rm -f testfile.img 326 - dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1 327 - DEVICE=$(losetup --show -f testfile.img) 328 - mkfs.btrfs -f $DEVICE 329 - mkdir -p tmpmnt 314 + rm -f testfile.img 315 + dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1 316 + DEVICE=$(losetup --show -f testfile.img) 317 + mkfs.btrfs -f $DEVICE 318 + mkdir -p tmpmnt 330 319 331 - FAILTYPE=fail_function 332 - FAILFUNC=open_ctree 333 - echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject 334 - echo -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval 335 - echo N > /sys/kernel/debug/$FAILTYPE/task-filter 336 - echo 100 > /sys/kernel/debug/$FAILTYPE/probability 337 - echo 0 > /sys/kernel/debug/$FAILTYPE/interval 338 - echo -1 > /sys/kernel/debug/$FAILTYPE/times 339 - echo 0 > /sys/kernel/debug/$FAILTYPE/space 340 - echo 1 > /sys/kernel/debug/$FAILTYPE/verbose 320 + FAILTYPE=fail_function 321 + FAILFUNC=open_ctree 322 + echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject 323 + echo -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval 324 + echo N > /sys/kernel/debug/$FAILTYPE/task-filter 325 + echo 100 > /sys/kernel/debug/$FAILTYPE/probability 326 + echo 0 > /sys/kernel/debug/$FAILTYPE/interval 327 + echo -1 > /sys/kernel/debug/$FAILTYPE/times 328 + echo 0 > /sys/kernel/debug/$FAILTYPE/space 329 + echo 1 > /sys/kernel/debug/$FAILTYPE/verbose 341 330 342 - mount -t btrfs $DEVICE tmpmnt 343 - if [ $? -ne 0 ] 344 - then 331 + mount -t btrfs $DEVICE tmpmnt 332 + if [ $? -ne 0 ] 333 + then 345 334 echo "SUCCESS!" 346 - else 335 + else 347 336 echo "FAILED!" 348 337 umount tmpmnt 349 - fi 338 + fi 350 339 351 - echo > /sys/kernel/debug/$FAILTYPE/inject 340 + echo > /sys/kernel/debug/$FAILTYPE/inject 352 341 353 - rmdir tmpmnt 354 - losetup -d $DEVICE 355 - rm testfile.img 342 + rmdir tmpmnt 343 + losetup -d $DEVICE 344 + rm testfile.img 356 345 357 346 358 347 Tool to run command with failslab or fail_page_alloc ··· 365 354 Examples: 366 355 367 356 Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab 368 - allocation failure. 357 + allocation failure:: 369 358 370 359 # ./tools/testing/fault-injection/failcmd.sh \ 371 360 -- make -C tools/testing/selftests/ run_tests 372 361 373 362 Same as above except to specify 100 times failures at most instead of one time 374 - at most by default. 363 + at most by default:: 375 364 376 365 # ./tools/testing/fault-injection/failcmd.sh --times=100 \ 377 366 -- make -C tools/testing/selftests/ run_tests 378 367 379 368 Same as above except to inject page allocation failure instead of slab 380 - allocation failure. 369 + allocation failure:: 381 370 382 371 # env FAILCMD_TYPE=fail_page_alloc \ 383 372 ./tools/testing/fault-injection/failcmd.sh --times=100 \ 384 - -- make -C tools/testing/selftests/ run_tests 373 + -- make -C tools/testing/selftests/ run_tests 385 374 386 375 Systematic faults using fail-nth 387 376 --------------------------------- 388 377 389 378 The following code systematically faults 0-th, 1-st, 2-nd and so on 390 - capabilities in the socketpair() system call. 379 + capabilities in the socketpair() system call:: 391 380 392 - #include <sys/types.h> 393 - #include <sys/stat.h> 394 - #include <sys/socket.h> 395 - #include <sys/syscall.h> 396 - #include <fcntl.h> 397 - #include <unistd.h> 398 - #include <string.h> 399 - #include <stdlib.h> 400 - #include <stdio.h> 401 - #include <errno.h> 381 + #include <sys/types.h> 382 + #include <sys/stat.h> 383 + #include <sys/socket.h> 384 + #include <sys/syscall.h> 385 + #include <fcntl.h> 386 + #include <unistd.h> 387 + #include <string.h> 388 + #include <stdlib.h> 389 + #include <stdio.h> 390 + #include <errno.h> 402 391 403 - int main() 404 - { 392 + int main() 393 + { 405 394 int i, err, res, fail_nth, fds[2]; 406 395 char buf[128]; 407 396 ··· 424 413 break; 425 414 } 426 415 return 0; 427 - } 416 + } 428 417 429 - An example output: 418 + An example output:: 430 419 431 - 1-th fault Y: res=-1/23 432 - 2-th fault Y: res=-1/23 433 - 3-th fault Y: res=-1/12 434 - 4-th fault Y: res=-1/12 435 - 5-th fault Y: res=-1/23 436 - 6-th fault Y: res=-1/23 437 - 7-th fault Y: res=-1/23 438 - 8-th fault Y: res=-1/12 439 - 9-th fault Y: res=-1/12 440 - 10-th fault Y: res=-1/12 441 - 11-th fault Y: res=-1/12 442 - 12-th fault Y: res=-1/12 443 - 13-th fault Y: res=-1/12 444 - 14-th fault Y: res=-1/12 445 - 15-th fault Y: res=-1/12 446 - 16-th fault N: res=0/12 420 + 1-th fault Y: res=-1/23 421 + 2-th fault Y: res=-1/23 422 + 3-th fault Y: res=-1/12 423 + 4-th fault Y: res=-1/12 424 + 5-th fault Y: res=-1/23 425 + 6-th fault Y: res=-1/23 426 + 7-th fault Y: res=-1/23 427 + 8-th fault Y: res=-1/12 428 + 9-th fault Y: res=-1/12 429 + 10-th fault Y: res=-1/12 430 + 11-th fault Y: res=-1/12 431 + 12-th fault Y: res=-1/12 432 + 13-th fault Y: res=-1/12 433 + 14-th fault Y: res=-1/12 434 + 15-th fault Y: res=-1/12 435 + 16-th fault N: res=0/12
+20
Documentation/fault-injection/index.rst
··· 1 + :orphan: 2 + 3 + =============== 4 + fault-injection 5 + =============== 6 + 7 + .. toctree:: 8 + :maxdepth: 1 9 + 10 + fault-injection 11 + notifier-error-inject 12 + nvme-fault-injection 13 + provoke-crashes 14 + 15 + .. only:: subproject and html 16 + 17 + Indices 18 + ======= 19 + 20 + * :ref:`genindex`
+11 -7
Documentation/fault-injection/notifier-error-inject.txt Documentation/fault-injection/notifier-error-inject.rst
··· 14 14 PM notifier error injection module 15 15 ---------------------------------- 16 16 This feature is controlled through debugfs interface 17 - /sys/kernel/debug/notifier-error-inject/pm/actions/<notifier event>/error 17 + 18 + /sys/kernel/debug/notifier-error-inject/pm/actions/<notifier event>/error 18 19 19 20 Possible PM notifier events to be failed are: 20 21 ··· 23 22 * PM_SUSPEND_PREPARE 24 23 * PM_RESTORE_PREPARE 25 24 26 - Example: Inject PM suspend error (-12 = -ENOMEM) 25 + Example: Inject PM suspend error (-12 = -ENOMEM):: 27 26 28 27 # cd /sys/kernel/debug/notifier-error-inject/pm/ 29 28 # echo -12 > actions/PM_SUSPEND_PREPARE/error ··· 33 32 Memory hotplug notifier error injection module 34 33 ---------------------------------------------- 35 34 This feature is controlled through debugfs interface 36 - /sys/kernel/debug/notifier-error-inject/memory/actions/<notifier event>/error 35 + 36 + /sys/kernel/debug/notifier-error-inject/memory/actions/<notifier event>/error 37 37 38 38 Possible memory notifier events to be failed are: 39 39 40 40 * MEM_GOING_ONLINE 41 41 * MEM_GOING_OFFLINE 42 42 43 - Example: Inject memory hotplug offline error (-12 == -ENOMEM) 43 + Example: Inject memory hotplug offline error (-12 == -ENOMEM):: 44 44 45 45 # cd /sys/kernel/debug/notifier-error-inject/memory 46 46 # echo -12 > actions/MEM_GOING_OFFLINE/error ··· 51 49 powerpc pSeries reconfig notifier error injection module 52 50 -------------------------------------------------------- 53 51 This feature is controlled through debugfs interface 54 - /sys/kernel/debug/notifier-error-inject/pSeries-reconfig/actions/<notifier event>/error 52 + 53 + /sys/kernel/debug/notifier-error-inject/pSeries-reconfig/actions/<notifier event>/error 55 54 56 55 Possible pSeries reconfig notifier events to be failed are: 57 56 ··· 64 61 Netdevice notifier error injection module 65 62 ---------------------------------------------- 66 63 This feature is controlled through debugfs interface 67 - /sys/kernel/debug/notifier-error-inject/netdev/actions/<notifier event>/error 64 + 65 + /sys/kernel/debug/notifier-error-inject/netdev/actions/<notifier event>/error 68 66 69 67 Netdevice notifier events which can be failed are: 70 68 ··· 79 75 * NETDEV_PRECHANGEUPPER 80 76 * NETDEV_CHANGEUPPER 81 77 82 - Example: Inject netdevice mtu change error (-22 == -EINVAL) 78 + Example: Inject netdevice mtu change error (-22 == -EINVAL):: 83 79 84 80 # cd /sys/kernel/debug/notifier-error-inject/netdev 85 81 # echo -22 > actions/NETDEV_CHANGEMTU/error
+120
Documentation/fault-injection/nvme-fault-injection.rst
··· 1 + NVMe Fault Injection 2 + ==================== 3 + Linux's fault injection framework provides a systematic way to support 4 + error injection via debugfs in the /sys/kernel/debug directory. When 5 + enabled, the default NVME_SC_INVALID_OPCODE with no retry will be 6 + injected into the nvme_end_request. Users can change the default status 7 + code and no retry flag via the debugfs. The list of Generic Command 8 + Status can be found in include/linux/nvme.h 9 + 10 + Following examples show how to inject an error into the nvme. 11 + 12 + First, enable CONFIG_FAULT_INJECTION_DEBUG_FS kernel config, 13 + recompile the kernel. After booting up the kernel, do the 14 + following. 15 + 16 + Example 1: Inject default status code with no retry 17 + --------------------------------------------------- 18 + 19 + :: 20 + 21 + mount /dev/nvme0n1 /mnt 22 + echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times 23 + echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability 24 + cp a.file /mnt 25 + 26 + Expected Result:: 27 + 28 + cp: cannot stat ‘/mnt/a.file’: Input/output error 29 + 30 + Message from dmesg:: 31 + 32 + FAULT_INJECTION: forcing a failure. 33 + name fault_inject, interval 1, probability 100, space 0, times 1 34 + CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc8+ #2 35 + Hardware name: innotek GmbH VirtualBox/VirtualBox, 36 + BIOS VirtualBox 12/01/2006 37 + Call Trace: 38 + <IRQ> 39 + dump_stack+0x5c/0x7d 40 + should_fail+0x148/0x170 41 + nvme_should_fail+0x2f/0x50 [nvme_core] 42 + nvme_process_cq+0xe7/0x1d0 [nvme] 43 + nvme_irq+0x1e/0x40 [nvme] 44 + __handle_irq_event_percpu+0x3a/0x190 45 + handle_irq_event_percpu+0x30/0x70 46 + handle_irq_event+0x36/0x60 47 + handle_fasteoi_irq+0x78/0x120 48 + handle_irq+0xa7/0x130 49 + ? tick_irq_enter+0xa8/0xc0 50 + do_IRQ+0x43/0xc0 51 + common_interrupt+0xa2/0xa2 52 + </IRQ> 53 + RIP: 0010:native_safe_halt+0x2/0x10 54 + RSP: 0018:ffffffff82003e90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdd 55 + RAX: ffffffff817a10c0 RBX: ffffffff82012480 RCX: 0000000000000000 56 + RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 57 + RBP: 0000000000000000 R08: 000000008e38ce64 R09: 0000000000000000 58 + R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff82012480 59 + R13: ffffffff82012480 R14: 0000000000000000 R15: 0000000000000000 60 + ? __sched_text_end+0x4/0x4 61 + default_idle+0x18/0xf0 62 + do_idle+0x150/0x1d0 63 + cpu_startup_entry+0x6f/0x80 64 + start_kernel+0x4c4/0x4e4 65 + ? set_init_arg+0x55/0x55 66 + secondary_startup_64+0xa5/0xb0 67 + print_req_error: I/O error, dev nvme0n1, sector 9240 68 + EXT4-fs error (device nvme0n1): ext4_find_entry:1436: 69 + inode #2: comm cp: reading directory lblock 0 70 + 71 + Example 2: Inject default status code with retry 72 + ------------------------------------------------ 73 + 74 + :: 75 + 76 + mount /dev/nvme0n1 /mnt 77 + echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times 78 + echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability 79 + echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/status 80 + echo 0 > /sys/kernel/debug/nvme0n1/fault_inject/dont_retry 81 + 82 + cp a.file /mnt 83 + 84 + Expected Result:: 85 + 86 + command success without error 87 + 88 + Message from dmesg:: 89 + 90 + FAULT_INJECTION: forcing a failure. 91 + name fault_inject, interval 1, probability 100, space 0, times 1 92 + CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.15.0-rc8+ #4 93 + Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 94 + Call Trace: 95 + <IRQ> 96 + dump_stack+0x5c/0x7d 97 + should_fail+0x148/0x170 98 + nvme_should_fail+0x30/0x60 [nvme_core] 99 + nvme_loop_queue_response+0x84/0x110 [nvme_loop] 100 + nvmet_req_complete+0x11/0x40 [nvmet] 101 + nvmet_bio_done+0x28/0x40 [nvmet] 102 + blk_update_request+0xb0/0x310 103 + blk_mq_end_request+0x18/0x60 104 + flush_smp_call_function_queue+0x3d/0xf0 105 + smp_call_function_single_interrupt+0x2c/0xc0 106 + call_function_single_interrupt+0xa2/0xb0 107 + </IRQ> 108 + RIP: 0010:native_safe_halt+0x2/0x10 109 + RSP: 0018:ffffc9000068bec0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 110 + RAX: ffffffff817a10c0 RBX: ffff88011a3c9680 RCX: 0000000000000000 111 + RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 112 + RBP: 0000000000000001 R08: 000000008e38c131 R09: 0000000000000000 113 + R10: 0000000000000000 R11: 0000000000000000 R12: ffff88011a3c9680 114 + R13: ffff88011a3c9680 R14: 0000000000000000 R15: 0000000000000000 115 + ? __sched_text_end+0x4/0x4 116 + default_idle+0x18/0xf0 117 + do_idle+0x150/0x1d0 118 + cpu_startup_entry+0x6f/0x80 119 + start_secondary+0x187/0x1e0 120 + secondary_startup_64+0xa5/0xb0
-116
Documentation/fault-injection/nvme-fault-injection.txt
··· 1 - NVMe Fault Injection 2 - ==================== 3 - Linux's fault injection framework provides a systematic way to support 4 - error injection via debugfs in the /sys/kernel/debug directory. When 5 - enabled, the default NVME_SC_INVALID_OPCODE with no retry will be 6 - injected into the nvme_end_request. Users can change the default status 7 - code and no retry flag via the debugfs. The list of Generic Command 8 - Status can be found in include/linux/nvme.h 9 - 10 - Following examples show how to inject an error into the nvme. 11 - 12 - First, enable CONFIG_FAULT_INJECTION_DEBUG_FS kernel config, 13 - recompile the kernel. After booting up the kernel, do the 14 - following. 15 - 16 - Example 1: Inject default status code with no retry 17 - --------------------------------------------------- 18 - 19 - mount /dev/nvme0n1 /mnt 20 - echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times 21 - echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability 22 - cp a.file /mnt 23 - 24 - Expected Result: 25 - 26 - cp: cannot stat ‘/mnt/a.file’: Input/output error 27 - 28 - Message from dmesg: 29 - 30 - FAULT_INJECTION: forcing a failure. 31 - name fault_inject, interval 1, probability 100, space 0, times 1 32 - CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc8+ #2 33 - Hardware name: innotek GmbH VirtualBox/VirtualBox, 34 - BIOS VirtualBox 12/01/2006 35 - Call Trace: 36 - <IRQ> 37 - dump_stack+0x5c/0x7d 38 - should_fail+0x148/0x170 39 - nvme_should_fail+0x2f/0x50 [nvme_core] 40 - nvme_process_cq+0xe7/0x1d0 [nvme] 41 - nvme_irq+0x1e/0x40 [nvme] 42 - __handle_irq_event_percpu+0x3a/0x190 43 - handle_irq_event_percpu+0x30/0x70 44 - handle_irq_event+0x36/0x60 45 - handle_fasteoi_irq+0x78/0x120 46 - handle_irq+0xa7/0x130 47 - ? tick_irq_enter+0xa8/0xc0 48 - do_IRQ+0x43/0xc0 49 - common_interrupt+0xa2/0xa2 50 - </IRQ> 51 - RIP: 0010:native_safe_halt+0x2/0x10 52 - RSP: 0018:ffffffff82003e90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdd 53 - RAX: ffffffff817a10c0 RBX: ffffffff82012480 RCX: 0000000000000000 54 - RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 55 - RBP: 0000000000000000 R08: 000000008e38ce64 R09: 0000000000000000 56 - R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff82012480 57 - R13: ffffffff82012480 R14: 0000000000000000 R15: 0000000000000000 58 - ? __sched_text_end+0x4/0x4 59 - default_idle+0x18/0xf0 60 - do_idle+0x150/0x1d0 61 - cpu_startup_entry+0x6f/0x80 62 - start_kernel+0x4c4/0x4e4 63 - ? set_init_arg+0x55/0x55 64 - secondary_startup_64+0xa5/0xb0 65 - print_req_error: I/O error, dev nvme0n1, sector 9240 66 - EXT4-fs error (device nvme0n1): ext4_find_entry:1436: 67 - inode #2: comm cp: reading directory lblock 0 68 - 69 - Example 2: Inject default status code with retry 70 - ------------------------------------------------ 71 - 72 - mount /dev/nvme0n1 /mnt 73 - echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times 74 - echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability 75 - echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/status 76 - echo 0 > /sys/kernel/debug/nvme0n1/fault_inject/dont_retry 77 - 78 - cp a.file /mnt 79 - 80 - Expected Result: 81 - 82 - command success without error 83 - 84 - Message from dmesg: 85 - 86 - FAULT_INJECTION: forcing a failure. 87 - name fault_inject, interval 1, probability 100, space 0, times 1 88 - CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.15.0-rc8+ #4 89 - Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 90 - Call Trace: 91 - <IRQ> 92 - dump_stack+0x5c/0x7d 93 - should_fail+0x148/0x170 94 - nvme_should_fail+0x30/0x60 [nvme_core] 95 - nvme_loop_queue_response+0x84/0x110 [nvme_loop] 96 - nvmet_req_complete+0x11/0x40 [nvmet] 97 - nvmet_bio_done+0x28/0x40 [nvmet] 98 - blk_update_request+0xb0/0x310 99 - blk_mq_end_request+0x18/0x60 100 - flush_smp_call_function_queue+0x3d/0xf0 101 - smp_call_function_single_interrupt+0x2c/0xc0 102 - call_function_single_interrupt+0xa2/0xb0 103 - </IRQ> 104 - RIP: 0010:native_safe_halt+0x2/0x10 105 - RSP: 0018:ffffc9000068bec0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 106 - RAX: ffffffff817a10c0 RBX: ffff88011a3c9680 RCX: 0000000000000000 107 - RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 108 - RBP: 0000000000000001 R08: 000000008e38c131 R09: 0000000000000000 109 - R10: 0000000000000000 R11: 0000000000000000 R12: ffff88011a3c9680 110 - R13: ffff88011a3c9680 R14: 0000000000000000 R15: 0000000000000000 111 - ? __sched_text_end+0x4/0x4 112 - default_idle+0x18/0xf0 113 - do_idle+0x150/0x1d0 114 - cpu_startup_entry+0x6f/0x80 115 - start_secondary+0x187/0x1e0 116 - secondary_startup_64+0xa5/0xb0
+48
Documentation/fault-injection/provoke-crashes.rst
··· 1 + =============== 2 + Provoke crashes 3 + =============== 4 + 5 + The lkdtm module provides an interface to crash or injure the kernel at 6 + predefined crashpoints to evaluate the reliability of crash dumps obtained 7 + using different dumping solutions. The module uses KPROBEs to instrument 8 + crashing points, but can also crash the kernel directly without KRPOBE 9 + support. 10 + 11 + 12 + You can provide the way either through module arguments when inserting 13 + the module, or through a debugfs interface. 14 + 15 + Usage:: 16 + 17 + insmod lkdtm.ko [recur_count={>0}] cpoint_name=<> cpoint_type=<> 18 + [cpoint_count={>0}] 19 + 20 + recur_count 21 + Recursion level for the stack overflow test. Default is 10. 22 + 23 + cpoint_name 24 + Crash point where the kernel is to be crashed. It can be 25 + one of INT_HARDWARE_ENTRY, INT_HW_IRQ_EN, INT_TASKLET_ENTRY, 26 + FS_DEVRW, MEM_SWAPOUT, TIMERADD, SCSI_DISPATCH_CMD, 27 + IDE_CORE_CP, DIRECT 28 + 29 + cpoint_type 30 + Indicates the action to be taken on hitting the crash point. 31 + It can be one of PANIC, BUG, EXCEPTION, LOOP, OVERFLOW, 32 + CORRUPT_STACK, UNALIGNED_LOAD_STORE_WRITE, OVERWRITE_ALLOCATION, 33 + WRITE_AFTER_FREE, 34 + 35 + cpoint_count 36 + Indicates the number of times the crash point is to be hit 37 + to trigger an action. The default is 10. 38 + 39 + You can also induce failures by mounting debugfs and writing the type to 40 + <mountpoint>/provoke-crash/<crashpoint>. E.g.:: 41 + 42 + mount -t debugfs debugfs /mnt 43 + echo EXCEPTION > /mnt/provoke-crash/INT_HARDWARE_ENTRY 44 + 45 + 46 + A special file is `DIRECT` which will induce the crash directly without 47 + KPROBE instrumentation. This mode is the only one available when the module 48 + is built on a kernel without KPROBEs support.
-38
Documentation/fault-injection/provoke-crashes.txt
··· 1 - The lkdtm module provides an interface to crash or injure the kernel at 2 - predefined crashpoints to evaluate the reliability of crash dumps obtained 3 - using different dumping solutions. The module uses KPROBEs to instrument 4 - crashing points, but can also crash the kernel directly without KRPOBE 5 - support. 6 - 7 - 8 - You can provide the way either through module arguments when inserting 9 - the module, or through a debugfs interface. 10 - 11 - Usage: insmod lkdtm.ko [recur_count={>0}] cpoint_name=<> cpoint_type=<> 12 - [cpoint_count={>0}] 13 - 14 - recur_count : Recursion level for the stack overflow test. Default is 10. 15 - 16 - cpoint_name : Crash point where the kernel is to be crashed. It can be 17 - one of INT_HARDWARE_ENTRY, INT_HW_IRQ_EN, INT_TASKLET_ENTRY, 18 - FS_DEVRW, MEM_SWAPOUT, TIMERADD, SCSI_DISPATCH_CMD, 19 - IDE_CORE_CP, DIRECT 20 - 21 - cpoint_type : Indicates the action to be taken on hitting the crash point. 22 - It can be one of PANIC, BUG, EXCEPTION, LOOP, OVERFLOW, 23 - CORRUPT_STACK, UNALIGNED_LOAD_STORE_WRITE, OVERWRITE_ALLOCATION, 24 - WRITE_AFTER_FREE, 25 - 26 - cpoint_count : Indicates the number of times the crash point is to be hit 27 - to trigger an action. The default is 10. 28 - 29 - You can also induce failures by mounting debugfs and writing the type to 30 - <mountpoint>/provoke-crash/<crashpoint>. E.g., 31 - 32 - mount -t debugfs debugfs /mnt 33 - echo EXCEPTION > /mnt/provoke-crash/INT_HARDWARE_ENTRY 34 - 35 - 36 - A special file is `DIRECT' which will induce the crash directly without 37 - KPROBE instrumentation. This mode is the only one available when the module 38 - is built on a kernel without KPROBEs support.
+1 -1
Documentation/process/4.Coding.rst
··· 298 298 fail; these failures can be restricted to a specific range of code. 299 299 Running with fault injection enabled allows the programmer to see how the 300 300 code responds when things go badly. See 301 - Documentation/fault-injection/fault-injection.txt for more information on 301 + Documentation/fault-injection/fault-injection.rst for more information on 302 302 how to use this facility. 303 303 304 304 Other kinds of errors can be found with the "sparse" static analysis tool.
+1 -1
Documentation/translations/it_IT/process/4.Coding.rst
··· 314 314 possono essere ridotti ad uno specifico pezzo di codice. Procedere con 315 315 l'inserimento dei fallimenti attivo permette al programmatore di verificare 316 316 come il codice risponde quando le cose vanno male. Consultate: 317 - Documentation/fault-injection/fault-injection.txt per avere maggiori 317 + Documentation/fault-injection/fault-injection.rst per avere maggiori 318 318 informazioni su come utilizzare questo strumento. 319 319 320 320 Altre tipologie di errori possono essere riscontrati con lo strumento di
+1 -1
Documentation/translations/zh_CN/process/4.Coding.rst
··· 205 205 启用故障注入后,内存分配的可配置百分比将失败;这些失败可以限制在特定的代码 206 206 范围内。在启用了故障注入的情况下运行,程序员可以看到当情况恶化时代码如何响 207 207 应。有关如何使用此工具的详细信息,请参阅 208 - Documentation/fault-injection/fault-injection.txt。 208 + Documentation/fault-injection/fault-injection.rst。 209 209 210 210 使用“sparse”静态分析工具可以发现其他类型的错误。对于sparse,可以警告程序员 211 211 用户空间和内核空间地址之间的混淆、big endian和small endian数量的混合、在需
+1 -1
drivers/misc/lkdtm/core.c
··· 15 15 * 16 16 * Debugfs support added by Simon Kagstrom <simon.kagstrom@netinsight.net> 17 17 * 18 - * See Documentation/fault-injection/provoke-crashes.txt for instructions 18 + * See Documentation/fault-injection/provoke-crashes.rst for instructions 19 19 */ 20 20 #include "lkdtm.h" 21 21 #include <linux/fs.h>
+1 -1
include/linux/fault-inject.h
··· 11 11 12 12 /* 13 13 * For explanation of the elements of this struct, see 14 - * Documentation/fault-injection/fault-injection.txt 14 + * Documentation/fault-injection/fault-injection.rst 15 15 */ 16 16 struct fault_attr { 17 17 unsigned long probability;
+1 -1
lib/Kconfig.debug
··· 1701 1701 called lkdtm. 1702 1702 1703 1703 Documentation on how to use the module can be found in 1704 - Documentation/fault-injection/provoke-crashes.txt 1704 + Documentation/fault-injection/provoke-crashes.rst 1705 1705 1706 1706 config TEST_LIST_SORT 1707 1707 tristate "Linked list sorting test"
+1 -1
tools/testing/fault-injection/failcmd.sh
··· 42 42 --interval=value, --space=value, --verbose=value, --task-filter=value, 43 43 --stacktrace-depth=value, --require-start=value, --require-end=value, 44 44 --reject-start=value, --reject-end=value, --ignore-gfp-wait=value 45 - See Documentation/fault-injection/fault-injection.txt for more 45 + See Documentation/fault-injection/fault-injection.rst for more 46 46 information 47 47 48 48 failslab options: