lol

nixos/test-driver: restructure error classes

After a discussion with tfc, we agreed that we need a distinction
between errors where the user isn't at fault (e.g. OCR failing - now
called `MachineError`) and errors where the test actually failed (now
called `RequestedAssertionFailed`).

Both get special treatment from the error handler, i.e. a `!!!` prefix
to make it easier to spot visually.

However, only `RequestedAssertionFailed` gets the shortening of the
traceback, `MachineError` exceptions may be something to report and
maintainers usually want to see the full trace.

Suggested-by: Jacek Galowicz <jacek@galowicz.de>

+23 -22
+6 -2
nixos/lib/test-driver/src/test_driver/driver.py
··· 11 11 from typing import Any 12 12 from unittest import TestCase 13 13 14 - from test_driver.errors import RequestedAssertionFailed, TestScriptError 14 + from test_driver.errors import MachineError, RequestedAssertionFailed 15 15 from test_driver.logger import AbstractLogger 16 16 from test_driver.machine import Machine, NixStartScript, retry 17 17 from test_driver.polling_condition import PollingCondition ··· 182 182 symbols = self.test_symbols() # call eagerly 183 183 try: 184 184 exec(self.tests, symbols, None) 185 - except TestScriptError: 185 + except MachineError: 186 + for line in traceback.format_exc().splitlines(): 187 + self.logger.log_test_error(line) 188 + sys.exit(1) 189 + except RequestedAssertionFailed: 186 190 exc_type, exc, tb = sys.exc_info() 187 191 filtered = [ 188 192 frame
+12 -15
nixos/lib/test-driver/src/test_driver/errors.py
··· 1 - class TestScriptError(Exception): 1 + class MachineError(Exception): 2 2 """ 3 - The base error class to indicate that the test script failed. 4 - This (and its subclasses) get special treatment, i.e. only stack 5 - frames from `testScript` are printed and the error gets prefixed 6 - with `!!!` to make it easier to spot between other log-lines. 3 + Exception that indicates an error that is NOT the user's fault, 4 + i.e. something went wrong without the test being necessarily invalid, 5 + such as failing OCR. 7 6 8 - This class is used for errors that aren't an actual test failure, 9 - but also not a bug in the driver, e.g. failing OCR. 7 + To make it easier to spot, this exception (and its subclasses) 8 + get a `!!!` prefix in the log output. 10 9 """ 11 10 12 11 13 - class RequestedAssertionFailed(TestScriptError): 12 + class RequestedAssertionFailed(AssertionError): 14 13 """ 15 - Subclass of `TestScriptError` that gets special treatment. 14 + Special assertion that gets thrown on an assertion error, 15 + e.g. a failing `t.assertEqual(...)` or `machine.succeed(...)`. 16 16 17 - Exception raised when a requested assertion fails, 18 - e.g. `machine.succeed(...)` or `t.assertEqual(...)`. 19 - 20 - This is separate from the base error class, to have a dedicated class name 21 - that better represents this kind of failures. 22 - (better readability in test output) 17 + This gets special treatment in error reporting: i.e. it gets 18 + `!!!` as prefix just as `MachineError`, but all stack frames that are 19 + not from `testScript` also get removed. 23 20 """
+5 -5
nixos/lib/test-driver/src/test_driver/machine.py
··· 18 18 from queue import Queue 19 19 from typing import Any 20 20 21 - from test_driver.errors import RequestedAssertionFailed, TestScriptError 21 + from test_driver.errors import MachineError, RequestedAssertionFailed 22 22 from test_driver.logger import AbstractLogger 23 23 24 24 from .qmp import QMPSession ··· 129 129 ) 130 130 131 131 if ret.returncode != 0: 132 - raise TestScriptError( 132 + raise MachineError( 133 133 f"Image processing failed with exit code {ret.returncode}, stdout: {ret.stdout.decode()}, stderr: {ret.stderr.decode()}" 134 134 ) 135 135 ··· 140 140 screenshot_path: str, model_ids: Iterable[int] 141 141 ) -> list[str]: 142 142 if shutil.which("tesseract") is None: 143 - raise TestScriptError("OCR requested but enableOCR is false") 143 + raise MachineError("OCR requested but enableOCR is false") 144 144 145 145 processed_image = _preprocess_screenshot(screenshot_path, negate=False) 146 146 processed_negative = _preprocess_screenshot(screenshot_path, negate=True) ··· 163 163 capture_output=True, 164 164 ) 165 165 if ret.returncode != 0: 166 - raise TestScriptError(f"OCR failed with exit code {ret.returncode}") 166 + raise MachineError(f"OCR failed with exit code {ret.returncode}") 167 167 model_results.append(ret.stdout.decode("utf-8")) 168 168 169 169 return model_results ··· 922 922 ret = subprocess.run(f"pnmtopng '{tmp}' > '{filename}'", shell=True) 923 923 os.unlink(tmp) 924 924 if ret.returncode != 0: 925 - raise TestScriptError("Cannot convert screenshot") 925 + raise MachineError("Cannot convert screenshot") 926 926 927 927 def copy_from_host_via_shell(self, source: str, target: str) -> None: 928 928 """Copy a file from the host into the guest by piping it over the