# Copyright (c) 2026. Jose Manuel Requena Plens """Fail on a result that keeps the caller's array instead of a copy of its own. ``np.asarray`` hands back the very array it was given when that array is already of the asked type, and so does every validation helper built on it. A result that stores such an array shares memory with the caller: changing the caller's array afterwards (``levels[0] = 0.0``, ``freqs *= 2``) changes a result that was computed, judged and perhaps printed before, and nothing raises. A view does the same (``levels[:, 0]``, ``levels.T``, ``reshape``). ``read_only`` applied to such an array does not help: it clears the flag on the caller's own array, or on a view that still shares the caller's memory. The rule: an array a function was handed, or a view of one, is kept only as a copy of its own. Every public record that can hold an array inherits ``phonometry._internal.frozen.OwnsArrays``, which makes that copy when the record is built, whoever builds it: a function of the package or a caller writing the record by hand. Anything else that keeps an array (a plain class, a private record) copies it through ``phonometry._internal.frozen.read_only_copy``. The check has four parts. The first reads the classes: a public record (a dataclass or a named tuple that a public module defines or lists in ``__all__``) with a field annotated as anything that may hold an array, and that does not inherit ``OwnsArrays``, itself or through a record it extends, fails. No trail of the second part reaches a record a caller builds by hand, so the record has to copy for itself. :data:`RECORD_EXEMPT` is that part's escape hatch. The second reads every function of the package and follows each parameter through the statements of its body: the numpy calls that may hand back their argument or a view of it (``asarray``, ``atleast_1d``, ``reshape``, ``ravel``, ``diag`` and their kin), basic indexing, attribute access, containers and a dictionary's or a list's shallow ``copy``, records, and the package's own functions whose return carries a parameter through (the validation helpers first among them, summarised once for the whole tree). A copy (``np.array``, an array's ``.copy()``, ``astype`` with its default ``copy=True``), indexing by a mask or by positions, or any arithmetic ends the trail; a helper of the package counts as positions only when every return of its own is a mask or positions, so one that returns a ``slice`` selects a view. A parameter annotated with a record of the package is read field by field, and a field the record copies (every field of one that inherits ``OwnsArrays``, or one its own ``__post_init__`` copies) carries nothing; what a record that inherits ``OwnsArrays`` reads back from ``self`` in its ``__post_init__`` is its own copy already. An annotation is read through the package's type aliases, so ``float | Field2D`` may hold an array when ``Field2D`` is ``NDArray[np.float64]``. Five places are where an array is kept: * a field of a public dataclass or named tuple, passed to its constructor (``cls(...)`` and ``type(self)(...)`` included), when the field is annotated as an array or anything that may hold one and the record does not copy it itself; * ``object.__setattr__(record, name, value)``: in a method of such a record, where the value read back from ``self`` in ``__post_init__`` is the constructor's argument, so a field normalised with ``np.asarray(self.levels)`` keeps the caller's array, and anywhere else on a public record, such as a factory patching the record it just built; * ``dataclasses.replace(record, ...)`` on a record parameter, which keeps every array field it is not given, unless the record inherits ``OwnsArrays`` and copies them again; * an attribute of a plain public class set in any of its methods (``self.name = value``, ``self.name.append(value)``, or ``self.name[key] = value`` on an attribute annotated as a list or a mapping), when the attribute is annotated as an array, or, with no annotation of its own, when a parameter the value carries is annotated as one or not at all: ``self._distances = distances`` keeps the caller's array for every method that hands it out later, and ``self.fs = fs`` with ``fs: float`` keeps a number; * ``read_only(value)`` anywhere. The third keeps the copy from being made twice: an argument handed to a public record that inherits ``OwnsArrays`` and written as a copy (``read_only_copy(x)``, ``np.copy(x)``, ``x.astype(t)``, ``read_only`` around a copy, or ``x.copy()`` for a field annotated as an array alone) fails, because the record copies it again; a type that has to change is written ``np.asarray(x, dtype=t)``, which converts only when it must. So does a local name handed over bare or through ``read_only`` when every binding of it is such a copy and nothing in the function, or in a helper of the package it is passed to, writes into it, binds it again or keeps it or a view of it: the copy then does nothing the record does not do. The reading of that last part only vouches for what it can see (arithmetic, comparisons, sizes and shapes, numpy calls that build a new array, ``math``, a few builtins), so a copy it cannot vouch for is left alone and the rule never asks for a copy that keeps the caller's array whole to be dropped. The fourth vouches for the one array a record keeps without a copy: one a factory passes through ``phonometry._internal.frozen.handed_over``, so that the frames of a simulation or the field of a march are not held twice while the record is built. That is right only when nothing else can reach the array, and each call is proved so from the syntax or fails: it is an argument of a record that inherits ``OwnsArrays``, built in the ``return`` statement; its argument is a name bound once, in the function's own scope and to it alone, to a new array (``np.zeros``, ``np.empty``, ``np.ones``, or ``None`` in one branch of a conditional); the function is no generator and does not hand out its frame (``locals()``, ``vars()``, ``eval``, ``exec``); and every other read of the name writes into the array (``name[...] = value``, ``+=`` included), tests it against ``None``, or passes it to a function of the package, with no decorator, whose parameter is used the same way, down to three calls deep. A view, a second name, a container, a closure or a lambda rules it out. The function is found however it is spelt: imported under another name, read as an attribute of a module bound by any import (``_internal.frozen.handed_over``), or named in a string (``getattr``). A call anywhere else (at module level, in a lambda, the function named but not called) fails too, and this part has no exemption: an array that cannot be proved alone is copied, as every other one is. A private helper is held to the rule whatever its callers pass today: the next caller may hand it the caller's array. The analysis reads names, not types, so a scalar that travels the same path as an array is read as one where the field it lands in could hold an array. :data:`EXEMPT` is the escape hatch, keyed by file and function, and each entry carries the reason it is one; an entry that covers nothing fails the check too. What the check does not read: a result that holds a record it was given whole (``BandPath`` objects in a prediction's ``paths``) holds that record, not a copy of it, and the record answers for its own arrays; a function that returns the caller's array bare, not inside a result, is outside the rule; a record whose fields are all annotated as something that holds no array is not required to inherit ``OwnsArrays``, so an array a caller puts in its ``float`` field anyway is kept as it was handed; a copy made by a helper that returns it (``return values.copy()``) and handed to a record is not read as made twice, since the helper's other callers may need it; and a value whose path runs through code outside the package (a callback, a library call it does not know) is read as a new value. The fourth part reads names, so a frame reached by introspection (``sys._getframe``, ``inspect.currentframe``) from code a factory calls could still reach an array handed over; no module of the package does that. """ from __future__ import annotations import argparse import ast import pathlib import sys from dataclasses import dataclass, field from typing import TYPE_CHECKING from ast_scan import ROOT, exempted, python_files, relative if TYPE_CHECKING: from collections.abc import Iterable, Mapping, Sequence #: Where the package lives; the default when no path is given. SOURCE = ROOT / "src" / "phonometry" #: The base a record inherits to copy every array it is built with #: (``phonometry._internal.frozen.OwnsArrays``). MECHANISM = "OwnsArrays" #: The function through which a factory hands the record it returns an array #: it built, for the record to keep without a copy #: (``phonometry._internal.frozen.handed_over``). HANDOVER = "handed_over" #: Public records that can hold an array and do not inherit :data:`MECHANISM`, #: keyed by file and class, each with the reason. RECORD_EXEMPT: dict[tuple[str, str], str] = { ("src/phonometry/io/_signal.py", "Signal"): ( "copies its samples itself, into the C-ordered float64 layout it " "promises, and keeps them writeable for processing in place, as " "io.read's contract says (tests/io/test_io_read.py); its other " "fields hold no array" ), } #: Functions that may keep a parameter's array, keyed by file and qualified #: function, each with the reason. EXEMPT: dict[tuple[str, str], str] = { ("src/phonometry/_internal/frozen.py", "_adopted"): ( f"the {MECHANISM} mechanism keeping an array handed over: it seals " f"the array {HANDOVER} marked, which the check's fourth part proves " "nothing but the record can reach, and any other array it returns " "None for, so the record copies it" ), ("src/phonometry/_internal/frozen.py", "_take_ownership"): ( f"the {MECHANISM} mechanism itself: what it stores on the record is " "the read-only copy _owned makes of each array, or the value as it " "was when it holds none; the check reads _owned's pass-through branch " "for the values with no array as the caller's array" ), } #: The numpy functions that may return their first argument, or a view of it. _NUMPY_ALIASING = frozenset( { "asarray", "asanyarray", "ascontiguousarray", "asfortranarray", "asarray_chkfinite", "require", "atleast_1d", "atleast_2d", "atleast_3d", "ravel", "squeeze", "reshape", "transpose", "matrix_transpose", "permute_dims", "broadcast_to", "expand_dims", "moveaxis", "swapaxes", "rollaxis", "flip", "flipud", "fliplr", "rot90", "diagonal", "diag", "trim_zeros", "real", "imag", "real_if_close", "split", "array_split", "hsplit", "vsplit", "dsplit", "unstack", "sliding_window_view", "as_strided", "frombuffer", } ) #: The numpy functions that return one view per array argument. _NUMPY_ALIASING_EACH = frozenset( {"broadcast_arrays", "atleast_1d", "atleast_2d", "atleast_3d"} ) #: The numpy functions whose result indexes an array by position or by mask, #: so that indexing with it copies rather than views. _NUMPY_INDEX = frozenset( { "argsort", "argwhere", "nonzero", "flatnonzero", "searchsorted", "digitize", "arange", "isfinite", "isnan", "isinf", "isclose", "isin", "logical_and", "logical_or", "logical_not", "logical_xor", "greater", "greater_equal", "less", "less_equal", "equal", "not_equal", } ) #: The numpy functions that build an array from what they are given, whose #: result selects by position when what they are given does or when they are #: asked for integers or booleans. _NUMPY_BUILDERS = frozenset( {"array", "asarray", "asanyarray", "ascontiguousarray", "atleast_1d", "fromiter"} ) #: The numpy functions that allocate a new array from a shape alone, the only #: arrays a factory may hand over (:data:`HANDOVER`). _NUMPY_FRESH = frozenset({"zeros", "empty", "ones"}) #: Array methods that may return the array itself or a view of it. _METHOD_ALIASING = frozenset( { "reshape", "ravel", "view", "squeeze", "transpose", "swapaxes", "diagonal", "__array__", # A mapping or a sequence of arrays hands its items out unchanged. "get", "values", "items", "pop", "setdefault", } ) #: Attributes of an array that are plain numbers or metadata, never its data. _SCALAR_ATTRIBUTES = frozenset( {"size", "shape", "ndim", "dtype", "itemsize", "nbytes", "flags", "strides"} ) #: Builtins whose result holds the items of their argument unchanged. _PASS_THROUGH_BUILTINS = frozenset( { "tuple", "list", "sorted", "reversed", "iter", "next", "dict", "set", "frozenset", "max", "min", } ) #: Calls that wrap a container without copying what it holds. _WRAPPERS = frozenset({"MappingProxyType", "read_only", "cast"}) #: The numpy functions that write into the array handed to them first #: (``np.add.at`` and the other ``ufunc.at`` read as ``at``). _NUMPY_WRITERS = frozenset( { "copyto", "put", "place", "putmask", "put_along_axis", "fill_diagonal", "shuffle", "at", } ) #: Array methods that write into the array they are called on. _METHOD_WRITERS = frozenset( {"sort", "fill", "resize", "put", "itemset", "setfield", "partition", "byteswap"} ) #: Array methods whose result is a number or a new array, never the array #: they are called on nor a view of it. _METHOD_FRESH = frozenset( { "all", "any", "argmax", "argmin", "argsort", "astype", "clip", "conj", "conjugate", "cumprod", "cumsum", "dot", "item", "max", "mean", "min", "nonzero", "prod", "round", "searchsorted", "std", "sum", "tolist", "var", } ) #: Attributes of an array that are views of its data. _VIEW_ATTRIBUTES = frozenset({"T", "mT", "real", "imag", "flat"}) #: Builtins whose result holds none of the array they are handed. _READING_BUILTINS = frozenset( { "abs", "bool", "complex", "float", "format", "int", "isinstance", "len", "repr", "str", } ) #: Methods that put their argument into the container they are called on. _CONTAINER_FILLERS = frozenset( {"append", "extend", "insert", "update", "add", "appendleft"} ) #: Annotation words that say a field may hold an array. _ARRAY_WORDS = ("ndarray", "NDArray", "ArrayLike", "Any", "object", "Real", "Complex") Labels = frozenset[str] _CLEAN: Labels = frozenset() @dataclass(frozen=True) class Shaped: """A tuple whose items are known one by one, as ``return a, b`` builds.""" items: tuple[Value, ...] @dataclass(frozen=True) class Fields: """A record whose fields are known one by one. Built by its constructor, it carries what each field was given; read from a parameter annotated with a record class, each field that can hold an array carries the parameter, and :attr:`rest` is what any other attribute (a property, say) may carry. """ items: tuple[tuple[str, Value], ...] rest: Labels = frozenset() def get(self, name: str) -> Value: """What one field carries. A name that is not a field the constructor was given is a default or a property; a property computes a new value from the fields, except on a record the caller handed over, where it may hand one of them out. """ for key, value in self.items: if key == name: return value return self.rest @dataclass(frozen=True) class Many: """A sequence whose items all carry the same, as ``Sequence[Record]`` does.""" item: Value Value = Labels | Shaped | Fields | Many def flat(value: Value) -> Labels: """Every parameter a value may carry, whatever its shape.""" out: set[str] = set() if isinstance(value, Shaped): for item in value.items: out |= flat(item) return frozenset(out) if isinstance(value, Fields): for _, item in value.items: out |= flat(item) return frozenset(out | value.rest) if isinstance(value, Many): return flat(value.item) return value def union(*values: Value) -> Value: """The value that may be any of *values*. A value that carries nothing adds nothing, so it keeps the shape of a tuple or a record the other branches agree on (``return None`` beside ``return a, b``). """ carrying = [v for v in values if isinstance(v, Shaped | Fields) or v] if len(carrying) == 1: return carrying[0] shaped = [v for v in carrying if isinstance(v, Shaped)] if ( shaped and len(shaped) == len(carrying) and len({len(v.items) for v in shaped}) == 1 ): return Shaped( tuple( union(*parts) for parts in zip(*(v.items for v in shaped), strict=True) ) ) records = [v for v in carrying if isinstance(v, Fields)] if records and len(records) == len(carrying): merged: dict[str, Value] = {} rest: set[str] = set() for record in records: rest |= record.rest for key, item in record.items: merged[key] = union(merged[key], item) if key in merged else item return Fields(tuple(merged.items()), frozenset(rest)) sequences = [v for v in carrying if isinstance(v, Many)] if sequences and len(sequences) == len(carrying): return Many(union(*(v.item for v in sequences))) out: set[str] = set() for value in carrying: out |= flat(value) return frozenset(out) @dataclass(frozen=True) class Finding: """One place where a parameter's array is kept without a copy.""" path: str function: str line: int sink: str parameters: tuple[str, ...] @property def key(self) -> tuple[str, str]: """The file relative to the repository, and the qualified function.""" return self.path, self.function @dataclass class ClassInfo: """What the check needs to know of a class of the package.""" name: str module: str record: bool bases: tuple[str, ...] line: int = 0 fields: list[tuple[str, str]] = field(default_factory=list) attributes: dict[str, str] = field(default_factory=dict) #: The fields ``__post_init__`` stores again, each with whether every #: value it stores there is a copy: those the record owns, whoever built it. owned: dict[str, bool] = field(default_factory=dict) #: The same fields, each with whether every value stored there is passed #: through ``read_only`` or ``read_only_copy``: a copy nobody can write #: into, which a result may share without sharing anyone's edits. sealed: dict[str, bool] = field(default_factory=dict) @dataclass class FunctionInfo: """One function of the package, with the module it is read in.""" module: str qualname: str node: ast.FunctionDef | ast.AsyncFunctionDef path: str owner: ClassInfo | None kind: str # "function", "method", "classmethod" or "staticmethod" #: Defined inside another function, and read there, with its closure. nested: bool = False def _is_record_decorator(node: ast.expr) -> bool: target = node.func if isinstance(node, ast.Call) else node if isinstance(target, ast.Name): return target.id == "dataclass" return isinstance(target, ast.Attribute) and target.attr == "dataclass" def _base_name(node: ast.expr) -> str: if isinstance(node, ast.Name): return node.id if isinstance(node, ast.Attribute): return node.attr if isinstance(node, ast.Subscript): return _base_name(node.value) return "" def _module_name(path: pathlib.Path, root: pathlib.Path) -> str: parts = list(path.relative_to(root.parent).with_suffix("").parts) if parts[-1] == "__init__": parts.pop() return ".".join(parts) class Tree: """Every module, class and function of the package, and how names resolve.""" def __init__(self, files: Sequence[pathlib.Path], root: pathlib.Path) -> None: self.package = root.name self.modules: dict[str, ast.Module] = {} self.paths: dict[str, str] = {} self.packages: set[str] = set() self.classes: dict[tuple[str, str], ClassInfo] = {} self.class_by_name: dict[str, list[ClassInfo]] = {} self.functions: dict[tuple[str, str], FunctionInfo] = {} self.imports: dict[str, dict[str, tuple[str, str | None]]] = {} self.numpy_names: dict[str, set[str]] = {} self.numpy_functions: dict[str, dict[str, str]] = {} self.exported: dict[str, tuple[str, ...]] = {} self.typed_cache: dict[tuple[str, str, str], Value] = {} #: Type aliases of the package, by name, as their definition reads. self.aliases: dict[str, str] = {} #: The aliases that stand for an array or for anything that may hold one. self.array_aliases: set[str] = set() #: The functions whose every return is a mask or an array of positions. self.positions: set[tuple[str, str]] = set() #: For each function read for a copy made twice, each node of its #: body with the node that holds it, and each name with its uses. self.syntax: dict[ ast.AST, tuple[dict[ast.AST, ast.AST], dict[str, list[ast.Name]]] ] = {} for path in files: module = _module_name(path, root) tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) self.modules[module] = tree self.paths[module] = relative(path) if path.name == "__init__.py": self.packages.add(module) for module, tree in self.modules.items(): self._read_module(module, tree) self._read_aliases() self.public: set[tuple[str, str]] = set() for (module, qualname), info in self.classes.items(): if ( not info.name.startswith("_") and "." not in qualname and _public_path(module) ): self.public.add((module, qualname)) # A class of a private module reaches a caller only when a public # module lists it in ``__all__``; importing it to use it is not that. for module, names in self.exported.items(): if not _public_path(module): continue for name in names: resolved = self.resolve(module, name) if resolved in self.classes: self.public.add(resolved) def _read_module(self, module: str, tree: ast.Module) -> None: imports: dict[str, tuple[str, str | None]] = {} numpy_names: set[str] = set() numpy_functions: dict[str, str] = {} package = module if module in self.packages else module.rpartition(".")[0] for node in ast.walk(tree): if isinstance(node, ast.Import): for alias in node.names: if alias.name == "numpy": numpy_names.add(alias.asname or "numpy") elif _inside(alias.name, self.package) and alias.asname: imports[alias.asname] = (alias.name, None) elif isinstance(node, ast.ImportFrom): if node.level: base = package.split(".") base = base[: len(base) - node.level + 1] source = ".".join([*base, node.module] if node.module else base) else: source = node.module or "" if source.startswith("numpy"): for alias in node.names: numpy_functions[alias.asname or alias.name] = alias.name continue if not _inside(source, self.package): continue for alias in node.names: local = alias.asname or alias.name if f"{source}.{alias.name}" in self.modules: imports[local] = (f"{source}.{alias.name}", None) else: imports[local] = (source, alias.name) self.imports[module] = imports self.exported[module] = _exported(tree) self.numpy_names[module] = numpy_names self.numpy_functions[module] = numpy_functions self._read_body(module, tree.body, "", None) def _read_aliases(self) -> None: """Which type aliases of the package may hold an array. ``Field2D = NDArray[np.float64]`` makes ``c: float | Field2D`` an array as surely as the alias's own text; an alias of an alias counts too. """ for tree in self.modules.values(): for node in _module_statements(tree.body): found = _alias(node) if found is not None: self.aliases[found[0]] = found[1] changed = True while changed: changed = False for name, text in self.aliases.items(): if name not in self.array_aliases and self.holds_array(text): self.array_aliases.add(name) changed = True def holds_array(self, annotation: str | None) -> bool: """Whether something so annotated may hold an array, aliases read.""" if holds_array(annotation): return True return annotation is not None and any( word in self.array_aliases for word in _words(annotation) ) def _read_body( self, module: str, body: Iterable[ast.stmt], scope: str, owner: ClassInfo | None, *, nested: bool = False, ) -> None: path = self.paths[module] for node in body: if isinstance(node, ast.ClassDef): qualname = f"{scope}.{node.name}" if scope else node.name record = any( _is_record_decorator(d) for d in node.decorator_list ) or any(_base_name(b) == "NamedTuple" for b in node.bases) info = ClassInfo( node.name, module, record, tuple(_base_name(b) for b in node.bases), node.lineno, ) for item in node.body: if isinstance(item, ast.AnnAssign) and isinstance( item.target, ast.Name ): annotation = ast.unparse(item.annotation) if annotation.startswith(("ClassVar", "typing.ClassVar")): continue if annotation in {"KW_ONLY", "dataclasses.KW_ONLY"}: continue if _excluded_from_init(item.value): info.attributes[item.target.id] = annotation continue info.fields.append((item.target.id, annotation)) info.attributes[item.target.id] = annotation self.classes[(module, qualname)] = info self.class_by_name.setdefault(node.name, []).append(info) self._read_body(module, node.body, qualname, info) elif isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef): qualname = f"{scope}.{node.name}" if scope else node.name kind = "function" if owner is not None: kind = "method" for decorator in node.decorator_list: name = _base_name(decorator) if name in {"classmethod", "staticmethod"}: kind = name self.functions[(module, qualname)] = FunctionInfo( module, qualname, node, path, owner, kind, nested=nested ) if owner is not None and not owner.record: # ``self.p: Field2D = ...`` in a method annotates the # attribute for every method of a plain class. for inner in ast.walk(node): if ( isinstance(inner, ast.AnnAssign) and isinstance(inner.target, ast.Attribute) and isinstance(inner.target.value, ast.Name) and inner.target.value.id == "self" ): owner.attributes.setdefault( inner.target.attr, ast.unparse(inner.annotation) ) self._read_body(module, node.body, qualname, None, nested=True) elif isinstance(node, ast.If | ast.Try | ast.With | ast.For | ast.While): self._read_body( module, _nested_statements(node), scope, owner, nested=nested ) def resolve(self, module: str, name: str, depth: int = 0) -> tuple[str, str] | None: """The ``(module, qualname)`` a bare name of *module* stands for.""" if (module, name) in self.functions or (module, name) in self.classes: return module, name target = self.imports.get(module, {}).get(name) if target is None or depth > 8: return None source, attribute = target if attribute is None: return None return self.resolve(source, attribute, depth + 1) def functions_named(self, module: str) -> set[str]: """The bare names *module* binds to functions of the package.""" names = {q for (m, q) in self.functions if m == module and "." not in q} names |= { local for local, (source, attribute) in self.imports.get(module, {}).items() if attribute is not None } return names def resolve_module(self, module: str, name: str) -> str | None: """The module a bare name of *module* stands for, if it is one.""" target = self.imports.get(module, {}).get(name) if target is not None and target[1] is None: return target[0] return None def record_fields(self, info: ClassInfo) -> list[tuple[str, str]]: """The constructor's fields of a record, the inherited ones first.""" fields: list[tuple[str, str]] = [] for base in info.bases: for candidate in self.class_by_name.get(base, []): if candidate.record: fields.extend(self.record_fields(candidate)) break names = {name for name, _ in info.fields} return [f for f in fields if f[0] not in names] + info.fields def copies_arrays(self, info: ClassInfo) -> bool: """Whether a record inherits :data:`MECHANISM`, itself or through a record it extends, and so copies every array it is built with. """ if MECHANISM in info.bases: return True return any( candidate.record and self.copies_arrays(candidate) for base in info.bases for candidate in self.class_by_name.get(base, []) ) def owns(self, info: ClassInfo, name: str) -> bool: """Whether a record copies a field itself: every field of one that inherits :data:`MECHANISM`, or one its own or a base's ``__post_init__`` copies. """ if info.owned.get(name) or self.copies_arrays(info): return True for base in info.bases: for candidate in self.class_by_name.get(base, []): if candidate.record and self.owns(candidate, name): return True return False def seals(self, info: ClassInfo, name: str) -> bool: """Whether a record publishes a field as a read-only copy of its own.""" if self.copies_arrays(info) or ( info.sealed.get(name) and self.owns(info, name) ): return True for base in info.bases: for candidate in self.class_by_name.get(base, []): if candidate.record and self.seals(candidate, name): return True return False def attribute_annotation(self, info: ClassInfo, name: str) -> str | None: """The annotation of an attribute of a class, looked up its bases too.""" if name in info.attributes: return info.attributes[name] for base in info.bases: for candidate in self.class_by_name.get(base, []): found = self.attribute_annotation(candidate, name) if found is not None: return found return None def _inside(module: str, package: str) -> bool: """Whether a dotted module name belongs to *package*.""" return module == package or module.startswith(f"{package}.") def _exported(tree: ast.Module) -> tuple[str, ...]: """The names a module's ``__all__`` lists, however it is spelt.""" names: list[str] = [] for node in tree.body: if isinstance(node, ast.Assign | ast.AugAssign | ast.AnnAssign): targets = node.targets if isinstance(node, ast.Assign) else [node.target] if not any(isinstance(t, ast.Name) and t.id == "__all__" for t in targets): continue if node.value is None: continue names.extend( item.value for item in ast.walk(node.value) if isinstance(item, ast.Constant) and isinstance(item.value, str) ) return tuple(names) def _public_path(module: str) -> bool: """Whether every part of a dotted module name is public.""" return not any(part.startswith("_") for part in module.split(".")) def _excluded_from_init(value: ast.expr | None) -> bool: """Whether a dataclass field is declared ``field(init=False)``.""" if not isinstance(value, ast.Call) or _base_name(value.func) != "field": return False return any( k.arg == "init" and isinstance(k.value, ast.Constant) and k.value.value is False for k in value.keywords ) def _module_statements(body: Iterable[ast.stmt]) -> Iterable[ast.stmt]: """A module's statements, those under ``if TYPE_CHECKING:`` and the like too.""" for node in body: if isinstance(node, ast.If | ast.Try): yield from _module_statements(_nested_statements(node)) else: yield node def _alias(node: ast.stmt) -> tuple[str, str] | None: """The name and the text of a type alias a statement defines, if it does. ``type Name = ...``, ``Name: TypeAlias = ...`` and a plain assignment whose value reads as a type (a subscript, a union, a dotted name) are aliases; a call, a number or a tuple of values is not. """ if isinstance(node, ast.TypeAlias) and isinstance(node.name, ast.Name): return node.name.id, ast.unparse(node.value) if ( isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name) and node.value is not None and _base_name(node.annotation) == "TypeAlias" ): return node.target.id, ast.unparse(node.value) if ( isinstance(node, ast.Assign) and len(node.targets) == 1 and isinstance(node.targets[0], ast.Name) and isinstance(node.value, ast.Subscript | ast.BinOp | ast.Attribute) and ( not isinstance(node.value, ast.BinOp) or isinstance(node.value.op, ast.BitOr) ) ): return node.targets[0].id, ast.unparse(node.value) return None def _nested_statements(node: ast.stmt) -> list[ast.stmt]: out: list[ast.stmt] = [] for name in ("body", "orelse", "finalbody", "handlers"): for child in getattr(node, name, []) or []: if isinstance(child, ast.ExceptHandler): out.extend(child.body) else: out.append(child) return out #: Annotation words that say a field holds a container, whose ``copy`` is #: the container's. _CONTAINER_WORDS = frozenset( {"dict", "Mapping", "MappingProxyType", "list", "Sequence", "tuple", "set"} ) def _method_copy(node: ast.Call) -> bool: """Whether a call is ``something.copy()``, with no argument.""" return ( isinstance(node.func, ast.Attribute) and node.func.attr == "copy" and not node.args and not node.keywords ) def _method_astype(node: ast.Call) -> bool: """Whether a call is ``something.astype(...)`` that copies. It copies unless ``copy=False`` is written, even when the array already has the type asked for. """ return ( isinstance(node.func, ast.Attribute) and node.func.attr == "astype" and not any( keyword.arg == "copy" and isinstance(keyword.value, ast.Constant) and keyword.value.value is False for keyword in node.keywords ) ) def holds_array(annotation: str | None) -> bool: """Whether a field so annotated may hold an array.""" if annotation is None: return True return any(word in annotation for word in _ARRAY_WORDS) @dataclass class Summary: """What a function's return value carries of its parameters.""" value: Value = _CLEAN class Reader: """Follows the parameters of one function through its body.""" def __init__( self, tree: Tree, function: FunctionInfo, summaries: Mapping[tuple[str, str], Summary], findings: list[Finding] | None, closure: Mapping[str, Value] | None = None, ) -> None: self.tree = tree self.function = function self.closure = dict(closure or {}) self.module = function.module self.summaries = summaries self.findings = findings self.returned: Value = _CLEAN self.constants: dict[str, tuple[str, ...]] = {} self.index_names: set[str] = set() self.container_names: set[str] = set() #: Local names bound to a record built here, with its class. self.record_names: dict[str, ClassInfo] = {} #: Whether each ``return`` read so far hands out a mask or positions. self.index_returns: list[bool] = [] self.post_init = ( function.owner is not None and function.node.name == "__post_init__" ) self.plain_method = ( function.owner is not None and not function.owner.record and function.kind == "method" ) # -- the parameters ------------------------------------------------- def start(self) -> dict[str, Value]: """Every parameter carries itself; ``self`` and ``cls`` carry nothing.""" arguments = self.function.node.args names = [ a.arg for a in (*arguments.posonlyargs, *arguments.args, *arguments.kwonlyargs) ] if arguments.vararg: names.append(arguments.vararg.arg) if arguments.kwarg: names.append(arguments.kwarg.arg) annotations = { a.arg: a.annotation for a in (*arguments.posonlyargs, *arguments.args, *arguments.kwonlyargs) } env: dict[str, Value] = dict(self.closure) for index, name in enumerate(names): if index == 0 and self.function.kind in {"method", "classmethod"}: env[name] = _CLEAN continue env[name] = self.typed(name, annotations.get(name)) owner = self.function.owner if self.post_init and owner is not None: # A record that copies its arrays has done so before its own # ``__post_init__`` runs: what that reads back from ``self`` is # the record's copy, not the constructor's argument. copied = self.tree.copies_arrays(owner) for name, _ in self.tree.record_fields(owner): env[f"self.{name}"] = _CLEAN if copied else frozenset({f"self.{name}"}) return env def typed(self, name: str, annotation: ast.expr | None) -> Value: """What a parameter carries, read through its annotation. A record of the package carries the parameter in each field that can hold an array and that the record does not copy for itself; a sequence of records carries one such record per item. Anything else carries the parameter whole. """ cache = self.tree.typed_cache key = (self.function.module, self.function.qualname, name) if key not in cache: cache[key] = self._typed(name, annotation) return cache[key] def _typed(self, name: str, annotation: ast.expr | None) -> Value: label: Labels = frozenset({name}) if annotation is None: return label shape = _record_annotation(annotation) if shape is None: return label word, many = shape candidates = [i for i in self.tree.class_by_name.get(word, []) if i.record] if len(candidates) != 1: return label info = candidates[0] record = Fields( tuple( ( field_name, label if self.tree.holds_array(annotation_text) and not self.tree.seals(info, field_name) else _CLEAN, ) for field_name, annotation_text in self.tree.record_fields(info) ), label, ) return Many(record) if many else record def run(self) -> Value: """Read the body, report the kept arrays, and return what is returned.""" self.block(self.function.node.body, self.start()) return self.returned # -- statements ------------------------------------------------------ def block( self, body: Iterable[ast.stmt], env: dict[str, Value] ) -> dict[str, Value]: for statement in body: env = self.statement(statement, env) return env def statement(self, node: ast.stmt, env: dict[str, Value]) -> dict[str, Value]: if isinstance(node, ast.Assign): value = self.expr(node.value, env) for target in node.targets: self.assign(target, value, env, node.value) return env if isinstance(node, ast.AnnAssign): if node.value is not None: value = self.expr(node.value, env) if isinstance(node.target, ast.Attribute): self.keep_attribute( node.target, value, ast.unparse(node.annotation) ) else: self.assign(node.target, value, env, node.value) return env if isinstance(node, ast.AugAssign): # ``items += [x]`` keeps x in a list; ``levels += x`` adds x's # values into an array. value = self.expr(node.value, env) if ( isinstance(node.target, ast.Name) and node.target.id in self.container_names ): env[node.target.id] = union(env.get(node.target.id, _CLEAN), value) return env if isinstance(node, ast.Expr): self.expr(node.value, env) return env if isinstance(node, ast.Return): self.index_returns.append( node.value is not None and self.indexes(node.value) ) if node.value is not None: self.returned = union(self.returned, self.expr(node.value, env)) return env if isinstance(node, ast.If): self.expr(node.test, env) return merge( self.block(node.body, dict(env)), self.block(node.orelse, dict(env)) ) if isinstance(node, ast.For | ast.AsyncFor): return self.loop(node, env) if isinstance(node, ast.While): self.expr(node.test, env) after = env for _ in range(2): after = merge(after, self.block(node.body, dict(after))) return merge(after, self.block(node.orelse, dict(after))) if isinstance(node, ast.With | ast.AsyncWith): for item in node.items: value = self.expr(item.context_expr, env) if item.optional_vars is not None: self.assign(item.optional_vars, value, env, item.context_expr) return self.block(node.body, env) if isinstance(node, ast.Try | ast.TryStar): body = self.block(node.body, dict(env)) branches = [ body, self.block(node.orelse, dict(body)), *( self.block(handler.body, merge(env, body)) for handler in node.handlers ), ] joined = merge(*branches) return self.block(node.finalbody, joined) if isinstance(node, ast.Match): self.expr(node.subject, env) return merge( env, *(self.block(case.body, dict(env)) for case in node.cases) ) if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef): # A nested function is read where it is defined, with what its # closure holds there. inner = self.tree.functions.get( (self.module, f"{self.function.qualname}.{node.name}") ) if inner is not None and inner.node is node: Reader(self.tree, inner, self.summaries, self.findings, env).run() return env if isinstance(node, ast.Delete): for target in node.targets: if isinstance(target, ast.Name): env.pop(target.id, None) return env return env def loop( self, node: ast.For | ast.AsyncFor, env: dict[str, Value] ) -> dict[str, Value]: names = _constant_strings(node.iter, self.constants) if self.post_init and names is not None and isinstance(node.target, ast.Name): # ``for name in ("a", "b"): object.__setattr__(self, name, ...)`` for name in names: self.constants[node.target.id] = (name,) env = self.block(node.body, env) self.constants.pop(node.target.id, None) return self.block(node.orelse, env) element = self.element(node.iter, env) after = env for _ in range(2): inner = dict(after) self.assign(node.target, element, inner, None) after = merge(after, self.block(node.body, inner)) return merge(after, self.block(node.orelse, dict(after))) def element(self, node: ast.expr, env: dict[str, Value]) -> Value: """What one item of an iteration over *node* carries.""" if isinstance(node, ast.Call) and isinstance(node.func, ast.Name): if node.func.id == "enumerate" and node.args: return Shaped((_CLEAN, self.element(node.args[0], env))) if node.func.id == "zip": return Shaped(tuple(self.element(a, env) for a in node.args)) if ( isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr == "items" ): return Shaped((_CLEAN, flat(self.expr(node.func.value, env)))) value = self.expr(node, env) if isinstance(value, Shaped): return union(*value.items) if value.items else _CLEAN if isinstance(value, Many): return value.item return value def assign( self, target: ast.expr, value: Value, env: dict[str, Value], source: ast.expr | None, ) -> None: if isinstance(target, ast.Name): env[target.id] = value strings = ( _constant_strings(source, self.constants) if source is not None else None ) if strings is not None: self.constants[target.id] = strings self.index_names.discard(target.id) self.container_names.discard(target.id) self.record_names.pop(target.id, None) if source is not None and self.indexes(source): self.index_names.add(target.id) if source is not None and self.builds_container(source): self.container_names.add(target.id) built = self.built_record(source) if built is not None: self.record_names[target.id] = built return if isinstance(target, ast.Tuple | ast.List): if ( isinstance(value, Shaped) and len(value.items) == len(target.elts) and not any(isinstance(e, ast.Starred) for e in target.elts) ): for element, item in zip(target.elts, value.items, strict=True): self.assign(element, item, env, None) return for element in target.elts: inner = element.value if isinstance(element, ast.Starred) else element self.assign(inner, flat(value), env, None) return if isinstance(target, ast.Subscript): # ``out[i] = x`` keeps x in a list or a dictionary built here, and # copies x's values into an array. container = _root_name(target.value) if container is not None and container in self.container_names: env[container] = union(env.get(container, _CLEAN), flat(value)) if isinstance(target.value, ast.Attribute): self.keep_attribute(target.value, value, None, into="item") return if isinstance(target, ast.Attribute): self.keep_attribute(target, value, None) return def keep_attribute( self, target: ast.Attribute, value: Value, annotation: str | None, *, into: str = "", ) -> None: """``self.name = value`` in a method of a plain public class. The attribute's own annotation, in the class body or on an assignment in any of its methods, says whether it may hold an array; an attribute with neither is read through the parameters the value carries, so that ``self._distances = distances`` is held to the annotation of ``distances`` and ``self.fs = fs`` to that of ``fs``. *into* is ``"item"`` for ``self.name[key] = value`` and the method's name for ``self.name.append(value)`` and its kin: the value is kept in a list or a mapping the object holds. An item written into an attribute that is not annotated as one is taken for an element of an array, which copies. """ owner = self.function.owner if ( not self.plain_method or owner is None or not isinstance(target.value, ast.Name) or target.value.id != "self" or not self.owner_public() ): return if annotation is None: annotation = self.tree.attribute_annotation(owner, target.attr) sink = f"self.{target.attr}" if into == "item": if annotation is None or not _collection_text(annotation): return sink += "[...]" elif into: sink += f".{into}(...)" if annotation is not None: if self.tree.holds_array(annotation): self.flag(target, sink, value) return if any(self.parameter_holds_array(label) for label in flat(value)): self.flag(target, sink, value) def parameter_holds_array(self, name: str) -> bool: """Whether a parameter of the function read may hold an array.""" arguments = self.function.node.args for argument in ( *arguments.posonlyargs, *arguments.args, *arguments.kwonlyargs, ): if argument.arg == name: if argument.annotation is None: return True return self.tree.holds_array(ast.unparse(argument.annotation)) return True def built_record(self, node: ast.expr | None) -> ClassInfo | None: """The record class a call builds, if *node* is such a call.""" if not isinstance(node, ast.Call): return None func = node.func owner = self.function.owner if isinstance(func, ast.Name) and func.id == "cls": return owner if owner is not None and owner.record else None key = self.callee(func) info = self.tree.classes.get(key) if key is not None else None return info if info is not None and info.record else None def callee(self, func: ast.expr) -> tuple[str, str] | None: """The function or class of the package a call names, if it names one.""" if isinstance(func, ast.Name): return self.tree.resolve(self.module, func.id) if isinstance(func, ast.Attribute): target = self.method_target(func) if target is not None: return target module = self.module_of(func.value) if module is not None: return self.tree.resolve(module, func.attr) return None # -- what kind of thing a name holds ---------------------------------- def indexes(self, node: ast.expr) -> bool: """Whether *node* is a mask or an array of positions. Indexing with one copies what it selects, where a slice, a number or an ellipsis hands out a view. """ if isinstance(node, ast.Compare | ast.List | ast.ListComp): return True if isinstance(node, ast.UnaryOp): return isinstance(node.op, ast.Invert | ast.Not) if isinstance(node, ast.BinOp): return isinstance(node.op, ast.BitAnd | ast.BitOr | ast.BitXor) if isinstance(node, ast.Name): return node.id in self.index_names if isinstance(node, ast.Tuple): return any(self.indexes(element) for element in node.elts) if isinstance(node, ast.Call): numpy_name = self.numpy_function(node.func) if numpy_name is not None: if numpy_name in {"where", "clip"}: return bool(node.args) and ( len(node.args) == 1 or self.indexes(node.args[0]) ) if numpy_name in _NUMPY_BUILDERS: # ``np.asarray(indices, dtype=np.intp)`` of positions, or # any array of integers or booleans, selects by position. return _dtype_bool_or_int(node) or ( bool(node.args) and self.indexes(node.args[0]) ) return numpy_name in _NUMPY_INDEX if isinstance(node.func, ast.Attribute) and node.func.attr in { "argsort", "nonzero", "astype", }: return node.func.attr != "astype" or _names_bool_or_int(node) # A helper of the package selects by position only when every # return of its own is a mask or positions; one that hands out a # slice, or anything else, is read as a view. return self.callee(node.func) in self.tree.positions return False def builds_container(self, node: ast.expr) -> bool: """Whether *node* builds a new list, tuple, set or dictionary.""" if isinstance( node, ast.List | ast.Tuple | ast.Set | ast.Dict | ast.ListComp | ast.SetComp | ast.DictComp | ast.GeneratorExp, ): return True return ( isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id in {"list", "tuple", "dict", "set", "sorted", "zip"} ) def is_collection(self, node: ast.expr) -> bool: """Whether *node* is a mapping, list or set rather than an array.""" if self.is_container(node): return True if not isinstance(node, ast.Name): return False arguments = self.function.node.args for argument in ( *arguments.posonlyargs, *arguments.args, *arguments.kwonlyargs, ): if argument.arg == node.id and argument.annotation is not None: return _collection_annotation(argument.annotation) return False def is_container(self, node: ast.expr) -> bool: """Whether *node* is a list, tuple or the like built in this function.""" if isinstance(node, ast.Name): return node.id in self.container_names return self.builds_container(node) # -- expressions ----------------------------------------------------- def expr(self, node: ast.expr, env: dict[str, Value]) -> Value: if isinstance(node, ast.Name): return env.get(node.id, _CLEAN) if isinstance(node, ast.Attribute): if ( self.post_init and isinstance(node.value, ast.Name) and node.value.id == "self" ): return env.get(f"self.{node.attr}", _CLEAN) inner = self.expr(node.value, env) if node.attr in _SCALAR_ATTRIBUTES: return _CLEAN if isinstance(inner, Fields): return inner.get(node.attr) return flat(inner) if isinstance(node, ast.Subscript): value = self.expr(node.value, env) self.expr(node.slice, env) if isinstance(value, Shaped) and isinstance(node.slice, ast.Constant): index = node.slice.value if isinstance(index, int) and -len(value.items) <= index < len( value.items ): return value.items[index] if self.indexes(node.slice): return _CLEAN if isinstance(value, Many): return value if isinstance(node.slice, ast.Slice) else value.item return flat(value) if isinstance(node, ast.Call): return self.call(node, env) if isinstance(node, ast.IfExp): self.expr(node.test, env) return union(self.expr(node.body, env), self.expr(node.orelse, env)) if isinstance(node, ast.BoolOp): return union(*(self.expr(v, env) for v in node.values)) if isinstance(node, ast.Tuple | ast.List | ast.Set): items = tuple(self.expr(e, env) for e in node.elts) if isinstance(node, ast.Tuple) and not any( isinstance(e, ast.Starred) for e in node.elts ): return Shaped(items) return union(_CLEAN, *items) if isinstance(node, ast.Starred): return flat(self.expr(node.value, env)) if isinstance(node, ast.Dict): parts = [self.expr(v, env) for v in node.values] for key in node.keys: if key is not None: self.expr(key, env) keys = [ key.value for key in node.keys if isinstance(key, ast.Constant) and isinstance(key.value, str) ] if len(keys) == len(node.keys): # Keyword arguments in waiting: ``Record(**{"a": x})``. return Fields(tuple(zip(keys, parts, strict=True))) return union(_CLEAN, *parts) if isinstance( node, ast.ListComp | ast.SetComp | ast.GeneratorExp | ast.DictComp ): scope = dict(env) for generator in node.generators: self.assign( generator.target, self.element(generator.iter, scope), scope, None ) for condition in generator.ifs: self.expr(condition, scope) if isinstance(node, ast.DictComp): self.expr(node.key, scope) return flat(self.expr(node.value, scope)) return flat(self.expr(node.elt, scope)) if isinstance(node, ast.NamedExpr): value = self.expr(node.value, env) self.assign(node.target, value, env, node.value) return value if isinstance(node, ast.Await): return self.expr(node.value, env) # Arithmetic, comparisons, f-strings, lambdas, constants: a new value. for child in ast.iter_child_nodes(node): if isinstance(child, ast.expr) and not isinstance(node, ast.Lambda): self.expr(child, env) return _CLEAN def call(self, node: ast.Call, env: dict[str, Value]) -> Value: func = node.func arguments = [self.expr(a, env) for a in node.args] keywords = {k.arg: self.expr(k.value, env) for k in node.keywords} first = arguments[0] if arguments else _CLEAN numpy_name = self.numpy_function(func) if numpy_name is not None: if numpy_name == "array": copy = _keyword_constant(node, "copy", default=True) return flat(first) if copy is not True else _CLEAN if numpy_name in _NUMPY_ALIASING_EACH and len(arguments) > 1: return Shaped( tuple( _CLEAN if self.is_container(a) else flat(v) for a, v in zip(node.args, arguments, strict=True) ) ) if numpy_name in _NUMPY_ALIASING: # A list or a tuple built here becomes a new array. if node.args and self.is_container(node.args[0]): return _CLEAN return flat(first) return _CLEAN if isinstance(func, ast.Attribute): receiver = self.expr(func.value, env) name = func.attr if name == "replace" and _base_name(func.value) == "dataclasses": return self.replace_sink(node, arguments, keywords) if name == "__setattr__" and _base_name(func.value) == "object": self.setattr_sink(node, arguments, env) return _CLEAN if name in _CONTAINER_FILLERS: container = _root_name(func.value) if container is not None: env[container] = union( env.get(container, _CLEAN), *map(flat, arguments), *keywords.values(), ) if isinstance(func.value, ast.Attribute): self.keep_attribute( func.value, union(_CLEAN, *map(flat, arguments), *keywords.values()), None, into=name, ) return _CLEAN if name == "astype": copy = _keyword_constant(node, "copy", default=True) return flat(receiver) if copy is not True else _CLEAN if name == "copy" and self.is_collection(func.value): # A dictionary's or a list's ``copy`` is shallow: the arrays # it holds are the same ones. return receiver if name in _METHOD_ALIASING: return flat(receiver) target = self.method_target(func) if target is not None: return self.apply(target, node, arguments, keywords, bound=True) module = self.module_of(func.value) if module is not None: resolved = self.tree.resolve(module, func.attr) if resolved is not None: return self.apply(resolved, node, arguments, keywords, bound=False) if name in _WRAPPERS: return flat(first) return _CLEAN if isinstance(func, ast.Name): name = func.id if name == "getattr" and arguments: if ( self.post_init and isinstance(node.args[0], ast.Name) and node.args[0].id == "self" ): names = ( _constant_strings(node.args[1], self.constants) if len(node.args) > 1 else None ) if names is not None: return union( _CLEAN, *(env.get(f"self.{n}", _CLEAN) for n in names) ) return union(first, *arguments[2:]) if ( name in {"list", "tuple", "sorted", "reversed"} and len(arguments) == 1 and isinstance(first, Many) ): return first if name in _PASS_THROUGH_BUILTINS or name in {"enumerate", "zip"}: return union(_CLEAN, *map(flat, arguments)) if name == "cls" and self.function.owner is not None: if not self.function.owner.record: return _CLEAN return self.constructor_sink( self.function.owner, node, arguments, keywords, public=self.owner_public(), ) if name == "replace" and name not in self.tree.functions_named(self.module): return self.replace_sink(node, arguments, keywords) resolved = self.tree.resolve(self.module, name) if resolved is not None: if name == "read_only" or resolved[1] == "read_only": self.flag(node, "read_only(...)", first) return flat(first) return self.apply(resolved, node, arguments, keywords, bound=False) if name in _WRAPPERS: return flat(first) return _CLEAN if ( isinstance(func, ast.Call) and isinstance(func.func, ast.Name) and func.func.id == "type" and self.function.owner is not None ): if not self.function.owner.record: return _CLEAN return self.constructor_sink( self.function.owner, node, arguments, keywords, public=self.owner_public(), ) self.expr(func, env) return _CLEAN def owner_public(self) -> bool: """Whether the class this method belongs to reaches a caller.""" owner = self.function.owner if owner is None: return False qualname = self.function.qualname.rpartition(".")[0] return (owner.module, qualname) in self.tree.public def numpy_function(self, func: ast.expr) -> str | None: """The numpy function a call names, or ``None``.""" if isinstance(func, ast.Name): return self.tree.numpy_functions[self.module].get(func.id) if isinstance(func, ast.Attribute): root = func.value while isinstance(root, ast.Attribute): root = root.value if ( isinstance(root, ast.Name) and root.id in self.tree.numpy_names[self.module] ): return func.attr return None def module_of(self, node: ast.expr) -> str | None: """The package module an expression names, if it names one.""" if isinstance(node, ast.Name): return self.tree.resolve_module(self.module, node.id) return None def method_target(self, func: ast.Attribute) -> tuple[str, str] | None: """The method ``self.name`` or ``cls.name`` names, in the class read.""" owner = self.function.owner if owner is None or not isinstance(func.value, ast.Name): return None if func.value.id not in {"self", "cls"} and func.value.id != owner.name: return None key = (owner.module, f"{self.function.qualname.rpartition('.')[0]}.{func.attr}") if key in self.tree.functions: return key return None def apply( self, target: tuple[str, str], node: ast.Call, arguments: list[Value], keywords: dict[str | None, Value], *, bound: bool, ) -> Value: """What a call to a function or a class of the package returns.""" if target in self.tree.classes: info = self.tree.classes[target] if info.record: return self.constructor_sink( info, node, arguments, keywords, public=target in self.tree.public ) return _CLEAN function = self.tree.functions[target] summary = self.summaries.get(target) if summary is None or not flat(summary.value): return _CLEAN mapping = bind(function, arguments, keywords, bound=bound) return substitute(summary.value, mapping) # -- the places an array is kept -------------------------------------- def constructor_sink( self, info: ClassInfo, node: ast.Call, arguments: list[Value], keywords: dict[str | None, Value], *, public: bool, ) -> Value: """Report what a record keeps, and return the record as a value.""" fields = self.tree.record_fields(info) annotations = dict(fields) if self.tree.copies_arrays(info): # Every array it is built with, however it is passed, is replaced # by a copy of its own: the record carries nothing of the caller. if public and self.findings is not None: self.copied_twice(info, node, fields) return Fields(tuple((name, _CLEAN) for name, _ in fields)) kept: dict[str, Value] = {} spilled: list[Value] = [] for index, value in enumerate(arguments): if isinstance(node.args[index], ast.Starred) or index >= len(fields): spilled.append(value) continue kept[fields[index][0]] = value for name, value in keywords.items(): if name is None and isinstance(value, Fields): for key, item in value.items: if key in annotations: kept[key] = item else: spilled.append(item) elif name is None or name not in annotations: spilled.append(value) else: kept[name] = value # A field the record copies itself, or one that cannot hold an array, # carries nothing a later read could keep. kept = { name: _CLEAN if self.tree.owns(info, name) or not self.tree.holds_array(annotations[name]) else value for name, value in kept.items() } if public: for name, value in kept.items(): if self.tree.holds_array(annotations[name]): self.flag(node, f"{info.name}.{name}", value) if any( self.tree.holds_array(annotation) for annotation in annotations.values() ): for value in spilled: self.flag(node, f"{info.name}(*...)", value) if spilled: extra = union(_CLEAN, *map(flat, spilled)) kept = {name: union(value, extra) for name, value in kept.items()} return Fields(tuple(kept.items())) def is_copy(self, node: ast.expr, annotation: str) -> bool: """Whether an argument is a copy made of an array. ``read_only_copy(x)``, ``np.copy(x)``, ``x.astype(t)`` (whose default ``copy=True`` copies even when ``x`` already has the type; written ``np.asarray(x, dtype=t)`` it converts only when it has to) and ``read_only`` around any of them or around ``np.array`` are copies whatever ``x`` is; ``x.copy()`` is one when the field is annotated as an array alone, since a mapping's or a list's ``copy`` copies the container. ``np.array(x)`` is not read as one: it is how a list becomes an array. """ if isinstance(node, ast.IfExp): return self.is_copy(node.body, annotation) or self.is_copy( node.orelse, annotation ) if not isinstance(node, ast.Call): return False name = node.func.id if isinstance(node.func, ast.Name) else None if ( name == "read_only_copy" or self.numpy_function(node.func) == "copy" or _method_astype(node) ): return True if name == "read_only" and node.args: inner = node.args[0] return isinstance(inner, ast.Call) and ( _method_copy(inner) or _method_astype(inner) or self.numpy_function(inner.func) in {"array", "copy"} ) return _method_copy(node) and self.array_only(annotation) def array_only(self, annotation: str) -> bool: """Whether a field so annotated holds an array and no container.""" words = _words(annotation) if any(word in _CONTAINER_WORDS for word in words): return False return any( word in {"ndarray", "NDArray", "ArrayLike"} or word in self.tree.array_aliases for word in words ) def copied_twice( self, info: ClassInfo, node: ast.Call, fields: Sequence[tuple[str, str]] ) -> None: """Report a copy made of an array handed to a record that copies it. The record makes a read-only copy of every array it is built with, so a ``read_only_copy`` (or a ``.copy()``) written around the argument copies the same array twice. So does a copy bound to a name first and handed over by that name, bare or through ``read_only``, when the copy does nothing else: see :meth:`copy_only_handed_over`. """ annotations = dict(fields) given: list[tuple[str, ast.expr]] = [ (fields[index][0], argument) for index, argument in enumerate(node.args) if index < len(fields) and not isinstance(argument, ast.Starred) ] given += [(k.arg, k.value) for k in node.keywords if k.arg in annotations] for name, argument in given: if self.is_copy(argument, annotations[name]): self.mark(argument, f"{info.name}.{name} (copied twice)") continue local = _handed_name(argument) if local is not None and self.copy_only_handed_over( local, annotations[name] ): self.mark(argument, f"{info.name}.{name} (copied twice, as {local})") # -- a copy bound to a name and handed over ---------------------------- def copy_only_handed_over(self, name: str, annotation: str) -> bool: """Whether a local name holds a copy that only reaches a record. Every binding of the name in the function is a copy (as :meth:`is_copy` reads one, for a field so annotated), the name is not a parameter, and every use of it is harmless (:meth:`harmless`): nothing writes into the copy, keeps it or a view of it, or hands it anywhere but to records that copy it again. Read this way, the copy is never what makes the code right, and dropping it changes nothing but the memory. Anything the reading cannot vouch for counts as a use that needs the copy, so the rule never asks for a copy that matters to be dropped. """ function = self.function.node if name in _parameter_names(function): return False parents, uses = self.syntax() if not all( isinstance(parents.get(use), ast.Assign | ast.AnnAssign) for use in uses.get(name, ()) if isinstance(use.ctx, ast.Store) ): return False bindings = _bindings(function, name) if not bindings or any( value is None or not self.is_copy(value, annotation) for value in bindings ): return False return self.uses_harmless(name, depth=0) def uses_harmless(self, name: str, depth: int) -> bool: """Whether every read of *name* in the function is harmless. A binding of the name is read as the caller's business: a parameter rebound in the function, or deleted, is not vouched for. """ parents, uses = self.syntax() for node in uses.get(name, ()): if isinstance(node.ctx, ast.Store | ast.Del): if depth > 0: return False continue if not self.harmless(node, parents, depth): return False return True def syntax(self) -> tuple[dict[ast.AST, ast.AST], dict[str, list[ast.Name]]]: """Each node of the function with the node that holds it, and each name with the places it is written, read once per function. """ node = self.function.node if node not in self.tree.syntax: parents: dict[ast.AST, ast.AST] = {} uses: dict[str, list[ast.Name]] = {} for parent in ast.walk(node): for child in ast.iter_child_nodes(parent): parents[child] = parent if isinstance(parent, ast.Name): uses.setdefault(parent.id, []).append(parent) self.tree.syntax[node] = (parents, uses) return self.tree.syntax[node] def harmless( self, node: ast.expr, parents: Mapping[ast.AST, ast.AST], depth: int ) -> bool: """Whether what *node* holds (the array, or a view of it) is only read. Read: an operand of arithmetic or of a comparison, a truth test, a number in an f-string, its size or shape, a method or a numpy function whose result is a new array, an index into something else, a builtin that reads it, a record that copies it, and a function of the package whose parameter is used the same way. A view of it (indexing, ``.T``, ``read_only``, ``np.asarray``) is followed to its own use. Anything else is not. """ parent = parents.get(node) if isinstance(parent, ast.BinOp | ast.UnaryOp | ast.Compare): return True if isinstance(parent, ast.BoolOp): # ``a or levels`` is ``levels`` itself when ``a`` is false. return self.harmless(parent, parents, depth) if isinstance(parent, ast.FormattedValue): return True if isinstance(parent, ast.If | ast.While | ast.Assert) and parent.test is node: return True if isinstance(parent, ast.IfExp): return parent.test is node or self.harmless(parent, parents, depth) if isinstance(parent, ast.Subscript): if parent.value is not node: # An index into another array: indexing by an array copies. return parent.slice is node return isinstance(parent.ctx, ast.Load) and self.harmless( parent, parents, depth ) if isinstance(parent, ast.Attribute): if not isinstance(parent.ctx, ast.Load): return False if parent.attr in _SCALAR_ATTRIBUTES - {"flags", "strides"}: return True if parent.attr in _VIEW_ATTRIBUTES: return self.harmless(parent, parents, depth) call = parents.get(parent) if not isinstance(call, ast.Call) or call.func is not parent: return False if any(k.arg == "out" for k in call.keywords): return False if parent.attr in _METHOD_FRESH: return _keyword_constant(call, "copy", default=True) is True if parent.attr in _METHOD_ALIASING: return self.harmless(call, parents, depth) return False if isinstance(parent, ast.keyword): call = parents.get(parent) if ( parent.arg is None or parent.arg == "out" or not isinstance(call, ast.Call) ): return False return self.harmless_argument(call, node, parent.arg, parents, depth) if ( isinstance(parent, ast.Call) and parent.func is not node and any(argument is node for argument in parent.args) ): return self.harmless_argument(parent, node, None, parents, depth) if isinstance(parent, ast.List | ast.Tuple) and isinstance( parent.ctx, ast.Load ): # ``np.concatenate([levels, extra])``: the items go into a new # array when the list goes straight to a numpy function that # builds one. call = parents.get(parent) return ( isinstance(call, ast.Call) and any(argument is parent for argument in call.args) and self.numpy_function(call.func) not in { None, "array", *_NUMPY_ALIASING, *_NUMPY_ALIASING_EACH, *_NUMPY_WRITERS, } ) return False def harmless_argument( self, call: ast.Call, node: ast.expr, keyword: str | None, parents: Mapping[ast.AST, ast.AST], depth: int, ) -> bool: """Whether a call only reads the array *node* holds.""" func = call.func if isinstance(func, ast.Name) and func.id in {"read_only", "cast"}: return self.harmless(call, parents, depth) if isinstance(func, ast.Name) and func.id in _READING_BUILTINS: return True if ( isinstance(func, ast.Attribute) and isinstance(func.value, ast.Name) and func.value.id == "math" ): return True numpy_name = self.numpy_function(func) if numpy_name is not None: first = keyword is None and bool(call.args) and call.args[0] is node if numpy_name in _NUMPY_WRITERS and first: return False if numpy_name in _NUMPY_ALIASING | _NUMPY_ALIASING_EACH or ( numpy_name == "array" and _keyword_constant(call, "copy", default=True) is not True ): return self.harmless(call, parents, depth) return True info = self.built_record(call) if info is not None: return self.tree.copies_arrays(info) target = self.callee(func) if target is None or target not in self.tree.functions or depth >= 3: return False function = self.tree.functions[target] parameter = _bound_parameter(function, call, node, keyword) if parameter is None: return False reader = Reader(self.tree, function, self.summaries, None) return reader.uses_harmless(parameter, depth + 1) # -- an array handed over without a copy -------------------------------- def handover_refused(self, call: ast.Call) -> str | None: """Why the reading cannot vouch for an array handed over, or ``None``. ``handed_over(name)`` lets the record keep the array without a copy, which is right only when nothing else can reach it, now or later. That is proved here from the syntax alone: the call is an argument of a record that inherits :data:`MECHANISM`, built in the ``return`` statement, so nothing of the function runs after the record holds the array; the name is bound once, in the function's own scope, to an array a numpy allocation makes (:data:`_NUMPY_FRESH`, or ``None`` in one branch of a conditional); and every other read of it only writes into the array (:meth:`writes_into`). No view, no second name and no container ever holds it, so when the function returns, the record's is the only reference left. """ parents, uses = self.syntax() holder = parents.get(call) record = parents.get(holder) if isinstance(holder, ast.keyword) else holder if not isinstance(record, ast.Call) or not ( (isinstance(holder, ast.keyword) and holder.arg is not None) or any(argument is call for argument in record.args) ): return "it is not an argument of a record" info = self.built_record(record) if info is None or not self.tree.copies_arrays(info): return f"it is not an argument of a record that inherits {MECHANISM}" if not isinstance(parents.get(record), ast.Return): return "the record is not built in the return statement" if ( len(call.args) != 1 or call.keywords or not isinstance(call.args[0], ast.Name) ): return "its argument is not a name" name = call.args[0].id function = self.function.node if name in _parameter_names(function): return f"{name} is a parameter" if _frame_escapes(function): return "the function can reach its own frame (a generator, locals())" bindings = _bindings(function, name) stores = [ use for use in uses.get(name, ()) if not isinstance(use.ctx, ast.Load) ] store = stores[0] if len(stores) == 1 else None if ( len(bindings) != 1 or bindings[0] is None or not self.allocates(bindings[0]) or store is None or not _single_target(parents.get(store), store) or not _own_scope(store, parents, function) ): return ( f"{name} is not bound once, in the function and to it alone, to a " f"new array (np.{', np.'.join(sorted(_NUMPY_FRESH))})" ) for use in uses.get(name, ()): if use is call.args[0] or use is store: continue if not self.writes_into(use, parents, depth=0): return f"{name} is read where it may be kept (line {use.lineno})" return None def allocates(self, node: ast.expr) -> bool: """Whether *node* is a new array a numpy allocation makes, or one of two such branches with ``None`` in the other. """ if isinstance(node, ast.IfExp): branches = (node.body, node.orelse) return any(not _is_none(b) for b in branches) and all( _is_none(b) or self.allocates(b) for b in branches ) return ( isinstance(node, ast.Call) and self.numpy_function(node.func) in _NUMPY_FRESH and not any(k.arg in {None, "like"} for k in node.keywords) ) def writes_into( self, node: ast.Name, parents: Mapping[ast.AST, ast.AST], depth: int ) -> bool: """Whether a read of a handed-over name only writes into the array. The array is written into (``name[...] = value``, or ``+=``), tested against ``None``, or passed to a function of the package whose parameter is used the same way, read in its own body down to three calls deep. Nothing else: not a view, not a container, not a closure of a nested function or a lambda, and not a function behind a decorator, whose wrapper is code the body does not show and may keep what it is passed. """ if not _own_scope(node, parents, self.function.node): return False parent = parents.get(node) if isinstance(parent, ast.Subscript): return parent.value is node and isinstance(parent.ctx, ast.Store) if isinstance(parent, ast.Compare): operands = (parent.left, *parent.comparators) return all(isinstance(op, ast.Is | ast.IsNot) for op in parent.ops) and all( operand is node or _is_none(operand) for operand in operands ) if isinstance(parent, ast.keyword): call, keyword = parents.get(parent), parent.arg if keyword is None: return False else: call, keyword = parent, None if not isinstance(call, ast.Call) or all(a is not node for a in call.args): return False if not isinstance(call, ast.Call) or depth >= 3: return False target = self.callee(call.func) if target is None or target not in self.tree.functions: return False function = self.tree.functions[target] parameter = _bound_parameter(function, call, node, keyword) if ( parameter is None or function.node.decorator_list or _frame_escapes(function.node) ): return False reader = Reader(self.tree, function, self.summaries, None) return reader.only_written(parameter, depth + 1) def only_written(self, name: str, depth: int) -> bool: """Whether a parameter is never bound again and only written into.""" parents, uses = self.syntax() return all( isinstance(use.ctx, ast.Load) and self.writes_into(use, parents, depth) for use in uses.get(name, ()) ) def replace_sink( self, node: ast.Call, arguments: list[Value], keywords: dict[str | None, Value] ) -> Value: """``dataclasses.replace``: the new record keeps every field not named. A record that copies its arrays copies the kept ones again when the new record is built, so nothing is kept there. """ if not node.args: return _CLEAN original = arguments[0] info = self.annotated_record(node.args[0]) if info is not None and not self.tree.copies_arrays(info): annotations = dict(self.tree.record_fields(info)) for name, annotation in annotations.items(): if name in keywords or not self.tree.holds_array(annotation): continue carried = ( original.get(name) if isinstance(original, Fields) else original ) if self.tree_public(info): self.flag(node, f"{info.name}.{name} (kept by replace)", carried) if isinstance(original, Fields): items = dict(original.items) items.update({k: v for k, v in keywords.items() if k is not None}) return Fields(tuple(items.items())) return union(original, *keywords.values()) def annotated_record(self, node: ast.expr) -> ClassInfo | None: """The record class a parameter is annotated with, if it names one.""" if not isinstance(node, ast.Name): return None arguments = self.function.node.args for argument in ( *arguments.posonlyargs, *arguments.args, *arguments.kwonlyargs, ): if argument.arg != node.id or argument.annotation is None: continue for word in _words(ast.unparse(argument.annotation)): for info in self.tree.class_by_name.get(word, []): if info.record: return info return None def tree_public(self, info: ClassInfo) -> bool: """Whether a record class reaches a caller.""" return any(self.tree.classes.get(key) is info for key in self.tree.public) def setattr_sink( self, node: ast.Call, arguments: list[Value], env: dict[str, Value] ) -> None: owner = self.function.owner if len(node.args) < 3: return names = _constant_strings(node.args[1], self.constants) or () target = node.args[0] if owner is None or not (isinstance(target, ast.Name) and target.id == "self"): self.foreign_setattr(node, target, names, arguments[2]) return if self.post_init: stored = node.args[2] sealed = isinstance(stored, ast.Call) and _base_name(stored.func) in { "read_only", "read_only_copy", } for name in names: owner.owned[name] = owner.owned.get(name, True) and not flat( arguments[2] ) owner.sealed[name] = owner.sealed.get(name, True) and sealed if not self.owner_public(): for name in names: if self.post_init: env[f"self.{name}"] = arguments[2] return for name in names: annotation = self.tree.attribute_annotation(owner, name) if self.tree.holds_array(annotation): self.flag(node, f"{owner.name}.{name}", arguments[2]) if self.post_init: # The field holds what was stored, for the reads that follow. env[f"self.{name}"] = arguments[2] def foreign_setattr( self, node: ast.Call, target: ast.expr, names: tuple[str, ...], value: Value, ) -> None: """``object.__setattr__(record, name, value)`` outside the record's methods. A factory that patches a field of the record it just built keeps the value as surely as the constructor would. The record is known when it was built here or came in annotated; one that is neither is held to the rule whatever it is, since a field set this way is seldom a number. """ info = ( self.record_names.get(target.id) if isinstance(target, ast.Name) else None ) if info is None: info = self.annotated_record(target) if info is not None and not self.tree_public(info): return label = info.name if info is not None else ast.unparse(target) for name in names or ("?",): annotation = ( self.tree.attribute_annotation(info, name) if info is not None else None ) if self.tree.holds_array(annotation): self.flag(node, f"{label}.{name}", value) def mark(self, node: ast.AST, sink: str) -> None: """Record a finding that is not about a parameter carried.""" if self.findings is None: return self.findings.append( Finding( self.function.path, self.function.qualname, getattr(node, "lineno", 0), sink, (), ) ) def flag(self, node: ast.AST, sink: str, value: Value) -> None: carried = flat(value) if not carried or self.findings is None: return self.findings.append( Finding( self.function.path, self.function.qualname, getattr(node, "lineno", 0), sink, tuple(sorted(carried)), ) ) def merge(*envs: dict[str, Value]) -> dict[str, Value]: """The names after any of several branches, each carrying all it may.""" out: dict[str, Value] = {} for env in envs: for name, value in env.items(): out[name] = union(out[name], value) if name in out else value return out _SEQUENCES = frozenset( { "Sequence", "list", "tuple", "Iterable", "Collection", "Iterator", "set", "frozenset", } ) def _record_annotation(node: ast.expr) -> tuple[str, bool] | None: """The class an annotation names, and whether it is a sequence of it. ``Record``, ``Record | None``, ``Sequence[Record]`` and ``tuple[Record, ...]`` are read; anything else is not. """ if isinstance(node, ast.Constant) and isinstance(node.value, str): try: node = ast.parse(node.value, mode="eval").body except SyntaxError: return None if isinstance(node, ast.BinOp) and isinstance(node.op, ast.BitOr): sides = { _record_annotation(n) for n in (node.left, node.right) if not _is_none(n) } return sides.pop() if len(sides) == 1 else None if isinstance(node, ast.Name | ast.Attribute): return _base_name(node), False if isinstance(node, ast.Subscript) and _base_name(node.value) in _SEQUENCES: inner = node.slice if isinstance(inner, ast.Tuple): if len(inner.elts) != 2 or not ( isinstance(inner.elts[1], ast.Constant) and inner.elts[1].value is Ellipsis ): return None inner = inner.elts[0] found = _record_annotation(inner) if found is None or found[1]: return None return found[0], True return None #: Annotations of a mapping, a sequence or a set, whose ``copy`` is shallow. _COLLECTIONS = frozenset( { "dict", "Dict", "Mapping", "MutableMapping", "defaultdict", "OrderedDict", "ChainMap", "Counter", "list", "List", "Sequence", "MutableSequence", "set", "Set", "MutableSet", "deque", } ) def _collection_annotation(node: ast.expr) -> bool: """Whether an annotation names a mapping, a list or a set (or ``None`` of one).""" if isinstance(node, ast.Constant) and isinstance(node.value, str): try: node = ast.parse(node.value, mode="eval").body except SyntaxError: return False if isinstance(node, ast.BinOp) and isinstance(node.op, ast.BitOr): sides = [n for n in (node.left, node.right) if not _is_none(n)] return bool(sides) and all(_collection_annotation(n) for n in sides) return _base_name(node) in _COLLECTIONS def _collection_text(annotation: str) -> bool: """:func:`_collection_annotation` on an annotation's text.""" try: node = ast.parse(annotation, mode="eval").body except SyntaxError: return False return _collection_annotation(node) def _is_none(node: ast.expr) -> bool: return isinstance(node, ast.Constant) and node.value is None def _words(text: str) -> list[str]: """The identifiers in an annotation's text.""" out: list[str] = [] word = "" for character in text: if character.isalnum() or character == "_": word += character continue if word: out.append(word) word = "" if word: out.append(word) return out def _dtype_bool_or_int(node: ast.Call) -> bool: """Whether an array-building call asks for booleans or integers.""" dtype = next((k.value for k in node.keywords if k.arg == "dtype"), None) if dtype is None and len(node.args) > 1: dtype = node.args[1] if dtype is None: return False text = ast.unparse(dtype) return "bool" in text or "int" in text def _names_bool_or_int(node: ast.Call) -> bool: """Whether an ``astype`` call converts to booleans or integers.""" for argument in node.args[:1]: text = ast.unparse(argument) return "bool" in text or "int" in text return False def _root_name(node: ast.expr) -> str | None: while isinstance(node, ast.Subscript | ast.Attribute): node = node.value return node.id if isinstance(node, ast.Name) else None def _handed_name(node: ast.expr) -> str | None: """The local name an argument hands over, bare or through ``read_only``.""" if ( isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id == "read_only" and len(node.args) == 1 and not node.keywords ): node = node.args[0] return node.id if isinstance(node, ast.Name) else None def _parameter_names(function: ast.FunctionDef | ast.AsyncFunctionDef) -> set[str]: arguments = function.args names = { a.arg for a in (*arguments.posonlyargs, *arguments.args, *arguments.kwonlyargs) } names.update(a.arg for a in (arguments.vararg, arguments.kwarg) if a is not None) return names def _names_in(target: ast.AST, name: str) -> bool: """Whether *name* is among the names a target or an expression holds.""" return any(isinstance(n, ast.Name) and n.id == name for n in ast.walk(target)) def _own_scope( node: ast.AST, parents: Mapping[ast.AST, ast.AST], function: ast.FunctionDef | ast.AsyncFunctionDef, ) -> bool: """Whether *node* is in the function's own scope, not in a nested function, a lambda, a class body or a comprehension. """ current = parents.get(node) while current is not None and current is not function: if isinstance( current, ast.FunctionDef | ast.AsyncFunctionDef | ast.Lambda | ast.ClassDef | ast.ListComp | ast.SetComp | ast.DictComp | ast.GeneratorExp, ): return False current = parents.get(current) return current is function def _single_target(statement: ast.AST | None, target: ast.Name | None) -> bool: """Whether *statement* binds *target* alone (``a = b = ...`` binds two).""" if isinstance(statement, ast.Assign): return len(statement.targets) == 1 and statement.targets[0] is target return isinstance(statement, ast.AnnAssign) and statement.target is target #: Builtins through which a function hands out its own local names. _FRAME_BUILTINS = frozenset({"locals", "vars", "eval", "exec"}) def _frame_escapes(function: ast.FunctionDef | ast.AsyncFunctionDef) -> bool: """Whether a function's locals may be reached from outside it: a coroutine or a generator, whose frame outlives the call, or a body that calls ``locals()``, ``vars()``, ``eval`` or ``exec``. """ if isinstance(function, ast.AsyncFunctionDef): return True for node in ast.walk(function): if isinstance(node, ast.Yield | ast.YieldFrom | ast.Await): return True if isinstance(node, ast.Name) and node.id in _FRAME_BUILTINS: return True return False def _bindings( function: ast.FunctionDef | ast.AsyncFunctionDef, name: str ) -> list[ast.expr | None]: """What each binding of *name* in the function binds it to. A plain assignment binds it to its value; any other binding (an augmented assignment, unpacking, a loop or ``with`` target, ``:=``, an import, a nested definition, ``global``) to ``None``, a value the reading does not know. """ found: list[ast.expr | None] = [] for node in ast.walk(function): if isinstance(node, ast.Assign): for target in node.targets: if isinstance(target, ast.Name) and target.id == name: found.append(node.value) elif isinstance(target, ast.Tuple | ast.List | ast.Starred) and ( _names_in(target, name) ): found.append(None) elif ( isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name) and node.target.id == name and node.value is not None ): found.append(node.value) elif _binds_unknown(node, name, function): found.append(None) return found def _binds_unknown( node: ast.AST, name: str, function: ast.FunctionDef | ast.AsyncFunctionDef ) -> bool: """Whether *node* binds *name* to something other than a plain value.""" if isinstance(node, ast.AugAssign | ast.NamedExpr): return isinstance(node.target, ast.Name) and node.target.id == name if isinstance(node, ast.For | ast.AsyncFor | ast.comprehension): return _names_in(node.target, name) if isinstance(node, ast.withitem): return node.optional_vars is not None and _names_in(node.optional_vars, name) if isinstance(node, ast.ExceptHandler): return node.name == name if isinstance(node, ast.Import | ast.ImportFrom): return any((a.asname or a.name).split(".")[0] == name for a in node.names) if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef | ast.ClassDef): return node is not function and node.name == name if isinstance(node, ast.Global | ast.Nonlocal): return name in node.names return False def _bound_parameter( function: FunctionInfo, call: ast.Call, node: ast.expr, keyword: str | None ) -> str | None: """The parameter of *function* a call binds *node* to, if it can tell.""" arguments = function.node.args positional = [a.arg for a in (*arguments.posonlyargs, *arguments.args)] bound = function.kind == "classmethod" or ( function.kind == "method" and isinstance(call.func, ast.Attribute) and isinstance(call.func.value, ast.Name) and call.func.value.id == "self" ) if bound: positional = positional[1:] if keyword is not None: named = {a.arg for a in (*arguments.args, *arguments.kwonlyargs)} return keyword if keyword in named else None for index, argument in enumerate(call.args): if isinstance(argument, ast.Starred): return None if argument is node: return positional[index] if index < len(positional) else None return None def _keyword_constant(node: ast.Call, name: str, *, default: object) -> object: for keyword in node.keywords: if keyword.arg == name: if isinstance(keyword.value, ast.Constant): return keyword.value.value return None return default def _constant_strings( node: ast.expr | None, constants: Mapping[str, tuple[str, ...]] ) -> tuple[str, ...] | None: if isinstance(node, ast.Constant) and isinstance(node.value, str): return (node.value,) if isinstance(node, ast.Name) and node.id in constants: return constants[node.id] if isinstance(node, ast.Tuple | ast.List): out: list[str] = [] for element in node.elts: found = _constant_strings(element, constants) if found is None: return None out.extend(found) return tuple(out) return None def bind( function: FunctionInfo, arguments: list[Value], keywords: dict[str | None, Value], *, bound: bool, ) -> dict[str, Value]: """What each parameter of *function* receives from one call.""" spec = function.node.args positional = [a.arg for a in (*spec.posonlyargs, *spec.args)] if bound and function.kind in {"method", "classmethod"}: positional = positional[1:] mapping: dict[str, Value] = {} extra: list[Value] = [] for index, value in enumerate(arguments): if index < len(positional): mapping[positional[index]] = value else: extra.append(value) if spec.vararg is not None: mapping[spec.vararg.arg] = union(_CLEAN, *extra) else: for value in extra: for name in positional: mapping[name] = union(mapping.get(name, _CLEAN), value) names = set(positional) | {a.arg for a in spec.kwonlyargs} spill: list[Value] = [] unpacked: list[Value] = [] for keyword, value in keywords.items(): if keyword is None: unpacked.append(value) elif keyword in names: mapping[keyword] = value else: spill.append(value) # ``**options`` can only reach the parameters nothing else has filled. for value in unpacked: for parameter in names - set(mapping): mapping[parameter] = flat(value) spill.append(value) if spec.kwarg is not None: mapping[spec.kwarg.arg] = union(_CLEAN, *spill) return mapping def substitute(value: Value, mapping: Mapping[str, Value]) -> Value: """A summary's value with each parameter replaced by what the call gave it.""" if isinstance(value, Shaped): return Shaped(tuple(substitute(item, mapping) for item in value.items)) if isinstance(value, Fields): return Fields( tuple((key, substitute(item, mapping)) for key, item in value.items), flat(substitute(value.rest, mapping)), ) if isinstance(value, Many): return Many(substitute(value.item, mapping)) out: set[str] = set() for label in value: out |= flat(mapping.get(label, _CLEAN)) return frozenset(out) def find_positions(tree: Tree) -> None: """Which functions return a mask or an array of positions, to a fixed point. Indexing with what such a function returns copies; indexing with what any other function returns, a ``slice`` say, may hand out a view. """ empty = {key: Summary() for key in tree.functions} for _ in range(20): found: set[tuple[str, str]] = set() for key, function in tree.functions.items(): if function.nested: continue reader = Reader(tree, function, empty, None) reader.run() if reader.index_returns and all(reader.index_returns): found.add(key) if found == tree.positions: break tree.positions = found # What the pass above learnt of the records' own copies was read without # the summaries; it is read again on them. for info in tree.classes.values(): info.owned.clear() info.sealed.clear() tree.typed_cache.clear() def summarise(tree: Tree) -> dict[tuple[str, str], Summary]: """What each function's return carries of its parameters, to a fixed point.""" summaries = {key: Summary() for key in tree.functions} for _ in range(20): changed = False for key, function in tree.functions.items(): if function.nested: continue value = Reader(tree, function, summaries, None).run() grown = union(summaries[key].value, value) if grown != summaries[key].value: summaries[key] = Summary(grown) changed = True if not changed: break return summaries def findings_in( files: Sequence[pathlib.Path], root: pathlib.Path = SOURCE, *, tree: Tree | None = None, ) -> list[Finding]: """Every place a function keeps an array it was handed without a copy. *tree* is the package already read from *files*, when the caller has it. """ if tree is None: tree = Tree(files, root) find_positions(tree) summaries = summarise(tree) # What each record's ``__post_init__`` copies, read once more on the # final summaries so that a factory is not held to a copy the record # already makes. for info in tree.classes.values(): info.owned.clear() info.sealed.clear() for function in tree.functions.values(): if function.node.name == "__post_init__": Reader(tree, function, summaries, None).run() tree.typed_cache.clear() found: list[Finding] = [] for function in tree.functions.values(): if not function.nested: Reader(tree, function, summaries, found).run() return sorted(set(found), key=lambda f: (f.path, f.line, f.sink)) def records_without_mechanism( files: Sequence[pathlib.Path], root: pathlib.Path = SOURCE, *, tree: Tree | None = None, ) -> list[Finding]: """Every public record that can hold an array and does not copy it. A record a caller builds by hand never passes through a function of the package, so no trail the analysis above follows reaches it; the record has to copy for itself. One that inherits :data:`MECHANISM`, directly or through a record it extends, does; any other public record with a field annotated as something that can hold an array is reported, at its class. *tree* is the package already read from *files*, when the caller has it. """ if tree is None: tree = Tree(files, root) found: list[Finding] = [] for key in sorted(tree.public): info = tree.classes[key] if not info.record or tree.copies_arrays(info): continue annotations = dict(tree.record_fields(info)) | info.attributes holding = sorted( name for name, annotation in annotations.items() if tree.holds_array(annotation) ) if holding: found.append( Finding( tree.paths[key[0]], key[1], info.line, f"{info.name} can hold an array ({', '.join(holding)}) and " f"does not inherit {MECHANISM}", (), ) ) return found def handovers_in( files: Sequence[pathlib.Path], root: pathlib.Path = SOURCE, *, tree: Tree | None = None, ) -> list[Finding]: """Every array handed over to a record that the reading cannot vouch for. Each call to :data:`HANDOVER`, however it is spelt (:func:`_names_target`), is read where it stands (:meth:`Reader.handover_refused`); a call outside a function the tree reads (at module level, in a class body, in a lambda), and the function named anywhere but as the callee of a call, are refused outright, since nothing can then be proved of the array. No exemption covers this part: an array that cannot be proved alone is copied, as every other one is. *tree* is the package already read from *files*, when the caller has it. """ if tree is None: tree = Tree(files, root) target = (f"{tree.package}._internal.frozen", HANDOVER) by_node = {function.node: function for function in tree.functions.values()} found: list[Finding] = [] for module, syntax in tree.modules.items(): named = [ node for node in ast.walk(syntax) if _names_target(node, tree, module, target) ] if not named: continue parents = { child: parent for parent in ast.walk(syntax) for child in ast.iter_child_nodes(parent) } for node in named: call = parents.get(node) scope = parents.get(node) while scope is not None and not isinstance( scope, ast.FunctionDef | ast.AsyncFunctionDef | ast.Lambda ): scope = parents.get(scope) function = ( by_node.get(scope) if isinstance(scope, ast.FunctionDef | ast.AsyncFunctionDef) else None ) reason: str | None if not isinstance(call, ast.Call) or call.func is not node: reason = "the function is named where it is not called" elif function is None: reason = "it is not called in a function" else: reason = Reader(tree, function, {}, None).handover_refused(call) if reason is not None: found.append( Finding( tree.paths[module], function.qualname if function is not None else "", getattr(node, "lineno", 0), f"{HANDOVER}(...): {reason}", (), ) ) return sorted(set(found), key=lambda f: (f.path, f.line, f.sink)) def _names_target( node: ast.AST, tree: Tree, module: str, target: tuple[str, str] ) -> bool: """Whether *node* may name the function *target* in *module*. A bare name imported from its module under any name counts, and so, spelt as the function's own name, does any other bare name, any attribute whatever it is read from (``frozen.handed_over``, ``_internal.frozen.handed_over``, a module bound by any import) and the string itself (``getattr(frozen, "handed_over")``). The reading errs on the side of the proof: a spelling it cannot follow to the module is held to it all the same, since nothing else in the package bears that name. """ if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Load): return node.id == target[1] or tree.resolve(module, node.id) == target if isinstance(node, ast.Attribute): return node.attr == target[1] return isinstance(node, ast.Constant) and node.value == target[1] def main(argv: Sequence[str] | None = None) -> int: """Report every array kept without a copy of its own.""" parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( "package", nargs="?", type=pathlib.Path, default=SOURCE, help="the package directory to read (default: src/phonometry); the " "whole package is read, because a helper in one module carries an " "array into another", ) arguments = parser.parse_args(argv) root = arguments.package.resolve() files = python_files([], root) tree = Tree(files, root) unequipped, stale_records = exempted( records_without_mechanism(files, root, tree=tree), RECORD_EXEMPT ) found, stale = exempted(findings_in(files, root, tree=tree), EXEMPT) unproved = handovers_in(files, root, tree=tree) if not (found or stale or unequipped or stale_records or unproved): print( "Every array a result keeps is a copy of its own, or an array " "nothing else can reach." ) return 0 if unequipped: print( "::error::a public record can hold an array and keeps whatever " "array a caller hands it" ) for finding in unequipped: print(f" {finding.path}:{finding.line}: {finding.sink}") print( f" -> inherit {MECHANISM} from phonometry._internal.frozen, last " "among the bases; a record that has to keep the caller's array goes " "in RECORD_EXEMPT at the top of scripts/check_array_aliasing.py " "with its reason." ) if found: print("::error::a result keeps an array it was handed, not a copy of it") for finding in found: carried = ( f" <- {', '.join(finding.parameters)}" if finding.parameters else "" ) print( f" {finding.path}:{finding.line}: {finding.function}: " f"{finding.sink}{carried}" ) print( f" -> hand a record that inherits {MECHANISM} the array as it is (it " "makes the copy), or np.asarray(value, dtype=...) for a type it has " "to change; anything else stores read_only_copy(value) from " "phonometry._internal.frozen (a copy of its own, read only); a " "function that has to keep the caller's array goes in EXEMPT at the " "top of scripts/check_array_aliasing.py with its reason." ) if unproved: print( f"::error::an array is handed to a record through {HANDOVER} and " "nothing proves that the record is the only one to hold it" ) for finding in unproved: print( f" {finding.path}:{finding.line}: {finding.function}: {finding.sink}" ) print( " -> hand the record the array as it is, and the record copies it; " f"{HANDOVER} is only for an array the function allocates itself " "(np.zeros, np.empty, np.ones), only writes into, there or in the " "helpers it passes it to, and hands to the record it returns." ) for key in stale: print(f"::error::EXEMPT lists {key}, which keeps no array any more") for key in stale_records: print(f"::error::RECORD_EXEMPT lists {key}, which needs no exemption any more") return 1 if __name__ == "__main__": sys.exit(main())