"""``execute`` — a persistent IPython kernel, presented as an ACP v2 terminal. The tool the agent actually lives in. A spawned crow agent is handed ``execute`` and nothing else; its file, web, memory, vision and delegation work happens through subtools ambient in the kernel. So this is not one tool among many, it is the surface. What changed from ``mcp/execute`` is the output contract, and it changed because ACP v2 gave terminals to the agent. v1 returned a string and that was the end of it — the client rendered the same text the model read. v2 splits the audience in two, which means splitting the payload in two: ``raw_bytes_b64`` Every byte the cell wrote, ANSI intact. The agent lifts this out and turns it into ``TerminalUpdate.output`` (snapshot) or has already streamed it as ``terminal_output_chunk`` (live). It never reaches the model. ``output`` The same bytes ANSI-stripped and capped. This is what the model reads, and what lands in ``ToolCallUpdate.raw_output``. Live streaming rides MCP progress notifications: each iopub chunk is base64'd into the notification's ``message`` field as it arrives, and the agent relays it as a ``terminal_output_chunk``. This is the same mechanism crow-mcp's Rust terminal tool used — ``ProgressNotificationParam.message`` as the data channel — and the same one its renderer was written against. A caller that sent no ``progressToken`` simply gets no notifications; the snapshot in the result covers it either way, and a client that DID watch the bytes arrive is the one that knows not to render the snapshot again. The kernel work runs in a thread. v1 called the blocking ZMQ drain directly inside an ``async def`` tool, which froze the MCP server's event loop for the whole cell — every other session's tool call queued behind it. Here the loop stays live, which is also what makes relaying progress possible at all. """ from __future__ import annotations import asyncio import base64 import concurrent.futures import json import os import sys from collections.abc import Callable from logging import getLogger from pathlib import Path from typing import Optional from fastmcp import Context from crow_cli.mcp2.server import mcp from crow_cli.mcp_meta import meta_get, rail_unread from crow_cli.tools import PRELUDE_V2 from .kernel import CrowKernel, KernelDead logger = getLogger(__name__) # Kernels keyed by ACP session id: one persistent REPL per session. A bare # caller (no _meta) is keyed by cwd instead. _kernels: dict[str, CrowKernel] = {} def _kernel_context( ctx: Context, ) -> tuple[ str, str, Optional[str], Optional[str], Optional[str], Optional[str], str, int ]: """Return ``(key, cwd, session_id, parent_tcid, db_uri, images_dir, redis_url, rlm_depth)`` from the caller-injected ``_meta``. The agent injects these so the kernel is keyed per session and starts in the session's working directory. The Context parameter is filtered out of the LLM-facing schema, so the model never sees — and cannot forge — any of it. No meta means a bare caller: key and cwd fall back to the server process's cwd. ``parent_tcid`` (the execute call's own ACP tool-call id) and ``db_uri`` feed the per-cell begin_cell prologue: they stamp subtool register entries and point the write-through sink at crow.db so the server-side drain can re-emit in-cell tool calls to the ACP client. ``images_dir`` points the kernel's image sink at the session's ImageStore directory so vision bytes land where the drain hydrates them. ``redis_url`` is the wake bus an in-cell task publishes on after it commits a delivery, so the owner wakes now instead of at its next backstop poll; "" is the bus being off. ``rlm_depth`` is how many delegations deep the calling session is, so an in-cell rlm can enforce its budget. """ meta = ctx.request_context.meta if ctx.request_context else None cwd = meta_get(meta, "cwd") or os.getcwd() session_id = meta_get(meta, "session_id") parent_tcid = meta_get(meta, "tool_call_id") db_uri = meta_get(meta, "db_uri") images_dir = meta_get(meta, "images_dir") bus = meta_get(meta, "redis_url") depth = meta_get(meta, "rlm_depth") if rail_unread(meta, session_id): logger.warning( "execute got a _meta it could not read (%s) — kernels fall back to " "cwd keying and the in-cell tools lose their identity", type(meta).__name__, ) return ( session_id or cwd, cwd, session_id, parent_tcid, db_uri, images_dir, bus or "", depth if isinstance(depth, int) and depth > 0 else 0, ) def _progress_token(ctx: Context) -> Optional[object]: """The caller's progressToken, or None if it did not send one. No token means no live stream is possible, so there is no point encoding chunks nobody will receive — the snapshot in the result covers that caller. """ meta = ctx.request_context.meta if ctx.request_context else None return meta_get(meta, "progressToken") if meta is not None else None def _meta_reset(ctx: Context) -> bool: """True when the CALLER — not the model — asked for the kernel to be reset. ``reset`` is a model-facing argument, so the model is the only one who can pull it, and pulling it costs a tool call it has to decide to make. That is the wrong shape for the two callers who actually know a reset is needed: a client reacting to a wedged session, and the agent recovering a connection it just had to rebuild. Neither should have to speak through the model, and neither can — a turn in flight is exactly when the model is unavailable. So this rides ``_meta``, the same channel as ``session_id`` and ``cwd``: injected by the agent, filtered out of the LLM-facing schema, and therefore neither visible to nor forgeable by the model. It resets the KERNEL and nothing else — the MCP client, the session, the tool surface all stay up, which is the entire point of having a signal for it. """ meta = ctx.request_context.meta if ctx.request_context else None return bool(meta_get(meta, "reset_kernel")) def _prologue( session_id: Optional[str], parent_tcid: Optional[str], db_uri: Optional[str], images_dir: Optional[str], redis_url: str = "", rlm_depth: int = 0, ) -> str: """Identity injection prepended to every non-empty cell: stamps the subtool register before the model's code runs. Fail-open — a broken prologue warns on stderr and the cell still runs; the tools work without identity (entries record None).""" return ( "try:\n" " from crow_cli.tools.register import begin_cell as _crow_begin\n" f" _crow_begin(session_id={session_id!r}," f" parent_tool_call_id={parent_tcid!r}, db_uri={db_uri!r}," f" images_dir={images_dir!r}, redis_url={redis_url!r}," f" rlm_depth={rlm_depth!r})\n" "except Exception as _crow_e:\n" " import sys as _crow_sys\n" " print(f'crow prologue: {_crow_e}', file=_crow_sys.stderr)\n" ) def get_kernel( key: str, cwd: str, prelude_path: str | None = None, python_path: str | None = None, ) -> CrowKernel: """Get or lazily start the persistent kernel for a key.""" kernel = _kernels.get(key) if kernel is not None and kernel.dead: # A kernel whose channels failed is not reusable, and it is not the # caller's job to notice: replace it here so the next call just works. logger.warning("Replacing the dead kernel for %s", key) _kernels.pop(key, None) kernel.shutdown() kernel = None if kernel is None: logger.info("Starting new IPython kernel for %s in %s", key, cwd) kernel = CrowKernel(python_path=python_path or sys.executable, cwd=cwd) _run_prelude(kernel, prelude_path) _kernels[key] = kernel return kernel def _run_prelude(kernel: CrowKernel, prelude_path: str | None = None) -> None: """Start a fresh kernel with Crow's built-in prelude, or a custom one. A custom prelude replaces the built-in imports, which may not be available in an alternate interpreter. A broken prelude is logged, not fatal.""" try: prelude = Path(prelude_path).read_text() if prelude_path else PRELUDE_V2 result = kernel.execute(prelude) if result.text.strip(): logger.warning("Kernel prelude produced output: %s", result.text[:200]) except Exception: logger.warning("Kernel prelude failed", exc_info=True) def shutdown_all() -> None: """Shut down every live kernel (test cleanup / server teardown).""" for key in list(_kernels): kernel = _kernels.pop(key, None) if kernel: kernel.shutdown() @mcp.tool async def execute( ctx: Context, code: str, reset: bool = False, timeout: float | None = 30.0, prelude_path: str | None = None, python_path: str | None = None, ) -> str: """Execute one cell in this session's persistent IPython kernel. This is a Jupyter-style REPL, not a fresh Python process per call. Every non-reset call runs in the same kernel for this session, so variables, imports, function definitions, open handles, and the working directory survive from one call to the next. Write multi-step work as successive cells and reuse names already defined. Do not re-import or redefine setup merely because this is a new tool call; inspect ``dir()`` if unsure what the kernel already contains. The kernel is isolated per session id, so state does not leak between sessions. A fresh kernel starts with Crow's async helpers (``fs``, ``memory``, ``rlm``, ``vision``, ``web``, ``write``, ``edit``, and ``reload``) already available. They are async: call them with ``await``. ``reload()`` refreshes Crow's tool modules in the current kernel without clearing your variables, imports, or cwd. Use ``reset=True`` only when you intentionally want a completely new kernel; it discards all state and starts the prelude again. ``python_path`` selects the interpreter for that new kernel, and ``prelude_path`` replaces Crow's built-in prelude with the contents of a Python file. Both are honored only with ``reset=True``; later plain resets return to Crow's interpreter and built-in prelude. The default interpreter is crow-cli's own, so crow's libraries are available and you can interrogate crow.db. Use it for multi-step Python work where you want to build on previous results. Args: code: Python code to execute. print() is how the code talks back: the returned output is everything the cell wrote, or its traceback on error. The last expression's value is NOT returned (no REPL Out[n]) — assign it, print it, or use it. reset: If True, shut down the kernel and start a fresh one, clearing all state (variables, imports). Use if the kernel is in a bad state. With empty code, just resets. Cannot clear a single variable — reset clears everything. prelude_path: Optional Python file to run as the new kernel's prelude instead of Crow's built-in imports. Honored only with reset=True; later plain resets use the built-in prelude. python_path: Optional interpreter path for the new kernel. Honored only with reset=True; later plain resets use Crow's interpreter. timeout: seconds to wait for the cell to finish (default 30). The cell is NOT killed on timeout — IPython keeps running it — so a long-running process returns a "(kernel busy: …)" notice and its output stays collectable on a later call. Pass a larger value, or None to wait indefinitely, for a cell you know runs long: a blocking rlm() delegation (slow on a local model), a build, a big query, a sleep. Returns: A JSON object: ``exit_code``, ``output`` (ANSI-stripped text), ``timed_out``, and ``raw_bytes_b64`` (the same output with ANSI intact, for the client to render). The agent lifts ``raw_bytes_b64`` into the ACP Terminal schema and strips it before the model sees the result. Vision tools running inside the cell have hydrated image blocks prepended by the server — the one and only addition to execute's output. Examples: execute("x = 42") # set a variable (no output) execute("print(x * 2)") # -> "84" execute("import sqlalchemy; print(sqlalchemy.__version__)") execute("", reset=True) # fresh kernel, all state cleared execute("", reset=True, python_path="/env/bin/python") execute("", reset=True, prelude_path="/env/prelude.py") """ key = None try: ( key, cwd, session_id, parent_tcid, db_uri, images_dir, redis_url, rlm_depth, ) = _kernel_context(ctx) # The caller's signal joins the model's argument here, before anything # branches on reset: from this line down they are the same request, and # a _meta reset honours prelude_path/python_path exactly as reset=True # does. Logged separately because the two have different authors and # "who asked for this kernel to die" is the first question an incident # asks. if not reset and _meta_reset(ctx): reset = True logger.info("Kernel reset requested via _meta for %s", key) if (prelude_path or python_path) and not reset: return json.dumps( { "exit_code": 1, "output": "Error: prelude_path and python_path describe the kernel " "a reset creates — pass reset=True with them.", "timed_out": False, "raw_bytes_b64": "", } ) if reset: old = _kernels.pop(key, None) if old: old.shutdown() logger.info("Reset kernel for %s", key) if not code.strip() and not (prelude_path or python_path): return json.dumps( { "exit_code": 0, "output": "Kernel reset successfully. All previous state lost.", "timed_out": False, "raw_bytes_b64": "", } ) kernel = get_kernel(key, cwd, prelude_path, python_path) if reset and not code.strip(): message = "Kernel reset successfully. All previous state lost." if prelude_path: message += f" Prelude loaded from {prelude_path}." if python_path: message += f" Interpreter: {python_path}." return json.dumps( { "exit_code": 0, "output": message, "timed_out": False, "raw_bytes_b64": "", } ) if code.strip(): code = ( _prologue( session_id, parent_tcid, db_uri, images_dir, redis_url, rlm_depth ) + code ) result = await _execute_streaming(ctx, kernel, code, timeout) # The same shape crow-mcp's Rust terminal tool returns, because the # agent-side lift and the client-side dedupe are both written against # it: exit_code / output / timed_out / raw_bytes_b64. # # raw_bytes_b64 is ALWAYS present, even when every byte already went # out live as a progress notification. That is not redundancy, it is # the split of responsibility: the server does not know who is # listening (a second client may attach mid-cell, a resume replays # from the snapshot), so it always ships the whole buffer, and the # client — which does know whether it watched the chunks arrive — # decides not to render it twice. return json.dumps( { "exit_code": result.exit_code, "output": result.text or "[no output]", "timed_out": result.status == "busy", "raw_bytes_b64": base64.b64encode(result.raw).decode("ascii"), }, ensure_ascii=False, ) except KernelDead as exc: # The channels are gone, which is not the cell's fault and not # something a retry of THIS call can fix. Drop the kernel so the next # call builds a fresh one, and say so. logger.error("Kernel dead: %s", exc, exc_info=True) if key is not None: _kernels.pop(key, None) return json.dumps( { "exit_code": 1, "output": f"Error: the kernel's channels died ({exc})." " It is replaced on the next call — call execute again.", "timed_out": False, "raw_bytes_b64": "", }, ensure_ascii=False, ) except Exception as exc: logger.error("Execute error: %s", exc, exc_info=True) # An empty str(exc) — queue.Empty from a timed-out get_shell_msg, the # bare "Error:" that read as "broken" — names its type instead, so the # failure is never invisible. detail = str(exc).strip() or type(exc).__name__ return json.dumps( { "exit_code": 1, "output": f"Error: {detail}", "timed_out": False, "raw_bytes_b64": "", }, ensure_ascii=False, ) async def _run_cell( kernel: CrowKernel, code: str, timeout: Optional[float], on_output: Optional[Callable[[bytes], None]] = None, ): """``kernel.execute`` off-loop — and interrupted if the caller goes away. ``to_thread`` cannot cancel its thread. A cancelled call raises here and leaves the worker polling to its deadline with the kernel's lock held, so every call in between answers ``(kernel busy)``. With ``timeout=None`` — the documented way to block on a long cell — that is forever. Abandoning the cell is the difference: the worker interrupts it, the cell ends, the lock is free in about a second instead of at the deadline. A flag on the kernel rather than a call into it, because this is the ``except CancelledError`` path — the surrounding cancel scope re-delivers cancellation to anything that checkpoints, so an ``await`` here would raise before doing the work, and the thread that owns the socket is the only one that knows whether a cell has started yet. """ try: return await asyncio.to_thread(kernel.execute, code, timeout, on_output) except asyncio.CancelledError: logger.info("execute cancelled mid-cell — abandoning it") kernel.abandon() raise async def _execute_streaming( ctx: Context, kernel: CrowKernel, code: str, timeout: Optional[float] ): """Run the cell off-loop, relaying each chunk as an MCP progress note. The kernel drain is blocking ZMQ, so it runs in a thread and the loop stays live — which is the only reason relaying is possible at all. Chunks are marshalled back with ``run_coroutine_threadsafe``, which preserves submission order, and the queued notifications are awaited before the result returns so a client never sees the completion before the bytes. """ token = _progress_token(ctx) if token is None: return await _run_cell(kernel, code, timeout) loop = asyncio.get_running_loop() sent: list[concurrent.futures.Future] = [] counter = [0] def on_output(chunk: bytes) -> None: counter[0] += 1 sent.append( asyncio.run_coroutine_threadsafe( ctx.report_progress( float(counter[0]), None, base64.b64encode(chunk).decode("ascii"), ), loop, ) ) try: return await _run_cell(kernel, code, timeout, on_output) finally: if sent: # run_coroutine_threadsafe hands back a *concurrent* Future; # wrap it so gather can await it on this loop. await asyncio.gather( *(asyncio.wrap_future(f) for f in sent), return_exceptions=True )