From eba4a6504865feab973e8ecc7dbd3c582b4bfffc Mon Sep 17 00:00:00 2001 From: kt Date: Tue, 13 Oct 2026 01:49:19 -0700 Subject: [PATCH] make /reload re-read the session live inputs (#3393) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * make /reload re-read the session's live inputs instead of a wire-compat no-op The daemon 'reload' answered the TS success without re-reading anything, while its own doc promised settings, provider auth, and the MCP user-server config — after a /login from another process, the running session kept serving the stale credential from its build-time provider target. The handler now rebinds the request auth from the credential store (the same block a model switch runs) and re-reads the MCP manager's settings and auth store. Settings and the model catalog resolve per use and need nothing. * fix /reload: keep the live model, park the auth reload off the runtime The reload rebuilt the provider target from a fresh resolve_model call, which retargeted the live request path (a routed image episode lost its serving model; a changed settings default could switch an unflagged session) and called the MCP auth store blocking_lock on the async worker runtime (a panic). The reload now refreshes only the stored credentials and headers for the LIVE model, keeps its service tier, and parks the whole reload on a blocking thread like get_mcp_connections. The compile and fmt failures of the first push (private visibility, formatting) ride along; the test fixtures load a valid models.json (custom providers require apiKey) and build full wire models. * address review: commit the auth refresh on the live slot, refresh the armed route, guard the async path with a test The auth refresh now lands in place on the CURRENT slot contents under the write lock (a racing set_model, route swap, or retirement is never clobbered by an older snapshot; a changed model re-resolves for whatever is live now). A routed image episode re-applies its armed target on every model-turn attempt, so the reload refreshes that snapshot auth from the same store. A real-engine async dispatch test now guards the blocking thread posture: it fails with a runtime block panic if the reload moves back onto the executor. * address re-review: fence the route install against the reload, refresh the saved session target, clean up the test dir apply_armed_image_route now holds the route lock through the slot write (the clone-and-install is one critical section), and the reload refreshes the armed route BEFORE the live slot: an attempt either installs the refreshed route target or lands before the slot refresh, never over it. The armed route saved session-target fallback re-binds from the same store, so a failing route clear never restores the pre-login key. The real-engine async test builds its worker under a TempDir. * fence every armed-route install against the reload The failover switch and restore paths cloned the armed route under the route lock, released it, then wrote the clone into the provider slot: a /reload refreshing the route and the slot in between reinstalled the old credentials after the reload answered. Both installs now go through install_armed_route_target, which holds the route lock through the slot write (the same fence apply_armed_image_route uses). The restore path no longer holds the provider-slot lock across the route read either, so every path that touches both locks takes them route-then-slot. * re-resolve the failover primary request auth at restore The failover capture kept the pre-switch key and headers for the whole episode; a credential rotated (and reloaded) during the failover served the old auth again once the turn settled back on the primary. The restore now re-resolves the request auth from the store for the captured model and falls back to the capture only when nothing resolves — the same never-clear semantics the reload refresh uses. * retrigger ci: the heartbeat snapshot fallback flake (sigstop race; passes 3/3 locally on this head) * a resolved credential replaces the request headers, it never inherits the old ones A credential whose resolution succeeds rebinds the pair in place: the key AND the headers, so a rotated credential without a team header clears the stale X-Prime-Team-ID instead of serving it on the new key. The failover primary restore falls back to the captured pair only when the store resolves nothing - a resolved key must never be paired with the captured headers. * assert the reload auth with boolean guards: the resolved key never formats into a failure message CodeQL flags the resolved credential flowing into an assert_eq! failure format (cleartext logging of sensitive information): the new header tests compare without the formatting sink. * surface the auth store reload failure: the reload reports what the session applied reload_auth_storage returns the storage reload error (a malformed auth.json or an unacquirable lock leaves the MCP manager on its previous credentials), reload_live_inputs propagates it, and handle_reload answers the client with the failure instead of a success the session did not apply. The connections view keeps its best-effort posture. * gate the target rebinds on the auth store reload: a failed reload keeps the last-good request auth The auth store re-read now runs first in reload_live_inputs: a malformed document or an unacquirable lock fails the reload before any request-auth resolution, so the live target keeps its working stored credential instead of falling to the configured fallback key over the unreadable store. The settings re-read still applies on the failed path; the error still propagates to the client. * serialize /reload: the last reload to run leaves the target serving the newest store Overlapping reload_live_inputs calls each resolve the store at their own read times, so an earlier reload could install its already-resolved key/header pair over a later reload's fresher pair (the model-unchanged branch trusts its pre-read). A session-level reload lock serializes the whole refresh; the armed route and live slot keep their existing lock discipline. * retrigger ci: the concurrent harness-state refinement flake (passes 3/3 locally; main green at 49480faa0) * the failover restore keeps the captured pair when the auth store read fails An unreadable auth.json makes the fresh resolution fall to the configured fallback key, which is not a stored credential - restored_primary_target must not replace the captured last-good pair with it, exactly the overwrite /reload's gate avoids. The resolution now reports the store's read health; the capture backs the restore when nothing resolves or the store failed. The create-config key override stays store-independent. * retrigger ci: the supervisor tool-results stats flake (same test flaked on main's own PR f67c5cf9 and merged green after retrigger) * a failed store read keeps the override key with the captured headers, and the restore serializes with /reload The create-config key override stands in memory regardless of the store's health, but its resolved headers still ride the store read: on a failed read the restore keeps the captured headers (the last-good team context). The failover restore also takes the session reload lock across its resolve-and-install, so a /reload landing mid-restore is never clobbered by the restore's older pair - whichever runs last leaves the newest store standing. * the override test asserts without formatting the resolved pair: no sensitive data in failure messages CodeQL flags the restored pair (the registry-resolved auth) flowing into an assert message: the header comparison keeps its boolean guard without the formatting sink. * retrigger ci: the catalog-slices supervisor timing flake (same protocol_breadth_b10_b11 binary as the earlier flake; passes 3/3 locally) * document the restore's blocking-pool posture: the turn (this closure included) runs on spawn_blocking, never an async executor worker The serializer thread asked for the locked resolution-and-install to be offloaded to a blocking task: the whole turn already runs inside one (worker/turn.rs parks the turn on spawn_blocking), so the synchronous credential resolution, OAuth refresh included, never blocks an executor worker - only a concurrently reloading session waits on the lock span, which is the serialization the clobber-race finding required. * fence the route clear against /reload: the clear serializes its resolve-and-install like the failover restore clear_image_route resolved fresh auth and wrote the slot without the reload lock, so a /reload landing in that window was clobbered by the clear's older pair - a login that just reloaded never reached the next turn (the same race the failover restore's fence closed). The clear now takes the session reload lock across its resolve-and-install; it runs on the turn's blocking thread, so the synchronous resolution never touches an async executor worker. Also merges main 7f646a40f (tombstoned-worker retirement). * fence the failover switch against /reload: the last unfenced resolve-and-install in the turn path The failover switch resolved the backup model's key/headers and wrote the slot without the reload lock, so a /reload landing in that window was clobbered by the switch's older pair - the restore callback in the same function already takes the lock for exactly this. The switch now serializes its resolve-and-install the same way (blocking-pool turn, no executor contact). * retrigger ci: the owned-stop kill-timing flake (passes 3/3 locally on 784520b0f; main green at afe8d14c4) * retrigger ci: the mid-provider-wait abort flake (different timing test each run; green in 10+ prior CI runs of this code; 974/975 in the binary) * rebind only off live store auth: the models.json fallback key is never a stored credential The reload path and the failover restore treated any resolved key as store auth, so a healthy store that no longer holds the provider's credential (or a second read racing the gate's) resolved the configured models.json fallback and overwrote the target's last-good pair - violating the PR's own 'nothing stored never clears' contract. The resolution now reports whether the store actually holds a live credential for the provider on a healthy read (one read cycle, answering both questions); the fallback key never replaces a serving pair anywhere, and the override keeps the captured headers when the store cannot serve. * retrigger ci: the deny job hit Docker Hub's unauthenticated pull rate limit (429) — same infra failure main's own CI shows at 6808008ac and 763094e1b * fence the remaining slot writers against /reload and state the live-credential invariant switch_model, build_session, and arm_image_turn_route resolved credentials and wrote the slot (or the armed route) without the reload lock, so a concurrent /reload landing between their resolve and install was clobbered by the older pair - the same race the failover callbacks, route clear, and failover switch fences close. All three now serialize their resolve-and-install; every writer is verified non-reentrant with the lock (build_session's scoped guard never rides an await; the arm returns before the turn's fenced paths). The store-auth docs now state the true invariant: the rebind runs onto a live credential - stored, or the ambient environment/runtime override, like a fresh resolution after a logout - and never onto the models.json fallback placeholder. * retrigger ci: the post-switch TUI render flake (screen-content assertion; passes 3/3 locally on this exact head; no stall signature) * hold the reload fence through route publication, and from before the route take on clear The arm's lock was scoped inside the match arm and dropped before the armed route published, so a /reload in that gap refreshed credentials the arm then overwrote with its older pair; and clear took the route before locking, leaving a concurrent reload unable to refresh the detached pair while the clear's fallback restore could clobber the reload's slot refresh. The arm now holds the lock from before the resolution through the route install; the clear takes it before the route take. Lock order is reload -> route -> slot everywhere. * fence the apply's session-target capture: the last route writer joins the reload serialization apply_armed_image_route captured the session target and installed the route without the reload lock, so an apply landing in the reload's window between its route refresh and slot rebind captured the pre-reload pair - a later clear with a failed resolve would restore those stale credentials. The apply now holds the lock across capture and install; every provider-target and route writer now serializes under it. * retrigger ci: the loaded-runner perf flakes (drag_select frame cost + protocol linearity; both pass locally on this head; both green on the prior head's CI) * retrigger ci: the contended-runner drag_select perf flake, roll 5 (identical pa-tui source green on main and locally; the PR-runner pool is contended tonight) * retrigger ci: the contended-runner perf-band flakes, roll 6 (the failing timing tests are main's code, green on main and locally; PR pool still contended) * retrigger ci: the contended-runner perf-band flakes, roll 7 * merge-compat: the reload dispatch test builds the WorkerConfig the Decision API child flag added * the reload lock becomes a tokio mutex: the async build path parks instead of blocking the executor build_session took the std reload lock on an async path, so a concurrent /reload holding it across auth-file I/O and a possible OAuth refresh stalled the executor worker until the reload finished. The lock is now tokio::sync::Mutex: the async build locks by .await (the task parks, no worker stall), and every blocking-pool site (the reload's spawn_blocking, the turn machinery's fences, switch_model) uses blocking_lock - the same posture as the session and mcp locks. Comments tightened per review: the fence rationale stays, the step-order narration goes. * Revert "the reload lock becomes a tokio mutex: the async build path parks instead of blocking the executor" This reverts commit 81dc3619ef23fd2ac0d2caa60d7fa9cbcb9bc1b7. * the build's async path takes the reload lock by yielding, never by blocking the worker Reverting the tokio-mutex conversion: the turn machinery runs inside the engine's own runtime.block_on, so blocking_lock panicked in the failover path (provider_failover_e2e failed locally and on CI). The std lock returns everywhere; the build's async acquire becomes a try_lock-and-yield loop - a reload holding the lock across auth I/O parks the TASK, not the executor worker; a poisoned lock acquires instantly (the holder is gone). The comment tightening from the reverted commit is kept. * tighten the fence comments: the rationale stays, the step-order narration goes * retrigger ci: the refinement concurrency flake (passes 3/3 locally; main's code, untouched by this PR) * retrigger ci: the loaded-runner flake family (supervisor stats race + perf bands; all pass locally, all main's code) * retrigger ci: the tool_results stats count race (3-vs-4, the same flake main's own PR f67c5cf9 hit; passes 3/3 locally) * recover the poisoned lock's guard instead of deadlocking on it: the Poisoned error carries the acquired guard try_lock's Poisoned error already carries the acquired guard; calling .lock() inside the match arm waited on the guard the scrutinee still held - a self-deadlock after any reload_lock holder panicked. Recover the guard with poison.into_inner(). * retrigger ci: the contention-flake family, roll continues (the timing tests pass locally; pool still degraded) * the clear path restores the captured pair when the store holds no live credential clear_image_route installed its fresh resolution unconditionally, so an episode settling after a logout overwrote the serving pair with the models.json fallback key - the same install the reload's store-auth gate rejects. The clear now takes the resolution with its store-auth flag: a live credential installs the fresh pair, otherwise the captured session target restores (the resolve_model-Err arm's fallback). * the clear installs the fresh model with its chain-resolved auth; the capture only backs a failed rebuild The prior clear-path gating restored the captured session target whenever the store held no live credential - but the capture is from the episode's start model: a mid-episode switch was dropped, the create-config override was discarded, and a missing capture left the routed image target serving after the episode. The clear returns to the CURRENT session state instead: the freshly resolved model with its chain-resolved auth (override or configured fallback included, what a fresh session of that model serves); the captured target only backs a resolution that cannot rebuild the model. * park the build's lock retry, and read the store-auth flag after the resolution's own OAuth refresh The try_lock retry hot-spun with yield_now (the task stays immediately runnable while a reload holds the lock across auth I/O and a possible OAuth refresh) - it now sleeps 10ms per retry, parking the worker. The store-auth flag was read before the registry's own credential lookup, so a failed-then-refreshed OAuth token left the flag false and the callers discarded the fresh token; the check now runs after the resolution, on the same registry the refresh updated. * retrigger ci: two registered flake classes (refinement harness-state lock contention + wave_b10 heartbeat sigstop race; both previously flaked and cleared on retrigger, both pass 3/3 locally on this head) * merge-compat: drop the decision_child field the Decision API revert removed * retrigger ci: the owned-stop kill-timing flake (registered class, failed at 784520b0f and cleared; passes locally on this head) * retrigger ci: registration kill-timing family (4/4 local post-thaw) + the TS-side goal_backoff incomplete observation (receipt 11779755212) — both runner timing classes, branch code sound * retrigger ci: draft_disconnect ts-reference-side refusal timeout (pinned TS binary identical to main's green run; my Rust diff cannot touch the TS side) * retrigger ci: contributor-trust gate infra failure (setup-nu could not download a Nushell release; the audit then failed by design) --- .../pa-core/src/mcp/manager_catalog/tests.rs | 2 +- crates/pa-core/src/mcp/mod.rs | 17 +- crates/pa-daemon/src/agent_engine.rs | 8 + .../pa-daemon/src/agent_engine/lifecycle.rs | 25 + crates/pa-daemon/src/agent_engine/model.rs | 80 +- crates/pa-daemon/src/agent_engine/reload.rs | 121 +++ .../src/agent_engine/session_engine_impl.rs | 8 + crates/pa-daemon/src/agent_engine/tests.rs | 1 + .../src/agent_engine/tests/reload.rs | 729 ++++++++++++++++++ .../pa-daemon/src/agent_engine/turn/model.rs | 71 +- crates/pa-daemon/src/image_route.rs | 61 +- crates/pa-daemon/src/mcp_connections.rs | 6 +- crates/pa-daemon/src/session_custom.rs | 81 +- crates/pa-daemon/src/worker/commands.rs | 2 +- 14 files changed, 1178 insertions(+), 34 deletions(-) create mode 100644 crates/pa-daemon/src/agent_engine/reload.rs create mode 100644 crates/pa-daemon/src/agent_engine/tests/reload.rs diff --git a/crates/pa-core/src/mcp/manager_catalog/tests.rs b/crates/pa-core/src/mcp/manager_catalog/tests.rs index afa080c9b..d338a8ec7 100644 --- a/crates/pa-core/src/mcp/manager_catalog/tests.rs +++ b/crates/pa-core/src/mcp/manager_catalog/tests.rs @@ -764,7 +764,7 @@ fn api_key_credential_views_see_a_cross_instance_write_after_the_reload() { "the cached copy does not see the other instance's write" ); // The view open's reload picks the stored key up. - manager.reload_auth_storage(); + manager.reload_auth_storage().expect("the reload applies"); assert!( manager .api_key_credential_views() diff --git a/crates/pa-core/src/mcp/mod.rs b/crates/pa-core/src/mcp/mod.rs index a38107e9e..18529b006 100644 --- a/crates/pa-core/src/mcp/mod.rs +++ b/crates/pa-core/src/mcp/mod.rs @@ -357,11 +357,24 @@ impl McpManager { /// runtime): a credential another process wrote — the interactive /// client's `/mcp` key flow stores through its own storage instance — /// becomes visible to the next view read, like the settings re-read - /// above. - pub fn reload_auth_storage(&mut self) { + /// above. The reload's failure surfaces to the caller: a malformed + /// document or a lock that could not be acquired leaves the manager + /// serving its previous credentials, and a successful reload must + /// not be reported over that. + /// + /// # Errors + /// + /// Returns the storage reload's error when the document cannot be + /// read or parsed (a malformed `auth.json`, an unacquirable lock): + /// the manager keeps its previous credentials in that case. + pub fn reload_auth_storage(&mut self) -> Result<(), String> { let storage = self.auth_storage.clone(); let mut handle = storage.blocking_lock(); handle.reload(); + match handle.load_error() { + Some(error) => Err(error.to_string()), + None => Ok(()), + } } /// The resolved integrations (login resolution and status displays). diff --git a/crates/pa-daemon/src/agent_engine.rs b/crates/pa-daemon/src/agent_engine.rs index 02f366ddd..3d60c705f 100644 --- a/crates/pa-daemon/src/agent_engine.rs +++ b/crates/pa-daemon/src/agent_engine.rs @@ -71,6 +71,7 @@ pub(crate) use config::CreateSessionResources; pub use config::SupervisorLinkConfig; use config::{GoalRuntimeHandles, ProducerUsageSink, RestoredSessionModel, StartupScope}; +mod reload; mod session_engine_impl; mod turn; @@ -285,6 +286,13 @@ pub struct AgentSessionEngine { /// every model-turn attempt so retries keep serving it; cleared when the /// episode settles. pub(crate) image_route: std::sync::Mutex>, + /// Serializes `/reload`'s whole live-input refresh: overlapping reloads + /// resolve the store at their own read times, so an earlier reload's + /// already-resolved key/header pair must never land over a later + /// reload's fresher pair (the last reload to run leaves the target + /// serving the newest store). + pub(crate) reload_lock: std::sync::Mutex<()>, + /// Turn-boundary runs register their controller here; /// [`SessionEngine::abort_auto_compaction`] aborts whatever holds it. pub(crate) auto_compaction_abort: std::sync::Mutex>>, diff --git a/crates/pa-daemon/src/agent_engine/lifecycle.rs b/crates/pa-daemon/src/agent_engine/lifecycle.rs index 7f434b551..7c75a2acb 100644 --- a/crates/pa-daemon/src/agent_engine/lifecycle.rs +++ b/crates/pa-daemon/src/agent_engine/lifecycle.rs @@ -164,6 +164,7 @@ impl AgentSessionEngine { pending_branch: std::sync::Mutex::new(None), provider_target: std::sync::Arc::new(std::sync::RwLock::new(None)), image_route: std::sync::Mutex::new(None), + reload_lock: std::sync::Mutex::new(()), own_summary: std::sync::Arc::new(std::sync::Mutex::new(None)), create_resources: std::sync::RwLock::default(), autonomous: std::sync::Arc::new(tokio::sync::Mutex::new( @@ -875,6 +876,30 @@ impl AgentSessionEngine { // rebuild. let stream_fn = switchable_stream_fn(std::sync::Arc::clone(&self.provider_target)); { + // The build's resolve-and-install serializes with `/reload`'s + // live-input refresh (the other slot writers' fence): a reload + // landing between this resolution and the slot write must not + // be clobbered by the build's older pair. The lock is + // acquired by yielding, never by blocking this worker: a + // reload holds the lock across auth-file I/O and a possible + // OAuth refresh, so a blocking acquire on an async path + // would stall the executor for that span. + let _reload_serialized = loop { + match self.reload_lock.try_lock() { + Ok(guard) => break guard, + Err(std::sync::TryLockError::WouldBlock) => {} + // The poisoned error CARRIES the acquired guard (the + // panicking holder is gone, the lock is free): recover + // it — a fresh .lock() here would wait on the guard + // this match's scrutinee still holds. + Err(std::sync::TryLockError::Poisoned(poison)) => break poison.into_inner(), + } + // Park, never hot-spin: a reload holds the lock across + // auth-file I/O and a possible OAuth refresh, and + // yield_now would leave this task immediately runnable + // through all of it. + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + }; let (api_key, headers) = self.resolve_request_key_and_headers(model); let mut target = self.provider_target.write().expect("provider target lock"); *target = Some(ProviderTarget { diff --git a/crates/pa-daemon/src/agent_engine/model.rs b/crates/pa-daemon/src/agent_engine/model.rs index 2977f5fc2..c980cf037 100644 --- a/crates/pa-daemon/src/agent_engine/model.rs +++ b/crates/pa-daemon/src/agent_engine/model.rs @@ -371,18 +371,53 @@ impl AgentSessionEngine { ) -> ( Option, Option>, + ) { + let (api_key, headers, _store_healthy) = + self.resolve_request_key_and_headers_and_store_health(model); + (api_key, headers) + } + + /// [`Self::resolve_request_key_and_headers`] plus whether the + /// resolution serves a live credential for the model's provider on a + /// healthy read — stored, or the ambient environment/runtime override + /// (a logout that empties the store leaves the session on the env + /// key, exactly like a fresh resolution). `false` means the resolved + /// key is the configured `models.json` fallback — served when the + /// store is unreadable OR resolves no credential from any live + /// source — which is not a credential and must never replace a + /// serving pair (the create-config override is source-independent; + /// callers apply it themselves). + pub(crate) fn resolve_request_key_and_headers_and_store_health( + &self, + model: &Model, + ) -> ( + Option, + Option>, + bool, ) { let auth = pa_core::auth::AuthStorage::create(&self.config.agent_dir); + let store_healthy = auth.load_error().is_none(); let mut registry = pa_core::models::ModelRegistry::create(auth, self.config.agent_dir.join("models.json")); let resolved = registry.get_api_key_and_headers(model, model.headers.as_ref()); + // The credential check runs AFTER the resolution: the registry's + // lookup may OAuth-refresh and store a fresh token, so a check + // before it would read the stale one and discard the fresh pair. + let stored_credential = registry + .auth + .get_api_key_with_source_token(&model.provider, false) + .api_key + .is_some(); + let store_auth = store_healthy && stored_credential; if let Some(api_key) = &self.current_selection().api_key { // The create-config key override pins the key, never the headers: // the registry's merged headers still ship, exactly like the TS - // `getApiKeyAndHeaders` override path. - return (Some(api_key.clone()), resolved.headers); + // `getApiKeyAndHeaders` override path. The flag stays the + // store's: a caller pairing the override key with headers + // keeps its last-good headers when the store cannot serve. + return (Some(api_key.clone()), resolved.headers, store_auth); } - (resolved.api_key, resolved.headers) + (resolved.api_key, resolved.headers, store_auth) } pub(crate) fn resolve_request_api_key(&self, model: &Model) -> Option { @@ -396,6 +431,45 @@ impl AgentSessionEngine { .get_api_key_and_headers(model, model.headers.as_ref()) .api_key } + + /// The primary target a failover episode restores: the captured model + /// with the request auth RE-RESOLVED from the store. The captured + /// pair only backs the restore when the store cannot serve a live + /// credential; the create-config key override stands in memory + /// regardless, keeping the captured headers off a failed read. + pub(crate) fn restored_primary_target( + &self, + primary: &Model, + captured_api_key: Option, + captured_headers: Option>, + ) -> pa_core::session_engine::provider_adapter::ProviderTarget { + let override_key = self.current_selection().api_key; + let (api_key, headers, store_auth) = + self.resolve_request_key_and_headers_and_store_health(primary); + // The capture backs the restore whenever the store cannot serve a + // live credential (a failed read, or a healthy store with nothing + // stored — both resolve the configured `models.json` fallback key, + // which is not a credential); a live stored credential REPLACES + // the pair, headers included. + let (api_key, headers) = if let Some(override_key) = override_key { + if store_auth { + (Some(override_key), headers) + } else { + (Some(override_key), captured_headers.or(headers)) + } + } else { + match api_key { + Some(api_key) if store_auth => (Some(api_key), headers), + _ => (captured_api_key, captured_headers), + } + }; + pa_core::session_engine::provider_adapter::ProviderTarget { + service_tier: *self.service_tier.read().expect("service tier lock"), + api_key, + model: primary.clone(), + headers, + } + } } /// The saved model context of a session file: the pinned `(provider, diff --git a/crates/pa-daemon/src/agent_engine/reload.rs b/crates/pa-daemon/src/agent_engine/reload.rs new file mode 100644 index 000000000..d10267cfa --- /dev/null +++ b/crates/pa-daemon/src/agent_engine/reload.rs @@ -0,0 +1,121 @@ +//! The session's live-input reload (`/reload`'s daemon half): re-read the +//! request auth from the credential store and the MCP manager's settings +//! and auth state — the pieces another process (a `/login`, a `/mcp` key +//! flow) can change under a running session. +use super::AgentSessionEngine; +use pa_core::session_engine::provider_adapter::ProviderTarget; + +impl AgentSessionEngine { + /// The stored request auth for the target's OWN model, applied in + /// place: only the key and headers change, and only when the + /// resolution serves a live credential for the model's provider — + /// stored, or the ambient environment/runtime override (a logout + /// that empties the store leaves the session on the env key, like a + /// fresh resolution). The configured `models.json` fallback key — + /// what the resolution serves when the store is unreadable OR + /// resolves no credential from any live source — is not a credential + /// and never replaces what the target already serves with. A + /// resolved credential REPLACES the pair: its headers land even when + /// it carries none, so a rotated credential without a team header + /// clears the previous credential's `X-Prime-Team-ID` instead of + /// serving it on the new key. + fn refresh_request_auth(&self, target: &mut ProviderTarget) { + let (api_key, headers, store_auth) = + self.resolve_request_key_and_headers_and_store_health(&target.model); + if store_auth { + if let Some(api_key) = api_key { + target.api_key = Some(api_key); + target.headers = headers; + } + } + } + + /// `/reload`'s session half: re-read the live inputs another process + /// may have changed — the request auth from the credential store + /// (rebinding the live target and an armed image route's pair in + /// place: the model and service tier never change), and the MCP + /// manager's settings and auth state. A rebind lands only off live + /// STORE auth (stored, or the ambient environment/runtime override); + /// the configured `models.json` fallback key never replaces a + /// last-good pair. A failed auth-store read fails the reload before + /// any target is touched. + pub(crate) fn reload_live_inputs(&self) -> Result<(), String> { + // The whole refresh is serialized against every other + // resolve-and-install (the session's reload lock): overlapping + // writers resolve the store at their own read times, so the last + // writer to run must be the one leaving the newest store. + let _serialized = self + .reload_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + // The auth store re-read gates the rebinds below: the settings + // re-read still applies on a failed reload. + { + let mut manager = self + .mcp + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let auth_reload = manager.reload_auth_storage(); + manager.refresh(); + auth_reload?; + } + // The armed route refreshes FIRST, the live slot SECOND: the + // route lock fences the route's clone-and-install against this + // refresh order (an attempt either installs the refreshed route + // or lands before the slot refresh — never over it), and the + // route's saved session target is the fallback a later + // `clear_image_route` restores when resolution fails, so its auth + // re-binds from the same store read. + if let Some(route) = self + .image_route + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .as_mut() + { + self.refresh_request_auth(&mut route.target); + if let Some(saved) = route.session_target.as_mut() { + self.refresh_request_auth(saved); + } + } + let live_model = self + .provider_target + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .as_ref() + .map(|target| target.model.clone()); + if let Some(model) = live_model { + let (api_key, headers, store_auth) = + self.resolve_request_key_and_headers_and_store_health(&model); + if store_auth { + let mut slot = self + .provider_target + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some(target) = slot.as_mut() { + // The model may have moved under the store read: a + // switch or route swap installed a new target. The + // auth then re-resolves for whatever is live NOW, + // under the same gate; a cleared slot stays cleared. + let (api_key, headers) = if target.model == model { + (api_key, headers) + } else { + match self.resolve_request_key_and_headers_and_store_health(&target.model) { + (Some(key), headers, true) => (Some(key), headers), + // The moved-to model's provider has no live + // stored credential: the target keeps its + // last-good pair. + _ => (None, None), + } + }; + // A resolved credential replaces the pair: headers + // clear when the fresh credential carries none. + if let Some(api_key) = api_key { + target.api_key = Some(api_key); + target.headers = headers; + } + } + } + } + Ok(()) + } +} diff --git a/crates/pa-daemon/src/agent_engine/session_engine_impl.rs b/crates/pa-daemon/src/agent_engine/session_engine_impl.rs index 53d238765..dd3a5a3f9 100644 --- a/crates/pa-daemon/src/agent_engine/session_engine_impl.rs +++ b/crates/pa-daemon/src/agent_engine/session_engine_impl.rs @@ -414,6 +414,14 @@ impl SessionEngine for AgentSessionEngine { return false; }; { + // The switch's resolve-and-install serializes with `/reload`'s + // live-input refresh (the failover callbacks' fence): a reload + // landing between this resolution and the slot write must not + // be clobbered by the switch's older pair. + let _reload_serialized = self + .reload_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); let (api_key, headers) = self.resolve_request_key_and_headers(&model); let mut target = self.provider_target.write().expect("provider target lock"); *target = Some(ProviderTarget { diff --git a/crates/pa-daemon/src/agent_engine/tests.rs b/crates/pa-daemon/src/agent_engine/tests.rs index eae8a650b..78ce6a5e7 100644 --- a/crates/pa-daemon/src/agent_engine/tests.rs +++ b/crates/pa-daemon/src/agent_engine/tests.rs @@ -13,6 +13,7 @@ mod compaction; mod goal; mod model_resolution; mod quota_park; +mod reload; mod rlm_children; mod saved_context; mod session; diff --git a/crates/pa-daemon/src/agent_engine/tests/reload.rs b/crates/pa-daemon/src/agent_engine/tests/reload.rs new file mode 100644 index 000000000..55f6462b1 --- /dev/null +++ b/crates/pa-daemon/src/agent_engine/tests/reload.rs @@ -0,0 +1,729 @@ +//! `/reload`'s session half: the request auth rebinds from the credential +//! store, the store a `/login` from another process writes. +use super::*; +use pa_core::session_engine::provider_adapter::ProviderTarget; + +/// A custom provider whose configured key is a placeholder: the stored +/// credential wins over it, so the request auth follows the store (the +/// re-login scenario the reload must cover). +fn write_credential_backed_provider(agent_dir: &std::path::Path) { + std::fs::create_dir_all(agent_dir).unwrap(); + std::fs::write( + agent_dir.join("models.json"), + serde_json::json!({ + "providers": { + "battery": { + "api": "openai-completions", + "baseUrl": "http://127.0.0.1:9", + "apiKey": "sk-config-fallback", + "models": [ + { + "id": "mock-1", + "name": "Mock 1", + "api": "openai-completions", + "contextWindow": 128_000, + "maxTokens": 4096, + } + ] + } + } + }) + .to_string(), + ) + .unwrap(); +} + +/// One stored OAuth credential for the custom provider. +fn write_oauth_credential(agent_dir: &std::path::Path, access: &str) { + std::fs::create_dir_all(agent_dir).unwrap(); + std::fs::write( + agent_dir.join("auth.json"), + serde_json::json!({ + "battery": { + "type": "oauth", + "access": access, + "refresh": "r", + "expires": 4_102_444_800_000i64, + } + }) + .to_string(), + ) + .unwrap(); +} + +fn credential_backed_engine_with_override( + dir: &std::path::Path, + api_key: Option<&str>, +) -> AgentSessionEngine { + AgentSessionEngine::new(AgentEngineConfig { + cwd: dir.to_path_buf(), + agent_dir: dir.join("agent"), + provider: Some("battery".to_string()), + model: Some("mock-1".to_string()), + api_key: api_key.map(str::to_string), + thinking: None, + session_dir: None, + session_file: None, + faux_script: None, + supervisor_link: None, + telemetry_disabled: Some(true), + cron_store: None, + queued_steering_probe: None, + }) + .expect("engine") +} + +fn credential_backed_engine(dir: &std::path::Path) -> AgentSessionEngine { + AgentSessionEngine::new(AgentEngineConfig { + cwd: dir.to_path_buf(), + agent_dir: dir.join("agent"), + provider: Some("battery".to_string()), + model: Some("mock-1".to_string()), + api_key: None, + thinking: None, + session_dir: None, + session_file: None, + faux_script: None, + supervisor_link: None, + telemetry_disabled: Some(true), + cron_store: None, + queued_steering_probe: None, + }) + .expect("engine") +} + +/// A credential another process wrote (the `/login` scenario) reaches the +/// running session through the reload: the provider target rebinds. +#[test] +fn reload_rebinds_the_request_auth_from_the_credential_store() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + write_credential_backed_provider(&agent_dir); + write_oauth_credential(&agent_dir, "stale-access"); + let engine = credential_backed_engine(dir.path()); + let model = engine.resolve_model().expect("the custom model resolves"); + // The session's build-time target: the credential the engine saw. + *engine + .provider_target + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(ProviderTarget { + service_tier: None, + api_key: Some("stale-access".to_string()), + model, + headers: None, + }); + // A re-login from another process replaces the stored credential. + write_oauth_credential(&agent_dir, "fresh-access"); + engine.reload_live_inputs().expect("the reload applies"); + let target = engine + .provider_target + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + .expect("the reload keeps the target set"); + assert_eq!( + target.api_key.as_deref(), + Some("fresh-access"), + "the reload rebinds the request auth to the fresh stored credential" + ); +} + +/// With nothing stored for the live model the reload keeps the slot +/// untouched (a models.json lost mid-session loses nothing) — the MCP +/// re-read alone still runs. +#[test] +fn reload_with_nothing_stored_keeps_the_target() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + std::fs::create_dir_all(&agent_dir).unwrap(); + let engine = credential_backed_engine(dir.path()); + assert!( + engine.resolve_model().is_err(), + "no models.json: nothing resolves" + ); + let model: pa_types::ai::Model = + pa_core::session_engine::provider_adapter::json_round_trip(&serde_json::json!({ + "id": "mock-1", + "name": "Mock 1", + "api": "openai-completions", + "provider": "battery", + "baseUrl": "http://127.0.0.1:9", + "reasoning": false, + "input": ["text"], + "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, + "contextWindow": 128_000, + "maxTokens": 4096, + })) + .expect("the model converts"); + *engine + .provider_target + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(ProviderTarget { + service_tier: None, + api_key: Some("stale-access".to_string()), + model, + headers: None, + }); + engine.reload_live_inputs().expect("the reload applies"); + let target = engine + .provider_target + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + .expect("the target stays"); + assert_eq!( + target.api_key.as_deref(), + Some("stale-access"), + "an unresolvable model leaves the target untouched" + ); +} + +/// The two-model pair the retarget guard uses: the text model the +/// selection pins and a second model standing in for a routed episode's +/// serving target. +fn write_two_model_provider(agent_dir: &std::path::Path) { + std::fs::create_dir_all(agent_dir).unwrap(); + std::fs::write( + agent_dir.join("models.json"), + serde_json::json!({ + "providers": { + "battery": { + "api": "openai-completions", + "baseUrl": "http://127.0.0.1:9", + "apiKey": "sk-config-fallback", + "models": [ + { + "id": "mock-1", + "name": "Mock 1", + "api": "openai-completions", + "contextWindow": 128_000, + "maxTokens": 4096, + }, + { + "id": "mock-vision", + "name": "Mock Vision", + "api": "openai-completions", + "contextWindow": 128_000, + "maxTokens": 4096, + } + ] + } + } + }) + .to_string(), + ) + .unwrap(); +} + +/// The live model never retargets through a reload: a routed image +/// episode serves on its own target and an unflagged session must not +/// follow a changed settings default — only the request auth rebinds. +#[test] +fn reload_keeps_the_live_model_and_rebinds_its_auth() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + write_two_model_provider(&agent_dir); + write_oauth_credential(&agent_dir, "fresh-access"); + let engine = credential_backed_engine(dir.path()); + // The selection pins mock-1; the slot holds a routed episode's target + // on a DIFFERENT model. + let session_model = engine.resolve_model().expect("the pinned model resolves"); + assert_eq!(session_model.id, "mock-1"); + let routed: pa_types::ai::Model = + pa_core::session_engine::provider_adapter::json_round_trip(&serde_json::json!({ + "id": "mock-vision", + "name": "Mock Vision", + "api": "openai-completions", + "provider": "battery", + "baseUrl": "http://127.0.0.1:9", + "reasoning": false, + "input": ["text"], + "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, + "contextWindow": 128_000, + "maxTokens": 4096, + })) + .expect("the routed model converts"); + *engine + .provider_target + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(ProviderTarget { + service_tier: None, + api_key: Some("stale-access".to_string()), + model: routed, + headers: None, + }); + engine.reload_live_inputs().expect("the reload applies"); + let target = engine + .provider_target + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + .expect("the reload keeps the target set"); + assert_eq!( + target.model.id, "mock-vision", + "the reload never retargets the live model" + ); + assert_eq!( + target.api_key.as_deref(), + Some("fresh-access"), + "the live model's request auth rebinds from the store" + ); +} + +/// Before the session's first build there is no live target to refresh: +/// the reload installs nothing (the build binds its own target) — the MCP +/// re-read alone runs. +#[test] +fn reload_without_a_live_target_installs_nothing() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + write_credential_backed_provider(&agent_dir); + write_oauth_credential(&agent_dir, "fresh-access"); + let engine = credential_backed_engine(dir.path()); + engine.reload_live_inputs().expect("the reload applies"); + assert!( + engine + .provider_target + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .is_none(), + "the reload never installs a target" + ); +} + +/// A routed image episode re-applies its armed target on every model-turn +/// attempt (overflow retries, failover restores): the reload must refresh +/// the armed route's auth too, or the next attempt reinstalls the stale +/// credentials a re-login just replaced. +#[test] +fn reload_refreshes_the_armed_image_route_target_auth() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + write_two_model_provider(&agent_dir); + write_oauth_credential(&agent_dir, "fresh-access"); + let engine = credential_backed_engine(dir.path()); + let session_model = engine.resolve_model().expect("the pinned model resolves"); + let routed: pa_types::ai::Model = + pa_core::session_engine::provider_adapter::json_round_trip(&serde_json::json!({ + "id": "mock-vision", + "name": "Mock Vision", + "api": "openai-completions", + "provider": "battery", + "baseUrl": "http://127.0.0.1:9", + "reasoning": false, + "input": ["text"], + "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, + "contextWindow": 128_000, + "maxTokens": 4096, + })) + .expect("the routed model converts"); + let override_model: pa_agent::types::Model = + pa_core::session_engine::provider_adapter::json_round_trip(&routed) + .expect("the override model converts"); + *engine + .image_route + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = + Some(crate::image_route::ImageRoute { + target: ProviderTarget { + service_tier: None, + api_key: Some("stale-access".to_string()), + model: routed, + headers: None, + }, + agent_override: pa_agent::agent::AgentModelOverride { + model: override_model, + thinking_level: pa_agent::types::ThinkingLevel::default(), + }, + // The saved session target a failing route clear restores: it + // carries the pre-login key and must re-bind with the rest. + session_target: Some(ProviderTarget { + service_tier: None, + api_key: Some("stale-session-access".to_string()), + model: session_model, + headers: None, + }), + }); + engine.reload_live_inputs().expect("the reload applies"); + let armed = engine + .image_route + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .take() + .expect("the reload keeps the armed route"); + assert_eq!( + armed.target.model.id, "mock-vision", + "the armed route keeps its routed model" + ); + assert_eq!( + armed.target.api_key.as_deref(), + Some("fresh-access"), + "the armed route's request auth rebinds from the store" + ); + let saved = armed + .session_target + .as_ref() + .expect("the saved session target stays"); + assert_eq!( + saved.api_key.as_deref(), + Some("fresh-access"), + "the saved session target's request auth rebinds from the store" + ); +} + +/// A failover's primary restore re-resolves the request auth from the +/// store: a credential rotated (and reloaded) during the failover serves +/// from the store, never the pre-failover capture; the capture only backs +/// a resolution that yields nothing. +#[test] +fn restored_primary_target_serves_the_store_auth() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + write_credential_backed_provider(&agent_dir); + write_oauth_credential(&agent_dir, "fresh-access"); + let engine = credential_backed_engine(dir.path()); + let primary = engine.resolve_model().expect("the primary resolves"); + let restored = + engine.restored_primary_target(&primary, Some("stale-capture".to_string()), None); + assert_eq!( + restored.api_key.as_deref(), + Some("fresh-access"), + "the restored primary serves the stored credential" + ); + assert_eq!( + restored.model.id, primary.id, + "the restore keeps the primary model" + ); + + // A provider with nothing configured anywhere: the capture backs the + // restore, not an empty key. + let ghost: pa_types::ai::Model = + pa_core::session_engine::provider_adapter::json_round_trip(&serde_json::json!({ + "id": "ghost-1", + "name": "Ghost 1", + "api": "openai-completions", + "provider": "ghost", + "baseUrl": "http://127.0.0.1:9", + "reasoning": false, + "input": ["text"], + "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, + "contextWindow": 128_000, + "maxTokens": 4096, + })) + .expect("the ghost model converts"); + let restored = engine.restored_primary_target(&ghost, Some("stale-capture".to_string()), None); + assert_eq!( + restored.api_key.as_deref(), + Some("stale-capture"), + "a resolution that yields nothing falls back to the capture" + ); + + // An unreadable store keeps the captured pair: the fresh resolution + // would serve the configured fallback key, which is not a credential + // — the same overwrite /reload's gate avoids. + std::fs::write(agent_dir.join("auth.json"), "not json").unwrap(); + let restored = engine.restored_primary_target( + &primary, + Some("captured-key".to_string()), + Some(team_headers("captured-team")), + ); + assert!( + restored.api_key.as_deref() == Some("captured-key"), + "a failed store read keeps the last-good captured key" + ); + assert!( + restored.headers.as_ref() == Some(&team_headers("captured-team")), + "a failed store read keeps the last-good captured headers" + ); +} + +/// A custom `prime-inference` provider: the one provider whose stored +/// credentials can carry request headers (the team header), so the +/// reload's header replacement is observable against a real resolution. +fn write_prime_inference_provider(agent_dir: &std::path::Path) { + std::fs::create_dir_all(agent_dir).unwrap(); + std::fs::write( + agent_dir.join("models.json"), + serde_json::json!({ + "providers": { + "prime-inference": { + "api": "openai-completions", + "baseUrl": "http://127.0.0.1:9", + "apiKey": "sk-config-fallback", + "models": [ + { + "id": "mock-1", + "name": "Mock 1", + "api": "openai-completions", + "contextWindow": 128_000, + "maxTokens": 4096, + } + ] + } + } + }) + .to_string(), + ) + .unwrap(); +} + +/// One stored `prime-inference` API-key credential, with or without the +/// team header `get_provider_headers` derives from `primeTeam`. +fn write_prime_inference_credential(agent_dir: &std::path::Path, key: &str, team_id: Option<&str>) { + std::fs::create_dir_all(agent_dir).unwrap(); + let mut credential = serde_json::json!({ "type": "api_key", "key": key }); + if let Some(team_id) = team_id { + credential["primeTeam"] = + serde_json::json!({ "teamId": team_id, "name": format!("Team {team_id}") }); + } + std::fs::write( + agent_dir.join("auth.json"), + serde_json::json!({ "prime-inference": credential }).to_string(), + ) + .unwrap(); +} + +fn team_headers(team_id: &str) -> std::collections::BTreeMap { + [("X-Prime-Team-ID".to_string(), team_id.to_string())] + .into_iter() + .collect() +} + +fn prime_inference_backed_engine(dir: &std::path::Path) -> AgentSessionEngine { + AgentSessionEngine::new(AgentEngineConfig { + cwd: dir.to_path_buf(), + agent_dir: dir.join("agent"), + provider: Some("prime-inference".to_string()), + model: Some("mock-1".to_string()), + api_key: None, + thinking: None, + session_dir: None, + session_file: None, + faux_script: None, + supervisor_link: None, + telemetry_disabled: Some(true), + cron_store: None, + queued_steering_probe: None, + }) + .expect("engine") +} + +/// A rotated credential REPLACES the request headers: a fresh key that +/// carries no team header must clear the previous credential's +/// `X-Prime-Team-ID`, never serve the stale team context on the new key. +#[test] +fn reload_clears_headers_when_the_fresh_credential_carries_none() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + write_prime_inference_provider(&agent_dir); + write_prime_inference_credential(&agent_dir, "team-key", Some("team-1")); + let engine = prime_inference_backed_engine(dir.path()); + let model = engine.resolve_model().expect("the custom model resolves"); + // The session's build-time target: the team credential the engine saw. + *engine + .provider_target + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(ProviderTarget { + service_tier: None, + api_key: Some("team-key".to_string()), + model, + headers: Some(team_headers("team-1")), + }); + // A re-login without the team replaces the stored credential. + write_prime_inference_credential(&agent_dir, "solo-key", None); + engine.reload_live_inputs().expect("the reload applies"); + let target = engine + .provider_target + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + .expect("the reload keeps the target set"); + // The resolved credential never formats into a failure message: an + // api key is sensitive data (CodeQL's cleartext-logging rule), so the + // assertion compares without the assert_eq! formatting sink. + assert!( + target.api_key.as_deref() == Some("solo-key"), + "the reload rebinds the request auth to the fresh stored credential" + ); + assert!( + target.headers.is_none(), + "the fresh credential clears the stale team header" + ); +} + +/// A failover's primary restore serves the resolved pair, never the +/// resolved key paired with the captured headers: a credential rotated +/// during the failover carries its own (empty) headers, clearing the +/// captured `X-Prime-Team-ID` the pre-failover credential pinned. +#[test] +fn restored_primary_target_serves_the_resolved_header_pair() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + write_prime_inference_provider(&agent_dir); + write_prime_inference_credential(&agent_dir, "solo-key", None); + let engine = prime_inference_backed_engine(dir.path()); + let primary = engine.resolve_model().expect("the primary resolves"); + let restored = engine.restored_primary_target( + &primary, + Some("captured-key".to_string()), + Some(team_headers("old-team")), + ); + // The resolved credential never formats into a failure message: an + // api key is sensitive data (CodeQL's cleartext-logging rule), so the + // assertion compares without the assert_eq! formatting sink. + assert!( + restored.api_key.as_deref() == Some("solo-key"), + "the restored primary serves the stored credential" + ); + assert!( + restored.headers.is_none(), + "the resolved pair replaces the captured headers" + ); +} + +/// A reload over a malformed auth store reports the failure: the store +/// cannot be parsed, so the MCP manager keeps its previous credentials +/// and the reload answers with the storage error instead of a success +/// the session did not apply. +#[test] +fn reload_reports_the_auth_store_failure() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + write_credential_backed_provider(&agent_dir); + write_oauth_credential(&agent_dir, "stale-access"); + let engine = credential_backed_engine(dir.path()); + let model = engine.resolve_model().expect("the custom model resolves"); + *engine + .provider_target + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(ProviderTarget { + service_tier: None, + api_key: Some("stale-access".to_string()), + model, + headers: None, + }); + // Another process corrupts the store: the reload's re-read fails. + std::fs::write(agent_dir.join("auth.json"), "not json").unwrap(); + let error = engine + .reload_live_inputs() + .expect_err("the malformed store fails the reload"); + assert!(!error.is_empty(), "the reload surfaces the storage error"); + // The failed reload never touched the target: the store gate ran + // before any rebind, so the session keeps its last-good request auth + // instead of falling to the provider's configured fallback key. + let target = engine + .provider_target + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + .expect("the failed reload keeps the target set"); + assert!( + target.api_key.as_deref() == Some("stale-access"), + "a failed reload preserves the last-good request auth" + ); + // A repaired store reloads cleanly again. + write_oauth_credential(&agent_dir, "fresh-access"); + engine + .reload_live_inputs() + .expect("the repaired store reloads"); + let target = engine + .provider_target + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + .expect("the reload keeps the target set"); + assert!( + target.api_key.as_deref() == Some("fresh-access"), + "the repaired store rebinds the request auth" + ); +} + +/// A create-config key override stands in memory regardless of the store: +/// when the store read fails, the restore keeps the override key and the +/// captured headers (the last-good team context), never the header-less +/// resolution the broken store produced. +#[test] +fn restored_primary_target_keeps_the_override_key_and_captured_headers_off_a_failed_store() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + write_credential_backed_provider(&agent_dir); + write_oauth_credential(&agent_dir, "stored-key"); + let engine = credential_backed_engine_with_override(dir.path(), Some("override-key")); + let primary = engine.resolve_model().expect("the primary resolves"); + // The store becomes unreadable mid-failover. + std::fs::write(agent_dir.join("auth.json"), "not json").unwrap(); + let restored = engine.restored_primary_target( + &primary, + Some("override-key".to_string()), + Some(team_headers("team-1")), + ); + assert!( + restored.api_key.as_deref() == Some("override-key"), + "the override key stands in memory off a failed store" + ); + assert!( + restored.headers.as_ref() == Some(&team_headers("team-1")), + "a failed store read keeps the captured headers" + ); + // A healthy store still resolves the override pair fresh. + write_oauth_credential(&agent_dir, "stored-key"); + let restored = engine.restored_primary_target( + &primary, + Some("override-key".to_string()), + Some(team_headers("team-1")), + ); + assert!( + restored.api_key.as_deref() == Some("override-key"), + "the healthy restore keeps the override key" + ); +} + +/// A healthy store that no longer holds the provider's credential +/// leaves the target untouched: the resolution would serve the +/// configured models.json fallback key, which is not a stored +/// credential — the same never-clears contract a broken store keeps. +#[test] +fn reload_keeps_the_target_when_the_credential_leaves_the_store() { + let dir = tempfile::TempDir::new().unwrap(); + let agent_dir = dir.path().join("agent"); + write_credential_backed_provider(&agent_dir); + write_oauth_credential(&agent_dir, "live-access"); + let engine = credential_backed_engine(dir.path()); + let model = engine.resolve_model().expect("the custom model resolves"); + *engine + .provider_target + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(ProviderTarget { + service_tier: None, + api_key: Some("live-access".to_string()), + model, + headers: None, + }); + // Another process removes the credential: the store stays healthy, + // but nothing is stored for the provider. + std::fs::write(agent_dir.join("auth.json"), "{}").unwrap(); + engine.reload_live_inputs().expect("the reload applies"); + let target = engine + .provider_target + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + .expect("the reload keeps the target set"); + assert!( + target.api_key.as_deref() == Some("live-access"), + "an empty store keeps the last-good stored credential" + ); + // A re-login restores the credential and the rebind resumes. + write_oauth_credential(&agent_dir, "fresh-access"); + engine.reload_live_inputs().expect("the reload applies"); + let target = engine + .provider_target + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + .expect("the reload keeps the target set"); + assert!( + target.api_key.as_deref() == Some("fresh-access"), + "a restored credential rebinds the request auth" + ); +} diff --git a/crates/pa-daemon/src/agent_engine/turn/model.rs b/crates/pa-daemon/src/agent_engine/turn/model.rs index 9c2d403e8..495756cd5 100644 --- a/crates/pa-daemon/src/agent_engine/turn/model.rs +++ b/crates/pa-daemon/src/agent_engine/turn/model.rs @@ -347,13 +347,24 @@ impl AgentSessionEngine { // The stream's provider target follows the switch (the same slot // `set_model` swaps). { + // The switch's resolve-and-install serializes + // with `/reload`'s live-input refresh — the + // restore path in the sibling callback takes + // the same lock: a reload landing between + // this resolution and its slot write must not + // be clobbered by the switch's older pair. The + // turn runs on the blocking pool, so the + // synchronous resolution never touches an + // async executor worker. + let _reload_serialized = self + .reload_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); // A routed episode keeps serving the route's - // target across the failover switch. - if let Some(route) = self.armed_image_route() { - let mut target = - self.provider_target.write().expect("provider target lock"); - *target = Some(route.target); - } else { + // target across the failover switch — through + // the fenced install (the route lock covers + // the clone and the slot write together). + if !self.install_armed_route_target() { let (api_key, headers) = self.resolve_request_key_and_headers(&next); let mut target = @@ -395,20 +406,40 @@ impl AgentSessionEngine { let agent_model = json_round_trip(&primary_model) .ok_or_else(|| anyhow::anyhow!("model conversion failed"))?; { - let mut target = - self.provider_target.write().expect("provider target lock"); - if let Some(route) = self.armed_image_route() { - *target = Some(route.target); - } else { - *target = Some(ProviderTarget { - service_tier: *self - .service_tier - .read() - .expect("service tier lock"), - api_key: primary_api_key, - model: primary_model.clone(), - headers: primary_headers, - }); + // The restore serializes with `/reload`'s + // live-input refresh: without this fence a + // reload landing between the restore's + // resolution and its slot write is clobbered + // by the restore's older pair — the lock + // makes whichever runs last leave the newest + // store standing. The whole turn (this + // closure included) runs on the blocking + // pool — the worker parks the turn there + // (`worker/turn.rs`) — so the synchronous + // credential resolution below, OAuth refresh + // and all, never touches an async executor + // worker; only a concurrently reloading + // session waits on this span. + let _reload_serialized = self + .reload_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + // The armed-route install goes through the + // fenced helper (no provider-slot lock held + // across the route read). + if !self.install_armed_route_target() { + // The primary's request auth re-resolves at + // restore: a credential rotated (and + // reloaded) during the failover serves from + // the store, never the pre-failover + // capture. + let target = self.restored_primary_target( + &primary_model, + primary_api_key, + primary_headers, + ); + *self.provider_target.write().expect("provider target lock") = + Some(target); } } agent.set_model(agent_model).await; diff --git a/crates/pa-daemon/src/image_route.rs b/crates/pa-daemon/src/image_route.rs index d394d0c39..6af3c3ded 100644 --- a/crates/pa-daemon/src/image_route.rs +++ b/crates/pa-daemon/src/image_route.rs @@ -89,6 +89,13 @@ impl AgentSessionEngine { /// at every model-turn attempt; a text-only session model with an /// unusable or missing `settings.imageModel` returns the refusal. pub(crate) fn arm_image_turn_route(&self, carries_images: bool) -> Result<(), String> { + // The lock spans the route's auth resolution and its publication: + // a `/reload` in between would refresh a route the arm then + // overwrites with the older pair. + let _reload_serialized = self + .reload_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); let route = self .resolve_image_turn_route(carries_images) .map_err(|error| format!("{error:#}"))?; @@ -153,7 +160,22 @@ impl AgentSessionEngine { /// Apply the armed route to a model turn (after the session build, so /// the build-time target cannot clobber it): the provider target and /// the agent's per-run override swap to the routed image model. + /// + /// The route lock is held through the slot write: a `/reload` that + /// refreshes the armed route's auth and then the slot's can never be + /// overtaken by an older clone of this route installing itself after + /// the refresh — the clone and its install are one critical section. pub(crate) fn apply_armed_image_route(&self, agent: &std::sync::Arc) { + // The lock spans the session-target capture AND the route install: + // `/reload` refreshes the armed route, releases the route lock, + // then rebinds the live slot — an apply in that window would + // capture the pre-reload session target (and a later clear would + // restore it over the reload's pair). Both run under the same + // lock: whichever runs last leaves the newest store standing. + let _reload_serialized = self + .reload_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); let mut slot = self .image_route .lock() @@ -170,15 +192,46 @@ impl AgentSessionEngine { .clone_from(&self.provider_target.read().expect("provider target lock")); } let route = route.clone(); - drop(slot); agent.set_model_override(Some(route.agent_override)); *self.provider_target.write().expect("provider target lock") = Some(route.target); } + /// Install the armed route's target as the serving slot: the route + /// lock is held through the slot write, so the clone and its install + /// are one critical section — a `/reload` that refreshes the armed + /// route and then the slot can never be overtaken by an older clone + /// (the failover switch and restore install through here too). + /// Returns `false` when no route is armed. + pub(crate) fn install_armed_route_target(&self) -> bool { + let slot = self + .image_route + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let Some(route) = slot.as_ref() else { + return false; + }; + let target = route.target.clone(); + *self.provider_target.write().expect("provider target lock") = Some(target); + true + } + /// Clear the armed route and restore the session's serving target (a /// fresh resolution, so a mid-episode model switch is honored): the /// next dispatched batch re-evaluates the routing against it. pub(crate) fn clear_image_route(&self) { + // The lock spans the route take AND the clear's resolve-and-install: + // the reload refreshes the armed route in place, so a clear that + // took the route first would leave the reload unable to refresh + // the detached pair, and the clear's fallback restore would then + // clobber the reload's slot refresh with the pre-reload + // `session_target`. Both run under the same lock — whichever runs + // last leaves the newest store standing. This runs on the turn's + // blocking thread (the worker parks the turn there), so the + // synchronous resolution never touches an async executor worker. + let _reload_serialized = self + .reload_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); let route = self .image_route .lock() @@ -194,6 +247,12 @@ impl AgentSessionEngine { if let Some(agent) = agent { agent.set_model_override(None); } + // The cleared slot returns to the CURRENT session state: the + // freshly resolved model (a mid-episode switch is honored) with + // its chain-resolved auth — the create-config override or the + // configured fallback key included, exactly what a fresh session + // of this model would serve. The captured session target only + // backs a resolution that cannot rebuild the model at all. let mut target = match self.resolve_model() { Ok(model) => { let (api_key, headers) = self.resolve_request_key_and_headers(&model); diff --git a/crates/pa-daemon/src/mcp_connections.rs b/crates/pa-daemon/src/mcp_connections.rs index 3aaabb7b3..af0004fa7 100644 --- a/crates/pa-daemon/src/mcp_connections.rs +++ b/crates/pa-daemon/src/mcp_connections.rs @@ -44,8 +44,10 @@ impl Worker { // The auth store re-read comes FIRST: the interactive client // stores the api-key credentials through its own storage // instance (the `/mcp` key flow runs client-side), so the - // view's reads must reload the store to see them. - manager.reload_auth_storage(); + // view's reads must reload the store to see them. A failed + // reload serves the roster's last-good credentials — the + // view reports what the manager has, not the reload's error. + let _auth_reload = manager.reload_auth_storage(); manager.refresh(); ( manager.connection_roster(), diff --git a/crates/pa-daemon/src/session_custom.rs b/crates/pa-daemon/src/session_custom.rs index 9b2313695..1383a0a0b 100644 --- a/crates/pa-daemon/src/session_custom.rs +++ b/crates/pa-daemon/src/session_custom.rs @@ -305,13 +305,45 @@ impl Worker { response_success(None, "refine", Some(result)) } - /// `reload`: re-read the session's live inputs — settings, provider auth, - /// the MCP user-server config. This port resolves each per use, so the - /// command answers the TS success. - pub(crate) fn handle_reload(&self) -> DaemonResponse { + /// `reload`: re-read the session's live inputs — the request auth + /// rebinds from the credential store (a `/login` from another process + /// never reaches the running session's request path otherwise), and + /// the MCP manager re-reads its settings and auth state. Settings and + /// the model catalog resolve per use and need no re-read. The reload + /// touches the MCP auth store, whose snapshot takes a blocking lock — + /// it parks on a blocking thread, the same posture as + /// `get_mcp_connections`. + pub(crate) async fn handle_reload(&self) -> DaemonResponse { if let Err(response) = self.require_created("reload") { return response; } + if let Some(engine) = &self.agent_engine { + let engine = std::sync::Arc::clone(engine); + // The blocking thread's reload can fail two ways: the auth + // store's reload (a malformed document, an unacquirable lock) + // or the join itself (a panic). Both answer the client's + // reload with the failure, never a success the session did + // not apply. + match tokio::task::spawn_blocking(move || engine.reload_live_inputs()).await { + Ok(Ok(())) => {} + Ok(Err(error)) => { + return response_failure( + None, + "reload", + &format!("session reload failed: {error}"), + None, + ); + } + Err(error) => { + return response_failure( + None, + "reload", + &format!("session reload failed: {error}"), + None, + ); + } + } + } response_success(None, "reload", None) } } @@ -970,4 +1002,45 @@ mod tests { assert!(response.success, "failed: {response:?}"); assert!(response.data.is_none()); } + + /// The reload parks the engine's blocking work off the async executor: + /// with a REAL agent engine behind the worker (the faux-script engine) + /// the reload reaches the MCP auth store's blocking lock, which panics + /// when taken on the runtime — this fails if the blocking work moves + /// back onto the executor. + #[allow(clippy::await_holding_lock)] // the faux registry is process-global: the guard must span the async flow + #[tokio::test] + async fn reload_parks_the_engine_reload_off_the_runtime() { + let _faux = crate::agent_engine::tests::FAUX_TEST_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + // A TempDir keeps the worker's files from piling up in /tmp. + let dir = tempfile::TempDir::new().unwrap(); + let dir_path = dir.path().to_path_buf(); + let config = crate::worker::WorkerConfig { + socket_path: dir_path.join("worker.sock"), + supervisor_socket_path: std::path::PathBuf::new(), + token: "token".to_string(), + worker_instance_id: String::new(), + active_session_id: "custom-session".to_string(), + agent_dir: dir_path.join("agent"), + recovery_journal_path: dir_path.join("recovery.jsonl"), + telemetry_disabled: None, + script: Some(json!({ "engine": "faux", "responses": ["ack"] })), + }; + let worker = Arc::new(crate::worker::Worker::new(config, None)); + let created = worker + .dispatch("create", &json!({ "noSession": true, "cwd": "/tmp" })) + .await; + assert!(created.success, "create failed: {created:?}"); + assert!( + worker.agent_engine.is_some(), + "the faux script must build a real agent engine" + ); + let response = worker + .dispatch("reload", &json!({ "activeSessionId": "custom-session" })) + .await; + assert!(response.success, "failed: {response:?}"); + assert!(response.data.is_none()); + } } diff --git a/crates/pa-daemon/src/worker/commands.rs b/crates/pa-daemon/src/worker/commands.rs index 26ae1c203..a53c01ca9 100644 --- a/crates/pa-daemon/src/worker/commands.rs +++ b/crates/pa-daemon/src/worker/commands.rs @@ -161,7 +161,7 @@ impl Worker { "restore_next_turn" => self.handle_restore_next_turn(payload), "restore_actions" => self.handle_restore_actions(payload), "refine" => self.handle_refine(payload).await, - "reload" => self.handle_reload(), + "reload" => self.handle_reload().await, "cancel_rlm_child" => self.handle_cancel_rlm_child(payload).await, "delete_rlm_subagent" => self.handle_delete_rlm_subagent(payload).await, // The engine call blocks on the engine runtime, so it -- knotmirror 0.0.1