Plan: Bringing Results & Previews into the Assistant

Status: living document. Captures the architecture direction and the sequenced work-breakdown agreed in design discussion.

Goal

Let the assistant (portal/) compute and show preview insights inline and browse full-run results, while reducing the number of moving parts and moving compute off R over time.

Target architecture (down from 6 pieces to 4):

Piece	Role
`portal/` (Next.js, Neon)	Chat UI, project/state, results browsing
`engine/` (R plumber API)	Heavy compute, jobs/artifacts in S3
S3 (`fprsstore`)	Job results + insight artifacts
Neon	Projects, model versions, job index

fprs-app (Shiny) is the thing being retired. fprspy (Python) is the model parse/validate layer and the long-term home for compute as R is unwound.

Key design decisions

Preview and full are separate jobs, distinguished by type metadata ("preview" / "full"). The jobs table filters ?type=full by default; previews are retrievable but hidden from the headline list.
Insights, not raw data. The engine already has an insight registry (09_insights.R) that persists display-ready artifacts via save_artifact. The assistant reads artifacts; it never sees a 1e7×N matrix.
Preview persists like full (job_id + build_job_insights), so the assistant has one read path for both.
Lifecycle via opaque owner_id + explicit cleanup. Engine stays domain-agnostic; cleanup_storage(rules) sweeps owner-less/expired/ marked-for-deletion jobs. fprs-app's previews become owner-less → swept.
Models sent inline, not via Sheets/S3 links. Requires a new data_source.type = "inline". Enabled by the sparse transport (below).
Model storage in Neon JSONB, hash-deduped. Sparse payload (~0.66 MB) makes inline JSONB comfortable.

Results-section hierarchy

project → model (version snapshot) → results. A "model" node in the Results tree is an immutable version snapshot, not a mutable entity:

Editing a model produces a new model node on top, with no results yet; older versions keep their results nested beneath them. Nesting is what preserves run history across edits — nothing is orphaned.
Models and results both carry an auto-generated name the user can rename (never block a run on naming).
A result pins the exact content_hash it ran against; the model node it nests under is that version.
Engine owner_id = project_id (opaque to the engine). The portal owns the project→model→result mapping and does fine-grained cascades.

Schema:

Entity	Key fields	Why
`projects`	id, name, owner, created_at	Top-level grouping; cleanup unit (`owner_id`)
`models`	id, `project_id`, name (auto), `content_hash`, created_at	One row per version snapshot; a node in the Results tree
`model_contents`	`content_hash` (PK), payload JSONB	Dedup store; content, not identity
`results`	id, `model_id`, name (auto), `kind` (`preview`/`full`), `engine_job_id`, status, created_at	A run, nested under its model version

Deletion cascade: delete project → all its models + results + engine jobs (DELETE /jobs/<id>); delete model → its results + jobs; delete result → just its job.

Sequenced work

Done

PR	Scope	Status
#107	Remove legacy R parser (`01_initialize_data.R`, `validate_sheet.R`); fprspy is the only parse path; migrate/prune dependent tests; add `check-fprspy` + `check-portal` CI	Merged
#108	Sparsify transport payload (~4.4 MB → ~0.66 MB); both consumers re-inflate	Open

Engine API (next)

Item	File(s)	Notes
`data_source.type = "inline"`	`engine/inst/api.R`, `engine/R/python_bridge.R`	Route to `parse_model_transport`; keeps gsheets/excel for transition
`POST /preview`	`engine/inst/api.R`	Wrap `compute_preview`; bearer auth; sync `{job_id, results}`
Persist preview	`engine/R/compute_preview.R`	Accept `storage`/`job_id`/`owner_id`; `type="preview"`; call `build_job_insights`
Insight builders null-return	`engine/R/09_insights.R`	Each builder early-returns `NULL` when inputs absent; `build_job_insights` skips NULL — no `scope` arg
`cleanup_storage(rules)` + `POST /cleanup`	`store/R/*`, `engine/inst/api.R`	Metadata-driven: `expires_at`, owner-less preview TTL, `marked_for_deletion`
`DELETE /jobs/<id>`	`engine/inst/api.R`	`delete_job` exists in `StorageInterface`
Artifact HTTP endpoints	`engine/inst/api.R`	`GET /jobs/<id>/artifacts[/<artifact_id>]`; accept query params → `materialize_job_artifact`
Jobs list `?type=` filter	`store/R/*`	`list_all_jobs` filter param

Portal (assistant)

Item	Notes
Schema: `projects`, `models` (version snapshot, auto-name, `content_hash`), `model_contents` (hash-deduped JSONB, `storage_kind` enum escape hatch), `results` (auto-name, `kind`, `engine_job_id`)	See Results-section hierarchy above; portal owns the mapping
Typed engine client	`/preview`, `/compute`, `/status`, `/jobs/<id>/artifacts[...]`, `DELETE /jobs/<id>`
LLM tools	`list_artifacts`, `get_artifact`, `run_preview`, `run_full`, `get_job_status` (+ later `focus_card`, `open_card_with_params`)
Results tree (project → model → results); chat scoped to project; active-result panel	v1 uses generic artifact fallback renderer; editing a model adds an empty model node on top
Cascading deletes → `DELETE /jobs/<id>` per affected result	Or `marked_for_deletion` + cleanup sweep

Later

Bespoke insight cards (per artifact_id), parameterized via params_key.
Port simulate_characteristics / optimization to Python; retire R engine.
Reconsider model.schema_version only if a real incompatibility appears (not needed today — inflation accepts dense and sparse).

Insight preview/full split

Preview computes init → simulate → virtual orgs → validate (no optimization). Insights computable from preview data vs. needing full optimization:

Works on preview	Needs full optimization
`job_summary`, `problem_definition_tables`, `org_specs`, `simulation_features`*, `org_ev_summary`, `org_comparison_matrix`	`portfolio_summary_metrics`, `portfolio_allocations_`, `multipliers_information_value`, `funding_sensitivity_cube`, `funding_determinants_`

* depends on portfolios; treat as full-only unless preview path is extended.

Resolved

Hierarchy: project → model (version snapshot) → results. Project at the top (cleanup unit). Editing a model adds a new empty model node on top; old versions keep their nested results.
Names: auto-generated for models and results; user-renameable; never block a run on naming.
Stale versions: handled structurally by nesting — a new model node appears with no results; older versions remain with theirs.
Deletion: project → cascade all; model → its results; result → its job. Each affected result triggers DELETE /jobs/<id>.
/preview: synchronous for v1.

Open questions

Default TTL for owner-less preview jobs (24 h proposed) — only matters for fprs-app's transition-period previews, not portal-created ones.