Deployment & Hosting

Deployment guidance keeps local development, hosted documentation, and platform operations aligned. The production platform is split across two hosts: the backend runs on Google Cloud Run, the frontend on Vercel, and this documentation site builds to static output.

Platform hosting (Google Cloud)

The backend migrated off Render to Google Cloud in 2026-07. Render is fully retired and no longer appears in any deploy path, runbook, or CI job. (2026-07)

Backend — Cloud Run

• Service moltrace-backend, project moltrace-prod, region us-central1, running scale-to-zero (no idle instance is kept warm).
• The container listens on Cloud Run’s injected $PORT (image default 8080). Migrations never run in the container entrypoint — they are a separate Cloud Run job, moltrace-migrate, invoking alembic upgrade head.
• Non-secret runtime config (APP_ENV, LOG_LEVEL, ALLOWED_ORIGINS) is set as plain env vars; every credential-class value is injected from Secret Manager at revision start.

Database — Cloud SQL for PostgreSQL 16, private IP only

• The database is Cloud SQL for PostgreSQL 16 on a private IP with no public interface — there is no public endpoint to reach, scan, or misconfigure an allow-list against.
• Cloud Run reaches it over Direct VPC egress (--network / --subnet / --vpc-egress) rather than a Serverless VPC connector: the same private path into the VPC, without billable connector instances.
• The instance was Alembic-stamp head-ed once at bootstrap; every release since applies upgrade head through the moltrace-migrate job.

Object storage, secrets, and keys

• Cloud Storage — the immutable raw-FID vault bucket moltrace-raw-vault (versioned), plus buckets for model weights and the DVC remote.
• Secret Manager — the source of every credential-class value (DATABASE_URL, API_KEY, ADMIN_EMAILS, AUDIT_SIGNING_KEY, SSO_ENCRYPTION_KEY, MFA_ENCRYPTION_KEY, PASSWORD_PEPPER), mounted into the revision via --set-secrets. Nothing credential-class is baked into the image or held in a repo file.
• Cloud KMS — holds the field-encryption key behind the application’s envelope encryption.
• Artifact Registry — container images; Cloud Build builds and pushes them.

Frontend — Vercel and the same-origin API proxy

The frontend (moltrace.co) remains on Vercel. Browsers never call the Cloud Run URL directly: API traffic goes through a same-origin /api/backend proxy, so credentials stay first-party and the backend origin is not a public browser surface.

CI/CD — keyless deploy behind the fail-closed gate

• The backend deploy job runs only on a green push to main and needs the frontend tests, the backend tests, the fail-closed deployment gate, and the provenance-verification job. A failure in any of them blocks the deploy — the gate cannot be bypassed by a rerun.
• GitHub Actions authenticates to Google Cloud by exchanging its OIDC token through Workload Identity Federation for the moltrace-deployer@ service account. There is no stored service-account key anywhere — not in the repo, not in Actions secrets — the same keyless principle as the Sigstore/Fulcio signing used for build provenance.
• Deploy sequence: Cloud Build builds and pushes the image tagged with the commit SHA → the moltrace-migrate Cloud Run job applies the Alembic deltas → gcloud run deploy rolls the new revision. Because gcloud run deploy --image preserves the service’s existing env, secrets, and VPC configuration, CI changes only the image; configuration changes go through the deploy runbook (moltrace_backend/deploy/README.md).
• The frontend deploy fires a Vercel deploy hook from the same gated job set, so the frontend and backend releases share one release control.

Raw-FID vault on serverless (Cloud Storage backend)

Cloud Run’s filesystem is ephemeral — anything written locally is lost when an instance recycles — so the write-once ALCOA+ raw-FID vault cannot live on the container filesystem in a serverless deployment. A Cloud Storage storage backend implements the same RawStorageBackend contract against GCS and preserves every write-once invariant the local backend encodes:

• Create-only writes — each object is uploaded under an if_generation_match=0 precondition, so the write succeeds only if nothing exists at that key. Atomic and race-safe; an existing archive is never overwritten.
• Reuse + verify — when the precondition fails (the archive is already vaulted), the stored object’s SHA-256 is checked against the incoming bytes before the existing object is reused.
• Post-write hash verification — the stored bytes are re-read and re-hashed after upload, so a truncated or corrupted upload fails loudly instead of silently entering the vault. Under strict-immutable mode the freshly written object is deleted and the write raises.
• WORM — durable immutability rests on the bucket’s retention policy / object versioning; a bucket with neither is flagged in the integrity report, because immutability would then rest only on the create-only precondition.
• Verify-before-read — a read refuses to return bytes unless the integrity report passes.

Selected with RAW_VAULT_BACKEND=gcs plus RAW_VAULT_BUCKET (production: moltrace-raw-vault, versioned); RAW_VAULT_BACKEND=gcs without a bucket is a startup error. The local filesystem backend remains the default (RAW_VAULT_BACKEND=local, RAW_VAULT_DIR), so local development and non-serverless deployments are unchanged. google-cloud-storage is an optional extra (nmrcheck[gcs]) imported lazily, so the package, its tests, and the local backend all run without it installed. (2026-07)

Documentation build

cd moltrace_docs
npm run build

The generated documentation output is written to dist/.

Port expectations

• Frontend development: 3000
• Backend development: 8000
• Docs development: the Astro dev server URL, usually 4321

Keep externally mapped hosting ports separate from local development conventions so developer workflows remain predictable. In production the backend does not pin a port: Cloud Run injects $PORT (the image defaults to 8080) and the container binds to it.

Hosting checks

• Confirm the /guides/ route is available.
• Confirm favicon, manifest, logo assets, and Open Graph preview assets resolve.
• Confirm search index generation completes during the static build.
• Confirm no sidebar or header links point at missing pages.

Backup & disaster recovery

What is backed up

• Primary database — all tenant data, the tamper-evident audit ledger, and the security-event stream. Covered by the managed platform’s automated daily backups. Point-in-time recovery is not enabled on the production instance — see Where the stated RPO is not met today. Cross-region replication and immutable / object-lock retention are a documented hardening seam, not a shipped guarantee.
• Raw-data vault — the write-once vendor archives, held with the object store’s durability and replication (the production bucket is versioned; bucket retention / object lock is not configured).
• Secrets — deliberately not in backups. API_KEY, AUDIT_SIGNING_KEY, and the IdP / MFA secrets are re-provisioned from the secret store / KMS during a restore.
• Code and infrastructure definitions — git, plus the signed supply chain (SBOM + SLSA provenance verified before any deploy).

The database is the system of record, and the audit ledger inside it is the integrity oracle for every restore.

Recovery objectives (RTO / RPO)

• Database (tenant data + audit ledger) — RPO ≤ 24 h on daily backups, or ≤ 5 min with point-in-time recovery; RTO ≤ 4 h to restore, integrity-verify, and cut over.
• Application (stateless services) — RPO 0, because the app is rebuilt from git and the verified supply chain; RTO ≤ 1 h to redeploy from main.

These are objectives, not SLAs. They are validated by the restore drill below and revised against measured drill results. A region-loss event recovers by restoring the database into a secondary region and redeploying the stateless app there.

Where the stated RPO is not met today

Read the ≤ 5 min figure above as the target the operational program is built toward, not a current capability. Finding MT-VULN-2026-003 (Medium, open) records the gap plainly:

• Point-in-time recovery is not enabled. On Cloud SQL, PITR is a per-instance toggle that the deploy runbook does not currently set. The production instance is also zonal, not regional.
• The real database RPO is therefore the ≤ 24 h daily-backup window, not ≤ 5 min.
• The raw-evidence vault has object versioning but no retention or bucket lock, so its write-once property rests on the create-only precondition and versioning rather than an enforced retention policy.

Closing the database half means enabling PITR on the instance and moving it to a regional configuration; closing the vault half means configuring bucket retention / bucket lock. Until both land, quote the ≤ 24 h figure to customers, not ≤ 5 min. (v0.62.1, 2026-06-26)

Container-image vulnerability scanning gap

Finding MT-VULN-2026-002 (Medium, open). The CI security gates scan dependencies and Dockerfile misconfiguration — they never scan the built image.

The gap opened at the migration. On the retired Render deployment the documentation recorded image scanning as “N/A — no Dockerfile”, which was accurate for buildpack-based deploys. Since the move to Google Cloud, an image is built (Cloud Build → Artifact Registry → Cloud Run), so the N/A no longer holds and was withdrawn. Closing it means adding an image scanner against Artifact Registry in the release path. (v0.62.1, 2026-06-26)

Restore-integrity verification

A restore is not finished until it is proven intact, and the tamper-evident audit chain is the natural oracle: a restored database whose per-row SHA-256 chain, HMAC anchors, and signed high-water mark still verify is provable evidence that nothing was lost or altered in transit (see Security Policy → Audit trail integrity).

nmrcheck.dr_verify is a library + CLI with no API surface and no new dependencies. verify_restore re-runs the audit-chain verification and adds four restore-sanity checks:

1. audit_chain — the full chain, anchors, and signed head re-verify. A break means the restore lost or altered records.
2. audit_history_present — the restored database actually contains chained audit events, which catches an empty or wrong-database restore.
3. signing_key_not_dev — the restored deployment uses the production signing key, not the dev fallback; a dev key means the chain’s tamper-evidence cannot be trusted.
4. row_counts_meet_baseline — core tables meet the pre-loss baseline captured from the last good backup, guarding against a partial or wrong-snapshot restore.

Run it against the restored database: python -m nmrcheck.dr_verify --min-rows audit_events=1,users=1 → exit 0 integrity verified, 1 a check failed, 2 could not connect. The assess decision is a pure function and unit-tested; verify_restore is tested against both a seeded (clean) and a deliberately tampered database.

Restore drills and game days

The runbook (moltrace_backend/docs/security/backup_dr.md) prescribes a drill quarterly and after any major schema change: pick a recovery point, capture the core-table row-count baseline, restore into an isolated non-production target (never overwrite production), run dr_verify, smoke the app against the restored database (/health, GET /admin/audit/verify, a representative read), then record the drill date, recovery point, measured RTO/RPO, and verifier result — with any gap filed into the findings register. A DR game-day template covers the declared region-loss scenario end to end.

Honest boundary

Backup storage, cross-region replication, immutable / object-lock retention, and executing a region-loss restore are operational — they live in the cloud provider’s console and a secondary region, not in this repository. What ships in-repo is the restore-integrity verifier, the RTO/RPO targets, and the drill / game-day runbooks. Read the numbers above as the objectives the operational program is built to meet, not as guarantees. (v0.60.0, 2026-06-26)

Transport security (TLS / HSTS)

Browser-hardening response headers are emitted on every API response, and HSTS over HTTPS is asserted with max-age=63072000 (2 years) + includeSubDomains + preload. The TLS / HTTPS detection is keyed off the TLS-terminating edge’s X-Forwarded-Proto, so plain-HTTP local dev is never pinned to HTTPS (the header is omitted on plain HTTP and present + correct on X-Forwarded-Proto: https). The full header set: Strict-Transport-Security (HTTPS-only, configurable via HSTS_* env), X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: strict-origin-when-cross-origin, and Permissions-Policy: geolocation=(), microphone=(), camera=() (set via setdefault, so a route may override).

In production that edge is the Cloud Run front end, which terminates TLS and sets X-Forwarded-Proto on the forwarded request.

TLS 1.3, modern ciphers, certificate issuance + rotation, and service-to-service mTLS are the deployment edge’s responsibility and are captured as a documented posture + adoption runbook (moltrace_backend/docs/ops_tls_posture.md). Settings: HSTS_ENABLED, HSTS_MAX_AGE_SECONDS, HSTS_INCLUDE_SUBDOMAINS, HSTS_PRELOAD. (v0.48.0, 2026-06-16)

Secrets management (provider seam + CI gate)

Every credential-class config value — DATABASE_URL, REDIS_URL, API_KEY, the SSO / MFA encryption keys, the password pepper — is read through a single in-process secrets provider seam (secrets_provider.resolve_secret / resolve_secret_strict). The env-backed default is byte-for-byte identical to the prior os.getenv reads, so the prod startup guards still fire (empty API_KEY → None; missing strict-required vars raise). The documented swap path is a managed store — Vault, AWS / GCP Secrets Manager — and short-lived dynamic DB credentials behind the same interface, selected via SECRETS_BACKEND. No call site changes are required to move to a managed store; the adoption runbook is moltrace_backend/docs/ops_secrets_management.md.

Layered with a CI / pre-commit secret-scanning gate that blocks the build on any committed secret — see Security Policy → Secrets management. (v0.47.0, 2026-06-16)

That documented swap is now in place in production: Cloud Run injects every credential-class value from Google Secret Manager at revision start, and the field-encryption key lives in Cloud KMS — see Platform hosting. No call sites changed; the seam absorbed the move. (2026-07)

GSD experimental backend rollout (v0.6 soak loop)

The opt-in Global Spectral Deconvolution backend (POST /spectrum/analyze/gsd, see NMR Interpretation) ships behind a per-request experimental: true flag while it accumulates a soak record on real-tenant traffic. The full pipeline from per-call telemetry to per-tenant graduation is feature-complete: the readiness panel renders in two API calls and the entire policy is owned by the backend.

Pipeline

1. Per-call audit event — every opt-in GSD invocation writes a structured spectrum.analyze_gsd audit event capturing the request shape (level, nucleus, declared solvent, field_mhz, input_point_count, wall_ms) and outcome shape (peak / environment counts by category, detected solvents, error_kind on failure). Tenants can query their own events via GET /audit/events?event_type=spectrum.analyze_gsd. (v0.6.3)
2. Aggregate rollup — GET /spectrum/analyze/gsd/telemetry-summary?window_days=N (admin-only; default 90 days, clamped [1, 365]) returns a pre-aggregated SpectrumGSDTelemetrySummary — invocations, error rate, median / p95 wall time, solvent auto-detect rate, plus per-nucleus / per-level / per-error-kind slice breakdowns. The readiness panel reads off the rollup; the raw event stream stays available for tenant-scoped per-event inspection. (v0.6.4)
3. Flip-readiness verdict — the rollup carries flip_readiness_verdict ("insufficient_data" | "clear" | "blocked"), flip_readiness_reasons (human-readable strings the FE shows verbatim), and flip_readiness_policy (the threshold snapshot: min_invocations=500, max_error_rate=0.05, min_solvent_detect_rate=0.95). A future policy tightening is a one-line backend change with no FE deploy required. (v0.6.5)
4. Per-tenant scope — GET /spectrum/analyze/gsd/telemetry-summary?actor_user_id=<id> (admin-only) computes the same verdict over only that user’s audit stream. The response echoes scope_actor_user_id so cached or replayed responses are self-describing. (v0.6.6)
5. Graduation action — POST /admin/users/{user_id}/gsd-graduation with body {"graduated": bool, "reason": str} (reason required, 1–500 chars — regulatory-relevant audit evidence). Idempotent on repeat-graduate. users.gsd_graduated_at is a nullable timestamp (None = still experimental; timestamp = graduated at that moment); spectrum_analyze_gsd consults it at request time so graduated tenants get experimental: false in both the response and the soak-telemetry audit event. API-key callers (no user attached) stay on experimental: true. (v0.6.7)
6. Adoption telemetry — the rollup carries graduated_user_count (full platform count globally; 0 or 1 when scoped). The FE readiness panel can render “X tenants graduated” from a single API call. (v0.6.8)
7. Per-tenant graduation history — GET /admin/users/{user_id}/gsd-graduation-history (admin-only) returns the full graduate / ungraduate sequence newest-first, each event carrying the admin’s documented reason and structured before/after state. Auditors reconstruct every decision without filtering the global audit stream client-side. (v0.6.9)
8. Adoption velocity — the rollup carries newly_graduated_in_window (unique users with a graduate event inside the window; repeat graduations dedup, ungraduate events don’t count). The readiness panel renders “X tenants graduated this quarter” alongside the snapshot count. (v0.6.10)

Operational summary

• Two-call FE readiness panel — rollup (/telemetry-summary) plus per-tenant graduation history (/gsd-graduation-history) covers the full v0.6 story for the readiness review meeting.
• Backend-owned policy — the flip decision lives in _compute_flip_readiness_verdict; tightening (for example, raising min_invocations to 2000) is a one-line backend change that lands in every caller’s rollup with no FE deploy.
• Audit-first — every graduation decision is reconstructible end-to-end without a separate reporting pipeline.