AI Model Lifecycle

SpectraCheck’s AI predictions are produced by versioned, governed models — never a single hard-coded predictor. This page catalogs the model-lifecycle backend that makes every AI-assisted result reproducible, auditable, and continuously improving: a model registry and inference router, a licence-aware datasets pipeline, an evaluation harness with a dominance gate, domain fine-tuning, closed-loop feedback, active learning, and the MLOps monitoring + deployment gate.

Across every stage the deterministic structure verifier remains the sole arbiter of correctness — models sharpen ranking and routing, they never override the science. The release timeline at the end gives chronological context.

Backend capabilities

Model registry & inference router

• Append-only model registry — moltrace.spectroscopy.ai.registry versions every model artifact (role, semantic version, SHA-256, training-data lineage = dataset-snapshot hash + row count, lifecycle status) with deterministic content addressing. Entries are immutable, duplicate ids are rejected, and lifecycle changes append as events; promotion to production auto-retires the incumbent for the same (role, nucleus). One pluggable store backs PostgreSQL in production and SQLite in tests. (v0.12.0, 2026-06-07)
• 5-layer inference router — InferenceRouter.predict_shifts_routed resolves each atom LoRA → NMRNet → HOSE fallback (the fine-tuned adapter is used only when a production adapter exists for the nucleus and the conformer-ensemble uncertainty is within its validated confidence band). Every prediction carries a complete, deterministic model_versions map that feeds the audit trail verbatim — one prediction, one immutable provenance record — so a result is reproducible bit-for-bit from the registry + lineage, and a reviewer sees which artifact produced each number and why one layer was chosen. (v0.12.0, 2026-06-07)

Public datasets pipeline

• Licence-aware ingestion + version pinning — moltrace.spectroscopy.data.datasets_pipeline turns the canonical public datasets (NMRShiftDB2, HMDB, BMRB, MassBank EU, GNPS, METLIN, QM9-NMR, 2DNMRGym, AIST SDBS) into a deduplicated, version-pinned corpus. Each source pins its upstream version + licence and is content-hashed — a changed upstream hash raises UpstreamChangedError rather than being silently accepted — and non-redistributable sources (SDBS, METLIN) are never written into a redistributable corpus. Normalization is RDKit-standardised SMILES + InChIKey with dedup by (InChIKey, spectral-hash); invalid records are quarantined with reasons, not dropped. (v0.13.0, 2026-06-07)
• Leakage-free frozen holdout — freeze_splits produces deterministic, seeded train/val/test splits grouped by InChIKey skeleton so a molecule never straddles splits. The test split is the sacred evaluation holdout — experimental-only, checksummed, and returned as a hash-exclusion set; assert_training_excludes_holdout is the guard every fine-tune must call. Computed (QM9) records are train-only and are dropped when they share a molecule with the eval set. (v0.13.0, 2026-06-07)

Evaluation harness & dominance gate

• The ten-metric dominance gate — moltrace.spectroscopy.eval.harness promotes a model version only when its full metric vector dominates the incumbent on a frozen, checksum-locked 100-spectrum gold set (60 NMRShiftDB2 + 20 HMDB + 20 in-house). The ten metrics: top-1 / top-3 structure accuracy, ¹H / ¹³C shift MAE, ECE, false-confirmation rate, retrieval recall@k, error-vs-uncertainty AUROC, perturbation robustness, reviewer-agreement rate, and end-to-end latency p50 / p95. No model ships on a single improved number, and the safety-critical metrics (false-confirmation rate, calibration) may never regress. (v0.14.0, 2026-06-07)
• CI promotion gate — gate_for_ci wraps evaluate + dominance against the production incumbent with exit codes 0 promotable / 1 not promotable / 2 gold-set checksum drift; GoldSet.assert_integrity() aborts the run if the holdout size or checksum drifts, so it can never be silently contaminated. (v0.14.0, 2026-06-07)

Domain fine-tuning — LoRA

• LoRA domain adapters with K-fold validation — moltrace.spectroscopy.ai.finetune trains a LoRA adapter (low rank r=8–16; train the adapter, freeze the base) on accumulated reviewer-validated in-house spectra, validates it with K-fold cross-validation (GAMP 5 Appendix D11), and registers it with full lineage. Each run freezes an immutable, content-addressed training snapshot that subtracts the holdout at freeze time — a leaked record raises HoldoutLeakageError. The run_id is a path-independent content address of the full manifest (hyperparameters, per-fold + aggregate MAE, snapshot hash, base id, code git sha, adapter SHA-256); GPU-hours and cost are logged. (v0.17.0, 2026-06-07)
• Gated registration, never auto-promoted — register_if_eligible evaluates the dominance gate on the frozen gold set and registers the adapter as candidate always, promotes to shadow only if it does not regress the incumbent, and never auto-promotes to production (human sign-off required). Registration refuses if the snapshot’s gold checksum disagrees with the live gold set — no adapter is registered without complete lineage. The trainer is injectable, so the whole pipeline runs deterministically on a CPU-only host; adapter weights are cached out of git, with only the SHA-256 + lineage persisted. (v0.17.0, 2026-06-07)

Hyperparameter optimization, calibration & contradiction detection

• Bayesian hyper-parameter optimization — optimize_hyperparameters runs Optuna (seeded TPE) over the five LoRA knobs, budgeted to ~10 trials (a budget, not a sweep). Each trial runs a full K-fold CV scored by a mean-CV-MAE + calibration objective, every trial is logged, and the resulting HPOStudy is reproducible (its id excludes wall-clock time). The best config feeds finetune_lora, so a trained adapter is traceable back to the search that chose its hyper-parameters. (v0.18.0, 2026-06-07)
• Confidence calibration as a promotion gate — temperature / Platt calibration heads, with ECE enforced as a first-class gate: a candidate whose gold-set ECE exceeds max_ece is not promotable even if it dominates on accuracy. A model that is more accurate but lies about its confidence does not ship. (v0.18.0, 2026-06-07)
• Contradiction detection — a deterministic detector plus a trained, calibrated classifier flag internal inconsistencies: no single consistent structure (from verifier verdicts), cross-modal NMR-vs-MS disagreement, and intra-spectral impossibilities (integration vs proton count, multiplicity vs coupling neighbours via the n+1 rule, a shift outside its plausible window). It complements, never replaces, the deterministic verifier — surfacing to the reviewer and feeding the active-learning queue. (v0.18.0, 2026-06-07)

Leak-proof cross-validation

• GroupKFold across every CV loop — the fold partitioner is group-aware: whole molecules (the InChIKey connectivity skeleton) or an explicit sample / batch key are assigned to a single fold, so a physical sample’s multiple scans never straddle the train and eval sides of a fold. This closes the cross-batch leakage that makes naive per-spectrum K-fold report optimistic, untrustworthy metrics, and is threaded through fine-tuning, HPO, and the contradiction trainer. With no grouping signal it reproduces the historical per-record split byte-for-byte, and every run manifest records its cv_strategy (group_kfold / kfold, group key, group count) for auditable lineage. (v0.18.1, 2026-06-07)

Closed-loop feedback & RLHF

• In-app feedback capture — moltrace.spectroscopy.feedback rates every AI output (predicted shift, proposed structure, peak label, purity call) with thumbs up/down + an optional free-text correction + a structured reason, persisted as an immutable, content-addressed event carrying the exact model_versions that produced it. Corrections fan out to the labeled-example store (training data), bare overrides to the active-learning queue, and usage / override analytics roll up where the model is weakest. (v0.19.0, 2026-06-07)
• RLHF reward model — advisory, never authoritative — corrections + accept/reject signals become Bradley-Terry preference pairs and train a deterministic reward model that advisorily re-ranks the reasoner’s candidates and prioritizes the annotation queue. Verifier supremacy is enforced structurally: a verifier-accepted candidate always ranks above a rejected one — reward only orders within a verdict class. (v0.19.0, 2026-06-07)
• A/B champion vs challenger, no auto-deploy — deterministic sticky hash routing (shadow = never served / canary = controlled served fraction), dominance-gated promotion with reviewer-acceptance + override guards, human sign-off required before the registry is ever mutated, and instant rollback modeled as a routing-layer canary-kill that never touches the append-only registry. (v0.19.0, 2026-06-07)
• Structured “why was it wrong?” taxonomy — a closed 7-value reason vocabulary (wrong_shift, wrong_multiplicity, wrong_structure, missed_impurity, wrong_integration, calibration_off, other) shared by the in-app control, the API contract, and the feedback engine, so override analytics segment by exactly where the model is weak. Surfaced through the AI-inference API — see Backend / API Contract. (v0.19.1, 2026-06-07)

Active-learning loop

• The flywheel — moltrace.spectroscopy.ai.active_learning turns every reviewer override into labeled training data and actively chooses the most informative spectra for scarce expert attention. capture_override records full provenance in one append-only, idempotent record (raw-FID hash, processed spectrum + content hash, the producing model_versions, the AI output, the human correction, reviewer id + timestamp); disagreement_score blends vote-split on the top-1 structure, predicted-shift variance, and confidence spread across model variants; build_annotation_queue ranks candidates by disagreement, greedily de-dups near-identical spectra, and slices to a labeling budget. (v0.20.0, 2026-06-08)
• Self-triggering retrains + loop yield — retraining_trigger fires on a monthly schedule or a volume of new labels and wires the fine-tune chain (snapshot → LoRA → gated registration); loop_yield_metrics reports labeled examples / month, the override-rate trend over consecutive windows (a falling trend is direct, auditable evidence the model is improving), and accuracy lift per retrain — emitted to the audit trail for the operations dashboard. (v0.20.0, 2026-06-08)

MLOps: monitoring, drift & the deployment gate

• Continuous drift monitoring + lineage dashboard — moltrace.spectroscopy.ops runs four monitors and returns a worst-of ok | warn | breach report: input drift (population-stability index of nucleus / field / solvent + molecular-weight PSI vs the training snapshot — a large PSI flags new chemistry the model never saw), confidence drift, override-rate drift (reusing the loop-yield trend), and latency (p50 / p95 vs SLO), paging an injectable alerter on each breach. The registry-backed lineage dashboard reads the registry as the source of truth — per production model: its version, training-snapshot hash, gold metric vector, promotion record + supersession, and current live drift status. (v0.21.0, 2026-06-08)
• Fail-closed deployment gate — run_deployment_gate allows a deploy only if all four checks pass: dominance (no safety regression), audit-chain integrity (provenance intact), test-suite-green, and data-leakage (the training snapshot is bound to the gold checksum and its record hashes are disjoint from the holdout). Every input defaults to the failing state, so an under-specified call is blocked — it fails closed. It is wired into CI as a deployment-gate job the deploy job depends on, so a regression in the release-control logic fails CI before it can ever let a bad model reach production. (v0.21.0, 2026-06-08)

Release timeline

A chronological summary; see each subsection above for substantive detail.

Version	Date	Headline
v0.21.0	2026-06-08	MLOps: monitoring, drift detection & the fail-closed deployment gate
v0.20.0	2026-06-08	Closed active-learning loop
v0.19.1	2026-06-07	Structured feedback reason taxonomy → AI-inference API
v0.19.0	2026-06-07	Closed-loop feedback: capture, RLHF reward model & A/B rollout
v0.18.1	2026-06-07	Leak-proof GroupKFold cross-validation
v0.18.0	2026-06-07	Bayesian HPO, calibration head & contradiction detection
v0.17.0	2026-06-07	LoRA domain fine-tuning pipeline
v0.14.0	2026-06-07	Evaluation harness: the ten metrics + dominance gate
v0.13.0	2026-06-07	Public-datasets pipeline: ingestion, versioning & frozen splits
v0.12.0	2026-06-07	Model registry + 5-layer inference router