Prompt & Context Management
Treat prompts, retrieval sources and model choices as versioned configuration.
- ▲
Versioned configurations
Share of prompts / retrieval sources / model choices under version control in a registry.
- ≈
Context size per request
Tokens of context assembled per call (system prompt + retrieved docs + history).
Evaluation
Measure quality before you ship a change and continuously after.
- ▲
Golden-dataset score
Pass rate / mean rubric score on the curated regression set.
- ▲
Eval coverage
Share of priority use-case scenarios represented in the golden dataset.
- ▲
Judge–human agreement
Correlation between LLM-as-a-judge scores and human labels on a sample.
Deployment & Rollout
Change the system gradually and reversibly.
- ▲
Gated rollout rate
Share of changes released via canary or shadow before going wide.
- ▼
Time to roll back
Minutes from detecting a bad change to reverting it.
Monitoring & Observability
See what the system is actually doing in production.
- ▲
Sampled live quality
Scored quality on a sample of real production traffic.
- ▼
Drift indicator
Divergence of input distribution or output quality from baseline.
- ▼
Error & refusal rate
Share of requests erroring or being refused / falling back.
- ▼
Latency (p50 / p95)
Time to first token and total response time.
Cost & Performance
Keep the bill and the speed under deliberate control.
- ▼
Cost per request
Average spend per request (input + output tokens × model price).
- ≈
Cheap / expensive model mix
Share of traffic served by small vs frontier models.
- ▲
Cache / reuse rate
Share of requests served from cached or reused results.
Incident & Governance
When something breaks, catch it fast and learn; keep the whole system accountable.
- ▼
Time to detect / resolve
Time from a regression starting to detection, and to resolution.
- ▲
Audit-trail coverage
Share of consequential actions with a complete, reviewable record.