Feedback Loop
Shipped is not done. The value comes from the loop after launch.
Procedure
The Loop
Instrument → Measure → Review → Adjust → Repeat (weekly)Every step is required. Skipping any one breaks the loop.
1 — Instrument
Before launch, decide what to measure. Three layers:
| Layer | Tells you | Examples |
|---|---|---|
| Business | Did the change move the needle? | Activation, retention, revenue |
| Product | Did users do the thing? | Button clicks, feature adoption |
| System | Did the system work? | Errors, latency, uptime |
For a new feature:
- 1 business metric (the one that justifies the feature)
- 2-3 product metrics (the funnel steps)
- 3-5 system metrics (so you know it works)
Add tracking BEFORE launch. Not after.
2 — Measure
Don't stare at a dashboard. Measure by collecting the number once.
Two formats:
- Daily metric sheet (automated): email/Slack at 9am with yesterday's key numbers
- Weekly review doc (one page): trends and anomalies
If you can't name the number you expect, you didn't define success.
3 — Review Weekly
30 minutes. Same time every week. Same template:
Week of [date]
[Big metric]: X (vs. target Y, vs. last week Z)
- Why did it move?
- What surprised me?
[Secondary metric 1]: ...
[Secondary metric 2]: ...
Actions:
1. [One thing we'll do this week]
2. [One thing we'll stop doing]Don't skip weeks. Skipping = loop is dead.
Anti-patterns
- Launching without a pre-declared success metric.
- Averaging per-row ratios instead of summing.
- Pooling results across model or runtime versions.
- Reviewing only when someone remembers.
References
- details
Verification Checklist
- [ ] Business, product, and system metrics are defined before launch.
- [ ] The expected direction is stated.
- [ ] Ratio aggregation uses sums.
- [ ] Review has an owner, time, and template.
- [ ] Results are segmented by config and version.
Aggregation Rules
- A ratio is
sum(numerator) / sum(denominator). Never average per-row means. - Combine only axes whose denominator is greater than zero. One missing axis must not null the others.
- Weight feedback by feedback rows, not by all rows in the group.
- Calibrate judge scores against human feedback per skill; report the bins and sample counts.
- Never pool scores across model revisions, runtime versions, or config fingerprints.