# Measure useful output and full cost

Practices: full-cost-accounting, price-token-categories, matched-comparison, scoped-findings.

Choose one cohort and observation period. Define acceptance and an independent deliverable before counting. Store PRs, tasks and deliverables as separate identifiers; a task can split into multiple PRs.

Record all attempts, stops, reviews, repairs and shared coordination. For every model use the actual identity, fresh input, output, cache reads and cache-write TTL. Retain modifiers and pricing date. Deduplicate streamed usage by message ID using the maximum available counter per category. Do not add cached input twice.

Cash cost includes actual charges and explicitly attributed subscription/tools/infrastructure. API-equivalent value is the selected token mix priced under a dated schedule. Quota consumption is a meter observation. Never rename one of these as another.

Record active human minutes across preparation, questions, review, debugging and integration. Concurrent tasks must not double-count the same human interval. Agent execution hours and wall time are separate fields.

Calculate accepted work/human hour only when its denominator is measured. Compare unit costs and rates only with compatible work mix, acceptance, price basis and scope. A zero or absent baseline supports an absolute result, not a multiplier.

Report repair burden and attributable defects over a fixed follow-up window. Show immature cohorts separately. Use [the worksheet](https://hamedtaheri.com/playbook/v0.2.0/templates/run-ledger.csv) and [local tools](https://hamedtaheri.com/playbook/v0.2.0/tools/README.md); their calculations do not make an unsupported causal claim valid.

## Cost per accepted task

Read [the metric definitions and benchmark bridge](https://hamedtaheri.com/playbook/v0.2.0/reference/cost-per-task.md).

Before execution, define task boundaries, parent/child relationships, work class, acceptance checks and retry limits. Pick one counting level. Join task IDs to run IDs, PRs and verification evidence. Deduplicate acceptance by task ID; never count accepted ledger rows as tasks. Retain all failed, stopped and repair attempts in the numerator.

Report cost per attempt separately from all-cohort cost per distinct accepted task. With repeated trials, distinguish successful trials from distinct solved tasks. Divide by the actual outcome count, not the accounting-row count. An unknown denominator stays unknown; zero successes give an undefined unit cost. Disclose missing costs and exclusions rather than silently treating them as zero.

For shared coordinator cost, keep a cohort total and disclose any per-task allocation rule. For subscription cash, record paid period, seat count, outside-cohort work and allocation basis. Human minutes and labor rates need their own evidence. Never multiply the detailed cohort's unit cost by a broader account-wide PR count.

For comparisons, hold starting state, task specification, acceptance tests, retry policy and price basis constant. Report work classes, acceptance rate, first-attempt success, repair/revert burden and human minutes. Use a public benchmark's tasks and grading protocol before claiming a benchmark cost advantage.

## Current results and the 20× target

Read [the metric availability audit](https://hamedtaheri.com/playbook/v0.2.0/reference/delivery-metrics.html) and [aggregates](https://hamedtaheri.com/playbook/v0.2.0/evidence/delivery-metrics.json). The observation window is October 1–19, 2026, Vancouver. Current counts are 275 merged task units and 185 parent tasks with all recorded parts merged. The 770 weekly merged PRs include 462 internal task-branch merges; the approximately 188× comparison to DX is raw PR volume only.

Keep review/test flags distinct from confirmed defects. First logged activity to merge includes waiting and is not human work time or production lead time. Absence of a recorded repair worker is not first-pass acceptance or zero intervention. Existing dated unattended-run accounts are not a measured developer-away completion rate. Preserve unknown acceptance, human-time and memory-causal metrics as null.

When the workflow changes how work is packaged, PR and total-commit counts change meaning. One PR per task, repair PRs into task branches and integration merges multiply both without multiplying the work. Compare periods across such a change with non-merge commits, code and test lines changed (generated files, lockfiles and data excluded) and new design documents, per calendar day, and report merge-commit share and non-merge commits per merged PR beside them to show the packaging shift. Rerun `scripts/playbook-activity.py` from the source repository to refresh [the monthly series](https://hamedtaheri.com/playbook/v0.2.0/evidence/activity-history.json). Call the result an activity ratio, never a productivity multiplier; lines describe code footprint, not human effort.

For an external PR comparison such as DX's 4.1 merged PRs a week, compute merged PRs per calendar day × 7 / the benchmark. After a packaging change, also report the last comparable month's multiple scaled by the rise in non-merge commits per day, and show the as-counted value beside it. DX publishes no PR size, so call every such value a PR-volume comparison with a dated cohort. Report only observed periods; never present a projection as a measurement.

October 2026 is the complete benchmark month. Its 2,155 merged PRs read 118.7× as counted; scaled from September's multiple, 16.0× by non-merge commits per day and 16.1× by code lines per day. Report 16× with the as-counted value beside it, and never call either productivity or the 20× result. Per calendar day, October's non-merge commits and code lines ran about 3.0× the April–September average; call that activity. Task, flow and cost aggregates for the month are in [october-2026.json](https://hamedtaheri.com/playbook/v0.2.0/evidence/october-2026.json) with their definitions: finished tasks count epics apart; lead time runs from filing to done; first-attempt acceptance covers only tasks with ledger rows. Keep the two cost measures apart: ledger cost per finished task covers recorded pipeline runs, while all API-equivalent compute divided by finished tasks includes interactive sessions and subagents. Decision wait in that file is a snapshot taken at collection, not an October measure.

Explain 20× as comparable accepted parent tasks per developer-week against an explicit baseline, with all participating human effort, work mix, quality and full cost recorded. Do not derive it by discounting a larger PR ratio. Prepare any additional instrumentation as a proposal for the human; do not change operational records or start paid runs without authorization.

## Explain the result before the diagnostics

Lead with three questions: what work got through, what evidence makes it trustworthy, and what it cost. Keep requested outcomes, implementation units, PRs and human-effort denominators distinct. The current 275 merged units include 213 classified as reviewed with passing tests (146 deep, 67 standard); the other 62 have different evidence categories, including 14 audit flags. Do not call the full cohort fully tested or accepted.

For each requested outcome, link definition, implementation revision, behavioral verification, resolved review findings, integration checks and independent acceptance. Report the last evidenced step. For bugs, retain reproduction and regression evidence; for feature slices, demonstrate the acceptance criteria. Classify work size and category before execution; never infer equivalent human hours from lines changed. Team memory connects the records, while tests and acceptance establish the result.