Conflict Lens — document extraction prototype
Reports to Muhammad Asim
Re: Allexive call prep (CRM contact 628)
muhammadasim@codeaza.org · codeaza.com
Demo: Tue 17 Nov, 7:00am PKT
Build a working, public example of how Codeaza tests document extraction when two pages give different answers for the same field. It must be live on a URL by Monday evening.
The prospect, Lukin Ackroyd (Principal AI Architect, Allexive), asked us exactly this: “How do you test document extraction, especially when two pages give different answers for the same field? Is there a public example we could talk through?” He cares about evidence over demos, human correction time and cost, and turning corrections into reviewed rules. This prototype answers his question in a form he can click through himself.
Success = Asim can show it in 3 minutes on the call and send Lukin the link afterwards.
Synthetic dataset + golden answers
Script-generate about 8 fictional NZ home/motor policy PDFs, 3–5 pages each: schedule, terms, and an endorsement dated later.
- Sum insured: schedule $650,000 vs later endorsement $720,000 → endorsement wins
- Excess: $500 on schedule vs $750 in terms → genuinely ambiguous, must be flagged
- “Jane M. Smith” vs “Jane Smith” → same person, not a conflict
- 03/04/2026 is 3 April (NZ format), never 4 March
- 2 clean documents with no conflicts as a control group
golden.json: expected value or"FLAG"per document and field
Fields (8): policy number, insured name, start date, end date, sum insured, excess, premium, address.
Extraction with provenance
Claude (claude-sonnet-5-5) extracts every candidate value for each field across all pages, each with its page number and exact quoted text. Use Claude's PDF citations, or a verbatim quote matched against the PDF text layer.
- Precedence rules live in
rules.yamland are applied in code, not hidden in the prompt - Default rule: sources disagree and no rule applies → FLAG for human review
- Cache results so the live demo never waits on the API
Conflict view, rule loop, scoreboard, deploy
- Conflict view: PDF on the left, fields on the right. Clicking a field jumps to the page and highlights the source text. Conflicts get a red badge and show both sources side by side with how they were resolved
- Correction → rule loop: on a flagged field, pick the right value and type a one-line rule. It saves to
rules.yaml, all docs re-run, and the scoreboard updates - Scoreboard: correct / wrong / flagged against
golden.json, conflict cases vs clean cases, cost and time per document, before/after when a rule is added - Deploy to a public URL, write the README, freeze at 6pm PKT
claude-haiku-5-5) cost/accuracy comparison, and a JSON export of each run.
react-pdf / pdf.js for viewing and highlightsOut of scope: auth, a database, real customer documents, OCR of scanned images, multi-tenancy, design polish beyond clean shadcn defaults.
golden.json, with zero wrong values on the planted ambiguous casesDeadline. Code freeze Monday 16 November, 6pm PKT. Anything unfinished by then is cut, not rushed in overnight.
Check-ins. A short update to Asim at the end of each day: what works, what's at risk, any cut you'd suggest.
Ownership. The repo sits under the Codeaza GitHub org. It is public, so treat every commit as client-facing.
Confidentiality. Nothing from client work goes in: no client names, documents, prompts or code. All data is fictional.
Cost. Keep API spend small. Cache every extraction and re-run the full set only when rules change.
Blockers. Raise them the same day. If the deadline is at risk, drop the nice-to-haves first, then polish, never the must-have checks.