Build log · not for filing, about filing

Advice Document Classifier & Filing System

Working notes on how this got built, in the order it actually happened — what runs, what broke, what got fixed, and what's still open. Written to be read alongside the code, one build session at a time. The idea and domain spec came from a collaborator; the build, the bugs, and the fixes below are mine.

Snapshot: 4 Sep 2026 Build sequence: step 3 of 7 Public on GitHub — github.com/D3lK1ch1/FinancialAdviceProjectFiling

10sample PDFs, hand-labelled

4doc types in scope

40tests across 5 files

9issues logged as roadmap, not left unwritten

What this is

A tool for Australian financial-advice firms. It reads client documents (PDF/Word), works out what each one is :Statement of Advice, Record of Advice, Financial Services Guide, Product Disclosure Statement and proposes where to file it and what to rename it, and flags anything it can't confidently place for a human to look at.

It classifies documents that already exist. It does not draft, suggest, or generate financial advice.

The knowledge base is the source of truth. Document rules live in knowledge_base.json, never hardcoded in Python.
Everything is a proposal. Nothing files itself. Low confidence routes to _Needs review, visibly.
Storage stays abstracted. Local folder today, a cloud bucket later — the classifier shouldn't know or care which.
Display mode, filing mode, and the classification engine stay independent — none of the three may depend on the others.
ROAs have four legislative bases (further advice, hold/no-action, small investment, no buy/sell) — an unresolved ROA is a "confirm which kind" flag, not an error.

How it's built

One unit at a time, each one run before the next starts. A unit is a function, a test file, a documented decision — small enough to hold in one head, evaluated by running it or explaining it back, not by trusting a "looks good." Commits land at the end of an evaluated unit, not at the end of a session, so a bad file never has to drag four good ones down with it when something needs undoing.

That's why the timeline below reads the way it does: bug found, fix made, verified against real samples, then move on — rather than a block of "features added" with the reasoning stripped out.

Timeline

20–21 Aug

Knowledge base & sample set

knowledge_base.json (v0.3) — document types, classifier hints, filing rules, and the 11-rule edge-case flag set — built first, as the thing everything else reads from. Ten real, verified-native-text sample PDFs sourced across SOA/ROA/FSG/PDS, ground-truthed in sample_documents.labelling.md. harness.py written as a first, keyword-matching prototype of the full pipeline.

Caught & fixed Two ROA samples were stale 2021 versions and one was mislabelled and replaced with current ASIC versions, relabelled nochange (was further_advice).

22 Aug

Live app track begins — parser + scope gate

parser.py (pypdf), app.py (FastAPI, POST /ingest), scope_gate.py (deterministic SOA/ROA/FSG/PDS title check), a drop-zone UI. First end-to-end run of a real PDF through the live track.

Bug 1 — reading order pypdf's default extraction follows content-stream order, not visual order — on FSG_AustralianSuper.pdf a footer interleaved mid-sentence and split a word. Fixed with extraction_mode="layout", verified the title phrase stayed intact across FSG/SOA/ROA/PDS before adopting it.

Bug 2 — cross-reference false positive scope_gate.py first labelled an FSG as soa, because the FSG's own text explains what an SOA is — a legitimate cross-reference, not noise. FSGs mention PDS and SOA; SOAs mention ROA — this is routine in the domain. Fixed by scoring matches by pattern length and restricting the check to the first 500 characters (the title region), not the whole document. Verified against 6 real samples.

25 Aug

Real classifier — Ollama wired in

classifier.py — replaces the scope gate's title-only guess with a real local llama3.1 call via Ollama, grounded in knowledge_base.json's classifier_hints, returning strict {doc_type, confidence, matched_signals}. Verified live against PDS_AustralianSuper.pdf at 100% confidence, correct type.

27 Aug

Status check — unit breakdown written

TO_DO_LIST.md rewritten against the architecture doc: 14 units, ordered by actual data-flow dependency. All 10 sample PDFs re-verified as selectable text directly through parser.py. The git-repo item was logged resolved here — see the next entry.

28 Aug — today

Test suite: 4 files, unit + real end-to-end

tests/test_parser.py, test_scope_gate.py, test_classifier.py (mocked, no Ollama needed), and test_e2e.py — the last one hitting live Ollama on purpose, no mocking, because that's the only test that proves the wired pipeline actually works end to end. README.md updated with the run command and a Windows PATH gotcha (python -m pytest, not bare pytest).

Discrepancy caught TO_DO_LIST.md logged the "no git repo" item as resolved on 27 Aug. It wasn't — git status still returns not a git repository. Flagged rather than trusted; see Open gaps below.

Test bug, found and fixed The out-of-scope e2e fixture used a blank pypdf page with no /Contents stream at all, which crashes layout-mode extraction (KeyError: '/Contents') — a real pypdf edge case, but not the one under test. Fixed with an explicit empty content stream instead.

Timeout fixed, not just changed Live-Ollama runs intermittently timed out at 60s — not cold-start only, failure count varied run to run (1/4, 3/4, 3/4). Bumped classifier.py's requests.post timeout to 180s, then re-ran the full e2e suite to confirm rather than assuming: 5/5 passed, ~2m37s, no timeouts.

Failure log built failure_log.py — log_failure() appends JSON Lines to failure_log.jsonl, field names matching 2_ARCHITECTURE.md's SQLite failure_log table so it drops in later without a schema change. No review UI exists yet to capture human corrections, so tests/test_e2e.py is the current caller — it already has a real prediction and a hand-labelled correct answer for every sample. Unit-tested separately in test_failure_log.py.

Frontend redesigned static/index.html restyled around the PRD's own review vocabulary: confidence as a tiered meter (high / medium / needs review), matched_signals surfaced as an explicit "Why" list, "Proposed classification — confirm before filing" stated up front rather than implied. app.py now resolves doc types to their full names via knowledge_base.json instead of the frontend hardcoding them. Verified live in-browser against a real sample, not just visually reviewed.

Track decision made Live app track has explicit priority. harness.py's fate — kept as an offline validator, or replaced — is deliberately deferred until the live track's results make that call obvious.

31 Aug

Repo goes public — history and backlog on the record

git init, CHANGELOG.md added, README.md filled in with run instructions and a dependency table, first two commits pushed to GitHub. The "no git repo yet" gap logged below on 28 Aug is now closed for real — checked against a fresh git status, not assumed.

Caught during this refresh .gitignore had a bare docs line meant to keep client-filed output out of git, but it also matched this planning folder — docs/BUILD_LOG.html itself included. It was never actually on GitHub despite two pushes. Fixed by scoping the ignore to samples/ only, which is what the "classified documents" comment was actually about.

4 Sep

Nine issues opened — the honest backlog, not a to-do buried in a file

Rather than let known gaps sit as prose in TO_DO_LIST.md only, they're now filed as scoped GitHub issues — security hardening, correctness bugs, format support, persistence sequencing, and CI. See Roadmap below for the grouping.

Two build tracks, one now prioritised

Two parallel implementations of the same pipeline exist. As of 28 Aug, the live app track has explicit priority — unit work happens there. harness.py's fate is a deliberately deferred decision, not an accident: it gets made once the live track's results make it obvious, not before.

Live app track

app.py · parser.py · scope_gate.py · classifier.py

Single-document, real FastAPI + drop-zone UI
Parser → scope gate → live Ollama classifier, all wired
Runs against real samples/ PDFs
No filing, no flagging engine, no persistence yet

Harness / validation track

harness.py · harness_demo_fixture.json

Batch, keyword-matcher prototype
Runs against synthetic fixture text, not real samples
Two known, unfixed bugs: KB-array-order tie-breaking; string-sorted (not date-parsed) event grouping
11-rule edge_case_flags engine exists in the KB but is read nowhere in code

Test suite added this session

Zero tests existed before this session. Four files now cover the four live-track modules, deliberately split by how much of the real world each one needs to talk to.

File	Covers	Needs	Result
test_parser.py	All 10 real samples parse with selectable text; response shape	Nothing external	11 / 11
test_scope_gate.py	All 10 samples classify by filename-prefix type; FSG/SOA regression pinned; out-of-scope text case	Nothing external	12 / 12
test_classifier.py	JSON parsing (plain / fenced / malformed); success, defaulting, and all three exception paths	`requests.post` mocked	9 / 9
test_e2e.py	Full `POST /ingest`, one real sample per doc type, plus a text-less PDF; logs a failure-log entry on any misclassification	Live Ollama, `llama3.1`	5 / 5
test_failure_log.py	`log_failure()` appends correctly-shaped JSON Lines; handles a `None` prediction; appends without overwriting	Nothing external	3 / 3

Run everything: python -m pytest from the project root. Details in README.md.

Where the project actually stands

Fourteen units, ordered by data-flow dependency rather than diagram order, each sized for one build-and-stop cycle. This is the honest version — most of it is still ahead.

#	Unit	Status
1	Parse PDF → text	Done
2	Title-pattern pre-filter (scope gate)	Done
3	LLM classify — 4 in-scope types	Done, live-tested
4	Flag: low-confidence + unknown-type	Spec'd, not built
5	Date extraction	Not built
6	Client identity (manual, no DB)	Not built
7	Advice-event identity	Blocked on 5, 6
8	Filing: naming + path proposal	Blocked on 4, 7
9	Persistence: documents / classifications / flags	Not built
10	Persistence: clients / advice-events	Extends 9
11	Real `_Needs review` queue	Blocked on 9
12	Approve / Correct / Reflag + failure log	Log mechanism done, UI blocked on 11
13	Remaining 9 flagging rules	Blocked on 10
14	Missing-record detection	Deferred (Phase 2)

Next unit is #4 — the only one with zero blockers left.

Open gaps, on the record

No requirements.txt — tracked as issue #3.

What's actually installed and in use is documented in README.md's dependency table for now.

Roadmap — nine issues, grouped

Filed on GitHub rather than left as prose, so the backlog is scoped, linkable, and closeable one at a time — the same one-unit discipline as the build itself, applied to planning.

#	Theme	What it covers
#1	Self-review	README-only PR documenting security, hygiene, and readiness findings before touching code — one branch, one reviewable change
#2	Security	Auth on `/ingest`, upload size limits, magic-byte validation, gating raw extracted text, documenting the local-Ollama privacy tradeoff
#3	File hygiene	`requirements.txt`, tighter `.gitignore`, de-identification rule for `failure_log.jsonl`, harness-fate decision recorded in `CHANGELOG.md`
#4	Correctness	Case-insensitive scope gate, position-based tie-break, verifying matched signals actually appear in text, per-type review thresholds, wiring reviewer corrections into `log_failure()`
#5	Format support	OCR fallback, `.docx` handling, a real `parse_document()` dispatcher, confidence capping on OCR'd text
#6	Client identity	Multi-field identity resolution (name, DOB, address, account), scored candidates over silent picks, `ambiguous_client` flag, family-key grouping without merging records
#7	Reproducibility	Pinned Ollama model digest (not a floating tag), `engine_version` stamped on every classification, config moved to env vars, a single `kb.py` loader
#8	Statefulness	Single-document flags before cross-document ones, day-first date parsing for Australian dates, SQLite persistence sequenced deliberately behind that
#9	CI	Tiered test suite (fixture-only vs full), LLM test marked non-deterministic and opt-in, GitHub Actions on PRs, a mechanical check rejecting any PR that adds PDFs or docs outside the policy

Reading this together

This file is meant to be walked through, not just skimmed because the goal is that whoever reads it with me can point at a decision here, ask "why," and get an answer that isn't "the AI decided." Start with the timeline above in order; every entry names the bug or the reasoning, not just the file that changed.

README.mdHow to actually run it : server, Ollama, tests.

CLAUDE.mdGround rules this build doesn't break.

docs/Spec and architecture this build follows.

tests/The suite this log describes — run it, don't take the numbers on trust.

GitHub IssuesThe nine-item roadmap above, each one closeable on its own.