Reference · (after pdf2json UAT regression)
Status Overview
The parser strategy, known issues, and feature-testing status as they stood mid-saga — the multi-bank detour's high-water mark before the rollback.
Why this exists
Like all my projects, it started out as a personal problem wanting to know how much I spend per month dowwn to the last detail. I wanted to know what I actually spend without scrolling through the bank app date by date, hunting for merchant and price that overwhelmed me because I would lose any pattern and gain nothing. More than once during the build someone asked "doesn't the bank app already do that?" and until late in the build, the honest answer was no until I saw 'Spend Summary' in my own bank app.
Not wanting the project to go to waste, it still bothered me and thus it just proved the real gap was finer than a monthly total. By-day and by-week detail, down to the transaction, was still missing — which is what the Calendar View (Jul 09) ended up solving.
The multi-bank arc
Nine sessions, one attempt at ANZ/CBA/NAB support, one regression, one rollback, one pivot. Click a node to jump to that session.
Parser strategy
| Path | Library | Used in API | Real PDF UAT |
|---|---|---|---|
| Text | pdf-parse → parseTransactions(text) | No (removed) | Was poor for CBA (0 txs) |
| Layout | pdf2json → parseTransactionsFromPdf | Yes | Regressed CBA/NAB May 2026 |
| Probe | probe-pdf2json.ts | No (reference only) | Used to inspect x/y columns |
Known issues & fixes
| Issue | Root cause | Solution | Status |
|---|---|---|---|
| PDF parser returns incorrect amounts | Regex / column guess on flat text | pdf2json x/y columns | Partial |
| Missing transaction filtering | Page headers, totals | filterSummaryRows() | Done |
| Categorization | Keyword matching | getCategoryFromDescription | Done |
| Multiple bank formats | Different headers/columns | Structural header + pdf2json | In progress |
| CBA real PDF | Mixed dates, split header, multi-row | Block merger attempted | Failed UAT May 2026 |
| NAB real PDF | Dots, multi-row, orphan row drop | Block merger attempted | Failed UAT May 2026 |
| ANZ official multi-row | Merchant + EFTPOS on separate y | mergeDescriptionLines | Not re-UAT'd |
| Tests vs reality | Synthetic fixtures only | Need probe-derived fixtures | Not started |
| No API fallback | pdf2json-only route | pdf-parse fallback when 0 txs | Recommended |
Feature testing status
| Feature | Unit | Integration | UAT | Notes |
|---|---|---|---|---|
| File upload | Yes | Yes | Yes | PDF only |
| PDF parsing (pdf2json) | Yes (synthetic rows) | Partial | Failed CBA/NAB | See 2026-05-19 |
| PDF parsing (text fallback) | Yes | N/A | Not wired | Still in codebase |
| Transaction parsing (ANZ) | Yes | Yes | Unknown | Re-test after block merger |
| Transaction parsing (CBA/NAB) | Yes (fictional) | No | Failed | — |
| Categorization | Yes | Yes | Partial | misc if description truncated |
| Monthly grouping | Yes | Yes | Partial | — |
| Database persistence | No | No | No | Blocked |
Session log (multi-bank)
| Date | Session | Outcome |
|---|---|---|
| Apr 29 | Multi-bank scaffold | Column map wired |
| May 02 | CBA/NAB bug investigation | 3 text-parser bugs identified |
| May 04–05 | Text parser fixes | multiBank 17/17 synthetic |
| May 18 | blank→0.00, plural headers | Tests green |
| May 19 | pdf2json API + block merger | UAT worse for CBA/NAB |
- pdf2json UAT fixes — orphan rows, opening balance, CBA header, API fallback
- Real PDF fixtures from probe output (not hand-written ideal rows)
- Database integration — schema not finalized
- Export reports — not implemented
Next session priorities (as of May 19)
- pdf-parse fallback in
parse-data/route.ts - Fix orphan-row drop + opening-balance block isolation in
pdf2json-extractor.ts - Probe snapshot → test fixture for one CBA page + one NAB page
- Re-UAT all four PDFs; update Notes.md probe table
Reference ·
Architecture & Parser Notes
Standing reference: file map, the parsing pipeline, category rules, and the running log of test-fixing and diagnosis sessions folded into this file over time.
Architecture
Frontend
Backend
Core problem: PDF bank statement parsing
Pipeline
extractor.ts — ANZ assumptions
| Component | Code | ANZ assumption |
|---|---|---|
| Date regex | /^\d{1,2}\s+[A-Z]{3}/i | Expects DD MMM like "15 JAN" |
| Amount columns | numericTokens.slice(-3) | Last 3 values = withdrawal, deposit, balance |
| Description | substring before amounts | Text between date and amounts |
| Amount detection | checks withdrawal OR deposit | Which column has a number |
detector.ts — format detection
Auto-detects between two formats: column (headers contain Balance + Withdrawal + Deposit) and line (single $ amount with keyword-based type detection).
Why other banks won't work as-is
| Bank | Required changes |
|---|---|
| Westpac | Date format DD/MM/YY — regex needs update |
| CommBank | Description first, amounts at different positions |
| NAB | Different column order entirely |
| International | Different date formats (MM/DD/YY), currency symbols in different spots |
Root issue: PDF text extraction gives flat text — no column information. The parser assumes date always at start, description in the middle, and 3 amount columns at the end.
Categories (lib/categories.ts)
Single source of truth for CATEGORY_KEYWORDS, CATEGORY_COLORS, and getCategoryFromDescription().
- friends — mobile banking payment, transfer to/from, pay anyone, osko, pay id
- food — restaurant, cafe, mcdonalds, hungry jacks, kfc, snack bar, pizza, burger, sushi, noodle, bakery, coffee…
- utilities — vodafone, telstra, optus, energy, electricity, water, gas, internet
The AI can't read PDFs directly, so the workflow is console-logging past the transaction header to figure out format: multi-column table vs. single-line per transaction. Reminder: find other bank files to see whether the same formatting can be reused.
Transaction improvements (ideas, not yet built)
- Categorization without hardcoding — richer rule-set, config-driven rules (JSON/TS) allowing multiple match kinds per category
- A per-user database that expands over time, storing user-specific overrides — hardcoded strings that become data over time
- An ML model, possibly trained on budgeting/transaction datasets (Kaggle), for merchant/category classification
- Exact substring keywords vs. regex patterns vs. metadata per category
- Deterministic evaluation order with explicit bank/merchant-code prefixes
- Per-user or per-project override layer: overrides → global rules → fallback
- Identify common ANZ description structures (
EFTPOS WITHDRAWAL <MERCHANT> <SUBURB>) and encode as generic patterns — e.g. "EFTPOS" + food → food, "OSKO"/"PAY ID" → friends/transfer - Track frequency of "misc" descriptions by normalized merchant token
Testing (ATTD)
Test fixing session — March 20, 2026
Error: Failed to resolve import "@testing-library/jest-dom" from "tests/setup.ts"
Investigation: checked package.json (malformed entry, line 30) → checked tests/setup.ts (unnecessary jest-dom import) → removed both → 22 failures remained → analyzed each failure category systematically.
Issue 1 — malformed jest-dom package entry
// BEFORE (broken)
"@testing-library/jest-dom": "file:testing-library/user-event@^14.5.0",
// AFTER — removed, not needed
package.json, tests/setup.ts
Issue 2 — missing "woolies" keyword
// lib/transactionParser.ts line 215
groceries: ['coles', 'woolworths', 'woolies', 'iga', 'supermarket'],
Why: .includes() does substring matching — "woolies" doesn't match "woolworths".
Issue 3 — column format parsing
# 3-column format (explicit deposit column)
2 Jan CARDLESS CREDIT 345.67 0.00 123456.78
# 2-column format (deposit implied zero)
2 Jan CARDLESS CREDIT 345.67 123456.78
Solution — rewrite extractTransaction(): regex /(\d[\d,]*(?:\.\d{1,2})?)/g to find all amounts, filter out the date's day number, take the last 3 amounts as withdrawal/deposit/balance, use 0.00 placeholders for missing columns in fixtures.
Issue 4 — label accessibility in FileUploader
// BEFORE
<label {...getRootProps()}>
// AFTER
<label {...getRootProps()} htmlFor="dropzone-file">
Why: tests use getByLabelText(), which requires programmatic association.
Issue 5 — header detection for line format
// Added to headerPatterns
/date.*transaction.*description.*amount/i,
/date.*transaction.*amount/i,
Why: Transaction Reports have "Amount" instead of "Withdrawal/Deposit".
Test data format requirements
The parser requires the 3-column format for column-based statements — always include 0.00 placeholders; the parser uses the last 3 numeric values as amounts; the date's day number (e.g. "2" in "2 Jan") is filtered out.
Date Transaction Detail Withdrawal Deposit Balance
2 Jan CARDLESS CREDIT 345.67 0.00 123456.78
12 Jan Salary Deposit 0.00 3500.00 126786.78
Final test results (March 20)
| Test suite | Passed | Failed | Total |
|---|---|---|---|
| Categorization | 92 | 0 | 92 |
| Transaction Parser | 23 | 0 | 23 |
| GroupByMonth | 9 | 1 | 10 |
| Summary | ~4 | ~8 | 14 |
| FileUploader | ~9 | 2 | 11 |
| Total | 140 | 10 | 150 |
- Understand the test data format before debugging the parser — check what the parser expects, check what the test data provides, align them, don't force the parser to handle bad data.
- Use
toBeCloseTo()for float comparisons, nottoBe(). - Mock files need a proper
sizeproperty:Object.defineProperty(file, 'size', {'{'} value: 1000, configurable: true {'}'}). - react-dropzone testing needs
userEvent, mocked callbacks, or testing behavior without triggering the dropzone directly.
Setup & test commands (previous testing content)
npm install -D vitest @testing-library/react@^16.0.0 @testing-library/jest-dom @testing-library/user-event@^14.5.0 jsdom
npm run test # watch mode
npm run test:run # once (headless)
npm run test:coverage # with coverage report
Note: @testing-library/react@^16 required for React 19; user-event latest stable is v14.
Test structure
Test fixing session — March 24, 2026
18 tests failing across 3 files: summary.test.ts (8), groupByMonth.test.ts (8), FileUploader.test.tsx (2).
Issue 1 — hardcoded year in tests vs runtime year in code
// lib/parser/group.ts:24
const key = `${'$'}{new Date().getFullYear()}-${'$'}{monthNum.toString().padStart(2, '0')}`;
Tests used hardcoded 2025, but the run date was 2026-03-24. Fix: use new Date().getFullYear() in tests instead of hardcoding.
Issue 2 — test data format didn't match parser requirements
// lib/parser/extractor.ts:126-140
if (numericTokens.length >= 3) {
withdrawalAmt = parseNum(numericTokens[numericTokens.length - 3]);
depositAmt = parseNum(numericTokens[numericTokens.length - 2]);
balanceAmt = parseNum(numericTokens[numericTokens.length - 1]);
}
Parser expects exactly 3 numeric values for column format. Fixture rows were missing the 0.00 placeholder for whichever column was empty.
Issue 3 — test expectations didn't match test data
Example: 2 debit transactions totalling 176.23, but the assertion expected deposits === 3500.00 — leftover from a different fixture. Fixed by verifying transaction structure rather than mismatched totals.
Issue 4 — react-dropzone integration with testing-library
| Approach | Result | Notes |
|---|---|---|
fireEvent.change(input, {'{'} files {'}'}) | Failed | Doesn't trigger dropzone |
userEvent.upload(input, file) | Partial | Works for single files, not validation |
| Mock react-dropzone hook | Complex | Requires act() wrapping |
Wrap in act() | Warning | "not configured to support act" |
| Skip untestable tests | Pass | Practical solution — used |
Before vs after (March 24)
| Metric | Before | After |
|---|---|---|
| Total tests | 150 | 149 |
| Passed | 132 | 148 |
| Failed | 18 | 0 |
| Skipped | 0 | 1 |
- Don't hardcode years in tests — use
new Date().getFullYear(). - Match test data to parser assumptions before debugging either side.
- Test data must actually produce the values the assertions expect — verify, don't assume.
- Some component libraries (react-dropzone) don't test well in jsdom — prefer unit tests for logic, E2E for integration, skip flaky tests rather than leaving them broken.
- Use descriptive test names ("should calculate deposits correctly for mixed transactions", not "test case 1").
When to seek help vs. debug further
| Situation | Recommendation |
|---|---|
| Clear assertion error | Debug — compare actual vs expected |
| Test times out repeatedly | Check jsdom support for the library |
| Same test fails everywhere | Likely a code bug, not a test issue |
| Fails only in CI/local | Environment-specific issue |
| 3+ attempts still failing | Consider skipping and documenting |
Regex analysis: extractMerchant failures — April 25, 2026
| Test | Input | Expected | Actual |
|---|---|---|---|
| 1 | EFTPOS AUSTRALIA XIN DONG BEI PT\CLAYTON VIC AU | AUSTRALIA XIN DONG BEI | AUSTRALIA |
| 2 | VISA DEBIT PURCHASE CARD 1127 SMOLBITEZ PTY LTD MELBOURNE | SMOLBITEZ | SMOLBITEZ MELBOURNE |
Root cause — ADDRESS_SUFFIX regex too greedy (lib/categories.ts:30):
const ADDRESS_SUFFIX = /\s+(STREET|ST|ROAD|RD|AVENUE|AVE|DRIVE|DR|LANE|LN|COURT|CT)?\s*[A-Z]+(\s+[A-Z]+)*\s*(VIC|NSW|QLD|SA|WA|TAS|ACT|NT)?\s*(AU AUS)\s*$/i;
Trace for test 1: ANZ_PREFIXES strips EFTPOS → BUSINESS_NOISE strips PT\CLAYTON → ADDRESS_SUFFIX's unbounded (\s+[A-Z]+)* greedily consumes XIN DONG BEI VIC AU, leaving only AUSTRALIA. The pattern is meant for real addresses like "123 SMITH STREET MELBOURNE VIC AU" but strips any words between the start and the state code, including merchant name parts.
| Option | Description | Tradeoff |
|---|---|---|
| 1 | Narrow ADDRESS_SUFFIX to street type + single state | Can't handle multi-word suburbs |
| 2 | Two-pass: split cleaning into controlled stages | More code, more explicit |
| 3 | Accept current behavior, adjust test expectations | May mis-categorize future merchants |
Multi-bank PDF parsing diagnosis — May 12, 2026
Run via npx tsx probe-pdfs.ts — runs all PDFs through the parser, prints count + first 5 transactions. Key finding: sampleCBAFormatText and sampleNABFormatText in fixtures are fictional clean formats — they pass tests but don't reflect what pdf-parse actually outputs from real PDFs.
Bug 1 — plural column names (ANZ, NAB)
detector.ts:42-49 — ANZ header: "Withdrawals ($) Deposits ($)"; NAB header: "Debits Credits". indexOf('withdrawal')/indexOf('deposit') return -1 against plural tokens, so the extractor falls back to positional guessing — card/reference numbers in descriptions get picked up as amounts, displacing the real value.
const withdrawalAliases = ['withdrawal', 'withdrawals', 'debit', 'debits'];
const depositAliases = ['deposit', 'deposits', 'credit', 'credits'];
Bug 2 — CBA mixed date formats → 0 transactions
detector.ts:73, extractor.ts:13 — CBA mixes 01 Jul 2021 OPENING BALANCE (has year) with regular lines like 28 Oct THE BODY SHOP... (no year). Date-format sampling hits the opening-balance line first, locks to DD MMM YYYY, and every regular line then fails the year-required regex.
Bug 3 — CBA header split across lines
detector.ts:3 — header is 4 separate lines (Date / Transaction / Debit / Credit Balance). isTransactionHeader() checks one line for date+description+amounts at once and never fires; the date column index ends up wrong.
Bug 4 — NAB dot padding in descriptions
extractor.ts:51-61 — lines like X Li .....20.00 get their dot padding concatenated straight into the description. Fix: description.replace(/\.{'{'}3,{'}'}/g, '').trim().
Bug 5 — File_000.pdf columns fused
pdf-parse renders adjacent columns with no space (DateTransaction DetailsWithdrawalsDeposits), and merges bank name into date (05 MARANZ MOBILE BANKING). A pdf-parse rendering artifact specific to this file's encoding — not easily fixable without a post-processing space-reinjection pass. Low priority.
Probe output summary
| File | Transactions | Notes |
|---|---|---|
| 2025-01-06.pdf (ANZ) | 145 | All typed credit (wrong), card numbers leak into amounts |
| 2025-07-04.pdf (ANZ) | 191 | Mixed, same column bug |
| 2026-01-06.pdf (ANZ) | 282 | Same as above |
| Australia Commonwealth J C.pdf (CBA) | 0 | Bug 2 kills all parsing |
| Australia NAB.pdf (NAB) | 14 | Dot padding, wrong column mapping |
| File_000.pdf (ANZ) | 44 | Dates show as "05 MARANZ" (fused render) |
Fix order (by impact)
detector.ts— plural aliases for withdrawal/deposit → fixes ANZ + NAB column mappingdetector.ts—detectDateFormatdoesn't lock on opening-balance year → fixes CBA 0-transaction bugextractor.ts— strip dot sequences from descriptions → cleans NAB output
Fixed
Test suite fixes
A broken package.json entry cascaded into 22 test failures. Fixed the root cause, then the parser assumptions the tests actually depended on.
Error: Failed to resolve import "@testing-library/jest-dom" from "tests/setup.ts"
Root cause
package.json line 30 had a malformed entry:
"@testing-library/jest-dom": "file:testing-library/user-event@^14.5.0",
Fixes applied
| Issue | File | Fix |
|---|---|---|
| jest-dom import | tests/setup.ts | Removed unused import |
| jest-dom package | package.json | Removed malformed entry |
| "woolies" not categorized | transactionParser.ts | Added 'woolies' to groceries keywords |
| FileUploader label a11y | FileUploader.tsx | Added htmlFor="dropzone-file" |
| Column parsing logic | transactionParser.ts | Rewrote to regex-based amount extraction |
| Header detection | transactionParser.ts | Added "Date…Amount" pattern |
| Amount threshold | transactionParser.ts | Changed to val >= 0 to handle 0.00 |
| Test fixture | sample-transactions.ts | Updated to 3-column format |
| Test tolerance | transactionParser.test.ts | toBe() → toBeCloseTo() |
Final result
Test Files: 2 failed | 3 passed (5)
Tests: 10 failed | 140 passed (150)
Remaining: Summary tests (8 — inline data doesn't match 3-column format) and FileUploader tests (2 — fireEvent doesn't trigger react-dropzone callbacks).
- Always run
npm installafter package.json changes - Test data must match parser assumptions exactly
- Float comparisons need
toBeCloseTo(), nottoBe() - react-dropzone tests need
userEventor proper mocking - Fix test data vs. changing the parser — don't over-engineer the parser for bad test data
Scaffolded
Multi-bank support scaffold
First step of the multi-bank attempt: column-position + date-format detection architected in, deliberately not bank-specific hardcoding.
What was built
- Architecture decision: column-position detection + date-format inference, not bank-specific hardcoding
DateFormatunion type added totypes.tscolumnMapanddateFormatwired throughdetector.ts→extractor.ts→index.tsTRANSACTION_REGEXconverted from a constant to a function that switches ondateFormat- Skeleton for
detectDateFormat()created but not implemented - All 159 tests pass (fallback behavior intact)
Bugs to fix next session
detectDateFormat(detector.ts:64) — pattern array declared but hardcoded to return'unknown'. Needs to sample data rows, not the header line.extractColumnFormat(extractor.ts:134) —columnMapvalues are character offsets but code uses them as token indices. Needs offset→index conversion, or nearest-token lookup.
Good: reasoned through the architecture before touching code — identified column-position vs. date-format as separate concerns without being told. Made all the changes without asking AI to write the logic.
Watch: left detectDateFormat body empty — skeleton started but not finished; the commit captured incomplete work silently. Next time: finish it, or leave an explicit // TODO.
Watch: added extra DateFormat variants beyond what was discussed — good initiative, but untested additions are future bugs.
Ways to work better
When a function is scaffolded but not implemented, leave an explicit // TODO: so the gap is visible in review. After writing a function, ask "what input would break this?" — that question on detectDateFormat would have caught the always-'unknown' return immediately.
Diagnosed
Multi-bank format bug investigation
Uploaded real CBA and NAB statements to the UI and compared parsed output to actual PDF values. Three root causes found.
Bug 1 — year leaking as withdrawal amount
lib/parser/extractor.ts — extractColumnFormat(). CBA transactions showed 2021 as the withdrawal amount with empty descriptions. amountRegex matches all digit sequences, including the year in DD MMM YYYY dates; the line-126 filter only removes the day number, not the year.
Fix needed: strip the full date (year included, if parts[2] matches /^\d{'{'}4{'}'}$/) from fullLine before running amountRegex.
Bug 2 — NAB format detected as 'unknown'
lib/parser/detector.ts — detectFormat(). NAB uses Debit/Credit headers instead of withdrawal/deposit, falling through to 'unknown'.
Fix needed: add a branch — hasDebit && hasCredit && hasBalance → 'column'.
Bug 3 — "Particulars" column not mapped to description
lib/parser/detector.ts — findTransactionHeader(). headerTokens.indexOf('description') returns -1 when the header says "Particulars".
Fix needed: alias lookup trying ['description', 'particulars', 'details', 'narrative'] in order.
Tests scaffolded
tests/unit/multiBank.test.ts — 14 tests covering CBA and NAB parsing. Fixtures added: sampleCBAFormatText, sampleNABFormatText.
9 failed | 5 passed
Fix order
- Bug 1 (year strip) — unblocks most CBA failures and description tests
- Bug 2 (detectFormat aliases) — unblocks NAB format detection
- Bug 3 (column name aliases) — unblocks NAB description extraction
Debit/credit type detection for 2-column formats (e.g. NAB salary = credit, Coles = debit) requires knowing which column the amount appeared in. A right-to-left token approach can't distinguish this without character offsets. Needs a decision next session.
Partial
Multi-bank parsing fixes (CBA/NAB)
Applied the three bugs diagnosed on May 2 — 11 of 15 new tests pass, but debit/credit typing for two-token rows is still wrong.
What was built
- Fixed summary tests (14/14) by adding a pre-formatted
textfield tosummaryTestData - Added
DD MMM YYYYdate format support for CBA statements - Added
Debit/Creditcolumn aliases indetector.ts - Added
Particularscolumn alias for NAB description mapping - Restructured
extractor.tsto check CBA/NAB format first, before ANZ fallback logic - Cleaned up duplicate dead code in
extractor.ts(275 → 202 lines)
Test status after fixes
✔ tests/unit/summary.test.ts (14 tests)
✔ tests/unit/groupByMonth.test.ts (9 tests)
✔ tests/unit/transactionParser.test.ts (23 tests)
✘ tests/unit/multiBank.test.ts (11 passed | 4 failed)
4 remaining failures
| Test | Issue |
|---|---|
| Cash Deposit should be credit 4700.00 | type = 'debit' instead of 'credit' |
| Credit Interest should be credit 3.21 | type = 'debit' instead of 'credit' |
| Salary Credit should be credit 2000.00 | type = 'debit' instead of 'credit' |
| should have both debits and credits | credits.length = 0 |
Root cause: when numericTokens.length === 2 (amount + balance, one column empty), the 2-token case still defaults to type: 'debit' despite the CBA/NAB branch now running first.
Next session plan
- Debug what
numericTokensactually contains for the CBA "Cash Deposit" line - Confirm
columnMapvalues are character positions, not token indices (they are — the code just treats them as indices) - Fix the character-position → token-index conversion in
extractColumnFormat()
Better: test each bug fix individually before moving to the next — three bugs were fixed at once, making it hard to isolate which fix broke what.
Better: add debug logging before restructuring code, not after — restructuring first made debugging harder.
Watch: the file got corrupted with duplicate code during edits — verify file state with a read after each edit.
173 passing
Structural header detection + multi-bank fixes
Replaced rigid hardcoded header regexes with structural detection. All multiBank tests green — but real-world gaps were flagged before committing.
What was built
- Replaced the 5-regex
HEADER_PATTERNSarray withisTransactionHeader(), checking 3 independent concepts: date + description + amounts - Alias lookup for columns: withdrawal/debit, deposit/credit, description/particulars/details/narrative
- Fixed
detectDateFormatto sample data lines (was sampling the header line) - Added
DD MMM YYYYto theDateFormatunion extractColumnFormatnow receivescolumnMapanddateFormatfor position-aware parsing- CBA/NAB branch correctly handles 3-token and 2-token cases
- Added
isCreditTransactionheuristic (credit, deposit, interest, salary keywords) - 15 new
multiBank.test.tstests; removed debugconsole.log
✔ 173 tests passing, 1 skipped — all multiBank tests green (previously 4 failures)
Real-world testing issues — not ready to commit yet
1. ANZ fallback always returns 'debit'
extractor.ts:178-187 — the 2-token path (amount + balance, no 0.00 placeholders) has both branches returning 'debit'. isCreditTransaction exists but isn't applied here. A real ANZ file with "Debit" in a credit transaction's description would misclassify it.
2. CBA real file — only opening balance shows in app
Unknown root cause. Needs: checking whether the real file has 0.00 placeholders or blank columns, whether detectDateFormat identifies the real date format correctly, and whether rows are silently failing to parse.
3. filter.ts doesn't filter "Opening Balance"
Pattern /^balance/i only catches lines starting with "Balance" — "Opening Balance" slips through. Fix: add /opening balance/i.
All files in a single commit: "Add multi-bank statement format support (CBA, NAB)". Skip: .claude/, agents/, sessions/, Notes.md — session artifacts.
Next session priorities
- Fix ANZ fallback — apply
isCreditTransactionto the 2-token path - Add "Opening Balance" to filter patterns
- Investigate the CBA real file — get sample lines or debug in-app
- Commit after fixes land
17/17 green
Regression test fixes + Bug 5 applied
Compile errors from language cross-contamination (a PHP/Rust arrow bled into TypeScript), a mis-transcribed test amount, and the blank→0.00 fix from the May 12 diagnosis, finally applied.
1. Diagnosed multiBank.test.ts compile errors
| # | Line | Error | Fix |
|---|---|---|---|
| 1 | 135 | () -> wrong arrow syntax (PHP/Rust bleed-in) | () => |
| 2 | 136–140 | Single-quoted multiline string — not valid JS | Backtick template literal |
| 3 | 155 | Expected 130000.00 — extra zero misread from 13,000.00 | 13000.00 |
| 4 | 157 | Missing ; after closing {'}'}) | {'}'}); |
2. Rewrote the plural-header test fixture
Single-line format never worked for the credit test: the 2-token path (13,000.00 + 15,302.99) falls back to the isCreditTransaction keyword heuristic, and "TRANSFER FROM" doesn't contain credit/deposit/interest/salary. Switched to multiline format with blank markers (as pdf-parse actually outputs), so the blank→0.00 conversion produces 3 tokens and column position determines type correctly.
3. Applied Bug 5 — blank marker → 0.00
extractor.ts:65-67 — blank lines were silently skipped before; now a blank line appends ' 0.00', forming a 3-token structure the CBA/NAB branch can read.
// BEFORE
if (!BLANK_LINE_REGEX.test(lines[nextIndex])) {
fullLine += ' ' + lines[nextIndex];
}
// AFTER
if (BLANK_LINE_REGEX.test(lines[nextIndex])) {
fullLine += ' 0.00';
} else {
fullLine += ' ' + lines[nextIndex];
}
✔ tests/unit/multiBank.test.ts — 17/17 passing
Cross-check against plan and prior sessions
Phase 1 status
| Fix | Status | Notes |
|---|---|---|
| Fix 1 — plural aliases | Done | detector.ts — from May 05 |
| Fix 2 — decimal-required amountRegex | Done | extractor.ts:138 — from May 05 |
| Fix 3 — detectDateFormat skips opening balance | Typo | detector.ts:74: "opening blance" (missing 'a') — passes tests since no fixture has that line, but breaks on real CBA PDFs |
| Fix 4 — page break stop regexes | Done | extractor.ts — from May 05 |
| Bug 5 — blank → 0.00 | Done | extractor.ts — this session |
From May 05 next-session priorities
| Item | Status |
|---|---|
| Fix ANZ 2-token fallback (isCreditTransaction) | Done — line 190 |
| Add "Opening Balance" to filter patterns | Already done — filter.ts:7 |
| Investigate CBA real file | Not done |
| Commit after fixes land | Not done |
Pre-existing, unrelated: groupByMonth.test.ts has 2 failures not caused by this session's changes.
What's left before this feature is done
Immediate: fix the "opening blance" typo at detector.ts:74; commit Phase 1 + Bug 5; investigate the 2 pre-existing groupByMonth failures.
Medium term: investigate the CBA real file (only opening balance shows — likely the same typo/date-format lock); begin Phase 2, pdf2json-extractor.ts using x/y coordinates — probe-pdf2json.ts had already produced column x-positions for all banks.
The single-quote vs. backtick distinction mirrors Python's single-quoted string limitation, except Python multiline uses triple-quotes. In JS/TS, the moment a string needs newlines, reach for backticks. The -> arrow is a language cross-contamination sign — worth a personal cheat-sheet entry.
Regressed
pdf2json UAT regression + multi-bank feasibility notes
The API switched from flat-text pdf-parse to x/y-aware pdf2json, plus a multi-row block merger — and real CBA/NAB uploads got worse, even with 189+ unit tests green. The turning point of the saga.
Executive summary
Unit tests still pass because fixtures are small, hand-built row layouts — not copies of real pdf2json output. Multi-bank parsing is feasible, but only if each bank is validated against real PDF geometry (probe snapshots → committed fixtures → fixes). Synthetic tests alone keep producing false confidence.
Architecture timeline
flowchart TB
subgraph phase1 [Phase 1 — Text parser]
pp[pdf-parse flat text]
det[detector.ts]
ext[extractor.ts merges continuation lines]
pp --> det --> ext
end
subgraph phase2 [Phase 2 — pdf2json production]
p2j[pdf2json x/y fragments]
p2e[pdf2json-extractor.ts]
api[parse-data/route.ts]
p2j --> p2e --> api
end
subgraph phase2b [Phase 2b — Block merger UAT fix]
blk[groupRowsIntoBlocks]
merge[mergeBlockColumns]
p2e --> blk --> merge
end
phase1 -->|replaced in API| phase2
phase2 --> phase2b
| When | Change | Files |
|---|---|---|
| Apr 29 | Column map + date format scaffold | detector.ts, extractor.ts, types.ts |
| May 02–05 | CBA/NAB text fixes, multiBank.test.ts | detector.ts, extractor.ts, fixtures |
| May 18 | Plural headers, blank→0.00, 17/17 multiBank | extractor.ts, detector.ts |
| May 19 (earlier) | API uses pdf2json only | pdf2json-extractor.ts, route.ts |
| May 19 (later) | Block merger, sort, multi-row fixtures | pdf2json-extractor.ts, bank-rows.ts |
Real UAT observations
| Bank / file | Reported behavior | Likely cause |
|---|---|---|
| CBA | Failed to load | pdf2json warnings on 850KB PDF; possibly 0 transactions, or a parse error swallowed as 500. Split header + mixed date formats still fragile. |
| NAB | First visible tx ~16 Sep; many rows missing | groupRowsIntoBlocks drops orphan rows — rows before 16 Sep never form a valid block |
| NAB | Description stuck on opening balance | Opening-balance row may start a block or merge into the first one; filter may run on the wrong field |
| ANZ | Not re-tested after block merger | Risk of the same multi-row issues recurring |
Why tests pass but UAT fails
| Tests | Real PDFs |
|---|---|
| Ideal x/y rows, 1–5 per scenario | Hundreds of rows, irregular y-rounding, split headers, wrapped descriptions |
| Fictional flat text (text parser) | Not used by the API anymore |
| No test loads actual PDF buffers in CI | CBA/NAB geometry never asserted |
Lesson: a passing pdf2json-extractor.test.ts does not prove production PDFs work.
Root causes in pdf2json-extractor.ts
1. Orphan row drop (NAB skips)
groupRowsIntoBlocks (~432–466): a row that's neither a transaction start nor a continuation gets the current block finalized and is never attached anywhere. Real NAB/CBA lines often lack a date in the date column on the first visual row.
2. Strict transaction-start detection
rowStartsTransaction requires a date at left or in the date column; CBA rows like "28 Oct" (no year) can fail if fragments split across columns.
3. Opening balance bleed
isSkipRow skips rows whose joined text matches "opening balance" — but split fragments may not join to that exact phrase, so a block can still absorb the balance-column amount or description text.
4. CBA header / page 1
Header split across 4 lines; if page 1 has no valid header, activeBounds stays null and the entire page is skipped.
5. No pdf-parse fallback
app/api/parse-data/route.ts only calls parseTransactionsFromPdf — any pdf2json failure is a 500 or empty state, no degraded path.
Is multi-bank feasible?
pdf2json is the right direction for column debit/credit — text-only parsing can't fix CBA/NAB layout. But one extractor for all banks, without per-bank real fixtures and iterative UAT, will keep regressing.
Pragmatic path: (1) pdf-parse fallback for UX, (2) pdf2json per bank behind flags, (3) probe snapshots as tests, (4) fix one bank at a time.
Recommended next steps
Immediate: hybrid API (pdf2json primary, pdf-parse fallback on 0 txs/throw); hard-skip opening-balance blocks before column conversion; never silently drop orphan rows with an amount.
Short term: capture real probe fixtures for one failing CBA page and one NAB page; add PDF_PARSE_DEBUG=1 logging per block skip reason; re-run the CBA header/column probe.
Medium term: re-UAT ANZ multi-row EFTPOS; Westpac/File_000 stay out of scope until ANZ/CBA/NAB are stable; update the Notes.md probe table after each fix.
Process: don't merge parser changes without at least one new real-layout fixture from probe output; commit session notes and code separately once a bank hits its success criteria.
Success criteria (unchanged — still not met for CBA/NAB)
| Target | |
|---|---|
| CBA | Loads; no large date gaps; WORLDREMIT + COSTCO as separate transactions |
| NAB | All rows; no year in description; not opening-balance text |
| ANZ official | Merchant + EFTPOS in description |
| ANZ unofficial | Chronological order |
Better: one bank per PR, with a probe snapshot test before merging.
Better: keep the pdf-parse fallback until pdf2json matches probe counts on all three banks.
Watch: large block-merger refactors without real PDF fixtures — tests green, UAT red.
Watch: assuming a chronological sort fixes "missing" rows — it only reorders what was actually parsed.
Improved, not finished
Hybrid PDF fallback + multi-bank parser hardening
Picked up from May 19: restored a usable path when pdf2json fails, and stopped the parser from dropping split rows before they had enough data to become a transaction.
What changed
| Area | Change | Files |
|---|---|---|
| Upload API | Tightened PDF validation — requires both MIME type and .pdf extension | route.ts |
| Upload API | Removed any size cast, used native File.size | route.ts |
| Upload API | Consistent JSON error responses | route.ts |
| Parser API | Hybrid parsing: pdf2json first, pdf-parse fallback on throw/zero txs | lib/parser/index.ts |
| Types | Local declaration for untyped pdf-parse entrypoint | lib/types/pdf-parse.d.ts |
| pdf2json parser | Preserved pending split rows instead of dropping incomplete ones | pdf2json-extractor.ts |
| Tests | Regression coverage for split-date orphan rows + opening-balance non-leakage | bank-rows.ts, pdf2json-extractor.test.ts |
Verification
npm test -- --run
→ 191 passed, 1 skipped
npm run lint
→ still failing — pre-existing project-wide issues, not this session's changes
(components/ui/input.tsx empty interface, next.config.ts require,
probe-pdf2json.ts any, FileUploader.test.tsx any, tests/setup.ts any)
Important implementation notes
The pdf-parse fallback imports pdf-parse/lib/pdf-parse directly — importing the package root made Vitest execute the package's debug path and try to open a test PDF that doesn't exist in this repo.
pdf2json remains the correct primary parser (it's the only one with x/y layout for debit/credit/balance columns); pdf-parse restores usability when layout parsing fails but can't reliably solve every multi-column statement on its own.
The split-row fix is conservative: a row with an amount, description, or partial day value and no current block is kept pending; if the next row starts a transaction and the pending block has no date yet, it's merged in rather than forcing a new boundary.
This is a bank transaction analyser: upload a PDF statement, extract transaction rows, filter out summary lines like opening balance or totals, categorize each transaction, group by month, return a spending summary. The hard part isn't the dashboard — it's PDF parsing. Bank PDFs look tabular but extraction libraries return fragmented text, and different banks place dates/descriptions/debits/credits/balances at different x/y positions. That's why the parser is split into detection, extraction, filtering, grouping, and summarizing. The project now has both a text-parser path (pdf-parse) and a layout-parser path (pdf2json) — layout is primary, text is fallback.
Portfolio assessment
Framed correctly, this is a good portfolio project — not a CRUD app or charting demo, but practical engineering around messy real-world data: parser design, file-upload validation, typed contracts, tests, incremental hardening from real failures. To make it portfolio-worthy: real redacted pdf2json fixture snapshots per bank; per-bank accuracy reporting (transaction count, known merchant rows, debit/credit correctness, no opening-balance leakage); a small parser-confidence/warnings UI section; a clean lint/test/build; a README with architecture diagram, screenshots, supported banks, limitations, test evidence. Keep scope tight before adding AI or database storage — a reliable parser is more impressive than a broad, inaccurate finance app.
Next recommended steps
- Capture one redacted pdf2json probe fixture each for ANZ, CBA, and NAB
- Add tests asserting the real-bank success criteria from the May 19 session
- Run real PDF UAT again after the fallback + split-row fixes
- Fix existing lint failures
- Update README/CHANGELOG once real-fixture validation is done
Rolled back
ANZ-only rollback and official ANZ PDF blocker
Multi-bank support removed from active code and tests. File 000 (unofficial ANZ) works; the official ANZ statement is still blocked — by corrupted PDF extraction, not a parser logic bug.
Executive summary
This session rolled the parser direction back toward ANZ-only support. File 000 is considered working. The official ANZ 2025-01-06 file is still blocked: most rows collapse because the parsed output contains corrupted amounts such as 2024.00 (from "EFFECTIVE DATE … 2024") and 432919512.00 (from a transfer/account reference). The parser now rejects those leaked values — but it can't recover the true amount once the PDF text extraction has already lost it.
Until the official ANZ raw extraction is inspected and either recovered or reported clearly as unsupported.
What changed in working tree
| Area | Change | Files |
|---|---|---|
| Parser scope | Removed active multi-bank DateFormat/columnMap plumbing | types.ts, detector.ts, extractor.ts, index.ts |
| Tests | Deleted CBA/NAB multi-bank unit test | multiBank.test.ts |
| Fixtures | Removed CBA/NAB fixture exports; ANZ fixture uses 3 amount columns | sample-transactions.ts |
| ANZ normalization | Support merged File 000 prefixes: "05 MARANZ", "04 MARVISA", "02 MARPAYMENT" | index.ts |
| Official ANZ table shape | Vertical header detection: Date/Description/Category/Withdrawal/Deposit/Amount | detector.ts |
| Official ANZ table shape | Ignored category-only continuation lines; normalized standalone "-" placeholders | index.ts |
| Amount parsing | Anchored to trailing ANZ amount columns; accepted signs, $, commas, decimals | extractor.ts |
| Safety guard | Rejected leaked description-number values (years near "EFFECTIVE DATE", large account refs) | extractor.ts |
| Regression tests | Official ANZ + File 000 regression coverage added | transactionParser.test.ts |
Verification
npm run test:run -- tests/unit/transactionParser.test.ts → 27 passed
npm run test:run → 162 passed, 1 skipped
git diff --check → passed
npm run lint still fails on pre-existing project-wide issues outside this rollback (input.tsx, extract-pdf.js, next.config.ts, probe-pdf2json.ts, test any casts).
Current blocker: official ANZ 2025-01-06
Observed bad output:
08 JUL
VISA DEBIT PURCHASE CARD 2606 ... EFFECTIVE DATE 04 JUL
food
-$2024.00
-
$-2024.00
09 JUL
ANZ M-BANKING FUNDS TFER TRANSFER 311463 FROM
friends
-$432919512.00
-
$-432919512.00
The parser now rejects these as leaked values rather than displaying them as real spending. One row survives validation — $12.00 against "VODAFONE AUSTRALIA … EFFECTIVE DATE 28 DEC 2024" — because it's a plausible real amount, not a leaked year/reference. This explains why monthly grouping showed only one transaction: grouping works, but only one row survives validation.
Key diagnosis
File 000 works because the parser receives real amount tokens (-$23.32 0.00, 0.00 +$2000.00). The official ANZ file doesn't, because rows appear to reach the app already corrupted: 20.24 renders as 2024.00 (decimal loss / date-year leakage), and 432919512 appears as an amount on transfer rows. The true amounts for those rows aren't present in the displayed table output at all.
Parser-only code must not guess missing financial amounts. It can reject impossible rows and preserve correct ones, but it cannot reconstruct cents that are absent from extraction.
The deeper lesson here wasn't any single bug — it was the architecture choice. The parser tried to work out which bank's format it was looking at from the text itself, and every format added to that detection made it more likely to misfire on a format that already worked. Getting CBA/NAB right kept regressing ANZ, and vice versa, until the regex collisions between formats made the whole thing less reliable than just supporting one bank well.
Next time: don't make the parser guess. Put a bank picker in the UI and let the user select the format before parsing, then run only that bank's rules. A little UX friction beats a parser silently misreading its own input.
Next required step
Capture the raw pdf-parse text for the official ANZ PDF before parseTransactions() transforms it. If it contains real tokens like $20.24, fix normalization/extraction to preserve them. If it already only contains corrupted tokens, the fix has to move to layout-aware extraction or a clear unsupported-format/low-confidence warning.
- Add a local-only debug capture for uploaded PDF raw text, gated by an env var (
DEBUG_PDF_TEXT=1) - Save raw extraction to a non-committed debug file or log during local dev
- Create a redacted fixture from the real raw text around failed official rows
- Add parser tests from that raw fixture
- Only commit once official ANZ raw text either parses correctly or fails with a clear warning instead of fake transactions
Tracked parser/test changes still uncommitted. Untracked project/session files intentionally left alone: .claude/, agents/, sessions/, Notes.md, probe-pdf2json.ts. Do not use git reset --hard or broad cleanup commands unless explicitly requested — the working tree contains useful uncommitted rollback work.
Shipped
Calendar View feature + groupByMonth year-rollover fix
Started from a portfolio-veracity worry, resolved that the parser pipeline itself is the real skill signal, then shipped a genuinely different angle: a calendar grid that doubles as a categorization QA tool — and caught a real year-rollover bug along the way.
Executive summary
ANZ's own banking app already has a merchant/category spend summary, raising the question of whether this project still demonstrates anything beyond a CRUD app — the same question that kicked off the whole project (see Why this exists on the Overview page): the bank app answers "how much, by merchant, this month," but never by day or by week, which was the original itch. Resolved that the existing parser pipeline (PDF → categorize → 160+ tests → CI) is the actual skill signal, not novelty — but added a genuinely different angle anyway: a Google-Calendar-style day grid showing every transaction on its real date, color-coded by category, doubling as a QA tool for spotting categorization drift while lib/categories.ts is still being expanded.
That QA angle wasn't theoretical — laying transactions out on a calendar made it obvious, at a glance, exactly which merchant strings were landing in misc instead of their real category, in a way that scrolling a flat table never surfaced. Bugs that were invisible in a list became visible the moment they had a date and a color next to them.
A real correctness bug was found and fixed in groupByMonth (transactions defaulting to today's real-world year instead of the statement's actual year) — then found again in a different form during real UAT: the first fix assumed ascending chronological order, but real ANZ statements can list months descending, which ran the year away to 2030 on an actual uploaded statement. Both directions are now handled.
What changed in working tree
| Area | Change | Files |
|---|---|---|
| Bug fix | groupByMonth no longer defaults every transaction to today's calendar year; tracks year via an explicit year token when present, or rollover-detection (only on an actual Dec↔Jan adjacency, either direction) as fallback | lib/parser/group.ts |
| New helper | groupByDay — buckets one month's transactions by day, attaches merchant via extractMerchant | group.ts, index.ts |
| New component | CalendarView — month grid, day chips (Merchant · Category · ±$Amount), colored via existing CATEGORY_COLORS, prev/next navigation | components/CalendarView.tsx |
| Integration | New "Calendar View" collapsible section, same pattern as "Monthly & Category Breakdown" | TransactionDisplay.tsx |
| Tests | Rollover tests (ascending + descending), explicit-year-token precedence, groupByDay bucketing/merchant/field-preservation | groupByMonth.test.ts, groupByDay.test.ts (new) |
| Docs | Calendar view feature note | README.md, CHANGELOG.md |
No new dependencies, no persistence changes — reads from the same in-memory parsedData the app already produces per upload.
Decisions made this session
- Color palette: reused the existing
CATEGORY_COLORSmap rather than hand-rolling a new one — already covers all 11 categories, already used elsewhere, zero new design decisions. - Scope cut (v1): no click-to-recategorize from the calendar, no multi-month wall view, no mobile layout pass, no persistence change. Deferred, not forgotten.
- Year-rollover fix: accepted as an internal-consistency fix, not a guarantee of absolute year correctness — there's no statement-period metadata anywhere in the parser to derive the true year when no year token exists at all.
Bugs found during UAT (real ANZ PDF, not synthetic fixtures)
1. Year runaway to 2030
The first groupByMonth fix treated any month-number decrease as a year rollover. The real statement lists months descending (July, then June, ...), so every single-month step looked identical to an actual Dec→Jan wrap, incrementing the year every time. Fixed by only triggering on an exact Dec(12)↔Jan(1) adjacency, in whichever direction the statement is ordered — a normal one-month step, either direction, never touches the year now.
2. Day chips truncated with no way to read the full line
CalendarView used Tailwind truncate (ellipsis) plus a native title hover tooltip — not discoverable; the user could only see "Vodafone…" with no price visible and no working hover/click affordance. Fixed by dropping truncate and letting chip text wrap (whitespace-normal break-words); day cells grow to fit instead of hiding content.
Neither bug was something the still-green test suite could have caught — the descending-order case and the visual truncation weren't exercised by synthetic fixtures. Both were caught only by uploading a real statement and looking at the render.
Verification
npm test -- --run
→ 169 passed, 1 skipped (started session at 161/1; +8 net)
npx tsc --noEmit
→ clean, no errors
Manual UAT: real ANZ PDF (3–4 months, descending order) uploaded through the running app at localhost:3000. Confirmed after fixes: correct year per month, full chip text visible without hover, category colors matching the table view, month navigation working across the whole statement.
Not yet committed — drafting README/CHANGELOG/session log first, then writing the commit message by hand. Untracked files present from before this session, intentionally left alone: .claude/, Notes.md, agents/, probe-misc-rate.ts, probe-pdf2json.ts, sessions/ (this file is the one new addition).