A locally-run research aid that finds companies worth a cold email — and refuses to tell you anything it can't cite. This page documents how it was built, what broke, and what's still honestly unsolved, using the same evidence discipline the tool itself is built around.
Exhibit A — what it actually outputs
Illustrative, not a live company result — a composite example built from this project's real, shipped output shape, kept generic on purpose so no real small business's name sits next to a tool-generated judgment call about it. The finding structure, field names, and wording style are exactly what the tool actually writes.
“CTO” found on this page, near the text: “...About the team. Meet the leadership behind our managed IT support...”
A result shaped exactly like this didn't exist until a real session fixed a false positive that pointed at a blog post about installing Microsoft Teams instead. See Case 01.
Brief
There are plenty of job boards, official and unofficial, and plenty of automation tools to make applying faster. But they all work the roughly 20% of jobs that get posted publicly, not the much larger hidden market filled through word of mouth — or through someone bold enough to reach out first.
The idea came from one piece of advice: search for the company, find the CTO or the hiring manager, work out a problem you could solve for them, and send a cold email to start a conversation. That's a different kind of appeal from a resume and cover letter fed into an ATS that mostly leads nowhere.
My problem was finding those companies — especially small to mid-sized ones, which are more likely than the big names everyone competes for to take on a junior, entry-level candidate on an international visa. So I specced this project to cast a net over a location for those companies, including non-tech businesses as long as they have a tech team, and take my chances.
Given a location and a technical interest, Prospect Scout finds publicly visible companies that are plausible prospects, collects verifiable case-study and hiring signals from their own sites, and writes a local report for a human to review before reaching out. It is deliberately not a lead-generation guarantee — a search can return zero eligible companies when public evidence doesn't support one, and that's treated as a correct answer, not a failure.
The one rule everything else follows from: it must never invent a company, a case study, a problem, or a contact. Every finding ships with a source URL, an excerpt, and a confidence level the human is meant to actually distrust a little.
Log
Search widening, ranking, and a results page worth reading
OSM discovery widened to libraries and research institutes alongside offices; scout/research.py gained bounded concurrency so a 250-candidate run finishes in minutes instead of tens of minutes; scout/ranking.py shipped — eligible before needs-review, then location and evidence strength, reasons shown per company instead of an opaque score. Role matching switched from substring to whole-word after a real false positive: “software engineer” was matching inside “software engineering,” a degree name, not a role.
Baseline: fetcher, discovery, evidence pipeline, reporting
The polite fetcher (scout/fetcher.py) — honours robots.txt, identifies itself honestly, skips login-gated content. Sitemap-first page discovery, sector classification, location verification, and local Markdown/JSON reporting. First documented baseline; the web app and CLI both usable end to end.
Profile-aware discovery and a named contact, only when the site actually publishes one
OSM's single fixed office=* query became a per-profile tag set — the technology profile now asks for office=it/research/engineer, not everything. Team/leadership-page contact extraction shipped, deliberately scoped to a company's own published team page — no third-party enrichment, no guessed emails. Caught two real bugs in testing before shipping: naive matching once flagged “Sales Director” as a “CTO” hit (both words contain the substring “cto”), and a name-extraction heuristic once grabbed “Our Head” instead of the real name sitting two words later.
Live-testing the live tool, and fixing what it found
This session. Ran real searches, read every result by hand, and didn't like some of them — see Cases 01–03 below. Shipped from that: a word-boundary fix for a false-positive contact page, a minimum-evidence floor before the tool will assert a sector, contact-aware and MSP-language-aware wording on its weakest finding type, and a full redesign of outreach guidance — from a composed, sendable draft to personalized talking points, after a first attempt over-corrected into being generic.
Platform-detection and broken-link signals, an ATS lookup set aside, docs caught up
Platform-detection (detect_platform_signal()) flags Shopify/Squarespace via confirmed literal fingerprints, closing gap #5. A broken-link signal surfaces a dead sitemap/homepage link as low-confidence evidence, deliberately excluding this project's own guessed fallback paths. A Greenhouse/Lever job-board lookup adapter was prototyped and deliberately left unwired — recorded as "deprioritized, not rejected," since ATS-hosted companies skew toward funded/scaled employers, the opposite of this project's small/informal focus. (Correction, 2026-09-26: that prototype was never committed, so the decision survives but the code doesn't.) A private research log and this page were both started this session to catch documentation up to code that had gotten ahead of it.
A robots.txt parser that silently ignored real rules
stdlib's urllib.robotparser only implements the 1996 draft and has no support for the */$ wildcard syntax RFC 9309 requires — confirmed live via a real CPython tracker issue, then reproduced directly: a Disallow: /*.pdf$ rule was silently ignored (fetch allowed) under stdlib and correctly honoured under Protego, the RFC 9309-compliant parser Scrapy itself switched to for the same reason. Swapped in scout/fetcher.py; all 8 existing robots tests passed unchanged. Also evaluated and shelved a promising new discovery lead (tenders.vic.gov.au) after the whole domain started returning a Cloudflare challenge to every fetch — not ruled out permanently, just not viable to build against right now.
A live crash, caught and fixed the same day it was found
The technology profile's 17 tag pairs were combined into one 68-clause Overpass request, which timed out outright against Melbourne's full metro bounding box — confirmed live via direct reproduction. The failure was silently swallowed and reported to the user as "no businesses found... double-check the spelling," which was actively misleading since geocoding had succeeded both times. Fixed by splitting into one request per tag pair, with per-request failure isolation so one slow tag doesn't blank out the rest. Verified live: the same search that previously returned 0 domains returned 190 afterward. Also replaced scout/sitemap.py's hand-rolled robots-sitemap parsing with the same Protego property used above, and formally recorded AAGE (gap #2) as considered and not pursued, closing out a recommendation the research log had left open.
The fix for the fix: batching, and three more confirmed false positives from actually reading the output
The 2026-08-21 fix traded a crash for a real slowdown — 17 sequential Overpass requests, each individually safe, adds up. Live-trial-confirmed a batch size (4 tag pairs per request; 8 came in right at the timeout edge, 12+ failed outright) that cuts a technology-profile search to 4 requests instead of 16, with real margin, not a guess. Also dropped office=software from the technology tag list (confirmed zero Melbourne hits) and corrected a stale comment that misattributed the profile's wider, deliberately-generic tag set to research it didn't actually come from — that reasoning (catching BHP-shaped companies hiding behind a generic office=company tag) lived only in this page's Case 03 until now. Reading real results again surfaced Case 04 below, plus three fresh confirmed instances of gap #10 and three brand-new gaps (#14, #15, #16) — see the open ledger.
robots.txt Crawl-delay finally honoured, and a real sector-language collision found via a specific-company check
Closed gap #4: scout/fetcher.py now reads a site's declared Crawl-delay and waits, lock-guarded per host so concurrent company fetches (gap #7) can't both slip through the same delay window at once. Separately, a scout inspect run against a real Melbourne accounting firm surfaced a genuine terminology collision, confirmed across five of the firm's own pages: every use of the word "technical" in a professional-services job ad or careers page meant tax/audit domain depth, never software skill — the opposite of what the word means when the same tool is hunting for a developer role. hidden_need_finding() now adds that caveat automatically whenever result.sector == "professional services", and scout/outreach.py's weakest finding type stopped repeating its own "Suggested next step" verbatim as the "Outreach angle" — it now gives the one cold-email tip that actually fits a low-confidence, no-contact result (a small ask, not a job ask), drawn from a set of real cold-email tips collected for the project instead of restating itself. Also recorded a new, deliberately unsolved gap: word-of-mouth/internal-referral hiring — common at small startups and scaleups — produces no public text on any source, by design, which is a harder and different problem than any discovery-source coverage gap (see the open ledger).
Gap #17 confirmed a second time, and three real drifts caught by re-reading the project's own docs
A real, ongoing job-search conversation surfaced a second independent confirmation of gap #17, this time from the other side of the hiring process: a real small tech company's actual reasons for its hiring choices — cost, legislation, and first-time risk tolerance on one hand, "value compatibility" over demonstrated technical insight on the other — sat entirely in a private conversation and a subjective read of a candidate, with zero trace on the company's own site. Not a new gap; the same permanent ceiling gap #17 already named, confirmed against a second real case rather than left as a one-off. Company left unnamed here, same policy as Exhibit F's note on Cases 01–04.
Separately, re-reading README.md and CHANGELOG.md against the actual code found three real drifts: CHANGELOG.md was three days behind four already-shipped changes (this page's own 2026-09-02 entry above had already caught up, the changelog hadn't); README.md had accidentally lost its "Privacy and etiquette" section and its link to docs/MVP_SPEC.md during an unrelated session-progress commit, and had never documented the /inspect specific-company search mode at all despite it being fully wired since scout/webapp.py's discovery-skip work; and protego — a real runtime dependency declared in requirements.txt — was missing from the local .venv, silently failing 5 of 151 test modules at import time rather than surfacing as a clear dependency error. All three fixed this session; none were code bugs, all were the project's own record of itself falling behind its own code.
Two real evidence-extraction bugs were fixed the same day. A theme's mobile menu built as <div role="navigation"> instead of a real <nav> slipped past the chrome stripping, so the same site-wide menu text matched a role term on every page and produced identical excerpts for three different case studies. And location verification only ever checked the homepage, so a company whose address sat on its /careers or /locations page was wrongly marked as needs-review — now every page already fetched in the run is checked, with no extra requests.
Research that survives a restart, and a use for sitemap dates
Every run had been throwaway: research a company, write a report, forget it. scout/store.py added a local SQLite store where rerunning a company replaces its old entry instead of stacking duplicates, so the store always reflects the latest research. Separately, sitemap <lastmod> dates now travel with each finding as observed_at, and ranking uses them as a final tie-breaker — a fresher dated signal wins a tie, and a page with no declared date gets None, never a made-up fetch-time stamp.
A map of everything researched so far
OpenStreetMap discovery already knew where each candidate was, then threw that away. Coordinates are now carried from discovery through to each result, and a new /map page plots every saved company with a coordinate on a Leaflet map (OSM tiles, no API key). Companies without one are listed underneath instead of silently dropped. For the specific-company mode, which skips discovery and so never had a coordinate, the tool now reads the company's own schema.org geo data if the homepage publishes it — a real but partial source, confirmed present on one of four sites checked, with no fuzzy geocoding fallback when it's missing. One bug caught by using it the same day: the results page had no link to the map at all.
Paused
Put on hiatus with 191 tests passing and nothing half-built. The reason is in the verdict below.
Exhibits B–E
None of these came from a code review. They came from running real searches against real companies and refusing to accept a result just because it didn't crash. Company names below are illustrative, not the real domains this was found against — the bugs and fixes are real and unedited; the specific small businesses involved aren't named here on purpose (see the note under Exhibit F).
Team-contact discovery matched pages by keyword: "team" among them. It flagged /blog/how-to-install-microsoft-teams/ as a leadership page — because "team" is a literal substring of "teams," the software product, not the org-chart concept. The extracted "contact" was Business Needs, pulled from the headline “Why Your Small Business Needs a CIO.”
Fixed with a word-boundary match, deliberately scoped to team-page keywords only — the case-study keyword list keeps its existing substring behaviour on purpose, since one of its terms ("success-stor") is an intentional truncation to catch both "story" and "stories." Re-ran the exact same company afterward: it found the real about page, and a real name.
Sector inference counts keyword hits per category and reports the winner, no matter how thin the margin. A local council library's homepage scored two hits under fashion/retail: "collection" and "clothing." A library collection isn't a clothing collection — but the word is identical either way.
Added a floor: a sector needs at least two independent keyword hits before it gets reported as anything but unknown. Re-ran the library afterward, expecting it fixed.
Prompted by a real story: someone found a tech role at a mining company through a networking event, not a search. Checked directly whether OpenStreetMap even had the data — it did. BHP's Melbourne HQ is a real, recently-verified OSM node, tagged office=company, with a live website tag pointing at bhp.com.
The technology profile's OSM query only asks for office=it/research/engineer — not office=company, the tag every generic corporate HQ gets. BHP was findable and invisible to this profile at the same time. Filed as evidence for a real, separately-scoped question: tech work increasingly lives inside companies that don't look like tech companies at all — not something to patch into this session's fixes by accident.
The broken-link signal reported two real companies' own image URLs (/wp-content/uploads/.../customer-service.png-shaped paths) as dead links — evidence text literally read “failed to load.” Both images loaded fine. The fetcher treats any non-HTML response as an error, and that error was being reported using the same wording as an actual dead link, with no distinction between the two.
Fixed by checking the actual response status before calling something broken, not just whether the page turned out to be usable as evidence. A response that loaded successfully but isn't HTML is now silently skipped — not usable as page text, but not a dead link either.
broken_link_signal finding.Contract
Not a style guide — an enforced shape. A finding missing any of these isn't a weaker finding, it's not a finding.
| Field | Requirement |
|---|---|
| source_url | Canonical URL after redirects — always the company's own site. |
| evidence | Short observed text, actually present on the fetched page. Never paraphrased into a claim. |
| confidence | High, medium, or low — a name match with no name nearby is low, always. |
| suggestion | A specific next action tied to that evidence, framed as a hypothesis, never a certainty. |
| status | eligible / needs_review / rejected. Only eligible companies get recommended — no evidence, no recommendation. |
Open ledger
Recorded on purpose, not discovered by a reader. A tool that only admits its limits when asked isn't actually being evidence-based about itself. Seven of seventeen shown below, numbered as in the private gap log — the rest live in the project's private gap log, not repeated here to avoid two copies drifting apart.
OpenStreetMap doesn't have every company mapped
Confirmed real misses, including a large multinational. This is the ceiling on discovery today, not a bug — the fix is a second, name-verification-only source, not a bigger query. That source (Australian Business Register lookups) was specced and then paused along with the project, before any code was written. One promising-looking candidate (AAGE's graduate-employer list) was researched and deliberately not pursued — it skews toward large companies running formal graduate programs, the opposite of the small/informal profile this gap is actually about. A live-trial-confirmed alternative exists (Wikidata's public SPARQL endpoint, 810+ Melbourne-headquartered organisations with a website property) but carries the same large/notable-organisation skew for a different reason — see gap #17 below for the harder problem neither one touches.
A tagged site can outlive the business it was tagged for
One confirmed case: an OSM record pointed at a domain since hijacked for SEO spam in another language. Nothing today tells "the fetch succeeded" apart from "this is still who OSM says it is."
Keyword matching can't tell intent from coincidence
A university's own career-services page can contain the exact phrase you searched for as a degree name, not a hiring signal. Word-boundary matching fixed the false certainty; it can't fix the ambiguity itself — that's left to the human review step on purpose. Reconfirmed 2026-08-29 on three more real small-business sites, same reasoning, no new fix attempted.
A real, currently-open role can still be invisible
If a company only lists a vacancy on a third-party job board and nowhere on its own site, this tool won't find it — job-board scraping is explicitly out of scope, not an oversight.
Selling tech isn't the same as needing tech talent
A Shopify storefront selling PCs matched via shop=computer — a correct tag that says nothing about whether the business hires engineers. No literal-text fix closes this; worth checking whether OSM even has a tag that draws this distinction before building anything.
A blog title can satisfy a "real word" boundary check and still be the wrong page
A real company's own /outsourced-it-vs-internal-team/ article — a genuine blog post, not a staff page — passed the same word-boundary check that correctly excludes "microsoft-teams." A real fix direction exists this time: the site's own sitemap already separates posts from pages.
Word-of-mouth and internal-referral hiring is invisible to any public-evidence method, not just OSM
Prompted by a real case: a friend's own interview experience at a Melbourne accounting firm suggested a genuine internal technical need that no public page anywhere — the firm's own site, its job ads, its Seek listings — showed any trace of. A company that hires this way produces no public text to find, on any source, by design. This isn't a coverage gap a better discovery source closes (see gap #02) — it's a permanent ceiling on what public-evidence methodology can verify, and it disproportionately affects the exact small startup-to-scaleup company size this project is most interested in surfacing. Confirmed a second time, 2026-09-04, from the hiring side rather than the candidate side (see build log) — the real reason behind a real hiring decision again sat entirely in private conversation, nowhere on the company's own site.
Exhibit F — Research trail
Pulled from the project's private research log. Small businesses are described rather than named, for the same reason as Cases 01–04. Not every row has its own URL — most small/mid companies were only ever surfaced through a shared list page, never fetched individually, and that distinction is kept honest below rather than implying a per-company check that didn't happen. Rows here are neutral, factual confirmations (membership lists, exchange status, public directories) about companies large or public enough that a factual mention carries no real characterization risk — a different bar from Cases 01–04 above, which involved a tool-generated judgment ("weaker lead," "no engineering language") about a real small business, and were made illustrative for that reason instead.
| Company | Live URL(s) actually fetched | Outcome & match to this project |
|---|---|---|
| Three small Melbourne tech companies | none — no mechanism found | The original discovery-gap examples: small and informal, too small for AAGE, ASX or government registers to catch. Still unsolved — the exact company size this project was built for. |
| Odoo | none — no mechanism found | Private multinational; no AAGE/ASX/government hit. Same gap, different company size. |
| Alcon | none — flagged only, never fetched | Large enough to plausibly run an AAGE grad program — named as worth checking, never actually checked. |
| NAB | aage.com.au/employer-members | Confirmed AAGE member; also a current ASX-20 constituent. The strongest single hit in the set. |
| Suburban Rail Loop Authority | aage.com.au/employer-members | Confirmed AAGE member; also a VIC statutory authority. |
| JBS | aage.com.au/employer-members (as “JBS Pork Australia”) | Confirmed AAGE member. Never ASX-listed (lists on B3/NYSE instead). |
| Vicroads | aage.com.au/employer-members | Confirmed AAGE member. |
| Ausnet | aage.com.au/employer-members + thebigmeet.com.au/melbourne | Confirmed by two independent fetches, plus AER regulated-network status by secondary corroboration. Not currently ASX-listed — delisted 2022. |
| DuluxGroup | aage.com.au/employer-members | Confirmed AAGE member. Delisted from ASX in 2019 — an ASX-only mechanism would have missed it entirely. |
| Essential Energy | aage.com.au/employer-members + thebigmeet.com.au/melbourne | Confirmed both. HQ'd in NSW, not Melbourne — shows these mechanisms can surface a company that recruits into Melbourne without being based there. |
| Fulton Hogan | aage.com.au/employer-members | Confirmed AAGE member. Privately held, never ASX-listed. |
| Jemena | aage.com.au/employer-members (as “Jemena/Zinfra”) | Confirmed AAGE listing, plus AER regulated-network status by secondary corroboration. |
| Zinfra | aage.com.au/employer-members (shared “Jemena/Zinfra” entry) | Weaker, shared hit — the AAGE list doesn't list it independently, flagged as an ambiguous name-mapping case. |
| Thoughtworks | aage.com.au/employer-members (as “Thoughtworks Australia”) | Confirmed AAGE member. Never ASX-listed (NASDAQ-listed US parent). |
| Sportsbet | responsiblewagering.com.au/about/ | Confirmed niche industry-body member. Only found because the sector was already suspected — doesn't generalise. |
| Kinetic IT | aiia.com.au/member_directory/kinetic-it-pty-ltd/ | Confirmed individual member page; the full AIIA directory's login-free bulk access was never confirmed. |
| Viasat | au.prosple.com/graduate-employers/viasat-australia | Noted but deliberately not counted — Prosple sits structurally adjacent to a job board, excluded per this project's own scraping rule. |
| Leica Biosystems | BioMelbourne Network directory — returned 403 Forbidden | Unconfirmed by direct access. Membership known only via a secondary Dun & Bradstreet profile. |
| Culture Amp | cultureamp.com/careers | Contrast example: real engineering-team language (“Engineering blog,” a named grad program) — used to shape the MSP-vs-product-team scoring signal. |
| HotDoc | practices.hotdoc.com.au/careers/ | Counter-example: a real target company whose careers page carries zero engineering language — the false-negative risk that blocked a hard MSP-exclusion filter. |
| BHP | bhp.com + Overpass node 2456680377 | Case 03. Findable in OSM under the generic office=company tag, invisible to the technology profile's tag set. |
| Ferocia | ferocia.com.au + ferocia.com.au/careers/ | No individual names anywhere on-site. CTO only confirmed off-site, via itnews.com.au trade-press coverage. |
| A small cloud consultancy | its own homepage | Run directly through analyse_company(), bypassing discovery entirely — same location_verified: false limitation later confirmed on BHP. |
| A Melbourne accounting firm | one job ad + four more of its own careers pages | Confirmed real terminology collision across every page checked: "technical" meant tax/audit domain depth, never software skill — the 2026-09-02 build-log entry above. Shipped as a sector-aware caveat in hidden_need_finding(). |
Verdict
The tool works. It finds companies, cites everything it says about them, admits when it finds nothing, and puts what it has researched on a map. What it couldn't do was the job it was built for: finding and verifying small to mid-sized companies from public evidence failed more often than plain word of mouth did.
Gap #17 in the ledger above is the reason, and it isn't a bug. The companies I most wanted to reach are often the ones that hire through referrals and conversations, so they leave no public text for any tool to find. A better data source wouldn't close that — the ceiling belongs to public-evidence discovery itself, and a tool built on the rule "never claim what you can't cite" hits it head-on.
So it's paused rather than abandoned, with nothing half-built. If I pick it back up, the next step is already specced: use the business register to verify companies I hear about through people, rather than trying to discover them. That would make the tool a support for word of mouth instead of a replacement for it — which is what the evidence says it should have been.
Under the hood
Language — Python 3
Web app — FastAPI + Uvicorn
Fetching — httpx, robots.txt-honouring, self-identifying user agent
Parsing — BeautifulSoup, hand-rolled sitemap XML discovery
Discovery — OpenStreetMap (Nominatim geocoding + Overpass queries)
Storage — local SQLite, one row per company, replaced on rerun
Map — Leaflet with OpenStreetMap tiles, no API key
Tests — stdlib unittest, fixture & fake-based, zero live network calls
Reports — local Markdown + JSON, nothing leaves the machine