BBB.org's business data isn't in the HTML — it's in an analytics script tag

Quick answer BBB.org doesn't render business data into the HTML table it looks like it does. Both the search-results page and every business profile page embed the real data as one inline JSON blob — webDigitalData — sitting inside a <script> tag meant for analytics, not for you. The visible <dt> / <dd> list underneath it only carries the fields the analytics layer left out (accreditation date, years in business). If you scrape the DOM table and ignore the script tag, you'll get a name and maybe a phone number and nothing else. If you scrape the script tag and ignore the DOM, you'll get ratings and IDs but no address, no years-in-business, no website. You need both, merged, per business. Why does the DOM only have half the data? 🕵️ When we built the BBB Business Leads Scraper , the first pass assumed BBB profile pages worked like most directory sites: a template with labeled fields you css_first() your way through. That assumption survives for exactly four fields — address, accreditation date, years in business, and website — which live nowhere except a <dt> / <dd> definition list further down the page. Everything that actually matters for lead scoring — business name, BBB letter grade, accreditation status, phone number, the internal IDs BBB uses to build its own canonical URL — comes from a different place entirely: a webDigitalData object, wired into the page for BBB's own analytics vendor, that happens to be complete, well-formed JSON: # actors/bbb-business-leads-scraper/src/parsers/common.py WEB_DIGITAL_DATA_MARKER = " webDigitalData " def extract_web_digital_data ( html : str ) -> dict | None : marker_pos = html . find ( WEB_DIGITAL_DATA_MARKER ) if marker_pos == - 1 : return None brace_start = html . find ( " { " , marker_pos ) raw = _extract_balanced_json ( html , brace_start ) ... return json . loads ( raw ) That is not a regex grabbing {.*} between two markers — a naive non-greedy match snaps shut on the first stray } inside a nested object, which this blo