How AI Web Scraping Handled Layout Changes Across 200 Sites Without Breaking
A growth-stage HR tech company sells labor market dashboards showing hiring demand, salary ranges, and skills trends by role and region. Its product depends on job posting data pulled from 200 sources. Customers renew based on one thing — whether the numbers on Monday morning are complete and current.
Zero pipeline outages in six months
Up from ~88% with in-house scrapers
Maintenance dropped from ~60 to under 5 hrs/week
Client Snapshot
A growth-stage HR tech company sells labor market dashboards that show hiring demand, salary ranges, and skills trends by role and region. Its product depends on job posting data pulled from 200 sources, from national job boards to niche industry boards and large employer career portals. Customers renew based on one thing — whether the numbers on Monday morning are complete and current.
The Challenge
The data team ran 200 custom scrapers, each built on hard-coded CSS selectors and XPath rules tied to a specific page structure. Job boards redesign constantly, run A/B tests on listing layouts, and migrate between applicant tracking systems, so the same scraper could work on Tuesday and fail on Wednesday. Across the portfolio, 8 to 15 scrapers broke every week.
Hard failures were the easy ones, since a scraper that returned nothing triggered an alert. The costly failures were silent — a moved salary field came back empty, a location landed in the job title column, or a pagination change quietly cut a source to its first page. These errors surfaced only when a customer asked why salaries for an entire region had dropped overnight.
Four engineers were spending roughly 60 hours a week on selector repairs instead of product work. Field completeness across the dataset hovered around 88%, and the company missed its weekly data delivery window 11 times in the six months before switching. Two enterprise renewals came with explicit questions about data reliability.
The Solution
APISCRAPY replaced the selector-based scrapers with AI extraction that identifies fields by meaning and page context rather than by fixed position in the HTML. The models learn what a job title, salary range, location, employment type, and posting date look like across many layouts, so a field that moves, gets renamed, or changes its markup is still recognized. Major job boards were covered from APISCRAPY's library of pre-built scrapers, and the remaining niche boards and career portals were configured during onboarding.
Rollout ran in three phases over ten weeks. APISCRAPY first ran in parallel with the legacy scrapers on 40 high-volume sources for 30 days, so the team could compare completeness and accuracy side by side. The remaining 160 sources then moved over in two waves, with the old scrapers retired only after each wave passed validation.
Clean, normalized data now arrives through an API feed and scheduled loads directly into the client's cloud data warehouse. Salary figures are standardized to annual values with currency preserved, duplicate postings syndicated across multiple boards are collapsed into one record, and every record carries its source URL and capture timestamp. The engineering team no longer owns a single scraper.
| Layer | What It Does |
|---|---|
| Semantic field detection | Locates each target field by content patterns and surrounding context, so cosmetic redesigns, renamed classes, and reordered sections do not break extraction. |
| Continuous change detection | Monitors every source for structural shifts, field completeness drops, record volume anomalies, and schema violations on each run. |
| Automatic re-mapping | When a layout shift is detected, the extraction model re-identifies fields on the new structure and validates output against the expected schema before delivery. |
| Human-in-the-loop QA | Records that fall below confidence thresholds are routed to APISCRAPY's data operations team for review, so uncertain data is checked before it ships. |
| Schema-locked delivery | Output always matches the client's fixed JSON schema, so downstream dashboards never see a column shift, whatever changed on the source site. |
The Results
In the first six months after full cutover, APISCRAPY's change detection flagged more than 1,100 layout and structural changes across the 200 sources. About 94% were absorbed automatically by the extraction models with no human involvement, and the rest were resolved by the data operations team within the same delivery cycle. Not one weekly delivery window was missed.
The freed engineering capacity went straight into the product. The team shipped a skills taxonomy feature and a regional salary benchmarking module in the following two quarters, both of which had been stalled for over a year. Both enterprise accounts that had raised reliability concerns renewed.
| Metric | Before APISCRAPY | After APISCRAPY |
|---|---|---|
| Scrapers breaking per week | 8 to 15 | 0 pipeline outages |
| Field completeness | About 88% | 99%+ |
| Engineering hours on scraper maintenance | About 60 per week | Under 5 per week |
| Missed weekly delivery windows (6 months) | 11 | 0 |
| Time from layout change to restored data | 1 to 3 days | Same delivery cycle |
We used to find out a job board had redesigned when a customer emailed us about missing salaries. Now the sites change and our data simply doesn't notice.
Tell us your sources and schema. We scope a pilot in days with a data sample you can evaluate before committing.