How a Marketplace Tracked Competitor Catalogs With Ecommerce Web Scraping
A US-based online marketplace sells laboratory and medical equipment. To plan its assortment and pricing, it needed a complete, structured view of a competing catalog of roughly 1.2 million laboratory equipment products. That catalog lived only on the competitor's website, so the web was the only place to get it.
99%+ data quality in client's own template
vs internal data collection approach
From kickoff to complete delivery
Client Snapshot
A US-based online marketplace sells laboratory and medical equipment. To plan its assortment and pricing, it needed a complete, structured view of a competing catalog of roughly 1.2 million laboratory equipment products. That catalog lived only on the competitor's website, so the web was the only place to get it.
The Challenge
The target site had no uniform structure and was not professionally built, so product pages changed layout from one category to the next. Every product followed a different format, and the full catalog ran to 1.2 million rows across more than 900 attribute columns. A standard template-based scraper would either break constantly or return a dump of irrelevant page data.
The marketplace also needed every product image downloaded and zipped into a single file, with each image link mapped back to its record in the CSV output. It wanted the data delivered in its own template rather than as raw HTML, and it needed a cost-efficient route instead of a large internal engineering project.
The Solution
APISCRAPY built a custom-programmed web crawler for the competing catalog. Because the site was unstructured, the team defined multiple business and logic rules so every required data point was captured, however an individual product page happened to be laid out.
Instead of dumping whole pages, the crawler extracted only the fields the marketplace needed and mapped them straight into the template it had designed. Product images were downloaded, zipped into one file, and linked back to their records in the CSV, which cut manual cleanup substantially.
The work ran as a managed ecommerce web scraping engagement, not a self-serve tool. APISCRAPY assigns a dedicated project manager at no extra cost, rotates requests across a pool of 100+ million proxies to avoid IP blocks, and uses outcome-based pricing, so clients pay for verified data delivered rather than crawler hours. Output can be delivered as CSV, JSON, Excel, or XML, or loaded directly into databases such as PostgreSQL, MySQL, BigQuery, and Redshift.
The Results
The marketplace received a structured dataset covering all 1.2 million laboratory equipment products, in its own template, with images packaged and linked to each record. Data quality exceeded 99%, the full delivery took three weeks, and the client saved around 80% on data collection costs.
Instead of a raw page dump, the team now had a competitor catalog it could filter, compare against its own listings, and load into its existing systems without another cleanup pass.
We didn't need another folder of raw pages. We needed a competitor's entire catalog in our own format, and that is exactly what arrived.
Tell us your sources and schema. We scope a pilot in days with a data sample you can evaluate before committing.