Scraping Infrastructure

What Is a User Agent?

A user agent is a string of text sent with a web request that identifies the client software making the request, such as a specific browser, version, and operating system.

scraping-infrastructure

Slug: user-agent

Role in scraping

Servers often inspect the user agent string to decide how to respond, and to detect non-browser traffic. Scrapers commonly set realistic user agent strings so requests resemble ordinary browser traffic rather than an obvious automated script.

Beyond the user agent

Modern anti-bot systems look well beyond the user agent string alone, examining broader browser fingerprints and behavioral signals, so user agent management is only one piece of sustainable data collection.

SEO
User Agent | APISCRAPY Glossary
A user agent is a string of text sent with a web request that identifies the client software making the request, such as a specific browser, version, an…