Python Web Scraping Engineer – Fully Remote – £70k
Please note – to be eligible for this role you must be UK-based with the unrestricted right to work in the UK. This organisation does not offer sponsorship.
My client is hiring a Senior Python Scraping Engineer to design, build, and operate high‑volume, highly resilient web scraping systems, primarily focusing on scraping Google at scale. This specialist role suits experienced engineers with deep expertise in bot detection and evasion.
The role sits at the intersection of data engineering, reverse engineering, and large‑scale systems reliability, delivering accurate, timely, and trusted data.
My client is a leader in adopting AI‑assisted and agentic coding practices, ideal for engineers who actively use AI tools to improve productivity, reasoning, and system design.
Key Responsibilities
- Design and operate large‑scale scraping systems handling 10+ million requests per day, primarily targeting Google and Google‑like platforms.
- Build robust scrapers for dynamic, JavaScript‑heavy environments using browser automation and hybrid approaches.
- Continuously adapt to changes in markup, request flows, ranking logic, and anti‑automation mechanisms.
- Engineer extraction pipelines with a strong emphasis on correctness, consistency, and observability.
- Implement and maintain advanced anti‑bot evasion strategies, including proxy and request routing strategies, browser and headless fingerprinting, CAPTCHA handling and mitigation.
- Monitor system health, detect anomalies early, and debug complex production issues.
- Optimize performance, cost, and latency across large‑scale scraping infrastructure.
- Collaborate closely with data engineers, data scientists, and product teams to ensure scraped data is reliable and usable.
- Produce clear documentation and operational runbooks to support long‑term maintainability.
Required Technical Skills
- Expert‑level web scraping skills using Python.
- Direct, hands‑on experience scraping Google at scale – essential.
- Deep understanding of anti‑bot and bot‑detection systems, browser and network fingerprinting, CAPTCHA systems and mitigation techniques, and scaling scraping infrastructure reliably.
- Strong knowledge of HTTP, TLS, cookies, headers, redirects, and browser networking behaviour.
- Experience with Playwright, Selenium, Puppeteer or equivalent frameworks.
- Comfortable designing asynchronous and concurrent scraping architectures.
- Proven experience running scraping systems at scale in cloud environments.
- Excellent debugging skills and the ability to reason about complex failure modes.
- Strong communication skills, with the ability to clearly explain complex technical behaviour.
#J-18808-Ljbffr