Quality Assurance Automation Engineer

Python QA Automation Engineer (AI & Agentic Systems)

Workload: Full time hours

Location: London (Onsite) – hybrid working

We need a Python QA Automation Engineer to assure quality, reliability, and compliance of AI‑driven enterprise systems exposing internal tooling to AI agents via Model Context Protocol (MCP).

Key Responsibilities:

• Build and maintain Python‑based automation frameworks to test MCP servers and AI workflows
• Design evaluation pipelines for LLM accuracy, safety, latency, and reliability
• Implement automated tests using Pytest and BDD frameworks
• Integrate quality gates into CI/CD pipelines for continuous assurance
• Detect and mitigate agentic AI failure modes such as hallucinations and incorrect tool usage
• Define quality metrics and acceptance criteria with engineering and product teams
• Contribute to Agile ceremonies within a Classic Agile delivery model
• Produce detailed quality reports and maintain test and validation documentation

Must‑Have Skills:

• Strong Python experience in test automation
• Advanced Pytest usage and BDD framework knowledge (Behave/Cucumber)
• Hands‑on LLM evaluation experience (RAGAS, DeepEval, or custom frameworks)
• Understanding of agentic AI risks and performance bottlenecks
• CI/CD integration for automated quality checks in distributed systems
• Experience with Model Context Protocol (MCP) or agent orchestration tools
• AI observability, monitoring, or logging exposure
• API and microservices testing experience
• Containerised or cloud‑native platform knowledge

#J-18808-Ljbffr…