Test Automation Specialist

AI Test Automation Engineer – FTC – Hybrid/London

This is a Fixed Term Contract until the end of 2026 but is highly likely to extend.

We are looking for an AI Test Automation Engineer to join on a fixed-term contract until the end of the year, with potential to extend. The role is based in London, with some office presence required.

You will help design and implement automated testing and evaluation frameworks for MCP Servers, LLM-based systems, and agentic AI workflows, ensuring accuracy, safety, reliability, and performance across complex AI-enabled platforms.

Key Responsibilities

• Develop automated testing frameworks for MCP Servers and related AI systems.
• Design evaluation strategies for LLM accuracy, safety, and reliability.
• Build automated tests using Python, Pytest, and BDD frameworks.
• Identify and address AI failure modes including hallucination, latency, and incorrect tool usage.
• Work with engineering, QA, and product teams to define quality metrics and acceptance criteria.
• Support Agile delivery, ensuring testing aligns with sprint goals.
• Produce reporting on quality outcomes, risks, and improvement areas.
• Maintain documentation for test cases, evaluation pipelines, and validation strategies.

Requirements

• Strong Python programming experience for test automation and evaluation.
• Expertise in Pytest and familiarity with BDD tools such as Behave or Cucumber.
• Knowledge of LLM evaluation approaches such as RAGAS, DeepEval, or custom evaluation pipelines.
• Understanding of agentic AI issues including hallucination, tool misuse, and performance bottlenecks.
• Experience testing AI workflows, distributed systems, or microservices environments.
• Strong knowledge of Agile delivery and CI/CD quality integration.
• Excellent communication, analytical, and problem-solving skills.

Nice to Have

• Experience with Model Context Protocol (MCP) or agent orchestration solutions.
• Exposure to observability, monitoring, or logging tools for AI systems.
• Background in API and service integration testing.
• Knowledge of containerised and cloud-native environments.
• Experience in enterprise AI automation or intelligent platform engineering.

#J-18808-Ljbffr…