Our benefits
- Share Options (EMI) scheme
- 25 days annual leave, plus flexible bank holidays and the opportunity to buy additional days
- Enhanced workplace Pension scheme - opt-in salary sacrifice scheme
- Life Insurance (3x annual salary)
- Employee Assistance Programme (EAP) and workplace wellbeing initiatives
- Private Healthcare cash-back scheme
- Flexible working hours and location
- Flexible benefits, such as: Cycle to Work, volunteer days/opportunities and charity events
- Enhanced parental leave packages and enhanced sick pay
- Training and development opportunities
- Engagement and celebration activities– anniversaries, birthdays, team building, company-wide events
Role specifics
- Salary range: Market rate
- Reporting to: Test Lead
- Key stakeholders: Tech and Product teams
- Organisational Framework Level: 2
About You / Job Summary
We’re looking for an experienced and curious QA Analyst to lead the testing of our AI and GenAI‑powered features. This role goes beyond traditional SaaS QA — it requires a mindset for dealing with uncertainty, edge cases, model behaviour, and testing for quality when outcomes are not always predictable. If you’re passionate about both automation and understanding how intelligent systems behave in the wild, we want to hear from you.
You’ll be embedded in a cross‑functional scrum team, working closely with engineers, data scientists, and product managers to define and execute test strategies for AI features. You’ll bring an automation‑first mindset but be comfortable with exploratory and manual testing when required — especially when testing model‑driven features. Your focus will include not just functional correctness, but also trust, safety, performance, and the overall quality of the AI experience.
Key Accountabilities
- Design, develop, and maintain test plans and automated test suites for AI/GenAI features across frontend and backend.
- Test complex AI‑driven systems where responses may vary — validate logic, relevance, completeness, tone, and safety.
- Build automation frameworks using Playwright and Cucumber with a focus on maintainability and scalability.
- Develop tools/scripts to validate model outputs and compare results across model versions (e.g., regression drift, hallucination checks).
- Define and execute non‑functional tests (e.g., performance under load, prompt injection resilience, fairness/bias testing).
- Log and manage defects effectively; collaborate in root cause analysis and prevention.
- Actively contribute to CI/CD testing pipelines to ensure fast, reliable releases.
- Stay current on trends in AI/ML testing and suggest improvements to our QA practices.
Key role measures
- Automation Coverage & Test Strategy Execution – Delivering and maintaining high levels of automated test coverage across AI/GenAI features, with a focus on scalable and maintainable test suites using Playwright and Cucumber.
- Model Output Quality & Safety Validation – Proactively identifying and validating issues in AI model outputs – including hallucinations, bias, and prompt injection vulnerabilities – to ensure safe, relevant, and trustworthy user experiences.
- CI/CD Integration & Test Reliability – Contributing to fast, stable releases by integrating tests into CI/CD pipelines and maintaining a low rate of flaky or failing tests.
- Exploratory Testing & Edge Case Discovery – Uncovering subtle bugs and edge cases through thoughtful exploratory testing, especially in areas where AI behaviour is less predictable – making a meaningful impact from early in the role.
Experience and skills
Our ideal candidate would have experience in the following areas:
- Proven experience testing AI/ML‑powered systems, ideally including GenAI features (e.g., LLMs, chatbots, classifiers).
- Strong proficiency with Playwright, Cucumber, and automation‑first test strategies.
- Understanding of common GenAI testing challenges: non‑determinism, bias, hallucinations, injection attacks.
- Experience with prompt testing, output validation, or model comparison tools.
- Familiarity with REST APIs, JSON, and integrating model endpoints into test frameworks.
- Inquisitive and detail‑oriented mindset – able to challenge assumptions and uncover subtle issues.
- Experience in Agile/Scrum environments and collaborative cross‑functional teams.
- Familiarity with CI/CD tooling (e.g., Azure DevOps, GitHub Actions, or similar).
- Strong written and verbal communication skills.
Nice to Have
- Background in machine learning or data science.
- Experience testing ethical/guardrail functionality in AI systems.
- Knowledge of data labelling, synthetic data generation, or test data management in AI contexts.
- Familiarity with tools like OpenAI’s evals, LangChain testing modules, or similar frameworks.
Diversity, Equity and Inclusion
If you’re excited about this role but your experience doesn’t align perfectly, we encourage you to apply anyway and tell us more about yourself. You may be just the right candidate for this or other roles.
We believe that seeing the world from all sorts of angles makes life better for all. We want you to know that the things that make you an individual, like your identity, age, ethnicity, religion, ability and background, are things that we choose to celebrate and support.
We are a scale‑up company, and as we continue to grow, we are passionate that having a diverse, inclusive and authentic workplace will remain at our core. We are creating an inclusive environment where our people can thrive.
Our values are aligned and at the heart of everything we do. We are respectful, united, rigorous, relentless and ethical.
#J-18808-Ljbffr