Senior Test Analyst

The Senior QA Engineer will drive the quality strategy for agentic AI systems, including applications powered by LLMs, autonomous agents, RAG pipelines, tool-calling, memory modules, and complex AI decisioning flows.

The role ensures that AI-driven capabilities meet strict enterprise-grade standards of accuracy, safety, reliability, and compliance, while collaborating closely with engineering, product, and AI/ML teams globally.

Job Responsibilities

Write automation alongside development to test unit level, API and e2e UI.

Implement backend services supporting agent runtimes, tool execution, and memory layers

Integrate LLMs into business workflows and internal systems

Contribute to orchestration logic connecting tools, APIs, and external services

Write clean, maintainable, well-tested code

Participate in architectural discussions

Collaborate with AI engineers to productionize experimental features

Contribute to monitoring, logging, and performance optimization

Requirements

Strong hands-on experience testing LLM-based or agentic AI applications.

Expertise in Python automation (PyTest, Playwright, Postman).

Understanding of prompts, embeddings, vector stores, tokenization. • Experience testing RAG pipelines and grounding data workflows.

Proficiency with APIs, microservices, and cloud environments (Azure/AWS/GCP).

Ability to evaluate unstructured AI output using structured scoring techniques.

Strong analytical and debugging skills in complex AI-driven workflows.

Desirable Skills

Experience with AI evaluation tools: TruLens, PromptLayer, MLFlow, W&B.;

Familiarity with agentic frameworks: LangChain, AutoGen, Semantic Kernel, CrewAI.

Understanding of LLM safety concepts, adversarial testing, guardrails, red teaming.

Knowledge of model fine-tuning, RLHF, and observability pipelines.

Exposure to distributed architecture and enterprise platforms.

Ability to mentor QA engineers on AI testing methods.

Qualifications

5+ years of professional software engineering experience

Hands-on work with generative AI and agentic systems.

Experience working in cross-functional product teams

Degree in Computer Science, Engineering, or related field (or equivalent experience)

Strong problem-solving skills and ability to work in fast-evolving technical domains

Interest in AI-native software architectures and continuous learning mindset

Certifications such as ISTQB Advanced, Azure/AWS/Google AI certifications (optional).

Equality Statement

SmartStream is an equal opportunities employer. We are committed to promoting equality of opportunity and following practices which are free from unfair and unlawful discrimination.