AI Evaluation Engineer

AI Evaluation Engineer

Location: Pune, India

Seniority: Mid-level (3-6 years)

Purpose: Execute comprehensive evaluations of FNZ's AI agents across the six-pillar framework, working as a generalist while developing specialist expertise in 1-2 pillars.

Key Responsibilities:

Design and conduct evaluations covering Task Performance, Safety, Efficiency, Groundedness, Robustness, and Suitability

Create "golden sets" of test examples representing expert judgment on desired agent behaviour

Develop evaluation rubrics and scoring criteria aligned to FNZ Evaluation Framework principles

Build comprehensive test suites covering happy paths, edge cases, and adversarial inputs

Evaluate multi-step agentic workflows: planning, tool selection, execution, error handling

Assess agent groundedness: verify outputs against knowledge bases, detect hallucinations

Document findings with clear evidence; collaborate with development teams on remediation

Contribute to building automated evaluation platform and CI/CD integration

Skills and Experience:

3-6 years in software testing, QA engineering, AI/ML development, or data science

Hands-on test automation skills, experience with ML frameworks highly valuable

Practical experience evaluating LLM applications, RAG systems, or AI agents

Understanding of prompt engineering, retrieval-augmented generation, and agent architectures

Analytical mindset to decompose complex agent behaviours and identify failure modes

Strong documentation and presentation skill

About FNZ

FNZ is committed to opening up wealth so that everyone, everywhere can invest in their future on their terms. We know the foundation to do that already exists in the wealth management industry, but complexity holds firms back.

We created wealth’s growth platform to help. We provide a global, end-to-end wealth management platform that integrates modern technology with business and investment operations. All in a regulated financial institution.

We partner with the world’s leading financial institutions, with over US$2.4 trillion in assets on platform (AoP).

Together with our clients, we empower nearly 30 million people across all wealth segments to invest in their future.