Senior AI Test Automation Engineer - Hybrid Madrid

Role description

We are looking for the very Top Talent… and we would be delighted if you were to join our team!

More in detail, UST is a multinational company based in North America, certified as a Top Employer and Great Place to Work, with over 35,000 employees worldwide and presence in more than 35 countries. We are leaders in digital technology services, delivering large-scale technological solutions to top-tier companies.

What we look for?

We are looking for a Senior AI Test Automation Engineer to join the ML & AI ART, owning the end-to-end test automation strategy for an advanced AI/MLOps platform. This platform powers business-critical AI capabilities such as LLM-based assistants, RAG systems, agentic workflows, and ML pipelines.

Main tasks and accountabilities will be:

Define and evolve the test automation architecture aligned with the Bank’s Test Strategy

Design scalable, reusable automation frameworks and patterns across teams

Analyse requirements and ensure testability during Agile ceremonies (PI Planning, Backlog Refinement, etc.)

Design and automate test cases (functional & non-functional) based on risk and coverage

Implement automation using Python frameworks (Playwright, pytest, Behave/BDD)

Develop AI/ML-specific testing approaches (LLM evaluation, non-deterministic behaviour, RAG workflows)

Build and maintain API contract tests and microservices validation

Integrate and run automated tests in CI/CD pipelines (GitLab)

Use Docker/Kubernetes for reproducible test environments

Manage test execution across environments (DEV, INT, UAT, pre-PROD)

Monitor test health, handle flaky tests, and ensure reliable reporting

Report defects in Jira with proper traceability and evidence

Produce audit-ready test artefacts and ensure compliance with governance standards

Collaborate with cross-functional teams (POs, Engineers, MLOps, Data Engineers)

Act as Test Automation Ambassador within the ART

Ensure end-to-end traceability across requirements, tests, and results

Maintain and evolve test suites as systems (APIs, models, prompts) change

What do we expect from you?

Strong experience in Python-based test automation (Playwright, pytest, BDD frameworks)

Proven ability to design and own test frameworks, not just test scripts

Experience integrating automation into CI/CD pipelines (GitLab preferred)

Knowledge of parallel execution, reporting tools, and flaky test management

Solid understanding of Git, clean code, and code review practices

Experience with Docker and basic knowledge of Kubernetes

Exposure to AI/ML testing (LLMs, RAG, non-deterministic systems) or strong motivation to learn

Understanding of APIs, microservices, event-driven architectures

Familiarity with SAFe, Agile, and DevOps principles

Experience with Jira and test management tools (Xray, Agile Hive, etc.)

Strong analytical mindset with end-to-end system understanding

Awareness of data privacy, security, and compliance in regulated environments

Work Location

Hybrid position in Madrid (3 days at client´s office, Sol area, next to the underground).

Work Schedule

Business hours (40 hours weekly with 1 hour for lunch).

What can we offer?

23 days of Annual Leave plus the 24th and 31st of December as discretionary days!

Numerous benefits (Health Care Plan, teleworking compensation, Life and Accident Insurances).

`Retribución Flexible´ Program: (Meals, Kinder Garden, Transport, online English lessons, Health Care Plan…)

Free access to several training platforms

Professional stability and career plans

UST also, compensates referrals from which you could benefit when you refer professionals.

The option to pick between 12 or 14 payments along the year.

Real Work Life Balance measures (flexibility, WFH or remote work policy, compacted hours during summertime…)

UST Club Platform discounts and gym Access discounts

If you would like to know more, don’t hesitate to apply and we’ll get in touch to fill you in detail. We are waiting for you!

In UST we are committed to equal opportunities in our selection processes and do not discriminate based on race, gender, disability, age, religion, sexual orientation or nationality. We have a special commitment to Disability & Inclusion, so we are interested in hiring people with disability certificate.

Skills

Playwright, Python, API Testing, pytest, Machine Learning, GitLab CI, Test Data Management, RAG

About UST

UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact - touching billions of lives in the process.