QA Engineer, AI

Job Title: QA Engineer, AI

FLSA status: Regular Full-Time Remote (US-Based)

Reports to: QA Manager

About TrueLearn

TrueLearn is an educational software company headquartered in Charlotte, NC, dedicated to empowering medical institutions and individual learners to maximize performance on high-stakes licensure exams. Through a suite of innovative, learning science–driven tools, TrueLearn supports clinicians at every stage of their careers - from board preparation through continuing education.

To date, TrueLearn has prepared over 70,000 healthcare professionals in their pursuit of board certification and works closely with hundreds of medical and health science institutions and residency programs throughout the United States. Our learning science–driven platform supports clinicians at every stage of their careers, from board preparation through continuing education. The company is rapidly growing, continuously innovating its technology platform and expanding its portfolio to meet the evolving needs of the healthcare community.

At the heart of everything TrueLearn does is a simple but powerful truth: the work touches the people who touch patients, which means quality here is never cosmetic. Every line of code, every model, and every experiment has the potential to ripple outward into better care, better outcomes, and better lives.

TrueLearn's culture is built around authenticity, empowerment, and a relentless commitment to advocating for learners. Strong core values guide decisions, priorities, and tradeoffs as the company scales - ensuring that growth never comes at the cost of alignment, integrity, or impact on healthcare professionals and patient care.

We are looking for a QA Engineer to help us protect and raise the quality of our learning experiences. This is a hands-on role where you will work day-to-day inside an engineering team - close to the code, the product, and the learners we support.

Position Summary

Testing AI features presents new challenges, but at its core, it relies on solid testing judgment. Your job is to help us determine what "good" looks like and to catch issues before a learner sees them. Your work spans three areas:

Testing AI-powered features. You work out how we measure whether an AI feature is actually good - building test sets, defining what “good” means when there’s no single right answer, and catching it when quality slips as prompts or models change. You learn to tell normal AI variation apart from a real bug.

Using AI to test better. You use AI tools in your own work - to help write test cases, build automation, sort through failures, and generate test data - and you build a clear sense of where they help and where they mislead.

Testing fundamentals. Good test strategy, reliable automation, clear bug reports, and the judgment to know what’s worth testing and what isn’t. This is the foundation everything else builds on.

Key Responsibilities

Build the feedback loop. Partner with engineers on observability for AI features (tracing, prompt/completion logging, and production monitoring) so real interactions feed back into test sets and live evals.

Own the test strategy for our features - from unit and integration coverage through end-to-end checks - and make quality a shared habit on the team.

Get involved early and stay vocal: bring a quality lens to planning and design, not just to the code at the end (shift-left), and stay close to features all the way through to production.

Build the systems that check output at scale - test sets, automated grading (including using an LLM as a judge), test harnesses, and dashboards that show whether quality is trending up or down over time.

Test for the failure modes that matter in a clinical-education context - factual accuracy, hallucination, inappropriate difficulty, prompt and retrieval regressions, and guardrail breaks - and partner with clinical/learning-science reviewers to validate correctness efficiently.

Understand the impact of latency and token cost in production AI features and when they become a failure point.

Build and maintain automated test coverage across the stack (UI, API, and service layers) and keep it integrated into CI so failures surface early and clearly.

Stay up to date with modern testing tools and help establish which ones belong in the team’s workflow.

Advocate for quality with evidence - clear repro steps, risk framed in terms of learner impact, and data that helps the team decide what to fix and when.

Partner closely with engineers, product, and data/ML - embedded in the team, testing with them rather than at them.

What We're Looking For

We care about strong testing fundamentals and a hunger to learn new technologies. If you have solid testing instincts and are genuinely curious about how AI can transform education, we want to talk - regardless of whether your experience is primarily with traditional software or if you've already started experimenting with AI testing.

A solid foundation in software testing. You understand test design and automation, and you have good instincts for what’s worth testing and what isn’t. You aim your effort at the riskiest problems, not the loudest ones.(Make It Count)

A curious mindset toward new technology. You're comfortable with ambiguity, ready to learn how we evaluate AI models, and eager to apply your testing expertise to a new frontier. (Own the Outcome)

An evidence-driven way of working. You drive decisions with data rather than a gut feeling, and you change your mind when the evidence says so.(Lead with Evidence)

An enthusiastic learner. You keep up with new tools and approaches, and you're always looking for ways to improve your testing toolkit. (Always Learning)

Comfortable with modern test tooling, or eager to get there. Some exposure to frameworks like Playwright, Cypress, or Selenium; pytest, Jest, or similar; plus API testing and CI. You can read and understand the code you’re testing - our stack is React, Angular, Node.js, and ASP.NET - though you don’t need to have shipped in all of it.

A collaborative partner who speaks up early. You embed with engineers, raise quality concerns clearly, and get involved from planning onward rather than waiting until the end. You make it safe to surface problems.(Help Others Succeed)

Genuine interest in the mission. You don’t need a healthcare background, but you should care that a wrong AI-generated question can mislead someone who is caring for patients - and that catching it matters.

You don’t need years of AI-testing experience to be right for this role. You need strong testing fundamentals, real curiosity, and the drive to grow into the AI side quickly - we’ll help you get there.

Why Should You Apply

This is a chance to do quality engineering at the frontier of AI, inside a company with a genuinely meaningful mission. You’ll build the evaluation systems that decide whether an AI-generated question is good enough to put in front of a future clinician - and you’ll do it with a team that wants to work the way good teams should work: with trust, with rigor, with humor, and with each other.

If that sounds like the kind of work you want to do, we’d love to talk.

The Perks

Highly Competitive benefits including medical, dental, vision

Additional benefits for employees to opt into

Health Savings Account & Flexible Spending Account

Retirement – 401K

Life Insurance

Self-regulated PTO

Parental Leave

Job Type: Full-time

Pay: $80,000.00 - $100,000.00 per year

Benefits:

401(k)

401(k) matching

Dental insurance

Employee assistance program

Flexible spending account

Health insurance

Health savings account

Life insurance

Paid time off

Parental leave

Retirement plan

Vision insurance

Location:

United States (Required)

Work Location: Remote