Senior AI QA Engineer with Python (Manual & Automation)
Skills
About the role
We are seeking a Senior AI QA Engineer to validate the performance of AI-driven video analysis systems, focused on detecting key moments in sports content. The role combines manual and automated testing, working with pre-labeled video assets provided by the customer.
Responsibilities
Audit live sports games (NBA, MLB, NFL, NHL) to ensure the AI/Inference Service correctly tracks and labels major sports moments such as touchdowns, home runs, and buzzer-beaters as they happen
Work with timecodes, video frames, transcriptions, and captions, applying sports-related contexts, lingo, and metrics
Act as the human expert who catches when the AI hallucinates, misinterprets a sports rule, or produces errors
Collaborate directly with AWS engineers to detail bugs and validate resolutions
Review AI-generated labels and metadata tags to ensure they make sense for the sport and meet advertising industry standards (IAB rules), keeping content brand-safe for ad placement
Compare the inference system's outputs against customer-provided labels to determine accuracy and identify missed or incorrectly detected events
Execute both manual test cases (for nuanced or edge cases) and automated test cases (for validating outputs at scale and ensuring consistency)
Log discrepancies, defects, and gaps between expected and actual results, and collaborate with the inference/development team to triage and resolve issues
Maintain clear QC documentation, track accuracy metrics, and help build a smooth testing process that bridges traditional sports broadcasting with new AI technology
Serve as a communication bridge between the customer and technical teams, clarifying results and expectations
Repeat validation iteratively as new labeled assets or model updates are provided to ensure ongoing accuracy and reliability
Requirements
3+ years of experience in both manual and automation QA, preferably in video-focused or AI-driven environments
Knowledge of testing LLMs and understanding of their workflow
Proficiency in automation scripting with Python, Selenium, or similar tools for comparing JSON outputs to ground truth at scale
Understanding of software development and QA cycles, including defect logging, triage, and reporting
Ability to interpret labeled data and validate model outputs
Familiarity with concepts like computer vision and transcript analysis (no need to understand internal model workings)
Strong communication skills for stakeholder interaction and reporting
Detail-oriented and iterative approach to testing
Proficiency in English at an Upper-Intermediate level (B2) or higher
Nice to have
Experience with video testing tools or frameworks
Prior work in sports analytics or media technology
We offer
International projects with top brands
Work with global teams of highly skilled, diverse peers
Healthcare benefits
Employee financial programs
Paid time off and sick leave
Upskilling, reskilling and certification courses
Unlimited access to the LinkedIn Learning library and 22,000+ courses
Global career opportunities
Volunteer and community involvement opportunities
EPAM Employee Groups
Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn
Questions about this role
Want AI Applyd to auto-apply to roles like this?
We tailor your resume per posting, fill the forms, and track replies for you.