Big Data Engineer

We’re Hiring: Big Data Engineer

Are you passionate about building scalable big data solutions and high-performance data pipelines? Join our team and work with cutting-edge technologies to develop enterprise-grade data platforms.

Experience Required

4+ years of software engineering experience developing Big Data pipelines.

3+ years of hands-on experience programming in Scala and PySpark.

3+ years of experience building solutions on AWS, including AWS Glue.

3+ years of experience working with Big Data file formats, including Parquet.

1+ years of experience with Data Validation, Data Profiling, or building Rule-Based Data Engines.

1+ years of experience leveraging AI coding assistants or code-generation tools (e.g., GitHub Copilot, Codex) in daily software development tasks.

Key Responsibilities

Design, build, and optimize a highly scalable, rule-based data validation engine.

Develop robust, high-performance Big Data pipelines using Scala, PySpark, and AWS Glue.

Write clean, well-tested code utilizing optimized formats like Parquet for efficient storage and retrieval.

Perform detailed data validation and profiling to ensure data quality and consistency across data systems.

Use enterprise-approved AI tools to streamline software development workflows and automate repetitive tasks.

Collaborate with cross-functional teams to translate business requirements into scalable technical solutions.

Evaluate emerging AI and Big Data technologies to drive innovation.

Participate in code reviews, testing, debugging, and performance optimization.

Required Skills

Scala

PySpark

AWS

AWS Glue

Parquet

Big Data Pipelines

Data Validation

Data Profiling

AI Coding Assistants (GitHub Copilot/Codex)