Big Data Engineer

Job Title

Big Data Engineer (EL3)

Experience Required

4+ Years

Employment Type

Full-Time

Job Summary

We are looking for an experienced Big Data Engineer to design, develop, and optimize scalable big data pipelines and data validation solutions. The ideal candidate should have strong hands-on experience in Scala, PySpark, AWS, and AWS Glue, along with expertise in data validation, profiling, and modern big data technologies. The role also involves leveraging AI-assisted development tools to improve engineering productivity and software quality.

Required Skills & Experience

4+ years of software engineering experience developing Big Data pipelines.

3+ years of hands-on programming experience using Scala and PySpark.

3+ years of experience building cloud-based data solutions on AWS, including AWS Glue.

3+ years of experience working with Big Data file formats such as Parquet.

1+ years of experience in data validation, data profiling, or developing rule-based data validation engines.

1+ years of experience using AI-powered coding assistants or code-generation tools such as GitHub Copilot, OpenAI Codex, or similar tools in daily software development.

Key Responsibilities

Design, develop, and optimize scalable, rule-based data validation engines capable of validating data across diverse data pipelines.

Build robust, high-performance Big Data pipelines and processing platforms using Scala, PySpark, AWS, and AWS Glue.

Develop clean, maintainable, and well-tested code while utilizing optimized storage formats such as Parquet for efficient data storage and retrieval.

Perform comprehensive data validation, data profiling, and quality assurance to ensure data accuracy, consistency, and reliability across multiple data sources.

Utilize enterprise-approved AI development tools to streamline software development, automate repetitive tasks, and improve engineering productivity.

Collaborate with cross-functional teams including Business Analysts, Data Engineers, Architects, and Product Owners to translate business requirements into scalable technical solutions.

Evaluate emerging Big Data technologies and AI-driven engineering practices to enhance solution architecture and innovation.

Participate in code reviews, unit testing, debugging, performance tuning, and optimization to ensure high-quality software delivery.

Technical SkillsProgramming Languages

Scala

PySpark

Python (preferred)

Big Data Technologies

Apache Spark

PySpark

Parquet

Data Validation Frameworks

Data Profiling

Cloud Platforms

AWS

AWS Glue

Data Engineering

ETL/ELT Development

Data Pipelines

Data Quality

Rule-Based Validation Engines

Data Processing

Development Practices

Code Reviews

Unit Testing

Debugging

Performance Optimization

Software Development Best Practices

AI-Assisted Development

GitHub Copilot

OpenAI Codex

AI Coding Assistants

Preferred Qualifications

Experience designing scalable data validation frameworks.

Strong understanding of distributed data processing and optimization techniques.

Familiarity with enterprise software development lifecycle (SDLC) and Agile methodologies.

Excellent analytical, problem-solving, and communication skills.

Ability to work effectively in cross-functional and collaborative teams.

Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)