Data Engineer

Role : Data Engineer Work Locations: Pune, Mumbai, Chennai, Bangalore CTC: Up to 19 LPA Experience: 6–10 Years Domain: Cards & Payments Background Verification: Pre-Onboarding

Job Summary

We are seeking a highly skilled Senior Data Engineer with strong expertise in PySpark, Apache Spark, Databricks, Airflow, and Cloud Data Platforms. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing pipelines, building robust ETL frameworks, and supporting data engineering initiatives in the Cards & Payments domain.

The candidate should have hands-on experience working with distributed data processing systems, cloud-based data lakes, and workflow orchestration tools while ensuring scalability, performance, and reliability of data solutions.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using PySpark and Apache Spark.

Build and optimize ETL/ELT workflows for processing large-scale datasets.

Develop and manage data solutions using Databricks, including Jobs, Workflows, and Delta Lake.

Create and maintain workflow orchestration processes using Apache Airflow.

Design and optimize complex SQL queries, stored procedures, and data warehouse solutions.

Develop and support cloud-based data lake architectures using Amazon S3 or equivalent cloud storage platforms.

Deploy and manage Spark workloads on EMR Serverless or other managed Spark environments.

Implement data quality checks, monitoring, and performance tuning across data pipelines.

Work closely with business stakeholders, analysts, and cross-functional teams to understand data requirements and deliver solutions.

Ensure adherence to data governance, security, and compliance standards.

Support real-time and batch processing use cases within enterprise data platforms.

Mandatory Skills

Strong hands-on experience with PySpark and Apache Spark internals.

Experience working with Databricks, including Jobs, Workflows, and Delta Lake.

Strong knowledge of Apache Airflow for workflow orchestration.

Advanced SQL skills and experience with Data Warehouse (DWH) systems.

Experience running Spark workloads on EMR Serverless or managed Spark platforms.

Hands-on experience with cloud data lakes using Amazon S3 or equivalent storage solutions.

Strong programming skills in Python.

Experience with Big Data technologies and distributed computing frameworks.

Desired Skills

Exposure to streaming frameworks such as:

Spark Structured Streaming

Apache Kafka

Experience with performance tuning and optimization of Spark applications.

Knowledge of CI/CD practices for data engineering deployments.

Familiarity with cloud-native data architectures and modern data platforms.

Preferred Experience

6–10 years of overall IT experience.

Experience in the Cards & Payments domain is preferred.

Strong understanding of data modeling, ETL frameworks, and data lake architectures.

Excellent problem-solving, analytical, and communication skills.

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Pay: Up to ₹1,900,000.00 per year

Application Question(s):

What is your total years of experience?

How many years of experience do you have with PySpark and Apache Spark?

What is your current CTC and expected CTC?

What is your notice period? Are you an immediate joiner?

Are you comfortable working from Pune, Mumbai, Chennai, or Bangalore in a hybrid work model?

Have you worked with EMR Serverless or any managed Spark platform?

How strong are your SQL skills? Have you worked with Data Warehousing solutions?

Can you explain your experience with Apache Airflow?

Work Location: In person