Data Engineer (Snowflake & AWS)

Job Summary

We are looking for an experienced Data BAU Lead – Data Engineering to lead data operations, production support, and continuous improvement initiatives within a cloud-first data environment. The ideal candidate will have strong technical leadership capabilities, hands-on expertise in AWS, Snowflake, Data Engineering, and BAU operations, with proven experience managing large-scale data platforms, production incidents, and data transformation initiatives.

The role requires a strong understanding of modern data architecture, cloud technologies, ETL pipelines, data lakes, data warehouses, and the ability to lead engineering teams while collaborating with business stakeholders, architects, data scientists, and DevOps teams.

Key Responsibilities

Data Engineering & Platform Management

Lead the Data Engineering BAU team responsible for daily operations, production support, incident management, and platform stability.

Design, develop, and maintain scalable data pipelines and data processing solutions.

Develop tools and frameworks to improve data movement between internal/external systems and enterprise data platforms.

Build robust and reusable data ingestion pipelines to collect, cleanse, transform, harmonize, and consolidate data from multiple sources.

Support and enhance existing data applications, infrastructure, and architecture.

Develop and maintain datasets, data models, and data management processes.

Improve data quality, reliability, performance, and operational efficiency.

Cloud & Data Platform Migration

Support migration of existing data transformation workloads from Oracle and MS SQL environments to Snowflake.

Lead migration of legacy transformation processes from Oracle, Hive, and Impala into modern cloud-based solutions using Spark, Python, and AWS Glue.

Design and optimize cloud-based data solutions using AWS services.

Evaluate and recommend data technologies, platforms, and tools aligned with business strategy.

Production Support & BAU Operations

Manage critical production support activities, ensuring timely resolution of incidents and operational issues.

Establish and improve BAU processes, monitoring, alerting, and operational controls.

Collaborate with DevOps teams to enhance CI/CD deployment processes and system monitoring.

Ensure proper documentation of processes, workflows, technical solutions, and operational procedures.

Stakeholder & Team Leadership

Work closely with business stakeholders to understand data requirements, availability, scalability, and accessibility needs.

Lead requirement analysis and deliver effective data solutions.

Collaborate with Data Scientists, Solution Architects, Engineers, and Business Teams on analytics initiatives.

Mentor and guide data engineers, promoting coding best practices, design principles, and engineering excellence.

Manage stakeholder communication including senior leadership updates when required.

Required Technical Skills

Cloud & Data Platforms

Strong hands-on experience with AWS Cloud Data Services, including:

Amazon S3

AWS Glue

AWS DMS

AWS MWAA (Managed Workflows for Apache Airflow)

IAM

Amazon RDS

Amazon Kinesis

AWS Lambda

AWS Step Functions

Strong understanding of cloud architecture design and optimization techniques.

Extensive knowledge of Snowflake architecture, performance optimization, and implementation practices.

Experience working with enterprise-scale Data Lakes and Data Warehouses.

Data Engineering Skills

Strong programming experience in:

SQL

Python

PySpark

Unix/Linux Shell Scripting

Strong experience with:

ETL/ELT development

Data ingestion frameworks

Data integration

Data transformation

Data modelling

Data quality management

Good understanding of distributed systems and scalable data processing.

Experience with data streaming technologies is preferred.

Workflow & DevOps

Strong knowledge of:

Apache Airflow / AWS MWAA

CI/CD pipelines

Git-based deployments

Agile development methodologies

SDLC lifecycle

Experience & Qualification Requirements

8+ years of experience in Data Engineering.

5+ years of recent hands-on coding experience as a Lead Engineer managing production support and BAU operations.

Minimum 2+ years of experience with large-scale datasets, Data Lakes, and Data Warehouse technologies.

Strong preference for hands-on experience with Snowflake.

Minimum 3+ years of experience with AWS Data Engineering services including AWS Glue, S3, RDS, Lambda, Kinesis, Step Functions, and Airflow.

Bachelor's degree in Computer Science, Information Technology, Engineering, or related STEM discipline.

Experience working in Agile, dynamic, customer-focused environments.