Application Developer

Job Description

Application Developer

Job Location: Pune

Location Flexibility: Primary Location Only

Req Id: 10313

Posting Start Date: 7/20/26

Role- Application developer

Job Description - Data Engineer Consultant - Airflow / Astronomer Migration

Job Title: Data Engineer Consultant - Airflow / Astronomer Migration

Experience: 3 to 5 years

Location / Shift:

Location: Pune, India / Remote / Hybrid as per project need

Working Mode: Full-time assignment

Client working hours may apply

Role Summary:

You will work as a Data Engineer Consultant .

You will support the migration of orchestration workflows from Azure Data Factory (ADF) to Astronomer / Apache Airflow .

You will design, build, test, and document data workflows.

You will work closely with the Principal Data Engineer and the Data Engineering team.

You will also suggest improvements for performance, cost, reliability, and maintainability.

Primary Skills:

Must Have

Apache Airflow / Astronomer

Airflow DAG design and development

Python and SQL

dbt

Snowflake

Azure Blob Storage

Data pipeline orchestration

Workflow testing and validation

Secondary Skills:

Good to Have

Azure Data Factory (ADF)

ADF to Airflow migration experience

Data pipeline modernization experience

Snowflake performance tuning

dbt model and flow optimization

Git / Azure DevOps

CI/CD for data pipelines

Monitoring and alerting for data workflows

Key Responsibilities:

1) Requirement Understanding

Understand existing ADF pipelines and orchestration flows.

Analyze current workflow schedules, triggers, parameters, and dependencies.

Understand ingestion steps, transformation logic, and downstream impact.

Work with the Principal Data Engineer and team to clarify open points.

Identify risks, blockers, assumptions, and dependencies early.

2) Solution Design

Design the migration approach from ADF to Astronomer / Airflow .

Convert existing orchestration logic into clean Airflow DAG design.

Define DAG structure, task dependencies, retry logic, and schedule patterns.

Design workflows that are easy to maintain and support.

Suggest improvements instead of only doing one-to-one migration.

3) Development / Implementation

Design and build DAGs in Apache Airflow / Astronomer .

Develop and adapt data pipelines as per migration requirement.

Modify existing ingestion steps in Astronomer where required.

Modify existing dbt flows and transformations when needed.

Use Python and SQL for workflow logic, validation, and automation.

Follow coding standards and project guidelines.

4) Integration / Configuration

Configure schedules and dependencies for Airflow DAGs.

Integrate Airflow workflows with dbt , Snowflake , and Azure Blob Storage .

Ensure dbt jobs are properly triggered and monitored through Airflow.

Validate integration between ingestion, transformation, and consumption layers.

Configure workflow parameters, environment settings, and required connections.

5) Testing & Validation

Perform unit testing for DAGs and workflow components.

Perform integration testing for end-to-end data workflows.

Validate migrated workflows against existing ADF output wherever applicable.

Check data accuracy, completeness, and processing status.

Fix defects and retest before production readiness.

Prepare test evidence and validation notes.

6) Performance Optimization

Review existing workflows and identify improvement areas.

Optimize Airflow DAG performance and execution time.

Improve reliability using proper retries, failure handling, and dependency management.

Suggest cost-efficient execution patterns.

Improve maintainability by creating reusable workflow components.

Proactively recommend better design wherever useful.

7) Security, Compliance & Governance

Follow client security and data handling guidelines.

Use secure connection and access patterns as per project standards.

Avoid hardcoding secrets, passwords, or sensitive values.

Ensure access and workflow configurations are controlled and traceable.

Follow required governance process for data pipeline changes.

8) Deployment & Release Management

Support deployment of Airflow / Astronomer workflows across environments.

Prepare deployment steps and release notes.

Support production readiness checks before go-live.

Coordinate with the Data Engineering team during release activities.

Validate workflows after deployment.

Support rollback or quick fix approach if any deployment issue occurs.

9) Production Support & RCA

Monitor workflow execution and identify failures.

Debug failures using Airflow logs, task history, dbt logs, and Snowflake queries.

Perform root cause analysis for recurring issues.

Implement preventive fixes to improve workflow stability.

Support production issues during migration and stabilization phase.

10) Documentation & Knowledge Transfer

Prepare technical documentation for developed DAGs and workflows.

Document schedule, dependency, parameter, and configuration details.

Prepare development and testing documentation.

Maintain handover notes and support instructions.

Provide knowledge transfer to the Data Engineering team.

Explain design decisions, known issues, and support steps clearly.

11) Agile Delivery & Collaboration

Work closely with the Principal Data Engineer and Data Engineering team.

Provide regular status updates on progress, risks, and blockers.

Collaborate with team members in a structured and transparent way.

Take ownership of assigned tasks and deliver on time.

Work as a hands-on team reinforcement for the migration project.

Tools / Technologies:

Cloud / Platform: Azure

Orchestration Tools: Apache Airflow, Astronomer, Azure Data Factory

Data Transformation: dbt

Storage: Azure Blob Storage

Database / Warehouse: Snowflake

Programming Languages: Python, SQL

DevOps Tools: Git / Azure DevOps, if used in project

Monitoring Tools: Airflow UI, Astronomer monitoring, dbt logs, Snowflake query history

Documentation Tools: Confluence / SharePoint / Project documentation repository, as applicable

Qualification:

BE / BTech / MCA / MSc / BSc / BCA or equivalent practical experience

Relevant Data Engineering, Cloud, Snowflake, dbt, or Airflow certification is good to have

Soft Skills:

Autonomy and ownership

Curiosity to understand existing systems

Proactive mindset

Ability to suggest practical improvements

Good communication

Strong collaboration with team members

Ability to integrate smoothly with existing team

Good documentation discipline

Preferred Candidate Profile:

3 to 5 years of Data Engineering experience.

Strong hands-on experience in Airflow / Astronomer .

Good working knowledge of dbt, Snowflake, Azure Blob Storage, Python, and SQL .

Prior experience in orchestration migration or data pipeline modernization.

Ability to understand ADF workflows and redesign them in Airflow.

Good delivery ownership and clear communication.

Able to work independently with minimum supervision.

Relocation Supported: No

Visa Sponsorship Approved: No