Lead Data Engineer - Microsoft Fabric

The Lead Data Engineer is responsible for designing, developing, and leading the implementation of scalable data engineering solutions using Microsoft Fabric. The ideal candidate should have strong expertise in modern data engineering practices, ETL/ELT architecture, cloud-based data platforms, data warehousing, and data governance. The role also involves providing technical leadership, ensuring adherence to best practices, and driving the successful delivery of enterprise data solutions.

Key Responsibilities

Lead the design and implementation of enterprise-scale data engineering solutions using Microsoft Fabric.

Design, develop, and optimize ETL/ELT pipelines for processing.

Architect and implement metadata-driven data ingestion and transformation frameworks.

Develop and maintain scalable PySpark notebooks for data processing.

Design and implement robust data warehousing solutions using Microsoft Fabric.

Provision and manage Microsoft Fabric workspaces, ensuring appropriate security, governance, and environment configuration.

Implement and manage data governance, cataloging, metadata management, and data lineage using Microsoft Purview.

Ensure data quality, security, governance, and compliance, including handling Personally Identifiable Information (PII).

Collaborate with business stakeholders, solution architects, and reporting teams to understand business requirements and translate them into technical solutions.

Optimize data pipelines for performance, scalability, reliability, and maintainability.

Support Power BI reporting by developing optimized data models and curated datasets.

Work closely with DevOps teams to establish CI/CD pipelines, deployment strategies, and release management processes.

Required Qualifications

Bachelor’s degree in computer science, Information Technology, Engineering, or a related discipline.

8+ years of experience in Data Engineering with at least 3–4 years of hands-on experience in Microsoft Fabric or similar cloud-based data platforms.

Proven experience in designing and implementing enterprise ETL/ELT solutions.

Strong understanding of modern data architecture and cloud-based analytics platforms.

Hands-on experience with Microsoft Purview for data governance, data cataloging, metadata management, and data lineage.

Experience provisioning and administering Microsoft Fabric workspaces.

Good understanding of metadata management concepts and their implementation in relational database systems.

Familiarity with DevOps methodologies, CI/CD processes, Git-based source control, and deployment automation.

Strong analytical, problem-solving, leadership, and communication skills.

Technical Skills

Microsoft Fabric

Microsoft Purview

Fabric Workspace Provisioning and Administration

Fabric Data Pipelines

PySpark Notebooks

Metadata-Driven Design

Metadata Management in Relational Databases

OneLake

Fabric Data Warehouse

Data Engineering and ETL/ELT Architecture

Data Modeling

SQL

Python

Power BI

PII Handling and Data Governance

Azure DevOps

Git

CI/CD Pipeline Implementation

Performance Optimization and Data Pipeline Monitoring

Pay: $120,000.00 - $150,000.00 per year

Experience:

Microsoft Fabric: 5 years (Required)

Data Engineer: 8 years (Required)

ETL: 6 years (Preferred)

Work Location: Remote