Senior Data Engineer

Job Title: Senior Data Engineer – GenAI Data Platforms

Job Summary

We are seeking a highly skilled Senior Data Engineer to architect and build next-generation Generative AI (GenAI) data platforms that power Large Language Model (LLM)-based applications, AI copilots, and Retrieval-Augmented Generation (RAG) systems. The ideal candidate will have extensive experience in modern data engineering, distributed data processing, Azure cloud technologies, and production-grade machine learning pipelines.

This role requires hands-on expertise in designing scalable data platforms capable of processing structured, unstructured, and multi-modal datasets while enabling advanced AI capabilities such as vector search, embeddings, semantic retrieval, and intelligent agent workflows.

Key ResponsibilitiesGenAI Data Platform Development

Design, build, and maintain scalable GenAI-ready data platforms for enterprise AI applications.

Develop robust pipelines for structured, semi-structured, unstructured, and multi-modal data.

Build production-ready Retrieval-Augmented Generation (RAG) pipelines, including document ingestion, chunking, embedding generation, indexing, retrieval, and prompt orchestration.

Develop AI data services that support LLM-based applications, enterprise search, and AI copilots.

Optimize vector indexing and semantic search performance for low-latency AI applications.

Machine Learning Platform Engineering

Own the end-to-end Machine Learning lifecycle, including:

Data ingestion

Data preprocessing

Feature engineering

Model training

Model evaluation

Deployment

Monitoring

Retraining

Rollback strategies

Build production-grade ML pipelines using Azure-native services with automated CI/CD.

Implement experiment tracking, model versioning, and model governance using Azure Machine Learning and MLflow.

Data Engineering

Design and implement scalable batch and real-time data pipelines using Apache Spark, PySpark, and Azure Databricks.

Build streaming architectures using Apache Kafka and Spark Structured Streaming.

Process large-scale datasets including:

JSON

XML

PDFs

Documents

Images

Audio

Application logs

Multi-modal enterprise data

Develop high-performance ETL/ELT pipelines optimized for scalability and reliability.

Ensure data quality, lineage, governance, and monitoring across all pipelines.

Generative AI & Agentic AI

Design and deploy enterprise-grade GenAI applications using Azure OpenAI Service.

Develop embedding pipelines using Azure OpenAI embedding models and other embedding frameworks.

Build intelligent Agentic AI workflows with:

Multi-step reasoning

Tool calling

Function orchestration

Memory management

Guardrails

Observability

Reliability engineering

Cost optimization

Implement prompt engineering and prompt management strategies.

Vector Databases & Search

Design and manage enterprise vector search systems using:

Azure AI Search

Pinecone

FAISS

ChromaDB

Weaviate (Preferred)

Optimize semantic retrieval performance.

Develop hybrid search architectures combining keyword and vector search.

Cloud & DevOps

Build cloud-native architectures on Microsoft Azure.

Implement Infrastructure as Code (IaC) using Terraform or Bicep (preferred).

Develop CI/CD pipelines using Azure DevOps or GitHub Actions.

Automate deployment, monitoring, rollback, and infrastructure provisioning.

Ensure platform security, scalability, reliability, and cost optimization.

Required Technical SkillsProgramming Languages

Python

PySpark

SQL

Scala (Preferred)

Big Data Technologies

Apache Spark

Spark Structured Streaming

Apache Kafka

Apache Airflow

Delta Lake

Azure Databricks

Cloud Technologies (Azure Preferred)

Azure Data Factory

Azure Databricks

Azure Machine Learning

Azure OpenAI Service

Azure AI Search

Azure Blob Storage

Azure Data Lake Storage Gen2

Azure Event Hubs

Azure Key Vault

Azure Monitor

Azure Functions

Machine Learning & MLOps

Azure Machine Learning

MLflow

Model Registry

Experiment Tracking

CI/CD for ML

Model Monitoring

Feature Engineering

Model Deployment

Feature Stores

Generative AI

Large Language Models (LLMs)

Retrieval-Augmented Generation (RAG)

Embeddings

Prompt Engineering

Semantic Search

AI Copilots

Agentic AI

Function Calling

LangChain

LangGraph (Preferred)

LlamaIndex

Vector Databases

Vector Search Technologies

Azure AI Search

Pinecone

FAISS

ChromaDB

Milvus (Preferred)

Weaviate (Preferred)

Data Engineering

ETL / ELT

Data Lake Architecture

Data Warehousing

Data Governance

Data Quality

Metadata Management

Batch Processing

Streaming Pipelines

DevOps & Version Control

Git

Azure DevOps

GitHub Actions

Docker

Kubernetes (Preferred)

Terraform (Preferred)

Required Qualifications

Bachelor's degree in Computer Science, Information Technology, Data Engineering, Software Engineering, or a related field.

Master's degree in Data Science, Artificial Intelligence, or Computer Science is preferred.

Microsoft Azure Data Engineer Associate or Azure AI Engineer certification is an advantage.

Experience

6–8 years of experience in Data Engineering.

Strong expertise in Python, PySpark, SQL, Apache Spark, Kafka, and Airflow.

Hands-on experience with Azure cloud services.

Experience building scalable enterprise data platforms.

Practical experience working with unstructured and multi-modal data (JSON, logs, PDFs, images, audio, and documents).

Proven experience developing RAG pipelines, vector search systems, and embedding workflows.

Exposure to production-grade LLM and Generative AI solutions.

Experience with MLOps, ML lifecycle management, and Azure Machine Learning.

Preferred Skills

Experience with LangChain, LangGraph, or LlamaIndex.

Familiarity with Hugging Face Transformers and open-source LLMs.

Knowledge of Kubernetes-based AI deployments.

Experience implementing observability for AI systems.

Understanding of Responsible AI principles, governance, and AI security.

Exposure to distributed systems and microservices architecture.

Key Competencies

Strong software engineering fundamentals.

Advanced problem-solving and analytical thinking.

Ability to design highly scalable distributed systems.

Excellent communication and stakeholder management skills.

Strong understanding of cloud architecture and AI infrastructure.

Passion for innovation in Artificial Intelligence and Generative AI.

Key Performance Indicators (KPIs)

Reliability and scalability of data pipelines.

Pipeline execution success rate and latency.

Data quality and governance compliance.

Performance of RAG and vector search systems.

ML deployment frequency and model reliability.

Infrastructure uptime and cost optimization.

AI application response accuracy and retrieval relevance.

Reduction in operational overhead through automation.

Why Join Us?

Build cutting-edge GenAI platforms powering enterprise AI products.

Work with Azure OpenAI, RAG, vector databases, and Agentic AI technologies.

Collaborate with experienced AI engineers, data scientists, and cloud architects.

Drive innovation in next-generation AI applications.

Competitive compensation, continuous learning opportunities, and career growth in AI and cloud engineering.

Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)