Senior Data Engineer – GenAI & Unstructured Data Pipelines

Job Title: Senior Data Engineer – GenAI & Unstructured Data Pipelines

Experience: 6–8 Years Location: Offshore (2 Positions) Employment Type: Full-Time

Job Summary

We are seeking an experienced Senior Data Engineer to build and optimize next-generation data platforms that power Generative AI and Large Language Model (LLM) applications. The ideal candidate will have strong expertise in designing scalable data pipelines for structured, unstructured, and multi-modal data while enabling Retrieval-Augmented Generation (RAG), vector search, embeddings, and AI-powered copilots.

This role requires hands-on experience in modern data engineering, Azure cloud technologies, MLOps, and GenAI workflows to develop production-grade AI data platforms and intelligent applications.

Key Responsibilities

Design, develop, and maintain scalable data pipelines for structured, unstructured, and multi-modal datasets.

Own the end-to-end machine learning lifecycle, including data ingestion, feature engineering, model training, evaluation, deployment, monitoring, retraining, and rollback.

Build and manage production-grade ML pipelines using Azure-native services with CI/CD automation and MLOps best practices.

Utilize Azure Machine Learning and MLflow for experiment tracking, model registry, versioning, and governed model deployment across development, testing, and production environments.

Design and implement Generative AI solutions using Azure OpenAI, embeddings, Retrieval-Augmented Generation (RAG), and vector search technologies.

Develop Agentic AI workflows with multi-step reasoning, tool integration, guardrails, observability, reliability, and cost optimization.

Build scalable batch and real-time data processing pipelines using Azure Databricks, Apache Spark, and Kafka.

Develop robust ingestion pipelines for text, documents, PDFs, logs, JSON, and other unstructured data sources.

Design and optimize RAG pipelines including document chunking, embedding generation, indexing, and semantic retrieval.

Implement and manage vector databases and vector search platforms such as Azure AI Search and Pinecone.

Collaborate with data scientists, AI engineers, and application teams to deliver scalable AI-ready data platforms.

Monitor pipeline performance, data quality, scalability, and system reliability while implementing continuous improvements.

Follow DevOps, MLOps, and Agile best practices for secure and automated deployments.

Required Skills & Qualifications

Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related field.

6–8 years of experience in Data Engineering with expertise in large-scale data processing.

Strong programming skills in Python, PySpark, and SQL.

Hands-on experience with:

Apache Spark

Apache Airflow

Apache Kafka

Azure Databricks

Experience building batch and streaming data pipelines.

Strong experience processing unstructured and semi-structured data including JSON, logs, documents, PDFs, and other text-based content.

Experience with cloud platforms, preferably Microsoft Azure.

Exposure to Azure Machine Learning, MLflow, and MLOps practices.

Hands-on experience with Generative AI concepts including:

Large Language Models (LLMs)

Retrieval-Augmented Generation (RAG)

Embeddings

Vector Databases

Semantic Search

Experience with vector search platforms such as Azure AI Search, Pinecone, or similar technologies.

Understanding of CI/CD pipelines, automation, monitoring, and production deployment strategies.

Strong analytical, problem-solving, and communication skills.

Preferred Qualifications

Experience with Azure OpenAI Service and AI-powered application development.

Knowledge of Agentic AI architectures and AI orchestration frameworks.

Experience working with multi-modal data pipelines and AI copilots.

Familiarity with data governance, security, and cloud architecture best practices.

Experience working in Agile development environments.

Azure Data Engineering, Databricks, or Azure AI certifications are a plus.

Preferred Technologies

Python

PySpark

SQL

Apache Spark

Apache Airflow

Apache Kafka

Azure Databricks

Azure Machine Learning

MLflow

Azure OpenAI

Azure AI Search

Pinecone

Vector Databases

Retrieval-Augmented Generation (RAG)

Embeddings

Large Language Models (LLMs)

CI/CD

Git

Docker

Azure DevOps

This role is ideal for data engineering professionals who are passionate about building scalable GenAI data platforms, enabling enterprise AI applications, and delivering production-ready data pipelines that power next-generation intelligent solutions.

Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)