Senior Data Engineer – GenAI & Unstructured Data Pipelines

Experience: 6–8 Years Employment Type: Full-time

Job Summary

We are seeking a highly skilled Senior Data Engineer – GenAI & Unstructured Data Pipelines to build and scale next-generation AI data platforms that power Large Language Model (LLM) applications and intelligent AI solutions. The ideal candidate will have strong expertise in data engineering, cloud technologies, and modern AI/ML architectures, with hands-on experience in designing pipelines for unstructured and multi-modal data.

You will play a key role in developing Retrieval-Augmented Generation (RAG) pipelines, vector search systems, AI copilots, and production-grade machine learning workflows while collaborating closely with data scientists, ML engineers, and application teams.

Key ResponsibilitiesData Engineering & Pipeline Development

Design, develop, and maintain scalable data pipelines for structured, semi-structured, and unstructured data, including documents, PDFs, logs, JSON, and multimedia content.

Build high-performance batch and real-time data ingestion pipelines using Apache Spark, PySpark, Kafka, and Azure Databricks.

Optimize data processing workflows for scalability, reliability, and performance.

Machine Learning & MLOps

Own the end-to-end machine learning lifecycle, including data ingestion, feature engineering, model training, evaluation, deployment, monitoring, retraining, and rollback.

Develop production-grade ML pipelines with CI/CD automation and governance.

Utilize Azure Machine Learning and MLflow for experiment tracking, model registry, and controlled deployments across development, testing, and production environments.

Generative AI & LLM Solutions

Design and implement Generative AI solutions using Azure OpenAI Services.

Build Retrieval-Augmented Generation (RAG) pipelines including document chunking, embeddings generation, indexing, retrieval, and response orchestration.

Develop AI copilots and intelligent applications powered by Large Language Models (LLMs).

Design agentic AI workflows with multi-step reasoning, tool integration, observability, guardrails, reliability, and cost optimization.

Vector Search & Knowledge Retrieval

Design and manage vector databases and semantic search solutions.

Implement embedding generation, similarity search, and retrieval workflows using Azure AI Search, Pinecone, or similar vector databases.

Optimize search relevance and retrieval performance for enterprise AI applications.

Cloud & Platform Engineering

Develop cloud-native data engineering solutions on Microsoft Azure.

Build scalable and secure architectures leveraging Azure Databricks, Azure Machine Learning, Azure OpenAI, and Azure AI Search.

Implement infrastructure automation and deployment best practices.

Collaboration & Best Practices

Work closely with Data Scientists, ML Engineers, AI Architects, and Business stakeholders to deliver enterprise AI solutions.

Follow software engineering best practices, code reviews, testing, and documentation standards.

Continuously evaluate emerging GenAI technologies and recommend improvements to the AI platform.

Required Skills & Qualifications

6–8 years of experience in Data Engineering.

Strong hands-on experience with:

Python

PySpark

SQL

Apache Spark

Apache Airflow

Apache Kafka

Experience handling unstructured data including JSON, documents, PDFs, logs, and text datasets.

Strong experience with Microsoft Azure cloud services.

Hands-on exposure to:

Large Language Models (LLMs)

Retrieval-Augmented Generation (RAG)

Embeddings

Vector Databases

Semantic Search

Experience building scalable batch and streaming data pipelines.

Good understanding of MLOps, CI/CD, and production deployment practices.

Preferred Qualifications

Experience with Azure Machine Learning and MLflow.

Hands-on knowledge of Azure OpenAI Services.

Experience with Azure AI Search, Pinecone, or other vector databases.

Familiarity with agentic AI frameworks and AI orchestration platforms.

Knowledge of cloud-native architecture, infrastructure automation, and DevOps practices.

Strong problem-solving, analytical, and communication skills.

Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)