AI Data Engineer

Senior Data Engineer – GenAI & Unstructured Data Pipelines

Job Type: Full-time Experience: 6–8 Years Location: Offshore (Remote)

Job Summary

We are seeking an experienced Senior Data Engineer – GenAI & Unstructured Data Pipelines to build next-generation AI data platforms that power Large Language Model (LLM) applications. The ideal candidate will have strong expertise in designing scalable data pipelines for unstructured and multi-modal data while enabling Retrieval-Augmented Generation (RAG), embeddings, vector search, and AI copilots.

This role is ideal for professionals passionate about modern data engineering, cloud-native architectures, and Generative AI technologies.

Key Responsibilities

Design, build, and maintain scalable data pipelines for structured, unstructured, and multi-modal data.

Own and support the end-to-end machine learning lifecycle, including data ingestion, feature engineering, model training, evaluation, deployment, monitoring, retraining, and rollback.

Develop production-grade ML pipelines using Azure native services with CI/CD automation and best practices.

Utilize Azure Machine Learning and MLflow for experiment tracking, model registry, and governed deployments across environments.

Design and implement Generative AI solutions using Azure OpenAI, embeddings, vector search, and Retrieval-Augmented Generation (RAG).

Build Agentic AI workflows with multi-step reasoning, tool integration, observability, guardrails, reliability, and cost optimization.

Develop scalable batch and streaming data pipelines using Azure Databricks, Apache Spark, and Kafka.

Create efficient RAG pipelines, including document chunking, embedding generation, indexing, and retrieval workflows.

Design and manage vector databases and search solutions such as Azure AI Search and Pinecone.

Build high-performance batch and real-time data ingestion pipelines using Spark and Kafka.

Required Skills & Qualifications

6–8 years of experience in Data Engineering.

Strong hands-on expertise with:

Python

PySpark

SQL

Apache Spark

Apache Airflow

Apache Kafka

Experience processing and managing unstructured data, including JSON, logs, documents, and PDFs.

Strong experience with cloud platforms, preferably Microsoft Azure.

Hands-on exposure to Generative AI technologies, including:

Large Language Models (LLMs)

Retrieval-Augmented Generation (RAG)

Embeddings

Vector Databases

Azure OpenAI

Azure AI Search

Pinecone

Experience working with Azure Databricks and modern data engineering frameworks.

Knowledge of CI/CD pipelines, MLflow, and Azure Machine Learning is highly preferred.

Preferred Skills

Experience building AI copilots and Agentic AI applications.

Understanding of data governance, monitoring, and production-grade ML operations.

Excellent analytical, problem-solving, and communication skills.

Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)