Hadoop Data Engineer

Hadoop Data Engineer responsible for designing, developing, and maintaining large-scale data processing systems within a distributed Hadoop ecosystem. The role focuses on enabling data-driven decision-making across banking operations, risk management, compliance, and customer analytics.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using Hadoop ecosystem tools (HDFS, Hive, Spark, Sqoop, Kafka).

Build and optimize ETL/ELT processes to support data ingestion from multiple banking systems.

Develop and manage big data solutions for structured and unstructured data.

Collaborate with data analysts, data scientists, and business stakeholders to deliver data solutions.

Ensure data quality, integrity, and governance aligned with banking and regulatory standards.

Perform performance tuning and optimization of Hadoop/Spark jobs.

Implement data security controls to comply with financial regulations (e.g., PCI, SOX).

Support real-time and batch data processing frameworks.

Troubleshoot production issues and provide continuous support for data platforms.

Work with cloud platforms (e.g., AWS, Azure) for modern data solutions.

Required Skills & Qualifications

Technical Skills

Strong experience with:

Hadoop ecosystem (HDFS, MapReduce, Hive, HBase)

Apache Spark (Scala/Python)

SQL & NoSQL databases

ETL tools (Informatica, Talend, or similar)

Kafka or other streaming tools

Proficiency in programming:

Python / Java / Scala

Experience with:

Data warehousing concepts

Workflow orchestration tools (Airflow, Oozie)

Unix/Linux environments

Knowledge of cloud data platforms (AWS EMR, Azure Data Lake) is a plus

Domain Knowledge

Understanding of banking and financial services data

Exposure to risk, compliance, or fraud analytics is preferred

Soft Skills

Strong problem-solving and analytical abilities

Excellent communication and collaboration skills

Ability to work in Agile/Scrum environments

Education & Experience

Bachelor’s or Master’s degree in:

Computer Science, Information Technology, or related field

Typically 5–10 years of experience in data engineering or big data development