Senior Data Engineer - 20 months contract

We are currently building a cutting-edge, large-scale environmental data platform. We are looking for a talented Senior Data Engineer to help us drive system resiliency, optimize CI/CD pipelines, and implement robust Site Reliability Engineering (SRE) practices across our cross-agency initiatives.

What You Will Do

As a Senior Data Engineer, you will bridge the gap between complex business requirements and highly resilient technical execution. Your core responsibilities will fall into three key pillars:

1. Data Pipeline Development & Architecture

Architect & Build: Design and build robust batch and real-time ingestion pipelines to collect, clean, merge, and harmonise diverse data streams from various source systems.

Translate Requirements: Partner with business users to translate their functional needs into scalable technical specifications, building reusable data models for end-users.

Innovate: Research and propose cutting-edge technologies and processes to continuously upgrade our core data infrastructure.

Secure Access: Design secure access mechanisms to ensure end-users and integrated systems can safely retrieve data assets.

2. Data Quality & Automated Testing

Enforce Integrity: Implement end-to-end data quality checks, validation, and reconciliation processes across all pipeline stages.

Automate Testing: Create and maintain automated test suites (covering functional, regression, and exploratory testing) integrated with CI/CD pipelines to catch bugs before they hit production.

Support UAT: Coordinate with stakeholders to facilitate smooth User Acceptance Testing (UAT), ensuring data products fully meet business requirements.

3. Operations, Governance & SRE

Optimize Operations: Conduct day-to-day monitoring, performance tuning, capacity planning, and issue diagnosis of databases and ETL systems.

Data Governance: Collaborate with data stewards to maintain data catalogues, lineage, metadata, and enforce strict security measures (encryption, masking, access control).

Drive DevOps: Champion CI/CD and Site Reliability Engineering (SRE) practices to ensure maximum system uptime and delivery efficiency.

What We Are Looking ForQualifications & Technical Skills

Education: Bachelor’s Degree in Computer Science, Software Engineering, Information Technology, or a related discipline.

Core Fundamentals: Deep understanding of system design, data structures, algorithms, data access, and data modeling.

Programming & SQL: Highly proficient in Python, Shell Scripting, and SQL.

Data Tools & Cloud:

Experience with Databricks and orchestration frameworks (e.g., Airflow, Azure Data Factory).

Hands-on experience with major cloud providers (AWS, Azure, or GCP).

DevOps & QA:

Familiarity with DevOps tools such as Docker, Git, and Terraform.

Experience with testing automation, CI/CD integration, and defect tracking tools (like JIRA or equivalent).

Added Advantages (Nice-to-Haves)

Experience working within public sector frameworks and government systems relating to data security/governance policies.

Domain knowledge or prior experience working with Climate and Weather domains.

Pay: $8,000.00 - $9,000.00 per month

Work Location: In person