Senior Data Platform Engineer CR 12/11/26

• We are looking for a Senior Data Platform Engineer to help drive the evolution of our Azure-based Data Platform and Lakehouse architecture
• This role is focused on the platform itself. The professional will own and evolve the foundations that data product teams build on:

Responsibilities

• Apache Airflow orchestration (migrating from ADF)
• Azure resource management
• (Key Vault, identity and access, networking, storage), Azure DevOps CI/CD and Infrastructure as Code
• Databricks and Unity Catalog governance
• observability
• cost efficiency, while remaining
• and close to the teams that use the platform.
• Orchestration Platform (Apache Airflow & ADF)
• Own the

Apache Airflow

• platform: environments, deployment, upgrades, configuration, connections, variables, pools and DAG standards
• Work on the migration from Azure Data Factory to Airflow and the evolution of the
• Metadata-Driven Orchestration Framework
• (DAG generation, reusable operators, dependency and scheduling patterns).
• Ensure Airflow is
• reliable and observable: SLAs, alerting, retries, backfills, health monitoring and clear failure ownership.
• Azure Resource Management
• Manage Azure resources with
• Infrastructure as Code (Terraform): Databricks workspaces, Data Lake Storage, Data Factory, Key Vault, networking and supporting services.
• Own
• secrets and identity management: Key Vault, service principals and managed identities, secret scopes and rotation.
• Own
• access management: Azure RBAC, Entra ID groups, Unity Catalog grants and ownership, following least privilege, plus access requests and periodic reviews.
• Own
• Azure DevOps: repositories, CI/CD pipelines, branch policies and multi-environment promotion, including deployment validation and Terraform state.
• Track
• cost and governance
• of Azure resources: tagging, budgets, right-sizing and cost attribution (e.g. query tagging).

Hands-on Data Platform evolution

• Contribute to the implementation of the Data Catalog and Data Quality Framework and it’s lineage across the platform.
• observability: pipelines that are observable, debuggable and alert-driven, with clear ownership of failures.
• Operate the
• Databricks
• Unity Catalog
• structure, cluster and job policies, Auto Loader ingestion patterns, and performance and cost optimization of workloads.
• Participate in
• architecture reviews
• and technical decision-making, stating
• trade-offs
• (performance, cost, reliability, complexity) with data.

Define and promote

• engineering standards: naming conventions, patterns and guidelines, through code reviews and design reviews.
• Collaborate with Architects, Product Owners and data engineers in technical maturation of data products.
• Build and maintain
• developer tooling and automation, including AI-assisted engineering workflows (shared agent instructions, skills, automated PR review).
• Support & Operations (Run Team Collaboration)
• Provide
• second-line support
• to the Run/Operations team for complex production topics.

Troubleshoot and resolve

• platform incidents, failures and performance issues; lead root cause analysis and post-mortems.
• handover, documentation and operational readiness of platform changes.

runbooks, monitoring and operational processes

• Key Vault, managed identities and service principals, secret management
• Access management: Azure RBAC, Entra ID, Unity Catalog permissions
• Azure DevOps (repos, pipelines, service connections, environments) and CI/CD for data platforms
• Infrastructure as Code (Terraform) and multi-environment promotion
• Hands-on experience with Databricks and Unity Catalog (compute, jobs, governance)
• Observability and monitoring, alerting and incident management

Requirements

• 6+ years of experience in data engineering, with significant time on platform or infrastructure topics
• Strong technical expertise with Python and SQL (advanced), including PySpark and Delta Lake semantics
• Deep hands-on experience with Apache Airflow in production: DAG design, deployment, operations, upgrades, and orchestration of Databricks and ADF workloads

ADLS, networking and security basics, Azure Data Factory

• Solid understanding of Data Lakehouse architecture, Medallion architecture and metadata-driven architecture; awareness of cost management and performance
• Ability to work in cross-functional teams and to influence without authority
• Ownership mindset and attention to detail
• Good communication skills with both technical and business stakeholders: leads with the recommendation, quantifies, and is transparent about uncertainty