jobapplier

P1 - GP - Senior Data Engineer / Data Architect

atlas

Senior Data Engineer / Data Architect About the role We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks.

Responsibilities

  • The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system.
  • You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications.
  • You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation.
  • build incremental loads from the source platform into Bronze using change timestamps and watermarks, with safe handling of late-arriving rows and full retention of raw history.
  • implement a source-agnostic business model of dimensions, fact tables with different history rules and reference data, built as performant Delta tables for Power BI and application use.
  • support a single cutover from the legacy system, and load and freeze the history of companies that will not move.
  • work with the data model owner on the item catalog, dimensions (period, currency, fund and entity), cash transactions and derived measures such as multiples and FX conversion.
  • build checks and quality flags, reconcile new figures against the legacy system before the cutover and run regression checks when the model changes.
  • publish Gold to downstream consumers, including Lakebase (Postgres) synced tables for applications and direct Delta access for BI.
  • orchestrate and monitor the pipelines, manage access in Unity Catalog, and keep documentation, lineage and runbooks current.
  • work closely with the vendor's data team, reporting and application teams, and finance and portfolio operations users on definitions and issues.
  • facts at a clear grain, dimensions, surrogate keys, slowly changing dimensions.
  • A track record of data quality work: deduplication, reconciliation, and tracing a wrong number back to its source.
  • Experience with Unity Catalog or a comparable governance layer: permissions, lineage, documentation.
  • Performance tuning for BI workloads, such as file layout, clustering and query plans.
  • Clear written communication, including explaining data issues and trade-offs to non-technical users.
  • Nice to have Private equity, asset management or fund administration data: portfolio monitoring, valuations, cash flows, fund and co-invest structures.
  • LTM and YTD periods, fiscal year ends, budgets and forecasts, restatements, currency conversion.
  • Databricks Lakebase or PostgreSQL, including serving data to applications.
  • Our environment Area Tools Lakehouse Databricks, Delta Lake, Unity Catalog Languages Spark SQL, Python, PySpark, T-SQL Application serving Databricks Lakebase (Postgres) synced tables Legacy and transition Azure SQL Database Reporting Power BI Cloud Microsoft Azure

Requirements

  • model source data close to its native structure, with deterministic deduplication, type-aware unit conversion, sign rules and quarantine of suspect rows rather than silent fixes.
  • What you'll bring 5+ years building production data pipelines, including 2+ years on Databricks with Delta Lake.
  • SQL (Spark SQL and T-SQL) and Python or PySpark.
  • Hands-on experience with a medallion (Bronze, Silver, Gold) architecture and incremental processing: watermarks, MERGE, change data feed.
  • Lakeflow (Delta Live Tables) or Databricks Workflows orchestration.
  • Azure SQL and migrating SQL Server logic to Databricks.
  • Power BI data modeling, or working closely with BI developers.
  • Infrastructure as code and CI/CD for data pipelines, for example Databricks Asset Bundles.
Easy Apply