P1 - GP - Senior Data Engineer / Data Architect
atlas
- Bengaluru, India
- On-site
- Lead/Staff
- Posted
Senior Data Engineer / Data Architect About the role We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks.
Responsibilities
- The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system.
- You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications.
- You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation.
- build incremental loads from the source platform into Bronze using change timestamps and watermarks, with safe handling of late-arriving rows and full retention of raw history.
- implement a source-agnostic business model of dimensions, fact tables with different history rules and reference data, built as performant Delta tables for Power BI and application use.
- support a single cutover from the legacy system, and load and freeze the history of companies that will not move.
- work with the data model owner on the item catalog, dimensions (period, currency, fund and entity), cash transactions and derived measures such as multiples and FX conversion.
- build checks and quality flags, reconcile new figures against the legacy system before the cutover and run regression checks when the model changes.
- publish Gold to downstream consumers, including Lakebase (Postgres) synced tables for applications and direct Delta access for BI.
- orchestrate and monitor the pipelines, manage access in Unity Catalog, and keep documentation, lineage and runbooks current.
- work closely with the vendor's data team, reporting and application teams, and finance and portfolio operations users on definitions and issues.
- facts at a clear grain, dimensions, surrogate keys, slowly changing dimensions.
- A track record of data quality work: deduplication, reconciliation, and tracing a wrong number back to its source.
- Experience with Unity Catalog or a comparable governance layer: permissions, lineage, documentation.
- Performance tuning for BI workloads, such as file layout, clustering and query plans.
- Clear written communication, including explaining data issues and trade-offs to non-technical users.
- Nice to have Private equity, asset management or fund administration data: portfolio monitoring, valuations, cash flows, fund and co-invest structures.
- LTM and YTD periods, fiscal year ends, budgets and forecasts, restatements, currency conversion.
- Databricks Lakebase or PostgreSQL, including serving data to applications.
- Our environment Area Tools Lakehouse Databricks, Delta Lake, Unity Catalog Languages Spark SQL, Python, PySpark, T-SQL Application serving Databricks Lakebase (Postgres) synced tables Legacy and transition Azure SQL Database Reporting Power BI Cloud Microsoft Azure
Requirements
- model source data close to its native structure, with deterministic deduplication, type-aware unit conversion, sign rules and quarantine of suspect rows rather than silent fixes.
- What you'll bring 5+ years building production data pipelines, including 2+ years on Databricks with Delta Lake.
- SQL (Spark SQL and T-SQL) and Python or PySpark.
- Hands-on experience with a medallion (Bronze, Silver, Gold) architecture and incremental processing: watermarks, MERGE, change data feed.
- Lakeflow (Delta Live Tables) or Databricks Workflows orchestration.
- Azure SQL and migrating SQL Server logic to Databricks.
- Power BI data modeling, or working closely with BI developers.
- Infrastructure as code and CI/CD for data pipelines, for example Databricks Asset Bundles.
