Senior Data Engineer (Databricks and Pyspark)
- Kuala Lumpur, Malaysia
- Senior Level
- 5+ years
- Posted
Skills
Required
- Databricks
- PySpark
- Delta Lake
- Azure Data Lake / ADF / Azure Functions / AWS
- Python
- APIs
- CI/CD
- cloud-based data pipelines
- Lakehouse/Medallion architectures
- Data Engineering
- SQL
- S3
- cloud storage
- Git
Preferred
- Gas / trading analytics
- SPARTA
- KPLER
- IIR
- OPEC
- Power BI / data visualization
- Terraform / Scala / PostgreSQL
- SharePoint development / .NET
- Medallion architecture
Position : Senior Data Engineer.
Responsibilities
- Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python.
- Implement Delta Lake and Medallion architecture for scalable enterprise data platforms.
- Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
- Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
- Optimize Spark jobs for performance, reliability, and cost efficiency.
- Implement data validation, quality checks, and monitoring with automated alerts and retries.
- Design secure and governed datasets using Unity Catalog and cloud security best practices.
- Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
- Provide technical leadership and guidance to junior team members.
Requirements
- Location : Malaysia and Singapore.
- Language : Mandarin Language must.
- We are looking for a Senior Data Engineer with strong expertise in Databricks, PySpark, Delta Lake, and cloud-based data pipelines. implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical leadership and hands-on architecture experience.
- 5–8+ years of experience in Data Engineering.
- Strong hands-on experience with Databricks, PySpark, Delta Lake, SQL, Python.
- Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents (S3, Lambda).
- Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage.
- Knowledge of ETL/ELT concepts, Lakehouse/Meddalion architecture, and distributed processing.
- Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines.
- Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).
Preferred
- Exposure to Oil & Gas or trading analytics (SPARTA, KPLER, IIR, OPEC).
- Knowledge of Power BI or data visualization concepts.
- Familiarity with Terraform, Scala, or PostgreSQL.
- Experience with SharePoint development or .NET (optional)
Pay
Salary: unknown currency 15,000–25,000
Senior Data Engineer (Databricks and Pyspark)
Unison Group · Kuala Lumpur, Malaysia · Full-time · Posted 2026-11-05
Role : Full Time Perm role + one month Bonus.