jobapplier

IN_Sr. Associate _Azure Databricks Engineer_D&A _Advisory _Pune

PwC Asia

This job is closed: the employer no longer lists it.

Skills

Required

  • Databricks
  • PySpark
  • Python
  • data processing
  • Azure
  • Data Lake Services
  • ADLS
  • Data Lake architecture
  • data engineering
  • analytics
  • Unity
  • Catalog
  • data governance
  • Fabric

Preferred

  • Handson
  • Datalake
  • Data Lake and Lakehouse architectures
  • data storage
  • Medalion Architecture
  • Delta Format
  • Data Processing and Transformation
  • lifecycle
  • Data Management

Responsibilities

  • Key Responsibilities: 1.

Data Lake and Lakehouse Implementation

  • o Design, implement, and manage Data Lake and Lakehouse architectures.
  • (Must have) o Develop and maintain scalable data pipelines and workflows.
  • (Must have) o Utilize Azure Data Lake Services (ADLS) for data storage and management.
  • (Must have) o Knowledge on Medalion Architecture, Delta Format.
  • (Must have) 2.

Data Processing and Transformation

  • o Use PySpark for data processing and transformations.
  • (Must have) o Implement Delta Live Tables for real-time data processing and analytics.
  • (Good to have) o Ensure data quality and consistency across all stages of the data lifecycle.
  • (Must have) 3.

Data Management and Governance

  • o Employ Unity Catalog for data governance and metadata management.
  • (Good to have) o Ensure robust data security and compliance with industry standards.
  • (Must have) 4.

Data Integration

  • o Extract, transform, and load (ETL) data from multiple sources (Must have) including SAP (Good to have), Dynamics 365 (Good to have), and other systems. o Utilize Azure Data Factory (ADF) and Synapse Analytics for data integration and orchestration.
  • (Must have) o Performance Optimization of the Jobs.
  • (Must have) 5.

Data Storage and Access

  • o Implement and manage Azure Data Lake Storage (ADLS) for large-scale data storage.
  • (Must have) o Optimize data storage and retrieval processes for performance and cost-efficiency.
  • (Must have) 6.

Collaboration and Communication

  • o Work closely with data scientists, analysts, and other stakeholders to understand data requirements.
  • (Must have) o Provide technical guidance and mentorship to junior team members.
  • (Good to have) 7.

Continuous Improvement

  • o Stay updated with the latest industry trends and technologies in data engineering and cloud computing.
  • (Good to have) o Continuously improve data processes and infrastructure for efficiency and scalability.
  • (Must have)

Requirements

  • Mandatory skill sets
  • Technical Skills: o Proficient in PySpark and Python for data processing and analysis. o Strong experience with Azure Data Lake Services (ADLS) and Data Lake architecture. o Hands-on experience with Databricks for data engineering and analytics. o Knowledge of Unity Catalog for data governance. o Expertise in Delta Live Tables for real-time data processing. o Familiarity with Azure Fabric for data integration and orchestration. o Proficient in Azure Data Factory (ADF) and Synapse Analytics for ETL and data warehousing. o Experience in pulling data from multiple sources like SAP, Dynamics 365, and others. · Soft Skills: o Excellent problem-solving and analytical skills. o Strong communication and collaboration abilities. o Ability to work independently and as part of a team. o Attention to detail and commitment to data accuracy and quality.

Certifications required

  • Certification in Azure Data Engineering or relevant Azure certifications. o DP203 (Must have) · Certification in Databricks. o Databricks certified Data Engineer Associate (Must have) o Databricks certified Data Engineer Professional (Good Have) Years of experience required: 4 to 7 Years
  • Educational Qualification B.E. / / MCA/ M.E/ MBA/ PGDM.
  • All qualifications should be in regular full-time mode with no extension of course duration due to backlogs.

Preferred

  • Preferred skill sets
  • Handson ETL / Datalake is good to have..

Job Position Title

  • IN_Sr.
  • Associate _Azure Databricks Engineer_D&A _Advisory _Pune Responsibilities: