IN_Senior Associate_Azure Data Engineer_D&A_Advisory_Ahmedabad
- Ahmedabad, India
- Mid Level
- 3+ years
- Master's degree
- Posted
Skills
Required
- Azure
- integration
- SQL
- Delta Lake
- Azure Data Factory
- Azure data services
- orchestration
- data governance
- CI/CD
- data engineering
- PySpark
- Spark
- Python
- Git
Preferred
- cloud
- analytics
- streaming
- real-time
- data processing
- Stream Analytics / Kafka
- machine learning pipeline support
- ML
- Docker
- Kubernetes
PwC India is looking for a highly skilled Azure Data Engineer Senior Associate with hands-on experience in designing, developing, and managing scalable data solutions on the Azure cloud platform, with particular expertise in Databricks. This is a technical role focused on building robust data pipelines, optimizing cloud data infrastructures, and enabling advanced analytics for enterprise clients.
Responsibilities
IN_Senior Associate_Azure Data Engineer_D&A_Advisory_Ahmedabad
- Design, develop, and optimize scalable data pipelines and ETL/ELT processes on Azure using Databricks, Azure Data Factory, and related services.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver high-quality data solutions.
- Build and manage data lakes, data warehouses, and lakehouse architectures leveraging Azure Synapse Analytics, Azure Data Lake Storage, and Databricks Delta Lake.
- Implement data ingestion, cleansing, transformation, and integration workflows from diverse data sources including SQL, NoSQL, and streaming platforms.
- Monitor and fine-tune data pipelines and Databricks clusters to ensure performance, cost-efficiency, and scalability.
- Apply data governance, security, and compliance best practices throughout data lifecycle management.
- Support continuous integration and continuous deployment (CI/CD) of data workflows and infrastructure as code (IaC).
- Prepare and maintain technical documentation including architecture diagrams, data flow designs, and operational guides.
- Stay updated with latest Azure data services, Databricks features, and industry trends to recommend innovative solutions.
Requirements
- Hands-on experience in designing and implementing data engineering solutions on Azure, specifically with Azure Databricks.
- Strong skills in PySpark, Spark SQL, and Delta Lake for building and optimizing distributed data pipelines.
- Proficiency in Azure Data Factory and other Azure data services for orchestration and integration.
- Solid understanding of data warehousing, lakehouse concepts, and big data architectures.
- Experience with scripting languages such as Python and SQL for data manipulation and automation.
- Knowledge of data governance, security, and compliance standards in cloud environments.
- Familiarity with version control systems (Git) and CI/CD tools like Azure DevOps.
- Strong analytical and problem-solving capabilities with attention to detail.
- 3 - 7 Years
- BE, B.Tech, MCA, MBA, M.Tech
Preferred
- Experience with real-time data processing and streaming technologies such as Azure Stream Analytics or Kafka.
- Exposure to machine learning pipeline support and integration with Azure ML or similar platforms.
- Working knowledge of containerization (Docker) and orchestration (Kubernetes) within cloud data environments.
- Prior consulting experience delivering cloud data modernization and analytics projects.
