Senior Data Engineer

Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time data processing.
• Architect end-to-end ETL/ELT solutions for data ingestion, transformation, and consumption across enterprise systems.
• Build and optimize cloud-native data platforms using AWS services such as S3, EMR, Redshift, Glue, Athena, and Airflow.
• Define and implement enterprise data strategies, including data sourcing, data flow, storage, governance, and consumption frameworks.
• Collaborate with Data Analytics, Business Intelligence, Product, and Technology teams to understand and fulfill data requirements.
• Design and implement scalable data models supporting reporting, analytics, and business intelligence initiatives.
• Monitor, troubleshoot, and support production data pipelines to ensure high availability and reliability.
• Lead performance tuning initiatives for complex SQL queries, ETL jobs, and large-scale data processing workloads.
• Ensure data quality, validation, governance, security, and compliance standards are adhered to.
• Drive cloud cost optimization initiatives while maintaining performance and user experience.
• Present technical architecture, solutions, and recommendations to business and technology stakeholders.
• Mentor junior engineers and coordinate tasks across project teams.
• Prepare and maintain technical documentation, architecture diagrams, and operational runbooks.

Requirements

• Technical Skills
• Data Engineering & Data Warehousing
• Enterprise Data Warehousing Concepts
• Data Lake & Data Lakehouse Architecture
• ETL/ELT Design and Development
• Data Integration and Data Migration
• Data Quality & Validation Frameworks
• Programming & Big Data
• Python
• Scala
• Apache Spark (PySpark/Spark SQL)
• SQL Query Optimization
• Distributed Data Processing
• AWS Data Stack
• Amazon S3
• Amazon EMR
• Amazon Redshift
• AWS Glue
• Amazon Athena
• Apache Airflow
• IAM & AWS Security Best Practices
• Data Modeling & Architecture
• Dimensional Data Modeling
• Star Schema & Snowflake Schema
• Data Architecture Design
• Metadata Management
• Data Governance Frameworks
• Scheduling & Orchestration
• Apache Airflow
• Enterprise Job Scheduling Frameworks
• Workflow Automation
• Monitoring & Production Support
• Data Pipeline Monitoring
• Incident Management
• Root Cause Analysis
• SLA Management
• Security
• Data Security Principles
• Data Encryption & Access Controls
• Regulatory and Compliance Awareness

Experience Requirements

• Minimum 7+ years of overall Data Engineering experience .
• Strong experience in Enterprise Data Warehousing (7+ years) .
• Hands-on expertise in building scalable ETL pipelines using Spark, Scala, and Python (7+ years) .
• Strong SQL development and performance tuning experience.
• Experience in Data Modeling, Data Architecture, and Data System Design.
• Extensive experience working with AWS Data Services including EMR, Redshift, S3, Athena, Glue, and Airflow.
• Experience supporting production environments and handling critical incidents.

Leadership & Soft Skills

• Experience leading and mentoring data engineering teams.
• Strong stakeholder management and communication skills.
• Ability to present complex technical solutions to leadership and cross-functional teams.
• Strong analytical and problem-solving capabilities.
• Excellent documentation and technical writing skills.
• Ability to manage multiple priorities and deliver under tight timelines.

Preferred

• Exposure to banking, financial services, fintech, or large enterprise environments preferred.

Preferred Qualifications

• BE/B.Tech/M.Tech from reputed Tier-1 Institutes.
• AWS Certifications (Solutions Architect, Data Engineer, Developer Associate, etc.).
• Certifications in Spark, Big Data, or Cloud Technologies will be an added advantage.