Lead Data Engineer

Designs, develops, implements and operates highly complex data pipelines, data products and data management solutions across modern cloud, warehouse and lakehouse platforms.

Responsibilities

• Applies strong software-engineering, data-modeling and distributed-processing practices to deliver solutions that are scalable, secure, resilient, observable and cost efficient.
• Independently resolves ambiguous engineering problems, contributes to technical direction and reusable standards, and mentors other engineers while partnering across product, architecture, governance, cybersecurity and platform teams.
• Lead the detailed engineering design, development, testing, deployment and production support of complex data pipelines and reusable data products.
• Develop maintainable data-processing and transformation solutions using advanced SQL and programming languages such as Python.
• Engineer solutions using Snowflake, Databricks, dbt and comparable modern data platforms; select appropriate warehouse, lakehouse or hybrid patterns based on workload needs.
• Design analytical storage solutions using relational databases, cloud object storage, columnar formats and open table formats such as Apache Iceberg.
• Implement batch, micro-batch, change data capture (CDC), event-driven and streaming patterns using technologies such as Apache Kafka or comparable platforms.
• Integrate data from relational and mainframe systems, APIs, event streams, SaaS platforms, files and object storage with appropriate security and operational controls.
• Apply normalized, dimensional, denormalized and data-product-oriented modeling approaches based on access, performance and lifecycle requirements.
• Snowflake, Databricks/Spark and SQL workloads for performance, scalability, reliability and cost, including partitioning, parallelism, data movement and compute utilization.
• Build data quality, reconciliation, metadata, lineage, observability, alerting and service-level controls into production solutions.
• Design for restartability, idempotency, retry, replay, recovery and failure isolation; lead diagnosis and root-cause analysis for complex production incidents.
• Apply source control, peer review, automated testing, CI/CD, environment promotion and infrastructure/configuration automation to data engineering delivery.
• Create reusable libraries, templates, implementation patterns and reference solutions that improve consistency and engineering velocity.
• Evaluate design alternatives and clearly document tradeoffs involving security, privacy, resiliency, maintainability, performance, cost and operational complexity.
• Contribute to technical standards, identify technical debt and recommend pragmatic modernization or remediation approaches.
• Mentor engineers through design reviews, code reviews, pairing and technical problem solving; influence engineering practices across the team and adjacent teams.
• Partner with product owners, architects, governance, cybersecurity, platform and consuming teams to translate business and data requirements into scalable solutions.
• Develop technical epics and stories, estimate complex work, surface dependencies and risks, and support iterative delivery through Agile practices.
• Provide technical leadership and mentorship without direct people-management responsibility.
• Understand and adhere to the Company’s risk and regulatory standards, policies and controls in accordance with the Company’s Risk Appetite.
• Design, implement, maintain and enhance internal controls to mitigate risk on an ongoing basis.
• Identify risk-related issues requiring escalation to management.
• Maintain internal control standards, including timely implementation of internal and external audit points and issues raised by external regulators, as applicable.
• Promote an environment that supports belonging and reflects the Company brand.
• Complete other related duties as assigned.

Requirements

• Associate’s degree and a minimum of 7 years’ systems analysis and/or application development work experience or Bachelor's degree and a minimum of 5 years' systems analysis and/or application development work experience.
• In lieu of a degree, a combined minimum of 9 year’s education and/or relevant work experience, including a minimum of 5 years’ system analysis and/or application development work experience.
• Expert proficiency in a minimum of 1 relevant programming language and professional proficiency in a minimum of 1 additional relevant programming language.
• Advanced SQL skills, including complex transformations, analytical queries and workload optimization.
• Experience with Python or another general-purpose programming language used for production data engineering.
• Hands-on engineering experience with Snowflake and/or Databricks.
• Experience with modern ELT/ETL and transformation practices, including dbt or a comparable framework.
• Experience with data modeling for warehouse, lakehouse and data-product use cases.
• Experience with distributed data processing using Apache Spark/PySpark or a comparable framework.
• Experience with Git-based development, automated testing, code review and CI/CD.
• Experience with batch and streaming architectures, orchestration and production operations.
• Knowledge of cloud computing and object-storage concepts, preferably in Microsoft Azure.
• Experience working with structured and semi-structured data using formats such as Parquet, JSON, Avro, CSV and XML.
• Demonstrated ability to independently design and deliver complex data solutions spanning ingestion, processing, persistence, quality and consumption.
• Demonstrated ability to troubleshoot complex distributed-processing, data-quality and production reliability issues.
• Demonstrated ability to balance performance, cost, scalability, resiliency, security and maintainability in technical decisions.
• Demonstrated ability to review and improve technical designs and code developed by other engineers.
• Demonstrated ability to communicate complex designs and tradeoffs to engineering, architecture, product and business audiences.

Preferred

• Advanced production experience with Snowflake, Databricks, Apache Spark/PySpark and dbt.
• Experience implementing Apache Iceberg or another open table format, including schema evolution, partitioning, snapshots and table maintenance.
• Experience with Apache Kafka, CDC and event-driven architectures.
• Experience with orchestration technologies such as Apache Airflow, Azure Data Factory, Databricks Workflows or dbt orchestration.
• Experience with Infrastructure as Code and configuration automation, including Terraform or comparable tooling.
• Experience with data catalogs, metadata management, lineage, data observability and automated data-quality frameworks.
• Experience building governed enterprise data products or domain-oriented data solutions.
• Experience working in a highly regulated environment such as financial services.
• Master’s degree in Computer Science, Engineering, Data Engineering, Information Systems or a related technical field.

Pay

• M&T Bank is committed to fair, competitive, and market-informed pay for our employees.
• The pay range for this position is $116,400.00 - $194,000.00 Annual (USD).
• The successful candidate’s particular combination of knowledge, skills, and experience will inform their specific compensation.

Location

Buffalo, New York, United States of America