Lead Data Engineer

Responsibilities

• Architect, build, and optimize scalable data pipelines using Databricks, PySpark, and Python
• Write and review complex SQL queries for data transformation, validation, and performance tuning
• Design and maintain data models and ETL/ELT workflows across Databricks environments
• Support or lead development of ML models/pipelines for data products (feature engineering, model training, deployment, and monitoring)
• Build data product applications using AI-assisted "vibe coding" tools (Claude Code, Cursor, or similar) to rapidly prototype, develop, and ship internal tools, dashboards, and data-driven applications
• Establish and enforce best practices for code quality, version control, and CI/CD using Git
• Champion the adoption of AI-assisted development tools across the team to improve developer velocity, code review efficiency, and documentation
• Conduct code reviews, architecture reviews, and technical design sessions
• Own and maintain the infrastructure underlying data products — compute clusters, transactional databases, job orchestration, environments, deployment pipelines, and monitoring
• Ensure data pipeline reliability, observability, and cost optimization across Databricks
• Manage access controls, environment configuration, and platform scaling for data product apps
• Troubleshoot infrastructure issues end-to-end (compute, storage, networking, connectivity)
• Lead, mentor, and grow a team of 4 engineers, providing technical guidance and career development support
• Conduct 1:1s, performance reviews, and skill-development planning for team members
• Foster a culture of continuous learning and rapid upskilling in response to evolving tech/business needs
• Encourage and enable the team to adopt new tools and frameworks quickly
• Own sprint planning, backlog grooming, and sprint retrospectives for the team (Agile/Scrum)
• Break down business requirements into well-scoped technical tasks and user stories
• Track sprint velocity, manage blockers, and ensure timely delivery of data engineering initiatives
• Coordinate with cross-functional stakeholders (analytics, product, business teams) to align priorities

Requirements

• 8+ years of experience in data engineering, with at least 2+ years in a technical leadership or team lead role
• Strong hands-on expertise in Databricks (notebooks, clusters, job orchestration, Delta Lake)
• Proficiency in PySpark and Python for large-scale data processing
• Advanced SQL skills (query optimization, stored procedures, performance tuning)
• Experience with Microsoft SQL Server (T-SQL, database design, troubleshooting)
• Experience with OCR data extraction (Azure Document Intelligence, AWS Textract)
• Solid understanding of Git workflows (branching strategies, PR reviews, merge conflict resolution)
• Hands-on experience building applications using AI coding tools / vibe coding (Claude Code, Cursor, GitHub Copilot, or similar)
• Experience maintaining and troubleshooting infrastructure supporting data products (compute, orchestration, deployment)
• Working knowledge of Machine Learning concepts and lifecycle (model training, deployment, monitoring)
• Proven experience running Agile sprint planning, backlog management, and team ceremonies
• Experience managing and mentoring a small engineering team (3-5 members)
• Demonstrated ability to quickly learn and adopt new technologies in response to business needs

Soft Skills

• Strong communication skills to bridge technical and non-technical stakeholders
• Ability to balance hands-on coding with people management responsibilities
• High adaptability — comfortable navigating ambiguity and fast-changing tech/business priorities
• Problem-solving mindset with a focus on scalable, maintainable solutions

Preferred

Experience with cloud platforms (Azure preferred, Databricks stack)

• Familiarity with CI/CD pipelines (Azure DevOps, GitHub Actions, etc.)
• Experience with data governance frameworks and data quality tools
• Bachelor's/Master's degree in Computer Science, Engineering, or related field