Lead Software Engineer- Data Engineer

Lead – Data Platform About the Role We are seeking a highly skilled Software Engineering Lead – Data Platform to lead the development of Caterpillar’s next-generation Digital Manufacturing Data Platform.

About Cat (CaterpillarCareers)

• Technology, Digital and Data Job Description: Your Work Shapes the World at Caterpillar Inc.
• When you join Caterpillar, you're joining a global team who cares not just about the work we do – but also about each other.
• We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable communities.
• We don't just talk about progress and innovation here – we make it happen, with our customers, where we work and live.
• Together, we are building a better world, so we can all enjoy living in it.
• This platform enables large-scale data ingestion, transformation, and analytics across manufacturing, supply chain, and engineering ecosystems.

Responsibilities

• The ideal candidate will bring deep expertise in data engineering, large-scale data ingestion, AWS-based architectures, and Snowflake , along with strong leadership and software engineering discipline.
• Leadership & Delivery Lead and mentor a team of data engineers and platform developers

Agile execution and ensure predictable, high-quality delivery

• Establish engineering best practices, code quality, and CI/CD standards Data Platform & Architecture Architect scalable and secure data platforms on AWS Design robust data ingestion frameworks for batch and near real-time pipelines
• Define best practices in data modeling, governance, and metadata management Data Engineering & Ingestion Lead design and development of scalable ingestion pipelines (structured and unstructured data)
• Build and optimize Snowflake-based data platforms for performance and cost
• Enable ingestion of diverse sources (databases, APIs, files, streaming data)
• & Platform Engineering Leverage AWS services (S3, Glue, Lambda, EMR, Redshift, etc.) for end-to-end pipelines

Implement CI/CD pipelines using

• Azure DevOps / Jenkins Ensure system scalability, resiliency, and operational readiness
• Excellence Enforce software engineering principles (modular design, code quality, testing, version control)
• Drive automation and continuous improvement

Promote reusable frameworks for ingestion and transformation

• Collaboration Partner with product managers, SMEs, and business stakeholders
• Promote integration of AI capabilities into data engineering pipelines (e.g., GenAI, MCP, ATA).

Requirements

• Translate business needs into scalable data solutions Must-Have Skills 10+ years of experience in Data Engineering / Data Platform roles
• Strong experience in AWS data ecosystem (S3, Glue, Lambda, EMR, Redshift) Deep expertise in Snowflake (architecture, optimization, data modeling)
• Strong programming skills in Python and SQL Extensive experience with data ingestion pipelines and ETL/ELT frameworks
• Exposure to real-time streaming (Kafka, Spark Streaming) Experience with CI/CD tools (GitHub, Jenkins, AWS CloudFormation etc.) Solid understanding of distributed systems and scalable architectures
• Strong foundation in software engineering principles (Git, testing, design patterns)

Experienced in working with Agile teams

• Collaborate with Data Science and AI teams to operationalize ML models and analytics workflows.
• Support real-time analytics and edge AI use cases in manufacturing environments.
• Use AI extensively in building and testing Data Ingestion and Data pipeline Nice-to-Have Skills Experience with Graph Databases (Neo4j, Neptune) Experience with Vector Databases (Milvus, OpenSearch) Knowledge of NVIDIA ecosystem and RAPIDS (cuDF, cuML, cuGraph)
• Experience integrating AI/ML pipelines or GenAI workflows
• Bachelor’s or Master’s degree in Computer Science / Engineering Proven track record in leading engineering teams and delivering at scale
• Strong leadership, communication, and stakeholder management skills
• This position requires working onsite five days a week.

Relocation is available for this position.

Posting Dates

November 4, 2026 - November 12, 2026 Caterpillar is an Equal Opportunity Employer. Qualified applicants of any age are encouraged to apply Not ready to apply? Join our Talent Community .