jobapplier

Data Engineer

Collinsongroup

Skills

Required

  • Data Engineering
  • Python
  • SQL
  • Snowflake
  • data transformation
  • AWS
  • Terraform
  • Kafka
  • data streaming
  • Git
  • CI/CD
  • automated testing
  • data pipelines

Preferred

  • Stream processing
  • Flink
  • Java
  • MongoDB
  • Atlas / SnapLogic / AWS DMS
  • deployment
  • migrations
  • backup
  • Docker

Responsibilities

  • Develop, maintain and troubleshoot data pipelines using Python, SQL, Snowflake and dbt.
  • Create and maintain tables, views and access permissions, while improving performance.
  • Work with Kafka topics, schemas and connectors to manage real-time data movement.
  • Use Terraform to deploy and maintain AWS resources and other cloud platform services.
  • Write Python scripts, create automated tests and manage CI/CD deployments using GitHub Actions.
  • Monitor pipelines and platform services, investigate failures, resolve incidents and participate in on-call support.
  • Follow security standards, manage access controls and improve platform stability and costs.
  • Participate in the on-call rota, diagnose production issues, coordinate incident response, and drive post-incident actions that prevent recurrence.
  • Maintain runbooks, architecture records and operational documentation for the services and workflows you own.
  • Contribute to agile planning, technical design and code reviews, and collaborate effectively with engineers and stakeholders across distributed teams.
  • Our core platform stack includes
  • Python and SQL; Snowflake and dbt; Confluent Cloud (Kafka, Schema Registry, Flink SQL and ksqlDB); AWS; Terraform; MongoDB Atlas; SnapLogic; Datadog and PagerDuty; GitHub Actions; Docker and ECS/Fargate; and JSON Schema data contracts with policy-as-code controls.

Requirements

Knowledge, skills and experience required

  • A university degree in computer science, engineering or a related discipline, or equivalent practical experience.
  • At least 4-5 years of hands-on Data Engineering Strong Python and SQL skills, with experience writing production code.
  • Practical experience with Snowflake and confluent for data transformation Hands-on experience with AWS and Terraform.
  • Understanding of Kafka, including topics, consumers and data streaming.
  • Experience with Git, CI/CD and automated testing.
  • Ability to troubleshoot data pipelines and production issues independently.
  • Understanding of cloud security, monitoring and access management.
  • Good communication, problem-solving and teamwork skills.

Preferred

  • Stream processing with Flink SQL or ksqlDB; Java for Flink user-defined functions is an advantage.
  • Administration or integration experience with MongoDB Atlas, SnapLogic or AWS DMS, including deployment, monitoring, migrations, backup and recovery.
  • Observability, incident response and delivery automation using Datadog, PagerDuty and GitHub Actions.
  • Docker, ECS/Fargate and AWS networking, including PrivateLink or Transit Gateway, with a focus on resilience, capacity and cost.
  • JSON Schema, data contracts and policy-as-code controls using OPA or Conftest.
  • AI-assisted software development tools and practices that improve delivery speed and quality, supported by appropriate review, testing, security and governance controls.
  • We welcome applicants with diverse perspectives, career paths and equivalent experience.