#111 - Data Engineer

We are seeking a hands-on Data Engineer to support a large-scale data platform modernization initiative. You will migrate approximately 1,000 datasets to Snowflake, adapt existing pipelines, and help deliver reliable orchestration with Apache Airflow 3.

Requirements

• The architecture and technical approach are already defined, with the role focused on implementation and migration delivery through March 2027.
• Migrate pipelines from Apache Airflow 2 to Apache Airflow 3. - Migrate Confluent connections between cloud regions. - Migrate pipelines from Hive to Iceberg using Snowpark Connect. - Adapt existing data pipelines to Snowflake while maintaining data quality and reliability. - Support workflow scheduling and monitoring in Apache Airflow. - Partner with the team to execute time-sensitive data migrations.
• 2+ years of experience. - Strong software engineering fundamentals. - Hands-on Python development experience. - Advanced SQL, including complex joins, CTEs, window functions, and performance tuning. - Snowflake data modeling. - Apache Airflow experience developing and testing pipelines in Airflow and Python. - Apache Kafka experience. - Apache Spark experience, including hands-on PySpark. - Experience with Iceberg and Parquet data formats. - AWS experience, particularly S3 data lakes, partitioning, and lifecycle policies. - Testing strategy experience and the ability to debug and test code hands-on. - At least 2 years of industry experience working with large-scale datasets, including Airflow and Spark. - Strong communication skills. - Ability to work independently.

Preferred

• Enterprise experience strongly preferred.
• Hands-on Snowflake migration experience, including data warehouse modernization and performance and cost optimization. - Apache Airflow migration experience, including version upgrades, executor changes, or platform migrations. - Experience using AI tools for development.
• Required Tools & Platforms - Python and SQL. - Snowflake, Apache Airflow, Apache Kafka, and Apache Spark/PySpark. - Iceberg and Parquet. - AWS S3. - The migration environment also includes Confluent, Hive, and Snowpark Connect. Location, Time & Engagement - Open to candidates in eligible EMEA and APAC locations. - Coverage until 12:00 PM Pacific Time is required. - Remote contract engagement for 40 hours per week. - Expected engagement through March 31, 2027; start date to be confirmed. - Two openings are available. - This is not currently a contract-to-hire opportunity.