Machine Learning Engineer – ML Evaluation & Experiment Design

Responsibilities

• The work involves analyzing ML experiments, datasets, metrics, and pipelines to determine whether challenges are technically sound, reproducible, appropriately difficult, and genuinely require strong machine learning reasoning.
• Experiment design and model selection
• Small and synthetic datasets
• Data quality and preprocessing
• Distribution shift and data contamination
• Label noise and feature leakage
• Model evaluation and metric selection
• Hyperparameter tuning
• Train / validation / test methodology
• Reviewing ML challenges and determining whether they are well designed and technically solvable
• Evaluating whether datasets contain meaningful and learnable signals
• Identifying unintended shortcuts or artifacts in synthetic datasets
• Determining whether tasks require genuine diagnosis of the underlying ML problem
• Reviewing evaluation metrics and improvement thresholds
• Detecting metric gaming, data leakage, and evaluation flaws
• Verifying reproducibility across the complete data → model → evaluation pipeline
• Assessing whether challenge difficulty is appropriately calibrated
• Providing clear recommendations for improving, recalibrating, or excluding problematic tasks

Requirements

• 3+ years of hands-on applied machine learning experience

Preferred

• Experience creating or participating in
• Kaggle, DrivenData, or similar ML competitions
• Experience designing benchmark datasets or ML challenges
• Background in data-centric AI or dataset quality

Experience with synthetic data generation and validation

• Familiarity with statistical testing, confidence intervals, and effect sizes
• Experience with ML evaluation pipelines, RLHF, or AI model evaluation
• Experience developing ML curricula or technical assessments
• Understanding of common ML failure modes such as:

You’ll review ML challenges involving

Reproducibility and deterministic pipelines

Statistical significance of model improvements

A key part of the role is determining whether a challenge actually rewards good ML reasoning, rather than simply being solvable through brute-force model selection or large hyperparameter searches.

Strong experience with

• ML experiment design
• Model selection
• Hyperparameter tuning
• Model evaluation
• Data preprocessing and validation
• Strong understanding of train, validation, and test splits

Ability to identify

• Data leakage
• Label noise
• Distribution shift
• Spurious correlations
• Feature leakage

Data contamination

• Experience evaluating whether performance improvements are statistically meaningful rather than random fluctuations
• Strong understanding of ML evaluation metrics and when different metrics are appropriate
• Experience debugging ML workloads across CPU and GPU environments
• Ability to analyze technical problems and provide clear written feedback

Nice to Have

• Shortcut learning
• Spurious correlations
• Goodhart’s Law
• Simpson’s paradox
• Metric gaming

What You’ll Be Responsible For

Engagement

Engagement

Part-time, project-based consulting

Focus

Applied machine learning, experiment design, data quality, and model evaluation

This role is a strong fit for ML engineers who enjoy debugging experiments, understanding why models succeed or fail, identifying problems in datasets and evaluation pipelines, and designing rigorous machine learning experiments.