Remote Python Senior Software Engineer, AI Training

Skills

You will assess how AI coding agents behave in real-world scenarios, focusing on:
Whether the response makes sense
Whether the preamble and reasoning are useful
Whether the output reflects strong engineering judgment
Whether the interaction feels right to an experienced developer

Judge whether outputs are useful, correct at a high level, and aligned with how a strong engineer would think
Assess the quality of explanations and reasoning, not just the code
Distinguish between levels of response quality (what makes something a 2 vs a 4)
Give clear, opinionated feedback in writing on what worked, what didn't, and what felt off or misleading
Help define what great looks like when working with tools like Cursor, Codex and Claude Code

Able to evaluate code without executing it or reviewing every line
Strong written and spoken English (B2 or above). You'll be writing detailed feedback and recording short video explanations, so this is a hard requirement
Comfortable giving direct, opinionated feedback
High bar for what good engineering looks like

Contract | Remote, worldwide | $100-$200/hour depending on experience and location | 10-20 hrs/week
Rate: $100-$200/hour depending on experience and location

Hours: 10-20 hours/week, flexible
Duration: ongoing. Projects run from about two weeks to a few months each, and we offer new ones to evaluators who do well

Please note that eligibility may vary depending on your location, as some regions may not be supported by the client.

This role is about engineering taste. Syntax correctness is the easy part.

Does this feel like something a strong engineer would actually say?

Is the model guiding the user well, or just dumping output?

You should be comfortable making subjective but rigorous judgments, and explaining them clearly.

Start: as soon as you clear the take-home and a project has an open seat
Process: one take-home evaluation exercise with a recorded Loom walkthrough. No interview