Fully Remote Project for Software Engineers/Coding Experts to Grade and Improve AI Efficiency

The Role:

This is not a traditional software engineering role. You will be helping build better AI by putting your knowledge to work in a structured, flexible, fully remote environment. The work is long-form and self-directed, and clear written reasoning matters as much as technical depth.

Responsibilities:

• Design challenging, realistic software engineering tasks drawn from your own day-to-day work: the scenario, a spec phrased the way you would brief a trusted teammate, and the codebase or supporting files an engineer would need (repositories you construct or adapt, tests, logs, API contracts, design notes), which you author yourself.
• Run those tasks through AI coding agents and evaluate the deliverable they produce (the diff, the tests, the design, the review) against the standard you would hold a teammate to.
• Compare two agent outputs on identical specs and codebases, decide which performed better, and document where each fell short.
• Write tests, rubrics, and success criteria that specify what a correct deliverable must contain (the right behavior preserved, the right edge cases handled, the right tests added, the right conventions followed), and explain in writing why a submission passes or fails each one.
• Flag concrete failures with evidence: brittle changes, missed edge cases, tests that pass for the wrong reason, thrashing in the trace, fabricated or ignored files, and off-spec interpretation of the ask.
• Contribute across your stack and adjacent ones, and review and refine tasks built by other engineers.

Qualifications Required:

• 2+ years of professional software engineering experience preferred, shipping and maintaining production code as part of a team.
• In progress Bachelor’s degree or higher.
• Expertise in at least one common stack: TypeScript or JavaScript, Python, Java, Go, Rust, C++, C#, Ruby, PHP, Swift, or Kotlin. Specialists are welcome: a strong
• Comfortable in a terminal with git, tests, and a debugger; able to read an unfamiliar codebase and orient quickly. Docker and CI experience is a plus.
• Strong judgment about what good code and a real model failure look like, and the ability to explain both in writing.
• No prior AI or machine learning experience is required. Engineering judgment and attention to detail matter most.
• Hands-on practitioner: you currently do (or recently did) the work yourself at an individual-contributor level, not solely in a managerial capacity.
• Full professional or native-level written and spoken English; you can articulate why a result is wrong, not only that it is.
• General familiarity with AI and LLM tools: you have used models like Claude or ChatGPT in professional work and can tell a well-reasoned answer from a plausible-sounding but incorrect one.
• Baseline tech literacy: comfortable with cloud file tools (e.g., Google Workspace), managing browser profiles, downloading and installing desktop apps (e.g., Claude), and everyday file handling (e.g., converting between Excel and Google Sheets, zipping files for sharing).
• A computer science degree is a plus but not required; shipped professional work outweighs credentials.

Process:: Assessment- 2-43 hours (unpaid)