Talent.com
Mindrift
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)Mindrift • Munich, BY, DE
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Mindrift • Munich, BY, DE
Vor 24 Tagen
Anstellungsart
  • Homeoffice
  • Quick Apply
Stellenbeschreibung

Please submit your CV in English and indicate your level of English proficiency.


Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.What this opportunity involves:
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.


You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT:

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for:

  • 8+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paidEffort estimate


Tasks for this project are estimated to take 30 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation:
Up to $200/hr equivalent, depending on level and pace. Tasks are estimated at ~30 hours each; you set your own schedule.

Jobalert für diese Suche erstellen

Senior Python Engineer - AI Coding Agent Evaluation (Freelance) • Munich, BY, DE

Ähnliche Stellen

Working Student Software Engineering Python (all genders)

Avelios Medical GmbHMunich, de

Our mission at Avelios is to unlock clinical data to power seamless healthcare operations for better patient care.To do so, we have built a modular software platform that digitizes and optimizes wo... Mehr anzeigen

 • Gesponsert

Junior AI Solutions Engineer (m/w/d)

Reply GroupMünchen, de

Junior AI Solutions Engineer (m/w/d).Als Junior AI Solutions Engineer (m/w/d) arbeitest du an der Entwicklung innovativer AI-Lösungen für unsere Kunden.Gleichzeitig baust du AI-Unterstützungen für ... Mehr anzeigen

 • Gesponsert

(Senior) Applied AI/ML Engineer (f/m/d)

Perelynmünchen, bayern, Deutschland

As an applied AI engineer at Perelyn, you will be involved in both internal and external engagements.Your responsibilities include:.Contributing to a wide range of AI-related projects, which span f... Mehr anzeigen

 • Gesponsert

Senior Software Engineer, AI (m/f/d)

Off GENmünchen, bayern, Deutschland

AI-powered productivity tools for consulting and finance teams, starting with PowerPoint.We're a small, ambitious team based in Munich, working with some of the world's leading firms.We move fast, ... Mehr anzeigen

 • Gesponsert

Senior AI Engineer (w/m/d)

Skalarmünchen, bayern, Deutschland

Build our core AI capabilities from v1 ->.Join the early core team of experienced entrepreneurs (scaled to millions of users, $100m+ exits).Build the future of tax & accounting with AI.Build our in... Mehr anzeigen

 • Gesponsert

(Senior-)AI Engineer (m/w/d)

Atruvia AGAschheim (München), de

Für unser Geschäftsfeld Data, Security and Identity im Tribe Advanced Intelligence (DSIADI) suchen wir dich in Voll- oder Teilzeit ab 30 Wochenstunden als (Senior-) AI Engineer (m/w/d) in Karlsruhe... Mehr anzeigen

 • Gesponsert

Founding AI Engineer

alagomünchen, bayern, Deutschland

The construction industry loses.Repeating the same mistakes over and over.We help teams eliminate double data entry and manual busywork by connecting to existing tools,.With every project, alago ge... Mehr anzeigen

 • Gesponsert

Unreal Engine Developer (Mid-Level / Senior) – Synthetic Data & AI Simulation

Autonomous Teamingmünchen, bayern, Deutschland

Work in an international, agile team creating the future of autonomous systems.Grow your career in a expanding and ambitious engineering team.Build innovative products using state-of-the-art techno... Mehr anzeigen

 • Gesponsert

(Senior) AI Engineer (m/w/d)

KI performance GmbHmünchen, bayern, Deutschland

Become our new (Senior) AI Engineer (m/f/d).Are you passionate about AI and cutting‑edge technologies? Do you want to help companies leverage AI to create real business impact? Are you ambitious, s... Mehr anzeigen

 • Gesponsert

(Senior) Full-stack Engineer | Carbmee

Carbmee GmbHmünchen, bayern, Deutschland

Carbmee builds the System of Action for carbon and environmental data in the world's most complex industrial sectors: automotive, defense, heavy manufacturing.Backed by €20M Series A, growing fast ... Mehr anzeigen

 • Gesponsert

Senior AI Engineer with focus on ML & LLMs (m/f/d)

Motiusmünchen, bayern, Deutschland

Senior AI Engineer with focus on ML & LLMs (m/f/d).Motius is an R&D company specialized in solving challenges with emerging tech to create longlasting value for customers.With the unique combinatio... Mehr anzeigen

 • Gesponsert

AI Solutions Architect

European Patent Office (EPO)Bad Tölz, Germany

Expertise in modern cloud/cloud-native architectures and a proven track record delivering production-grade AI/ML solutions (deep learning, generative AI) that solve real business problems.Hands-on ... Mehr anzeigen

 • Gesponsert

AI Platform Engineer – Developer Tools & LLM Operations (all gender)

ALTEN Consulting Services GmbHmünchen, bayern, Deutschland

Die ALTEN Consulting Services GmbH bietet mit über 20 Jahren Branchenerfahrung ihren namhaften Kunden aus Industrie und Wirtschaft hochspezialisierte Engineering- und IT-Dienstleistungen zur gemein... Mehr anzeigen

 • Gesponsert

Software/AI Engineer

Product PulseMunich, Bavaria, DE

An innovative company is the AI coworker.It lives in Slack and Microsoft Teams, connects to thousands of tools, and does real work for real companies: finance, marketing, ops, engineering.We're bui... Mehr anzeigen

Senior AI Software Engineer (m/f/d)

Peter Park System GmbHmünchen, bayern, Deutschland

We are looking for a forward-thinking Senior AI Software Engineer to drive intelligence into our core products and internal operations.Sitting at the intersection of robust backend engineering and ... Mehr anzeigen

 • Gesponsert

Senior Plattform Engineer (m/w/d)

Aramaz Digital GmbHMünchen, de

Senior Plattform Engineer (m/w/d).Platform Engineering | DevOps | Kubernetes | Cloud Infrastruktur | CI/CD | Energiewende | IT Architektur.Für unseren Kunden in München suchen wir einen erfahrenen ... Mehr anzeigen

 • Gesponsert

Senior Applied AI Engineer (m/f/d)

Agile Robots AGmünchen, bayern, Deutschland

Permanent employee,Full-time· Germany, Munich (HQ).As a Senior Applied AI Engineer (Intelligent Application Focus) at Agile Robots SE, you will design, build, and deploy AI solutions that enable ou... Mehr anzeigen

 • Gesponsert

Senior AI Engineer

Spheremünchen, bayern, Deutschland

We are looking for an experienced AI Engineer to join our growing team and help design, and deploy robust AI software solutions for our clients.You will lead and deliver high-impact AI projects in ... Mehr anzeigen