Talent.com
Mindrift
Freelance Agent Evaluation EngineerMindrift • Munich, BY, DE
Freelance Agent Evaluation Engineer

Freelance Agent Evaluation Engineer

Mindrift • Munich, BY, DE
Vor 30+ Tagen
Anstellungsart
  • Homeoffice
  • Quick Apply
Stellenbeschreibung

Please submit your CV in English and indicate your level of English proficiency.

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves

We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Effort estimate

Tasks for this project are estimated to take 20 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation

Up to $50/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.

Jobalert für diese Suche erstellen

Freelance Agent Evaluation Engineer • Munich, BY, DE

Ähnliche Stellen

Forward Deployed AI Engineer

Next Venturesmünchen, bayern, Deutschland

Forward Deployed AI Engineer - Enterprise AI Transformation - Munich - 150k.We’re looking for engineers who want to operate at the intersection of AI, engineering, and enterprise transformation.Thi... Mehr anzeigen

 • Gesponsert

Customer Engineer, Google Distributed Cloud, EMEA

Google Inc.münchen, bayern, Deutschland

Google welcomes people with disabilities.The preferred working location can be selected from:.Munich, Germany; Paris, France; London, UK; Copenhagen, Denmark; Madrid, Spain.Bachelor's degree or equ... Mehr anzeigen

 • Gesponsert

Legal and Product Engineer

Solve Intelligencemünchen, bayern, Deutschland

At Solve Intelligence, we’re building the AI-powered infrastructure for the global Intellectual Property landscape.AI innovation to ensure our engine is technically superior and legally airtight.Th... Mehr anzeigen

 • Gesponsert

Freelance Luxury Automotive Evaluator - Munich & Bavaria South, DE

CXGmünchen, bayern, Deutschland

Become a Luxury Automotive Experience Evaluator.As a Luxury Automotive Experience Evaluator, you’ll be invited to visit premium automotive showrooms and provide expert feedback on the customer jour... Mehr anzeigen

 • Gesponsert

Freelance Luxury Automotive Evaluator - Munich & Alpine Region (German-speaking)

CXG groupmünchen, bayern, Deutschland

Become a Luxury Automotive Experience Evaluator.As a Luxury Automotive Experience Evaluator, you’ll be invited to visit premium automotive showrooms and provide expert feedback on the customer jour... Mehr anzeigen

 • Gesponsert

AI Prompt Evaluator (Croatian) Limited Openings in Essen

Cross Border Talentsmünchen, bayern, Deutschland

AI Prompt Evaluator (Croatian) Limited Openings in Essen.Job Openings AI Prompt Evaluator (Croatian) Limited Openings in Essen.About the job AI Prompt Evaluator (Croatian) Limited Openings in Essen... Mehr anzeigen

 • Gesponsert

Forward Deployed Engineer (m/w/d)

Mayflower GmbHmünchen, bayern, Deutschland

Die Mayflower GmbH baut Individualsoftware mit Fokus auf Generative AI – seit 1998 inhabergeführt.Wir bringen komplette AI-Infrastrukturen in Produktion: Model-Serving, Agent-Gateways, Chat und Voi... Mehr anzeigen

 • Gesponsert

Forward Deployed Engineer (FDE)

Early-Stage AI Startup in Stealthmünchen, bayern, Deutschland

Our mission: Putting AI to work.We're not making another "AI app"; we're fundamentally changing how work gets done.We deploy agentic AI systems into real businesses (logistics, ops, healthcare) to ... Mehr anzeigen

 • Gesponsert

Customer Engineer, Sovereignty (German, English)

Googlemünchen, bayern, Deutschland

Customer Engineer, Sovereignty (German, English).Location: Munich, Germany; Berlin, Germany; +3 more; +2 more.Experience owning outcomes and decision making, solving ambiguous problems and influenc... Mehr anzeigen

 • Gesponsert

Staff AI Agent Engineer - Moveworks | Customer Deployment

Servicenowmünchen, bayern, Deutschland

It all started when engineer Fred Luddy wrote code that automated a tedious task for his coworker, Phyllis.That moment inspired Fred to build a company that could do that for everyone—freeing peopl... Mehr anzeigen

 • Gesponsert

Engineering Manager - AI Evaluation Engine

Resaromünchen, bayern, Deutschland

Resaro was founded on the belief that AI will change the world in ways we cannot even imagine - but every new technology needs safeguards to advance.We are an independent, third‑party AI assurance ... Mehr anzeigen

 • Gesponsert

Luxury Brand Evaluator in Munich - German Speaking (Mission-based)

CXG groupmünchen, bayern, Deutschland

Turn your passion for luxury into a career opportunity.Explore the world of premium brands and make a lasting impact in fashion, beauty, jewelry, or automobiles.As a luxury brand evaluator, you wil... Mehr anzeigen

 • Gesponsert

(Associate) Engineer (m/f/d) Development Consumables

Getingemünchen, bayern, Deutschland

Select how often (in days) to receive an alert:.Company: PULSION Medical Systems SE.Remote Work: 1-2 days at home (site based).Join our diverse teams of passionate people and a career that allows y... Mehr anzeigen

 • Gesponsert

Forward Deployed Engineer - EMEA

Tulip Interfacesmünchen, bayern, Deutschland

Location: 50% of time at a client location within assigned country.As a Forward Deployed Engineer (FDE), you work directly with manufacturers to design, build, and deploy solutions on Tulip's Front... Mehr anzeigen

 • Gesponsert

Forward Deployed Engineer, Agentic Platform (UK/Europe)

Coheremünchen, bayern, Deutschland

Cohere is the leading security-first enterprise AI company.We build cutting‑edge foundation AI models and end‑to‑end products that are designed to solve real‑world business problems.We’re training ... Mehr anzeigen

 • Gesponsert

Lead / Principal Deployment Engineer

Process Analytics Factory - PAFnow by Celonismünchen, bayern, Deutschland

As a Lead / Principal Deployment Engineer, you are the backbone of Celonis delivery.You lead the hands‑on implementation of Celonis solutions for enterprise customers, combining platform expertise ... Mehr anzeigen

 • Gesponsert

Luxury Brand Evaluator in Munich - German Speaking (Mission-based), DE

CXGmünchen, bayern, Deutschland

Turn your passion for luxury into a career opportunity.Explore the world of premium brands and make a lasting impact in fashion, beauty, jewelry, or automobiles.Join CXG, the global leader in custo... Mehr anzeigen

 • Gesponsert

Principal Forward Deployed Engineer – AI-Native Software Development (m/f/d)

IT-Systemhaus der Bundesagentur für Arbeitmünchen, bayern, Deutschland

Principal Forward Deployed Engineer – AI-Native Software Development (m/f/d).At JetBrains, code is our passion.Ever since we started, back in 2000, we have strived to make the strongest, most effec... Mehr anzeigen

 • Gesponsert

Customer Engineer, Google Distributed Cloud, EMEA

Googlemünchen, bayern, Deutschland

Customer Engineer, Google Distributed Cloud, EMEA.Munich, Germany; Paris, France; London, UK.Google welcomes people with disabilities.Bachelor's degree or equivalent practical experience.Experience... Mehr anzeigen

 • Gesponsert

Quality Engineer (m/w/d) – Medical Devices - English & German Speaking

BDmünchen, bayern, Deutschland

Quality Engineer (m/f/d) Medical Devices – Wangs, Switzerland.Support in all quality development activities for commercial production.Ensuring and complying with the quality system requirements def... Mehr anzeigen