Talent.com
Mindrift
Freelance Agent Evaluation EngineerMindrift • Frankfurt am Main, HE, DE
Freelance Agent Evaluation Engineer

Freelance Agent Evaluation Engineer

Mindrift • Frankfurt am Main, HE, DE
Vor 30+ Tagen
Anstellungsart
  • Homeoffice
  • Quick Apply
Stellenbeschreibung

Please submit your CV in English and indicate your level of English proficiency.


Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.


We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT:

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for:

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

Requirements and benefits

Educational qualifications

  • A Master’s Degree in Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields.
  • Bachelor’s degree is accepted if only candidate has 5 years of experience in the field.

Academic and/or Professional Experience

Candidates should have a minimum of 3 years of professional experience in related roles or domain - specifically for QA-automation/testing or cybersecurity roles

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Compensation:
Paid per accepted task. Your rate depends on the qualification tier you reach and how efficiently you complete tasks — up to the equivalent of $50/hr. Because payment is per task, a faster pace raises your effective hourly rate.

Why this freelance opportunity might be a great fit for you?

  • Take part in a part-time, remote, freelance project that fits around your primary professional or academic commitments.
  • Work on advanced AI projects and gain valuable experience that enhances your portfolio. - Influence how future AI models understand and communicate in your field of expertise.
Jobalert für diese Suche erstellen

Freelance Agent Evaluation Engineer • Frankfurt am Main, HE, DE

Ähnliche Stellen

Agent Engineer

Durlston Partnersfrankfurt, hesse, Deutschland

Backed by one of Europe’s leading investment firms, the company is expanding its Frankfurt operation and looking to bring on two commercially driven Sales Traders to own the German market.The role ... Mehr anzeigen

 • Gesponsert

Prompt Evaluator, Croatian Speaker, Essen, Germany

Cross Border Talentsfrankfurt, hesse, Deutschland

About the job Prompt Evaluator, Croatian Speaker, Essen, Germany.Are you a detail-oriented professional with a passion for language and technology? We are looking for a.Essen with excellent lifesty... Mehr anzeigen

 • Gesponsert

IT Operations AI Engineer (m/f/d)

Terma Groupdarmstadt, hessen, Deutschland

We are looking for an AI Engineer to apply AI and advanced analytics to EUMETSAT's infrastructure operations and to enable and evolve the AI/ML infrastructure of the multi-mission computing environ... Mehr anzeigen

 • Gesponsert

(Senior) GenAI Engineer

Capco - The Capital Markets CompanyFrankfurt am Main, de

So divers wie unsere Mitarbeiter*, sind auch unsere Lösungen, die wir unseren Kunden anbieten.Bringe Dich ein und setze Deine Ideen bei Capco in die Tat um als.In Frankfurt am Main, Düsseldorf oder... Mehr anzeigen

 • Gesponsert

Prompt Evaluators (m/f/d) with Japanese

TELUS Digitalfrankfurt, hesse, Deutschland

Prompt Evaluators (m/f/d) with Japanese.Japanese to support the development of AI technologies.Join us in enhancing the latest speech technology applications by refining our systems in Japanese.Res... Mehr anzeigen

 • Gesponsert

Senior AI Engineer | Agentic Systems (w/m/d)

FlexIT Consulting GmbHfrankfurt, hesse, Deutschland

KI wirksam einsetzen – als Enabler für modernes Software Engineering.Künstliche Intelligenz verändert die Softwareentwicklung grundlegend.In dieser Rolle unterstützt du ein etabliertes Finanzuntern... Mehr anzeigen

 • Gesponsert

Field Application Engineer - nVent Eriflex

nVentfrankfurt, hesse, Deutschland

We’re looking for people who put their innovation to work to advance our success – and their own.Join an organization that ensures a more secure world through connecting and protecting our customer... Mehr anzeigen

 • Gesponsert

Senior Cloud Engineer DevOps (Azure) (m/w/d)

GRAYOAKBad Nauheim, Germany

Abgeschlossenes Studium der (Wirtschafts-)Informatik oder vergleichbare Qualifikation.Mindestens 3-4 Jahre Berufserfahrung im Bereich Cloud Development, DevOps oder Platform Engineering.Fundierte E... Mehr anzeigen

 • Gesponsert

Technical Quality Engineer

Geely Auto Deutschlandfrankfurt, hesse, Deutschland

Responsible for overseeing the technical support function to ensure the network support and the achievement of after-sales department.This role plays a critical part in implementing Geely’s quality... Mehr anzeigen

 • Gesponsert

Automobil-Luxusmarken-Evaluator - Hessen - Rhein-Main

CXG groupfrankfurt, hesse, Deutschland

Haben Sie eine feine Wertschätzung für High-End-Performance-Fahrzeuge und das Kundenerlebnis, das sie umgibt?.Sind Sie mit den Standards, dem Service und der Atmosphäre vertraut, die in luxuriösen ... Mehr anzeigen

 • Gesponsert

Testia GmbH - NDT Inspector (m/f/d)

TESTIA GmbHFriedberg (Hessen), Hessen, DE

We are currently looking to hire a:.In the frame of the development of our Quality Innovation Initiative, Testia is looking for an NDT Inspector (m/w/d):.Carrying out non-destructive testing (NDT) ... Mehr anzeigen

 • Gesponsert

Senior Generative AI Engineer (M/W/D)

IQVIAfrankfurt, hesse, Deutschland

We are seeking a highly skilled Senior Generative AI Engineer to join our pioneering team.In this role, you will be instrumental in developing the Orion platform, focused on accelerating the develo... Mehr anzeigen

 • Gesponsert

Senior Cloud Developer mit Architektur-Fokus (Go / AWS) (w/m/d)

Publicare Marketing Communications GmbHBad Nauheim, Germany

Tiefes Tech-Fundament: Du verfgst ber mindestens 4 Jahre fundierte Berufserfahrung in der Softwareentwicklung, idealerweise im Cloud-Umfeld.Core-Skills: Du beherrschst Go (Golang) sicher und hast b... Mehr anzeigen

 • Gesponsert

Lead Agentic Systems Engineer (m/f/d)

Korro AIdarmstadt, hessen, Deutschland

Korro’s Technology and Innovation AI group translates applied research into product capabilities that directly impact pediatric care.Working at the intersection of Artificial Intelligence, Computer... Mehr anzeigen

 • Gesponsert

Spacecraft Operations Engineer for LEO and/or GEO satellites [26130]

Sopra Steria USAdarmstadt, hessen, Deutschland

As a member of a team providing services to EUMETSAT, you will have the following responsibilities:.Ensure proper in-flight operations for the spacecraft and provide 24/7 on‑call service.Performing... Mehr anzeigen

 • Gesponsert

Senior Software Developer (m/w/d) -.NET / C#

LORENZ Life Sciences GroupBad Nauheim, Germany

Du bringst mehrjhrige Erfahrung in der Entwicklung mit C#,.NET-kosystem mit und kennst dich sicher mit relationalen Datenbanken wie MS SQL, Oracle und PostgreSQL aus.Du bist sicher im Umgang mit ag... Mehr anzeigen

 • Gesponsert

AI-native Product Engineer (w/m/d)

LotumBad Nauheim, Hessen, Germany

Produktiv erprobte Expertise in der Gestaltung und Umsetzung agentischer Softwareentwicklung (Orchestrierung, Context Engineering, RAG, Workflows).Fhigkeit und Motivation, permanent und proaktiv ne... Mehr anzeigen

 • Gesponsert

Confluent Consulting Engineer

Infosysfrankfurt, hesse, Deutschland

Technology: Apache Kafka, Confluent Platform, Stream Processing.Role: Technology Lead / Confluent Consulting Engineer.Seniority level: Mid‑Senior level.Job function: Information Technology.Industri... Mehr anzeigen

 • Gesponsert

Requirements Engineer (m/w/d)

compeople AGfrankfurt, hesse, Deutschland

In der heutigen schnelllebigen digitalen Welt wissen wir, wie wichtig individuelle Erlebnisse und sinnvolle Verbindungen sind.Deshalb bieten wir nicht nur digitale Lösungen an – wir kombinieren uns... Mehr anzeigen

 • Gesponsert

Spacecraft Operations Engineer for LEO and/or GEO satellites [26130]

Sopra Steria DEdarmstadt, hessen, Deutschland

As a member of a team providing services to EUMETSAT, you will have the following responsibilities:.Ensure proper in-flight operations for the spacecraft and provide 24/7 on‑call service.Performing... Mehr anzeigen