AI Research Engineer - dscout

At Dscout, we’re building the most flexible and powerful UX research platform on the market—trusted by top brands across finance, healthcare, consumer goods, and tech. Our tools help teams deeply understand the humans behind their products so they can build better ones.

Dscout AI Training & Evals

Dscout AI Training & Evals builds human data and evaluations that demonstrate whether AI works for real people. For ten years, Dscout has captured how people behave in real contexts with consent and scientific rigor. AI Training & Evals directs that engine toward AI: consented human behavior and judgment turned into training data, evaluations, and benchmarks AI teams can trust.

We serve AI researchers, product builders, and research teams. We are a new unit inside an established company: live customers, a mature platform (React, Elixir, Python, Swift, Kotlin; SOC 2 and HITRUST certified), a small senior team, and room to build. We operate AI-native—agents write code, evals gate changes, and we move fast.

Why This Role

• Tackle Unsolved Problems: Measure whether AI works for real people over real time.
• Unique Data: Study real participants performing real tasks in their own contexts.
• Own the Benchmark: Found and design Dscout's first benchmark from the ground up.
• Publish: Publish methods and benchmarks to the research community with first authorship.
• Small Senior Team: Work closely with the Head of AI Research and the General Manager.

Responsibilities

This is a research-first role focused on questions where the right rubric, benchmark, or method does not yet exist. You will identify gaps, propose bold directions, run experiments, construct benchmarks, and publish findings, while ensuring the science functions within live customer programs.

Rubrics and QA Science

• Design rubrics and test validity, reliability, and sensitivity.
• Measure human rater reliability and distinguish noise from true signal.
• Calibrate AI judges against human judgment and study drift.
• Develop quality signals to substantiate shipped data points.

Long-Horizon, Real-User Benchmarks

• Design Dscout’s first benchmark focusing on real users in real context over long horizons (multi-week tasks across tools).
• Define protocol, task design, scoring, and release; collaborate with delivery teams on fieldwork.
• Maintain benchmark integrity over time (handling drift, contamination, and difficulty shifts).
• Author research papers and lead public releases.

Human Judgment as Training and Eval Signal

• Study when real-world human feedback improves a model versus adding noise.
• Contrast preference and judgment data from representative users against expert data.
• Translate findings into operational methods for forward-deployed engineers and delivery teams.

New Research Directions

• Propose new programs such as calibrating user simulations against real behavior, human-AI collaboration, and behavioral agent evaluation.
• Co-author studies with external research labs and academics; help facilitate the internal paper-reading group.

First-Year Milestones

• A QA and rubric framework running on live customer programs with documented validity.
• Benchmark v0 released with a paper submitted to a top-tier venue.
• Adoption or citation of our work by at least one external team.
• Delivery colleagues independently executing your research methods.

Requirements

• Publication Record: Published research, preprints, or workshop papers in evals, preference learning, reward modeling, human-AI interaction, measurement, or agent evaluation.
• Education/Experience: PhD, PhD in progress, or MS with 2+ research internships at top labs/industry groups (or an equivalent track record of creative ML research).
• Statistics: Strong applied statistics background (inter-rater reliability, confidence intervals, statistical power).
• Evaluation Experience: Hands-on experience with LLM-as-judge frameworks, rubric-based evaluation, and preference datasets.
• Software Skills: Strong Python programming skills with the ability to build evaluation pipelines and research prototypes. AI coding tools integrated into daily workflow.
• Messy Data: Experience working with real, uncurated human behavioral data.
• Communication: Clear technical writing skills across both research and product audiences.
• Location: Based in the San Francisco Bay Area.

Nice to Have

• Long-horizon or agentic evaluation experience.
• Background in psychometrics, item response theory (IRT), or human-subjects research.
• Open-source benchmark or dataset contributions.

Compensation & Benefits

• Total Compensation: $250,000 - $400,000 OTE + Equity.
• Built-in bonus and equity programs.
• Flexible PTO and 15 company holidays.
• 12 weeks of paid parental leave.
• 401(k) matching program.
• Education stipend and conference budget.
• Remote work stipend.

Interview Process

1. Initial conversation with the Head of AI Research.
2. Research presentation on a project you led.
3. Design exercise (rubric validation for long-horizon tasks).
4. Conversations with the GM, forward-deployed engineers, and delivery partners.