User Researcher, AI Evaluations - Notion

About the Role

We’re seeking an experienced UX Researcher to define and scale how we evaluate Notion’s AI-powered experiences. You will focus on defining what “good” looks like for model output quality and the end-to-end product experience, including discovery, goal setting, delegation, and building trust with AI.

This role sits at the intersection of research craft and evaluation operations. You will run studies to uncover user mental models and translate insights into reusable rubrics, workflows, and measurement approaches for cross-functional teams.

What You'll Achieve

• Define Frameworks & Rubrics: Establish clear, reusable evaluation criteria for helpfulness, trust, tone, control, and transparency.
• Run Recurring Evals: Lead longitudinal and feature-specific studies, including qualitative research and human-in-the-loop evaluations to benchmark improvements.
• Anchor in Real Workflows: Ensure evaluations reflect actual jobs-to-be-done and user intent rather than decontextualized feedback.
• Identify Failure Modes: Uncover system breakdowns and edge cases to provide actionable guidance for guardrails and prioritization.
• Operationalize Evaluation: Collaborate with Product, Design, Engineering, and Data Science to build scalable evaluation loops.

Skills You'll Need

• Operationalizing Insight: Ability to turn qualitative user expectations into concrete rubrics and observable metrics.
• AI Fluency: Experience evaluating AI-enabled products (LLMs, agents, generative UI) and working with Data Science/ML partners.
• Communication: Ability to align diverse partners around shared definitions of quality and drive product change.
• UX Research Craft: Proficiency in interviews, benchmarking, surveys, and experiments.
• Pragmatism: Ability to prioritize ruthlessly and work through ambiguity in a fast-moving environment.
• Experience: 5+ years of industry UX research experience.

Nice to Haves

• Familiarity with LLM-as-judge methods or “golden dataset” creation.
• Experience with AI research and observability tooling (e.g., Dovetail, Braintrust).
• Proficiency in SQL, Python, or statistical software.
• Master’s or PhD in HCI, Psychology, Behavioral Science, or related fields.