Skip to content
中文Sign in

From B2B product manager to AI PM: after the eval-set question

Example

Her resume said she led an AI support project. In the first round she got grilled on evaluation and metrics until she stalled. Before round two she fixed exactly that

WhoFour years as a B2B SaaS product manager, worked on an LLM customer-support project last year, now going for AI PM roles

Example scenario: written around a typical user. People, companies and numbers are made up to show how the product works. Not a real student.

01

Background

The strongest line on her resume: "Led an AI customer-support project, resolution rate up 30%". The project was real, but she owned scenario selection and the launch plan. The model and the evals were built by the ML team.

This is a common path into AI PM, and it has one classic failure mode: the more your resume sounds like you led the AI, the harder interviewers will test you as an AI PM.

02

The dossier: what went in

She created a job dossier in OfferSpar and added three things. Every later round's prep and debrief uses them.

ResumeHer general resume, with that project line unchanged
Job descriptionAI PM, customer support, at a large e-commerce company. "LLM shipping experience and evaluation skills preferred"
NotesThe recruiter said round one is with the product lead of the business unit and will dig into projects
03

Key calls in the prep

Before round one, the prep didn't hand her a list of common AI PM questions. It started with where this round was most likely to go wrong.

  1. "Up 30%" will be questioned

    A percentage with no baseline or time window is the favorite follow-up. The interviewer isn't checking whether the number is impressive; they're checking whether you did the work. Prepare one sentence: how the metric is defined, where the sample came from, what it's compared against.

  2. "Led an AI project" means you'll be asked about evals and model choice

    The JD says evaluation skills are preferred. That's what the team is missing right now, so they'll push on it. If you can't describe an eval set, interviewers assume the project never left the demo stage.

  3. Separate what the team did from what you did

    If the ML team built the evals, say so. Then put the weight on the calls you made: why support tickets first, who set the quality bar and how.

04

What happened in the interview

She handled the metric question well. The sentence she prepared did its job.

Then came: "How was the eval set built? How do you attribute bad cases?" She said the ML team ran it weekly, got pushed two more levels, and ended up admitting she didn't know.

Afterwards she logged the six questions and roughly what she said.

05

What the debrief said

The debrief compared the round against the prep and called out what worked and what didn't.

QuestionHow was the resolution rate calculated Strong

Next timeDefinition and baseline first, then before and after. The interviewer moved on. Keep this as your standard answer.

QuestionHow was the eval set built Lost points

Next timeDon't pretend you built it. Say which part you owned, such as the bad-case categories or the acceptance bar, then what you learned about evaluation from it.

QuestionWhat would you change if you did it again Weak

Next time"More time" isn't an answer. Name a real trade-off, such as adding human-handoff rate to the evals earlier.

06

What changed for the next round

After the debrief, "can't explain evaluation" went into her account memory as a weak spot. The round-two prep opened with it.

  1. Prepare an eval story: the bad-case categories she helped define and how they changed the launch plan
  2. Round two is likely with the business owner, so lead with the business call (why support tickets first) instead of retelling the project
  3. Say "the ML team built this part" up front, so she steers where the follow-ups go

Keep reading

Try it with your own resume and JD

Create a job dossier and get prep for your next round