From B2B product manager to AI PM: after the eval-set question
Her resume said she led an AI support project. In the first round she got grilled on evaluation and metrics until she stalled. Before round two she fixed exactly that
WhoFour years as a B2B SaaS product manager, worked on an LLM customer-support project last year, now going for AI PM roles
Example scenario: written around a typical user. People, companies and numbers are made up to show how the product works. Not a real student.
Background
The strongest line on her resume: "Led an AI customer-support project, resolution rate up 30%". The project was real, but she owned scenario selection and the launch plan. The model and the evals were built by the ML team.
This is a common path into AI PM, and it has one classic failure mode: the more your resume sounds like you led the AI, the harder interviewers will test you as an AI PM.
The dossier: what went in
She created a job dossier in OfferSpar and added three things. Every later round's prep and debrief uses them.
| Resume | Her general resume, with that project line unchanged |
|---|---|
| Job description | AI PM, customer support, at a large e-commerce company. "LLM shipping experience and evaluation skills preferred" |
| Notes | The recruiter said round one is with the product lead of the business unit and will dig into projects |
Key calls in the prep
Before round one, the prep didn't hand her a list of common AI PM questions. It started with where this round was most likely to go wrong.
"Up 30%" will be questioned
A percentage with no baseline or time window is the favorite follow-up. The interviewer isn't checking whether the number is impressive; they're checking whether you did the work. Prepare one sentence: how the metric is defined, where the sample came from, what it's compared against.
"Led an AI project" means you'll be asked about evals and model choice
The JD says evaluation skills are preferred. That's what the team is missing right now, so they'll push on it. If you can't describe an eval set, interviewers assume the project never left the demo stage.
Separate what the team did from what you did
If the ML team built the evals, say so. Then put the weight on the calls you made: why support tickets first, who set the quality bar and how.
What happened in the interview
She handled the metric question well. The sentence she prepared did its job.
Then came: "How was the eval set built? How do you attribute bad cases?" She said the ML team ran it weekly, got pushed two more levels, and ended up admitting she didn't know.
Afterwards she logged the six questions and roughly what she said.
What the debrief said
The debrief compared the round against the prep and called out what worked and what didn't.
QuestionHow was the resolution rate calculated Strong
Next timeDefinition and baseline first, then before and after. The interviewer moved on. Keep this as your standard answer.
QuestionHow was the eval set built Lost points
Next timeDon't pretend you built it. Say which part you owned, such as the bad-case categories or the acceptance bar, then what you learned about evaluation from it.
QuestionWhat would you change if you did it again Weak
Next time"More time" isn't an answer. Name a real trade-off, such as adding human-handoff rate to the evals earlier.
What changed for the next round
After the debrief, "can't explain evaluation" went into her account memory as a weak spot. The round-two prep opened with it.
- Prepare an eval story: the bad-case categories she helped define and how they changed the launch plan
- Round two is likely with the business owner, so lead with the business call (why support tickets first) instead of retelling the project
- Say "the ML team built this part" up front, so she steers where the follow-ups go
Keep reading
Try it with your own resume and JD
Create a job dossier and get prep for your next round