Example content; the people and companies are fictional. In real use, guides and debriefs are built from your own resume, the JD, and your notes from every round.
AI Product Manager · Round 1 · Peer interview
This round checks whether the numbers on your resume are really yours
The JD wants someone who puts LLMs into the purchase flow and is strict about metrics. A senior peer PM will dig into your support bot for details, numbers and what you did yourself. Strength: you built the handoff rules and guardrails. Risk: the 40% and 30% on your resume with no clear definition
A senior peer PM, most likely on the shopping-assistant team and someone you'd work with daily. They don't care how many terms you know; they care whether you've actually done the work and can pick up tasks on day one, so expect lots of follow-ups on details and numbers
Candidate: 5 years in product, built an AI support bot at a home-renovation platform, now interviewing for a shopping assistant
Show the job description
What this round really tests
Is the experience real
the '40% resolution lift' and '30% staffing cost cut' on your resume will get taken apart. They want definition, baseline and sample, not a bigger number
Source · Resume: “Resolution rate up 40%”Source · JD item 8Team vs. you
this was a dozen-person project. Expect them to follow your details three layers down to find what you did with your own hands
Source · Resume: “LLM support assistant”Source · JD item 7Knowing where LLMs stop
in e-commerce a wrong price or promo is a complaint and a loss. They want to see you know when not to let the model talk
Source · JD item 4Source · JD item 9Support to shopping
support absorbs problems, a shopping assistant helps people decide. Goals, metrics and the cost of a wrong answer all change. Can you see that unprompted?
Source · JD item 1Source · JD item 10
Know the business
Concepts to brush up on
| Concept | What it is | Where it comes up |
|---|---|---|
| Conversion funnel Source · JD item 5 | How many shoppers remain at each step from visit to chat to cart to order | To talk about the assistant's impact, say which step of the funnel it moves |
| Average order value Source · JD item 5 | The average amount of one order | The JD lists it next to conversion; recommendations move it directly |
| Pre-sale resolution rate Source · JD item 2 | Share of pre-sale questions solved without a human handoff or a repeat contact | Same family as the resolution rate on your resume; the definitions must line up |
| Bad-case loop Source · JD item 2 | Collect wrong answers, find the cause, fix, then re-test | The JD wants you to drive model, retrieval and knowledge-base fixes; this loop is how |
Business points that earn credit Ideas you can raise
Marketplaces usually build a shopping assistant to catch needs shoppers can't put into a search box; the goal is conversion, not cheaper support
Work it in When you describe the support project, add that shopping and support have different goals
Check · Use the product page and chat entry yourselfPlatforms like this typically pilot in high-ticket, high-consideration categories and keep rules in front of costly answers
Work it in When asked about scope for version one, use this to pick a category
Check · Check which categories the team's job posts mentionAcross the industry, shopping assistants are mostly still being proven; whoever can show the return on cost gets the budget
Work it in In your closing or questions, name the number you'd want to pin down first
Check · Their annual report, if public: what it says on AI
Questions they may follow with · 3
- How does the assistant relate to search and recommendations? Do they compete for traffic?AngleStart from when shoppers can't say what they want
- During a big sale, should the assistant be bolder or more careful?AngleWeigh the cost of a wrong answer at peak traffic against call costs
- What will a shopping assistant look like a year from now?AngleGive the evidence you'd go on, then one direction you'd bet on
Industry and company points are AI inferences. Check them before the interview
Your opening minute
A peer mines your intro for follow-ups. Lead with what you built and a number you can defend, not agent exploration
Likely questions and how to answer
01Tell me about yourself
What they want to know
The first two minutes set the frame, and they're scanning for hooks to dig into later
How to answer
Under 90 seconds: who you are, the most relevant stint, why this role. Center it on the Hearthwork support assistant; Freshbasket gets one line. Plant a hook you can defend, like 'I designed the handoff rules'. Don't plant 'some agent exploration'. You only scratched the surface there and one follow-up will show it
02Walk me through the support assistant
What they want to know
They want the big picture and where you stood in it
How to answer
One line of context (volume doubled at peak, old bot couldn't cope), one line of solution (LLM plus knowledge-base retrieval), then spend 70% of the time on your three pieces: knowledge-base cleanup (1,400 FAQs down to 380), the eval set (600 real conversations), handoff rules. One number each. Keep saying 'the team did X, I owned Y', never a blanket 'we'
03How was the 40% resolution lift calculated?
What they want to know
The most common credibility check. People who did the work start with the denominator; people who didn't just repeat the result. A number that shifts mid-interview discounts everything else you said
How to answer
Check the dashboard beforehand and memorize five things: the metric definition, baseline, new value, whether 40% is relative or absolute, and sample size and period. Say them in that order in 30 seconds, then volunteer one limitation (e.g. which conversations were excluded). If you can't find a number, say so. Never estimate on the spot
04What did you own? What did you decide?
What they want to know
They're listening for independent judgment versus execution and coordination
How to answer
Have one decision ready: high-risk intents don't get free-form model answers. Cover the disagreement (someone wanted prompt rules instead), your evidence (which wrong answers dominated the bad cases), the call and the outcome. If you didn't pick the model, say so plainly, then add what you required of it, e.g. every answer must trace back to a policy article
05What do you do when the model is wrong?
What they want to know
The core risk in e-commerce. They're checking whether you think in guardrails or only in wins
How to answer
This is your strongest material. Tier, guard, close the loop: three intent tiers, high-risk can only quote approved text, 200 bad cases a week tagged three ways. Then always name the cost: guardrails push handoff rate up, so how did you weigh that. Finish with the bridge: prices, stacked promos, stock and delivery times are the high-risk intents in shopping
06If you ran our shopping assistant, what would v1 aim for?
What they want to know
Can you carry your experience over, and can you make trade-offs
How to answer
One primary goal: conversations with the assistant convert to orders better than comparable traffic without it. Two guardrails: wrong-answer rate on prices and promos, and handoff rate not rising. Scope to one category and say why that one. Don't list five metrics and five features. They want to hear what you cut
Numbers to have ready
| Number | Where it's from | How they'll probe | Your definition |
|---|---|---|---|
| Resolution rate up 40% | Resume: “Resolution rate up 40%” | First how resolution is defined, then baseline, new value, relative or absolute, sample size and time window | Resolution means [your definition]. Baseline [your number], after launch [your number], so 40% is a [relative / absolute] lift. Sample: [conversations and time window], counting only [which inquiry types] |
| Staffing cost down 30% | Resume: “staffing cost down 30%” | Whether it's fewer handoffs or fewer people, and what it's worth in money | The 30% is a drop in [handoffs / actual headcount], compared against [time window], worth roughly [your amount]. Say plainly whether anyone was cut |
| 72% → 89% | Resume: “Ticket auto-routing” | Where the accuracy sample came from, who reviewed it, how routing time was measured | Accuracy was measured on 2,000 human-reviewed tickets, 72% to 89%. Average routing time went from 26 to 9 minutes, timed from [which step] |
Before the interview
Stories to have ready
- The full high-risk tiering story: trigger (deposit wrongly called refundable), the disagreement, your evidence, the decision, the cost, the result
- Ticket routing: 72% → 89% on 2,000 human-reviewed tickets, routing time 26 → 9 minutes. Your cleanest number; bring it up yourself
- Something you stopped or reworked, e.g. why the agent-based rescheduling didn't continue. A clear reason to stop is judgment too
Check these first
- Pull the dashboard: confirm the resolution definition, baseline and sample, and whether 40% is relative or absolute
- Figure out what 'staffing cost down 30%' really is: fewer handoffs, or fewer people hired, and how much money. Get a version that adds up
- Browse this platform's product pages and help entry points; picture where the shopping assistant would live
Where you're most likely to lose points
- Fuzzy definitions, or swapping numbers under follow-up. That makes every other claim suspect
- Telling the whole story as 'we' and only scrambling to separate your part when asked
- Showcasing RAG, agents and multi-turn as highlights. They'll pick one and ask about cost and trade-offs
Get these done
- Write definition, baseline and sample for both key numbers on one card; say it out loud three times, under 30 seconds each
- Split the support project into three columns: what the team did / what I owned / what I decided
- Cut 'agent exploration' from your intro and put the handoff rules in its place
- Prepare two questions to ask: where the shopping assistant is stuck today, and what this role should deliver in its first three months
Feedback & debrief
After the interview, log what you were asked and how you answered. You get: where each answer lost points, what to say next time, and what the next round will probe
Solid grasp of the project and good risk instincts. Your answer on wrong model outputs was the best of the session and clearly hands-on. Two things cost you. First, the two most visible numbers on your resume fell apart under follow-up: no clear definition, baseline, or relative vs. absolute, and you changed the number mid-answer, the biggest red flag there is. Second, you said 'we' for four questions straight; your best material only surfaced in question five, after the discount was applied. You passed mostly on that answer. Round 2 will take the numbers and your own decisions apart again
01Let's start with a quick intro
Strong
Kept it short, and the renewal number had a clear start and end point
Missed
'LLM plus RAG' and 'agent exploration' are traps you set for yourself. You barely touched agents and one follow-up exposes it. The hook you should have planted (you designed the handoff rules) never came up
Next time
02Walk me through the support assistant project
Strong
Clear context: volume doubled at peak and the old bot couldn't cope, so they understood why it mattered
Missed
Six terms in one breath: vector search, reranking, prompts, multi-turn, intent detection, agents, with no reason for any. All 'we', so they couldn't tell where you stood. Ending on agents set another trap
Next time
03How did you calculate that 40% lift in resolution rate?
Strong
The direction of the definition was right: no handoff counts as resolved, so you know roughly what it measures
Missed
Baseline 'thirties', new value 'seventies', then a retraction; relative vs. absolute unclear; finally handed off to the data team. A shifting number discounts all your other numbers. The prep listed checking this first, and it didn't happen
Next time
04Which parts did you own yourself? Any decision you made?
Strong
You said plainly that you didn't choose the model instead of claiming it. That earns trust
Missed
'Driving execution', 'getting everyone aligned', 'my manager set the direction' all say you had no judgment of your own. You did: the intent tiering in question five was your call, and you didn't think to use it here
Next time
05What happens when the model gets something wrong, say a refund policy?
Strong
Best answer of the session: a real failure (deposits called refundable), tiers, a fallback, and a weekly 200-case review tagged three ways. Detailed enough to survive follow-ups. That's what 'hands-on' sounds like
Missed
No cost or outcome: how far those errors dropped once high-risk went to quoted text, and how much handoffs rose. Guardrails always push handoffs up, and they'll want to know how you weighed it
Next time
06If you ran our shopping assistant, what would the first version aim for?
Strong
You knew conversion matters and thought about remembering preferences. The direction isn't off
Missed
Five metrics, five features, no trade-offs, nothing on how support differs from shopping. A wrong promo price is a direct loss; a weak recommendation is just a missed sale. Different costs mean different metrics and guardrails
Next time
Likely next round
- Walk me through the 40% and the 30% again: definition, baseline, sample, relative or absolute?
- The 30% staffing cost cut: how much money was that? Did headcount go down?
- After the risk tiering, how much did handoffs go up? How did you justify that cost to your manager?
- If a shopper returns something the assistant recommended, is that on the assistant?
- Why didn't the agent-based rescheduling go forward?
- Why move from home-renovation support to e-commerce shopping?
After the debrief, these go into your long-term memory and shape the next prep guide:
- Weak spots Fuzzy number definitions: mixes relative and absolute, no baseline or sample, changes figures under pressure
- Speaking habits Tells projects as 'we' throughout; only separates own work when asked
- Strengths Strong on guardrails: three intent tiers, quote-only for high risk, weekly bad-case tagging; holds up to probing
- Project stories Lead story: no free-form answers on high-risk intents, triggered by a wrong deposit answer; own decision
- Common follow-ups Often asked 'how was this number calculated' and 'which part was yours'
Same question, two answers
Your resume says resolution rate went up 40%. How was that calculated?
A general-purpose AI usually says
For questions like this, use the STAR method: describe the situation, explain your task and the actions you took, then quantify the result with data. Highlight your analytical skills and results-driven mindset, communicate clearly and logically, stay confident, and show deep business understanding and strong teamwork
What you get here
They don't want a prettier number; they're checking whether you did the work. People who did start with the denominator: how resolution is defined, the baseline, the sample, and whether 40% is relative or absolute. Yours is 31% on the old bot to 43% on the new one: 40% relative, 12 points absolute, about 18,000 conversations over a three-week rollout, pre-purchase only. Thirty seconds, then volunteer the caveat. The real danger is changing the number live. In round 1 you went from 'the seventies' back to '40%', and that one wobble discounts every other number you gave
Now try it with your own resume and JD
Create a job file and get a prep guide written for you before your next round, then a debrief right after.