Skip to content
中文Sign in

Example content; the people and companies are fictional. In real use, guides and debriefs are built from your own resume, the JD, and your notes from every round.

AI Product Manager · Round 1 · Peer interview

This round checks whether the numbers on your resume are really yours

SampleExperienced hire · 3-5 yrsAn e-commerce platform (fictional)

The JD wants someone who puts LLMs into the purchase flow and is strict about metrics. A senior peer PM will dig into your support bot for details, numbers and what you did yourself. Strength: you built the handoff rules and guardrails. Risk: the 40% and 30% on your resume with no clear definition

A senior peer PM, most likely on the shopping-assistant team and someone you'd work with daily. They don't care how many terms you know; they care whether you've actually done the work and can pick up tasks on day one, so expect lots of follow-ups on details and numbers

Candidate: 5 years in product, built an AI support bot at a home-renovation platform, now interviewing for a shopping assistant

Show the job description
What you'll do 1. Own the roadmap for our AI shopping assistant and support bot, from pre-purchase questions and product recommendations to post-purchase help, and design how LLMs fit into the purchase flow 2. Define success metrics and the evaluation system: eval sets, a bad-case taxonomy and launch bars; drive iteration across model, retrieval and knowledge base 3. Work with ML, engineering, support ops and category teams to balance answer quality, latency and inference cost 4. Design guardrails and human-handoff rules for high-risk topics (prices, promotions, refunds, delivery times) so wrong answers don't turn into complaints and losses 5. Run staged rollouts and A/B tests to measure impact on conversion, order value and handoff rate What we're looking for 1. 3+ years in product; conversational products, search and recommendations, or support systems a plus 2. Shipped at least one LLM application from zero to launch and can explain the trade-offs and the results 3. Strong with data: you define metrics, pull your own numbers, read conversation logs, and care how a number is calculated 4. A clear sense of what LLMs can and can't do: what to hand to the model and what must stay rule-based 5. E-commerce checkout or transaction experience is a bonus

What this round really tests

  1. Is the experience real

    the '40% resolution lift' and '30% staffing cost cut' on your resume will get taken apart. They want definition, baseline and sample, not a bigger number

    Source · Resume: “Resolution rate up 40%”Source · JD item 8
  2. Team vs. you

    this was a dozen-person project. Expect them to follow your details three layers down to find what you did with your own hands

    Source · Resume: “LLM support assistant”Source · JD item 7
  3. Knowing where LLMs stop

    in e-commerce a wrong price or promo is a complaint and a loss. They want to see you know when not to let the model talk

    Source · JD item 4Source · JD item 9
  4. Support to shopping

    support absorbs problems, a shopping assistant helps people decide. Goals, metrics and the cost of a wrong answer all change. Can you see that unprompted?

    Source · JD item 1Source · JD item 10

Know the business

Concepts to brush up on

ConceptWhat it isWhere it comes up
Conversion funnel
Source · JD item 5
How many shoppers remain at each step from visit to chat to cart to orderTo talk about the assistant's impact, say which step of the funnel it moves
Average order value
Source · JD item 5
The average amount of one orderThe JD lists it next to conversion; recommendations move it directly
Pre-sale resolution rate
Source · JD item 2
Share of pre-sale questions solved without a human handoff or a repeat contactSame family as the resolution rate on your resume; the definitions must line up
Bad-case loop
Source · JD item 2
Collect wrong answers, find the cause, fix, then re-testThe JD wants you to drive model, retrieval and knowledge-base fixes; this loop is how

Business points that earn credit Ideas you can raise

  1. Marketplaces usually build a shopping assistant to catch needs shoppers can't put into a search box; the goal is conversion, not cheaper support

    Work it in When you describe the support project, add that shopping and support have different goals

    Check · Use the product page and chat entry yourself
  2. Platforms like this typically pilot in high-ticket, high-consideration categories and keep rules in front of costly answers

    Work it in When asked about scope for version one, use this to pick a category

    Check · Check which categories the team's job posts mention
  3. Across the industry, shopping assistants are mostly still being proven; whoever can show the return on cost gets the budget

    Work it in In your closing or questions, name the number you'd want to pin down first

    Check · Their annual report, if public: what it says on AI
Questions they may follow with · 3
  • How does the assistant relate to search and recommendations? Do they compete for traffic?AngleStart from when shoppers can't say what they want
  • During a big sale, should the assistant be bolder or more careful?AngleWeigh the cost of a wrong answer at peak traffic against call costs
  • What will a shopping assistant look like a year from now?AngleGive the evidence you'd go on, then one direction you'd bet on

Industry and company points are AI inferences. Check them before the interview

Your opening minute

A peer mines your intro for follow-ups. Lead with what you built and a number you can defend, not agent exploration

About 60 seconds“Hi, I'm Ke Lin, five years in product. The last two at Hearthwork on our LLM support assistant: ML owned the model and retrieval, I owned knowledge-base cleanup, the eval set and the handoff rules. I cut 1,400 scattered FAQs to 380 articles, each with an owner and an expiry date, and built a 600-conversation eval set. Resolution went up 40%, measured as [your definition and baseline]. Before that, at Freshbasket, I took membership renewal from 29% to 36%. Shopping is closer to the sale than support and mistakes cost more, so that guardrail work is what I'd bring”

Likely questions and how to answer

01

Tell me about yourself

What they want to know

The first two minutes set the frame, and they're scanning for hooks to dig into later

How to answer

Under 90 seconds: who you are, the most relevant stint, why this role. Center it on the Hearthwork support assistant; Freshbasket gets one line. Plant a hook you can defend, like 'I designed the handoff rules'. Don't plant 'some agent exploration'. You only scratched the surface there and one follow-up will show it

02

Walk me through the support assistant

What they want to know

They want the big picture and where you stood in it

How to answer

One line of context (volume doubled at peak, old bot couldn't cope), one line of solution (LLM plus knowledge-base retrieval), then spend 70% of the time on your three pieces: knowledge-base cleanup (1,400 FAQs down to 380), the eval set (600 real conversations), handoff rules. One number each. Keep saying 'the team did X, I owned Y', never a blanket 'we'

Source · Resume: “LLM support assistant”Source · JD item 7
03

How was the 40% resolution lift calculated?

What they want to know

The most common credibility check. People who did the work start with the denominator; people who didn't just repeat the result. A number that shifts mid-interview discounts everything else you said

How to answer

Check the dashboard beforehand and memorize five things: the metric definition, baseline, new value, whether 40% is relative or absolute, and sample size and period. Say them in that order in 30 seconds, then volunteer one limitation (e.g. which conversations were excluded). If you can't find a number, say so. Never estimate on the spot

Source · Resume: “Resolution rate up 40%”Source · JD item 8
04

What did you own? What did you decide?

What they want to know

They're listening for independent judgment versus execution and coordination

How to answer

Have one decision ready: high-risk intents don't get free-form model answers. Cover the disagreement (someone wanted prompt rules instead), your evidence (which wrong answers dominated the bad cases), the call and the outcome. If you didn't pick the model, say so plainly, then add what you required of it, e.g. every answer must trace back to a policy article

Source · Resume: “human-handoff rules”
05

What do you do when the model is wrong?

What they want to know

The core risk in e-commerce. They're checking whether you think in guardrails or only in wins

How to answer

This is your strongest material. Tier, guard, close the loop: three intent tiers, high-risk can only quote approved text, 200 bad cases a week tagged three ways. Then always name the cost: guardrails push handoff rate up, so how did you weigh that. Finish with the bridge: prices, stacked promos, stock and delivery times are the high-risk intents in shopping

Source · JD item 4Source · JD item 9
06

If you ran our shopping assistant, what would v1 aim for?

What they want to know

Can you carry your experience over, and can you make trade-offs

How to answer

One primary goal: conversations with the assistant convert to orders better than comparable traffic without it. Two guardrails: wrong-answer rate on prices and promos, and handoff rate not rising. Scope to one category and say why that one. Don't list five metrics and five features. They want to hear what you cut

Source · JD item 1Source · JD item 5

Numbers to have ready

NumberWhere it's fromHow they'll probeYour definition
Resolution rate up 40%Resume: “Resolution rate up 40%”First how resolution is defined, then baseline, new value, relative or absolute, sample size and time windowResolution means [your definition]. Baseline [your number], after launch [your number], so 40% is a [relative / absolute] lift. Sample: [conversations and time window], counting only [which inquiry types]
Staffing cost down 30%Resume: “staffing cost down 30%”Whether it's fewer handoffs or fewer people, and what it's worth in moneyThe 30% is a drop in [handoffs / actual headcount], compared against [time window], worth roughly [your amount]. Say plainly whether anyone was cut
72% → 89%Resume: “Ticket auto-routing”Where the accuracy sample came from, who reviewed it, how routing time was measuredAccuracy was measured on 2,000 human-reviewed tickets, 72% to 89%. Average routing time went from 26 to 9 minutes, timed from [which step]

Before the interview

Stories to have ready

  • The full high-risk tiering story: trigger (deposit wrongly called refundable), the disagreement, your evidence, the decision, the cost, the result
  • Ticket routing: 72% → 89% on 2,000 human-reviewed tickets, routing time 26 → 9 minutes. Your cleanest number; bring it up yourself
  • Something you stopped or reworked, e.g. why the agent-based rescheduling didn't continue. A clear reason to stop is judgment too

Check these first

  • Pull the dashboard: confirm the resolution definition, baseline and sample, and whether 40% is relative or absolute
  • Figure out what 'staffing cost down 30%' really is: fewer handoffs, or fewer people hired, and how much money. Get a version that adds up
  • Browse this platform's product pages and help entry points; picture where the shopping assistant would live

Where you're most likely to lose points

  • Fuzzy definitions, or swapping numbers under follow-up. That makes every other claim suspect
  • Telling the whole story as 'we' and only scrambling to separate your part when asked
  • Showcasing RAG, agents and multi-turn as highlights. They'll pick one and ask about cost and trade-offs

Get these done

  • Write definition, baseline and sample for both key numbers on one card; say it out loud three times, under 30 seconds each
  • Split the support project into three columns: what the team did / what I owned / what I decided
  • Cut 'agent exploration' from your intro and put the handoff rules in its place
  • Prepare two questions to ask: where the shopping assistant is stuck today, and what this role should deliver in its first three months

Feedback & debrief

After the interview, log what you were asked and how you answered. You get: where each answer lost points, what to say next time, and what the next round will probe

3.0/ 5
Structure3
Credible2
Depth3
Role fit3
Delivery4

Solid grasp of the project and good risk instincts. Your answer on wrong model outputs was the best of the session and clearly hands-on. Two things cost you. First, the two most visible numbers on your resume fell apart under follow-up: no clear definition, baseline, or relative vs. absolute, and you changed the number mid-answer, the biggest red flag there is. Second, you said 'we' for four questions straight; your best material only surfaced in question five, after the discount was applied. You passed mostly on that answer. Round 2 will take the numbers and your own decisions apart again

01

Let's start with a quick intro

Strong

Kept it short, and the renewal number had a clear start and end point

Missed

'LLM plus RAG' and 'agent exploration' are traps you set for yourself. You barely touched agents and one follow-up exposes it. The hook you should have planted (you designed the handoff rules) never came up

Next time

'Five years in product, the last two building an AI support assistant at Hearthwork. I owned three pieces: knowledge base, eval set and handoff rules. Keeping the model from free-styling on high-risk questions was a call I pushed through. Before that, membership at Freshbasket, renewal from 29% to 36%. Support and shopping both help people decide inside a conversation, and I want to bring what I learned the hard way'
02

Walk me through the support assistant project

Strong

Clear context: volume doubled at peak and the old bot couldn't cope, so they understood why it mattered

Missed

Six terms in one breath: vector search, reranking, prompts, multi-turn, intent detection, agents, with no reason for any. All 'we', so they couldn't tell where you stood. Ending on agents set another trap

Next time

One line of context, one line of solution: 'LLM plus knowledge-base retrieval; ML owned retrieval and the model.' Then 70% on your three pieces: 'I cut 1,400 scattered FAQs to 380 articles, each with an owner and expiry date. I built a 600-conversation eval set from real chats that every change runs against. And I designed risk-tiered handoff rules.' One number each, then let them dig in
03

How did you calculate that 40% lift in resolution rate?

Strong

The direction of the definition was right: no handoff counts as resolved, so you know roughly what it measures

Missed

Baseline 'thirties', new value 'seventies', then a retraction; relative vs. absolute unclear; finally handed off to the data team. A shifting number discounts all your other numbers. The prep listed checking this first, and it didn't happen

Next time

'Resolution means no handoff and no repeat contact on the same issue within 24 hours. The old keyword bot was at 31%; after a three-week rollout the assistant hit 43%. That's 12 points absolute; the 40% on my resume is relative. Sample: 20% of traffic, about 18,000 conversations. One caveat: pre-purchase only, post-purchase wasn't counted.' Thirty seconds, caveat volunteered
04

Which parts did you own yourself? Any decision you made?

Strong

You said plainly that you didn't choose the model instead of claiming it. That earns trust

Missed

'Driving execution', 'getting everyone aligned', 'my manager set the direction' all say you had no judgment of your own. You did: the intent tiering in question five was your call, and you didn't think to use it here

Next time

'Three kinds of decisions. ML picked the model and retrieval; my requirement was that every answer trace back to a policy article. I owned the knowledge base and eval set. My call: no free-form answers on high-risk intents. ML wanted prompt rules. I went through two weeks of bad cases; deposit and refund errors were over half the high-risk ones, too many for prompts to hold. So: quote approved text or hand off'
05

What happens when the model gets something wrong, say a refund policy?

Strong

Best answer of the session: a real failure (deposits called refundable), tiers, a fallback, and a weekly 200-case review tagged three ways. Detailed enough to survive follow-ups. That's what 'hands-on' sounds like

Missed

No cost or outcome: how far those errors dropped once high-risk went to quoted text, and how much handoffs rose. Guardrails always push handoffs up, and they'll want to know how you weighed it

Next time

Add two lines: 'The cost was a higher handoff rate on those conversations, and we accepted it, because one wrong refund answer costs far more to clean up than one handoff. In shopping, the high-risk intents are prices, stacked promos, stock and delivery times.' Look up both numbers first: how much those errors dropped, and how many points handoffs rose
06

If you ran our shopping assistant, what would the first version aim for?

Strong

You knew conversion matters and thought about remembering preferences. The direction isn't off

Missed

Five metrics, five features, no trade-offs, nothing on how support differs from shopping. A wrong promo price is a direct loss; a weak recommendation is just a missed sale. Different costs mean different metrics and guardrails

Next time

'V1 gets one primary goal: conversations with the assistant convert better than comparable traffic without it. Two guardrails: near-zero wrong answers on price and promos, and no rise in handoffs. One high-ticket, high-consideration category, say large appliances, where people ask and compare the most. Prices and promos always come live from the catalog; the model never generates a number'

Likely next round

  • Walk me through the 40% and the 30% again: definition, baseline, sample, relative or absolute?
  • The 30% staffing cost cut: how much money was that? Did headcount go down?
  • After the risk tiering, how much did handoffs go up? How did you justify that cost to your manager?
  • If a shopper returns something the assistant recommended, is that on the assistant?
  • Why didn't the agent-based rescheduling go forward?
  • Why move from home-renovation support to e-commerce shopping?

After the debrief, these go into your long-term memory and shape the next prep guide:

  • Weak spots Fuzzy number definitions: mixes relative and absolute, no baseline or sample, changes figures under pressure
  • Speaking habits Tells projects as 'we' throughout; only separates own work when asked
  • Strengths Strong on guardrails: three intent tiers, quote-only for high risk, weekly bad-case tagging; holds up to probing
  • Project stories Lead story: no free-form answers on high-risk intents, triggered by a wrong deposit answer; own decision
  • Common follow-ups Often asked 'how was this number calculated' and 'which part was yours'

Same question, two answers

Your resume says resolution rate went up 40%. How was that calculated?

A general-purpose AI usually says

For questions like this, use the STAR method: describe the situation, explain your task and the actions you took, then quantify the result with data. Highlight your analytical skills and results-driven mindset, communicate clearly and logically, stay confident, and show deep business understanding and strong teamwork

What you get here

They don't want a prettier number; they're checking whether you did the work. People who did start with the denominator: how resolution is defined, the baseline, the sample, and whether 40% is relative or absolute. Yours is 31% on the old bot to 43% on the new one: 40% relative, 12 points absolute, about 18,000 conversations over a three-week rollout, pre-purchase only. Thirty seconds, then volunteer the caveat. The real danger is changing the number live. In round 1 you went from 'the seventies' back to '40%', and that one wobble discounts every other number you gave

Now try it with your own resume and JD

Create a job file and get a prep guide written for you before your next round, then a debrief right after.