3 Prompting lab: getting good answers, and knowing their limits
About 60 minutes
- The idea. A few extra sentences in your prompt (who it is for, the case, the task, the format, and “say what is missing”) get you an answer organised the way you think on rounds. That answer is faster to read and easier to check. It is not always right, so you check the one claim the plan depends on.
- What you do. Build a prompt for a task you need this month and keep the version that works. Then run one made-up case twice, bare and structured, and score both.
- What you keep. A five-part prompt template, saved in your “AI in Medicine” project, and one sentence for question 2 of your learning journal.
- Short on time? Read “Night float” and “The five-part prompt”, then do Part A (20 minutes). Part B can wait for later in the week.
By the end of this chapter you will have:
- Written a five-part prompt and kept it as a template you will use again.
- Changed one part at a time and seen what each part does to the answer.
- Run one made-up case twice in one tool, bare and structured, and said what the structure improved and what it did not.
- Checked the one claim a plan depends on against a calculator, a guideline, or a drug label.
Course competencies: AIM-5 (clinical use), AIM-2 (critical appraisal).
What you need. Your university ChatGPT Edu or Copilot account. Nothing to install. Made-up patients only. A blank document for notes. This chapter goes with Monday’s talk, “Getting good answers”: read it after the talk, or instead of it. The template you build here is the one to use for Learn it with AI this week.
3.1 Night float, 2:40 a.m.
You are the intern on night float. Your co-intern Sam comes back from seeing a 44-year-old woman in the emergency department. She is three weeks out from surgery to fix a broken ankle. She has had two days of sharp chest pain that is worse when she breathes in. Her heart rate is 108, her oxygen saturation is 95 percent, and her right calf is bigger than the left. Sam has already asked the chatbot on their phone, twice. The first time they typed the question the way you would say it out loud. The second time, after a lecture on prompting, they wrote it out properly: who they are, the case in clinical language, exactly what they want, and the format they want it in.
Sam hands you the phone. “Second one’s much better. I’m going to put it in the note.”
This presentation is concerning for a pulmonary embolism (PE), especially given the recent surgery, tachycardia, pleuritic pain, and unilateral calf swelling. Other possibilities include pneumonia, musculoskeletal pain, pericarditis, and pneumothorax. I would recommend getting an ECG and chest X-ray, checking a troponin, and ordering a D-dimer; if the D-dimer is negative, PE can be reasonably excluded. If it is elevated, proceed to CT pulmonary angiography. Consider a lower-extremity Doppler ultrasound for the calf swelling. Provide supplemental oxygen as needed and analgesia.
| # | Diagnosis | Supports | Opposes | Next step |
|---|---|---|---|---|
| 1 | Pulmonary embolism | Recent surgery, tachycardia, pleuritic pain, unilateral calf swelling | Normal BP, only mild hypoxaemia | D-dimer; CT pulmonary angiography if positive |
| 2 | Deep vein thrombosis without PE | Calf asymmetry, immobility | Chest symptoms present | Compression ultrasound |
| 3 | Pneumonia | Pleuritic pain | Afebrile, no cough | Chest X-ray |
| 4 | Pericarditis | Pleuritic pain | No positional component described | ECG |
| 5 | Musculoskeletal | Recent immobility | Tachycardia, hypoxaemia | Diagnosis of exclusion |
Overall: PE is most likely. Obtain a D-dimer; a negative result effectively rules out PE in this setting.
Read both before going on. Sam is right that the second answer is better. It is ranked, it gives the findings for and against each diagnosis, and it names a next step in every row. That change came from a few extra sentences in the prompt. Writing those sentences is the skill this chapter teaches.
Both answers also make the same mistake about the D-dimer, and the table makes it harder to see. By the end of the chapter you will know the line that belongs in Sam’s prompt, and the check that catches the error.
The skill has a name: prompting, writing what you type to a language model so that what comes back is more useful. A language model is the program behind ChatGPT, Copilot, Claude, and Gemini. It writes text one word-piece (a word or part of a word) at a time. The words you give it are the only thing it knows about your case.
It is your job to learn where these tools help your work, and you learn that by trying. If you would rather try first and read after, open your chatbot now and do Part A of the Do.
3.2 Why this matters
By next July you will be Sam. You will have a case, a phone, and a minute, and what you type will change what comes back.
Asking well takes a few minutes to learn, and you will use it every day. It gets you answers organised the way you think on rounds, that show their reasoning, and that say what they are missing. Used this way, the tool helps. In a randomized trial of 92 physicians working through management cases, those given GPT-4 scored higher than those with the usual resources alone (Goh et al., 2025). Checking is one step in that routine, not a reason to avoid the tool.
The skill is new to most students. In a study of 148 final-year medical students, only 5 percent had heard the phrase “clinical prompt engineering”. The students judged GPT-3.5’s answers to ten written cases, five of which the model had got wrong. On a typical answer, only 56 percent of the students judged correctly whether it was right (Waldock et al., 2025).
3.3 How it works
3.3.1 The five-part prompt
Learn the parts, not fixed wording.
| Element | What you write | Example |
|---|---|---|
| Role | Who the model is helping | “You are assisting a physician preparing for a case discussion.” |
| Context | The case, in clinical language | “Epigastric pain radiating to the back,” not “stomach pain.” Vitals, key negatives, medications. |
| Task | Exactly what you want | “Give a ranked differential with the three findings that most support and most oppose each entry.” |
| Format | The shape of the answer | “Table. One row per diagnosis.” |
| Uncertainty | What to do with gaps | “Where the case lacks information you would need, say what is missing instead of assuming it.” |
The first four rows are standard prompting advice. The fifth is this chapter’s addition: models tend to fill a gap with a complete-looking answer, and asking them to name the gap instead is the most useful single line a clinician can add, a claim you will test yourself in the Do.
One 2026 study asked chatbots for references on spine surgery in two ways: as a spine surgeon writing in clinical language, and as a patient writing in everyday words. Asked as a patient, ChatGPT made up references about ten times as often (McLaughlin et al., 2026). The study changed who was asking and the wording at the same time. It did not test people writing in a second language. So take it as a reason for a habit that helps everyone: say who you are, and write the case the way you would present it on rounds. The Context row of Table 3.1 is where you do that.
3.3.2 What structure changes, and what it does not
A prompt can change how an answer is organised: ranked or not, with findings for and against, with a next step, with what it does not know. This is almost entirely under your control. Ask for a table and you get a table. That is the reliable part of prompting, and it is worth having: an organised answer is faster to read and easier to check.
A prompt can also, sometimes, change whether an answer is correct. These are two different properties, and they do not change together. Correctness is limited by two things a prompt does not change: what the model knows, and what the case contains. A prompt cannot supply the piece of history you left out.
What a structured prompt can do is make the model show its reasoning, so that you can check it. When a Stanford study prompted GPT-4 to imitate styles of clinical reasoning, diagnostic accuracy did not change, but the reasoning became something a physician could read and judge (Savage et al., 2024). Prompt effects on accuracy are real, but they differ from model to model and they are unstable (studies in Go deeper). Better organisation is the effect that is reliable.
3.3.3 The error in Sam’s answer
Go back to Sam’s phone. A negative D-dimer rules out pulmonary embolism only when the pretest probability is low. Pretest probability is how likely the diagnosis is before you order any test. The original emergency-department study validated that strategy in patients whose clinical score put them in the low-probability group (Wells et al., 2001). Sam’s patient has had surgery in the last four weeks, has a heart rate over 100, and has a swollen calf. On the Wells score, which MDCalc will calculate for you, she is not low probability. The right next step is imaging, not a D-dimer, because a positive D-dimer would only delay the scan. Newer pathways, such as a two-level Wells score (“PE likely” or “PE unlikely”) or a D-dimer cutoff that rises with age, do not change this for Sam: she scores “PE likely”, and she is under 50. Both answers got this wrong. Only one of them put it in a column headed “Next step” beside a column of supporting findings.
That is why the neat answer needs more care, not less. Automation bias is trusting a machine’s suggestion more than it deserves, and even more when the output looks authoritative. You read the row, you agree with the findings, and you accept the next step without a separate check.
Two habits would have protected Sam. The fifth line of the prompt gives you a place to start, but it does not make the model catch its own error. The check does: test the one claim the plan depends on, here “a negative D-dimer rules it out”, against a source. For Sam, that is the Wells score in MDCalc.
A structured prompt gets you a better-organised answer that is easier to read and to check. It does not reliably make the answer right. Ask it to say what is missing, then check the one claim the plan depends on before anything goes out with your name on it.
3.4 Watch or listen (optional)
Podcast. NEJM AI Grand Rounds, “Medicine, Machines, and Magic: Dr. Jonathan Chen on Medical AI” (15 October 2025; 48 min). https://ai-podcast.nejm.org/e/medicine-machines-and-magic-dr-jonathan-chen-on-medical-ai/ Chen is an author of both randomized trials in Go deeper, and senior author of the first. The first twenty minutes are enough.
3.5 Do
Made-up patients only. What counts as patient information.
Run every prompt in this Do in a Temporary Chat, or a chat outside your course project. ChatGPT can remember earlier chats and read what is in a project. A fresh chat keeps each comparison fair.
3.5.1 Part A: build a prompt that works (about 20 minutes)
Pick something you will need in the next month: a topic you will present on rounds, a condition you want to explain to patients, or a question you expect at interviews. Fill in the brackets and run this in a Temporary Chat, or a chat outside your course project.
You are helping a fourth-year medical student going into [specialty].
Context: I am preparing a five-minute teaching talk on [topic] for
third-year students on their [specialty] rotation. They already know
[what they know].
Task: give me the three points they most need, one common mistake, and
one question to ask them at the end.
Format: a bulleted list, under 200 words.
Where you are unsure, or where guidelines differ or have changed
recently, say so instead of picking one.Then change one part at a time, each in a new Temporary Chat, or a new chat outside your course project, and compare:
- Change the context: make the audience patients, or an attending.
- Change the format: a table, one paragraph, or three slides.
- Delete the last line. What disappeared from the answer?
Write one line for each change: what it did. Keep the version you liked best. That is the start of your template.
3.5.2 Part B: one case, two prompts (about 20 minutes)
You will run one made-up case twice and score both answers. Use the same chatbot for both runs, each in its own Temporary Chat, or a chat outside your course project, so the second run cannot see the first.
3.5.2.1 Step 1. Pick a case (2 minutes)
Choose by the specialty you plan to enter, or the one whose trap you least expect to catch. Each case has one deliberate trap. The traps are of different kinds, and you will not be told which kind yours is until Step 4.
A 31-year-old woman presents with two days of right-sided chest pain that is sharp and worse on inspiration, and shortness of breath when climbing stairs. Heart rate 102, blood pressure 118/74, respiratory rate 20, oxygen saturation 96 percent on room air, temperature 37.1 °C. Lungs are clear. No cough, no haemoptysis, no leg swelling or calf tenderness. She describes herself as otherwise healthy and takes no regular medication that she mentions. She has not travelled recently.
An 82-year-old woman is brought from an assisted-living facility with two days of fluctuating confusion; staff say she is “not herself” and was found wandering at night. Temperature 36.9 °C, heart rate 84, blood pressure 138/78, oxygen saturation 97 percent. She is inattentive and disoriented to time but has no focal neurological findings. No dysuria, frequency, or flank pain reported; no cough. Urine dipstick: leukocyte esterase positive, nitrite positive. White cell count 8.1. Glucose, sodium, and calcium are normal. Medications: amlodipine 5 mg daily, atorvastatin 20 mg nightly, sertraline 50 mg daily, acetaminophen as needed, and oxybutynin 5 mg twice daily, started twelve days ago for urinary urgency.
A 66-year-old man attends pre-operative clinic ten days before elective repair of a right inguinal hernia. Past history: paroxysmal atrial fibrillation, hyperlipidaemia, type 2 diabetes, and “bronchitis” for which an urgent-care clinic started an antibiotic last week. Examination is unremarkable apart from the reducible hernia. Medications: apixaban 5 mg twice daily, amiodarone 200 mg daily, simvastatin 40 mg nightly, metformin 1000 mg twice daily, tamsulosin 0.4 mg daily, clarithromycin 500 mg twice daily (day 4 of 7). Question for the model: what needs attention before this operation?
A 23-year-old man is brought in by his roommate. For ten days he has slept two to three hours a night without feeling tired, talks rapidly and jumps between topics, and has spent about $4,000 on equipment for a business he describes in grand terms. For the last three days he has heard a voice commenting on what he is doing. Three weeks ago he was started on prednisone 60 mg daily for a severe contact dermatitis and is now on a taper at 30 mg. No prior psychiatric history. A maternal uncle has bipolar disorder. He uses cannabis most weekends. Vital signs and neurological examination are normal.
3.5.2.2 Step 2. The bare prompt (3 minutes)
Paste the case into a Temporary Chat, or a chat outside your course project, with nothing but this in front of it:
What’s the diagnosis and what should I do?
Copy the whole answer, word for word, into your document under Bare. If the tool replies with questions instead of a plan, do not answer them. The questions are the answer; copy them and score them like any other reply.
3.5.2.3 Step 3. The structured prompt (5 minutes)
Open a second Temporary Chat, or another new chat outside your project. Here is the five-part template from Table 3.1; fill in the case where marked.
You are assisting a physician preparing to present this case at morning report. Case:
<paste the case, unchanged>. Give a ranked differential diagnosis with, for each entry, the two or three findings that most support it and the two or three that most oppose it, then the single best next step. Format as a table, one row per diagnosis, followed by one paragraph on management. Where the case lacks information you would need to decide, say what is missing instead of assuming it.
Copy the answer word for word under Structured. Note the tool and, if it shows one, the model version.
3.5.2.4 Step 4. Score both (10 minutes)
Now open the trap for your case, below, and score each answer twice: once for organisation, once for correctness on the trap.
Case A: a critical omission. The case never says whether she is pregnant, whether she takes hormonal contraception or other oestrogen, whether she has been immobile, or whether she or her family have had a clot before. Every one of those changes the pretest probability of pulmonary embolism, and no score can be computed without them. The trap is whether the model asks for them or assumes them. An answer that computes a Wells score, or declares her “low risk”, from what is given has invented the missing history. Full marks require the answer to name at least two of the missing items as things it needs before choosing between D-dimer and imaging.
Case B: the common answer is wrong. The pattern-match is “urinary tract infection causing delirium; start antibiotics.” But bacteria in the urine without urinary symptoms is common in older women, especially in care facilities. The Infectious Diseases Society of America’s 2019 guideline specifically addresses older adults with delirium and no local urinary symptoms: look for other causes and observe rather than treat the urine (Nicolle et al., 2019). The clue that separates the diagnoses is in the medication list. Oxybutynin is a strongly anticholinergic drug. The American Geriatrics Society Beers Criteria list it as one to avoid in older adults, and it was started twelve days before the confusion (“American Geriatrics Society 2023 Updated AGS Beers Criteria® for Potentially Inappropriate Medication Use in Older Adults.” 2023). Full marks require the answer to put the new anticholinergic ahead of, or at least beside, the dipstick, and to say that the dipstick alone does not establish infection.
Case C: interactions hidden in the list. Clarithromycin strongly blocks the liver enzyme that clears simvastatin. The simvastatin label lists clarithromycin as a contraindication, and it caps the dose at 20 mg a day with amiodarone, so 40 mg is over the limit (DailyMed label). The American Heart Association’s statement on statin interactions says the same (Wiggins et al., 2016). Clarithromycin also raises apixaban exposure through the same pathway, and amiodarone with clarithromycin puts two QT-prolonging drugs together. The perioperative apixaban plan and metformin around surgery are the expected content. The trap is whether the model points out the interactions without being asked. Full marks require the statin-macrolide interaction to be named and acted on.
Case D: genuinely ambiguous. The correct answer is that two diagnoses remain: corticosteroid-induced mania with psychotic features, and a first episode of bipolar I disorder with psychotic features, with a cannabis contribution to exclude. What separates them is time. Symptoms that began after the steroid started and resolve after it stops point one way; symptoms that persist point the other. The family history raises the baseline risk without settling it. Psychiatric effects of corticosteroids are common, dose-related, usually early in a course, and usually resolve with reduction or withdrawal (Warrington & Bostwick, 2006). A model that declares “bipolar I disorder” as the diagnosis has decided too early. Full marks require both diagnoses to stay on the list with the distinguishing test named, and a safety plan that does not depend on which it is.
Organisation, 0 to 5. One point for each: ranked; findings for and against each entry; an explicit next step; a format you could read aloud on rounds; a statement of what information is missing.
Correctness on the trap, 2/1/0.
| Score | Meaning |
|---|---|
| 2 | The trap is caught and it changes the plan |
| 1 | The trap is mentioned but hidden, softened, or does not change the plan |
| 0 | The trap is missed, or the answer invents around it |
Write the scores in your document:
Tool: ______________ Case: A / B / C / D
Bare Structured
Organisation (0 to 5) ___ ___
Trap (2/1/0) ___ ___If your organisation score rose and your trap score did not, you have seen the main point of this chapter yourself. If both rose, write down which sentence did it; this chapter’s guess is the last one. If the bare answer caught the trap and the structured one did not, keep both transcripts. One case is two observations, not a study, because the same prompt does not always give the same answer (Wang et al., 2024).
3.5.2.5 Step 5, if you have time. Check the claim the plan depends on (10 minutes)
Take the better of your two answers and find the one claim its plan depends on: a threshold, a first-line drug, a “rules out”, an interaction, a dose. Check it against one source, in a browser:
- A calculator or score: MDCalc, for anything with a score or a threshold.
- A guideline: search PubMed with
<condition> AND (guideline[Publication Type] OR "practice guideline"), filtered to the last ten years, and take the society’s guideline, not a commentary on it. - A drug label: DailyMed (https://dailymed.nlm.nih.gov), the FDA’s label repository, for any interaction, contraindication, or dose.
Record it as supported, contradicted, or not addressed, with the source. “Not addressed” is a real result. A claim the sources do not make is a claim you should not be repeating either. Another chatbot is not a source.
No time now? Do this check later in the week, on an answer from your own work, and write it up in question 2 of your journal.
3.5.3 What you keep (5 minutes)
Your template. Write out the version from Part A you will use again, in this shape. Keep it in your “AI in Medicine” project, in a chat or a file there, so you can copy it next time.
Role: __________________________________________________
Context: __________________________________________________
Task: __________________________________________________
Format: __________________________________________________
Missing: Where the case lacks information you would need,
say what is missing instead of assuming it.
(or your Part A last line)Your finding. One sentence: what did structure change, and what did it not change? Put it in question 2 of this week’s learning journal, with the claim you checked, the source, and what you found.
3.6 Check
Score yourself against this before moving on.
| You have | Meets | Falls short |
|---|---|---|
| A kept template (objective 1) | All five parts of Table 3.1, and you can say which line is the fifth and what it is for | Four parts, or the fifth is “be accurate” |
| Parts changed one at a time (objective 2) | One line each on what changing the context, the format, and the last line did | One run, no changes |
| Two scored answers (objective 3) | Organisation and trap scored separately, the kind of trap named, and a one-sentence finding that names what changed and what did not | One overall impression of which answer was “better” |
| One checked claim (objective 4), here or on your own work | A named calculator, guideline, or label, and a verdict | “It matched what I remembered”, or a second chatbot |
3.7 What this changes for you
What does this change about how you’ll practice? Write it down before you close the chapter. Two specifics.
First, it is 2:40 a.m. again. Sam holds out the phone: “Second one’s much better. I’m going to put it in the note.” Answer Sam in three sentences. Say what really is better about the second answer, what you would check before it goes in the note, which line you would add to the prompt next time, and what you would say about typing a real patient into the phone.
Second, name one task in your own work next week where you will use the template you kept. Then name the one claim in that task you would check, and the source you would check it against.
3.8 Go deeper
What the trials show about physicians using the tool
How you use the answer matters too, not only how you ask. In a randomized trial of 50 physicians working through written cases, those given a language model scored no better than those without it. The model on its own, in an exploratory comparison, scored 16 points higher (Goh et al., 2024) (Figure 3.1). In a second trial by the same group, on management, 92 physicians with the model scored 6.5 points higher than those without (95 percent CI 2.7 to 10.2). The model alone scored about the same as them (no significant difference), and the physicians with the model also took about two minutes longer per case (Goh et al., 2025). Read the discussion sections for the authors’ own account of why the first trial showed no benefit.
Papers
- A single-centre observational study in which pharmacists ran 50 internal-medicine cases through ChatGPT and OpenEvidence, with and without a prompt template. The template made no significant difference to accuracy and completeness: the probability of an acceptable answer went from 0.54 to 0.60 for ChatGPT and from 0.64 to 0.52 for OpenEvidence (Yang et al., 2026).
- An orthopaedics study that asked several models about osteoarthritis guideline recommendations in several prompt styles. The best style depended on the model, and the same question asked five times gave answers whose agreement with each other ranged from none to near-perfect (Wang et al., 2024).
- A low-back-pain study in which targeted safety prompts improved the weaker model on every measure, and the stronger model on everything except safety (Luo et al., 2026).
- Diagnostic-reasoning prompts as a route to checkable answers (Savage et al., 2024), and the psychiatry group’s prompt built from reusable parts, where structure did improve the weaker model’s accuracy at pulling facts out of text (Verhees et al., 2026): the counter-example that limits this chapter’s claim.
- When 33 physicians across 17 specialties graded chatbot answers to 284 of their own questions, the median answer was almost completely correct, but 36 answers were rated mostly or completely wrong (Goodman et al., 2023).
- When oncologists checked chatbot cancer-treatment recommendations against guidelines, about a third of outputs mixed at least one non-guideline recommendation in among correct ones (Chen et al., 2023).
- The study of students judging GPT-3.5’s answers, behind the 56 percent (Waldock et al., 2025), and the published, reusable case-based LLM workshop from Harvard (Choy et al., 2026); this chapter’s Part B is a shorter version of it.
- A tutorial on prompt engineering written for health professionals, for the ten-rules material in article form (Meskó, 2023).
Listen
- NEJM AI Grand Rounds, “AI and the Evolution of Medical Thought with Dr. Adam Rodman” (19 June 2024; 53 min). Rodman is an author of both trials above; he talks about clinical reasoning and why he thinks the models reason at all. https://ai-podcast.nejm.org/e/ai-and-the-evolution-of-medical-thought-with-dr-adam-rodman
Talks
- Prompt engineering, ten rules with clinical examples, about fifteen slides. The Role, Context, Task, and Format rows of Table 3.1 come from it.
- AI in medical education, for the attribution question that arises the first time a chatbot’s table ends up in a presentation or a paper.