Show Me The Evidence Transistor (https://transistor.fm) https://feeds.transistor.fm/show-me-the-evidence Most training is sold on confidence. Show Me The Evidence is built on data. In every episode we take a single study, clinical trial, or systematic review and work through what it found, how it was designed, and what it means for the way we teach and assess skill. We focus on metrics-based training and proficiency-based progression, the approach that asks learners to demonstrate measurable competence before moving on, and we trace its results across surgical, medical, and professional education. This is a podcast for learning professionals and medical educators who want more than opinion. Expect plain-language breakdowns of the research, honest discussion of what the evidence does and does not support, and conversations with the people behind the studies. If you make decisions about how people are trained, we think you deserve to see the evidence first. 2026 OGC Metrics and Anthony G Gallahger. c7a3ba32-a396-5c86-a6e3-597f17500640 yes en Sun, 16 Aug 2026 08:00:21 +0100 Sun, 16 Aug 2026 08:01:36 +0100 https://img.transistorcdn.com/gpHp02Yy04HonZV1huOO8X8dWULaUCN8tveDOjkUxDw/rs:fill:0:0:1/w:1400/h:1400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS9lZGMy/MjkyM2UxYTkzZmM5/ZjVkMmI4MzAzOTM0/YTIzYi5wbmc.jpg Show Me The Evidence episodic Anthony G. Gallagher, Flux Learning Ltd Most training is sold on confidence. Show Me The Evidence is built on data. In every episode we take a single study, clinical trial, or systematic review and work through what it found, how it was designed, and what it means for the way we teach and assess skill. We focus on metrics-based training and proficiency-based progression, the approach that asks learners to demonstrate measurable competence before moving on, and we trace its results across surgical, medical, and professional education. This is a podcast for learning professionals and medical educators who want more than opinion. Expect plain-language breakdowns of the research, honest discussion of what the evidence does and does not support, and conversations with the people behind the studies. If you make decisions about how people are trained, we think you deserve to see the evidence first. Most training is sold on confidence. training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices Anthony G Gallagher No No Experience Is Not Proficiency: Measuring what surgeons actually do | Show Me The Evidence E10 10 10 Experience Is Not Proficiency: Measuring what surgeons actually do | Show Me The Evidence E10 full 7eb33bd4-ace1-45bc-b20b-51ee4ebffa1d https://share.transistor.fm/s/79d6dbab In this episode, Professor Anthony G. Gallagher is joined by Dr Rui Farinha, a senior consultant urologist in Lisbon who has completed fellowships in Spain, Germany and Belgium in laparoscopic and robotic surgery, and who trained with Professor Alexandre Mottrie in Aalst and at Orsi Academy in Belgium. The two met at Orsi in 2019.


The conversation begins with a practical question. Robots are expensive, so what do they actually give the surgeon? Rui sets out the case carefully: tremor filtering and motion scaling for precision, wristed instruments for dexterity in tight anatomy such as the pelvis, magnified stereoscopic vision, better ergonomics over long procedures, and a digital platform capable of recording and analysing performance. Then he adds the caveat that frames the rest of the hour. None of these technical advantages automatically produces a better clinical outcome.


From there the discussion turns to how surgeons are trained. Rui traces his own path through three eras: an apprenticeship model in open surgery that depended on which cases turned up and which consultant happened to be supervising, a laparoscopic era in which he discovered that open skills did not transfer and that basic skills belonged outside the operating theatre, and a robotic era that was structured from the start around defined objectives, simulation, proximate feedback and a demonstrated standard.


The heart of the episode is Rui's programme of research on robot-assisted partial nephrectomy (RAPN). He explains why he chose a technically demanding, high-risk procedure with clearly separable phases and direct consequences for the patient, and he sets out what the studies found. A complex operation can be deconstructed into observable phases, steps, errors and critical errors, with 100 per cent consensus from an international expert panel. Experienced surgeons made 69 per cent fewer total errors than novices. Within the experienced group, the low-error surgeons made 77 per cent fewer errors than the high-error surgeons, and that high-error expert group performed at roughly the level of the better novices. Procedure-specific binary metrics achieved high inter-rater reliability where a global rating scale did not. And a systematic review of partial nephrectomy training models found models widely rated as realistic and useful, but no randomised controlled trials and no evidence of skill transfer.


Rui's conclusion is direct. Realism is not evidence. A simulator is a vehicle, not a training programme. Anyone building a curriculum should define the performance they want first and select or construct the simulation second, which is the opposite of what usually happens. He describes a model he developed deliberately as a delivery vehicle for a metric-based curriculum, using readily available animal tissue and emulating eight of the eleven phases of the human procedure, on the grounds that the useful question is not how realistic a model looks but how much high-quality measurable practice it permits.


The episode closes on proficiency-based progression in the skills laboratory and in the operating room, on the value of proximate human feedback in an era of AI-delivered training, and on a central principle: the manufacturer's instructions for use teach the surgeon how the robot functions, while PBP determines whether the surgeon can use it proficiently.



Key Topics Covered

What the robot actually adds, and what it does not | 0:07

  • Guest introduction: senior consultant urologist in Lisbon, fellowships in Spain, Germany and Belgium, met Professor Gallagher at Orsi Academy in 2019
  • Robots do not replace the surgeon and do not operate independently; they act as an interface that translates the surgeon's movements
  • Precision through filtering of physiological tremor and scaling of movement, valuable in delicate dissection, suturing and vascular anastomosis | 1:46

Dexterity, vision and endurance | 2:34

  • Wristed instruments provide additional degrees of freedom where rigid laparoscopic instruments cannot, which matters most in narrow spaces such as the pelvis
  • Magnified high-definition three-dimensional view improves depth perception and tissue plane discrimination; the surgeon controls the camera directly | 3:25
  • An adjustable console reduces muscular strain and fatigue, supporting concentration and consistency through long procedures | 4:14

The digital platform, and the caveat that matters | 5:04

  • Because the surgeon's actions pass through a computer-controlled interface, robotic systems can record instrument movement, analyse technical performance and integrate imaging or navigation
  • Rui's caveat: these technical advantages do not automatically produce better clinical outcomes. Performance still depends on training, case selection, team coordination and appropriate use of the technology
  • Professor Gallagher returns to the Dwight Meglan episode and the view that the surgeon, not the platform, makes the decisions | 5:53

Three training eras: apprenticeship, laparoscopy, robotics | 6:43

  • Open surgery followed the apprenticeship model: observe, assist, then perform increasing parts of the operation under supervision
  • The limitations Rui experienced: dependence on case availability, inconsistent quality of supervision, and progression decided by an individual trainer's subjective judgement | 8:29

Laparoscopy and the skills that did not transfer | 9:20

  • Competence in open surgery did not translate into competence in minimally invasive surgery
  • Two-dimensional vision and reduced depth perception, the fulcrum effect, long rigid instruments with limited degrees of freedom, and bimanual coordination against an indirect image | 10:10
  • The realisation that basic laparoscopic skills could and should be learned outside the operating theatre, using box trainers, virtual reality, dry and animal laboratory work and standardised exercises | 10:58

Structured robotic training from 2019 | 11:48

  • Training with Professor Alexandre Mottrie: console controls, instrument management, camera control and clutching, multiple arms, collision avoidance, bedside communication, troubleshooting and emergency undocking
  • Maintaining situational awareness while physically separated from the patient, supported by online modules, simulation, observation, bedside assistance, supervised console practice and procedural prompting | 12:39

From time served to a defined standard | 13:28

  • The question shifts from how many cases the trainee has performed to whether the trainee can perform the procedure safely and consistently to a defined standard
  • Clearly defined learning objectives, procedure-specific benchmarks, objective assessment, simulation before patients, and deliberate practice with proximate feedback

What simulation cannot teach | 14:19

  • Clinical judgement, decision making under pressure, recognition of abnormal anatomy, management of bleeding and complications, adapting when the plan changes, and leading the operating team
  • Practice remains very different from centre to centre and department to department | 15:09

Simulation and the Halstedian model | 16:51

  • The apprenticeship model remains important but is no longer sufficient on its own
  • Professor Gallagher's position: simulation does not end Halstedian training, it improves it, because the consultant now supervises a pre-trained novice who arrives with ...
]]> In this episode, Professor Anthony G. Gallagher is joined by Dr Rui Farinha, a senior consultant urologist in Lisbon who has completed fellowships in Spain, Germany and Belgium in laparoscopic and robotic surgery, and who trained with Professor Alexandre Mottrie in Aalst and at Orsi Academy in Belgium. The two met at Orsi in 2019.


The conversation begins with a practical question. Robots are expensive, so what do they actually give the surgeon? Rui sets out the case carefully: tremor filtering and motion scaling for precision, wristed instruments for dexterity in tight anatomy such as the pelvis, magnified stereoscopic vision, better ergonomics over long procedures, and a digital platform capable of recording and analysing performance. Then he adds the caveat that frames the rest of the hour. None of these technical advantages automatically produces a better clinical outcome.


From there the discussion turns to how surgeons are trained. Rui traces his own path through three eras: an apprenticeship model in open surgery that depended on which cases turned up and which consultant happened to be supervising, a laparoscopic era in which he discovered that open skills did not transfer and that basic skills belonged outside the operating theatre, and a robotic era that was structured from the start around defined objectives, simulation, proximate feedback and a demonstrated standard.


The heart of the episode is Rui's programme of research on robot-assisted partial nephrectomy (RAPN). He explains why he chose a technically demanding, high-risk procedure with clearly separable phases and direct consequences for the patient, and he sets out what the studies found. A complex operation can be deconstructed into observable phases, steps, errors and critical errors, with 100 per cent consensus from an international expert panel. Experienced surgeons made 69 per cent fewer total errors than novices. Within the experienced group, the low-error surgeons made 77 per cent fewer errors than the high-error surgeons, and that high-error expert group performed at roughly the level of the better novices. Procedure-specific binary metrics achieved high inter-rater reliability where a global rating scale did not. And a systematic review of partial nephrectomy training models found models widely rated as realistic and useful, but no randomised controlled trials and no evidence of skill transfer.


Rui's conclusion is direct. Realism is not evidence. A simulator is a vehicle, not a training programme. Anyone building a curriculum should define the performance they want first and select or construct the simulation second, which is the opposite of what usually happens. He describes a model he developed deliberately as a delivery vehicle for a metric-based curriculum, using readily available animal tissue and emulating eight of the eleven phases of the human procedure, on the grounds that the useful question is not how realistic a model looks but how much high-quality measurable practice it permits.


The episode closes on proficiency-based progression in the skills laboratory and in the operating room, on the value of proximate human feedback in an era of AI-delivered training, and on a central principle: the manufacturer's instructions for use teach the surgeon how the robot functions, while PBP determines whether the surgeon can use it proficiently.



Key Topics Covered

What the robot actually adds, and what it does not | 0:07

  • Guest introduction: senior consultant urologist in Lisbon, fellowships in Spain, Germany and Belgium, met Professor Gallagher at Orsi Academy in 2019
  • Robots do not replace the surgeon and do not operate independently; they act as an interface that translates the surgeon's movements
  • Precision through filtering of physiological tremor and scaling of movement, valuable in delicate dissection, suturing and vascular anastomosis | 1:46

Dexterity, vision and endurance | 2:34

  • Wristed instruments provide additional degrees of freedom where rigid laparoscopic instruments cannot, which matters most in narrow spaces such as the pelvis
  • Magnified high-definition three-dimensional view improves depth perception and tissue plane discrimination; the surgeon controls the camera directly | 3:25
  • An adjustable console reduces muscular strain and fatigue, supporting concentration and consistency through long procedures | 4:14

The digital platform, and the caveat that matters | 5:04

  • Because the surgeon's actions pass through a computer-controlled interface, robotic systems can record instrument movement, analyse technical performance and integrate imaging or navigation
  • Rui's caveat: these technical advantages do not automatically produce better clinical outcomes. Performance still depends on training, case selection, team coordination and appropriate use of the technology
  • Professor Gallagher returns to the Dwight Meglan episode and the view that the surgeon, not the platform, makes the decisions | 5:53

Three training eras: apprenticeship, laparoscopy, robotics | 6:43

  • Open surgery followed the apprenticeship model: observe, assist, then perform increasing parts of the operation under supervision
  • The limitations Rui experienced: dependence on case availability, inconsistent quality of supervision, and progression decided by an individual trainer's subjective judgement | 8:29

Laparoscopy and the skills that did not transfer | 9:20

  • Competence in open surgery did not translate into competence in minimally invasive surgery
  • Two-dimensional vision and reduced depth perception, the fulcrum effect, long rigid instruments with limited degrees of freedom, and bimanual coordination against an indirect image | 10:10
  • The realisation that basic laparoscopic skills could and should be learned outside the operating theatre, using box trainers, virtual reality, dry and animal laboratory work and standardised exercises | 10:58

Structured robotic training from 2019 | 11:48

  • Training with Professor Alexandre Mottrie: console controls, instrument management, camera control and clutching, multiple arms, collision avoidance, bedside communication, troubleshooting and emergency undocking
  • Maintaining situational awareness while physically separated from the patient, supported by online modules, simulation, observation, bedside assistance, supervised console practice and procedural prompting | 12:39

From time served to a defined standard | 13:28

  • The question shifts from how many cases the trainee has performed to whether the trainee can perform the procedure safely and consistently to a defined standard
  • Clearly defined learning objectives, procedure-specific benchmarks, objective assessment, simulation before patients, and deliberate practice with proximate feedback

What simulation cannot teach | 14:19

  • Clinical judgement, decision making under pressure, recognition of abnormal anatomy, management of bleeding and complications, adapting when the plan changes, and leading the operating team
  • Practice remains very different from centre to centre and department to department | 15:09

Simulation and the Halstedian model | 16:51

  • The apprenticeship model remains important but is no longer sufficient on its own
  • Professor Gallagher's position: simulation does not end Halstedian training, it improves it, because the consultant now supervises a pre-trained novice who arrives with ...
]]> Sun, 16 Aug 2026 08:00:00 +0100 Anthony G. Gallagher, Flux Learning Ltd Anthony G. Gallagher, Flux Learning Ltd 3786 In this episode, Professor Anthony G. Gallagher is joined by Dr Rui Farinha, a senior consultant urologist in Lisbon who has completed fellowships in Spain, Germany and Belgium in laparoscopic and robotic surgery, and who trained with Professor Alexandre Mottrie in Aalst and at Orsi Academy in Belgium. The two met at Orsi in 2019.


The conversation begins with a practical question. Robots are expensive, so what do they actually give the surgeon? Rui sets out the case carefully: tremor filtering and motion scaling for precision, wristed instruments for dexterity in tight anatomy such as the pelvis, magnified stereoscopic vision, better ergonomics over long procedures, and a digital platform capable of recording and analysing performance. Then he adds the caveat that frames the rest of the hour. None of these technical advantages automatically produces a better clinical outcome.


From there the discussion turns to how surgeons are trained. Rui traces his own path through three eras: an apprenticeship model in open surgery that depended on which cases turned up and which consultant happened to be supervising, a laparoscopic era in which he discovered that open skills did not transfer and that basic skills belonged outside the operating theatre, and a robotic era that was structured from the start around defined objectives, simulation, proximate feedback and a demonstrated standard.


The heart of the episode is Rui's programme of research on robot-assisted partial nephrectomy (RAPN). He explains why he chose a technically demanding, high-risk procedure with clearly separable phases and direct consequences for the patient, and he sets out what the studies found. A complex operation can be deconstructed into observable phases, steps, errors and critical errors, with 100 per cent consensus from an international expert panel. Experienced surgeons made 69 per cent fewer total errors than novices. Within the experienced group, the low-error surgeons made 77 per cent fewer errors than the high-error surgeons, and that high-error expert group performed at roughly the level of the better novices. Procedure-specific binary metrics achieved high inter-rater reliability where a global rating scale did not. And a systematic review of partial nephrectomy training models found models widely rated as realistic and useful, but no randomised controlled trials and no evidence of skill transfer.


Rui's conclusion is direct. Realism is not evidence. A simulator is a vehicle, not a training programme. Anyone building a curriculum should define the performance they want first and select or construct the simulation second, which is the opposite of what usually happens. He describes a model he developed deliberately as a delivery vehicle for a metric-based curriculum, using readily available animal tissue and emulating eight of the eleven phases of the human procedure, on the grounds that the useful question is not how realistic a model looks but how much high-quality measurable practice it permits.


The episode closes on proficiency-based progression in the skills laboratory and in the operating room, on the value of proximate human feedback in an era of AI-delivered training, and on a central principle: the manufacturer's instructions for use teach the surgeon how the robot functions, while PBP determines whether the surgeon can use it proficiently.



Key Topics Covered

What the robot actually adds, and what it does not | 0:07

  • Guest introduction: senior consultant urologist in Lisbon, fellowships in Spain, Germany and Belgium, met Professor Gallagher at Orsi Academy in 2019
  • Robots do not replace the surgeon and do not operate independently; they act as an interface that translates the surgeon's movements
  • Precision through filtering of physiological tremor and scaling of movement, valuable in delicate dissection, suturing and vascular anastomosis | 1:46

Dexterity, vision and endurance | 2:34

  • Wristed instruments provide additional degrees of freedom where rigid laparoscopic instruments cannot, which matters most in narrow spaces such as the pelvis
  • Magnified high-definition three-dimensional view improves depth perception and tissue plane discrimination; the surgeon controls the camera directly | 3:25
  • An adjustable console reduces muscular strain and fatigue, supporting concentration and consistency through long procedures | 4:14

The digital platform, and the caveat that matters | 5:04

  • Because the surgeon's actions pass through a computer-controlled interface, robotic systems can record instrument movement, analyse technical performance and integrate imaging or navigation
  • Rui's caveat: these technical advantages do not automatically produce better clinical outcomes. Performance still depends on training, case selection, team coordination and appropriate use of the technology
  • Professor Gallagher returns to the Dwight Meglan episode and the view that the surgeon, not the platform, makes the decisions | 5:53

Three training eras: apprenticeship, laparoscopy, robotics | 6:43

  • Open surgery followed the apprenticeship model: observe, assist, then perform increasing parts of the operation under supervision
  • The limitations Rui experienced: dependence on case availability, inconsistent quality of supervision, and progression decided by an individual trainer's subjective judgement | 8:29

Laparoscopy and the skills that did not transfer | 9:20

  • Competence in open surgery did not translate into competence in minimally invasive surgery
  • Two-dimensional vision and reduced depth perception, the fulcrum effect, long rigid instruments with limited degrees of freedom, and bimanual coordination against an indirect image | 10:10
  • The realisation that basic laparoscopic skills could and should be learned outside the operating theatre, using box trainers, virtual reality, dry and animal laboratory work and standardised exercises | 10:58

Structured robotic training from 2019 | 11:48

  • Training with Professor Alexandre Mottrie: console controls, instrument management, camera control and clutching, multiple arms, collision avoidance, bedside communication, troubleshooting and emergency undocking
  • Maintaining situational awareness while physically separated from the patient, supported by online modules, simulation, observation, bedside assistance, supervised console practice and procedural prompting | 12:39

From time served to a defined standard | 13:28

  • The question shifts from how many cases the trainee has performed to whether the trainee can perform the procedure safely and consistently to a defined standard
  • Clearly defined learning objectives, procedure-specific benchmarks, objective assessment, simulation before patients, and deliberate practice with proximate feedback

What simulation cannot teach | 14:19

  • Clinical judgement, decision making under pressure, recognition of abnormal anatomy, management of bleeding and complications, adapting when the plan changes, and leading the operating team
  • Practice remains very different from centre to centre and department to department | 15:09

Simulation and the Halstedian model | 16:51

  • The apprenticeship model remains important but is no longer sufficient on its own
  • Professor Gallagher's position: simulation does not end Halstedian training, it improves it, because the consultant now supervises a pre-trained novice who arrives with ...
]]> training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices No The end of "see one, do one, teach one" — Prof Stefano Puliatti | Show Me The Evidence Episode 9 9 9 The end of "see one, do one, teach one" — Prof Stefano Puliatti | Show Me The Evidence Episode 9 full 5085ba25-1188-4ef4-a7b4-9f76059d64c4 https://share.transistor.fm/s/82c1546a Professor Stefano Puliatti is Professor of Urology at the University of Modena, a practising robotic surgeon, a faculty member of Surgquest, and the former Medical Director of Orsi Academy in Belgium, where much of the PBP research discussed here was carried out.


Key takeaways

  • Training quality should not depend on the luck of being assigned a gifted trainer. A shared methodology makes good outcomes reproducible.
  • Errors, not speed or step count alone, are the real indicator of surgical quality. Process measures on their own do not guarantee it.
  • A proficiency benchmark set at expert level is reachable by nearly all trainees, and usually faster than conventional training allows.
  • Good simulation does not have to be expensive. A validated methodology on a low-cost model can outperform costly kit used without one.
  • Diluting the method, for example by dropping the pre-lab benchmark, measurably slows learning and raises cost.


Evidence and further reading

The figures cited in this episode come from the peer-reviewed studies below. Please confirm the exact papers you want listed for this episode before publishing.

  • De Groote R, et al. Proficiency-based training and evidence-based methodology: a systematic review and meta-analysis. BJU International. doi:10.1111/bju.70333
  • Mazzone E, Puliatti S, et al. A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes. Annals of Surgery, 2021. PMID: 33630473
  • Puliatti S, et al. Can all surgical trainees be trained to proficiency for a robotic urethro-vesical anastomotic task using a chicken model? A prospective, randomized trial. PMID: 40351291
  • Randomised trial on the economic impact of proficiency-based progression versus conventional robotic surgical training. PMC12907777
  • Development and validation of the objective assessment of robotic suturing and knot tying skills for a chicken anastomotic model. Surgical Endoscopy, 2020. doi:10.1007/s00464-020-07918-5
  • De Groote R, et al. Proficiency-based progression training for robotic surgery skills training: a randomized clinical trial. BJU International, 2022. doi:10.1111/bju.15811


Credits

Show Me the Evidence is hosted by Professor Anthony G. Gallagher and produced by Flux Learning. 

]]> Professor Stefano Puliatti is Professor of Urology at the University of Modena, a practising robotic surgeon, a faculty member of Surgquest, and the former Medical Director of Orsi Academy in Belgium, where much of the PBP research discussed here was carried out.


Key takeaways

  • Training quality should not depend on the luck of being assigned a gifted trainer. A shared methodology makes good outcomes reproducible.
  • Errors, not speed or step count alone, are the real indicator of surgical quality. Process measures on their own do not guarantee it.
  • A proficiency benchmark set at expert level is reachable by nearly all trainees, and usually faster than conventional training allows.
  • Good simulation does not have to be expensive. A validated methodology on a low-cost model can outperform costly kit used without one.
  • Diluting the method, for example by dropping the pre-lab benchmark, measurably slows learning and raises cost.


Evidence and further reading

The figures cited in this episode come from the peer-reviewed studies below. Please confirm the exact papers you want listed for this episode before publishing.

  • De Groote R, et al. Proficiency-based training and evidence-based methodology: a systematic review and meta-analysis. BJU International. doi:10.1111/bju.70333
  • Mazzone E, Puliatti S, et al. A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes. Annals of Surgery, 2021. PMID: 33630473
  • Puliatti S, et al. Can all surgical trainees be trained to proficiency for a robotic urethro-vesical anastomotic task using a chicken model? A prospective, randomized trial. PMID: 40351291
  • Randomised trial on the economic impact of proficiency-based progression versus conventional robotic surgical training. PMC12907777
  • Development and validation of the objective assessment of robotic suturing and knot tying skills for a chicken anastomotic model. Surgical Endoscopy, 2020. doi:10.1007/s00464-020-07918-5
  • De Groote R, et al. Proficiency-based progression training for robotic surgery skills training: a randomized clinical trial. BJU International, 2022. doi:10.1111/bju.15811


Credits

Show Me the Evidence is hosted by Professor Anthony G. Gallagher and produced by Flux Learning. 

]]> Fri, 31 Jul 2026 18:21:51 +0100 Anthony G. Gallagher, Flux Learning Ltd Anthony G. Gallagher, Flux Learning Ltd 3036 Professor Stefano Puliatti is Professor of Urology at the University of Modena, a practising robotic surgeon, a faculty member of Surgquest, and the former Medical Director of Orsi Academy in Belgium, where much of the PBP research discussed here was carried out.


Key takeaways

  • Training quality should not depend on the luck of being assigned a gifted trainer. A shared methodology makes good outcomes reproducible.
  • Errors, not speed or step count alone, are the real indicator of surgical quality. Process measures on their own do not guarantee it.
  • A proficiency benchmark set at expert level is reachable by nearly all trainees, and usually faster than conventional training allows.
  • Good simulation does not have to be expensive. A validated methodology on a low-cost model can outperform costly kit used without one.
  • Diluting the method, for example by dropping the pre-lab benchmark, measurably slows learning and raises cost.


Evidence and further reading

The figures cited in this episode come from the peer-reviewed studies below. Please confirm the exact papers you want listed for this episode before publishing.

  • De Groote R, et al. Proficiency-based training and evidence-based methodology: a systematic review and meta-analysis. BJU International. doi:10.1111/bju.70333
  • Mazzone E, Puliatti S, et al. A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes. Annals of Surgery, 2021. PMID: 33630473
  • Puliatti S, et al. Can all surgical trainees be trained to proficiency for a robotic urethro-vesical anastomotic task using a chicken model? A prospective, randomized trial. PMID: 40351291
  • Randomised trial on the economic impact of proficiency-based progression versus conventional robotic surgical training. PMC12907777
  • Development and validation of the objective assessment of robotic suturing and knot tying skills for a chicken anastomotic model. Surgical Endoscopy, 2020. doi:10.1007/s00464-020-07918-5
  • De Groote R, et al. Proficiency-based progression training for robotic surgery skills training: a randomized clinical trial. BJU International, 2022. doi:10.1111/bju.15811


Credits

Show Me the Evidence is hosted by Professor Anthony G. Gallagher and produced by Flux Learning. 

]]> training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices No From Physics-Based Simulation to Surgical Robots| Show Me The Evidence E8- Dr Dwight Meglan 8 8 From Physics-Based Simulation to Surgical Robots| Show Me The Evidence E8- Dr Dwight Meglan full 99b3b872-365e-48a1-a80c-c0e537c3b4f8 https://share.transistor.fm/s/a692e7df Show Me the Evidence, Episode 8

Guest: Dr Dwight Meglan Topic: Physics-Based Simulation, Surgical Robotics and Why Simulators Still Don't Measure What Matters


Episode Summary

In this episode, Professor Tony Gallagher is joined by Dr Dwight Meglan, the engineer who developed one of the first physics-based virtual reality simulators for endovascular procedures in the late 1990s, and who has spent the last two decades building surgical robots. Together they trace 30 years of simulation-based training and ask why so little has changed: simulators are still not verified against real-world physics, the field still measures process rather than skill, and device manufacturers, not educators, still set the agenda. The conversation ranges from haptics and instrumented torquers to Likert scales, credentialing committees, telesurgery risk, and why autonomous surgical robots are a regulatory and financial impossibility rather than a technical one. It closes with the evidence for Proficiency-Based Progression (PBP) and the leadership needed to adopt it.


Key Topics Covered

1. Building the first physics-based VR simulators | 0:00

  • Meeting at Medicine Meets Virtual Reality in the late 1990s
  • Real-time physics of tool and tissue interaction, fluoroscopy and haptic feedback
  • Why the goal was to replicate reality, not design a user experience
  • Physics tests to verify simulator correctness still do not exist, 25 years on

2. Who really drives simulation: the device manufacturers | 2:24

  • Manufacturers pay for simulation, so manufacturers shape it
  • Training to use the device versus training to perform the procedure
  • Simulators in exhibition booths: marketing tools first, training tools second

3. Haptics and the sensory threshold problem | 3:55

  • The instrumented torquer: measuring what cardiologists actually feel
  • Just noticeable difference thresholds vary between clinicians
  • Still no published datasets on the forces a cardiologist feels during catheterisation
  • Clinicians praising the haptics on simulators where the haptics were switched off

4. Using devices safely: the stapler and the defibrillator | 8:52

  • A stapling device with a 6 to 27 per cent leak rate, where one third of patients who develop a leak die
  • Training to the instructions for use is device safety training, not surgical skills training
  • Cardiac defibrillator implantation: clinicians departing from the instructions for use
  • Construct validity findings: some very senior clinicians perform worse than the worst trainee when assessed with objective, peer-derived metrics

5. What should a simulator measure? | 13:19

  • The original approach: record everything, then find the measures that matter
  • Metrics for mechanical thrombectomy for acute stroke, developed from the human procedure with Mentice
  • How clinician-led metrics forced a redesign of contrast injection, later patented
  • It works when you insist on it, but you must start with the metrics

6. The state of simulation metrics today | 17:35

  • At a recent conference, almost none of the exhibited simulators had any metrics at all
  • Some manufacturers avoid measurement deliberately: plausible deniability
  • Validated metrics as a purchasing condition: if you cannot build them in, we will not buy

7. From simulation to surgical robotics | 21:52

  • Why simulation felt like a capped market and robotics did not
  • The analogous DNA of simulators and robots as real-time information processing systems
  • Verification versus validation: robots are bench-tested against dozens of specifications, simulators almost never are

8. Measuring process, not skill | 26:49

  • Motion tracking and path length: lessons not learned from laparoscopic surgery
  • AI-driven pattern hunting as a fishing expedition without a hypothesis
  • Suturing as the test case: the physics of tissue apposition has never been published
  • Physical intelligence and humanoid robotics will improve simulation from the outside in

9. Likert scales are not measurement | 36:47

  • Binary, procedure-specific metrics require scoring the entire video, reliably, in pairs
  • One-to-five ratings after watching a few minutes of video are hand waving, not assessment
  • Ring exercises on robotic simulators have never been verified against real forces

10. Whose job is it? Credentialing and privileging | 39:27

  • Manufacturers certify device use; professional societies and hospitals grant privileges
  • Per-procedure privileging in the United States versus broad qualification in Europe
  • The credentialing committee problem: standards set by the least experienced member
  • Case volume, fellowship length and reputation are social proof, not evidence of competence

11. A jumbo jet a day: the human cost | 44:43

  • Deaths from surgical skills deficits estimated as equivalent to a full jumbo jet crashing every day, consistent with evidence that around 4.2 million people die within 30 days of surgery each year (Nepogodiev et al., The Lancet, 2019)
  • Why one death at a time never makes headlines the way one crash does
  • The Bristol Royal Infirmary case: peers knew for a decade before the front pages forced action (The Bristol Royal Infirmary Inquiry, Kennedy Report, 2001)
  • Telesurgery at scale: how one underperforming surgeon could soon harm many patients quickly

12. The PBP evidence and the leadership gap | 47:47

  • Systematic review and meta-analysis evidence: PBP-trained clinicians make approximately 60 per cent fewer objectively assessed intraoperative errors
  • Training to proficiency in one third of the time and at less than half the cost of conventional training
  • "A failure of leadership": adoption at ORSI Academy, AANA, ERUS under Alberto Breda, and increasingly Medtronic cardiovascular came from senior leaders making it non-negotiable
  • The aviation lesson: flight simulation was mandated by government before the evidence existed, while surgery has the evidence and no mandate

Publication: Mazzone, E., Puliatti, S., Amato, M., Bunting, B., Rocco, B., Montorsi, F., Mottrie, A. and Gallagher, A.G. (2021). A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes. Annals of Surgery, 274(2), 281-289. DOI: 10.1097/SLA.0000000000004650

Publication: Puliatti, S., Rodriguez Peñaranda, N., Amato, M., De Groote, R., Farinha, R., Bunting, B., van Cleynenbreugel, B., Mottrie, A. and Gallagher, A.G. (2026). Randomised trial on the economic impact of proficiency-based progression vs conventional robotic surgical training. BJU International, 137, 493-501. https://doi.org/10.1111/bju.70130

13. The next 10 years: AI, automation and the autonomy myth | 54:08

  • AI will lower the barrier to building simulators and add semi-automated instruction
  • Part-task automation, such as a supervised stapler, is plausible; autonomous procedures are not
  • The barrier is financial and regulatory: manufacturers will not accept liability for practising medicine
  • The self-driving analogy: millions of hours of verified synthetic data would ...
]]> Show Me the Evidence, Episode 8

Guest: Dr Dwight Meglan Topic: Physics-Based Simulation, Surgical Robotics and Why Simulators Still Don't Measure What Matters


Episode Summary

In this episode, Professor Tony Gallagher is joined by Dr Dwight Meglan, the engineer who developed one of the first physics-based virtual reality simulators for endovascular procedures in the late 1990s, and who has spent the last two decades building surgical robots. Together they trace 30 years of simulation-based training and ask why so little has changed: simulators are still not verified against real-world physics, the field still measures process rather than skill, and device manufacturers, not educators, still set the agenda. The conversation ranges from haptics and instrumented torquers to Likert scales, credentialing committees, telesurgery risk, and why autonomous surgical robots are a regulatory and financial impossibility rather than a technical one. It closes with the evidence for Proficiency-Based Progression (PBP) and the leadership needed to adopt it.


Key Topics Covered

1. Building the first physics-based VR simulators | 0:00

  • Meeting at Medicine Meets Virtual Reality in the late 1990s
  • Real-time physics of tool and tissue interaction, fluoroscopy and haptic feedback
  • Why the goal was to replicate reality, not design a user experience
  • Physics tests to verify simulator correctness still do not exist, 25 years on

2. Who really drives simulation: the device manufacturers | 2:24

  • Manufacturers pay for simulation, so manufacturers shape it
  • Training to use the device versus training to perform the procedure
  • Simulators in exhibition booths: marketing tools first, training tools second

3. Haptics and the sensory threshold problem | 3:55

  • The instrumented torquer: measuring what cardiologists actually feel
  • Just noticeable difference thresholds vary between clinicians
  • Still no published datasets on the forces a cardiologist feels during catheterisation
  • Clinicians praising the haptics on simulators where the haptics were switched off

4. Using devices safely: the stapler and the defibrillator | 8:52

  • A stapling device with a 6 to 27 per cent leak rate, where one third of patients who develop a leak die
  • Training to the instructions for use is device safety training, not surgical skills training
  • Cardiac defibrillator implantation: clinicians departing from the instructions for use
  • Construct validity findings: some very senior clinicians perform worse than the worst trainee when assessed with objective, peer-derived metrics

5. What should a simulator measure? | 13:19

  • The original approach: record everything, then find the measures that matter
  • Metrics for mechanical thrombectomy for acute stroke, developed from the human procedure with Mentice
  • How clinician-led metrics forced a redesign of contrast injection, later patented
  • It works when you insist on it, but you must start with the metrics

6. The state of simulation metrics today | 17:35

  • At a recent conference, almost none of the exhibited simulators had any metrics at all
  • Some manufacturers avoid measurement deliberately: plausible deniability
  • Validated metrics as a purchasing condition: if you cannot build them in, we will not buy

7. From simulation to surgical robotics | 21:52

  • Why simulation felt like a capped market and robotics did not
  • The analogous DNA of simulators and robots as real-time information processing systems
  • Verification versus validation: robots are bench-tested against dozens of specifications, simulators almost never are

8. Measuring process, not skill | 26:49

  • Motion tracking and path length: lessons not learned from laparoscopic surgery
  • AI-driven pattern hunting as a fishing expedition without a hypothesis
  • Suturing as the test case: the physics of tissue apposition has never been published
  • Physical intelligence and humanoid robotics will improve simulation from the outside in

9. Likert scales are not measurement | 36:47

  • Binary, procedure-specific metrics require scoring the entire video, reliably, in pairs
  • One-to-five ratings after watching a few minutes of video are hand waving, not assessment
  • Ring exercises on robotic simulators have never been verified against real forces

10. Whose job is it? Credentialing and privileging | 39:27

  • Manufacturers certify device use; professional societies and hospitals grant privileges
  • Per-procedure privileging in the United States versus broad qualification in Europe
  • The credentialing committee problem: standards set by the least experienced member
  • Case volume, fellowship length and reputation are social proof, not evidence of competence

11. A jumbo jet a day: the human cost | 44:43

  • Deaths from surgical skills deficits estimated as equivalent to a full jumbo jet crashing every day, consistent with evidence that around 4.2 million people die within 30 days of surgery each year (Nepogodiev et al., The Lancet, 2019)
  • Why one death at a time never makes headlines the way one crash does
  • The Bristol Royal Infirmary case: peers knew for a decade before the front pages forced action (The Bristol Royal Infirmary Inquiry, Kennedy Report, 2001)
  • Telesurgery at scale: how one underperforming surgeon could soon harm many patients quickly

12. The PBP evidence and the leadership gap | 47:47

  • Systematic review and meta-analysis evidence: PBP-trained clinicians make approximately 60 per cent fewer objectively assessed intraoperative errors
  • Training to proficiency in one third of the time and at less than half the cost of conventional training
  • "A failure of leadership": adoption at ORSI Academy, AANA, ERUS under Alberto Breda, and increasingly Medtronic cardiovascular came from senior leaders making it non-negotiable
  • The aviation lesson: flight simulation was mandated by government before the evidence existed, while surgery has the evidence and no mandate

Publication: Mazzone, E., Puliatti, S., Amato, M., Bunting, B., Rocco, B., Montorsi, F., Mottrie, A. and Gallagher, A.G. (2021). A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes. Annals of Surgery, 274(2), 281-289. DOI: 10.1097/SLA.0000000000004650

Publication: Puliatti, S., Rodriguez Peñaranda, N., Amato, M., De Groote, R., Farinha, R., Bunting, B., van Cleynenbreugel, B., Mottrie, A. and Gallagher, A.G. (2026). Randomised trial on the economic impact of proficiency-based progression vs conventional robotic surgical training. BJU International, 137, 493-501. https://doi.org/10.1111/bju.70130

13. The next 10 years: AI, automation and the autonomy myth | 54:08

  • AI will lower the barrier to building simulators and add semi-automated instruction
  • Part-task automation, such as a supervised stapler, is plausible; autonomous procedures are not
  • The barrier is financial and regulatory: manufacturers will not accept liability for practising medicine
  • The self-driving analogy: millions of hours of verified synthetic data would ...
]]> Sat, 18 Jul 2026 10:46:12 +0100 Anthony G. Gallagher, Flux Learning Ltd Anthony G. Gallagher, Flux Learning Ltd 3684 Show Me the Evidence, Episode 8

Guest: Dr Dwight Meglan Topic: Physics-Based Simulation, Surgical Robotics and Why Simulators Still Don't Measure What Matters


Episode Summary

In this episode, Professor Tony Gallagher is joined by Dr Dwight Meglan, the engineer who developed one of the first physics-based virtual reality simulators for endovascular procedures in the late 1990s, and who has spent the last two decades building surgical robots. Together they trace 30 years of simulation-based training and ask why so little has changed: simulators are still not verified against real-world physics, the field still measures process rather than skill, and device manufacturers, not educators, still set the agenda. The conversation ranges from haptics and instrumented torquers to Likert scales, credentialing committees, telesurgery risk, and why autonomous surgical robots are a regulatory and financial impossibility rather than a technical one. It closes with the evidence for Proficiency-Based Progression (PBP) and the leadership needed to adopt it.


Key Topics Covered

1. Building the first physics-based VR simulators | 0:00

  • Meeting at Medicine Meets Virtual Reality in the late 1990s
  • Real-time physics of tool and tissue interaction, fluoroscopy and haptic feedback
  • Why the goal was to replicate reality, not design a user experience
  • Physics tests to verify simulator correctness still do not exist, 25 years on

2. Who really drives simulation: the device manufacturers | 2:24

  • Manufacturers pay for simulation, so manufacturers shape it
  • Training to use the device versus training to perform the procedure
  • Simulators in exhibition booths: marketing tools first, training tools second

3. Haptics and the sensory threshold problem | 3:55

  • The instrumented torquer: measuring what cardiologists actually feel
  • Just noticeable difference thresholds vary between clinicians
  • Still no published datasets on the forces a cardiologist feels during catheterisation
  • Clinicians praising the haptics on simulators where the haptics were switched off

4. Using devices safely: the stapler and the defibrillator | 8:52

  • A stapling device with a 6 to 27 per cent leak rate, where one third of patients who develop a leak die
  • Training to the instructions for use is device safety training, not surgical skills training
  • Cardiac defibrillator implantation: clinicians departing from the instructions for use
  • Construct validity findings: some very senior clinicians perform worse than the worst trainee when assessed with objective, peer-derived metrics

5. What should a simulator measure? | 13:19

  • The original approach: record everything, then find the measures that matter
  • Metrics for mechanical thrombectomy for acute stroke, developed from the human procedure with Mentice
  • How clinician-led metrics forced a redesign of contrast injection, later patented
  • It works when you insist on it, but you must start with the metrics

6. The state of simulation metrics today | 17:35

  • At a recent conference, almost none of the exhibited simulators had any metrics at all
  • Some manufacturers avoid measurement deliberately: plausible deniability
  • Validated metrics as a purchasing condition: if you cannot build them in, we will not buy

7. From simulation to surgical robotics | 21:52

  • Why simulation felt like a capped market and robotics did not
  • The analogous DNA of simulators and robots as real-time information processing systems
  • Verification versus validation: robots are bench-tested against dozens of specifications, simulators almost never are

8. Measuring process, not skill | 26:49

  • Motion tracking and path length: lessons not learned from laparoscopic surgery
  • AI-driven pattern hunting as a fishing expedition without a hypothesis
  • Suturing as the test case: the physics of tissue apposition has never been published
  • Physical intelligence and humanoid robotics will improve simulation from the outside in

9. Likert scales are not measurement | 36:47

  • Binary, procedure-specific metrics require scoring the entire video, reliably, in pairs
  • One-to-five ratings after watching a few minutes of video are hand waving, not assessment
  • Ring exercises on robotic simulators have never been verified against real forces

10. Whose job is it? Credentialing and privileging | 39:27

  • Manufacturers certify device use; professional societies and hospitals grant privileges
  • Per-procedure privileging in the United States versus broad qualification in Europe
  • The credentialing committee problem: standards set by the least experienced member
  • Case volume, fellowship length and reputation are social proof, not evidence of competence

11. A jumbo jet a day: the human cost | 44:43

  • Deaths from surgical skills deficits estimated as equivalent to a full jumbo jet crashing every day, consistent with evidence that around 4.2 million people die within 30 days of surgery each year (Nepogodiev et al., The Lancet, 2019)
  • Why one death at a time never makes headlines the way one crash does
  • The Bristol Royal Infirmary case: peers knew for a decade before the front pages forced action (The Bristol Royal Infirmary Inquiry, Kennedy Report, 2001)
  • Telesurgery at scale: how one underperforming surgeon could soon harm many patients quickly

12. The PBP evidence and the leadership gap | 47:47

  • Systematic review and meta-analysis evidence: PBP-trained clinicians make approximately 60 per cent fewer objectively assessed intraoperative errors
  • Training to proficiency in one third of the time and at less than half the cost of conventional training
  • "A failure of leadership": adoption at ORSI Academy, AANA, ERUS under Alberto Breda, and increasingly Medtronic cardiovascular came from senior leaders making it non-negotiable
  • The aviation lesson: flight simulation was mandated by government before the evidence existed, while surgery has the evidence and no mandate

Publication: Mazzone, E., Puliatti, S., Amato, M., Bunting, B., Rocco, B., Montorsi, F., Mottrie, A. and Gallagher, A.G. (2021). A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes. Annals of Surgery, 274(2), 281-289. DOI: 10.1097/SLA.0000000000004650

Publication: Puliatti, S., Rodriguez Peñaranda, N., Amato, M., De Groote, R., Farinha, R., Bunting, B., van Cleynenbreugel, B., Mottrie, A. and Gallagher, A.G. (2026). Randomised trial on the economic impact of proficiency-based progression vs conventional robotic surgical training. BJU International, 137, 493-501. https://doi.org/10.1111/bju.70130

13. The next 10 years: AI, automation and the autonomy myth | 54:08

  • AI will lower the barrier to building simulators and add semi-automated instruction
  • Part-task automation, such as a supervised stapler, is plausible; autonomous procedures are not
  • The barrier is financial and regulatory: manufacturers will not accept liability for practising medicine
  • The self-driving analogy: millions of hours of verified synthetic data would ...
]]> training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices No Dr Ruben DeGroote, From Proficiency to Wisdom in Surgical Training 7 7 Dr Ruben DeGroote, From Proficiency to Wisdom in Surgical Training full 5594e6fc-1758-4c5c-b5ca-57b0887be3d3 https://share.transistor.fm/s/7aa03d61 Show Me the Evidence

Guest: Dr Ruben De Groote
Topic: From Time to Competence: Proficiency-Based Progression and the Reinvention of Robotic Surgical Training

Episode Summary

In this episode, Professor Tony Gallagher sits down with Dr Ruben De Groote, consultant urologist at OLV in Aalst, Belgium, CEO of 4Health and its digital learning platform Surgquest, and the researcher behind a recently completed PhD on Proficiency-Based Progression (PBP) robotic surgical training across three surgical disciplines. Ruben and Tony first met in 2019 while developing and validating the metrics for the robot-assisted radical prostatectomy.

Together they examine an uncomfortable reality: the century-old Halstedian apprenticeship model can no longer produce surgeons who are ready to operate independently. Reduced theatre exposure, rising bureaucracy, and working-hours legislation have hollowed out the "see one, do one, teach one" paradigm, leaving as many as one in three residents unready for independent practice. Ruben makes the evidence-based case for PBP: a standardised, metric-driven approach that measures what a surgeon actually does, gives explicit formative feedback, and trains to a benchmark rather than to a clock. The conversation moves from the failings of subjective assessment through a multi-specialty randomised controlled trial, and on to the harder question of how surgeons progress from proficiency to genuine wisdom.

Key Topics Covered

1. Why the apprenticeship model is breaking down (0:50) The Halstedian "see one, do one, teach one" model relied entirely on graded theatre exposure. Over the last 15 to 20 years, bureaucracy and a legal cap on working hours (departments are penalised for exceeding roughly 60 hours a week on average) have eroded that exposure. The result is a vicious circle in which trainees get less time in the operating room, and fellowships originally meant for super-specialisation are being repurposed simply to reach independence.

2. Where the bureaucracy came from (6:06) A wider shift in medicine towards risk aversion and defensive practice. Tasks that were once handled verbally now require written orders, increasing the administrative burden for everyone and pulling ambitious residents out of theatre for half a day or more.

3. Robotics as both a challenge and an opportunity (7:52) Robotic surgery combines complex procedures with the mastery of technology, which raises the training bar. It also places a computer between the surgeon's eyes and the patient, making it possible to store video and surgical data, review procedures, give formative feedback, and measure kinematics. This makes robotics a powerful tool for objectively measuring and improving surgical quality.

4. Exposure is not enough: the case for structure (9:31) Watching a procedure is not the same as being trained to perform it. Ruben and Tony agree that robotics demands the structure the Halstedian approach once imposed, but delivered through universal, evidence-driven standards and benchmarking, rather than the reputation of a single centre or trainer.

5. The systemic problem: a lack of standardisation (12:39) Without standardised curricula, trainees are dependent on the goodwill of whichever consultant they are assigned. Ruben describes a fellowship with six fellows and nine consultants, each teaching the same procedure differently, and warns that patients are effectively used as training models for consultants who were not well trained themselves.

6. Who should set and police the standards (17:37) Standards should be set by rigorous scientific research, not opinion. Scientific societies should define the benchmark and authorities should make it mandatory, in the same way prescribing rights follow formal qualification. Ruben cautions against a large role for industry, citing the conflict of interest in paid proctoring, where a proctor can be pushed to guide a novice through complex steps they have not earned the right to attempt.

7. The multi-specialty randomised controlled trial (23:15) A blinded RCT deliberately included urologists, general surgeons, and gynaecologists to test the belief that some specialties are inherently more skilled. At baseline all three performed equally, and after training all three performed equally well. The methodology, not the specialty, predicted the skill set. As reported in the episode, 67 per cent of PBP trainees reached proficiency by the end of the day, compared with 17 per cent trained by the apprenticeship model. [See PROVESA / De Groote RCT publications below.]

8. Quantifying intraoperative performance: why subjective scales fail (28:18) Likert-based tools such as GEARS are subjective and prone to drift, with a trainer's scoring shifting depending on the video seen just before. For validity, inter-reader agreement should be 80 per cent or higher; in Ruben's study GEARS reached only around 30 per cent, which by default makes it invalid for assessing surgical quality. Binary metrics are the alternative: procedure-specific, zero or one, either a step was performed or a defined error was made. They force assessment of the whole procedure and remove the room to "cheat".

9. Formative feedback in practice (34:59) In Aalst, fellows meet every Thursday to review a recorded procedure on a split screen, with the surgery on one side and the validated metrics on the other. Ruben facilitates, translating the metrics to the procedure and pinpointing exactly where an error occurred. This is transparent, non-subjective feedback that the whole group learns from, and it neutralises the "God complex" that can distort eminence-based teaching, since even high-volume experts sometimes score poorly against objective metrics.

10. Is PBP genuinely better? (38:01) Ruben's position is unambiguous: a methodology associated with a 60 per cent reduction in intraoperative errors compared with the apprenticeship model has to be accepted as better, and fewer errors translate into better patient outcomes. [See Mazzone et al. meta-analysis below.]

11. PBP beyond residents and beyond technical skills (45:14) PBP applies to residents, novice and experienced consultants, and nurses. It has been shown to sharpen non-technical skills too, including a study by Dorothy Breen applying PBP metrics to ICU patient handover using the ISBAR system, which made the process more efficient and filtered out unhelpful information. [See Breen et al. below.]

12. From proficiency to wisdom, and the role of Surgquest (52:01) Proficiency means performing a standard procedure safely. Wisdom is the further step: the volume of experience needed to keep improving, the ability to manage the unexpected, and access to experienced support when a case turns difficult. Surgquest, the 4Health digital learning platform, curates global experts demonstrating not just standard procedures but genuinely challenging cases, helping trainers give fellows more console time in the knowledge that a mistake can be repaired.


Publications and Evidence Cited

De Groote, R., Puliatti, S., Amato, M., Mazzone, E., Rosiello, G., Farinha, R., Paludo, A., Desender, L., Van Cleynenbreugel, B., Bunting, B.P., Mottrie, A., Gallagher, A.G. (2022). Proficiency-based progression training for robotic surgery skills training: a randomized clinical trial. BJU International. DOI: 10.1111/bju.15811. https://doi.org/10.1111/bju.15811


De Groote, R., Puliatti, S., Amato, M., et al. (2025). Does surg...

]]> Show Me the Evidence

Guest: Dr Ruben De Groote
Topic: From Time to Competence: Proficiency-Based Progression and the Reinvention of Robotic Surgical Training

Episode Summary

In this episode, Professor Tony Gallagher sits down with Dr Ruben De Groote, consultant urologist at OLV in Aalst, Belgium, CEO of 4Health and its digital learning platform Surgquest, and the researcher behind a recently completed PhD on Proficiency-Based Progression (PBP) robotic surgical training across three surgical disciplines. Ruben and Tony first met in 2019 while developing and validating the metrics for the robot-assisted radical prostatectomy.

Together they examine an uncomfortable reality: the century-old Halstedian apprenticeship model can no longer produce surgeons who are ready to operate independently. Reduced theatre exposure, rising bureaucracy, and working-hours legislation have hollowed out the "see one, do one, teach one" paradigm, leaving as many as one in three residents unready for independent practice. Ruben makes the evidence-based case for PBP: a standardised, metric-driven approach that measures what a surgeon actually does, gives explicit formative feedback, and trains to a benchmark rather than to a clock. The conversation moves from the failings of subjective assessment through a multi-specialty randomised controlled trial, and on to the harder question of how surgeons progress from proficiency to genuine wisdom.

Key Topics Covered

1. Why the apprenticeship model is breaking down (0:50) The Halstedian "see one, do one, teach one" model relied entirely on graded theatre exposure. Over the last 15 to 20 years, bureaucracy and a legal cap on working hours (departments are penalised for exceeding roughly 60 hours a week on average) have eroded that exposure. The result is a vicious circle in which trainees get less time in the operating room, and fellowships originally meant for super-specialisation are being repurposed simply to reach independence.

2. Where the bureaucracy came from (6:06) A wider shift in medicine towards risk aversion and defensive practice. Tasks that were once handled verbally now require written orders, increasing the administrative burden for everyone and pulling ambitious residents out of theatre for half a day or more.

3. Robotics as both a challenge and an opportunity (7:52) Robotic surgery combines complex procedures with the mastery of technology, which raises the training bar. It also places a computer between the surgeon's eyes and the patient, making it possible to store video and surgical data, review procedures, give formative feedback, and measure kinematics. This makes robotics a powerful tool for objectively measuring and improving surgical quality.

4. Exposure is not enough: the case for structure (9:31) Watching a procedure is not the same as being trained to perform it. Ruben and Tony agree that robotics demands the structure the Halstedian approach once imposed, but delivered through universal, evidence-driven standards and benchmarking, rather than the reputation of a single centre or trainer.

5. The systemic problem: a lack of standardisation (12:39) Without standardised curricula, trainees are dependent on the goodwill of whichever consultant they are assigned. Ruben describes a fellowship with six fellows and nine consultants, each teaching the same procedure differently, and warns that patients are effectively used as training models for consultants who were not well trained themselves.

6. Who should set and police the standards (17:37) Standards should be set by rigorous scientific research, not opinion. Scientific societies should define the benchmark and authorities should make it mandatory, in the same way prescribing rights follow formal qualification. Ruben cautions against a large role for industry, citing the conflict of interest in paid proctoring, where a proctor can be pushed to guide a novice through complex steps they have not earned the right to attempt.

7. The multi-specialty randomised controlled trial (23:15) A blinded RCT deliberately included urologists, general surgeons, and gynaecologists to test the belief that some specialties are inherently more skilled. At baseline all three performed equally, and after training all three performed equally well. The methodology, not the specialty, predicted the skill set. As reported in the episode, 67 per cent of PBP trainees reached proficiency by the end of the day, compared with 17 per cent trained by the apprenticeship model. [See PROVESA / De Groote RCT publications below.]

8. Quantifying intraoperative performance: why subjective scales fail (28:18) Likert-based tools such as GEARS are subjective and prone to drift, with a trainer's scoring shifting depending on the video seen just before. For validity, inter-reader agreement should be 80 per cent or higher; in Ruben's study GEARS reached only around 30 per cent, which by default makes it invalid for assessing surgical quality. Binary metrics are the alternative: procedure-specific, zero or one, either a step was performed or a defined error was made. They force assessment of the whole procedure and remove the room to "cheat".

9. Formative feedback in practice (34:59) In Aalst, fellows meet every Thursday to review a recorded procedure on a split screen, with the surgery on one side and the validated metrics on the other. Ruben facilitates, translating the metrics to the procedure and pinpointing exactly where an error occurred. This is transparent, non-subjective feedback that the whole group learns from, and it neutralises the "God complex" that can distort eminence-based teaching, since even high-volume experts sometimes score poorly against objective metrics.

10. Is PBP genuinely better? (38:01) Ruben's position is unambiguous: a methodology associated with a 60 per cent reduction in intraoperative errors compared with the apprenticeship model has to be accepted as better, and fewer errors translate into better patient outcomes. [See Mazzone et al. meta-analysis below.]

11. PBP beyond residents and beyond technical skills (45:14) PBP applies to residents, novice and experienced consultants, and nurses. It has been shown to sharpen non-technical skills too, including a study by Dorothy Breen applying PBP metrics to ICU patient handover using the ISBAR system, which made the process more efficient and filtered out unhelpful information. [See Breen et al. below.]

12. From proficiency to wisdom, and the role of Surgquest (52:01) Proficiency means performing a standard procedure safely. Wisdom is the further step: the volume of experience needed to keep improving, the ability to manage the unexpected, and access to experienced support when a case turns difficult. Surgquest, the 4Health digital learning platform, curates global experts demonstrating not just standard procedures but genuinely challenging cases, helping trainers give fellows more console time in the knowledge that a mistake can be repaired.


Publications and Evidence Cited

De Groote, R., Puliatti, S., Amato, M., Mazzone, E., Rosiello, G., Farinha, R., Paludo, A., Desender, L., Van Cleynenbreugel, B., Bunting, B.P., Mottrie, A., Gallagher, A.G. (2022). Proficiency-based progression training for robotic surgery skills training: a randomized clinical trial. BJU International. DOI: 10.1111/bju.15811. https://doi.org/10.1111/bju.15811


De Groote, R., Puliatti, S., Amato, M., et al. (2025). Does surg...

]]> Fri, 03 Jul 2026 17:52:38 +0100 Anthony G. Gallagher, Flux Learning Ltd Anthony G. Gallagher, Flux Learning Ltd 3605 Show Me the Evidence

Guest: Dr Ruben De Groote
Topic: From Time to Competence: Proficiency-Based Progression and the Reinvention of Robotic Surgical Training

Episode Summary

In this episode, Professor Tony Gallagher sits down with Dr Ruben De Groote, consultant urologist at OLV in Aalst, Belgium, CEO of 4Health and its digital learning platform Surgquest, and the researcher behind a recently completed PhD on Proficiency-Based Progression (PBP) robotic surgical training across three surgical disciplines. Ruben and Tony first met in 2019 while developing and validating the metrics for the robot-assisted radical prostatectomy.

Together they examine an uncomfortable reality: the century-old Halstedian apprenticeship model can no longer produce surgeons who are ready to operate independently. Reduced theatre exposure, rising bureaucracy, and working-hours legislation have hollowed out the "see one, do one, teach one" paradigm, leaving as many as one in three residents unready for independent practice. Ruben makes the evidence-based case for PBP: a standardised, metric-driven approach that measures what a surgeon actually does, gives explicit formative feedback, and trains to a benchmark rather than to a clock. The conversation moves from the failings of subjective assessment through a multi-specialty randomised controlled trial, and on to the harder question of how surgeons progress from proficiency to genuine wisdom.

Key Topics Covered

1. Why the apprenticeship model is breaking down (0:50) The Halstedian "see one, do one, teach one" model relied entirely on graded theatre exposure. Over the last 15 to 20 years, bureaucracy and a legal cap on working hours (departments are penalised for exceeding roughly 60 hours a week on average) have eroded that exposure. The result is a vicious circle in which trainees get less time in the operating room, and fellowships originally meant for super-specialisation are being repurposed simply to reach independence.

2. Where the bureaucracy came from (6:06) A wider shift in medicine towards risk aversion and defensive practice. Tasks that were once handled verbally now require written orders, increasing the administrative burden for everyone and pulling ambitious residents out of theatre for half a day or more.

3. Robotics as both a challenge and an opportunity (7:52) Robotic surgery combines complex procedures with the mastery of technology, which raises the training bar. It also places a computer between the surgeon's eyes and the patient, making it possible to store video and surgical data, review procedures, give formative feedback, and measure kinematics. This makes robotics a powerful tool for objectively measuring and improving surgical quality.

4. Exposure is not enough: the case for structure (9:31) Watching a procedure is not the same as being trained to perform it. Ruben and Tony agree that robotics demands the structure the Halstedian approach once imposed, but delivered through universal, evidence-driven standards and benchmarking, rather than the reputation of a single centre or trainer.

5. The systemic problem: a lack of standardisation (12:39) Without standardised curricula, trainees are dependent on the goodwill of whichever consultant they are assigned. Ruben describes a fellowship with six fellows and nine consultants, each teaching the same procedure differently, and warns that patients are effectively used as training models for consultants who were not well trained themselves.

6. Who should set and police the standards (17:37) Standards should be set by rigorous scientific research, not opinion. Scientific societies should define the benchmark and authorities should make it mandatory, in the same way prescribing rights follow formal qualification. Ruben cautions against a large role for industry, citing the conflict of interest in paid proctoring, where a proctor can be pushed to guide a novice through complex steps they have not earned the right to attempt.

7. The multi-specialty randomised controlled trial (23:15) A blinded RCT deliberately included urologists, general surgeons, and gynaecologists to test the belief that some specialties are inherently more skilled. At baseline all three performed equally, and after training all three performed equally well. The methodology, not the specialty, predicted the skill set. As reported in the episode, 67 per cent of PBP trainees reached proficiency by the end of the day, compared with 17 per cent trained by the apprenticeship model. [See PROVESA / De Groote RCT publications below.]

8. Quantifying intraoperative performance: why subjective scales fail (28:18) Likert-based tools such as GEARS are subjective and prone to drift, with a trainer's scoring shifting depending on the video seen just before. For validity, inter-reader agreement should be 80 per cent or higher; in Ruben's study GEARS reached only around 30 per cent, which by default makes it invalid for assessing surgical quality. Binary metrics are the alternative: procedure-specific, zero or one, either a step was performed or a defined error was made. They force assessment of the whole procedure and remove the room to "cheat".

9. Formative feedback in practice (34:59) In Aalst, fellows meet every Thursday to review a recorded procedure on a split screen, with the surgery on one side and the validated metrics on the other. Ruben facilitates, translating the metrics to the procedure and pinpointing exactly where an error occurred. This is transparent, non-subjective feedback that the whole group learns from, and it neutralises the "God complex" that can distort eminence-based teaching, since even high-volume experts sometimes score poorly against objective metrics.

10. Is PBP genuinely better? (38:01) Ruben's position is unambiguous: a methodology associated with a 60 per cent reduction in intraoperative errors compared with the apprenticeship model has to be accepted as better, and fewer errors translate into better patient outcomes. [See Mazzone et al. meta-analysis below.]

11. PBP beyond residents and beyond technical skills (45:14) PBP applies to residents, novice and experienced consultants, and nurses. It has been shown to sharpen non-technical skills too, including a study by Dorothy Breen applying PBP metrics to ICU patient handover using the ISBAR system, which made the process more efficient and filtered out unhelpful information. [See Breen et al. below.]

12. From proficiency to wisdom, and the role of Surgquest (52:01) Proficiency means performing a standard procedure safely. Wisdom is the further step: the volume of experience needed to keep improving, the ability to manage the unexpected, and access to experienced support when a case turns difficult. Surgquest, the 4Health digital learning platform, curates global experts demonstrating not just standard procedures but genuinely challenging cases, helping trainers give fellows more console time in the knowledge that a mistake can be repaired.


Publications and Evidence Cited

De Groote, R., Puliatti, S., Amato, M., Mazzone, E., Rosiello, G., Farinha, R., Paludo, A., Desender, L., Van Cleynenbreugel, B., Bunting, B.P., Mottrie, A., Gallagher, A.G. (2022). Proficiency-based progression training for robotic surgery skills training: a randomized clinical trial. BJU International. DOI: 10.1111/bju.15811. https://doi.org/10.1111/bju.15811


De Groote, R., Puliatti, S., Amato, M., et al. (2025). Does surg...

]]> training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices No Göran Malmberg, Mentice: Why Simulation Without Metrics Falls Short 6 6 Göran Malmberg, Mentice: Why Simulation Without Metrics Falls Short full 2a5b8f23-f24b-4eeb-a037-41fb3c98979e https://share.transistor.fm/s/179d91ab Guest: Göran Malmberg, Group CEO and President of Mentice AB (Gothenburg, Sweden), the world leader in physics-based virtual reality simulation for endovascular therapies

Host: Professor Anthony G Gallagher

Topic: Why Simulation Without Metrics Falls Short, and the Unsettled Question of Who Owns Medical Training


Episode Summary

In this episode, Professor Tony Gallagher is joined by Göran Malmberg, who has led Mentice since 2008 and built it into the global leader in physics-based VR simulation for endovascular therapies. Drawing on more than two decades at the meeting point of engineering, medical devices and clinical training, Göran and Tony confront an uncomfortable structural problem: training is still treated as a cost item rather than a driver of value, and no one can say clearly who owns the responsibility for proving that a clinician is ready to perform a procedure. They make the case that simulation without validated metrics is a suboptimal tool, that proficiency-based progression (PBP) is the route to structured skill, and that the next move belongs to whoever is willing to lead, whether that is a major device manufacturer, a regulator or a professional society.


Key Topics Covered

1. From High-Tech to High-Stakes — 0:00

  • Göran's route into medical simulation from automotive and industrial B2B technology
  • Why selling advanced technology is broadly similar across sectors, and what was genuinely new about medicine
  • Mentice and the early years of VR simulation in medicine

2. Training as a Cost, Not a Value — 2:13

  • Why structured training is rarely connected to return on investment
  • How it has often depended on the goodwill and personal time of passionate clinicians
  • The wider concern about a business ethos moving into academic medical centres, and where training and quality assurance fit

3. Why Simulation Needs Metrics — 6:40

  • Göran's agreement with Tony's central conclusion: any simulation without metrics is a suboptimal tool for procedure-based training
  • The difference between general-purpose simulation and structured training to a defined level of skill
  • Where proficiency-based progression fits

4. The Hard Part: Defining and Validating Metrics — 7:26

  • Why the biggest hurdle is getting device companies to define and validate metrics before they engage the simulation provider
  • The mechanical thrombectomy example, where clinicians pushed Mentice to build and subsequently patent a new device
  • Who should own the metrics: industry, professional medicine, or departments of health
  • The evidence that PBP-trained operators perform around 60% better in the clinical environment

Publication: Seymour NE, Gallagher AG, Roman SA, O'Brien MK, Bansal VK, Andersen DK, Satava RM. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Annals of Surgery. 2002;236(4):458-464. doi:10.1097/00000658-200210000-00008

5. The Misunderstood Cost of Physics-Based Simulation — 11:12

  • Why a physics-based VR simulation is not a video game
  • The months of programming, engineering and validation behind a credible simulator
  • How to weigh that cost against the alternative cost of proctored real cases over three to six months
  • Evidence that PBP simulation training can cost roughly a third of conventional training, and the question of why medicine has not collated this

Publication: Puliatti S, Rodriguez Peñaranda N, Amato M, De Groote R, Farinha R, Bunting B, van Cleynenbreugel B, Mottrie A, Gallagher AG. Randomised trial on the economic impact of proficiency-based progression versus conventional robotic surgical training. BJU International. 2026;137(3):493-501. doi:10.1111/bju.70130

6. The Evidence Industry Finds Hard to Hear — 18:00

  • The Birkmeyer finding that suboptimal performance in experienced clinicians leads to worse outcomes, with a difference of roughly 50 to 80 per cent
  • The Mascheroni IMPROF trial, run with Medtronic in Switzerland, where metrics-based training to proficiency produced markedly fewer intraoperative errors than traditional simulation training, and where Tony notes that none of the standard-trained group reached the proficiency benchmark
  • A Leuven PhD viva on a circular stapler, where PBP training removed the leak rate seen with standard instructions-for-use training

Publication: Birkmeyer JD, Finks JF, O'Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJO. Surgical skill and complication rates after bariatric surgery. New England Journal of Medicine. 2013;369(15):1434-1442. doi:10.1056/NEJMsa1300625

Publication: Mascheroni J, Stockburger M, Patwala A, Mont L, Rao A, Retzlaff H, Garweg C, Verbelen T, Gallagher AG. Effect of Metrics-Based Simulation Training to Proficiency on Procedure Quality and Errors Among Novice Cardiac Device Implanters: The IMPROF Randomized Trial. JAMA Network Open. 2023;6(8):e2322750. doi:10.1001/jamanetworkopen.2023.22750

7. Whose Job Is It Anyway? — 22:40

  • The sensitivity of telling a clinician they are not ready yet, and why the metrics come from a clinician's own expert peers, not from the device manufacturer
  • The unclear allocation of responsibility across industry, regulators, societies and hospitals
  • The shift towards clinical leadership in US hospitals as a possible route forward
  • Why manufacturers already hold much of the information they need, through their human factors teams

8. AI and Real-Time Performance Feedback — 30:09

  • Whether AI will deliver real-time, metric-based scoring on a VR simulation, and on what timeframe
  • Göran's view of three years or less, set against Tony's more cautious five to ten
  • The risk that AI repeats the laparoscopic generation's mistake of measuring process rather than the quality of performance
  • Why simulation may be the most controllable environment in which to train AI

9. Marketing or Training? Why Industry Hesitates — 34:03

  • Why device manufacturers are comfortable using VR simulation for marketing but less comfortable using it for metric-based training
  • The mix of unclear responsibility, sensitivity around assessing clinician skill, and the cost and effort of defining a programme
  • The "speed to volume" argument: properly trained clinicians use a device more safely, more confidently, and ultimately buy more of it
  • Why this is an uncomfortable issue for industry across Europe, the US and Asia

10. Should Every Hospital Have a Simulation Suite? — 39:39

  • Lessons from the large multi-discipline skills centres funded in the US in the mid-2000s
  • Why simulation works best close to the cath lab and integrated into the weekly clinical routine, rather than used a few times a year

11. Mandatory, Funded and Lifelong — 43:19

  • The case for making physics-based simulation a non-negotiable, centrally funded part of residency, backed by Departments of Health or the European Commission
  • Why training has to be continuous and lifelong as devices and procedures change every year
  • The unresolved question of certification structures and who holds the authority, illustrated by the American Board of Internal Medicine experience

12. ERIS, EHRA and What Trainees Want ...

]]> Guest: Göran Malmberg, Group CEO and President of Mentice AB (Gothenburg, Sweden), the world leader in physics-based virtual reality simulation for endovascular therapies

Host: Professor Anthony G Gallagher

Topic: Why Simulation Without Metrics Falls Short, and the Unsettled Question of Who Owns Medical Training


Episode Summary

In this episode, Professor Tony Gallagher is joined by Göran Malmberg, who has led Mentice since 2008 and built it into the global leader in physics-based VR simulation for endovascular therapies. Drawing on more than two decades at the meeting point of engineering, medical devices and clinical training, Göran and Tony confront an uncomfortable structural problem: training is still treated as a cost item rather than a driver of value, and no one can say clearly who owns the responsibility for proving that a clinician is ready to perform a procedure. They make the case that simulation without validated metrics is a suboptimal tool, that proficiency-based progression (PBP) is the route to structured skill, and that the next move belongs to whoever is willing to lead, whether that is a major device manufacturer, a regulator or a professional society.


Key Topics Covered

1. From High-Tech to High-Stakes — 0:00

  • Göran's route into medical simulation from automotive and industrial B2B technology
  • Why selling advanced technology is broadly similar across sectors, and what was genuinely new about medicine
  • Mentice and the early years of VR simulation in medicine

2. Training as a Cost, Not a Value — 2:13

  • Why structured training is rarely connected to return on investment
  • How it has often depended on the goodwill and personal time of passionate clinicians
  • The wider concern about a business ethos moving into academic medical centres, and where training and quality assurance fit

3. Why Simulation Needs Metrics — 6:40

  • Göran's agreement with Tony's central conclusion: any simulation without metrics is a suboptimal tool for procedure-based training
  • The difference between general-purpose simulation and structured training to a defined level of skill
  • Where proficiency-based progression fits

4. The Hard Part: Defining and Validating Metrics — 7:26

  • Why the biggest hurdle is getting device companies to define and validate metrics before they engage the simulation provider
  • The mechanical thrombectomy example, where clinicians pushed Mentice to build and subsequently patent a new device
  • Who should own the metrics: industry, professional medicine, or departments of health
  • The evidence that PBP-trained operators perform around 60% better in the clinical environment

Publication: Seymour NE, Gallagher AG, Roman SA, O'Brien MK, Bansal VK, Andersen DK, Satava RM. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Annals of Surgery. 2002;236(4):458-464. doi:10.1097/00000658-200210000-00008

5. The Misunderstood Cost of Physics-Based Simulation — 11:12

  • Why a physics-based VR simulation is not a video game
  • The months of programming, engineering and validation behind a credible simulator
  • How to weigh that cost against the alternative cost of proctored real cases over three to six months
  • Evidence that PBP simulation training can cost roughly a third of conventional training, and the question of why medicine has not collated this

Publication: Puliatti S, Rodriguez Peñaranda N, Amato M, De Groote R, Farinha R, Bunting B, van Cleynenbreugel B, Mottrie A, Gallagher AG. Randomised trial on the economic impact of proficiency-based progression versus conventional robotic surgical training. BJU International. 2026;137(3):493-501. doi:10.1111/bju.70130

6. The Evidence Industry Finds Hard to Hear — 18:00

  • The Birkmeyer finding that suboptimal performance in experienced clinicians leads to worse outcomes, with a difference of roughly 50 to 80 per cent
  • The Mascheroni IMPROF trial, run with Medtronic in Switzerland, where metrics-based training to proficiency produced markedly fewer intraoperative errors than traditional simulation training, and where Tony notes that none of the standard-trained group reached the proficiency benchmark
  • A Leuven PhD viva on a circular stapler, where PBP training removed the leak rate seen with standard instructions-for-use training

Publication: Birkmeyer JD, Finks JF, O'Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJO. Surgical skill and complication rates after bariatric surgery. New England Journal of Medicine. 2013;369(15):1434-1442. doi:10.1056/NEJMsa1300625

Publication: Mascheroni J, Stockburger M, Patwala A, Mont L, Rao A, Retzlaff H, Garweg C, Verbelen T, Gallagher AG. Effect of Metrics-Based Simulation Training to Proficiency on Procedure Quality and Errors Among Novice Cardiac Device Implanters: The IMPROF Randomized Trial. JAMA Network Open. 2023;6(8):e2322750. doi:10.1001/jamanetworkopen.2023.22750

7. Whose Job Is It Anyway? — 22:40

  • The sensitivity of telling a clinician they are not ready yet, and why the metrics come from a clinician's own expert peers, not from the device manufacturer
  • The unclear allocation of responsibility across industry, regulators, societies and hospitals
  • The shift towards clinical leadership in US hospitals as a possible route forward
  • Why manufacturers already hold much of the information they need, through their human factors teams

8. AI and Real-Time Performance Feedback — 30:09

  • Whether AI will deliver real-time, metric-based scoring on a VR simulation, and on what timeframe
  • Göran's view of three years or less, set against Tony's more cautious five to ten
  • The risk that AI repeats the laparoscopic generation's mistake of measuring process rather than the quality of performance
  • Why simulation may be the most controllable environment in which to train AI

9. Marketing or Training? Why Industry Hesitates — 34:03

  • Why device manufacturers are comfortable using VR simulation for marketing but less comfortable using it for metric-based training
  • The mix of unclear responsibility, sensitivity around assessing clinician skill, and the cost and effort of defining a programme
  • The "speed to volume" argument: properly trained clinicians use a device more safely, more confidently, and ultimately buy more of it
  • Why this is an uncomfortable issue for industry across Europe, the US and Asia

10. Should Every Hospital Have a Simulation Suite? — 39:39

  • Lessons from the large multi-discipline skills centres funded in the US in the mid-2000s
  • Why simulation works best close to the cath lab and integrated into the weekly clinical routine, rather than used a few times a year

11. Mandatory, Funded and Lifelong — 43:19

  • The case for making physics-based simulation a non-negotiable, centrally funded part of residency, backed by Departments of Health or the European Commission
  • Why training has to be continuous and lifelong as devices and procedures change every year
  • The unresolved question of certification structures and who holds the authority, illustrated by the American Board of Internal Medicine experience

12. ERIS, EHRA and What Trainees Want ...

]]> Sun, 28 Jun 2026 09:13:23 +0100 Anthony G. Gallagher, Flux Learning Ltd Anthony G. Gallagher, Flux Learning Ltd 3758 Guest: Göran Malmberg, Group CEO and President of Mentice AB (Gothenburg, Sweden), the world leader in physics-based virtual reality simulation for endovascular therapies

Host: Professor Anthony G Gallagher

Topic: Why Simulation Without Metrics Falls Short, and the Unsettled Question of Who Owns Medical Training


Episode Summary

In this episode, Professor Tony Gallagher is joined by Göran Malmberg, who has led Mentice since 2008 and built it into the global leader in physics-based VR simulation for endovascular therapies. Drawing on more than two decades at the meeting point of engineering, medical devices and clinical training, Göran and Tony confront an uncomfortable structural problem: training is still treated as a cost item rather than a driver of value, and no one can say clearly who owns the responsibility for proving that a clinician is ready to perform a procedure. They make the case that simulation without validated metrics is a suboptimal tool, that proficiency-based progression (PBP) is the route to structured skill, and that the next move belongs to whoever is willing to lead, whether that is a major device manufacturer, a regulator or a professional society.


Key Topics Covered

1. From High-Tech to High-Stakes — 0:00

  • Göran's route into medical simulation from automotive and industrial B2B technology
  • Why selling advanced technology is broadly similar across sectors, and what was genuinely new about medicine
  • Mentice and the early years of VR simulation in medicine

2. Training as a Cost, Not a Value — 2:13

  • Why structured training is rarely connected to return on investment
  • How it has often depended on the goodwill and personal time of passionate clinicians
  • The wider concern about a business ethos moving into academic medical centres, and where training and quality assurance fit

3. Why Simulation Needs Metrics — 6:40

  • Göran's agreement with Tony's central conclusion: any simulation without metrics is a suboptimal tool for procedure-based training
  • The difference between general-purpose simulation and structured training to a defined level of skill
  • Where proficiency-based progression fits

4. The Hard Part: Defining and Validating Metrics — 7:26

  • Why the biggest hurdle is getting device companies to define and validate metrics before they engage the simulation provider
  • The mechanical thrombectomy example, where clinicians pushed Mentice to build and subsequently patent a new device
  • Who should own the metrics: industry, professional medicine, or departments of health
  • The evidence that PBP-trained operators perform around 60% better in the clinical environment

Publication: Seymour NE, Gallagher AG, Roman SA, O'Brien MK, Bansal VK, Andersen DK, Satava RM. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Annals of Surgery. 2002;236(4):458-464. doi:10.1097/00000658-200210000-00008

5. The Misunderstood Cost of Physics-Based Simulation — 11:12

  • Why a physics-based VR simulation is not a video game
  • The months of programming, engineering and validation behind a credible simulator
  • How to weigh that cost against the alternative cost of proctored real cases over three to six months
  • Evidence that PBP simulation training can cost roughly a third of conventional training, and the question of why medicine has not collated this

Publication: Puliatti S, Rodriguez Peñaranda N, Amato M, De Groote R, Farinha R, Bunting B, van Cleynenbreugel B, Mottrie A, Gallagher AG. Randomised trial on the economic impact of proficiency-based progression versus conventional robotic surgical training. BJU International. 2026;137(3):493-501. doi:10.1111/bju.70130

6. The Evidence Industry Finds Hard to Hear — 18:00

  • The Birkmeyer finding that suboptimal performance in experienced clinicians leads to worse outcomes, with a difference of roughly 50 to 80 per cent
  • The Mascheroni IMPROF trial, run with Medtronic in Switzerland, where metrics-based training to proficiency produced markedly fewer intraoperative errors than traditional simulation training, and where Tony notes that none of the standard-trained group reached the proficiency benchmark
  • A Leuven PhD viva on a circular stapler, where PBP training removed the leak rate seen with standard instructions-for-use training

Publication: Birkmeyer JD, Finks JF, O'Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJO. Surgical skill and complication rates after bariatric surgery. New England Journal of Medicine. 2013;369(15):1434-1442. doi:10.1056/NEJMsa1300625

Publication: Mascheroni J, Stockburger M, Patwala A, Mont L, Rao A, Retzlaff H, Garweg C, Verbelen T, Gallagher AG. Effect of Metrics-Based Simulation Training to Proficiency on Procedure Quality and Errors Among Novice Cardiac Device Implanters: The IMPROF Randomized Trial. JAMA Network Open. 2023;6(8):e2322750. doi:10.1001/jamanetworkopen.2023.22750

7. Whose Job Is It Anyway? — 22:40

  • The sensitivity of telling a clinician they are not ready yet, and why the metrics come from a clinician's own expert peers, not from the device manufacturer
  • The unclear allocation of responsibility across industry, regulators, societies and hospitals
  • The shift towards clinical leadership in US hospitals as a possible route forward
  • Why manufacturers already hold much of the information they need, through their human factors teams

8. AI and Real-Time Performance Feedback — 30:09

  • Whether AI will deliver real-time, metric-based scoring on a VR simulation, and on what timeframe
  • Göran's view of three years or less, set against Tony's more cautious five to ten
  • The risk that AI repeats the laparoscopic generation's mistake of measuring process rather than the quality of performance
  • Why simulation may be the most controllable environment in which to train AI

9. Marketing or Training? Why Industry Hesitates — 34:03

  • Why device manufacturers are comfortable using VR simulation for marketing but less comfortable using it for metric-based training
  • The mix of unclear responsibility, sensitivity around assessing clinician skill, and the cost and effort of defining a programme
  • The "speed to volume" argument: properly trained clinicians use a device more safely, more confidently, and ultimately buy more of it
  • Why this is an uncomfortable issue for industry across Europe, the US and Asia

10. Should Every Hospital Have a Simulation Suite? — 39:39

  • Lessons from the large multi-discipline skills centres funded in the US in the mid-2000s
  • Why simulation works best close to the cath lab and integrated into the weekly clinical routine, rather than used a few times a year

11. Mandatory, Funded and Lifelong — 43:19

  • The case for making physics-based simulation a non-negotiable, centrally funded part of residency, backed by Departments of Health or the European Commission
  • Why training has to be continuous and lifelong as devices and procedures change every year
  • The unresolved question of certification structures and who holds the authority, illustrated by the American Board of Internal Medicine experience

12. ERIS, EHRA and What Trainees Want ...

]]> training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices No Dr Richard Satava: The Science of Surgical Training- Time, Profit and Surgical Simulation 5 5 Dr Richard Satava: The Science of Surgical Training- Time, Profit and Surgical Simulation full af0c3147-82c1-4c60-87d5-00e961f6f9fa https://share.transistor.fm/s/41e13d4c The evidence that simulation and proficiency-based training produce better, safer surgeons has been settled for over two decades. So why has surgery been so slow to take it up?


Professor Tony Gallagher talks to surgical simulation pioneer Dr Richard "Rick" Satava, Professor Emeritus of Surgery at the University of Washington and a former DARPA programme manager, about the study they ran together at Yale, why the operating room is the wrong place to learn basic skills, and why the real barriers to change are time, leadership and money rather than the science. They close on the limits of what AI can assess today, and what comes next, from telesurgery to surgery in space.


(0:00) Thirty years, and one seminal study

(2:07) Simulation as a revolution, transposed from the military

(3:23) Why the operating room is the wrong place to learn basic skills

(6:30) Treating surgical education as a science is disruptive

(9:39) The 2002 Yale study, and the end of "see one, do one"

(11:58) Disruption keeps coming: telesurgery, space and directed energy

(13:01) The barrier is not the evidence, it is selling it

(16:21) Leadership: the skills lab before theatre

(18:06) Who should own the standards and credentialing

(21:00) Twenty-five years on: training hours and fellowships

(25:52) You cannot compress the time, people learn at different rates

(32:26) Why proficiency-based progression is demanding to deliver

(35:07) Protected time in practice: the Wednesday afternoon model

(44:18) Why proficiency-based progression has not gained traction

(45:08) Formative feedback and the meaning of progression

(50:28) Where AI actually is today

(52:02) Convincing the establishment that simulation is patient care

(58:49) Service over training, and the business of medicine


Studies mentioned:


Seymour, Gallagher, Satava et al. Virtual reality training improves operating room performance. Annals of Surgery, 2002. https://journals.lww.com/annalsofsurgery/abstract/2002/10000/virtual_reality_training_improves_operating_room.8.aspx


Satava, Stefanidis, Levy et al. Fundamentals of Robotic Surgery (FRS) skills curriculum trial. Annals of Surgery, 2020. https://pubmed.ncbi.nlm.nih.gov/30720503


Gallagher, Ritter, Satava et al. Proficiency-based training as a paradigm shift. Annals of Surgery, 2005. https://pubmed.ncbi.nlm.nih.gov/15650649


Satava and Gallagher. Proficiency-based progression for FRS curriculum development. Annals of Laparoscopic and Endoscopic Surgery, 2020. https://ales.amegroups.org/article/view/5782/html



Guest: Dr Richard M. Satava


Host: Professor Anthony G. Gallagher. LinkedIn: https://www.linkedin.com/in/anthony-g-gallagher/

]]> The evidence that simulation and proficiency-based training produce better, safer surgeons has been settled for over two decades. So why has surgery been so slow to take it up?


Professor Tony Gallagher talks to surgical simulation pioneer Dr Richard "Rick" Satava, Professor Emeritus of Surgery at the University of Washington and a former DARPA programme manager, about the study they ran together at Yale, why the operating room is the wrong place to learn basic skills, and why the real barriers to change are time, leadership and money rather than the science. They close on the limits of what AI can assess today, and what comes next, from telesurgery to surgery in space.


(0:00) Thirty years, and one seminal study

(2:07) Simulation as a revolution, transposed from the military

(3:23) Why the operating room is the wrong place to learn basic skills

(6:30) Treating surgical education as a science is disruptive

(9:39) The 2002 Yale study, and the end of "see one, do one"

(11:58) Disruption keeps coming: telesurgery, space and directed energy

(13:01) The barrier is not the evidence, it is selling it

(16:21) Leadership: the skills lab before theatre

(18:06) Who should own the standards and credentialing

(21:00) Twenty-five years on: training hours and fellowships

(25:52) You cannot compress the time, people learn at different rates

(32:26) Why proficiency-based progression is demanding to deliver

(35:07) Protected time in practice: the Wednesday afternoon model

(44:18) Why proficiency-based progression has not gained traction

(45:08) Formative feedback and the meaning of progression

(50:28) Where AI actually is today

(52:02) Convincing the establishment that simulation is patient care

(58:49) Service over training, and the business of medicine


Studies mentioned:


Seymour, Gallagher, Satava et al. Virtual reality training improves operating room performance. Annals of Surgery, 2002. https://journals.lww.com/annalsofsurgery/abstract/2002/10000/virtual_reality_training_improves_operating_room.8.aspx


Satava, Stefanidis, Levy et al. Fundamentals of Robotic Surgery (FRS) skills curriculum trial. Annals of Surgery, 2020. https://pubmed.ncbi.nlm.nih.gov/30720503


Gallagher, Ritter, Satava et al. Proficiency-based training as a paradigm shift. Annals of Surgery, 2005. https://pubmed.ncbi.nlm.nih.gov/15650649


Satava and Gallagher. Proficiency-based progression for FRS curriculum development. Annals of Laparoscopic and Endoscopic Surgery, 2020. https://ales.amegroups.org/article/view/5782/html



Guest: Dr Richard M. Satava


Host: Professor Anthony G. Gallagher. LinkedIn: https://www.linkedin.com/in/anthony-g-gallagher/

]]> Mon, 22 Jun 2026 02:54:18 +0100 Anthony G. Gallagher, Flux Learning Ltd Anthony G. Gallagher, Flux Learning Ltd 4246 The evidence that simulation and proficiency-based training produce better, safer surgeons has been settled for over two decades. So why has surgery been so slow to take it up?


Professor Tony Gallagher talks to surgical simulation pioneer Dr Richard "Rick" Satava, Professor Emeritus of Surgery at the University of Washington and a former DARPA programme manager, about the study they ran together at Yale, why the operating room is the wrong place to learn basic skills, and why the real barriers to change are time, leadership and money rather than the science. They close on the limits of what AI can assess today, and what comes next, from telesurgery to surgery in space.


(0:00) Thirty years, and one seminal study

(2:07) Simulation as a revolution, transposed from the military

(3:23) Why the operating room is the wrong place to learn basic skills

(6:30) Treating surgical education as a science is disruptive

(9:39) The 2002 Yale study, and the end of "see one, do one"

(11:58) Disruption keeps coming: telesurgery, space and directed energy

(13:01) The barrier is not the evidence, it is selling it

(16:21) Leadership: the skills lab before theatre

(18:06) Who should own the standards and credentialing

(21:00) Twenty-five years on: training hours and fellowships

(25:52) You cannot compress the time, people learn at different rates

(32:26) Why proficiency-based progression is demanding to deliver

(35:07) Protected time in practice: the Wednesday afternoon model

(44:18) Why proficiency-based progression has not gained traction

(45:08) Formative feedback and the meaning of progression

(50:28) Where AI actually is today

(52:02) Convincing the establishment that simulation is patient care

(58:49) Service over training, and the business of medicine


Studies mentioned:


Seymour, Gallagher, Satava et al. Virtual reality training improves operating room performance. Annals of Surgery, 2002. https://journals.lww.com/annalsofsurgery/abstract/2002/10000/virtual_reality_training_improves_operating_room.8.aspx


Satava, Stefanidis, Levy et al. Fundamentals of Robotic Surgery (FRS) skills curriculum trial. Annals of Surgery, 2020. https://pubmed.ncbi.nlm.nih.gov/30720503


Gallagher, Ritter, Satava et al. Proficiency-based training as a paradigm shift. Annals of Surgery, 2005. https://pubmed.ncbi.nlm.nih.gov/15650649


Satava and Gallagher. Proficiency-based progression for FRS curriculum development. Annals of Laparoscopic and Endoscopic Surgery, 2020. https://ales.amegroups.org/article/view/5782/html



Guest: Dr Richard M. Satava


Host: Professor Anthony G. Gallagher. LinkedIn: https://www.linkedin.com/in/anthony-g-gallagher/

]]> training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices No Geraldine Hennessy Dr. Richard Angelo: From Apprenticeship to Proficiency — Rethinking How We Train Surgeons 4 4 Dr. Richard Angelo: From Apprenticeship to Proficiency — Rethinking How We Train Surgeons full 9f9918d8-b33b-49a1-bed1-bddbbb7aef11 https://share.transistor.fm/s/53a80e77 Episode 4 — Dr. Richard Angelo

Guest

Dr. Richard (Rick) Angelo — Arthroscopic surgeon based in Seattle; former President of the Arthroscopic Association of North America (AANA). Holds a PhD in proficiency-based progression training.

Host

Tony (relationship with Rick spans ~15 years, originating from a chance meeting at a conference in Sweden)

Episode Overview

A deep-dive conversation on the fundamental failures of traditional surgical training and how proficiency-based progression (PBP) training offers a scientifically rigorous alternative. The discussion centres on the landmark Copernicus Study — the first study in medicine to use proficiency demonstration as an outcome measure.

Key Topics Covered

1. Limitations of the Traditional Apprenticeship Model

  • The "see one, do one, teach one" model lacks objective assessment
  • Despite decades of training and significant investment, AANA could not verify whether skill acquisition was actually occurring
  • Complication rates and suboptimal outcomes weren't improving with existing training efforts

2. The Founding Question

  • Rick, during his time in the AANA presidential line, asked: "Is there a better way to train surgical skills?"
  • This led to engagement with Tony's work on proficiency-based progression training

3. Proficiency-Based Progression (PBP) Training — Core Principles

  • Define a clear target: what does quality performance of a procedure look like?
  • Deconstruct tasks into discrete, trainable components
  • Develop objective, binary metrics (did it occur or not?) rather than global rating scales
  • Establish inter-rater reliability between assessors
  • Trainees must demonstrate a benchmark at each stage before progressing (including cognitive pre-course material — 83% threshold)
  • Errors and deviations from optimal performance are trained explicitly — not just steps

4. The Bankart Repair — Why It Was Chosen

  • Common procedure with a broad, transferable skill set
  • Suited to task deconstruction and partial task simulation
  • Chosen by Rick and endorsed by the AANA core group

5. Curriculum Before Simulation

  • A critical insight: the curriculum and metrics must be developed first; simulation is chosen to match, not the other way around
  • Contrast with the wider medical field's focus on "eye candy" VR simulators that lack meaningful metrics
  • The FAST model (Fundamentals of Arthroscopic Surgery Training) was developed with Rob Pedowitz for knot tying — a low-cost, highly accurate partial task trainer
  • Even a simple conical nail punch from a garage became an effective tool for measuring loop elongation

6. The Copernicus Study — Design & Results
Three study groups:

  • Group A (Traditional): Lectures, open-access knot-tying lab, cadaver session — standard AANA approach
  • Group B (Simulator only): Access to the simulator without the PBP curriculum or metrics
  • Group C (PBP): Proficiency benchmarks at every stage — cognitive, knot-tying, and shoulder model

Results:

  • Group B was 1.4× more likely than Group A to meet the benchmark (marginal)
  • Group C participants (assigned to PBP, even without passing all benchmarks): 5.5× more likely than Group A
  • Group C participants who met all proficiency benchmarks: 7.5× more likely to meet the final benchmark
  • Error reduction: ~56% decrease in Bankart errors; ~58% for rotator cuff repair
  • In one follow-up weekend cohort of 18 trainees: 89% demonstrated proficiency in Bankart repair; 83% in rotator cuff repair

7. Key Finding: The Deficiency is in Training, Not Trainees

  • Pre-study concern about a "weed-out process" proved unfounded
  • With quality training, almost all trainees can master the required skills
  • Referenced Frank Lewis (former Chair, American Board of Surgery) sharing the same observation
  • Stefano Pogliani's study demonstrated near-universal proficiency is achievable

8. The Role of Errors in Surgical Training

  • Distinguishing novice from expert performers is best predicted by error enactment, not step completion
  • Each deviation from optimal performance creates a cascade risk — even if consequences aren't immediate
  • Upcoming study expected to show errors are the best predictor of patient outcomes

9. Broader Applicability to Procedure-Based Medicine

  • Principles apply across disciplines — cardiology, robotics, and beyond
  • Contrast drawn with VR simulator manufacturers at the European Heart Rhythm Association Conference (Paris), where most simulations had no metrics
  • Chicken tissue models used successfully in robotic surgery training at €5 per chicken — effective without being high-tech

10. Credentialing and Quality Assurance

  • Discussion of whether PBP methodology could or should underpin credentialing for new procedures or devices
  • Device failures in the field often attributable to inadequate clinician preparation, not device defects
  • Practical challenges for societal credentialing (procedure selection, remediation pathways, cost of metric development, legal defensibility)
  • European Commission is moving toward micro-credentials for technical skills — awarded by universities, recognised across EU member states
  • Both speakers agree: medicine must develop objective, procedure-based performance assessment for the public good
  • Analogy: demonstrating more skill is required to get a driver's licence than is currently required of surgeons in terms of objective performance assessment
]]> Episode 4 — Dr. Richard Angelo

Guest

Dr. Richard (Rick) Angelo — Arthroscopic surgeon based in Seattle; former President of the Arthroscopic Association of North America (AANA). Holds a PhD in proficiency-based progression training.

Host

Tony (relationship with Rick spans ~15 years, originating from a chance meeting at a conference in Sweden)

Episode Overview

A deep-dive conversation on the fundamental failures of traditional surgical training and how proficiency-based progression (PBP) training offers a scientifically rigorous alternative. The discussion centres on the landmark Copernicus Study — the first study in medicine to use proficiency demonstration as an outcome measure.

Key Topics Covered

1. Limitations of the Traditional Apprenticeship Model

  • The "see one, do one, teach one" model lacks objective assessment
  • Despite decades of training and significant investment, AANA could not verify whether skill acquisition was actually occurring
  • Complication rates and suboptimal outcomes weren't improving with existing training efforts

2. The Founding Question

  • Rick, during his time in the AANA presidential line, asked: "Is there a better way to train surgical skills?"
  • This led to engagement with Tony's work on proficiency-based progression training

3. Proficiency-Based Progression (PBP) Training — Core Principles

  • Define a clear target: what does quality performance of a procedure look like?
  • Deconstruct tasks into discrete, trainable components
  • Develop objective, binary metrics (did it occur or not?) rather than global rating scales
  • Establish inter-rater reliability between assessors
  • Trainees must demonstrate a benchmark at each stage before progressing (including cognitive pre-course material — 83% threshold)
  • Errors and deviations from optimal performance are trained explicitly — not just steps

4. The Bankart Repair — Why It Was Chosen

  • Common procedure with a broad, transferable skill set
  • Suited to task deconstruction and partial task simulation
  • Chosen by Rick and endorsed by the AANA core group

5. Curriculum Before Simulation

  • A critical insight: the curriculum and metrics must be developed first; simulation is chosen to match, not the other way around
  • Contrast with the wider medical field's focus on "eye candy" VR simulators that lack meaningful metrics
  • The FAST model (Fundamentals of Arthroscopic Surgery Training) was developed with Rob Pedowitz for knot tying — a low-cost, highly accurate partial task trainer
  • Even a simple conical nail punch from a garage became an effective tool for measuring loop elongation

6. The Copernicus Study — Design & Results
Three study groups:

  • Group A (Traditional): Lectures, open-access knot-tying lab, cadaver session — standard AANA approach
  • Group B (Simulator only): Access to the simulator without the PBP curriculum or metrics
  • Group C (PBP): Proficiency benchmarks at every stage — cognitive, knot-tying, and shoulder model

Results:

  • Group B was 1.4× more likely than Group A to meet the benchmark (marginal)
  • Group C participants (assigned to PBP, even without passing all benchmarks): 5.5× more likely than Group A
  • Group C participants who met all proficiency benchmarks: 7.5× more likely to meet the final benchmark
  • Error reduction: ~56% decrease in Bankart errors; ~58% for rotator cuff repair
  • In one follow-up weekend cohort of 18 trainees: 89% demonstrated proficiency in Bankart repair; 83% in rotator cuff repair

7. Key Finding: The Deficiency is in Training, Not Trainees

  • Pre-study concern about a "weed-out process" proved unfounded
  • With quality training, almost all trainees can master the required skills
  • Referenced Frank Lewis (former Chair, American Board of Surgery) sharing the same observation
  • Stefano Pogliani's study demonstrated near-universal proficiency is achievable

8. The Role of Errors in Surgical Training

  • Distinguishing novice from expert performers is best predicted by error enactment, not step completion
  • Each deviation from optimal performance creates a cascade risk — even if consequences aren't immediate
  • Upcoming study expected to show errors are the best predictor of patient outcomes

9. Broader Applicability to Procedure-Based Medicine

  • Principles apply across disciplines — cardiology, robotics, and beyond
  • Contrast drawn with VR simulator manufacturers at the European Heart Rhythm Association Conference (Paris), where most simulations had no metrics
  • Chicken tissue models used successfully in robotic surgery training at €5 per chicken — effective without being high-tech

10. Credentialing and Quality Assurance

  • Discussion of whether PBP methodology could or should underpin credentialing for new procedures or devices
  • Device failures in the field often attributable to inadequate clinician preparation, not device defects
  • Practical challenges for societal credentialing (procedure selection, remediation pathways, cost of metric development, legal defensibility)
  • European Commission is moving toward micro-credentials for technical skills — awarded by universities, recognised across EU member states
  • Both speakers agree: medicine must develop objective, procedure-based performance assessment for the public good
  • Analogy: demonstrating more skill is required to get a driver's licence than is currently required of surgeons in terms of objective performance assessment
]]> Fri, 29 May 2026 18:10:18 +0100 Anthony G. Gallagher, Flux Learning Ltd Anthony G. Gallagher, Flux Learning Ltd 2901 Episode 4 — Dr. Richard Angelo

Guest

Dr. Richard (Rick) Angelo — Arthroscopic surgeon based in Seattle; former President of the Arthroscopic Association of North America (AANA). Holds a PhD in proficiency-based progression training.

Host

Tony (relationship with Rick spans ~15 years, originating from a chance meeting at a conference in Sweden)

Episode Overview

A deep-dive conversation on the fundamental failures of traditional surgical training and how proficiency-based progression (PBP) training offers a scientifically rigorous alternative. The discussion centres on the landmark Copernicus Study — the first study in medicine to use proficiency demonstration as an outcome measure.

Key Topics Covered

1. Limitations of the Traditional Apprenticeship Model

  • The "see one, do one, teach one" model lacks objective assessment
  • Despite decades of training and significant investment, AANA could not verify whether skill acquisition was actually occurring
  • Complication rates and suboptimal outcomes weren't improving with existing training efforts

2. The Founding Question

  • Rick, during his time in the AANA presidential line, asked: "Is there a better way to train surgical skills?"
  • This led to engagement with Tony's work on proficiency-based progression training

3. Proficiency-Based Progression (PBP) Training — Core Principles

  • Define a clear target: what does quality performance of a procedure look like?
  • Deconstruct tasks into discrete, trainable components
  • Develop objective, binary metrics (did it occur or not?) rather than global rating scales
  • Establish inter-rater reliability between assessors
  • Trainees must demonstrate a benchmark at each stage before progressing (including cognitive pre-course material — 83% threshold)
  • Errors and deviations from optimal performance are trained explicitly — not just steps

4. The Bankart Repair — Why It Was Chosen

  • Common procedure with a broad, transferable skill set
  • Suited to task deconstruction and partial task simulation
  • Chosen by Rick and endorsed by the AANA core group

5. Curriculum Before Simulation

  • A critical insight: the curriculum and metrics must be developed first; simulation is chosen to match, not the other way around
  • Contrast with the wider medical field's focus on "eye candy" VR simulators that lack meaningful metrics
  • The FAST model (Fundamentals of Arthroscopic Surgery Training) was developed with Rob Pedowitz for knot tying — a low-cost, highly accurate partial task trainer
  • Even a simple conical nail punch from a garage became an effective tool for measuring loop elongation

6. The Copernicus Study — Design & Results
Three study groups:

  • Group A (Traditional): Lectures, open-access knot-tying lab, cadaver session — standard AANA approach
  • Group B (Simulator only): Access to the simulator without the PBP curriculum or metrics
  • Group C (PBP): Proficiency benchmarks at every stage — cognitive, knot-tying, and shoulder model

Results:

  • Group B was 1.4× more likely than Group A to meet the benchmark (marginal)
  • Group C participants (assigned to PBP, even without passing all benchmarks): 5.5× more likely than Group A
  • Group C participants who met all proficiency benchmarks: 7.5× more likely to meet the final benchmark
  • Error reduction: ~56% decrease in Bankart errors; ~58% for rotator cuff repair
  • In one follow-up weekend cohort of 18 trainees: 89% demonstrated proficiency in Bankart repair; 83% in rotator cuff repair

7. Key Finding: The Deficiency is in Training, Not Trainees

  • Pre-study concern about a "weed-out process" proved unfounded
  • With quality training, almost all trainees can master the required skills
  • Referenced Frank Lewis (former Chair, American Board of Surgery) sharing the same observation
  • Stefano Pogliani's study demonstrated near-universal proficiency is achievable

8. The Role of Errors in Surgical Training

  • Distinguishing novice from expert performers is best predicted by error enactment, not step completion
  • Each deviation from optimal performance creates a cascade risk — even if consequences aren't immediate
  • Upcoming study expected to show errors are the best predictor of patient outcomes

9. Broader Applicability to Procedure-Based Medicine

  • Principles apply across disciplines — cardiology, robotics, and beyond
  • Contrast drawn with VR simulator manufacturers at the European Heart Rhythm Association Conference (Paris), where most simulations had no metrics
  • Chicken tissue models used successfully in robotic surgery training at €5 per chicken — effective without being high-tech

10. Credentialing and Quality Assurance

  • Discussion of whether PBP methodology could or should underpin credentialing for new procedures or devices
  • Device failures in the field often attributable to inadequate clinician preparation, not device defects
  • Practical challenges for societal credentialing (procedure selection, remediation pathways, cost of metric development, legal defensibility)
  • European Commission is moving toward micro-credentials for technical skills — awarded by universities, recognised across EU member states
  • Both speakers agree: medicine must develop objective, procedure-based performance assessment for the public good
  • Analogy: demonstrating more skill is required to get a driver's licence than is currently required of surgeons in terms of objective performance assessment
]]> training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices No Geraldine Hennessy From the FDA to the operating theatre: how proficiency rewrote the rules of surgical training 3 3 From the FDA to the operating theatre: how proficiency rewrote the rules of surgical training full 40959753-ac18-4e18-8eef-8056877efea6 https://share.transistor.fm/s/23ed31df

Guest: Professor Anthony G Gallagher Host: Patrick Kiely
Episode focus: two landmark studies, the 2004 JAMA carotid stenting paper and the Copernicus arthroscopy trial


Episode summary

Professor Tony Gallagher, the founder of Proficiency-Based Progression (PBP), joins Patrick Kiely to revisit two studies that changed how we think about surgical competence.

The first is the 2004 JAMA paper describing a closed-door meeting at which the US Food and Drug Administration agreed, for the first time, that simulation training should form part of how doctors are approved to perform a procedure. The second is the Copernicus trial in shoulder arthroscopy, which showed that a simulator only improves training when it is paired with validated metrics and a clear proficiency benchmark.

Together they make a simple, evidence-led case: competence should be measured by the skill a clinician can demonstrate, not by years served or cases counted.


Chapters

0:00 Introduction
0:50 Inside the 2004 closed-door FDA meeting on carotid stenting
4:43 Why carotid stenting forced the conversation
7:25 Skill over specialty: ending the turf war
9:44 The FDA precedent: simulation becomes part of credentialing
12:03 Why procedure volume is a crude proxy for competence
14:17 Why the argument had to appear in JAMA
16:22 A homogeneous skill set, devices, and patient safety
20:42 The Copernicus Initiative: a paradigm shift in training
22:53 Three groups, one lesson: a simulator alone is not enough
25:44 The results: 56 per cent fewer errors and the 7.5 times finding
27:53 The trainees who did not pass, and distributed training
30:37 Pass the cognitive exam before the skills lab
32:33 Task deconstruction: 45 steps and 77 possible errors
36:04 Errors versus sentinel errors: why minor errors matter most
39:09 Why fidelity is not the point
41:59 Why a multi-site trial mattered
44:20 Where to start: begin with the metrics


Key points

  • Carotid artery stenting is high risk and crossed three specialties, so the FDA needed a way to be sure each clinician was safe to perform it. PBP simulation let credentialing rest on demonstrated skill rather than on specialty or case numbers.
  • In 2004 the FDA accepted virtual reality simulation as part of the training package for a new device. This was the first time a regulator tied device approval to a training standard.
  • Procedure volume and hours logged are weak indicators of skill. Demonstrated proficiency is a far better one.
  • In the Copernicus arthroscopy trial, traditional training performed worst, adding a simulator alone helped only slightly, and PBP plus the simulator produced the strongest and safest performance.
  • The PBP group made roughly 56 per cent fewer errors, and residents who met every benchmark were 7.5 times more likely to reach the final standard.
  • Minor errors, not only critical ones, predict poor outcomes, so trainees are taught to avoid every avoidable error.
  • To build PBP: find people who are genuinely good at the task, define and validate the metrics, choose simulations that let trainees practise the key steps, train faculty on the metrics first, and require a pass on the online didactic before anyone enters the skills lab.


Studies referenced

  • Gallagher AG, Cates CU. Approval of virtual reality training for carotid stenting: what this means for procedural-based medicine. JAMA. 2004;292(24):3024-3026. Read on JAMA Network
  • Angelo RL, Ryu RKN, Pedowitz RA, et al. A Proficiency-Based Progression Training Curriculum Coupled With a Model Simulator Results in the Acquisition of a Superior Arthroscopic Bankart Skill Set. Arthroscopy. 2015;31(10):1854-1871. Read on the Arthroscopy journal


Connect and follow

]]>

Guest: Professor Anthony G Gallagher Host: Patrick Kiely
Episode focus: two landmark studies, the 2004 JAMA carotid stenting paper and the Copernicus arthroscopy trial


Episode summary

Professor Tony Gallagher, the founder of Proficiency-Based Progression (PBP), joins Patrick Kiely to revisit two studies that changed how we think about surgical competence.

The first is the 2004 JAMA paper describing a closed-door meeting at which the US Food and Drug Administration agreed, for the first time, that simulation training should form part of how doctors are approved to perform a procedure. The second is the Copernicus trial in shoulder arthroscopy, which showed that a simulator only improves training when it is paired with validated metrics and a clear proficiency benchmark.

Together they make a simple, evidence-led case: competence should be measured by the skill a clinician can demonstrate, not by years served or cases counted.


Chapters

0:00 Introduction
0:50 Inside the 2004 closed-door FDA meeting on carotid stenting
4:43 Why carotid stenting forced the conversation
7:25 Skill over specialty: ending the turf war
9:44 The FDA precedent: simulation becomes part of credentialing
12:03 Why procedure volume is a crude proxy for competence
14:17 Why the argument had to appear in JAMA
16:22 A homogeneous skill set, devices, and patient safety
20:42 The Copernicus Initiative: a paradigm shift in training
22:53 Three groups, one lesson: a simulator alone is not enough
25:44 The results: 56 per cent fewer errors and the 7.5 times finding
27:53 The trainees who did not pass, and distributed training
30:37 Pass the cognitive exam before the skills lab
32:33 Task deconstruction: 45 steps and 77 possible errors
36:04 Errors versus sentinel errors: why minor errors matter most
39:09 Why fidelity is not the point
41:59 Why a multi-site trial mattered
44:20 Where to start: begin with the metrics


Key points

  • Carotid artery stenting is high risk and crossed three specialties, so the FDA needed a way to be sure each clinician was safe to perform it. PBP simulation let credentialing rest on demonstrated skill rather than on specialty or case numbers.
  • In 2004 the FDA accepted virtual reality simulation as part of the training package for a new device. This was the first time a regulator tied device approval to a training standard.
  • Procedure volume and hours logged are weak indicators of skill. Demonstrated proficiency is a far better one.
  • In the Copernicus arthroscopy trial, traditional training performed worst, adding a simulator alone helped only slightly, and PBP plus the simulator produced the strongest and safest performance.
  • The PBP group made roughly 56 per cent fewer errors, and residents who met every benchmark were 7.5 times more likely to reach the final standard.
  • Minor errors, not only critical ones, predict poor outcomes, so trainees are taught to avoid every avoidable error.
  • To build PBP: find people who are genuinely good at the task, define and validate the metrics, choose simulations that let trainees practise the key steps, train faculty on the metrics first, and require a pass on the online didactic before anyone enters the skills lab.


Studies referenced

  • Gallagher AG, Cates CU. Approval of virtual reality training for carotid stenting: what this means for procedural-based medicine. JAMA. 2004;292(24):3024-3026. Read on JAMA Network
  • Angelo RL, Ryu RKN, Pedowitz RA, et al. A Proficiency-Based Progression Training Curriculum Coupled With a Model Simulator Results in the Acquisition of a Superior Arthroscopic Bankart Skill Set. Arthroscopy. 2015;31(10):1854-1871. Read on the Arthroscopy journal


Connect and follow

]]> Tue, 26 May 2026 00:21:41 +0100 Anthony G. Gallagher, Flux Learning Ltd Anthony G. Gallagher, Flux Learning Ltd 2782

Guest: Professor Anthony G Gallagher Host: Patrick Kiely
Episode focus: two landmark studies, the 2004 JAMA carotid stenting paper and the Copernicus arthroscopy trial


Episode summary

Professor Tony Gallagher, the founder of Proficiency-Based Progression (PBP), joins Patrick Kiely to revisit two studies that changed how we think about surgical competence.

The first is the 2004 JAMA paper describing a closed-door meeting at which the US Food and Drug Administration agreed, for the first time, that simulation training should form part of how doctors are approved to perform a procedure. The second is the Copernicus trial in shoulder arthroscopy, which showed that a simulator only improves training when it is paired with validated metrics and a clear proficiency benchmark.

Together they make a simple, evidence-led case: competence should be measured by the skill a clinician can demonstrate, not by years served or cases counted.


Chapters

0:00 Introduction
0:50 Inside the 2004 closed-door FDA meeting on carotid stenting
4:43 Why carotid stenting forced the conversation
7:25 Skill over specialty: ending the turf war
9:44 The FDA precedent: simulation becomes part of credentialing
12:03 Why procedure volume is a crude proxy for competence
14:17 Why the argument had to appear in JAMA
16:22 A homogeneous skill set, devices, and patient safety
20:42 The Copernicus Initiative: a paradigm shift in training
22:53 Three groups, one lesson: a simulator alone is not enough
25:44 The results: 56 per cent fewer errors and the 7.5 times finding
27:53 The trainees who did not pass, and distributed training
30:37 Pass the cognitive exam before the skills lab
32:33 Task deconstruction: 45 steps and 77 possible errors
36:04 Errors versus sentinel errors: why minor errors matter most
39:09 Why fidelity is not the point
41:59 Why a multi-site trial mattered
44:20 Where to start: begin with the metrics


Key points

  • Carotid artery stenting is high risk and crossed three specialties, so the FDA needed a way to be sure each clinician was safe to perform it. PBP simulation let credentialing rest on demonstrated skill rather than on specialty or case numbers.
  • In 2004 the FDA accepted virtual reality simulation as part of the training package for a new device. This was the first time a regulator tied device approval to a training standard.
  • Procedure volume and hours logged are weak indicators of skill. Demonstrated proficiency is a far better one.
  • In the Copernicus arthroscopy trial, traditional training performed worst, adding a simulator alone helped only slightly, and PBP plus the simulator produced the strongest and safest performance.
  • The PBP group made roughly 56 per cent fewer errors, and residents who met every benchmark were 7.5 times more likely to reach the final standard.
  • Minor errors, not only critical ones, predict poor outcomes, so trainees are taught to avoid every avoidable error.
  • To build PBP: find people who are genuinely good at the task, define and validate the metrics, choose simulations that let trainees practise the key steps, train faculty on the metrics first, and require a pass on the online didactic before anyone enters the skills lab.


Studies referenced

  • Gallagher AG, Cates CU. Approval of virtual reality training for carotid stenting: what this means for procedural-based medicine. JAMA. 2004;292(24):3024-3026. Read on JAMA Network
  • Angelo RL, Ryu RKN, Pedowitz RA, et al. A Proficiency-Based Progression Training Curriculum Coupled With a Model Simulator Results in the Acquisition of a Superior Arthroscopic Bankart Skill Set. Arthroscopy. 2015;31(10):1854-1871. Read on the Arthroscopy journal


Connect and follow

]]> training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices No Geraldine Hennessy The VR-OR Study — Proof That Simulation Training Transfers to the Operating Room & The Methodology of Proficiency-Based Progression 2 2 The VR-OR Study — Proof That Simulation Training Transfers to the Operating Room & The Methodology of Proficiency-Based Progression full 1b95422d-793c-41bd-9903-31eeb05a3d49 https://share.transistor.fm/s/3c24c4ab Guest: Professor Anthony G Gallagher
Topic: The VR-OR Study — Proof That Simulation Training Transfers to the Operating Room & The Methodology of Proficiency-Based Progression

Episode Summary

In this episode, Patrick Kiely sits down with Professor Tony Gallagher to examine two landmark papers that transformed simulation-based surgical training. The first — the 2002 Yale VR-OR study — provided the first prospective randomised blinded proof that virtual reality simulator training transfers directly to improved operating room performance. The second — a 2005 Annals of Surgery paper — provided the field with the recipe for how to actually implement it. Together, they form the scientific and methodological backbone of Proficiency-Based Progression. Tony explains why the design decisions that made these studies credible — blinding, objective metrics, proficiency benchmarks, construct validity — are the same decisions most training programs still fail to make today.

Key Topics Covered

1. The Problem VR Training Was Designed to Solve — 0:00

  • The apprenticeship model and why laparoscopic surgery broke it
  • The fundamental cognitive challenge of moving from direct vision to a monitor
  • The fulcrum effect: why instrument manipulation on a monitor creates a proprioceptive conflict the brain must automate
  • Rick Satava's proposal: acquire basic skills outside the OR, on simulators

2. The Simulator That Changed Things — 3:21

  • Johnson & Johnson's Ethicon simulator: an emulator, not a physics-based model
  • Why abstract psychomotor tasks work better than tissue simulation
  • The surgical community's scepticism — and why Yale provided the opportunity to test it properly

3. The Proficiency Benchmark: How It Was Set — 4:51

  • Rejecting time and trial number as training endpoints
  • Using objectively assessed performance of experienced (not world-class) surgeons as the benchmark
  • Mean vs. median performance, and how to handle outlier experts (>2 SD from mean are excluded)
  • Frank Lewis (American Board of Surgery) on why the benchmark is deliberately high — and why that's fine

4. The Results: What Happened in the OR — 6:57

  • VR-trained residents: six times fewer errors in the OR
  • Control group: nine times more likely to fail to progress during a procedure

5. Failure to Progress: What It Reveals — 7:23

  • Defining the metric: instruments moving but the procedure not advancing
  • Why it indicates the person was not ready to perform the task independently
  • How it predicted the need for online didactic preparation before the skills lab

6. Why the Study Had to Be Prospective, Randomised, and Blinded — 13:11

  • The gold standard language clinicians understand
  • Why senior figures in surgery said it wasn't doable — and why they were wrong
  • How double-blinding protected the integrity of intraoperative assessment
  • The study design that subsequently became the default methodology for evaluating simulation tools in medicine

7. Objective Metrics vs. Likert Scales — 15:22

  • Why Likert scales fail for technical skill assessment
  • Inter-rater reliability below .8 invalidates any assessment tool by default
  • The subjectivity problem: two surgeons from the same year, same school, scoring the same video differently
  • Why errors are the most sensitive measure of change as a result of training
  • Steps vs. errors: trainees learn what to do; what they don't learn systematically is what not to do

8. The 2005 Annals Paper: The Recipe for PBP — 27:33

  • Why the VR-OR paper alone wasn't enough — Randy Halleck: "You assume we know how to use the methodology"
  • What the 2005 paper added: how to develop metrics, who to involve, how to set the benchmark, how to validate
  • The core principles of PBP that remain unchanged today

Publication: Gallagher, A.G. & Seymour, N.E. (2002). Virtual reality training for laparoscopic surgery. Annals of Surgery, October 2002.
https://journals.lww.com/annalsofsurgery/abstract/2002/10000/virtual_reality_training_improves_operating_room.8.aspx

9. Education vs. Training: Why the Distinction Matters — 29:05

  • Education = knowledge transmission; Training = skill acquisition
  • Why medicine has done excellent education for centuries but apprenticeship-based training no longer fits the 21st century
  • The online didactic benchmark: trainees don't enter the skills lab until they've demonstrated knowledge to the level of experienced practitioners
  • What this saves in skills lab time — and what it tells supervisors about where to direct help

10. The Pre-Trained Novice and Attentional Capacity — 31:31

  • Chunking: how the brain compresses discrete information units into automated sequences
  • Why unautomated technical skills consume attentional capacity that should be available for situational awareness
  • The bicycle analogy: looking at the handlebars vs. seeing the pothole
  • Why automation must occur outside the OR — stress in the operating room compounds cognitive load

11. Case Volume as a Surrogate for Skill — 37:04

  • Why procedure numbers are a weak and noisy predictor of surgical competence
  • The Berkmar study: intraoperative performance, not experience, predicts patient outcomes
  • Building wisdom vs. accumulating numbers
  • Why you'd use procedure numbers when you can actually measure skill

12. Simulators Can Teach Bad Behaviour — 40:44

  • Buying the wrong simulator is a fundamental and common mistake
  • Simulators are built by engineers, not clinicians — metrics must precede procurement
  • Two concrete examples: fluoroscopy pedal use with no consequence; syringe plunger speed in mechanical thrombectomy training teaching dangerous injection technique
  • How insisting on metric-aligned design led a simulation company to patent an improved device

13. Why the Benchmark Should Not Be Set on the Top 1% — 45:20

  • Setting on top 1% means almost no trainee reaches it
  • The top 1% may not always be who you think — statistical identification of outliers
  • The Monday-to-Friday surgeon doing a first-class job is the right model
  • Trainees can develop beyond the benchmark; the goal is safe, competent, timely performance

14. Why Time Alone Is a Dangerous Metric — 47:10

  • Historical roots of speed as a surgical measure: pre-anaesthesia amputation
  • Speed-accuracy trade-off: faster = more errors
  • The stroke thrombectomy example: speed matters in triage, but a fast operator who lacerates a vessel causes a worse outcome
  • Training for skill automation produces speed as a downstream consequence — not the other way around

15. Where the 2005 Prediction Has Landed — 49:34

  • PBP applied across: laparoscopic surgery, robotic surgery, endovascular procedures, cardiology, radiology, anaesthetics, intensive care, communication skills
  • ~60% improvement in performance outcomes consistently across domains
  • The PLOS ONE ut...
]]> Guest: Professor Anthony G Gallagher
Topic: The VR-OR Study — Proof That Simulation Training Transfers to the Operating Room & The Methodology of Proficiency-Based Progression

Episode Summary

In this episode, Patrick Kiely sits down with Professor Tony Gallagher to examine two landmark papers that transformed simulation-based surgical training. The first — the 2002 Yale VR-OR study — provided the first prospective randomised blinded proof that virtual reality simulator training transfers directly to improved operating room performance. The second — a 2005 Annals of Surgery paper — provided the field with the recipe for how to actually implement it. Together, they form the scientific and methodological backbone of Proficiency-Based Progression. Tony explains why the design decisions that made these studies credible — blinding, objective metrics, proficiency benchmarks, construct validity — are the same decisions most training programs still fail to make today.

Key Topics Covered

1. The Problem VR Training Was Designed to Solve — 0:00

  • The apprenticeship model and why laparoscopic surgery broke it
  • The fundamental cognitive challenge of moving from direct vision to a monitor
  • The fulcrum effect: why instrument manipulation on a monitor creates a proprioceptive conflict the brain must automate
  • Rick Satava's proposal: acquire basic skills outside the OR, on simulators

2. The Simulator That Changed Things — 3:21

  • Johnson & Johnson's Ethicon simulator: an emulator, not a physics-based model
  • Why abstract psychomotor tasks work better than tissue simulation
  • The surgical community's scepticism — and why Yale provided the opportunity to test it properly

3. The Proficiency Benchmark: How It Was Set — 4:51

  • Rejecting time and trial number as training endpoints
  • Using objectively assessed performance of experienced (not world-class) surgeons as the benchmark
  • Mean vs. median performance, and how to handle outlier experts (>2 SD from mean are excluded)
  • Frank Lewis (American Board of Surgery) on why the benchmark is deliberately high — and why that's fine

4. The Results: What Happened in the OR — 6:57

  • VR-trained residents: six times fewer errors in the OR
  • Control group: nine times more likely to fail to progress during a procedure

5. Failure to Progress: What It Reveals — 7:23

  • Defining the metric: instruments moving but the procedure not advancing
  • Why it indicates the person was not ready to perform the task independently
  • How it predicted the need for online didactic preparation before the skills lab

6. Why the Study Had to Be Prospective, Randomised, and Blinded — 13:11

  • The gold standard language clinicians understand
  • Why senior figures in surgery said it wasn't doable — and why they were wrong
  • How double-blinding protected the integrity of intraoperative assessment
  • The study design that subsequently became the default methodology for evaluating simulation tools in medicine

7. Objective Metrics vs. Likert Scales — 15:22

  • Why Likert scales fail for technical skill assessment
  • Inter-rater reliability below .8 invalidates any assessment tool by default
  • The subjectivity problem: two surgeons from the same year, same school, scoring the same video differently
  • Why errors are the most sensitive measure of change as a result of training
  • Steps vs. errors: trainees learn what to do; what they don't learn systematically is what not to do

8. The 2005 Annals Paper: The Recipe for PBP — 27:33

  • Why the VR-OR paper alone wasn't enough — Randy Halleck: "You assume we know how to use the methodology"
  • What the 2005 paper added: how to develop metrics, who to involve, how to set the benchmark, how to validate
  • The core principles of PBP that remain unchanged today

Publication: Gallagher, A.G. & Seymour, N.E. (2002). Virtual reality training for laparoscopic surgery. Annals of Surgery, October 2002.
https://journals.lww.com/annalsofsurgery/abstract/2002/10000/virtual_reality_training_improves_operating_room.8.aspx

9. Education vs. Training: Why the Distinction Matters — 29:05

  • Education = knowledge transmission; Training = skill acquisition
  • Why medicine has done excellent education for centuries but apprenticeship-based training no longer fits the 21st century
  • The online didactic benchmark: trainees don't enter the skills lab until they've demonstrated knowledge to the level of experienced practitioners
  • What this saves in skills lab time — and what it tells supervisors about where to direct help

10. The Pre-Trained Novice and Attentional Capacity — 31:31

  • Chunking: how the brain compresses discrete information units into automated sequences
  • Why unautomated technical skills consume attentional capacity that should be available for situational awareness
  • The bicycle analogy: looking at the handlebars vs. seeing the pothole
  • Why automation must occur outside the OR — stress in the operating room compounds cognitive load

11. Case Volume as a Surrogate for Skill — 37:04

  • Why procedure numbers are a weak and noisy predictor of surgical competence
  • The Berkmar study: intraoperative performance, not experience, predicts patient outcomes
  • Building wisdom vs. accumulating numbers
  • Why you'd use procedure numbers when you can actually measure skill

12. Simulators Can Teach Bad Behaviour — 40:44

  • Buying the wrong simulator is a fundamental and common mistake
  • Simulators are built by engineers, not clinicians — metrics must precede procurement
  • Two concrete examples: fluoroscopy pedal use with no consequence; syringe plunger speed in mechanical thrombectomy training teaching dangerous injection technique
  • How insisting on metric-aligned design led a simulation company to patent an improved device

13. Why the Benchmark Should Not Be Set on the Top 1% — 45:20

  • Setting on top 1% means almost no trainee reaches it
  • The top 1% may not always be who you think — statistical identification of outliers
  • The Monday-to-Friday surgeon doing a first-class job is the right model
  • Trainees can develop beyond the benchmark; the goal is safe, competent, timely performance

14. Why Time Alone Is a Dangerous Metric — 47:10

  • Historical roots of speed as a surgical measure: pre-anaesthesia amputation
  • Speed-accuracy trade-off: faster = more errors
  • The stroke thrombectomy example: speed matters in triage, but a fast operator who lacerates a vessel causes a worse outcome
  • Training for skill automation produces speed as a downstream consequence — not the other way around

15. Where the 2005 Prediction Has Landed — 49:34

  • PBP applied across: laparoscopic surgery, robotic surgery, endovascular procedures, cardiology, radiology, anaesthetics, intensive care, communication skills
  • ~60% improvement in performance outcomes consistently across domains
  • The PLOS ONE ut...
]]> Mon, 25 May 2026 17:46:21 +0100 Anthony G. Gallagher, Flux Learning Ltd Anthony G. Gallagher, Flux Learning Ltd 3210 Guest: Professor Anthony G Gallagher
Topic: The VR-OR Study — Proof That Simulation Training Transfers to the Operating Room & The Methodology of Proficiency-Based Progression

Episode Summary

In this episode, Patrick Kiely sits down with Professor Tony Gallagher to examine two landmark papers that transformed simulation-based surgical training. The first — the 2002 Yale VR-OR study — provided the first prospective randomised blinded proof that virtual reality simulator training transfers directly to improved operating room performance. The second — a 2005 Annals of Surgery paper — provided the field with the recipe for how to actually implement it. Together, they form the scientific and methodological backbone of Proficiency-Based Progression. Tony explains why the design decisions that made these studies credible — blinding, objective metrics, proficiency benchmarks, construct validity — are the same decisions most training programs still fail to make today.

Key Topics Covered

1. The Problem VR Training Was Designed to Solve — 0:00

  • The apprenticeship model and why laparoscopic surgery broke it
  • The fundamental cognitive challenge of moving from direct vision to a monitor
  • The fulcrum effect: why instrument manipulation on a monitor creates a proprioceptive conflict the brain must automate
  • Rick Satava's proposal: acquire basic skills outside the OR, on simulators

2. The Simulator That Changed Things — 3:21

  • Johnson & Johnson's Ethicon simulator: an emulator, not a physics-based model
  • Why abstract psychomotor tasks work better than tissue simulation
  • The surgical community's scepticism — and why Yale provided the opportunity to test it properly

3. The Proficiency Benchmark: How It Was Set — 4:51

  • Rejecting time and trial number as training endpoints
  • Using objectively assessed performance of experienced (not world-class) surgeons as the benchmark
  • Mean vs. median performance, and how to handle outlier experts (>2 SD from mean are excluded)
  • Frank Lewis (American Board of Surgery) on why the benchmark is deliberately high — and why that's fine

4. The Results: What Happened in the OR — 6:57

  • VR-trained residents: six times fewer errors in the OR
  • Control group: nine times more likely to fail to progress during a procedure

5. Failure to Progress: What It Reveals — 7:23

  • Defining the metric: instruments moving but the procedure not advancing
  • Why it indicates the person was not ready to perform the task independently
  • How it predicted the need for online didactic preparation before the skills lab

6. Why the Study Had to Be Prospective, Randomised, and Blinded — 13:11

  • The gold standard language clinicians understand
  • Why senior figures in surgery said it wasn't doable — and why they were wrong
  • How double-blinding protected the integrity of intraoperative assessment
  • The study design that subsequently became the default methodology for evaluating simulation tools in medicine

7. Objective Metrics vs. Likert Scales — 15:22

  • Why Likert scales fail for technical skill assessment
  • Inter-rater reliability below .8 invalidates any assessment tool by default
  • The subjectivity problem: two surgeons from the same year, same school, scoring the same video differently
  • Why errors are the most sensitive measure of change as a result of training
  • Steps vs. errors: trainees learn what to do; what they don't learn systematically is what not to do

8. The 2005 Annals Paper: The Recipe for PBP — 27:33

  • Why the VR-OR paper alone wasn't enough — Randy Halleck: "You assume we know how to use the methodology"
  • What the 2005 paper added: how to develop metrics, who to involve, how to set the benchmark, how to validate
  • The core principles of PBP that remain unchanged today

Publication: Gallagher, A.G. & Seymour, N.E. (2002). Virtual reality training for laparoscopic surgery. Annals of Surgery, October 2002.
https://journals.lww.com/annalsofsurgery/abstract/2002/10000/virtual_reality_training_improves_operating_room.8.aspx

9. Education vs. Training: Why the Distinction Matters — 29:05

  • Education = knowledge transmission; Training = skill acquisition
  • Why medicine has done excellent education for centuries but apprenticeship-based training no longer fits the 21st century
  • The online didactic benchmark: trainees don't enter the skills lab until they've demonstrated knowledge to the level of experienced practitioners
  • What this saves in skills lab time — and what it tells supervisors about where to direct help

10. The Pre-Trained Novice and Attentional Capacity — 31:31

  • Chunking: how the brain compresses discrete information units into automated sequences
  • Why unautomated technical skills consume attentional capacity that should be available for situational awareness
  • The bicycle analogy: looking at the handlebars vs. seeing the pothole
  • Why automation must occur outside the OR — stress in the operating room compounds cognitive load

11. Case Volume as a Surrogate for Skill — 37:04

  • Why procedure numbers are a weak and noisy predictor of surgical competence
  • The Berkmar study: intraoperative performance, not experience, predicts patient outcomes
  • Building wisdom vs. accumulating numbers
  • Why you'd use procedure numbers when you can actually measure skill

12. Simulators Can Teach Bad Behaviour — 40:44

  • Buying the wrong simulator is a fundamental and common mistake
  • Simulators are built by engineers, not clinicians — metrics must precede procurement
  • Two concrete examples: fluoroscopy pedal use with no consequence; syringe plunger speed in mechanical thrombectomy training teaching dangerous injection technique
  • How insisting on metric-aligned design led a simulation company to patent an improved device

13. Why the Benchmark Should Not Be Set on the Top 1% — 45:20

  • Setting on top 1% means almost no trainee reaches it
  • The top 1% may not always be who you think — statistical identification of outliers
  • The Monday-to-Friday surgeon doing a first-class job is the right model
  • Trainees can develop beyond the benchmark; the goal is safe, competent, timely performance

14. Why Time Alone Is a Dangerous Metric — 47:10

  • Historical roots of speed as a surgical measure: pre-anaesthesia amputation
  • Speed-accuracy trade-off: faster = more errors
  • The stroke thrombectomy example: speed matters in triage, but a fast operator who lacerates a vessel causes a worse outcome
  • Training for skill automation produces speed as a downstream consequence — not the other way around

15. Where the 2005 Prediction Has Landed — 49:34

  • PBP applied across: laparoscopic surgery, robotic surgery, endovascular procedures, cardiology, radiology, anaesthetics, intensive care, communication skills
  • ~60% improvement in performance outcomes consistently across domains
  • The PLOS ONE ut...
]]> training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices No Geraldine Hennessy The Crisis in Surgical Training & Proficiency-Based Progression (PBP) 1 1 The Crisis in Surgical Training & Proficiency-Based Progression (PBP) full 853bfde5-1c20-4af2-90b3-5015099c7f40 https://share.transistor.fm/s/47916f2b Guest: Professor Anthony G Gallagher
Topic: The Crisis in Surgical Training & Proficiency-Based Progression (PBP)

Episode Summary

In this inaugural episode, Patrick Kiely sits down with Professor Tony Gallagher — founder of Proficiency-Based Progression and one of the world's leading researchers in surgical skills assessment and simulation-based training — to examine a deeply uncomfortable truth: that professional credentialing in medicine tells us almost nothing about actual clinical competence. Tony shares 30 years of evidence challenging the assumptions underpinning surgical and procedural training worldwide, and makes the case for Proficiency-Based Progression (PBP) as the superior — and inevitable — alternative.

Key Topics Covered

1. The Competence Problem in Surgery — 0:00

  • Why credentials don't equal competence
  • The Halsted training paradigm — developed in the late 19th/early 20th century — and why it's still in use
  • How to actually find out if your surgeon is good (hint: ask the theater sister)

2. Why Current Training Metrics Are Failing — 5:08

  • Procedure volume and hours logged as proxies for competence — and why they're wrong
  • The misuse of Likert-type scales in surgical assessment
  • Reduced work hours legislation (Europe/US) and its impact on trainee experience
  • The Libby Zion case (New York) and how it changed US residency hours

3. The Yale Study That Changed Everything — 9:53

  • The landmark 2002 Yale study showing simulator-trained residents made 60% fewer errors
  • Why it became a citation classic — and why change was still slow

  • Publication:  Gallagher & Seymour (2002). Virtual reality training for laparoscopic surgery. Annals of Surgery, October 2002. (Presented at American Surgical Association, April 2002) https://journals.lww.com/annalsofsurgery/abstract/2002/10000/virtual_reality_training_improves_operating_room.8.aspx


4. The American College of Surgeons Response — 12:29

  • Gerry Healy's pivotal leadership shift at the Boston meeting
  • The establishment of Accredited Educational Institutes (2006)
  • Why 100+ accredited simulation centers still aren't producing the training outcomes expected

5. The Experience ≠ Competence Myth — 16:47

  • Why procedure volume is a noisy surrogate for surgical skill
  • How some practicing consultants perform worse than residents in training
  • Objective intraoperative performance assessment as the gold standard

6. Proficiency-Based Progression: How It Works — 20:30

  • The mechanics of PBP: phases, steps, errors, critical errors, and the benchmark
  • Establishing benchmarks from experienced — not world-class — practitioners
  • Construct validity, inter-rater reliability, and why Likert scales fail
  • The role of deliberate practice (Ericsson) and why explicit, formative feedback accelerates learning

  • Publication: Mazzone, Elio; Puliatti, Stefano MD; Amato, Marco; Bunting, Brendan; Rocco, Bernardo; Montorsi, Francesco; Mottrie, Alexandre; Gallagher, Anthony G. PhD, DSc||. A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes. Annals of Surgery 274(2):p 281-289, August 2021. | DOI: 10.1097/SLA.0000000000004650

7. Why PBP Hasn't Been Adopted Universally — 35:27

  • "It's a failure of leadership"
  • Organisations that have adopted PBP: AANA, ERUS, ORSI Academy
  • Incentive structures in healthcare and medical device manufacturing that slow adoption
  • The Center for Medicare Services complication-rate accountability model as a potential lever

8. The Economics of PBP — 37:49


Publication: Puliatti, S., Rodriguez Peñaranda, N., Amato, M., De Groote, R., Farinha, R., Bunting, B., van Cleynenbreugel, B., Mottrie, A. and Gallagher, A.G. (2026), Randomised trial on the economic impact of proficiency-based progression vs conventional robotic surgical training. BJU Int, 137: 493-501. https://doi.org/10.1111/bju.70130 https://bjui-journals.onlinelibrary.wiley.com/doi/full/10.1111/bju.70130 — Cost-effectiveness analysis of PBP vs. conventional training. At 500 trainees/year: PBP ~€1.7M vs. conventional ~€3.5M; cost equivalence at just 25 trainees; 100% of PBP trainees reached proficiency vs. 58% conventional


9. Surgeon Skill Predicts Patient Outcomes — 40:24

  • PBP applied to communication skills: deteriorating patient handover study, Cork University Hospital
  • PBP applied to epidural training: 50%+ reduction in epidural failure rates for non-PBP trained group

10. PBP Beyond Medicine: The Utilities Sector — 45:42

  • Reach Active case study: PBP training for utility workers to safely identify and excavate buried cables
  • Over €1 million saved in avoided utility strikes in year one
  • Same methodology, same results — across a non-university workforce

11. How to Implement PBP in Your Organisation — 48:25

  • Start by identifying individuals who are objectively good at the task
  • Work out the metrics: phases, steps, errors, critical errors
  • Validate — consensus, not just agreement
  • Build or select simulation tools around the validated metrics
  • Train faculty on the metrics first
  • Require trainees to pass the online didactic benchmark before entering the skills lab
  • Train to benchmark — not to time, not to hours

12. AI, Robotics, and the Future of Training — 1:01:31

  • Why AI currently measures process, not performance
  • What AI will need to assess: granular, step-level surgical actions
  • The endovascular sphere as the likely first domain for AI integration
  • Why simulation and AI must be seen as tools, not solutions
  • The inevitability of PBP adoption — the question is only when


Connect & Follow

Timestamps

TopicTimeThe competence problem: credentials vs. skill | 0:00
Why current training metrics are wrong | 5:08
The 2002 Yale simulator study | 9:53
ACS response and simulation center rollout | 12:29
Experience ≠ competence | 16:47
Proficiency-based progression — mechanics | 20:30
Why PBP hasn't been widely adopted | 35:27
The economics of PBP | 37:49
Surgeon skill predicts patient outcomes (NEJM) | 40:24
PBP in utilities / Reach Active case study | 45:42
How to implement PBP in your organisation | 48:25
AI, robotics, and the future of training | 1:01:31

]]> Guest: Professor Anthony G Gallagher
Topic: The Crisis in Surgical Training & Proficiency-Based Progression (PBP)

Episode Summary

In this inaugural episode, Patrick Kiely sits down with Professor Tony Gallagher — founder of Proficiency-Based Progression and one of the world's leading researchers in surgical skills assessment and simulation-based training — to examine a deeply uncomfortable truth: that professional credentialing in medicine tells us almost nothing about actual clinical competence. Tony shares 30 years of evidence challenging the assumptions underpinning surgical and procedural training worldwide, and makes the case for Proficiency-Based Progression (PBP) as the superior — and inevitable — alternative.

Key Topics Covered

1. The Competence Problem in Surgery — 0:00

  • Why credentials don't equal competence
  • The Halsted training paradigm — developed in the late 19th/early 20th century — and why it's still in use
  • How to actually find out if your surgeon is good (hint: ask the theater sister)

2. Why Current Training Metrics Are Failing — 5:08

  • Procedure volume and hours logged as proxies for competence — and why they're wrong
  • The misuse of Likert-type scales in surgical assessment
  • Reduced work hours legislation (Europe/US) and its impact on trainee experience
  • The Libby Zion case (New York) and how it changed US residency hours

3. The Yale Study That Changed Everything — 9:53

  • The landmark 2002 Yale study showing simulator-trained residents made 60% fewer errors
  • Why it became a citation classic — and why change was still slow

  • Publication:  Gallagher & Seymour (2002). Virtual reality training for laparoscopic surgery. Annals of Surgery, October 2002. (Presented at American Surgical Association, April 2002) https://journals.lww.com/annalsofsurgery/abstract/2002/10000/virtual_reality_training_improves_operating_room.8.aspx


4. The American College of Surgeons Response — 12:29

  • Gerry Healy's pivotal leadership shift at the Boston meeting
  • The establishment of Accredited Educational Institutes (2006)
  • Why 100+ accredited simulation centers still aren't producing the training outcomes expected

5. The Experience ≠ Competence Myth — 16:47

  • Why procedure volume is a noisy surrogate for surgical skill
  • How some practicing consultants perform worse than residents in training
  • Objective intraoperative performance assessment as the gold standard

6. Proficiency-Based Progression: How It Works — 20:30

  • The mechanics of PBP: phases, steps, errors, critical errors, and the benchmark
  • Establishing benchmarks from experienced — not world-class — practitioners
  • Construct validity, inter-rater reliability, and why Likert scales fail
  • The role of deliberate practice (Ericsson) and why explicit, formative feedback accelerates learning

  • Publication: Mazzone, Elio; Puliatti, Stefano MD; Amato, Marco; Bunting, Brendan; Rocco, Bernardo; Montorsi, Francesco; Mottrie, Alexandre; Gallagher, Anthony G. PhD, DSc||. A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes. Annals of Surgery 274(2):p 281-289, August 2021. | DOI: 10.1097/SLA.0000000000004650

7. Why PBP Hasn't Been Adopted Universally — 35:27

  • "It's a failure of leadership"
  • Organisations that have adopted PBP: AANA, ERUS, ORSI Academy
  • Incentive structures in healthcare and medical device manufacturing that slow adoption
  • The Center for Medicare Services complication-rate accountability model as a potential lever

8. The Economics of PBP — 37:49


Publication: Puliatti, S., Rodriguez Peñaranda, N., Amato, M., De Groote, R., Farinha, R., Bunting, B., van Cleynenbreugel, B., Mottrie, A. and Gallagher, A.G. (2026), Randomised trial on the economic impact of proficiency-based progression vs conventional robotic surgical training. BJU Int, 137: 493-501. https://doi.org/10.1111/bju.70130 https://bjui-journals.onlinelibrary.wiley.com/doi/full/10.1111/bju.70130 — Cost-effectiveness analysis of PBP vs. conventional training. At 500 trainees/year: PBP ~€1.7M vs. conventional ~€3.5M; cost equivalence at just 25 trainees; 100% of PBP trainees reached proficiency vs. 58% conventional


9. Surgeon Skill Predicts Patient Outcomes — 40:24

  • PBP applied to communication skills: deteriorating patient handover study, Cork University Hospital
  • PBP applied to epidural training: 50%+ reduction in epidural failure rates for non-PBP trained group

10. PBP Beyond Medicine: The Utilities Sector — 45:42

  • Reach Active case study: PBP training for utility workers to safely identify and excavate buried cables
  • Over €1 million saved in avoided utility strikes in year one
  • Same methodology, same results — across a non-university workforce

11. How to Implement PBP in Your Organisation — 48:25

  • Start by identifying individuals who are objectively good at the task
  • Work out the metrics: phases, steps, errors, critical errors
  • Validate — consensus, not just agreement
  • Build or select simulation tools around the validated metrics
  • Train faculty on the metrics first
  • Require trainees to pass the online didactic benchmark before entering the skills lab
  • Train to benchmark — not to time, not to hours

12. AI, Robotics, and the Future of Training — 1:01:31

  • Why AI currently measures process, not performance
  • What AI will need to assess: granular, step-level surgical actions
  • The endovascular sphere as the likely first domain for AI integration
  • Why simulation and AI must be seen as tools, not solutions
  • The inevitability of PBP adoption — the question is only when


Connect & Follow

Timestamps

TopicTimeThe competence problem: credentials vs. skill | 0:00
Why current training metrics are wrong | 5:08
The 2002 Yale simulator study | 9:53
ACS response and simulation center rollout | 12:29
Experience ≠ competence | 16:47
Proficiency-based progression — mechanics | 20:30
Why PBP hasn't been widely adopted | 35:27
The economics of PBP | 37:49
Surgeon skill predicts patient outcomes (NEJM) | 40:24
PBP in utilities / Reach Active case study | 45:42
How to implement PBP in your organisation | 48:25
AI, robotics, and the future of training | 1:01:31

]]> Sat, 23 May 2026 09:00:00 +0100 Anthony G. Gallagher, Flux Learning Ltd Anthony G. Gallagher, Flux Learning Ltd 3876 Guest: Professor Anthony G Gallagher
Topic: The Crisis in Surgical Training & Proficiency-Based Progression (PBP)

Episode Summary

In this inaugural episode, Patrick Kiely sits down with Professor Tony Gallagher — founder of Proficiency-Based Progression and one of the world's leading researchers in surgical skills assessment and simulation-based training — to examine a deeply uncomfortable truth: that professional credentialing in medicine tells us almost nothing about actual clinical competence. Tony shares 30 years of evidence challenging the assumptions underpinning surgical and procedural training worldwide, and makes the case for Proficiency-Based Progression (PBP) as the superior — and inevitable — alternative.

Key Topics Covered

1. The Competence Problem in Surgery — 0:00

  • Why credentials don't equal competence
  • The Halsted training paradigm — developed in the late 19th/early 20th century — and why it's still in use
  • How to actually find out if your surgeon is good (hint: ask the theater sister)

2. Why Current Training Metrics Are Failing — 5:08

  • Procedure volume and hours logged as proxies for competence — and why they're wrong
  • The misuse of Likert-type scales in surgical assessment
  • Reduced work hours legislation (Europe/US) and its impact on trainee experience
  • The Libby Zion case (New York) and how it changed US residency hours

3. The Yale Study That Changed Everything — 9:53

  • The landmark 2002 Yale study showing simulator-trained residents made 60% fewer errors
  • Why it became a citation classic — and why change was still slow

  • Publication:  Gallagher & Seymour (2002). Virtual reality training for laparoscopic surgery. Annals of Surgery, October 2002. (Presented at American Surgical Association, April 2002) https://journals.lww.com/annalsofsurgery/abstract/2002/10000/virtual_reality_training_improves_operating_room.8.aspx


4. The American College of Surgeons Response — 12:29

  • Gerry Healy's pivotal leadership shift at the Boston meeting
  • The establishment of Accredited Educational Institutes (2006)
  • Why 100+ accredited simulation centers still aren't producing the training outcomes expected

5. The Experience ≠ Competence Myth — 16:47

  • Why procedure volume is a noisy surrogate for surgical skill
  • How some practicing consultants perform worse than residents in training
  • Objective intraoperative performance assessment as the gold standard

6. Proficiency-Based Progression: How It Works — 20:30

  • The mechanics of PBP: phases, steps, errors, critical errors, and the benchmark
  • Establishing benchmarks from experienced — not world-class — practitioners
  • Construct validity, inter-rater reliability, and why Likert scales fail
  • The role of deliberate practice (Ericsson) and why explicit, formative feedback accelerates learning

  • Publication: Mazzone, Elio; Puliatti, Stefano MD; Amato, Marco; Bunting, Brendan; Rocco, Bernardo; Montorsi, Francesco; Mottrie, Alexandre; Gallagher, Anthony G. PhD, DSc||. A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes. Annals of Surgery 274(2):p 281-289, August 2021. | DOI: 10.1097/SLA.0000000000004650

7. Why PBP Hasn't Been Adopted Universally — 35:27

  • "It's a failure of leadership"
  • Organisations that have adopted PBP: AANA, ERUS, ORSI Academy
  • Incentive structures in healthcare and medical device manufacturing that slow adoption
  • The Center for Medicare Services complication-rate accountability model as a potential lever

8. The Economics of PBP — 37:49


Publication: Puliatti, S., Rodriguez Peñaranda, N., Amato, M., De Groote, R., Farinha, R., Bunting, B., van Cleynenbreugel, B., Mottrie, A. and Gallagher, A.G. (2026), Randomised trial on the economic impact of proficiency-based progression vs conventional robotic surgical training. BJU Int, 137: 493-501. https://doi.org/10.1111/bju.70130 https://bjui-journals.onlinelibrary.wiley.com/doi/full/10.1111/bju.70130 — Cost-effectiveness analysis of PBP vs. conventional training. At 500 trainees/year: PBP ~€1.7M vs. conventional ~€3.5M; cost equivalence at just 25 trainees; 100% of PBP trainees reached proficiency vs. 58% conventional


9. Surgeon Skill Predicts Patient Outcomes — 40:24

  • PBP applied to communication skills: deteriorating patient handover study, Cork University Hospital
  • PBP applied to epidural training: 50%+ reduction in epidural failure rates for non-PBP trained group

10. PBP Beyond Medicine: The Utilities Sector — 45:42

  • Reach Active case study: PBP training for utility workers to safely identify and excavate buried cables
  • Over €1 million saved in avoided utility strikes in year one
  • Same methodology, same results — across a non-university workforce

11. How to Implement PBP in Your Organisation — 48:25

  • Start by identifying individuals who are objectively good at the task
  • Work out the metrics: phases, steps, errors, critical errors
  • Validate — consensus, not just agreement
  • Build or select simulation tools around the validated metrics
  • Train faculty on the metrics first
  • Require trainees to pass the online didactic benchmark before entering the skills lab
  • Train to benchmark — not to time, not to hours

12. AI, Robotics, and the Future of Training — 1:01:31

  • Why AI currently measures process, not performance
  • What AI will need to assess: granular, step-level surgical actions
  • The endovascular sphere as the likely first domain for AI integration
  • Why simulation and AI must be seen as tools, not solutions
  • The inevitability of PBP adoption — the question is only when


Connect & Follow

Timestamps

TopicTimeThe competence problem: credentials vs. skill | 0:00
Why current training metrics are wrong | 5:08
The 2002 Yale simulator study | 9:53
ACS response and simulation center rollout | 12:29
Experience ≠ competence | 16:47
Proficiency-based progression — mechanics | 20:30
Why PBP hasn't been widely adopted | 35:27
The economics of PBP | 37:49
Surgeon skill predicts patient outcomes (NEJM) | 40:24
PBP in utilities / Reach Active case study | 45:42
How to implement PBP in your organisation | 48:25
AI, robotics, and the future of training | 1:01:31

]]> training, metrics, evidence, data, proficiency, skills training, robotic surgery, medical devices No Geraldine Hennessy