Discussion about this post

User's avatar
Ann Somers Hogg's avatar

Dr. Wachter - Thanks for sharing this analysis. As a non-clinical researcher, I appreciated seeing this from a provider’s perspective. I’m a huge fan of yours and incredibly grateful for all of the advice you shared about your own practices to avoid COVID during the pandemic. I work for the Christensen Institute, so I was thrilled to see your reference to Clay in your article. I also recently wrote a piece on OpenEvidence’s disruptive potential, as compared to UpToDate. I wrote it before UpToDate released it’s Expert AI tool, so that’s not included. However, I’d love your thoughts: https://open.substack.com/pub/annsomershogg/p/health-cares-next-game-changer-isnt

While I propose that OE might win for a different reason than being a true Disruptive Innovation, I agree with your assessment that it may follow a similar trajectory to UpToDate displacing text books. And that’s not necessarily because it’s disruptive, but instead because it nails the Job to Be Done.

fa mo's avatar

There's no discussion given here to the possibility that foundation models will ever get good enough to do this job out of the box. I'm not so sure. They are a) getting better faster b) have a lot of incentive to develop guardrails for "compliance" heavy businesses like finance and healthcare. In Clayton Christenson terms, the risk to OE might be from "beneath" not "above"

But engaging on the question as posed-

It's extremely unlikely that UpToDate-based AI will be as good as OpenEvidence, as long as OE avoids platform decay (aka "enshitification") that is the cardinal sin of two-sided attention marketplaces 

Both models will be generally trained LLMs with post-training/ RAG/ system prompts/ guardrails/ inference on top, with one drawing most strength from human-curated summaries, the other more directly from the literature/ trials/ guidelines. 

[In other words, I'm willing to bet this is will not be true "UpToDate tool won’t scour the entire medical literature or the unfiltered internet for insights. Instead, UpToDate’s AI will draw its wisdom exclusively from its thousands of continuously updated chapters, written by experts."]

There is no way that those Paul Bunyan humans will be able to compete on scale and speed, and the most likely outcome is that what they paste into their updates will increasingly be OpenEvidence generated anyway. 

The observationally trained model (Epic corpus) will also start with general knowledge (much as medical doctors start as adult humans with bachelors degrees)- but the interesting question will be whether the experience of real world "good doctors" (not the average or "doctors like you") will be able to be harnessed for greater acceptability/ feasibility realism - especially if it deviates from the evidence base to a significant degree ("I know the CHF GDMT calls for 3 meds, but go slow in elderly patients").

IMO the real question is not whether UpToDate or Epic can leverage their *content* but whether they will be able to leverage their *distribution* to pose a challenge to OE; how expensive / difficult it will be for them to replicate the exact model-building, guardrails, RAG, prompting approach that OE has taken; and how tainted OE will become from a pharma-ad driven business model. 

15 more comments...

No posts

Ready for more?