Great article. Might be hard to get through, so I'll put out this quote which I enjoyed and compelled me to read the rest of it.
"Matthews had started a cystic-fibrosis treatment program as a young pulmonary specialist at Babies and Children’s Hospital, in Cleveland, in 1957, and within a few years was claiming to have an annual mortality rate that was less than two per cent. To anyone treating CF at the time, it was a preposterous assertion. National mortality rates for the disease were estimated to be higher than twenty per cent a year, and the average patient died by the age of three. Yet here was Matthews saying that he and his colleagues could stop the disease from doing serious harm for years. “How long [our patients] will live remains to be seen, but I expect most of them to come to my funeral,” he told one conference of physicians.
In 1964, the Cystic Fibrosis Foundation gave a University of Minnesota pediatrician named Warren Warwick a budget of ten thousand dollars to collect reports on every patient treated at the thirty-one CF centers in the United States that year—data that would test Matthews’s claim. Several months later, he had the results: the median estimated age at death for patients in Matthews’s center was twenty-one years, seven times the age of patients treated elsewhere. He had not had a single death among patients younger than six in at least five years."
"In this short speech was the core of Warwick’s world view. He believed that excellence came from seeing, on a daily basis, the difference between being 99.5-per-cent successful and being 99.95-per-cent successful."
I took notice of that passage also. But the main reason the 99.5 to 99.95 percentages is meaningful is that they were measures of a daily risk. So it quickly compounds into a meaningful percentages; the 99.95% rate meant an 83% chance patient would stay well this year, the 99.5% rate meant only a 16% chance of staying well for this year.
Many of the medical percentages that we see quoted in media are derived using a lifetime risk basis, _not_ a daily one. On a lifetime basis the difference between 99.95% and 99.5% _is_ negligible.
I wonder how many people reading this article are aware of the difference.
I have Cystic Fibrosis. I don't think it's fair to assume that the bell curve associated with treatment results is fair to attribute to the quality of care. Factors such as climate and pollution can significantly contribute to health, as can socio-economic and average distance patients travel, as well as access to insurance.
For instance, I'm from North Texas; I went to college in Oklahoma. Back then (1996) Texas had a type of insurance for children with CF (it actually extended to age 21) unavailable in Oklahoma. I ended up losing care, going off of my meds, and by the time I was back in Texas and had the income to support my needs (I came from a very poor family) I had lost 30 pounds (down to 98lbs) and had lost 25% of my lung capacity. Was this attributable to the clinic in Dallas? Of course not.
(FWIW, I'm up to 155lbs these days and my lung capacity is probably in the upper 5-10% of all CF patients)
Of course there are other effects that go into that bell curve (I'd be more worried about doctor-to-patient ratios and time per patient), but I think the article does an excellent job of highlighting the differences in care.
Very true. It's important to note that the majority of a (well maintained) CF patient's health is the result of the the breathing treatments and enzyme treatments; I could probably not see a doctor for years if I had my meds. (Probably true of many chronic conditions I'm sure, but speaking to my experience only)
I'm sure you know this but a good physician will closely track your lung function and on a dip will get a sputum test done and then get you on the right antibiotics asap. That is a very important part of CF treatment that does need regular trips to the clinic. BTW I'm glad you're doing so well :-)
I've had the fortune of being pretty consistent lung function-wise - the dips tended to be when I was doing poorly with my treatment regimen. (Of course there are many who have issues despite their best efforts)
Medicine is like any other profession - it shouldn't be surprising that some doctors are better than others, like some lawyers/engineers/teachers are better than others.
There are a lot of challenges with healthcare metrics. Medical problems are more complex, arising from more factors than practically any other problem. Most other sciences can be resolved by theory or relatively convergent empirical data. Medicine is empirical, but the scatter is huge.
Data is not tracked well. When it does exist, it is in formats that are not well conducive to analysis and sharing. This seems to be slowly (and expensively) improving. Pressure for privacy adds to inertia for innovations that could use medical data for improved outcomes.
With the title of the article, I was hoping there would be some more discussion about implications of some 'rating system' for doctors. It would probably superficially be a good idea, but such a system would obviously result in some level of gaming. Does a doctor in a rated system only take easy cases to keep a good track record? Are hard cases given more weighting? Who/ how is such a weighting decided? Such a system could offer a results-based compensation scheme for doctors, which is lacking from most publicly funded systems.
"Medicine is like any other profession - it shouldn't be surprising that some doctors are better than others"
Surely, some doctors are better than others, but
(for the sake of discussion :)) I don't think every professions or activities are equivalent with respect to 'rating'. Take the extreme example of walking. All human beings walk equally well. This is unlike playing music or solving math problems for instance.
I'm certainly not saying that medicine is as easy as walking, but maybe for a wide range of health problems, there are well known standard procedures or solutions that all doctors are competent enough to follow. And in doubt, they usually refer their patient to a specialist, who in turn can send them to a more competent doctor.
I think you got my point. Usually, you don't stop looking at people in the street thinking how great is their walking. Walking is a simple enough task so that in normal conditions you don't see a wide range of abilities. Unlike swimming, playing music or cooking where even in normal condition you can see that some people are incredibly better han others.
An other example that comes to mind are regular airline pilots: nobody wonders whether they are good or bad. We never have to worry that the pilot is unable to fly the plane.
Again, this is a terrible example. I'd say that a good third of passengers are wondering exactly that. (It's irrational to do so, but that's another story.)
And further, skilled pilots can be recognised by their landings, particularly in making soft ones in stormy conditions. Perhaps also in their customer service with witty commentary over the PA.
Divergent, but any skill I can think of will have a distribution of abilities. Some of these skills have a threshold beyond which incremental improvement gives little benefit. Walking example: if you can walk 100m in one go, you are better off than someone who can walk 10m. Likewise, 1000m > 100m. Does 500km give any more benefit than 50km? Maybe, but probably only for extreme use-cases (maybe competitive extreme-distance walking? I'd rather drive!). Similarly with technique - if walking technique is horrible it matter for mobility and injury risk. Beyond a certain level, it probably matters less. And maybe walking extremely well is a visual-social signal, as a marker for someone who is in good shape? I'd read somewhere research done on dancing as a marker of fitness for attractiveness to the opposite sex.
You have nothing at stake with the walking of others. I have friends who are massage therapists. They notice the walking quality of others and what it means closely. When I see a doctor I personally have a lot at stake, which is why I do my homework.
What about the equiv. of code reviews? For surgeons, take videos of each surgery, and have them rated by an impartial review panel. I can't find the link now, but I recently saw a segment posted on Youtube that compared a surgeon with good technique, to one with poor technique, and it correlated with the overall outcome of the patients.
I remember this, but I also can't find it. Part of it involved maintaining proper view of the field through laparoscopes, minimizing wasted movement, and avoiding unnecessary tissue damage. That last one has the most direct effect on outcomes, while the former two can lead to the third.
The lower ranked surgeons would have the scope jostling around and the laparoscopic instruments would contact nearby tissues, while the higher ranked surgeons just seemed to have more elegance in his skill; a laser-precision focus to borrow from this thread and an intention and purpose for every movement.
I had a similar impression watching that video on surgery technique to watching high level Starcraft player replays versus some afternoon ladder replays by mid-level players. When placed side-by-side, the differences were fairly obvious even if you've never done the act yourself.
What stuck out for me was the difference in attitude between the two example centers. It's "67% is still pretty good, so keep it up" vs "90% is a big decrease from 105%; this is a failure and we need to make adjustments." In articular, the former type of attitude is certainly the norm from my experience (with medical care in general, I have no experience with CF).
This idea applies to more than medicine. A boss once told me, speaking about a piece of work I thought was pretty good, "If it could be better, it's wrong." It stuck with me, and I've found that everyone I meet who is exceptional at some skill has this attitude.
When I hear "the perfect is the enemy of the good" it's usually in response to the sentiment, "I can't ship this yet; it's not 'perfect'".
In the context of this article: You have a patient who needs to be treated -- today. They can't wait for a hypothetical perfect treatment. Indeed, more fundamentally, the only way to discover a "perfect" treatment would be to use today's least-worst solution, with the determination to measure and improve it continually.
Also the point of the article was that it's not so much the techniques -- which all the CF centers know about and use -- but the aggressiveness of their consistent application.
This isn't about a perfect design. It's about doctors who are willing to coach/goad/persuade patients into doing consistently what they already know they need to do. (And who are willing to be sticklers for consistency to the point of being a bit of a PITA to their colleagues.)
Hmm, no. 'The perfect is the enemy of the good' is one way to sum up the law of diminishing returns, and when you're operating within fixed constraints that matters a great deal, because inefficiently allocated resources have to come from somewhere. I'm a perfectionist by inclination, but in a team environment perfectionism can manifest as tunnel vision and optimizing for a local maximum.
I met people literally unable to finish project until it was perfect. It was never perfect. They just kept iterating and iterating the first part of the projects and the whole thing was never finished.
They were not forced to produce crap by overly active manager, it was not that situation. Manager did not cared for months.
Perfect is the enemy of the good refers to the above problem.
In most contexts I have worked the risk comes not from building the system incorrectly but building the wrong system - i.e. getting the requirements rather than the implementation wrong.
I would generally rather have a less than perfect system that can be used now than a "perfect" system 3 months from now as, in my opinion, you only really validate what you have built when it is in production and in use and delivering value.
The counter-idiom is "Excellence is good enough.". I have used it as a mantra to counter the paralysing need not to act on things unless perfection can be achieved.
Iteration is a pretty good way to deal with this, IME. Get it to be serviceable but mediocre ASAP, then continue improving until better marginal benefits can be found elsewhere.
This reminds me of a true story I once read about the best car salesman in the country at the time. Single handedly he was selling more than some not-so-small entire dealerships. The difference was his incredible focus and approach to even the tiniest detail that could help a sale and willingness to try new things.
One thing that stands out in my memory about this was the fact the he made sure he always had 3 (!) pens for the customer to sign the sales agreements. Just in case 1 pen stopped working and the backup pen malfunctioned. He brought this sort of laser focus to absolutely every aspect of the business.
How many salesmen would have a spare pen, already in place, let alone 2 spares. How many would even have thought about it and what the right number might be?
Focus, attention to detail and continuous innovation.
Unfortunately, there's also a cargo-cult aspect to this kind of thing, when managers start saying: "Everyone is now required to have three pens at their station at all times."
I'm in a company where the suit-wearing side says "we need innovation" to the tech side, but they don't actually want a discussion about what they see as opportunities or directions. They just want to invoke it by rote ritual, or order it as if we kept some on shelves.
Patients with CF at Fairview got the same things that patients everywhere did—some nebulized treatments to loosen secretions and unclog passageways (a kind of mist tent in a mouth pipe), antibiotics, and a good thumping on their chests every day. Yet, somehow, everything he did was different.
I found this quote especially interesting. I've often found this to be true, that you look over at someone who's much better, and on one hand they're just doing the same things you're doing, but somehow they're just on a different level. It's very difficult to distinguish exactly how someone achieves better results, that is, until it's been explained. Here, we see this clearly in how the level of lung function is perceived. Both centers provide the same treatment, but at the Cincinnati center lung levels around 70% are "okay", but at Minnesota, any drop is unacceptable, even if the patient is already at above-normal levels.
It sort of reminds me of the "Don't deal with it, fix it" article that was also posted recently. You'll be coasting around at a certain stable level of engagement until one day you see something you've never considered before and all of a sudden you see a whole new world of possibilities.
Good article, but I feel Atul Gawande mischaracterized "evidence-based practice" in a way that does our whole planet a disservice when he poo-poohed it to lionize Warwick.
Evidence based practice isn't really new, it's been a thing since '92. And while it is about using studies to make treatment decisions, it's not about forbidding doctors from thinking for themselves.
But what really got my goat was the fact that evidence-based medicine is the new thing, and it's an essential step in the evolution of medicine, and more people need to understand it. Evidence based medicine is about applying statistics to determine when treatments cause more harm than good. With modern medicine's advances in imaging techniques, we're hitting all sorts of new and dangerous problems where the benign abnormalities we all accumulate over our lives are treated at great expense, inconvenience and loss of health.
I don't think he was "poo-poohing" it exactly. If everyone only follows evidence based medicine there will never be any improvement because there will never be any changes.
So someone has to be the person who ignores it, who tries new things. And that someone, in this field, is Warwick.
The evidence part comes in when that person also tracks their results! i.e. makes it a study, and Warsick certainly seems to do so.
Yes I got the impression his attitude about evidence-based medicine is similar to that saying about accounting -- that "accounting is the art of looking in the rear view mirror and dragging your ass into the future".
i.e. Accounting isn't worthless. It's necessary, but not sufficient.
“We are used to thinking that a doctor’s ability depends mainly on science and skill. The lesson from Minneapolis is that these may be the easiest parts of care. Even doctors with great knowledge and technical skill can have mediocre results; more nebulous factors like aggressiveness and consistency and ingenuity can matter enormously... What the best may have, above all, is a capacity to learn and adapt—and to do so faster than everyone else.”
Doesn't the central limit theorum imply that most sane distributions of quality of care (including all doctors are equally good) will result in a bell curve?
There are tests you can do to tell if the variance between care centers exceed that you would expect from chance, but the mere observation of a bell curve seems completely uninteresting.
Say each doctor's patients had a common constant death rate for that doctor, not a crazy assumption, and those constant death rates across doctors followed a gamma distribution (not hard, because I could fit my ass with a gamma), then the distribution of each doctor's patients' life expectancies would be exponential and the distribution of life expectancies of the whole group would be Pareto.
Exponential is completely positively skewed and thin-tailed, and Pareto is roughly the same shape but very fat-tailed.
I hope that more and more data becomes available around healthcare and outcomes!
Medicare recently produced this data http://www.nytimes.com/interactive/2014/04/09/health/medicar... and the center that my wife works at used it to to produce this graph http://blog.parathyroid.com/parathyroid-surgery-medicare/. They are quite a bit better at this particular operation than anywhere else in the US, not surprising given how many more they do, however it is often difficult for patients to know or understand that - who should they trust? More data please! Interestingly this operation is one that Atul Gawande (who wrote this article in 2004) specializes in too.
Significantly, this article neglects to consider how sample size (and demographics) might affect the measurement of hospital treatment quality.
tl;dr: If you don't adjust for the sample size, what may appear to be the best, or worse, hospital, may look like that only because of how they have (un)successfully treated a single patient, and thereby yield an unrealistic estimate of patient quality (ie; 100% cure rate of a single patient, which may not be reflective of the actual caliber of doctors).
Further reading, which is strongly recommended if you want to learn more;
Sorry...but I don't get what you're postulating...that the doctors and hospitals whose reputations are at stake...nevermind the lives of their patients who are affected by policy decisions...would be totally ignorant of that basic statistical caveat? That it's possible that the top-of-the-line hospital just happened to cure a couple of really lucky patients and everyone who has lauded that hospital has never heard of regression analysis?
I guess it's possible...but it's probably more likely that the New Yorker is not trying to be a reference on statistical methods. In any case, the statistical caveat you mention is arguably addressed in this catchall-paragraph that briefly describes the problem of quality-of-care statistics:
> In recent years, there have been numerous efforts to measure how various hospitals and doctors perform. No one has found the task easy. One difficulty has been figuring out what to measure. For six years, from 1986 to 1992, the federal government released an annual report that came to be known as the Death List, which ranked all the hospitals in the country by their death rate for elderly and disabled patients on Medicare. The spread was alarmingly wide, and the Death List made headlines the first year it came out. But the rankings proved to be almost useless. Death among the elderly or disabled mostly has to do with how old or sick they are to begin with, and the statisticians could never quite work out how to apportion blame between nature and doctors. Volatility in the numbers was one sign of the trouble. Hospitals’ rankings varied widely from one year to the next based on a handful of random deaths. It was unclear what kind of changes would improve their performance (other than sending their sickest patients to other hospitals). Pretty soon the public simply ignored the rankings.
Even with younger patients, death rates are a poor metric for how doctors do. After all, very few young patients die, and when they do it’s rarely a surprise; most already have metastatic cancer or horrendous injuries or the like. What one really wants to know is how we perform in typical circumstances. After I’ve done an appendectomy, how long does it take for my patients to fully recover? After I’ve taken out a thyroid cancer, how often do my patients have serious avoidable complications? How do my results compare with those of other surgeons?
(the author himself is a surgeon and has written a lot about the problems of reliably measuring quality of care performance)
To be clear, I'm not at all suggesting that doctors do this out of malice - only that effective statistical analysis is really, really, hard, and it's not immediately clear that surgeons (regardless of their international reputation) would have the background required to properly analyze these studies in a coherent framework, especially when it's not immediately relevant to a patients outcome in an operating theatre.
I'm not suggesting that they're ignorant of basic statistical facts, but I am definitely suggesting that they're not immediately aware of the subtle assumptions implicit in many of the statistical models they use.
For example, given the large numbers we're talking about, not only is it possible that the "number 1" hospital in any particular field is there because of statistical fluke, it's actually likely that this is the case.
The (mis)use of statistics certainly isn't limited to medicine, but it is one of the places where its misinterpretation has the biggest impact.
Ostensibly an article about medicine but really more about how to achieve true excellence at what you do ... definitely a long read but well worth the time
The same question came to my mind and I asked a doctor "Sir, what happens to the students in MBBS who just pass taking a border mark?" The doctor said you should still be ok because passing score itself is like 60% + lot of practice involved. I did not bother to check the pass mark required for the course but now-a-days I go to doctor who have at least 10+ years of experience.
With some else :) Keeping the joke aside. I would want them to assist senior doctor for some time and personally i always use second opinion. BTW, if you do not you can get anti-biotic off the shelf and most of the doctors will give you in first visit.
Really good takeaway near the end of the article: "What the best may have, above all, is a capacity to learn and adapt—and to do so faster than everyone else."
If competence is, as the article suggests, normally distributed, then there's no natural level at which to draw the cutoff line. But anyway, iirc, studies show no correlation between outcomes and class rank (nor selectiveness of medical school), at least for surgeons.
"Matthews had started a cystic-fibrosis treatment program as a young pulmonary specialist at Babies and Children’s Hospital, in Cleveland, in 1957, and within a few years was claiming to have an annual mortality rate that was less than two per cent. To anyone treating CF at the time, it was a preposterous assertion. National mortality rates for the disease were estimated to be higher than twenty per cent a year, and the average patient died by the age of three. Yet here was Matthews saying that he and his colleagues could stop the disease from doing serious harm for years. “How long [our patients] will live remains to be seen, but I expect most of them to come to my funeral,” he told one conference of physicians.
In 1964, the Cystic Fibrosis Foundation gave a University of Minnesota pediatrician named Warren Warwick a budget of ten thousand dollars to collect reports on every patient treated at the thirty-one CF centers in the United States that year—data that would test Matthews’s claim. Several months later, he had the results: the median estimated age at death for patients in Matthews’s center was twenty-one years, seven times the age of patients treated elsewhere. He had not had a single death among patients younger than six in at least five years."