16% > 11% represents a 31% decrease in the absolute number of hospitalizations. That's nearly a ONE THIRD improvement in the hospitalization rate. In places where hospitals are at or near capacity, that would make a huge difference.
Or were you trying to make a statement about the statistical validity of the result, because the outcomes differ by 5%? Because that's not how statistics work. A well constructed study of sufficient size can provide detect arbitrarily small differences in outcomes--you just need a bigger study size, the smaller the difference you want to detect. You would learn about this any 101-level statistics course, and a lot more useful information, too.
Intro to stats says that studies are very difficult and that bias is everywhere. Reinforces how stats can be whatever you want them to be.
Larger Number of people in the sample is a good start. However, bias is also a human element. Therefore you need that study reproduced independently, multiple times.
Researchers don't like this because studies are expensive and hard, and their boss is telling them to publish and get funding. The system is broken.
For starters, experimental bias isn't really an issue, here... This was an observational study, with crystal clear objective criteria for coding the dependent & independent variables.
A cheap, simple observational study like this is merely a first step. No reasonable expert is claiming that this study proves anything worth making medical policy changes over. This study's sole purpose is to establish whether it's worth investing in further studies, or if we can just shitcan the idea now. Subsequent studies will cost more money, and involve larger samples, better controls, and get a lot more scrutiny from peers.
Time and money are finite resources. You have to have some kind of system for deciding where to spend those resources.
This is how medical science works. It's got a ton of pitfalls, but we still do it this way because so far nobody has come up with a better alternative.
> experimental bias isn't really an issue, here... This was an observational study, with crystal clear objective criteria for coding the dependent & independent variables.
I have no position on this particular study, but in general you can make your coding as simple and objective as you like and it won't do much about experimental bias. Objective processing of bad data isn't better than subjective processing of bad data.
All studies have some level of risk for bias... But that doesn't mean all studies have the same level of risk.
The essay you linked is very good, but it's actually illustrating my point... The mouse study ran into bias problems because they had to go and directly measure something in nature, and then turn that into a number. That introduces several opportunities for error.
But this SSRI/COVID study isn't doing that. They're literally just looking at patient records, and counting hospitalization vs current SSRI usage. They're not picking up mice to count ticks... They're exporting records to a spreadsheet, and summing columns.
Now, these guys might have problems with the quality of the records they're relying on... Who knows whether the records are accurate or not. But that's a problem of data quality, not experimental bias.
You might say "Who cares? Either kind of error undermines the results, just the same!" But it definitely suggests that the other guy doesn't know what he's talking about... Because if he did understand stats, he would have used the right terminology.
> The essay you linked is very good, but it's actually illustrating my point... The mouse study ran into bias problems because they had to go and directly measure something in nature, and then turn that into a number. That introduces several opportunities for error.
> But this SSRI/COVID study isn't doing that. They're literally just looking at patient records, and counting hospitalization vs current SSRI usage. They're not picking up mice to count ticks... They're exporting records to a spreadsheet, and summing columns.
> Now, these guys might have problems with the quality of the records they're relying on... Who knows whether the records are accurate or not. But that's a problem of data quality, not experimental bias.
Well, no, I have to disagree with this analysis. The mouse study described in that essay didn't run into bias problems, and the reason it didn't is that it was counting the ticks itself. But other tick-counting studies did have bias problems (due to bad methodology), and the Lyme-disease studies based on the bad tick-counting studies had the same bias problems (because they were using the bad data). The bias doesn't go away when you wash it through two publications instead of one. It's not a different kind of problem -- or even a different instance of the same kind of problem! -- and there's no reason to give it a different name.
You sound like you just took a stats 101 class and now are bragging to a bunch of adults about how much you learned in stats 101. Sincerely, someone working in the field for last 7 years.
Sounds like I don't know nearly as as you... I minored in stats 20+ years ago, but since then I do a lot more arguing about stats than actually using them for work.
You're kinda right about one thing, though... I am definitely talking down to OP, and a couple of other folks.
I'm frustrated. There are real problems in how science uses statistics, and it sounds like OP & co have heard about those problems. But the way they talk about this study, they're just throwing crap against the wall to have something to say. They don't really seem to understand the problems they're talking about, or how those problems apply to the study we're currently talking about.
I should change my attitude, and stop taking out my frustrations on these folks. If they're wrong, I'm probably not going to change their minds by insulting them.
> For starters, experimental bias isn't really an issue, here... This was an observational study, with crystal clear objective criteria for coding the dependent & independent variables.
Just a nitpick: it was not an observational study, it was a true experiment, because they controlled the treatment: it was assigned by researchers at random. It is by definition not an observational study which have a distinctive feature that it doesn't control for a treatment.
I ran it through an AB test significance calculator using the upper bound for each.
p value was 0.9972 and the power was 92.40% it's a significant result.
"Variant B’s conversion rate (11.47%) was 30.08% lower than variant A’s conversion rate (16.40%). You can be 99% confident that variant B will perform worse than variant A."
> When the DSMC met on Aug 5, 2021, it recommended that the TOGETHER trial stop randomly assigning patients to the fluvoxamine group, as this comparison had met the prespecified superiority criterion for the primary endpoint (prespecified superiority threshold 97·6%).
That seems problematic. Did they stop the study because they had achieved statistical significance?
You'd be interested in learning about the origin and application of Chi-Squared tests (or Fisher's exact test). Detecting changes in population means, and making sure those changes aren't just random noise, is quite straightforward and can be done by hand.
It won't be possible for me to explain the answer to your satisfaction, if you lack at least a basic education in probability & statistics. Maybe Khan Academy has an intro level course, or a "Stats for Humanities" variant that can help you get started.
IMO, one of the largest hurdles that modern society is going to have to overcome is the average person's understanding of large (or small) numbers and basic statistics.
I think a large issue within that hurdle is that a lot of people simply won't want to learn, or apply the knowledge after learning.
And the funny thing is that the original commenter was looking at a trivial math problem. It wasn't even a question of statistical validity, it was just "What does it mean to compare percentages to each other?" That's grade-school mathematics.
I see this kind of error pretty frequently, and it's usually not a failure of mathematics... Most often, the person has some pre-existing negative emotional reaction to the stated conclusion, and their brain searches for something to criticize, in order to soothe those negative emotions.
I could only speculate as to why the original poster doesn't like hearing that an existing drug shows promise in treating COVID. Maybe they're a big Ivermectin booster, and they don't like the idea of another drug threatening their favorite?
I think you might want to reconsider your opinion of others. I believe the OP was referring to this result being on the edge of statistical significance given:
The effect size relative to the base population;
The fact that many similar trials are performed and discarded with similar drugs;
And as we see when looking deeper, there are other issues such as early stopping. It is not unreasonable to suspect the effect could disappear given more data.
YOU are raising some valid points... but based on the OP's other comments, they seem to have a very limited knowledge of statistics--and yet, they are pushing a very critical line on this topic.
OP is asking other posters to do quite a bit of homework for them... As in, "Is this study big enough?" <<YES>> "OK, but how big does it need to be to prove X?" ... At a certain point, it seems clear that OP is not arguing in good faith, because they feel compelled to make the critical statements, first, without understanding the topix.
I would caution you to not extend sympathy to OP's cause, simply because you happen to share a critical eye for this study. There is a HUGE difference between being right for the right reason, vs the wrong reason... and that's how we get crap like Ivermectin for COVID.
In a way it is though. There is a 84% chance you wouldn't go to the hospital either way, an 11% chance that you will go to the hospital either way, and a 5% chance that you will be saved from going to the hospital. So there is a 1 in 20 chance that this drug will save you from being hospitalized. It could also make your illness less severe in the other scenarios though.
Population and individual stats are different though. If someone is older and in a higher risk group, the drug could indeed reduce hospitalization risk for that individual much more than 5%, while for other low-risk groups it could do next to nothing.
Well thank you for stats 101 but what I was trying to ask was that if this level of confidence promising within medical/pharma field - i.e. do we have stats on studies that said something like out of the 767 or whatever studies that claimed a drug sounded promising because it was associated with 5% outcome improvement - 90% resulted in long term commercial success of the drug.
I mean take Remdesivir - https://www.nejm.org/doi/full/10.1056/nejmoa2007764 - statistically significant yet some other studies found it had no benefits. At least I don't think it was a slam dunk or promising in real life as day the vaccines.
It sounds like you're asking "What's the risk that this study will turn out to be wrong, in the long run?" That's a very interesting question, but sadly there's no way to answer it from the available data.
Different studies can show different results for a lot of reasons... Sometimes, it's just random chance. Other times, it's due to bad study design, or poor experimental controls, or deliberate P-hacking. There's no way to tell from the results of a single study.
But that's not what this study is for, anyway... This is a preliminary observational study to look for possible drug candidates that might impact COVID severity. It's only suggesting potential future directions of investigation, not advocating for immediately dosing everyone with SSRIs... Medical policy changes won't happen until a much larger body of evidence has been established.
It's the nature of science that not all experiments produce correct results. Sometimes, we get it wrong--but that's why we have to keep repeating experiments, and trying different variations, before we trust a conclusion.
In the meantime, though, medical decision makers will weigh the costs vs benefits of proceeding with an experimental treatment... If the risks are low, and the need is great, they'll probably let the door open to more experiments. Most people seem to think that's a pretty sensible way to do things.
Or were you trying to make a statement about the statistical validity of the result, because the outcomes differ by 5%? Because that's not how statistics work. A well constructed study of sufficient size can provide detect arbitrarily small differences in outcomes--you just need a bigger study size, the smaller the difference you want to detect. You would learn about this any 101-level statistics course, and a lot more useful information, too.