The conclusion is entirely not based on results from this study, if it can be called a proper study.
What can be told from the abstract, is that sample size is extremely small (19 individuals), there don't seem to be any control parameters and the distinction of "lonely" and "not lonely" is made based on self reporting, which is yet another source of potential errors.
Other than that, the subject studied was reaction speed, which can be interpreted as basic alertness, and even if we consider the results reliable, which we can't, the conclusion could be made almost entirely different - lonely people are more alert to their surroundings, while non-lonely are more calm and relaxed.
I'm not sure who are reviewing these studies, but it would never pass as a conclusive study by any standards in non-social sciences.
> is that sample size is extremely small (19 individuals)
This is not meaningful. I'm not sure why this keeps getting repeated, but it's entirely possible to get significant results from small sample sizes. No comment on the rest of the study, but I just don't think sample sizes are worth mentioning, and I wouldn't even call that "extremely small". This is what significance testing is for, after all.
With regards to self-reporting, I read a very nice defence on this in a book on the scientific study of consciousness [1], which basically says that we need to let go of this idea that self-reporting is somehow "unscientific", because some phenomena only exist as experiences (and I'd think loneliness is one of them), and therefore the scientific study of these experiences has to incorporate self-reporting on some level. You can't get a blood test for loneliness.
I don't think self-reporting is unscientific, but I do think it's not particularly reliable. Reported loneliness may correlate with actual loneliness, but they're different phenomena.
This was really driven home to me in doing product development and user testing. I've worked with some great user researchers, and they were extremely skeptical about self-reported characteristics. The classic example is, "Would you buy X?" People are terrible at answering that correctly and honestly. But there were a bunch more where we carefully rigged situations so we could measure behavior rather than self-reported internal state, because behavior ended up being a better measure of internal state for us. I could well believe the same is true for loneliness.
I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain. We currently may not have the tools to usefully read that physical state. We may not have a blood test for loneliness, but we may one day have a brain scan for it. Or, perhaps sooner, a pattern in the sort of data that will be gathered by the iPhone 15m (the "m" being for medical, of course).
> I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain.
Absolutely. But in order to map those experiences to those 'physical states' you have to trust self-reporting. I agree with you that questions such as "Would you buy X?" are incredibly unreliable when it comes to self-reporting, so I understand the skepticism, but I don't think loneliness falls into that category, because it's not something that can be externally tested (we can't observe the person to see if they're "truly" lonely or not).
Hmm. I'm not following your logic. It sounds like you're saying that reported loneliness isn't unreliable because we can't externally test it. I'd think it's just the opposite; if we have no external tests, that's a great reason to assume it's unreliable.
That said, I'm not as confident as you that it's not externally testable. These researchers, for example, believe they have found EEG test results that correlate with loneliness. It would seem to me that loneliness could change both self-generated behavior and response to stimuli, which could yield reliable measures. Sociality is a biologically mediated phenomenon, so there could be metabolic measures, like how cortisol is an indicator of stress. We expect loneliness to happen in the brain, so aside from EEG measures, perhaps we could find an FMRI test that exposes it.
> It sounds like you're saying that reported loneliness isn't unreliable because we can't externally test it. I'd think it's just the opposite; if we have no external tests, that's a great reason to assume it's unreliable.
I'm not disagreeing with you on that--there's always the possibility someone's lying to you, and that has to be designed into your study as best as possible. It's difficult. My point is more that we don't have any other choice if we want to do a scientific study of things like loneliness. Or anything else that only exists within the context of one's own head for that matter.
> That said, I'm not as confident as you that it's not externally testable.
Perhaps it is. I'm only wary of these things because it's very easy to find correlations. Studies of depression are full of this. You can find a certain brain region or gene or hormone level or whatever correlated with depression, but then you find someone who also feels depressed who doesn't exhibit those symptoms. So do you tell that person they're not "really" feeling depressed? Of course not. And fMRI studies are open to abuse too; I'm sure you've heard of the "dead salmon" study, but if not: http://www.wired.com/2009/09/fmrisalmon/
I'm actually optimistic, as you are, that there will be better diagnostic tests for a whole range of mental conditions/experiences/illnesses/etc., but the core criticism still holds: if you want to trust a diagnostic test of a mental state, ultimately, at some point, you will also have needed to trust the self-reporting of that mental state.
I think we broadly agree, but I can't go as far as agreeing with this: "If you want to trust a diagnostic test of a mental state, ultimately, at some point, you will also have needed to trust the self-reporting of that mental state."
I think it's true we can generally trust people's self reports. But we all know that's not always true. There's the classic comedy staple:
A: You sound upset.
B: I AM NOT UPSET!
But there's more subtlety there, too. Anybody who deals with children has had the experience of understanding what the child is feeling better than said child does. And I remember being on the other side of that as a kid. People would ask me how I was doing, and I would self-report whatever answer I thought they wanted, not my actual state, which was often mysterious to me.
And although that has lessened as I grow older, I still can look back on key moments as an adult where I thought I was feeling one thing but, upon later reflection, realize it was something else.
So I think the best we can do is to come up with a series of external tests that correlate well with self-reporting and with each other. With that, yes, I think we can tell people, "No, you're not really depressed." For example, in the months after I lost my mom to a brain tumor, I thought that I was depressed. I went to see a therapist and she said, "No, you're mourning an important loss. It's supposed to be like this." And she was right.
This! We find focus groups nearly useless; folks are so eager to please. You get nothing but "Yes this is great. Maybe change the color to yellow". But you learn nothing about what would really appeal or sell.
I'm thinking focus groups should change to some sort of psychological test situation. "You can take only one of these home for free. Which one do you want?" I.e. perceived scarcity (cost) would elicit honest responses.
Yeah, I have never found focus groups of any value. I don't understand how one separates social effects from actual effects. And many user researchers agree. E.g.:
User focus groups may not be useful, but user testing can be. I'm thinking of studies where you give users some tasks to do, and watch how they interact with the UI to accomplish that.
As part of an ongoing study I have been conducting my whole life, I have found self-reported cleverness, self-reported attractiveness, and self-reported indications that "the guy over there wants fight me" to all be positively correlated with alcohol intake.
Can you elaborate on why small sample sizes are not a cause for concern?
It's my understanding that when designing a study, you look to strengthen statistical power, which is determined by a combination of a) significance criterion (p values, confidence intervals) b) magnitude of the effect and c) sample size. Increasing the sample size decreases sampling error.
Just from a common-sense perspective, if the sample of the study was n=2, then any results derived from significance testing would be nonsensical - how can we extrapolate our findings to a broader population? Is n=19 good enough?
> It's my understanding that when designing a study, you look to strengthen statistical power, which is determined by a combination of a) significance criterion (p values, confidence intervals) b) magnitude of the effect and c) sample size. Increasing the sample size decreases sampling error.
I don't disagree with this at all. You always want to increase your sample size if possible. But as you also say, the magnitude of the effect you're observing is also important; if it's very strong, then it can be revealed by a small sample size. An absurd example could be "cyanide capsules kill (n=2)". Obviously, the smaller the sample size, the harder it is to draw reliable conclusions, but I wouldn't think n=19 is necessarily too small. I think many of phase I clinical trials aren't much more than that.
More importantly, mentioning a "small" sample size makes a nice soundbite, and is often used to discredit studies. These studies might be good or they might not be, but talking about sample size can be used as a "thought-terminating cliche". Perhaps a person is looking for evidence that reinforces their own beliefs (we all do it), and they read a criticism of a study, it mentions "small sample sizes", and they can walk away happy in a belief the study was flawed. But all that's really happened is that it's prevented a proper discussion of the study taking place. I'm not sure how much of a problem this is, but I've seen it happen frequently.
Got it, thanks for the explanation. I generally trust that the authors of such studies know what they're doing when they design an experiment with a certain n value in mind. Agreed that the insufficient n argument does crop up a lot around HN.
Also important, I recall a note from one of my statistics classes to the effect of, strong sample sizes are often counter-intuitively small.
I want to say the example was something like polling in a political race, and the math worked out that you could get reliable results polling only a few tens of people so long as you had random sampling.
One of my professors had a rule of thumb of discounting anything below n=25, but you absolutely need to ensure random sampling as its one of the fundamental assumptions of regression - making sure the probability of being part of the sample is uniform across the entire population.
> ... but it's entirely possible to get significant results from small sample sizes.
It is also possible to simply guess some effect. You don't need study for stating a result. A purpose of study is to prove something. A study that can not be told to be proven and is not replicable is not worth anything, as further scientific knowledge cannot be built on top of it. If a study do not prove or at least strengthen confidence in a theory, what is it worth? Not even meta-analysis will consider studies like these to prove some entirely different theory because it is just that unreliable.
Now I'm not saying this particular study is entirely like that, but the flaws everyone mentioned are concerning.
What can be told from the abstract, is that sample size is extremely small (19 individuals), there don't seem to be any control parameters and the distinction of "lonely" and "not lonely" is made based on self reporting, which is yet another source of potential errors.
Other than that, the subject studied was reaction speed, which can be interpreted as basic alertness, and even if we consider the results reliable, which we can't, the conclusion could be made almost entirely different - lonely people are more alert to their surroundings, while non-lonely are more calm and relaxed.
I'm not sure who are reviewing these studies, but it would never pass as a conclusive study by any standards in non-social sciences.