Today's article is a guest post Catherine Clavering of Kiss Me Deadly.
As a psychologist, particularly one of a therapeutic bent, I never felt much like a scientist. I mean, we're fluffy bunnies in the world of “proper” science, no matter how much we talk about epistemological theory or Feyerabend's ideas or squeeze gratuitous Greek letters and geometry style diagrams into our work.
As a lingerie professional, however, I'm sometimes quite scared about how little the industry seems to know about what it means to be scientific.
At its heart, science is about argument. You ask a billion questions, come up with an idea, and everyone – INCLUDING YOU – throws everything they can at it to try and disprove it. I'm very lucky that KMD fans seem to have learnt this in their cribs, and so will cheerfully debate you into the ground on why our survey respondents might not represent our core customer group, among other things.
Today, I'm going to talk about sampling (because it's either that or get a large bruise from hitting myself on the head the next time someone says something stupid about breast health research). No, not the sexy lingerie example sampling, but the choosing a bunch of people to test your idea on kind of sampling. Yes, I know this sounds terribly boring but bear with me, because if you stick with it, you too can have arguments with people on the internet about statistics! Which is everyone's idea of a good time, no? Okay, well, alternatively it means you'll be able to evaluate whether you really need to get worked up about whether bras cause IBS or not.
The main piece of research I'm going to use as an example today was published by the Portsmouth Breast Health Research and is named "The Evaluation of Professional Bra Fitting Criteria for Bra Selection and Fitting in the UK." I'm using this because I'm pretty sure this is the only non-commercial piece of research that's being quoted in all those “80% of women are wearing the wrong bra size!!!” things you see.
This research set out to compare two different methods of bra fitting; one was the plus 4 method, and the other is fitting by eye. I could criticise a whole heap of things about their research methods, but let's stick with the sampling. You'll notice that the title of the paper says “In the UK,” which one might reasonably assume means they want this research to be relevant to the whole of the UK.
As you've likely guessed, there are quite a few women in the UK. In fact, there are more than 32 million of us according to the 2011 census. That's a good many to get to try on bras, so of course no research will ever include all of them, just as the census doesn't catch all the people in the country.
Fortunately, because of the Magic Of Statistics (or Gaussian Distribution, if you'd like to go and look it up), we don't need to see all of the women to get a good idea of what they are like. We just need to get a really good example set of them. So here are some things to keep in mind when you're getting that example set (i.e. sample) of people.
- The number. Your sample size needs to reflect the total population that your research covers. In this case, that's women from the whole of the UK, a number of about 32 million.
- Geography. If your sample is supposed to represent the whole of the UK, then it should include people from all over the UK. A major issue in experiments is that the vast majority are done on people from one small area (namely, from around the universities) and then interpreted to represent the entire world!
- Socioeconomic Status. This means that if it's possible social status and income might make a difference in what you are testing, you should make sure your sample represents as much diversity in that area as possible. In the UK, there's an established link between income and weight (the richer are thinner), so for any kind of size research you really must do this.
- Other relevant demographic variables. In this case, age might be a significant variable; for example, we know that size changes much with age.
- How are you finding your sample? The methods you use to find your sample set can make a huge difference to results. Imagine if you were doing research on bra fit, and your sample only came from shops who told their unhappy customers to come and see you? If you surveyed women who just made a bra purchase they loved, you could get a completely different result!
So how does this research do?
Well, Portsmouth Breast Health Research recruited 45 women.
I bet even the least mathematical amongst you are saying, “Isn't 45 an awful lot less than 32 million?”
And yes. Yes, it is.
Now, it's a very long time since I had to calculate a sample size, and I'm sure many of you can do it much better than me, but I'm pretty certain that to get a decent sample you need at least 350 people. Furthermore, most research that covers the whole population needs at least 1000 people. So basically, 45 isn't really cutting the mustard.
This is especially true when you consider that all the study participants were from Portsmouth. Now, the UK is tiny and Portsmouth is a pretty nice place, but I can assure you that most people would not be impressed if we, say, let Portsmouth decide major government policies because it seems pretty representative of the rest of the country. Even worse, all the study participants were recruited from university staff and students, who are all likely to be fitted in the same place (and therefore run into the same fit issues).
Moving on, participants were recruited via poster and email. If you wanted to get very picky, it's important to know what the emails and posters said, as you'll get wildly different samples in response to “Not sure what bra size you are? Help us with our research!” vs. “We are looking for women to help test fitting methods for bras.” Although, you may be wondering, surely if you mention bra fit at all, you're likely to get people who aren't happy with their fit? If that was your first thought, give yourself several brownie points because one of the major problems in social science research is that volunteers are atypical! The good news though, is that at least they weren't paid to do it.
The researchers managed to recruit women aged from 19 to 58 (average age 32). They also have some means and standard deviations for weight and height, but nothing for socioeconomic status (class and/or income) or ethnicity. They also didn't include anyone who had breast fed or been pregnant in the last year, and I'd be interested to know if you thinks that's wise or not!
On the basis of sample size and geographic distribution alone, you can't really use this study to demonstrate much about the UK as a whole in terms of bra fitting, and you certainly can't use it to demonstrate anything about women all over the world. It's also worth noting that there's another study, done by a plastic surgeon on more than 100 women, that found a similar percentage of bra fit issues but in a completely different direction!
I'll stop here, but hopefully, in the future, when you hear someone spout off a statistic in bra marketing, you'll think, "Where did that number come from and how many people did they use to get it?"