Weird and Wacky Engineering
Meaningless surveys drive me crazy--and diminish real science
Bill Schweber
3/14/2012 10:50 PM EDT
Almost without exception, when I see a news report starting off with the expression "according to a recent survey," "according to a study," or "according to a report," I skip the story.
Why? Because most of the so-called items are nearly worthless pap. So many are half-baked and biased stalking horses for a political agenda, or thinly veiled prods more funding for a project or cause, and are often self-reported drivel.
For example, I just saw "More Than 15% Obese in Nearly All U.S. Metro Areas" and made the mistake of actually looking at it. It had a detailed table listing the purported obesity percentages in various cities: Poor McAllen-Edinburg-Mission, Texas, where residents were the most likely to be obese, at 38.8%.
Except for one thing: this is ridiculous accuracy and precision. As you read further into this Gallup survey—a supposedly respected organization, BTW—you find out this apparent clarity to three significant figures is based on—get this—"self-reported height and weight." As we say, "oh, well, never mind."
What's the accuracy of such data? Pretty lousy, I'd say. After all, many people, shocking to say, probably under-report their weight and over-report their height. And who reports it, anyway? It's probably not a representative cross-section or sample, I'd bet.
I suppose it is possible to argue that the many data errors will average out, as they sometimes do, but not in this case, I'm pretty sure. I think giving the results with a range of, say, "between 35 and 40%" for those Texas-area folks would be far more realistic
This is just one example of a news story based on a nearly meaningless survey; there are so many more. Why do these get so much coverage?
I think there are several reasons. First, reporters and the media like to cite supposedly scientific studies that support their personal political and social biases. Plus—and here's the dirty little secret—they are easy topics to cover and write up. You take the reported survey results from the press release, make a call to get a few quotes, write a summary with an attention-grabbing lead-in, and poof! your work is done: no need to do real research, reporting, checking of facts, analysis of method, or ask for other views from other experts. In short: many reporters are lazy, and this is an easy way to do an assignment.
Normally, I wouldn't care about these pseudo-scientific reports—but the problem is that what's "pseudo" one day soon brings everything which sounds similar down to its level. As a result, credible scientific reports with solid backing are soon lumped in with these semi-bogus ones, and soon all elicit the same "you must be kidding" response.
One of the first lessons engineers and scientists earn, either formally or via experience, is not to impugn too much precision into test results, and to recognize sources of error in experimental data analysis. A second lesson takes longer to learn: those who do things "quick and dirty" often diminish those nearby, who are trying to do it right and clean.
Have you seen any so-called studies, surveys, or reports that made you angry due to their idiocy? And worse, have you ever had to write one? ◊
|
|
|
|
|
agk
3/15/2012 4:20 AM EDT
Many times surveys were meaning full and lot of constructive actions were taken up. Similarly the test result's accuracy reflects the quality of the item relevant to the tested parameter. Doing any thing right and clean happens by doing the surveying and by testing the accuracy.
Sign in to Reply
dflovely
3/19/2012 9:34 AM EDT
Maybe we need a survey to find out what percentage of surveys are based on real science? :-)
Sign in to Reply
healthreform
3/15/2012 10:39 AM EDT
Actually the so called survey can be very misleading. You make a great point that "self reported" Height and weight is not going to give an accurate picture of obesity. In addition,obesity is more linked to food chemicals than to height and weight obsession
_______________________
Mark Lillson Nutritionist MD http://spirithappy.org
Sign in to Reply
Work to Ride comma Ride to Work
3/23/2012 4:42 PM EDT
You mean chemicals like sucrose, frucose, carbohydrates?
Sign in to Reply
Earl54
3/16/2012 2:27 PM EDT
"As a result, credible scientific reports with solid backing are soon lumped in with these semi-bogus ones, and soon all elicit the same "you must be kidding" response."
Except that, among the general public, the surveys do not elicit that response, and therein lies the problem. So many take the results of semi-bogus, or worse yet completely bogus, surveys, studies, and samples, as the absolute truth, and then can be led by the nose to reach any conclusion desired. I've been amused, appalled, or offended by the reactions of people to these studies, depending on the study and the importance of the results.
Sign in to Reply
Sparky_Watt
3/16/2012 6:51 PM EDT
I will take this one step further. Surveys usually very accurately report the opinion of thousands of people. The key word is opinion. The truth is often far different. So, the people are being led by other's opinions, not reality.
Sign in to Reply
skdarbha
3/16/2012 2:33 PM EDT
TV & Radio are full of bogus 'studies'. "A new study from the Univ of [Fill-In-The-Name] suggests that men who own convertibles are 18% more likely to develop colon cancer in the next 20 years."
Okay, I made that up.. but many stories aren't any more rigorous. The surveys are plain ridiculous - They look at 2 factors that might often have NOTHING in common - Then relate them simply bcuz they happened to be on the same questionnaire given to participants.
Sign in to Reply
tINY
3/16/2012 2:35 PM EDT
Keep in mind that "obese" has a specific meaning in medicine that isn't what most people think. Except for kickers, every single NFL pro is "obese" even if they have almost no body fat... So you have to consider the definitions used in your studies too. That lesson comes a little later, though.
Sign in to Reply
BrainiacV
3/16/2012 4:04 PM EDT
My spousal-unit worked at a few survey companies. One that dealt with health issues, strived for absolute accuracy. Another that did consumer surveys, cooked the numbers to whatever the client wanted.
Sign in to Reply
WKetel
3/17/2012 4:29 PM EDT
It would be simple to take a fairly accurate survey for obese people, just watch and count them on the sidewalk. The downside is it would only represent the folks on that sidewalk, heading to the pizza places or whatever. IT seems like a lot of surveys are not nearly that well organized, and I am certain that many are just plain made up-numbers pulled from thin air. But I never believe any surveys so it does not bother me much. Also, when they call on the phone, do they really think they are getting right answers?
Sign in to Reply
RWatkins
3/29/2012 6:18 PM EDT
All surveys have problems. In this case, "Is that guy obese, or did he have bad posture/scoliosis so was sticking his belly out?" It gets worse when the medical definition a few years ago was forced to leave the old height-weight charts, and then came up with the "new" body mass index charts that are basically the old height weight charts renamed and do not address percent body fat and physical condition. I once knew a Navy machinist mate who was a weight lifter and was probably under 5% body fat, and he was required to go on a diet because his body mass index was obese. At least as important as the survey or measurement itself is the criteria or metric in use. If obese means 210 pounds at 6 feet tall, there are a lot of people who are not "fat" who will be branded obese. Sadly, that is also used as an excuse by health insurance companies to raise rates.
Sign in to Reply
BicycleBill
3/18/2012 10:43 AM EDT
Sorry, just "looking" doesn't tell you if they are obese--just "large": as someone noted above, the 250-pound/all muscle/no fat footballer is not obese, but sure is big. You need to know their weight (mass) to set a body mass index. But you are right about that last comment--do people even give the right answers? Do they "fudge" to sound better, or maybe they don't know, so they make something upo?
Sign in to Reply
sudo
3/18/2012 7:26 PM EDT
Too right!
With the danger of sounding elitist, I think the underlying problem is that the wider public and certainly the media has a very poor understanding of stastics, probabilities and scientific concepts.
Good engineers and scientists have an understanding and appreciation for subtleties, probabilities and degrees of things. For example, something is found to be n% with a 95% confidence level. The average person just wants a simple answer.
Let's take a hotly debated issue, like global warming. There have been a lot of studies, the overwhelming majority of which reported a high probability of warming global climate.
Then, the debate started, with lobby groups on both sides with vested interests. Some media folks had the misguided belief that you have to give equal weight to both sides in the debate, even if it took digging up some pseudo-scientific opinions and most often they did not have the capacity of distinguishing good studies from bad ones. Some of the counterarguments were also spurious, like saying that "this year, where I live, we had an unusually cold winter so where is your global warming now?"
Then, there is the question of if it's happening, what should be done about it or should anything be done, at all? ...but let's not go there!
Once I heard an interview with a climate scientist who tried to rationally explain the details and also the uncertainties of the modeling and how to evaluate the collected data. The arguments were quite compelling about warming global temperatures and rising sea levels but what the interviewer took away was that since we can't say with 100% certainty that sea levels will rise x feet/metres by year y, it probably won't be happening at all.
So, opinions flourished with religious fervour at the extreme ends of the scale, which suited the media perfectly - that's what sells papers and gives pageviews and airtime.
Meanwhile, the average person would listen to this and ask: "So, is it happening or not?!"
Sign in to Reply
cdhmanning
3/28/2012 5:49 PM EDT
Even when the real survey is rigorous and statistically sound, journalists will boil out the goodness before serving up to their readership.
Percentages are useless metrics unless accompanied by confidence intervals etc, yet journalists will routinely drop these from their articles because their readership don't understand the relevance.
A change from 40% to 45% might or might not be statistically relevant, but that is all most reporters will focus on.
Sign in to Reply
BicycleBill
3/18/2012 9:38 PM EDT
The public suffers from what scientist Douglas R Hofstadter called "innumeracy", later popularized in a book of the same name by mathematician John Allen Paulos (see http://www.innumeracy.com/).
And it has gotten worse, even basic calculations such as making change are now done "automatically" by the cash register--or if the purchase is by credit/debit card, there is no change to figure, anyway.
Sign in to Reply
Jerry.Avins
3/19/2012 4:26 PM EDT
Too many people lack statistical savvy, so these surveys don't get the belly laughs they deserve. Language skills today are poor too. Many reporters illogically think that to "beg the question" means to ask for an answer. (If you think do too, see http://en.wikipedia.org/wiki/Begging_the_question. An air-conditioner technician wanted to replace a 1/3-hp motor with a 1/4-hp one (probably all he had in the truck) and justified its adequacy by stating that four is bigger than three, so it was even more powerful. Either he believed that or he hoped the customer would. She threw him out.
I sometimes use a humorously wrong word and so do you. What does "impugn too much precision" mean? Did you mean "impute"?
Sign in to Reply
Work to Ride comma Ride to Work
3/23/2012 4:54 PM EDT
Even ignoring the statistical errors and other mathematical issues, some surveys are crafted to get the desired answers. I recall humoring a telephone survey once (never again) and listening to the questions it became quite apparent to me that the questions were worded in such a way to get a specific answer. After about the third or fourth question, I told the person on the other end of the line that this survey was male bovine excrement (but I used the more familiar term), and that these questions were simply rigged to get a desired answer and I hung up. They didn't want people's opinion or position, they wanted cleverly fabricated data to "support" their a priori position.
Sign in to Reply