I'm listening to NPR this morning on the way into work, and they're interviewing a guy who wrote a book about some interrogations he did for the CIA. He's telling what sorts of approaches worked in his interrogations, and what things did not.
First thing I thought was:
the plural of anecdote is not data!
But is that really always true? Can we, should we always discard the anecdote purely based on the fact that it's a story someone is telling about his personal experience? When does an experienced person, discussing knowledge he's gained in the course of activity in his profession, become an authority we should listen to rather than a guy with an anecdote we should disregard? And when does that cross over into the Appeal to Authority fallacy?
So, D&Ders, when does anecdote become useful insight? When does claiming you have insight instead of stories become an appeal to authority? Is there a middle space in there somewhere?
Posts
Let 'em eat fucking pineapples!
Now, ideally this still means actually compiling it, not just saying you heard it somewhere. Some sort of survey of those reporting being tortured or an analysis of success via interrogation in the context of kinder or less kind(but not tortuous) treatment.
Lots of approaches can use anecdotes meaningfully, it's just usually people stop there.
Attempting to prove or support a broad or wide ranging notion with an anecdote (or even a few anecdotes) would be foolish. This kind of silly goosery comes up when people try to justify their racism/sexism/homophobia/whatever because "one time a black person/woman/gay/etc did ______." But, there are some unique experiences (I would classify being a CIA interrogator as one of these) that aren't going to be prevalent enough to construct an exhaustive and meaningful dataset either because there aren't enough people out there or because people aren't willing to talk about it.
Sometimes we need to just do the best we can with the information that we have available to us. I am not going to take the time to track down the numbers on it, but I bet there aren't many people who are performing interrogations for the CIA, and I would also guess that the people who do work in that capacity generally remain fairly reticent. As long as the news story didn't attempt to frame the discussion as "this is what it's like for all interrogators ever," I'm alright with them having a limited perspective for the segment.
The reason stereotypes are so often pervasive and intractable is because much of them are true; we just frame that data through a filter that assigns values to those traits, often without attempting to understand the origination of those behaviors or traits.
Maybe he's just really bad at it.
I'm not sure how you could be bad at torture.
The thing is, there's nothing to compare that to in a vacuum. Or him saying it does work. You need large groups of people trying other things before you really have anything you can glean anything from.
The reason anecdotes are not data is that they're not controlled, or usually compared to anything. People say "Well, this works!" without thinking about the times it didn't work (or vice versa, though I think that's rarer), and don't compare it to something else that may work.
So, to use the torture as an example, and (huge caveat) ignoring ethics entirely, to decide if torture works you just compare the percentage of times someone tells you what they want with torture to the percentage of times they tell you without torture. Then you have data*.
*You can't stop here though, you should compare torture to not torture to free blowjobs to treating the subject like normal people to treating them like they're in a five star hotel, if you're really interested in maximal effectiveness of interrogation technique.
Well I'm not sure that's the case in our example here. If I try to put a nail in a board with a fork I don't need to try every other tool in the house to determine its not working.
Of course it would be different if we were determining what type of hammer was best.
Generally I would say that when anecdotes are used to define a theory they can be disregarded as proof. As in "A banker just squandered my lifesavings, all bankers are criminals". When anecdotes are used to provide detail to a wider theory they should be something to listen to. "Careless behavior by bankers led to a lot of people losing their life savings. I myself lost $X because my banker only thought of personal gain".
In general, I would call what the original post describes unscientific. In my mind, that description makes it pretty questionable.
(Also, if I were to do this scientifically, I'd use two control groups. One group that knew something that wasn't tortured and one where the group doesn't know anything, but are tortured anyway. The latter would be especially interesting, because interrogation really suffers from expectation bias)
So, in your judgement there is no amount of experience that can render an anecdote useful for informational purposes?
Let's take an actual example; A century and some decades ago, there were professional doctors. Their job was medicine. There were also women(!), who gave birth(!?) and then in a depressingly large number of cases died from puerperal fever.
A professional within this noble field, named Ignaz Semmelweis, found this distressing. He worked at a obstreticians clinic, well, the 19th century equivalent anyway, in Vienna. He set out to examine why this was happening, and through careful collection of observational data, noticed that women treated by male doctors rather than female wetnurses were more likely to die contract the fever. One of the firs theories postulated to explain this was that the presence of the male upset the fragile female state of mind, and this upset led to their eventual death.
After an inspiringily tragic incident where an older doctor nicked his finger during an autopsy (that he was performing) and also died of fever, our friend Ignaz saw a connection. The first clue was that female wetnurses never performed or watched autopsies. After some experimentation and careful observation concluded that we could save a whole bunch of mothers from dying if all of these fine professionals were to wash their fucking hands after fondling dead corpses during autopsies - this was before germ theory I think, so the exact reason why this worked was a bit unclear, but that it worked was blatantly obvious.
Now, he had this scientifically documentated. There were graphs showing mortality rates. Yet he died embittered and insane in an asylum, after his attempts at convincing his fellow professionals to think of the mothers and wash their hands was met with scorn and disbelief. Because his fellow professionals weren't all that into science; They had professional experience and the very idea that true gentlemen would wash their hands all the godamn time was absurd - unlike that idea about the fragile temper of the fair sex, where the mere presence of a man during the birthing led to fever.
Experience in a given field does not immunize to confirmation bias; It didn't during Ignaz' era, and it doesn't now. A collection of anecdotes, whether from a professional doctor or a professional dowser, is not by it's nature scientific. Depending on what we're doing, the bar we set for how scientific something has to be varies, and my personal opinion is that torture is important enough to warrant the same kind of scientific rigor we award medicine; You do trials, and you do them under carefully monitored double blind conditions. You don't line up an experienced doctor whose mother eats the medicine and have him say "I have extensive experience within this field and I say this medicine is completely harmless; My mother eats it and so does her sisters, and they feel great!".
It was probably Ali Soufan, and his upcoming book is titled The Black Banners: The Inside Story of 9/11 and the War Against al-Qaeda.
It's something like if Peyton Manning wanted to talk about good quarterback play, I think we would all listen. Especially in a world where Football Outsiders didn't exist.
Now, I don't have a comprehensive theory of what "legitimate expertise" is beyond "I know it when I see it." If a person has years of experience in a field, or has written multiple peer-reviewed papers on a given subject, or has extensive education in a given field, I'm prepared to call that "legitimate expertise." So, I guess, a combination of education, on-the-job and otherwise lived experience, and publication history all make for legitimate expertise in my book (EDIT: and not just any publication history--self-published books would have less weight in this scenario than a book that was published through Oxford University Press, for instance. So there's maybe a "through established, credentialed, widely respected channels/institutions" addendum to those criteria).
I also think there are degrees of authority: I think that a woman has more authority to speak about the experiences of women than a man does, and that a person of color has more authority to speak about the experiences of PoC than a white person does, but I don't think that either of those individuals has ultimate or comprehensive authority on everything related to being a woman or a person of color respectively. This is because although they are unquestionably legitimate experts about being a woman, or being a person of color, there is not necessarily a broad consensus among either of those groups about most things.
I mean, not that you shouldn't study it anyway. With whatever you can get past IRB.
Yeah I cannot imagine any scientific study about torture ever, ever, ever passing IRB. You know. Because of the torture of test subjects. I guess theoretically people could give their informed consent to be tortured, but even then I can't imagine IRB passing it because of the ethical problems of what kind of compensation can you give? How could that compensation ever not be coercive? It's practically impossible just to TALK to certain populations.
This is probably going to sound a little snarky, but it's meant as a legitimate response.
Isn't that story about Ignatz just a single data point? I mean, couldn't you view it as an anecdote?
In other words, how often does this happen? A situation in which the majority of "experts" in a particular field use their experience to deduce a conclusion, rather than conducting extensive studies or experiments, and that conclusion turns out to be incorrect.
I'd venture to say that the more we know about the various sciences, the less likely we are to have a massive deductive consensus that is incorrect. To use the nail example, I don't need to conduct a study on whether hammers work better than forks for smacking a nail into a piece of wood; my knowledge of basic principles of physics tells me that the hammer will certainly be better.
This isn't to say it's not worth conducting studies on a lot of things. Certainly there will be many cases in which our deductions lead to false conclusions (probably because we are basing them off of faulty assumptions and don't know it), but as we learn more about the various sciences, I think the potential for that decreases over time.
So to bring this back on topic, I don't think there's anything wrong with accepting an expert's conclusions, so long as we sort of keep in mind that there is a potential for error (and so long as we confirm that the expert really is an expert).
For a more serious example of experience turning out to not only be bunk, but dangerous and unjust: Quite a few death row cases.
The execution of Cameron Todd Willingham for an arson that killed his daughters. Which no fire expert in the country actually believes was even arson, just an unlucky electrical fire. Fire investigators were simply firemen who were apprenticed to the previous fire investigator, with no scientific or formal training for their duties at all. And neither the prosecutor or the defense attorney, both of whom are still practicing law, seem to give a damn.
short version: http://youtu.be/P-cMpKfDPHg
long version: http://www.newyorker.com/reporting/2009/09/07/090907fa_fact_grann
There's also the West Memphis three, released this year, who were convicted with no physical evidence, and based on the confession of a minor with a severe learning disability after 12 hours of interrogation, whose confession contradicted the evidence at the crime scene and had to be coached by the police on what to confess. You'd think if we could trust the logical thinking of our police and prosecutors' offices they'd have never been tried, and if we could trust the public they'd have never been convicted. Oh and one of the victims' dads had his DNA recovered from the scene, and has a record of wives dying mysteriously. And yet they served 18 years in prison before finally getting out on an Alford plea.
http://youtu.be/JS1yzjNjJGs
Then there's the trial of Anthony Graves, which for some reason there's no good youtube videos about, but was sentenced to death in an absolutely insanely corrupt trial, and only just barely managed to finally win the chance for a new trial. And yet despite the years of struggle for anyone to grant his appeal, when it went to court the prosecutors asked for the trial to be dismissed because he was clearly innocent. His prosecutor is still practicing law as well.
http://www.texasmonthly.com/2010-10-01/feature2.php
When your mechanic goes "I've seen this before - your x is broken" do you ignore him because he has no study to demonstrate how often behavior y is caused by problem x? I mean, surely there's a line somewhere here, where you accept an authority's explanation based on his experience, rather than based on empirical data.
The line is when you have empirical data. People probably should have trusted doctors to not wash their hands before performing births before Ignatz published. But not after.
Otherwise, its probably wise to trust the expert, they have more data than you do, anec or regular, because the perform the operations and look at the situations more.
Edit: A better question is probably "when should we seek for better data?" because, of current, the answer seems to be "when you have anecdata that does not correspond with the experts anecdata"
Don't get me wrong, everything that is unscientific is not neccesarily incorrect, but that's kind of the point I originally wanted to make. Non-systematically treated data(/anecdotes) isn't scientific, and cannot support a scientific theory. Depending on what we're doing and the standard of rigour we're currently willing to accept, this is a bigger or a smaller problem.
But a consensus among scientists/professionals is not, in itself, science.
Argument from Authority is perfectly valid in many cases.
The history of hand-washing and doctors goes on and on, long after Ignatz. It will make you sick if you look into it further. Empirical data showed clearly for decades that doctors were refusing to wash their hands and were thus killing some of their patients, and doctors still ignored it and thought it was insulting to tell them to wash their hands. We were well into the digital age before significant headway was made into the problem, and the problem still exists today.
Anyway, there are so many philosophical approaches to this that I don't even know what to post. For starters, I gather that there is an implied contradiction in the OP that isn't clearly stated... "This guy said that torture worked, but we know from data that it doesn't!" However, almost every time I've heard someone claim that data or studies have shown that torture doesn't work, that person is actually just reciting something they've heard. They have no actual knowledge of any such studies or data. They've just heard that there are studies and/or data, and that they showed that torture doesn't work. So what we have is expert anecdote vs. hearsay that pretends to be data.
And that is practically inevitable. Scientific methods are great ways of ensuring a higher level of reliability and usefulness in how we discern truth. But not only is there a lot of room for error in the methods themselves, but more importantly the data and and experimental results are under constant risk of being turned into anecdote, whether it's in the variable that wasn't considered for isolation, or in how the conclusion was written, or how the media reported the results, or how it spread among people in word-of-mouth. When the rubber meets the road, most of us don't actually have a formal data set stored in our brain, we have a memory of something we heard from someone about something they read in the Post about something a scientist studied. And we think it far superior to "anecdote."
The OP said nothing about torture working. For all anyone just reading this thread (including me) knows, the guy could have said that torture doesn't work while other methods do.
It was my understanding that the intent of the OP was to challenge an expert claiming that torture worked when data says that it does not. Else I'm not sure how the example in the OP relates to the topic.
I agree, but what are you referring to here?
It was really quite good. Though I don't think this guy is trying to develop a science of interrogation, just relating what works in his experience (building rapport), and how the contractors that were brought in had a different ideology (breaking the prisoner down with enhanced interrogation techniques). In the segment he doesn't really comment on whether or not much actionable intelligence was gathered through enhanced interrogation techniques, but that a lot of very valuable intelligence (about KSM, Jose Padilla) was extracted before such techniques were applied, and enhanced interrogation was falsely credited.
In the expanded Frontline PBS interview, he goes on to say that torture "has not provided one actionable item of intelligence". The CIA and FBI refused to deny, confirm, or even interview for Frontline on this story.