As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/

Anecdote vs Experience

spool32spool32 Contrary LibraryRegistered User regular
edited September 2011 in Debate and/or Discourse
I'm listening to NPR this morning on the way into work, and they're interviewing a guy who wrote a book about some interrogations he did for the CIA. He's telling what sorts of approaches worked in his interrogations, and what things did not.

First thing I thought was: the plural of anecdote is not data!

But is that really always true? Can we, should we always discard the anecdote purely based on the fact that it's a story someone is telling about his personal experience? When does an experienced person, discussing knowledge he's gained in the course of activity in his profession, become an authority we should listen to rather than a guy with an anecdote we should disregard? And when does that cross over into the Appeal to Authority fallacy?

So, D&Ders, when does anecdote become useful insight? When does claiming you have insight instead of stories become an appeal to authority? Is there a middle space in there somewhere?

spool32 on

Posts

  • VeritasVRVeritasVR Registered User regular
    Experience is never more than a series of anecdotes (samples), but someone with a lot of them (as opposed to just one or two) becomes more and more credible. Eventually, they're going to have enough samples that their experience is statistically significant.

    CoH_infantry.jpg
    Let 'em eat fucking pineapples!
  • durandal4532durandal4532 Registered User regular
    Case studies and observational studies are useful when fully fledged experimental procedure is untenable. You can't establish torture as the independent variable without actually committing crimes against humanity, so it's necessary to compile data from alternate sources.

    Now, ideally this still means actually compiling it, not just saying you heard it somewhere. Some sort of survey of those reporting being tortured or an analysis of success via interrogation in the context of kinder or less kind(but not tortuous) treatment.

    Lots of approaches can use anecdotes meaningfully, it's just usually people stop there.

    Take a moment to donate what you can to Critical Resistance and Black Lives Matter.
  • MalyonsusMalyonsus Registered User regular
    The sum of anecdotes all taken together can end up being significant, but one of the large things that separates anecdotes from samples is the recall of the person with the anecdote. Confirmation bias tends to eliminate 'less illustrative' anecdotes, tainting whatever statistical significance the personal experience as a whole might have had.

  • SyrdonSyrdon Registered User regular
    The only concern I would have with someone using a large number of anecdotes is selective recall. Accurate recording relatively shortly after the fact should resolve that though (of course, at that point, you're starting down the path to collecting data)

  • LoveIsUnityLoveIsUnity Registered User regular
    Anecdotes aren't inherently bad; they do, however, need to be framed correctly. I would also suggest that the more unique a situation is, the closer an anecdote comes to data.

    Attempting to prove or support a broad or wide ranging notion with an anecdote (or even a few anecdotes) would be foolish. This kind of silly goosery comes up when people try to justify their racism/sexism/homophobia/whatever because "one time a black person/woman/gay/etc did ______." But, there are some unique experiences (I would classify being a CIA interrogator as one of these) that aren't going to be prevalent enough to construct an exhaustive and meaningful dataset either because there aren't enough people out there or because people aren't willing to talk about it.

    Sometimes we need to just do the best we can with the information that we have available to us. I am not going to take the time to track down the numbers on it, but I bet there aren't many people who are performing interrogations for the CIA, and I would also guess that the people who do work in that capacity generally remain fairly reticent. As long as the news story didn't attempt to frame the discussion as "this is what it's like for all interrogators ever," I'm alright with them having a limited perspective for the segment.

    steam_sig.png
  • AtomikaAtomika Live fast and get fucked or whatever Registered User regular
    edited September 2011
    It is worth mentioning that anecdotes, or personal experience, ends up shaping the way most of us perceive the world in "concrete" terms that many of us feel apply universally.

    The reason stereotypes are so often pervasive and intractable is because much of them are true; we just frame that data through a filter that assigns values to those traits, often without attempting to understand the origination of those behaviors or traits.

    Atomika on
  • Alfred J. KwakAlfred J. Kwak is it because you were insulted when I insulted your hair?Registered User regular
    What's the title of the book?

  • Styrofoam SammichStyrofoam Sammich WANT. normal (not weird)Registered User regular
    I guess it depends a lot on the context and source of the anecdotes. I mean if a guy who interrogates for a living says that in his experience torture doesn't work, that is anecdotal, but it would be silly to ignore it.

    wq09t4opzrlc.jpg
  • ChanusChanus Harbinger of the Spicy Rooster Apocalypse The Flames of a Thousand Collapsed StarsRegistered User regular
    I guess it depends a lot on the context and source of the anecdotes. I mean if a guy who interrogates for a living says that in his experience torture doesn't work, that is anecdotal, but it would be silly to ignore it.

    Maybe he's just really bad at it.

    Allegedly a voice of reason.
  • Styrofoam SammichStyrofoam Sammich WANT. normal (not weird)Registered User regular
    Chanus wrote:
    I guess it depends a lot on the context and source of the anecdotes. I mean if a guy who interrogates for a living says that in his experience torture doesn't work, that is anecdotal, but it would be silly to ignore it.

    Maybe he's just really bad at it.

    I'm not sure how you could be bad at torture.

    wq09t4opzrlc.jpg
  • ShivahnShivahn Unaware of her barrel shifter privilege Western coastal temptressRegistered User, Moderator mod
    edited September 2011
    I guess it depends a lot on the context and source of the anecdotes. I mean if a guy who interrogates for a living says that in his experience torture doesn't work, that is anecdotal, but it would be silly to ignore it.

    The thing is, there's nothing to compare that to in a vacuum. Or him saying it does work. You need large groups of people trying other things before you really have anything you can glean anything from.

    The reason anecdotes are not data is that they're not controlled, or usually compared to anything. People say "Well, this works!" without thinking about the times it didn't work (or vice versa, though I think that's rarer), and don't compare it to something else that may work.

    So, to use the torture as an example, and (huge caveat) ignoring ethics entirely, to decide if torture works you just compare the percentage of times someone tells you what they want with torture to the percentage of times they tell you without torture. Then you have data*.

    *You can't stop here though, you should compare torture to not torture to free blowjobs to treating the subject like normal people to treating them like they're in a five star hotel, if you're really interested in maximal effectiveness of interrogation technique.

    Shivahn on
  • Styrofoam SammichStyrofoam Sammich WANT. normal (not weird)Registered User regular
    So, to use the torture as an analogy, and (huge caveat) ignoring ethics entirely, to decide if torture works you just compare the percentage of times someone tells you what they want with torture to the percentage of times they tell you without torture.

    Well I'm not sure that's the case in our example here. If I try to put a nail in a board with a fork I don't need to try every other tool in the house to determine its not working.

    Of course it would be different if we were determining what type of hammer was best.

    wq09t4opzrlc.jpg
  • UrcbubUrcbub Registered User regular
    It would be hard to draw a definite line of where and when anecdotes is something to listen to or when to disregard.

    Generally I would say that when anecdotes are used to define a theory they can be disregarded as proof. As in "A banker just squandered my lifesavings, all bankers are criminals". When anecdotes are used to provide detail to a wider theory they should be something to listen to. "Careless behavior by bankers led to a lot of people losing their life savings. I myself lost $X because my banker only thought of personal gain".

  • CalixtusCalixtus Registered User regular
    So, to use the torture as an analogy, and (huge caveat) ignoring ethics entirely, to decide if torture works you just compare the percentage of times someone tells you what they want with torture to the percentage of times they tell you without torture.

    Well I'm not sure that's the case in our example here. If I try to put a nail in a board with a fork I don't need to try every other tool in the house to determine its not working.

    Of course it would be different if we were determining what type of hammer was best.
    Mostly because "every other tool" is a bad null hypothesis. If we went with "I look at the nail for the same amount of time as I attempt to manhandle it with a fork" we'd get a good "Does it work at all?" hypothesis testing going, and we could then select other tools for comparision to test the fork against something else.


    In general, I would call what the original post describes unscientific. In my mind, that description makes it pretty questionable.


    (Also, if I were to do this scientifically, I'd use two control groups. One group that knew something that wasn't tortured and one where the group doesn't know anything, but are tortured anyway. The latter would be especially interesting, because interrogation really suffers from expectation bias)


    -This message was deviously brought to you by:
  • spool32spool32 Contrary Library Registered User regular
    Calixtus wrote:
    So, to use the torture as an analogy, and (huge caveat) ignoring ethics entirely, to decide if torture works you just compare the percentage of times someone tells you what they want with torture to the percentage of times they tell you without torture.

    Well I'm not sure that's the case in our example here. If I try to put a nail in a board with a fork I don't need to try every other tool in the house to determine its not working.

    Of course it would be different if we were determining what type of hammer was best.
    Mostly because "every other tool" is a bad null hypothesis. If we went with "I look at the nail for the same amount of time as I attempt to manhandle it with a fork" we'd get a good "Does it work at all?" hypothesis testing going, and we could then select other tools for comparision to test the fork against something else.


    In general, I would call what the original post describes unscientific. In my mind, that description makes it pretty questionable.


    (Also, if I were to do this scientifically, I'd use two control groups. One group that knew something that wasn't tortured and one where the group doesn't know anything, but are tortured anyway. The latter would be especially interesting, because interrogation really suffers from expectation bias)


    So, in your judgement there is no amount of experience that can render an anecdote useful for informational purposes?

  • Styrofoam SammichStyrofoam Sammich WANT. normal (not weird)Registered User regular
    Yeah if thats the case then what is a professional opinion?

    wq09t4opzrlc.jpg
  • CalixtusCalixtus Registered User regular
    spool32 wrote:
    Calixtus wrote:
    So, to use the torture as an analogy, and (huge caveat) ignoring ethics entirely, to decide if torture works you just compare the percentage of times someone tells you what they want with torture to the percentage of times they tell you without torture.

    Well I'm not sure that's the case in our example here. If I try to put a nail in a board with a fork I don't need to try every other tool in the house to determine its not working.

    Of course it would be different if we were determining what type of hammer was best.
    Mostly because "every other tool" is a bad null hypothesis. If we went with "I look at the nail for the same amount of time as I attempt to manhandle it with a fork" we'd get a good "Does it work at all?" hypothesis testing going, and we could then select other tools for comparision to test the fork against something else.


    In general, I would call what the original post describes unscientific. In my mind, that description makes it pretty questionable.


    (Also, if I were to do this scientifically, I'd use two control groups. One group that knew something that wasn't tortured and one where the group doesn't know anything, but are tortured anyway. The latter would be especially interesting, because interrogation really suffers from expectation bias)


    So, in your judgement there is no amount of experience that can render an anecdote useful for informational purposes?
    It depends on how you define 'useful for informational purposes'. Many unscientific theories concerning generalities throughout history have been informational when applied within a certain framework, without neccesarily being objectively correct. The whole philsophical rationale for science and the scientific method is that there is a world, and this world works in a certain way, and that it is by scientific study that we can determine how this works.

    Let's take an actual example; A century and some decades ago, there were professional doctors. Their job was medicine. There were also women(!), who gave birth(!?) and then in a depressingly large number of cases died from puerperal fever.

    A professional within this noble field, named Ignaz Semmelweis, found this distressing. He worked at a obstreticians clinic, well, the 19th century equivalent anyway, in Vienna. He set out to examine why this was happening, and through careful collection of observational data, noticed that women treated by male doctors rather than female wetnurses were more likely to die contract the fever. One of the firs theories postulated to explain this was that the presence of the male upset the fragile female state of mind, and this upset led to their eventual death.

    After an inspiringily tragic incident where an older doctor nicked his finger during an autopsy (that he was performing) and also died of fever, our friend Ignaz saw a connection. The first clue was that female wetnurses never performed or watched autopsies. After some experimentation and careful observation concluded that we could save a whole bunch of mothers from dying if all of these fine professionals were to wash their fucking hands after fondling dead corpses during autopsies - this was before germ theory I think, so the exact reason why this worked was a bit unclear, but that it worked was blatantly obvious.

    Now, he had this scientifically documentated. There were graphs showing mortality rates. Yet he died embittered and insane in an asylum, after his attempts at convincing his fellow professionals to think of the mothers and wash their hands was met with scorn and disbelief. Because his fellow professionals weren't all that into science; They had professional experience and the very idea that true gentlemen would wash their hands all the godamn time was absurd - unlike that idea about the fragile temper of the fair sex, where the mere presence of a man during the birthing led to fever.


    Experience in a given field does not immunize to confirmation bias; It didn't during Ignaz' era, and it doesn't now. A collection of anecdotes, whether from a professional doctor or a professional dowser, is not by it's nature scientific. Depending on what we're doing, the bar we set for how scientific something has to be varies, and my personal opinion is that torture is important enough to warrant the same kind of scientific rigor we award medicine; You do trials, and you do them under carefully monitored double blind conditions. You don't line up an experienced doctor whose mother eats the medicine and have him say "I have extensive experience within this field and I say this medicine is completely harmless; My mother eats it and so does her sisters, and they feel great!".

    -This message was deviously brought to you by:
  • enlightenedbumenlightenedbum Registered User regular
    What's the title of the book?

    It was probably Ali Soufan, and his upcoming book is titled The Black Banners: The Inside Story of 9/11 and the War Against al-Qaeda.

    Self-righteousness is incompatible with coalition building.
  • enlightenedbumenlightenedbum Registered User regular
    As for the thread topic: while the plural of anecdote is not data, there is something valuable in expertise. This guy (assuming it was indeed Ali Soufan, which it probably was as he's been everywhere lately) was considered by the US government's primary counterintelligence agency to be an expert interrogator. We should take his experience under consideration, especially in a realm where there isn't a lot of good access to data.

    It's something like if Peyton Manning wanted to talk about good quarterback play, I think we would all listen. Especially in a world where Football Outsiders didn't exist.

    Self-righteousness is incompatible with coalition building.
  • taoist drunktaoist drunk Registered User regular
    edited September 2011
    Not every appeal to authority is fallacious. If a person is indeed a legitimate expert and she or he is espousing a belief that is commonly agreed on among a majority or large plurality of legitimate experts in the field using information that wasn't collected through rigorous scientific experimentation, I see no reason to discount what she or he has to say just because nobody ran any double-blind tests. There is, of course, reason to take it with a grain of salt, and what they're saying isn't automatically true by virtue of their expertise, but it's worth more serious consideration than the opinion of some schmo.

    Now, I don't have a comprehensive theory of what "legitimate expertise" is beyond "I know it when I see it." If a person has years of experience in a field, or has written multiple peer-reviewed papers on a given subject, or has extensive education in a given field, I'm prepared to call that "legitimate expertise." So, I guess, a combination of education, on-the-job and otherwise lived experience, and publication history all make for legitimate expertise in my book (EDIT: and not just any publication history--self-published books would have less weight in this scenario than a book that was published through Oxford University Press, for instance. So there's maybe a "through established, credentialed, widely respected channels/institutions" addendum to those criteria).

    I also think there are degrees of authority: I think that a woman has more authority to speak about the experiences of women than a man does, and that a person of color has more authority to speak about the experiences of PoC than a white person does, but I don't think that either of those individuals has ultimate or comprehensive authority on everything related to being a woman or a person of color respectively. This is because although they are unquestionably legitimate experts about being a woman, or being a person of color, there is not necessarily a broad consensus among either of those groups about most things.

    taoist drunk on
  • durandal4532durandal4532 Registered User regular
    Man, the thing is: what is your methodology for conducting torture trials, and why exactly does it require study? Are we going to start torturing people if it ends up providing a statistically significant amount of information? The significant policy aspect of torture is that it is damaging to human life and liberty and can't be condoned in a civilized society, it being (anecdotally) ineffective is simply the cherry on top.

    I mean, not that you shouldn't study it anyway. With whatever you can get past IRB.

    Take a moment to donate what you can to Critical Resistance and Black Lives Matter.
  • taoist drunktaoist drunk Registered User regular
    Man, the thing is: what is your methodology for conducting torture trials, and why exactly does it require study? Are we going to start torturing people if it ends up providing a statistically significant amount of information? The significant policy aspect of torture is that it is damaging to human life and liberty and can't be condoned in a civilized society, it being (anecdotally) ineffective is simply the cherry on top.

    I mean, not that you shouldn't study it anyway. With whatever you can get past IRB.

    Yeah I cannot imagine any scientific study about torture ever, ever, ever passing IRB. You know. Because of the torture of test subjects. I guess theoretically people could give their informed consent to be tortured, but even then I can't imagine IRB passing it because of the ethical problems of what kind of compensation can you give? How could that compensation ever not be coercive? It's practically impossible just to TALK to certain populations.

  • Hi I'm Vee!Hi I'm Vee! Formerly VH; She/Her; Is an E X P E R I E N C E Registered User regular
    Calixtus wrote:
    spool32 wrote:
    Calixtus wrote:
    So, to use the torture as an analogy, and (huge caveat) ignoring ethics entirely, to decide if torture works you just compare the percentage of times someone tells you what they want with torture to the percentage of times they tell you without torture.

    Well I'm not sure that's the case in our example here. If I try to put a nail in a board with a fork I don't need to try every other tool in the house to determine its not working.

    Of course it would be different if we were determining what type of hammer was best.
    Mostly because "every other tool" is a bad null hypothesis. If we went with "I look at the nail for the same amount of time as I attempt to manhandle it with a fork" we'd get a good "Does it work at all?" hypothesis testing going, and we could then select other tools for comparision to test the fork against something else.


    In general, I would call what the original post describes unscientific. In my mind, that description makes it pretty questionable.


    (Also, if I were to do this scientifically, I'd use two control groups. One group that knew something that wasn't tortured and one where the group doesn't know anything, but are tortured anyway. The latter would be especially interesting, because interrogation really suffers from expectation bias)


    So, in your judgement there is no amount of experience that can render an anecdote useful for informational purposes?
    It depends on how you define 'useful for informational purposes'. Many unscientific theories concerning generalities throughout history have been informational when applied within a certain framework, without neccesarily being objectively correct. The whole philsophical rationale for science and the scientific method is that there is a world, and this world works in a certain way, and that it is by scientific study that we can determine how this works.

    Let's take an actual example; A century and some decades ago, there were professional doctors. Their job was medicine. There were also women(!), who gave birth(!?) and then in a depressingly large number of cases died from puerperal fever.

    A professional within this noble field, named Ignaz Semmelweis, found this distressing. He worked at a obstreticians clinic, well, the 19th century equivalent anyway, in Vienna. He set out to examine why this was happening, and through careful collection of observational data, noticed that women treated by male doctors rather than female wetnurses were more likely to die contract the fever. One of the firs theories postulated to explain this was that the presence of the male upset the fragile female state of mind, and this upset led to their eventual death.

    After an inspiringily tragic incident where an older doctor nicked his finger during an autopsy (that he was performing) and also died of fever, our friend Ignaz saw a connection. The first clue was that female wetnurses never performed or watched autopsies. After some experimentation and careful observation concluded that we could save a whole bunch of mothers from dying if all of these fine professionals were to wash their fucking hands after fondling dead corpses during autopsies - this was before germ theory I think, so the exact reason why this worked was a bit unclear, but that it worked was blatantly obvious.

    Now, he had this scientifically documentated. There were graphs showing mortality rates. Yet he died embittered and insane in an asylum, after his attempts at convincing his fellow professionals to think of the mothers and wash their hands was met with scorn and disbelief. Because his fellow professionals weren't all that into science; They had professional experience and the very idea that true gentlemen would wash their hands all the godamn time was absurd - unlike that idea about the fragile temper of the fair sex, where the mere presence of a man during the birthing led to fever.


    Experience in a given field does not immunize to confirmation bias; It didn't during Ignaz' era, and it doesn't now. A collection of anecdotes, whether from a professional doctor or a professional dowser, is not by it's nature scientific. Depending on what we're doing, the bar we set for how scientific something has to be varies, and my personal opinion is that torture is important enough to warrant the same kind of scientific rigor we award medicine; You do trials, and you do them under carefully monitored double blind conditions. You don't line up an experienced doctor whose mother eats the medicine and have him say "I have extensive experience within this field and I say this medicine is completely harmless; My mother eats it and so does her sisters, and they feel great!".

    This is probably going to sound a little snarky, but it's meant as a legitimate response.

    Isn't that story about Ignatz just a single data point? I mean, couldn't you view it as an anecdote?

    In other words, how often does this happen? A situation in which the majority of "experts" in a particular field use their experience to deduce a conclusion, rather than conducting extensive studies or experiments, and that conclusion turns out to be incorrect.

    I'd venture to say that the more we know about the various sciences, the less likely we are to have a massive deductive consensus that is incorrect. To use the nail example, I don't need to conduct a study on whether hammers work better than forks for smacking a nail into a piece of wood; my knowledge of basic principles of physics tells me that the hammer will certainly be better.

    This isn't to say it's not worth conducting studies on a lot of things. Certainly there will be many cases in which our deductions lead to false conclusions (probably because we are basing them off of faulty assumptions and don't know it), but as we learn more about the various sciences, I think the potential for that decreases over time.

    So to bring this back on topic, I don't think there's anything wrong with accepting an expert's conclusions, so long as we sort of keep in mind that there is a potential for error (and so long as we confirm that the expert really is an expert).

    vRyue2p.png
  • KanaKana Registered User regular
    edited September 2011
    The immediate impulse is to lend a lot of weight to personal experience, especially when it's backed up by decades and decades of professionals agreeing it's the most effective action. And yet... Personally I'm a classically trained chef, and you'd think hundreds of years of cooking the same classical dishes would give chefs a pretty good idea on what happens to food. But there's tons of traditions passed down that, when they were finally studied scientifically by the new molecular gastronomy movement, were found to be totally false. Salted water makes your pasta taste better, but it was commonly credited for raising the boiling temperature allowing the pasta to cook faster. Totally false, even if you put in something like a pound of salt to a gallon of water, it raised the boiling temperature by like a hundredth of a degree. All they needed was a quality thermometer, cooks SHOULD have known that decades ago, simply by temp'ing the water... And yet nobody did. Pastry chefs always accepted that the best way to whip a meringue was in a copper bowl, but when tested in a lab there was no measurable difference between the end result from just using normal stainless steel. Many cooks still sear the outside of the meat to 'seal the moisture inside', but that assumes you can somehow compress water inside the steak. Herve This has an entire chapter in his book Molecular Gastronomy taking down that idea, including a lot of lab tests.

    For a more serious example of experience turning out to not only be bunk, but dangerous and unjust: Quite a few death row cases.

    The execution of Cameron Todd Willingham for an arson that killed his daughters. Which no fire expert in the country actually believes was even arson, just an unlucky electrical fire. Fire investigators were simply firemen who were apprenticed to the previous fire investigator, with no scientific or formal training for their duties at all. And neither the prosecutor or the defense attorney, both of whom are still practicing law, seem to give a damn.

    short version: http://youtu.be/P-cMpKfDPHg
    long version: http://www.newyorker.com/reporting/2009/09/07/090907fa_fact_grann

    There's also the West Memphis three, released this year, who were convicted with no physical evidence, and based on the confession of a minor with a severe learning disability after 12 hours of interrogation, whose confession contradicted the evidence at the crime scene and had to be coached by the police on what to confess. You'd think if we could trust the logical thinking of our police and prosecutors' offices they'd have never been tried, and if we could trust the public they'd have never been convicted. Oh and one of the victims' dads had his DNA recovered from the scene, and has a record of wives dying mysteriously. And yet they served 18 years in prison before finally getting out on an Alford plea.
    http://youtu.be/JS1yzjNjJGs

    Then there's the trial of Anthony Graves, which for some reason there's no good youtube videos about, but was sentenced to death in an absolutely insanely corrupt trial, and only just barely managed to finally win the chance for a new trial. And yet despite the years of struggle for anyone to grant his appeal, when it went to court the prosecutors asked for the trial to be dismissed because he was clearly innocent. His prosecutor is still practicing law as well.

    http://www.texasmonthly.com/2010-10-01/feature2.php

    Kana on
    A trap is for fish: when you've got the fish, you can forget the trap. A snare is for rabbits: when you've got the rabbit, you can forget the snare. Words are for meaning: when you've got the meaning, you can forget the words.
  • Hahnsoo1Hahnsoo1 Make Ready. We Hunt.Registered User regular
    An anecdote (or a single data point) becomes meaningful at the same time a scientific study becomes meaningful... when it gives you a reliable basis for predicting the behavior in question. This ability to judge based on past information received is the main measure (in my mind, opinion here) of meaningful data. An anecdote is scorned because it often isn't reproducible, often is biased, and only provides one example of behavior to model future interactions. But if it's an anecdote that allows you to make meaningful predictions, then it has value.

    8i1dt37buh2m.png
  • SyrdonSyrdon Registered User regular
    Yeah if thats the case then what is a professional opinion?
    Half the time, utter crap with an air of believability. Experts aren't any less likely to suffer from confirmation bias, silly stereotypes or just plain foolishness than everyone else. The way to avoid those is to conduct a serious study, or at least make a serious attempt at collecting data when studies aren't possible (ie: every torturer writes a report of his findings, those findings are then checked against other peoples' reports and against fact that you do know to be either true or false). Appeals to authority are bad because they assume that the authority has a much lower rate of being wrong than is actually found in humans who are restricting themselves to their subject matter. Appeals to data are generally suggested as the fix because its much harder for the numbers to be wrong if the method you gathered them through was valid.

  • Hahnsoo1Hahnsoo1 Make Ready. We Hunt.Registered User regular
    Syrdon wrote:
    Yeah if thats the case then what is a professional opinion?
    Half the time, utter crap with an air of believability. Experts aren't any less likely to suffer from confirmation bias, silly stereotypes or just plain foolishness than everyone else. The way to avoid those is to conduct a serious study, or at least make a serious attempt at collecting data when studies aren't possible (ie: every torturer writes a report of his findings, those findings are then checked against other peoples' reports and against fact that you do know to be either true or false). Appeals to authority are bad because they assume that the authority has a much lower rate of being wrong than is actually found in humans who are restricting themselves to their subject matter. Appeals to data are generally suggested as the fix because its much harder for the numbers to be wrong if the method you gathered them through was valid.
    Can you definitively say that professional opinion is "half the time, utter crap with an air of believability"? Are we supposed to just take your word on it? Where are your numbers? *grin* Just kidding. (Not poking fun at you, just thought it was ironic)

    8i1dt37buh2m.png
  • spool32spool32 Contrary Library Registered User regular
    Syrdon wrote:
    Yeah if thats the case then what is a professional opinion?
    Half the time, utter crap with an air of believability. Experts aren't any less likely to suffer from confirmation bias, silly stereotypes or just plain foolishness than everyone else. The way to avoid those is to conduct a serious study, or at least make a serious attempt at collecting data when studies aren't possible (ie: every torturer writes a report of his findings, those findings are then checked against other peoples' reports and against fact that you do know to be either true or false). Appeals to authority are bad because they assume that the authority has a much lower rate of being wrong than is actually found in humans who are restricting themselves to their subject matter. Appeals to data are generally suggested as the fix because its much harder for the numbers to be wrong if the method you gathered them through was valid.

    When your mechanic goes "I've seen this before - your x is broken" do you ignore him because he has no study to demonstrate how often behavior y is caused by problem x? I mean, surely there's a line somewhere here, where you accept an authority's explanation based on his experience, rather than based on empirical data.

  • GoumindongGoumindong Registered User regular
    edited September 2011
    spool32 wrote:

    When your mechanic goes "I've seen this before - your x is broken" do you ignore him because he has no study to demonstrate how often behavior y is caused by problem x? I mean, surely there's a line somewhere here, where you accept an authority's explanation based on his experience, rather than based on empirical data.

    The line is when you have empirical data. People probably should have trusted doctors to not wash their hands before performing births before Ignatz published. But not after.

    Otherwise, its probably wise to trust the expert, they have more data than you do, anec or regular, because the perform the operations and look at the situations more.

    Edit: A better question is probably "when should we seek for better data?" because, of current, the answer seems to be "when you have anecdata that does not correspond with the experts anecdata"

    Goumindong on
    wbBv3fj.png
  • CalixtusCalixtus Registered User regular
    Calixtus wrote:
    spool32 wrote:
    Calixtus wrote:
    So, to use the torture as an analogy, and (huge caveat) ignoring ethics entirely, to decide if torture works you just compare the percentage of times someone tells you what they want with torture to the percentage of times they tell you without torture.

    Well I'm not sure that's the case in our example here. If I try to put a nail in a board with a fork I don't need to try every other tool in the house to determine its not working.

    Of course it would be different if we were determining what type of hammer was best.
    Mostly because "every other tool" is a bad null hypothesis. If we went with "I look at the nail for the same amount of time as I attempt to manhandle it with a fork" we'd get a good "Does it work at all?" hypothesis testing going, and we could then select other tools for comparision to test the fork against something else.


    In general, I would call what the original post describes unscientific. In my mind, that description makes it pretty questionable.


    (Also, if I were to do this scientifically, I'd use two control groups. One group that knew something that wasn't tortured and one where the group doesn't know anything, but are tortured anyway. The latter would be especially interesting, because interrogation really suffers from expectation bias)


    So, in your judgement there is no amount of experience that can render an anecdote useful for informational purposes?
    It depends on how you define 'useful for informational purposes'. Many unscientific theories concerning generalities throughout history have been informational when applied within a certain framework, without neccesarily being objectively correct. The whole philsophical rationale for science and the scientific method is that there is a world, and this world works in a certain way, and that it is by scientific study that we can determine how this works.

    Let's take an actual example; A century and some decades ago, there were professional doctors. Their job was medicine. There were also women(!), who gave birth(!?) and then in a depressingly large number of cases died from puerperal fever.

    A professional within this noble field, named Ignaz Semmelweis, found this distressing. He worked at a obstreticians clinic, well, the 19th century equivalent anyway, in Vienna. He set out to examine why this was happening, and through careful collection of observational data, noticed that women treated by male doctors rather than female wetnurses were more likely to die contract the fever. One of the firs theories postulated to explain this was that the presence of the male upset the fragile female state of mind, and this upset led to their eventual death.

    After an inspiringily tragic incident where an older doctor nicked his finger during an autopsy (that he was performing) and also died of fever, our friend Ignaz saw a connection. The first clue was that female wetnurses never performed or watched autopsies. After some experimentation and careful observation concluded that we could save a whole bunch of mothers from dying if all of these fine professionals were to wash their fucking hands after fondling dead corpses during autopsies - this was before germ theory I think, so the exact reason why this worked was a bit unclear, but that it worked was blatantly obvious.

    Now, he had this scientifically documentated. There were graphs showing mortality rates. Yet he died embittered and insane in an asylum, after his attempts at convincing his fellow professionals to think of the mothers and wash their hands was met with scorn and disbelief. Because his fellow professionals weren't all that into science; They had professional experience and the very idea that true gentlemen would wash their hands all the godamn time was absurd - unlike that idea about the fragile temper of the fair sex, where the mere presence of a man during the birthing led to fever.


    Experience in a given field does not immunize to confirmation bias; It didn't during Ignaz' era, and it doesn't now. A collection of anecdotes, whether from a professional doctor or a professional dowser, is not by it's nature scientific. Depending on what we're doing, the bar we set for how scientific something has to be varies, and my personal opinion is that torture is important enough to warrant the same kind of scientific rigor we award medicine; You do trials, and you do them under carefully monitored double blind conditions. You don't line up an experienced doctor whose mother eats the medicine and have him say "I have extensive experience within this field and I say this medicine is completely harmless; My mother eats it and so does her sisters, and they feel great!".

    This is probably going to sound a little snarky, but it's meant as a legitimate response.

    Isn't that story about Ignatz just a single data point? I mean, couldn't you view it as an anecdote?

    In other words, how often does this happen? A situation in which the majority of "experts" in a particular field use their experience to deduce a conclusion, rather than conducting extensive studies or experiments, and that conclusion turns out to be incorrect.

    I'd venture to say that the more we know about the various sciences, the less likely we are to have a massive deductive consensus that is incorrect. To use the nail example, I don't need to conduct a study on whether hammers work better than forks for smacking a nail into a piece of wood; my knowledge of basic principles of physics tells me that the hammer will certainly be better.

    This isn't to say it's not worth conducting studies on a lot of things. Certainly there will be many cases in which our deductions lead to false conclusions (probably because we are basing them off of faulty assumptions and don't know it), but as we learn more about the various sciences, I think the potential for that decreases over time.

    So to bring this back on topic, I don't think there's anything wrong with accepting an expert's conclusions, so long as we sort of keep in mind that there is a potential for error (and so long as we confirm that the expert really is an expert).
    Arguments from authority is never scientific; Because the philosophical (axiomatic) basis for science and the scientific method does not recognize it. It's just not how actual, falsifiable, science works. Actual science is based on the idea that there exists a separate world, and that we can through observation learn truths about this world - there's no mechanism for voting, there's no aknowledgement of authority on a given subject.

    Don't get me wrong, everything that is unscientific is not neccesarily incorrect, but that's kind of the point I originally wanted to make. Non-systematically treated data(/anecdotes) isn't scientific, and cannot support a scientific theory. Depending on what we're doing and the standard of rigour we're currently willing to accept, this is a bigger or a smaller problem.

    But a consensus among scientists/professionals is not, in itself, science.

    -This message was deviously brought to you by:
  • shrykeshryke Member of the Beast Registered User regular
    No, but a concensus among scientists is most often used when the result is a product of science that you aren't capable of understanding or don't have the time to understand.

    Argument from Authority is perfectly valid in many cases.

  • YarYar Registered User regular
    edited September 2011
    Goumindong wrote:
    The line is when you have empirical data. People probably should have trusted doctors to not wash their hands before performing births before Ignatz published. But not after.

    The history of hand-washing and doctors goes on and on, long after Ignatz. It will make you sick if you look into it further. Empirical data showed clearly for decades that doctors were refusing to wash their hands and were thus killing some of their patients, and doctors still ignored it and thought it was insulting to tell them to wash their hands. We were well into the digital age before significant headway was made into the problem, and the problem still exists today.

    Anyway, there are so many philosophical approaches to this that I don't even know what to post. For starters, I gather that there is an implied contradiction in the OP that isn't clearly stated... "This guy said that torture worked, but we know from data that it doesn't!" However, almost every time I've heard someone claim that data or studies have shown that torture doesn't work, that person is actually just reciting something they've heard. They have no actual knowledge of any such studies or data. They've just heard that there are studies and/or data, and that they showed that torture doesn't work. So what we have is expert anecdote vs. hearsay that pretends to be data.

    And that is practically inevitable. Scientific methods are great ways of ensuring a higher level of reliability and usefulness in how we discern truth. But not only is there a lot of room for error in the methods themselves, but more importantly the data and and experimental results are under constant risk of being turned into anecdote, whether it's in the variable that wasn't considered for isolation, or in how the conclusion was written, or how the media reported the results, or how it spread among people in word-of-mouth. When the rubber meets the road, most of us don't actually have a formal data set stored in our brain, we have a memory of something we heard from someone about something they read in the Post about something a scientist studied. And we think it far superior to "anecdote."

    Yar on
  • jothkijothki Registered User regular
    Yar wrote:
    Goumindong wrote:
    The line is when you have empirical data. People probably should have trusted doctors to not wash their hands before performing births before Ignatz published. But not after.

    The history of hand-washing and doctors goes on and on, long after Ignatz. It will make you sick if you look into it further. Empirical data showed clearly for decades that doctors were refusing to wash their hands and were thus killing some of their patients, and doctors still ignored it and thought it was insulting to tell them to wash their hands. We were well into the digital age before significant headway was made into the problem, and the problem still exists today.

    Anyway, there are so many philosophical approaches to this that I don't even know what to post. For starters, I gather that there is an implied contradiction in the OP that isn't clearly stated... "This guy said that torture worked, but we know from data that it doesn't!" However, almost every time I've heard someone claim that data or studies have shown that torture doesn't work, that person is actually just reciting something they've heard. They have no actual knowledge of any such studies or data. They've just heard that there are studies and/or data, and that they showed that torture doesn't work. So what we have is expert anecdote vs. hearsay that pretends to be data.

    And that is practically inevitable. Scientific methods are great ways of ensuring a higher level of reliability and usefulness in how we discern truth. But not only is there a lot of room for error in the methods themselves, but more importantly the data and and experimental results are under constant risk of being turned into anecdote, whether it's in the variable that wasn't considered for isolation, or in how the conclusion was written, or how the media reported the results, or how it spread among people in word-of-mouth. When the rubber meets the road, most of us don't actually have a formal data set stored in our brain, we have a memory of something we heard from someone about something they read in the Post about something a scientist studied. And we think it far superior to "anecdote."

    The OP said nothing about torture working. For all anyone just reading this thread (including me) knows, the guy could have said that torture doesn't work while other methods do.

  • shrykeshryke Member of the Beast Registered User regular
    No, the OP just heavily implies it.

  • YarYar Registered User regular
    shryke wrote:
    No, the OP just heavily implies it.
    Right, as I said. I gather... implied... not clearly stated...

    It was my understanding that the intent of the OP was to challenge an expert claiming that torture worked when data says that it does not. Else I'm not sure how the example in the OP relates to the topic.

  • chrisnlchrisnl Registered User regular
    Just because you agree with the conclusion doesn't mean you have to agree with the methods used to reach that conclusion.

    steam_sig.png
  • YarYar Registered User regular
    edited September 2011
    chrisnl wrote:
    Just because you agree with the conclusion doesn't mean you have to agree with the methods used to reach that conclusion.

    I agree, but what are you referring to here?

    Yar on
  • DjeetDjeet Registered User regular
    Not to derail, but I think this was the segment that spool32 mentions: http://www.npr.org/2011/09/13/140401483/an-interrogator-writes-the-inside-story-of-9-11.

    It was really quite good. Though I don't think this guy is trying to develop a science of interrogation, just relating what works in his experience (building rapport), and how the contractors that were brought in had a different ideology (breaking the prisoner down with enhanced interrogation techniques). In the segment he doesn't really comment on whether or not much actionable intelligence was gathered through enhanced interrogation techniques, but that a lot of very valuable intelligence (about KSM, Jose Padilla) was extracted before such techniques were applied, and enhanced interrogation was falsely credited.

  • Mikey CTSMikey CTS Registered User regular
    edited September 2011
    Djeet wrote:
    Not to derail, but I think this was the segment that spool32 mentions: http://www.npr.org/2011/09/13/140401483/an-interrogator-writes-the-inside-story-of-9-11.

    It was really quite good. Though I don't think this guy is trying to develop a science of interrogation, just relating what works in his experience (building rapport), and how the contractors that were brought in had a different ideology (breaking the prisoner down with enhanced interrogation techniques). In the segment he doesn't really comment on whether or not much actionable intelligence was gathered through enhanced interrogation techniques, but that a lot of very valuable intelligence (about KSM, Jose Padilla) was extracted before such techniques were applied, and enhanced interrogation was falsely credited.

    In the expanded Frontline PBS interview, he goes on to say that torture "has not provided one actionable item of intelligence". The CIA and FBI refused to deny, confirm, or even interview for Frontline on this story.

    Mikey CTS on
    // PSN: wyrd_warrior // MHW Name: Josei //
Sign In or Register to comment.