What the #overlyhonestmethods tag tells us about the nature of science

Anyone who follows me on twitter has probably noticed my obsession with a new hashtag: #Overlyhonestmethods, where scientists describe how they actually run and report their experiments.  Some of these (particularly the stats ones) are frightening, all hilarious.  They also say something important how science works--and why it works that way.  (Or perhaps I'm just rationalizing my decision to spend my morning reading tweets).

[Edit 1/11/13]: It's been several days and the hashtag is continuing as strong as ever.  There's now a twitter account devoted completely to overly honest methods, and a cute little cartoon.  There's also been a blog trying to stir up trouble by writing about the potential for backlash, where the only backlash so far is this blog post itself.  I'd like to think there's been no backlash because most people on Twitter have a healthy sense of humor and appreciation for the realities of science, rather than because not enough people have seen #overlyhonestmethods yet.

By the way, this list is only semi-complete; I've left out a few tweets that I don't understand, don't think are funny, or don't think say anything interesting.  But it's still the most complete list I've seen so far. (Other lists include Huffington Post's, The UK Telegraph's, io9's, ASBMB Today, Scientific American Blogs', Beckie Port's, Occam's Typewriter's, Vit's, and Anne Buchanan's, which includes some interesting reflections on the whole phenomenon.  Oscillatory Thoughts composed his own "confessions" in the same style for neuroimagers).  A political science blog has collected social science ones here and here.  My comments on selected tweets are in brackets.

1) Science involves laziness and human error, because it's conducted by tired, rushed, overworked human beings.  In this respect, science isn't so different from any other career.

julezyme ‏@julezyme: I have no idea how much X I used because I lost the glove I wrote it on, but here's a number that seems reasonable.#overlyhonestmethods  [This is worthy of a new hashtag, #adhdscience.  Think it would trend?]

Sara Callori ‏@SaraDoesScience: Sample was air dried at room temperature because I put them in the heater but forgot to turn it on. #overlyhonestmethods [another candidate for #adhdscience]

RT @GavinHub: I didn't label the tubes, but by looking at the results, I'm pretty sure this one is the control#OverlyHonestMethods [#adhdscience redux]

Bill Manley ‏@disgruntledbill: I used half as much enzyme because I forgot to order more #overlyhonestmethods [#adhdscience]

Dominik ‏@comfort_eagle: I had to exclude 2 subjects from further EEG analysis because I forgot to press the "record" button . #overlyhonestmethods [I'm pretty sure this #adhdscience actually happened with one participant in the experiment I'm currently analyzing, as no data got recorded from them].  

janisue jones ‏@janisuejones: We used 50μl of positive control instead of 30μl because it is going bad and we forgot to order more. #overlyhonestmethods

Jennifer Schmitt ‏@JenSchmitt618: That survey question was written that way because we finished the questionnaire at 2 am and left for the field at 7 am#overlyhonestmethods

Paul Macklin ‏@MathCancer: This figure is not really relevant to the method, but making it took way too much effort to leave out. #overlyhonestmethods

Prerana Shrestha ‏@prerana123: all behavior procedures were carried out in the order of least stressful to most stressful for the experimenter.#overlyhonestmethods

Emily Burns ‏@emilyredhead: #overlyhonestmethods a 16h incubation was carried out because that's how many hours there are between 5pm and 9am.

Alex Chase ‏@aechase: Slices were left in a formaldehyde bath for over 48 hours, because I put them in on Friday and refuse to work weekends.#overlyhonestmethods

Geeky Girl Engineer ‏@gkygirlengineer: Samples were incubated for however long our time-wasting, productivity-sucking lab meetings lasted. #overlyhonestmethods

julezyme ‏@julezyme: The reaction mix was incubated for as long as it took me to go to the toilet and check my email. #overlyhonestmethods

Kirkwood Lab @ UOIT ‏@KirkwoodLab: Study lake selection was based on phys-chem criteria and the absence of annoying, barking dogs from local cottages#overlyhonestmethods

Dr Caz ‏@fruitflyCaz: I copy-pasted most of this methodology section from my previous paper on a similar topic #OverlyHonestMethods  [I've done this before...]

Melanie Slabinski ‏@MissSlabby: “@The_Shed: After the biological sample was dropped on the floor, the 3 second rule was applied. #overlyhonestmethods”

Scott Paquette ‏@scottpaq: Only papers available online were cited in the lit review. I haven't found the campus library yet. #overlyhonestmethods

Will Morgan@willtmorgan: The instrument was inoperable during this period because somebody hit one of the switches with their arse.#overlyhonestmethods

Dr. Twittenheimer@DrTwittenheimer: A modified protocol was implemented because a certain graduate student seems unable to follow simple instructions.#overlyhonestmethods

Emma@atomselectrons: This dye was selected because the bottle was within reach#overlyhonestmethods

dr leigh@dr_leigh: incubation lasted three days because this is how long the undergrad forgot the experiment in the fridge #overlyhonestmethods

Blas M. Benito ‏@BlasMBenito: We used method A instead of method B because my cat pooped on method B #overlyhonestmethods

Georgina_Milne ‏@Georgina_Milne: "The sample was biased because the plants were growing in a thorn bush and I didn't want to stick my hand in too far"#overlyhonestmethods

Adam J. Mosa ‏@MosaFacts: we studied the effects of the rhinovirus on adult neurogenesis because I sneezed inside the sterile hood #overlyhonestmethods

Jamie Quinn ‏@jmquinn: No SES [socioeconomic status] data were available for the sample, as the testers forgot to ask. #overlyhonestmethods

[And on a similar note...] @peds_id_doc: #overlyhonestmethods Side effects were limited 2 headache & nausea, because they were the only ones we asked about.

overlyhonestmethods ‏@overlyhnstmthds: the sample incubated for 20 extra hours in the oven because i said "4hrs" and the undergrad heard "24hrs" #overlyhonestmethods

Allie Kosela ‏@AllieKosela: Can you account for your sources of error? Yes, undergrads #overlyhonestmethods

Louise Saul ‏@AikiSaul: incubation times vary depending on how long a coffee/lunch break is needed #overlyhonestmethods

David Clarke ‏@Mozarticus: the crystal was grown via slow evaporation of dmso.... or i couldnt be bothered cleaning the nmr tube for 3 mths #overlyhonestmethods

Adam Shellard ‏@AdamShellard: Statistical analysis was carried out on excel #overlyhonestmethods  [included here because this basically reflects laziness].

Michelle Turco ‏@chelleomi: Crystals formed after 5 months... because I forgot to check between month 1 & 5 #overlyhonestmethods

Jamie Quinn ‏@jmquinn: we don't have the code used for the analyses because one of us is too good to use code and instead uses dropdown menus#overlyhonestmethods

Lija Greenseid ‏@Lija27: The interviews were transcribed verbatim, except for when I babbled about things that were totally off topic #overlyhonestmethods

Alex ‏@voting: Standard room pressure was assumed because nobody could be bothered to fight with the barometer. #overlyhonestmethods

Mariza de Souza ‏@marizageek: Further investigation could have been carried out if I hadn't knocked the apparatus over and misaligned it #overlyhonestmethods

Gregor Miller ‏@radianity: The experiment was stopped after the postdoc collapsed in tears from the realisation that he'll never become faculty#overlyhonestmethods

Alex ‏@voting: This research topic was chosen because I could essentially just rehash various pieces of work from the past few years.#overlyhonestmethods

@paulcoxon: collaborators were chosen by the quality of their research and if their labs are near a beach#overlyhonestmethods

Kane_WMC_Lab ‏@Kane_WMC_Lab: We would have reported stimulus size in terms of visual angle, rather than cm, but we always forget how to do that!#overlyhonestmethods

Alix Conway ‏@conwizzle42: Sites were randomly selected by how far I felt like walking that day#overlyhonestmethods

Sasha Feoktistov ‏@thisisdemocracy: "we adapted prev. est. protocol for our novel experiments" we had no clue what we were doing so we copied smarter ppl#overlyhonestmethods

@FortRory: All mathematical calculation contained in the following paper was outsourced. #OverlyhonestMethods

Helen Lee Lin, Ph.D. ‏@helenleelin: #overlyhonestmethods We only cite papers that are available as full-text online because going to the library and Xeroxing is hard.

Donaldo D. Canales ‏@psycho_ddc: Mostly citing meta-analyses in this manuscript because I don't have time to read 500+ papers #overlyhonestmethods

Teri Anne Stanley ‏@TeriAnneStanley: We used Female Rats because the technician checked the wrong box on the order form #overlyhonestmethods

Amrita Neelakantan ‏@amster04: rodent traps laid only on the ground as too many volunteers fell off the trees while trying to set them up in branches #overlyhonestmethods

 @sethzenz: Additional variables were not considered because everyone involved is tired of working on this paper.#overlyhonestmethods

Cool Science Videos ‏@CoolScienceVids: This method was essentially described by Jones et al (1964), but we didn't notice it and so are publishing it anyway#overlyhonestmethods

@swift528491 The tube was thawed at 37 degrees celsius". I tucked it into my bra cuz I was running late#overlyhonestmethods

A.C. Papageorgiou ‏@ACPapageorgiou: We sampled 7 populations of plant X. Two more were not sampled, because there is no road crossing their habitat#overlyhonestmethods

Arnaud Garcon ‏@ArnaudGarcon: Sample was shaken vigorously (dropped on the floor)#overlyhonestmethods

Marc Maxson ‏@marcmaxson: we didn't test as many clams as oysters because we're pretty sure someone found the samples and ate them #overlyhonestmethods

JörgB ‏@BldyJck: We tried to measure only axons.. but honestly, it's f**king hard to distinguish them from dendrites with our microscope#OverlyhonestMethods [I've wondered about this sort of thing].

Xylotheca ‏@Xylotheca: Measurements were accurate to 6 places... + or - error caused by vibration due to people walking past. #overlyhonestmethods [A perfectly controlled environment, sadly, is almost never possible.  I work with human subjects and would love to be able to make measurements accurate to 6 places plus or minus anything].

Jacquelyn Shelton ‏@jacquelyshelton: we published in journal B because the more prestigious journal A asked for more experiments and we were lazy#overlyhonestMethods

Bernadette Wright ‏@WrightEval: The site visit locations were chosen because we thought they would make fun vacation spots #overlyhonestmethods

[And of course....]: NCSU Linguistics Lab ‏@NCSULinguistics: "Analysis was delayed because the grad students got preoccupied coming up with #overlyhonestmethod tweets" #overlyhonestmethods

2) Science involves limited resources, primarily time and money. This may seem obvious but it prevents us from conducting optimal studies by limiting the number of participants we can test, the range of techniques we can employ, etc.

Kel Hinkle ‏@KelHinkle: Used our kids as subjects because it didn't require IRB approval. #overlyhonestmethods  [I know researchers who do this, and who also "volunteer" their kids for any experiment at the university for which they might be eligible.  Someone once suggested I look for professors' kids to pilot my honors thesis for similar reasons].  

@profserious: I used students as subjects because rats are expensive and you get too attached to them#overlyhonestmethods

James Grecian ‏@JamesGrecian: species datasets were really selected because they were all I could get hold of 1 month before my PhD thesis deadline#overlyhonestmethods

Elise ‏@eliseisawkward: We used 4 mice per sample because that's how many we had left over from the last experiment. #overlyhonestmethods

Steven Graves ‏@revwhitewolf: RT @paulcoxon The spectrometer was made by a company which no longer exists, in a country which also, no longer exists#overlyhonestmethods

Viet Le ‏@AmasianV: DNA was purified using either Promega or Qiagen kits...depending on which company was offering the better discount.#overlyhonestmethods

Miguel_delF ‏@yo_Antitwitter: We didn't read half (in real any) of  papers we cite because to print them is to expensive #overlyhonestmethods [To better understand the costs, scientists can cite anywhere from dozens to over a hundred papers, ranging from 2 or 3 to 40 or 50 pages depending on the type of paper and the discipline].

Anne Osterrieder@AnneOsterrieder: Buffers were prepared with the stock solutions inherited from the PhD student who left the lab a couple of years ago.#overlyhonestmethods

thecellularscale@cellularscale: We used software X because it is free.#overlyhonestmethods

Erin Kaufman ‏@ELKaufman1: Sample was biased b/c patients were at 3 diff hospitals, gas prices were over $4 and I wasn't getting reimbursed.#overlyhonestmethods

Christopher ‏@topher_hunter: 'Future studies will explore...' No they won't, because the funding ran out. #overlyhonestmethods

Allison D. Gamblin ‏@SVAllie: RT @Keated #overlyhonestmethods I couldn't access this paper behind the pay wall, but the abstract seemed like it was in the right area.

Sarah Grossman ‏@wanna_disco: the carefully crafted augment why a 2ndary analysis is the most suitable method=I dont have time to collect my own data#overlyhonestmethods

@davidkozin: N was reduced to save my marriage.#overlyhonestmethods

Ana B ‏@anabstar: With a 0 dollar budget and no grant, we were only able to find enough free coffee to obtain 50 data points, not 500.#overlyhonestmethods

Research Girl ‏@researchgirlca: "We used this approach because it was the only one we could afford with the end of our grant money." @grantsfacilitat#overlyhonestmethods

Jamie Murray ‏@JamieMurray86: We used ERPs because we are too poor to afford an MRI scanner#overlyhonestmethods [It's funny because it's true].

@researchremix: We used database X because it is the only one our library has a subscription to#overlyhonestmethods

@devillesylvain Sylvain Deville: we didn't read half of the papers we cite because they are behind a paywall #overlyhonestmethods #OA

Dr.Angela Pfammatter ‏@APfam: We chose these questionnaires because they didn't cost money#overlyhonestmethods

James Baker ‏@jbkr_: 0.68mg of reagent X was used because that was all that was left & we didn't want to order another kilo of the stuff. #OverlyHonestMethods

GK Lefkowitz ‏@GKLePhD: @upmicblog On a related note, we only repeated this once because we used up the free sample and PI wouldn't buy more #overlyhonestmethods

Piv Gopalasingam ‏@Cascade21:  Light-sensitive compounds were stored in a card box surrounded by tin foil 'cos we couldn't afford a fancy container#overlyhonestmethods

@NobyLeong: We used this chemical because the other one was too expensive #overlyhonestmethods” 

2b) Because of the expense of supplies, scientists often borrow them from each other--and have to take on faith that these supplies really are what they're supposed to be.

julezyme ‏@julezyme: The antibody was probably from Sigma, but I'm not sure because I just got an aliquot of it from someone down the hall.#overlyhonestmethods

Carl Carruthers@ProteinWrangler: Plasmids were a gracious gift from the Miser lab after many emails, phone calls, & drunken reminders at conferences.#overlyhonestmethods

dr leigh ‏@dr_leigh: they kinda just send us these rats, we take their word on what they actually are. #overlyhonestmethods #sigh

Chris Gallop ‏@scoutman21: Most reagents and solvents were begged/stolen/borrowed from neighbouring labs and further purified with no real care#overlyhonestmethods

3) Scientists use equipment or programming code that breaks or otherwise doesn't function as intended--generally at the most inconvenient times. The vicissitudes of weather and dangerous animals fill the same function for field/nature researchers.

Steven Graves ‏@revwhitewolf: RT @Camaceron1 pH remained constant throughout entire experiment (pH meter ran out of battery half way through)#overlyhonestmethods

Julian Rubino ‏@rubinojg: Oh there's a problem with that pH meter? Not like I used it to analyze all my samples or anything... #overlyhonestmethods

∆ Garcia-Castellanos ‏@danigeos: I can't reproduce my own old results because my code 'evolved' during the years #OverlyHonestMethods

Ian Holmes@ianholmes: You can download our code from the URL supplied. Good luck downloading the only postdoc who can get it to run, though#overlyhonestmethods

Dr Caz ‏@fruitflyCaz: We randomly selected our study site but when we arrived a lion had got there first so we moved to an arbitrary spot#overlyhonestmethods

Anthony (Tony)Martin ‏@Ichnologist: Only measured smaller alligator tracks at field site because we ran away whenever big tracks encountered. #overlyhonestmethods#ichnology

Alice Proverbio ‏@AliceProverbio: Stimulus ISI [interstimulus interval, or time between presentation of 2 different stimuli] was set at 2.5 secs, so that the PC wouldn't keep crashing #overlyhonestmethods

Maria Sundaram ‏@mariasundaram: Selected relevant sections of the forebrain were mounted on slides because we couldn't get the rest of them to stick.#overlyhonestmethods

Angela ‏@zolacnomiko: Subplot locations are approximate, because wild animals pulled up our plot markers and chewed on them. #OverlyHonestMethods#ecology

Amanda Warr ‏@Mandabirdstweet: Bird counts were not carried out on this day because there was a lion at the site and I didn't want to die. #overlyhonestmethods

Donaldo D. Canales ‏@psycho_ddc: Performed studies on a junk computer made in early 90s because PI doesn't trust brand new lab equiptment #overlyhonestmethods

Sam Urmy ‏@ElOceanografo: "Data were not collected during these 2 weeks because a gopher chewed into the buried power cable and exploded."#overlyhonestmethods [That one has to be true, because you just can't make up those details.  Also, "The Exploding Gophers" might make a good band name].

4) Science involves limited numbers of subjects, which are often uncooperative.  There's good reasons for the much-maligned "convenience samples."

Madhusudan Katti ‏@leafwarbler: We surveyed a sample of American college undergrads and extrapolated to all humans, because... #evolutionarypsychology#overlyhonestmethods  [Pretty much all American psychology labs do this unless they collaborate with labs in other countries or self-consciously do "cross-cultural research"].

Jon Brock ‏@DrBrocktagon: Participants were 19 undergraduate students aged 18 to 23 plus my mum, who's 57 #overlyhonestmethods  [Apparently undergrads do actually run their mom in studies, and professors seem to think it's a good thing.  So long as it's just undergrads, no problem].

a small bear ‏@hoorayabear: There were 14 participants in this study because that is how many people I could convince to take part. #overlyhonestmethods

Douglas ‏@Taikonaut: the Qs targeted differences in NZ, Scottish and Midwestern dialects because those were the subjects I had access to#OverlyHonestMethods

LNT ‏@LNT89: Mice were treated with 200uL subcutaneously. Apart from that wriggly one, not sure how much he got... 100uL? #overlyhonestmethods

Barbara Bell ‏@baulabarb: we modified the marking/identification method for use with snapping turtles because we wanted to keep our fingers.#overlyhonestmethods

Dr Caz ‏@fruitflyCaz: We're not going to mention how many marshmallows were used to bribe the monkeys to do our bidding #overlyhonestmethods

Lucy Wyatt ‏@LinkaNeo: Research organisms weren't reproducing as fast as I need them to so I tried playing Barry White while in lab. #overlyhonestmethods

Barbara Bell ‏@baulabarb: we didn't weigh the nesting female leatherback because none of us could lift 1500lb #overlyhonestmethods

Brett Schaerer ‏@astrobug75: samples were biased towards the sedentary/dead insects because the live ones flew away #overlyhonestmethods

Jon Brock ‏@DrBrocktagon: Four participants completed 5000 trials each. Coincidentally, their initials are the same as those of the authors#overlyhonestmethods  [In some fields, such as vision science, it's actually accepted for study authors to be subjects, on the (questionable) assumption that everyone sees pretty much the same way.  However, they're generally not the only participants].

Gregor Miller ‏@radianity: We assume our results apply generally, however we were only able to test on participants working on advanced degrees#overlyhonestmethods

eastcoastkids ‏@eastcoastkids: Participants were recruited from the first author's statistics class in exchange for baked goods. #OverlyHonestMethods

Gregor Miller ‏@radianity: Participants were selected based on their level of desperation for free chocolate bars #overlyhonestmethods

Robyn Powell ‏@DontDoDinosaurs: Participants selected from poor university students who will do anything we ask for ten bucks #overlyhonestmethods

Jessica ‏@BennyandMaia: Test subjects were freshman males. Only volunteered after I dressed slutty. Nobody volunteers to smell shit. #overlyhonestmethods

Emily ‏@mosaicofminds: We used N subjects because it was the most I could find within the time allotted to complete study #overlyhonestmethods

@Crommunist Human subjects were recruited from a pool consisting of BIO 205-A and BIO 205-B #OverlyHonestMethods

Ian Beatty ‏@idbeatty: We sampled weekly, then biweekly, then monthly because I got tired of nagging participants to complete their surveys.#overlyhonestmethods

Angela ‏@zolacnomiko: Blood samples were not taken from specimens 173, 462, and 535 because they escaped. Oops. #OverlyHonestMethods

Anne Cregan ‏@AnneCregan: Psychometric test data for Subject 100 was omitted bc he got sick of answering questions and just kept hitting return#overlyhonestmethods [There's always one of these.  In really boring experiments, particularly ones involving MRI scanners, some even fall asleep in the middle].

NCSU Linguistics Lab ‏@NCSULinguistics: "The subject did not wish to repeat the experiment because I asked her out and shit got weird" #overlyhonestmethods

Yani Malai Abdullah ‏@YaniMalai: In this study we had a LARGE number of ASD participants (n=10); matched on the enthusiasm of their parents.#overlyhonestmethods [This is sort of two-in-one as for most types of studies, ten participants is not a large number].

Dr. Nyk ‏@DrKushalNaik: #overlyhonestmethods All patients gave consent because we removed the 'I do not give consent.' checkbox from the informed consent document. [This one crosses the line into an outright misdeed, I think].

5) Science hasn't stopped relying on serendipity since penicillin was discovered in the mold growing on a scientist's old sandwich left lying around the lab.  Good scientists take advantage of this.

[This says it all]: Georgina ‏@geocorner: "Found these results totally by accident cause I mixed up 2 experiments. Formulated cool hypothesis. E voila. Nature.#overlyhonestmethods"

Jessica Wright ‏@Jessrw_NYC: We discovered binding to a heat shock protein because we overgrew our culture and did the experiment anyway. #overlyhonestmethods#truestory

NatC ‏@SciTriGrrl: We used the reagent in Experiment 1 due to a lazy mistake by an undergrad. But hey! Serendipity works! @dr_leigh#overlyhonestmethods

Marcus G ‏@NoMstn: we accidently discovered figure 7 and then filled in figures 1-6 to that it would all make sense in the paper #overlyhonestmethods

Keated ‏@Keated: #overlyhonestmethods This interesting result turned up in my last ever sweep, so I have no idea whether it's a fluke or not

Mason A. Rouser ‏@MasonRouser: Methodology discovered when paper accidentally dropped in experiment and it selectively removed excess reagent only#overlyhonestmethods

Tal Yarkoni @talyarkoni: we discovered the anxiogenic properties of our new drug accidentally, while trying to fuck with labmate Steve's coffee#overlyhonestmethods

 6) Scientists love clever wordings, puns, fun journal titles, cool noises, pretty graphs, and cute animals as much as the rest of us.  Humor and aesthetics are an important, if undervalued, motivator in science.

Claire Gomer ‏@ClaireGomer: I chose the study species due to their complex social interactions, and because they're dead cute. #overlyhonestmethods

James C. Kaufman ‏@JamesKaufman: I decided to study whether poets die younger than other writers so I could publish in a journal called Death Studies.#overlyhonestmethods

Rebecca Gill ‏@DrRebeccaGill: Wrote this paper on the effect of petitioner gender on success just to see the variable "petsex" in print. #overlyhonestmethods

isa ‏@isaontheway: Like how I chose the title of my thesis because I wanted every word to start with a "C". 

#overlyhonestmethods

Macrophagic ‏@Macrophagic: Reagent was selected because its name made us giggle.#OverlyHonestMethods

Julie Dugdale ‏@JulieDugdale: #overlyhonestmethods We included these graphs because, well, they look serious and quite pretty.

Cheng H. Lee ‏@chenghlee: MT @drugmonkeyblog 2 days to isolate the protein, 5 weeks to generate the hilarious double-entendre name for the gene.#overlyhonestmethods

@benosaka Blood samples were spun at 1500rpm because the centrifuge made a scary noise at higher speeds.#OverlyHonestMethods

Colleen ‏@straightedgersx: When observing baby chickens (chicks) behavior we named them Chicken Little, Chicken Pot Pie & Chicken Ala King#overlyhonestmethods

Nash Turley ‏@nashturley: I picked Solanum as a study species because I had worked with it before and I think it is pretty #overlyhonestmethods

ootastic ‏@ootastic: It may not be the most interesting data, but it makes the prettiest picture. #overlyhonestmethods

Pasquale ‏@mansquatchie: I chose to use the particular solvent because it smelled better than the others. #overlyhonestmethods

Nils Hoppe ‏@nilshoppe: "Team brainstormed excellent project acronym and then built grant proposal around it" #overlyhonestmethods

Rachel Beattie ‏@RachelLBeattie: We chose to study preschoolers because that gave us a really good excuse to buy lots of stickers #overlyhonestmethods

7) Scientists often don't understand statistics very well, and use procedures that aren't appropriate for the data being analyzed.

Cheng H. Lee ‏@chenghlee: Our data & conclusions are right b/c we have a p=0.049; competitor is totally wrong b/c they have a p=0.051. #OverlyHonestMethods

Nicolas Ward ‏@UltraNurd: We fit a normal distribution even though we know the data is multimodal, because it's easier to calculate in Excel.#overlyhonestmethods

Marga Corameta ‏@MargaCorameta: We applied this statistical analysis method because is the only one we know how to use in SPSS #overlyhonestmethods  [For those unfamiliar with the program, SPSS is a program involving no coding whatsoever, just drop-down menus...with a confusing array of different procedures you can use, whose names and attribute don't necessarily perfectly match their equivalents in the stats textbooks.  However, this is actually one of the easiest methods you can use to analyze data, so this confession is pretty sad].

Blas M. Benito ‏@BlasMBenito: Our dataset was non-normal, heterocedastic, and almost stochastic, but what the hell, we used linear regression#overlyhonestmethods

Scott Paquette ‏@scottpaq: A qualitative data approach was chosen since our graduate stats course has permanently scared us away from numbers#overlyhonestmethods

Jessica Bregant ‏@JessieBregant: Tests were chosen after thorough analytical review. We ran everything we could make SPSS do, then reviewed for p<.05.#overlyhonestmethods

miss_sapphyre ‏@miss_sapphyre: This statistical method was chosen because it's the only one my MD coauthors understand. #overlyhonestmethods

8) Scientists aren't always honest.  They'll bend the procedure or, particularly, the stats, to make the results look as good as possible.

Ian Bushfield ‏@ibushfield: This fitting method was used as it made my results work.#overlyhonestmethods

Brent Neal ‏@BrentN: Bayesian methods were used to evaluate data. Our choice of prior [assumptions] was made in order to give the right answer. #overlyhonestmethods

Dr Krystal ‏@dr_krystal: Data was analysed using Fisher's Exact t-test because it gave us a statistically significant p-value #overlyhonestmethods

Nik Papageorgiou ‏@upmicblog: n=3 for positive data. The other n=16 that didn't work were "experimental error". #overlyhonestmethods

Dr Caz ‏@fruitflyCaz: This methodology is full of jargon so you won't work out we just tweaked things until they worked #OverlyHonestMethods

Adam Watts ‏@fireecologist: Four outliers were excluded from analysis, because the third author thought they "messed with our R-squared." #overlyhonestmethods  [Outliers are people whose results were either way higher or way lower than the rest of the group.  The third author is complaining that the strength of a correlational analysis has been reduced by outliers].

Shinobi ‏@shinobi42: We removed some of this data from our sample because our model looks better without it. #overlyhonestmethods

Nikki Botting ‏@NicolaBotting: Finally a power calculation is included using some random values because no reviewer wlll actually attempt to check this #overlyhonestmethods

Nikki Botting ‏@NicolaBotting: A substantial number of participants were then excluded on relatively random grounds so that results match predictions #overlyhonestmethods

Adrian O'Hara ‏@gingerbiochem: The experiment was run at least 3 times - we kept trying until we got the answer we wanted/fitted the hypothesis #overlyhonestmethods  [One hopes they also did things to improve their procedure or data analysis each of those times, too].

Iain Gilmour ‏@Iain_Gilmour: Once got advice to plot data that correlated as large solid squares, non-correlated data as faint open circles. #overlyhonestmethods

Nial Wheate ‏@NialWheate: Only a section of the NMR spectrum is shown, because there was evidence of small impurities in the other sections #overlyhonestmethods

Guzmán Sánchez ‏@Guzmansanch: The experiment was relentlessly repeated until it supported our hypothesis #overlyhonestmethods


9) Scientists follow conventions, too.  They claim that this allows them to compare their results with others.  True, but I think it also reflects that scientists are subject to the same social pressures as the rest of us.

dr leigh@dr_leigh: we used [program] because doesn’t everyone else? #overlyhonestmethods

Emily ‏@mosaicofminds: We chose these parameters to filter & baseline correct ERP data because all the other labs do it (we don't know why).#overlyhonestmethods

Randy McLain ‏@randoramma: Used [method] as that's what we've always done, the only way we know to do it, and God help us if we change anything!#overlyhonestmethods

@droenn: I cited this paper because everyone else has cited it, though noone has ever seen an actual copy#overlyhonestmethods" so true lol

Patricia lockwood ‏@thepsychologist: We used this questionnaire measure because everyone else does#overlyhonestmethods

Michelle Dawson ‏@autismcrisis: We know this autism severity measure doesn't work, but a bunch of other people used it so we will too #overlyhonestmethods but true story

Jon Brock ‏@DrBrocktagon: #autism #overlyhonestmethods Groups were matched on verbal IQ and nonverbal IQ. We're not sure why but this is what everyone else does.  

10) Reviewers can be more annoying than helpful, and scientists invest a good deal of time and energy in placating them and stroking their egos.

Gregor Miller ‏@radianity: Half of the article's literature review cites irrelevant research because one of those authors might review the paper#overlyhonestmethods

Are you serious ‏@Enter_Chinaman: The following section contains marginally relevant citations by people who will most likely review our paper. #overlyhonestmethods

AprilPawluk ‏@AprilPawluk: "This utterly useless and redundant experiment was done thanks to Reviewer #3" #overlyhonestmethods

Jon Brock ‏@DrBrocktagon: Further analyses were performed to appease Reviewer 3 #overlyhonestmethods

Kane_WMC_Lab ‏@Kane_WMC_Lab: We thank reviewer 3 for suggesting this alternative analysis, but we really don't mean it. #overlyhonestmethods

Pepijn Vemer ‏@PepijnVemer: I added this reference during revision, because I believe one of the authors is a reviewer. #overlyhonestmethods

Dr Lyndon Walker ‏@DrLyndonWalker: My references seem out of date because this is the 10th journal I have submitted the article to #overlyhonestmethods

Adam Sowalsky ‏@sowalsky: We asked our competitor to be a co-author so he would be ineligible to review the paper #overlyhonestmethods

10) Scientists face extreme pressures to publish.  Sometimes, what's publishable lines up well with good science.  Sometimes, it...doesn't.

Scott Paquette ‏@scottpaq: The motivation for this paper was to increase our citation count, pre-tenure. #overlyhonestmethods

Joe Coules ‏@Joe_Coules: We was going to run a replication but nobody funds or publishes replications. So we did this instead #OverlyhonestMethods

Rob Ford ‏@robfordmancs: We used Bayesian latent variable multilevel regression because only flashy new methods get published in this journal#overlyhonestmethods

Patricia lockwood ‏@thepsychologist: We changed the title to an undescriptive pun as it was more likely to get published in a high impact journal #overlyhonestmethods

biochem belle@biochembelle: We decided to use Technique Y because it's new and sexy, plus hot and cool. And because we could. #overlyhonestmethods

Dr. Paul ‏@unMighty: Data not shown because I've reached this journal's limit on figures.#overlyhonestmethods

Blas M. Benito ‏@BlasMBenito: We decided to use the method X two because it was developed by the editor #overlyhonestmethods

Nikki Botting ‏@NicolaBotting: In this article, a large number of self citations are included,not bcs they are useful but to raise the authors H-index#overlyhonestmethods

Michelle Turco ‏@chelleomi: Even though the tech did the benchwork & the prof did the analysis, the grad student got 1st author so he can graduate#overlyhonestmethods

Crommunist ‏@Crommunist: This test was chosen because my supervisor is trying to pioneer it as part of her tenure bid #OverlyHonestMethods

Jon Brock ‏@DrBrocktagon: #autism #overlyhonestmethods The experiment took 30 mins. We also spent 4 hrs rediagnosing participants using ADOS & ADI so we could publish

Felicity Callard ‏@felicitycallard: Methods were creatively embellishd by PI, who said that if we accurately reported what we'd done, wd never be published #overlyhonestmethods

overlyhonestmethods ‏@overlyhnstmthds: submitted to a new journal b/c editors wanted more experiments and the only one who can do it was an undergrad who left #overlyhonestmethods

 Régis Alenda ‏@regis_alenda: My 16-years old son hadn't actualy done much for this paper, but he is listed as an author so that it'll make the news.#overlyhonestmethods

Paul Bretherton ‏@NeuroConsulting: The real expert on this paper was the tech who showed us all how to do it but isnt an author because they're the tech#overlyhonestmethods

11) The principal investigator or PI (professor who runs the lab and supervises studies) generally does not perform the experiments herself, and probably isn't the most knowledgeable person in the lab when it comes to practical methodology.  Most labs manage to function just fine anyway.

Neuroskeptic ‏@Neuro_Skeptic: The first author didn't write this Methods section and doesn't understand half of it. #overlyhonestmethods

Dr Caz ‏@fruitflyCaz: Our first author didn't conduct this research so this methodology isn't actually accurate #OverlyHonestMethods

Julie Scott ‏@jcscottie09: The lead researcher on the project spent 0 time in the field and has no idea where the study sites were #overlyhonestmethods

Jen ‏@JenDVM2012: The first author had no part in data collection since he can't be in the lab without supervision #overlyhonestmethods

Neuroskeptic ‏@Neuro_Skeptic: The first author didn't write this Methods section and doesn't understand half of it. #overlyhonestmethods

12) Part of science is not knowing what results you're going to get before you start, so you often get surprising results or ones that don't make sense.  You usually have to rethink your hypotheses, redo parts of your procedure, or otherwise improvise.  In my opinion, this flexibility is a sign of good science, but for some reason, journals want to publish articles that suggest the results and explanation were perfectly planned all along.  This creates a misleading picture of how science works--hence, the existence of an #overlyhonestmethods tag.

dr leigh ‏@dr_leigh: we did experiment 2 because we didn't know what the fuck to make of experiment 1 #overlyhonestmethods [I bet this happens sometimes in every lab].

[This point is actually quite profound.] Christina Palka ‏@Luckiturtle: successful experiments are actually failed experiements combined with a new hypothesis #overlyhonestmethods.

[And on the same lines]: Duncan Hill ‏@drduncanhill: #overlyhonestmethods a common route to new avenues of research is getting an experiment wrong, seeing something interesting and studying it

Dr Caz ‏@fruitflyCaz: This isn't the study we intended on doing but the results supported this angle better #OverlyHonestMethods

Michael Hale ‏@Hey_MickeyH: The methods were described in the 3rd person passive voice to create an illusion that everything went according to plan#overlyhonestmethods

Pavito ‏@pdodonov: We performed these fancy analyses because our original results (and hypotheses) just weren't interesting enough.#OverlyHonestMethods

@ProfLikeSubst Proflikesubstance: Though it all sounds tight now, we had no sweet fucking clue what we were doing when this started#overlyhonestmethods

Stuart R Jackson ‏@StuartRJackson: #overlyhonestmethods Results varied and we recommend further investigation (we have no idea what the hell just happened).

@dingram: "Empirically derived settings" sounds so much better than "we twiddled with the knobs until it worked"

[Here's the cynical way of looking at this situation]: Skeptical ‏@RC_Skeptic: Our Introduction is pure post-hoc bullshit, but it tells a nice story, doesn't it? #overlyhonestmethods

A.C. Papageorgiou ‏@ACPapageorgiou: The "aim of the study" was defined after writting the "discussion" part of this manuscript... #overlyhonestmethods

nrayko ‏@nrayko: The literature review was written after all the research was completed and analyzed so the results would make sense.#overlyhonestmethods  [Doesn't everyone do this?  No, seriously, don't they? You always end up adding new literature you would never have thought to include after finding results anyway, so why not just write the whole literature review after running the experiment?].

One blogger has argued that the errors discussed here result from perverse incentives in science.  They don't help, for sure, and probably contribute to #7-11.  But even with the best incentives, human error and laziness, technological failure, limited resources, and the sheer serendipity of science will still exist.  #Overlyhonestmethods will always be with us.