Protocol evolution or bad game of telephone

Nov 21 2012 Published by under [Education&Careers]

I am not in the lab everyday. In fact, I am basically never in my own lab with the purpose of doing science. We can argue over whether that is the most effective way for things to get done, but it's the reality of my situation. As a result, I'm not there to see what people are up to all the time.

Once in a while I'll either get methods text for a paper or see something in lab meeting and wonder "why did you do it that way?". Inevitably the following conversation will take place:

Student: "The normal protocol wasn't working so I found a workaround."

Me: "Did you test the effectiveness of your new protocol against other variants of the conditions you changed?"

Student: "No, this worked so I went with it."

Me: "Did you try and figure out what wasn't working for you with the old protocol."

Student: "Um, not really..."

Me: "Are you the only one using the changed protocol?"

Student: "No, I gave it to Undergrad X and Y."

I don't mind people trying to improve protocols in a systematic way, nor do I discourage people from troubleshooting lab issues. However, changing established protocols in a haphazard way and propagating those changes to others is the stuff that drives PIs crazy. Not only is it bad science, but there is enormous potential for unanticipated downstream issues. If you alter the concentration of reagent A in Part 1 of a protocol and it effects downstream process 4, it is entirely possible that a lot of time will be spent troubleshooting process 4 before the change to reagent A is mentioned and the link is made.

Even minor "tweaks" to a protocol have a way of propagating until one day the PI goes to read the methods for Standard Procedure One and it turns out that the game of lab telephone has turned it into Stendard Progressional Ounce.

Lab protocols can be changed, but systematically, and with everyone in the loop so that people are aware of the changes and all working from the same playbook.

15 responses so far

  • DrugMonkey says:

    Amen brother.

  • DJMH says:

    Equally fun, when someone changes common lab acquisition/analysis code and the change is propagated, with or without everyone's knowledge. Easier to fix than a reagent change, but harder to detect.

  • I don't even ask about any of this shitte anymore. I sleep better that way.

  • Alex says:

    CPP, let me tell you about the time when I didn't check a few lines of data analysis code because a student thought it would be easier to switch around the order of some calculations. We shared the data analysis code, somebody else checked those lines, and I wound up writing an erratum.

  • darchole says:

    You just have to make sure the people running the experiments have actually seen the written protocols. There is stuff I haven't even seen, and we have to have a written and approved protocol because of the various alaphabet agencies overseeing the research (i.e. NRC, IACUC, DEA, OSHA even). I know there is protocol drift, but I've got no idea what the protocol even IS. (I asked when I first started...gave up 5+ years later of ever seeing it, or having any input on how it's written.)

  • We have procedures in place to prevent those kinds of mistakes, but there is no way that a PI running any kind of diverse interdisciplinary research program can keep her hands on that level of experimental minutiae. Maybe if your lab has like three people in it and they all use exactly the same technical approaches, then you can be aware of that level of detail. But any bigger than that, and you have no choice but to trust people, and build in processes to verify.

  • Alex says:

    The procedure that we have in place is that all of the students on that aspect of the project have to look at everyone else's code, and the students have to submit me an outline of the code. I don't check every semicolon and bracket (usually) but they have to give me an outline of the procedure. I also require them to run it with a whole bunch of different inputs corresponding to cases where I know what the output has to be. It has to pass those tests.

    The problem is that they didn't translate between outline and procedure in the appropriate way, the mistake was small enough that it still passed the tests, and the other student on the project lacked the confidence to raise a question since I had signed off on the outline.

    The fix now is that everybody gets to hear the story about the erratum, and they are told that asking questions, even questions about things that I've already inspected, is way better than writing an erratum. To demonstrate my point, I give examples of people who faced no ill consequences for bringing issues to my attention. That, and I do a lot more spot-checking. I can't check everything (I agree with you that that level of detail is impossible), but I check things at random to see if they understand the outline as well as they think they understand it. If not, we have a teaching moment.

  • Alex says:

    Oh, and I don't tell them what the test cases are while they're designing it. I don't want them designing around the tests, because then they might focus on the special cases rather than the general procedure.

  • zy says:

    hmm. As a grad student I've done this. It's a fine line to walk when you are becoming more independent (because that's part of learning to be a researcher, you're PI is not around, etc). I can't email my PI every time I change something. And sometimes I make choices that don't work. When possible, I now go online and read about why different reagents are in a protocol. What does salt do in PCR? Why would you add more, or less? But for some things, like kits, it's impossible to know what Solution B does, or why you need X amount. In the past I changed things around a bit, until I wrote my first paper. My advisor explained to me that the easiest methods to write were the simplets. Always change protocols with the reviewers in mind. This has been a good rule of thumb. But it's deffinitally something that needed to be communicated to me explicitly. Otherwise my own insecurity mixed with a desire not to email my prof with every ninny problem would have led to a lot of creative modifications.

  • [...] blogs, especially for the name “Spandrel Shop” which so applies to this post) has a great post up about protocol changes, showing the dangers of Standard Procedure One becoming Stendard Progressional Ounce. The idea that [...]

  • postdoc says:

    I'm in computational biology, and one thing that has troubled me is that almost no one has ever looked at my code during development, although I post all of it online at press. It has been the source of much worry. I have found bugs after months of work and taken an extremely long time to develop any confidence in my results for fear that mistakes were still lurking. I almost always try to test my code on special cases, as Alex suggested above, but I'm sure others could come up with better ones. I'm also not a programmer by training, and it has taken me an indecently long time to pick up efficient algorithms and good habits, like version control.

    I'm now in the position of starting my own lab, and one of my goals is to keep my underlings from having to learn like I did. I'm not sure if it's best to have a "buddy" system, where a more experienced postdoc might look at a grad student's code, for instance, and the grad student would repay the favor down the line, perhaps with someone else. A more extreme buddy system would involve two people coding the same thing in parallel and then comparing outputs. I worry here that people might spend too much time on others' projects, which they might not feel are relevant--but from another perspective, this practice might help the lab's research be more cohesive (theory/computational labs often aren't) and could increase everyone's publication count. I'm also considering having a part-time lab tech who is effectively a developer, and who would serve as everyone's extreme coding buddy. Money for this developer would come from funding otherwise allocated to grad students and postdocs. Is it worth it?

    I realize I'm late to the comments here, but I would love another post on this topic and perhaps some anecdotes about what has worked in different groups. I am concerned about the volume of code individual researchers are expected to churn out on their own with minimal supervision.

  • [...] week I wrote a post about lab peeps arbitrarily changing established protocols based on the pursuit of fixing their current data ailment. Based on the comments and links, this is [...]

  • chkuo says:

    When I started my own lab I set up an internal wiki site (using Dokuwiki) for all of our protocols. The "edit" permission is give to the "trusted" members (e.g., full-time RA) but not the new-comers (e.g., summer interns). This way everyone has the access to the most recent version of everything and can figure out who changed what at when if necessary. I can keep an eye on what's going on by subscribing to the RSS feed to see all of the changes. So far this has worked very well for us.

  • [...] Prof-like Substance for “Protocol evolution or a bad game of telephone” (the right and the wrong way to tweak your experiments) [...]

  • […] Non-systematic alteration of parameters and processes in individual experiments can make it difficult to properly interpret downstream data. An electronic laboratory notebook (ELN) with protocol management options can help […]

Leave a Reply