Is there an alternative way to interview for TT jobs?

Feb 01 2013 Published by under [Education&Careers]

With minor variation, pretty much every job search for an academic tenure-track job works the same way:

1)Post an ad describing the type of research you are interested in adding to the department.

2) Wade through the resulting applications looking for metrics of success (publications, grant funding, awards)

3) (optional) Phone interview a subset of candidates (~10).

4) Invite 3-5 candidates for an on campus interview, during which they will meet with department members one-on-one, give a seminar about the work they have done, give some presentation on work they would do at the interviewing institution, eat a bunch of meals with the search committee.

From a logistical standpoint, getting the campus interviewee list right is pretty critical. If a committee goes through the campus interviews and doesn't find a suitable candidate, the search is either delayed while they scramble to bring different people in or is a failed search that may or may not re-appear in the following season. Neither option is good, so the committee really wants to get those invites right.

However, the metrics of success we use to evaluate people on paper don't always translate into strong interviews. It seems that every search has at least one interviewee who self-immolates in some way. I have been involved in searches in which only one candidate turned out to be acceptable to the committee.

This has led myself, and several others I have talked with informally, to wonder whether there is a better screening process by which we could identify candidates. There are so many good scientists out there who just need a shot at an interview, but can't get one.

It's an imperfect process, and yet I have never seen a department try anything different. What could we do better?

36 responses so far

  • How do they self immolate?
    A sample video of a talk the candidate gave could at least weed out those with utterly abysmal public speaking skills.

  • JB says:

    Self-immolating ... I like it. An example from my small liberal arts college department:

    Q: Although part of your job would be to teach classes A, B, and C, you have also experiences teaching courses D, E, and F, which we also offer. In case of a sabbatical or research buyout, would you be willing to substitute D, E, or F, for for A, B, or C for a semester? We will of course provide you with all course preps.

    A: No.

    There is a pretty hysterical thread on the Chronicle of Higher Ed forum about crazy stuff people say and do during interviews. Amazing.

    Pretty much all my colleagues in the humanities do conference interviews where two people stay in the conference hotel and do consecutive rounds of speed interviewing (20 - 25 min each). Not saying it works, but it is probably a way to weed out the crazies. A typical humanities job at my SLAC does get on the order of 200 - 300 applications, so there is a lot to weed out.

  • eeke says:

    " I have been involved in searches in which only one candidate turned out to be acceptable to the committee. " -- Aren't you only looking for one candidate? Isn't this what you want?

    You did not mention letters of support. Have you found, in any way, whether information provided in these letters predicts the performance of a candidate during an interview?

  • proflikesubstance says:

    Yes, we are looking for one person, but that person may have other offers and choose to go elsewhere. In that case, if you don't have a 2nd option, you are SOL.

    Letters are sometimes marginally helpful, but can serve as the basis for talking yourself in or out of a candidate, depending on what you want to take away from them.

    WRT the conference interviews, I know that is really common in the humanities, but I've never heard of it being done in science fields. Maybe it's because there isn't a single conference that everyone goes to in my field.

  • Jeramia says:

    I'm surprised it doesn't happen at SFN (maybe it does?). It's the right time of year, and is freaking huge.

    Also, there used to be a different, more efficient way: the old boy's network. A PI would call up his friend the chair of the search committee and tell him to hire his postdoc/grad student. I'm not sure you can have an efficient system that is equally fair to all the candidates.

  • Anonymous says:

    What about replacing a phone interview with a pre-interview Skype "interview" with an abbreviated talk by the candidate (20 min) and plenty of time for questions that might help you weed out the weirdos? It seems like body language might be something important that is missing on the phone. Although, this type of interview would undoubtably lead to more bias due to someone's appearance than a phone interview. However, in my experience people are usually more awkward on the phone than on Skype.

  • Dr Becca says:

    There really should be something at SfN, but I imagine the reason there isn't is that search committees can't quite get their shit together in time to make a medium-short list (a good chunk of app deadlines are Nov 1), plus everyone's so busy at SfN anyway--who's got time to conduct 20 15-min interviews?

  • iGrrrl says:

    Well, you could do what some institutions [sometimes] do, which is only hire people they already know, usually post-docs out of labs within the institution. "How else do you know who's good?"

    Worst case is to hire your own graduates, right out of the PhD with no post doc. They have no idea how the world works.

    Seriously, I don't know what could be better. I've seen people who did great right up to the questions after the seminar, and then fall apart. ("How do you know your electrode is in [brain region]?" they asked, expecting a histology answer. "I just know," he said, with dramatic gesture, and repeated himself when pushed. "I just know!" And at that point, so did the search committee and the rest of the department.)

  • eeke says:

    Actually, given the high number of applications you are getting, I am really surprised to hear that so many people fuck up the interview. Maybe no amount of information on paper is going to tell you how well a candidate behaves in person. Likewise, the way a person behaves during an interview does not always predict how well they will do on the job. Choice #2 or #3 may end up being far more successful than choice #1. If you're not finding a committee favorite after inviting only 3-5 people, you aren't inviting enough people, or you're being way too narrow in your search. My own supervisor was hired in her department despite having a research interest well outside the search committee's initial focus. She ended up being amazingly successful.

  • "It seems that every search has at least one interviewee who self-immolates in some way"

    PLS: I would like to know some examples of how the candidates self-immolated (if this is word) them selves :D

  • t says:

    I think there's a tacit assumption in your post that the impression committees and departments draw during the interview is what we actually want to base the hiring decision on, and that the point of all that initial wading through CVs and pre-screening is just a preliminary filter to get to the stage where committees can see what the candidates are really like. I understand that this is how it normally works, but it's at odds with what we know about how complex decisions ideally should be made.

    In particular, we know that this is exactly the kind of situation in which people tend to grossly overweight factors that ultimately are not very predictive of success. Job talks and personal impressions are two of the worst offenders. I can't count the number of times I've heard people say things like "oh, well she seemed really great on paper, but then she gave a terrible talk, and everyone thought she was extremely aloof", only to have that person passed over, go somewhere else, and do absolutely outstanding work--often of a highly collegial and collaborative nature. The reality is that hiring committees have a lot of relatively reliable evidence of what a person is like and what they can do in front of them in the form of CVs, publications, letters of recommendation, etc. And then they very often throw that away in favor of completely idiosyncratic quirks that empirical data suggest are not highly predictive of anything at all (e.g., clinical psychologists talk a lot about how they're good at picking up on clients' non-verbal cues, when available data suggest such cues probably only bias judgment).

    I'd argue that the real problem is not that we do the interviewing wrong so much as that we give much weight to the in-person interview process at all. Of course you want to make sure that the person you hire isn't a complete and utter jackass; so it's reasonable to bring people in just to make sure they don't have a clear personality disorder that's apparent to everyone who interacts with them. But short of that (and that's not a common occurrence), the idea that you can tell how well a person is going to fit in or how well they're going to do at your institution based on an hour or two of interaction in a highly unnatural situation is a pleasant fiction. If we really cared about getting it right, the way we would make hiring decisions is to approximate, as closely as possible, an actuarial process: write down the outcomes you care about ahead of time (productivity, collegiality, etc.), identify the best objective indicators of those outcomes (number of publications, letters of recommendations from as many people as possible affirming personal qualities, etc.), and weight them however you think is appropriate. Then, when the candidates show up to interview, give everyone with a say in the matter a numerical rating sheet that they have to fill out without discussing the candidate with anyone else who has a say (this is to ensure independence of observations--because hallways conversations have a huge effect on how a candidate is received). Then, after all the candidates are done interviewing, sum up the ratings, plug them into the model everyone agreed to ahead of time, and select the candidate who comes out on top. That's who you make an offer to.

    Of course, when you put it this way, it sounds ludicrous to most people, because we all prefer to believe that our brief interactions with people can actually trump the weight of evidence accumulated over many years. Which is probably why the process ends up looking so random. Everyone wants to hire someone they personally like (either personally or professionally), without realizing that liking someone is not a good predictor of occupational success, and that even if it was, your ability to actually determine how much you'll like a person once you have to work next door to them for 30 years is quite limited.

  • Driving By says:

    Older colleagues tell me that a number of candidates "self-immolate," that seems to be the rule, and it does make the job of selecting the top choice easier. We are having a search in my area right now. There are about 20 who look phenomenal on paper, what I call "the long short list," and in an ideal world we would interview all of them. Unfortunately, many of them won't get a chance to interview, as we are only allowed to bring in 3 people, which is an arbitrary number set by the Dean. The first candidate who interviewed turned out to be very green and very arrogant and very unprepared, and we are ticked off as he took the place of someone who could be less obnoxious, better prepared, and just as smart. I would love it if there was a way to sample the whole pool of 20 or so who all look very promising on paper.

    Maybe Skype pre-interviews with the recruitment cte, as suggested above. That sounds interesting.

  • proflikesubstance says:

    PLS: I would like to know some examples of how the candidates self-immolated (if this is word) them selves

    I don't have enough space to describe all the ways, but arrogance and lack of preparation are prime ways to light that match.

    WRT skype interviews, I've done that and it's not a whole lot better than phone interviews. The disadvantage is that the committee can't exchange notes and eye rolls for the rough ones.

  • AcademicLurker says:

    The thing that really surprised me the first time I saw the hiring process from the other end was the number of people who give fantastic seminars and then fall on their faces during the chalk talk. Not so much in the sense of having fatal flaws, but in the sense that they had no idea that the chalk talk is serious and not an afterthought.

    I still can't believe it when I see future plans of a vague "Well, I worked on this protein/gene/signalling pathway as a postdoc, and now I'll work on this other one" variety. Didn't their PIs tell them what a chalk talk is supposed to be like?

  • anotherpostdoc says:

    Re: "Worst case is to hire your own graduates, right out of the PhD with no post doc. They have no idea how the world works."

    You just described my PhD supervisor. Truer words have never been spoken. His career advice was "don't worry, you'll be hired as soon as you defend". That was almost 4 years ago.

    Re: " Then, after all the candidates are done interviewing, sum up the ratings, plug them into the model everyone agreed to ahead of time, and select the candidate who comes out on top. That's who you make an offer to."

    Sounds a bit like the Brazilian system, where each commitee member scores you on a variable combination of : written exam (!), CV (each paper worth 1 point, each conference abstract 0.25, so on), memorial talk (where you have to defend your career choices, why did you publish what you did, why your work is relevant, etc.), a sample undergraduate lecture, and/or a 30 min research project presentation. The committee members give the candidates scores, and the best scored gets the job.

    To be honest, I don't like it.

    1) It places a lot more pressure on the candidates and the committee (every single applicant has a right to go through the entire process, as long as they get the minimum acceptable scores in each category);

    2) The scoring system is often poorly thought out, with one specific category outweighing the others so much as to make them almost irrelevant;

    3) Many committees will still try to steer the final scores towards the candidate they want/like, and;

    4) Candidates can officially appeal on the results, which happens way too often, delaying the entire process even more.

    On the bright side, once it's over, you know exactly why you didn't get the job. After dozens of applications sent to north american institutions and pretty much no interviews, I still have no idea why I'm never shortlisted.

  • Jim Thomerson says:

    Back before the deluge, we would very carefully craft a notice of position available. This would be run through the university legal department. Once we hit the cutoff date, the committee would look at all the applications. Each member looking for one of the things listed in the job qualifications. If a committee member found an application which did not fit, it was immediately set aside.

    The committee would recommend some 10 applicants to the faculty. We would have requested letters of reference from the 10. The faculty would determine the three top candidates, and they would be invited for interview.

    During the interview, they would also teach an appropriate class lecture. They would have had information and opportunity to prepare before coming to the interview. The class would submit evaluations of the candidate. For one reason or another, we occasionally had unsuccessful searches.

    I made the list of ten in applying to a major institution. I would have replaced a friend who resigned. I talked with my friend and found that he had about an 80 hour a week work load. I got a phone call asking if I were still interested in coming for an interview. I told them no.

    We have discussed giving each candidate a number, and drawing numbers out of a hat, but not seriously.

  • proflikesubstance says:

    JT, that's how it's done in the color TV era too, with or without the teaching seminar.

  • DJMH says:

    I know it would introduce a different kind of bias, but perhaps it would be helpful to recruit applicants more aggressively at conferences? I.e. if you meet a postdoc who is clearly going on the market soon, and have enough time to chat and think they're pretty good, then chase after them to apply? If the in-person screening counts for so much, then it seems like it'd be good to do more of it in advance.

  • bashir says:

    Slow golf clap for t's comment.

  • We have recently instituted very aggressive soliciting for applications - faculty make a list of people we would like to apply, at least two people contact each person encouraging them to apply & why our department would be good for them.

    This has worked well for us and has ensured a good pool of acceptable candidates, which is especially difficult for Assoc/Full/Chair searches. (no failed searches yet) but I agree that 3 feels too few to bring to campus. 5 but with a shorter interview period ( we do two full days ) would be better I think.

  • proflikesubstance says:

    T - if all we were hiring was a person to share a departmental affiliation with and all we cared about was research output, then you are right. But that's not the case. Whereas research output is a major player, so are things like "can this person interact well with students here?" and "will this person be willing to carry their share of service and teaching load?" you know why this is important? Because we potentially have to deal with this person for the next 30 years. If we hire a shitty colleague, who do you think ends up picking up the slack for the things they won't/can't/don't do?

    Ever had a colleague ignore their 50 undergrad advisees? Where do you think those students go?

    Ever had a colleague estrange much of their lab?

    Ever had a PI throw everything in their lab into the hallway and leave without ever coming back (true story)?

    My point is that when things go bad it doesn't happen in a vacuum. Other people in the department are affected and the situation is more acute in a department with research is not the only concern. So if you want to argue that an application package gives you better feel for a person than meeting them and talking over two days, then I would be interested to hear how. And no, the letters rarely give a full story.

    Beyond THAT, I want to know where the person sees their research going. I want to be able to discuss it, see how well they have thought it out. I want to shoot the shit about their ideas over a beer and see if it sounds exciting. Are THEY excited?

    I have little doubt that anyone we interview can be successful in the right circumstances, but some aren't ready. I wasn't ready to lead my own lab when I had my first interview and I have interviewed people in the same boat. Another year or two as a postdoc makes a huge difference and I fully suspect that if I had left my postdoc two years earlier than I did, I would not be doing as well as I am. Others are ready earlier, and it is important to suss that out too. So while they may go on to do great things after missing out on a job, timing matters too.

  • t says:

    PLS, I'm not arguing that we should only care about research output because that's the only thing that can be measured objectively. I'm arguing that no matter what criteria a search committee cares about (and they certainly can and should include things like collegiality), there are much better ways to go about the process than to have five people get together in a room and discuss their qualitative impressions.

    For example, you want to pick a candidate in part based on how pleasant a colleague they are? Fine; that makes perfect sense. Decide up front how much you want to weight that relative to other factors. Then, when the candidates show up, give everyone who meets with them a detailed rating form with questions like "this person seemed friendly" and "I could see myself collaborating with this person in future". Have everyone fill out the form and submit it without talking to anyone else about the candidate, and immediately after meeting with the candidate. Then, don't review the ratings as a group until after all the candidates have visited (because there is reason to think there are huge order effects in the hiring process).

    Eventually, when the committee does get together, it should live and die by the data--i.e., compute the mean for each item across all respondents and plug that into the regression equation as previously agreed upon. Do not try to rationalize away results you don't like--e.g., "oh, well sure John Doe was pleasant enough and the ratings suggest that everyone liked him; but when I met with him, it was clear that he was just superficially charming and isn't really going to get along with people." That is precisely what gets people in trouble. This kind of thing happens all the time, and it's absolutely infuriating, because it often amounts to one person picking on some idiosyncratic feature of a candidate's behavior and privileging it over plenty of other much better data.

    To reiterate, I'm not arguing in favor of one criterion or another during the hiring process. I'm arguing that qualitative anecdotal impressions are a very bad way to estimate the things a committee cares about, including things like affability, collegiality, and so on. For instance, when you say things like this:

    Beyond THAT, I want to know where the person sees their research going. I want to be able to discuss it, see how well they have thought it out. I want to shoot the shit about their ideas over a beer and see if it sounds exciting. Are THEY excited?

    ...you're implying that you are somehow in a good position, based on one short conversation, to determine how excited and enthusiastic the candidate is, how capable they are at formulating a research program, etc. But the data on this type of thing (which mostly come from domains where people have years of expertise, not just sitting on a search committee once every two or three years) are unequivocal that even 'expert' decision-makers do very poorly in these kinds of situations, because one can't possibly begin to take into account all of the many situational and dispositional factors that play into an interaction. When you walk away from the bar, you might think "well, that guy doesn't really know where his research program is going"; but if you're honest with yourself, you should maybe ask yourself whether it's possible that you were the problem in that interaction (maybe you gave off bad vibes), or that the candidate just happened to be nervous (but wouldn't be nervous once they actually started working there), or that you just randomly phrased a question in a way that confused them (so they answered a different question than you thought you were asking), and so on and so forth.

    Again, I'm not saying you shouldn't care about the things you mention above; I'm saying that you're kidding yourself if you think that you're able to discern that accurately and then weight it appropriately next to many other factors. Should committees weigh how nice someone is? Absolutely. But they should decide how important that is to them up front, write that down on a piece of paper, and then stick to that. And they should base their estimates of how nice/collegial/focused each candidate is on the aggregate quantitative opinion of as many people as possible (faculty, students, search committee, etc.), not on the qualitative and non-independent judgments of 4 people sitting together in a room for an hour or two.

  • proflikesubstance says:

    T- You sound a lot like you've never been on a search committee. I think you would be surprised at how unprepared some people really are and how little they have thought about their future. I'm going to guess you're a postdoc and I can understand how frustrating some of this is, but it's surprisingly easy to talk to candidates and just find out where they are at. Sure, there might be interviewer influence, but guess what is funny about science: we constantly have to deal with exactly that kind of challenge both in person and in writing. So why should an interview be different?

  • t says:

    I'm not sure I understand your comment. I'm not arguing that things should be done differently for the sake of the interviewees; I'm arguing things should be done better for the sake of the interviewers. I'm simply responding to your question about how to do interviews in an alternative (and better) way. The search committee presumably has as its goal to identify the best candidate for the position according to some specific set of criteria. I'm pointing out that based on an enormous empirical literature, we have excellent reason to think that the approach of sitting in a room and comparing qualitative notes about candidates is a pretty bad approach--or at least, it's much worse than trying to use an actuarial approach. You haven't said anything to dispute that, so I'm not sure what your objection is. Surely it's not particularly difficult to try and make the process more quantitative; actually, I think it would reduce the workload quite dramatically--or at least cut a lot of arguments short. So what's the problem? If you accept that there might be interviewer influence (which is only a small part of what I'm pointing to, but fine), why wouldn't you want to do whatever you can to minimize that influence?

  • SearchChair says:

    I am search chair of a Microbiology department at a R1 institution. We have had 3 years of searches finally hiring 2 new faculty this year. Last year we interviewed 6 postdocs, all with K awards, either K22 or K99. All were awful for various reasons. All were from very respected labs and had published well. Just to give you an idea of what candidates do to screw up their interviews, here is a taste.

    1. Arrogant/Cocky/Overly confident responses to questions during your talk, professing that you know the right answer and there are no possible other explanations is usually bad.

    2. Giving the talk in the same monotone coma inducing tone that you would talk to a Telemarketer with is usually bad.

    3. When asked if you are interesting in collaborating with other people in the department and you answer emphatically "No". Not a good sign.

    4. You answer, "Yes" when asked if your current boss, a bigger than big cheese in the field, is going to continue to work on this project and directly compete with you. Not a good sign, you will lose that race.

    5. Running through your talk at such a high pace as you present every mouse experiment, western blot and ELISA you have done over the last 4 years while not really explaining why you are doing any of these assays, and then starting your third section of your talk with 5 minutes left in the hour, then proceed to blow through the allotted hour, and keep talking while the Search Chair stands next to you alerting you to being out of time, and you keep going and going only to end 20 minutes after the allotted time. After which no one asks questions because they just want to get out of the room. Not a good sign.

  • SearchChair says:

    I forgot the best one:

    6. When you put your flash drive in the computer to load your talk and your powerpoint file is mixed in with other files. And as the icon for the other files show up on screen the audience sees they are not other PPT files but pornographic pictures of what look like young girls. Not a good sign you are going to be asked back for a second visit.

  • DJMH says:

    T, you're talking about how to make decisions *after* the search committee has met with the candidates. PLS is talking about how to get better candidates to campus. Your conversation is at cross-purposes.

    Frugal ecologist, that's exactly what I'm thinking. Since the in-person matters a great deal (particularly for smaller departments, where the candidate's personality is a huge factor in their winningness), people should be putting in the legwork the year prior to a search, to chat up promising postdocs at conferences. Then you stand a chance of improving the pool invited to campus, which is huge in having second and third choices even if the first choice goes elsewhere.

  • t says:

    DJMH. that's fair, although I think it's mostly just a matter of degree. Obviously it's not feasible to use information on personal qualities when assessing candidates on paper, but the same basic philosophy should apply--i.e., once the first cull is made (which can only be done using heuristics for most searches given the volume of applications) committees should strive to make quantitative rather than qualitative judgments whenever possible. E.g., have everyone on the committee rate the key candidates on dimensions like productivity, perceived fit with the department, potential for collaboration, etc.; and then use the aggregate quantitative ratings as the basis for selection rather than sticking five people in a room and having them hash it out.

    SearchChair, most of your examples seem like precisely the kind of thing I'm talking about as behaviors that seem offensive in the moment, but are of very dubious utility in predicting long-term success and collegiality. In particular, numbers (2), (4), and (5) are, I would argue, almost certainly terrible things to base a hiring decision on. If giving a monotonic or disorganized job talk precluded academic success, a lot of well-established and very famous faculty would be out of jobs. I'm willing to bet there are people in your department you consider integral colleagues who routinely give very bad talks (and probably always have); knowing that, why would you conclude that a bad job talk (or even a terrible one) makes one an awful candidate? The fact that search committees routinely reject candidates for this kind of thing when (at least in some cases) they're stellar in most other respects strikes me as a discredit to the way we typically do things, not a defense of our hiring practices.

  • proflikesubstance says:

    T - One thing the committee is trying to assess is the future prospects of candidates as independent PIs. Critical to this is attempting, as best one can, to separate personal ability from lab environment. Everyone has a story of the person who came out of a famous pedigree and then struggled mightily when the infrastructure they took for granted wasn't there.

    This is the whole reason for the "chalk talk". To see what a candidate's vision is for their future work and what potential they have for funding. Therefore, things like SearchChair's #4 are critical, because it is generally a bad idea to jump into a field already populated by fully running labs unless you have a unique twist of direction that sets you apart. It takes a while to establish a new lab and you will not be able to directly compete toe-to-toe with giants in the field.

    Similarly, even though not giving a great talk might be a bad predictor of success, not being able to organize a coherent and organized talk does not speak well to one's ability to communicate effectively about their science. Since communicating your own science (and not that of the lab you just trained in where it is unclear how much writing was done by a candidate) in the form of papers and proposals is absolutely critical to the success of a lab, people use the talk as a proxy for one's individual ability in this area.

    Is it perfect? No. But neither is relying on past success as a member of an established lab with funding often obtained directly or indirectly (again, environment) on the reputation of someone else.

  • Gilly says:

    "Since communicating your own science (and not that of the lab you just trained in where it is unclear how much writing was done by a candidate) in the form of papers and proposals is absolutely critical to the success of a lab, people use the talk as a proxy for one's individual ability in this area."

    Yes, this. Also, let us not forget, presumedly this person will be asked to teach classes for undergraduates and possibly also graduate students. If they cannot organize their own work coherently and interestingly (which is the science one would expect they would know most intimately) would you really trust them to teach classes? That's just a nightmare waiting to happen.

  • t says:

    Again, you're misconstruing what I'm saying. I'm emphatically not suggesting that talk/chalk-talk performance should be completely ignored in hiring decisions. I most definitely think they should count for something. What I'm pointing out is that (a) allowing the decision to rest on entirely qualitative, subjective considerations of a small number of people is a sure-fire recipe for massive overweighting of personal impressions at the expense of other factors, and (b) people grossly over-estimate their own ability to reliably judge personal qualities from thin slices (i.e., even if you thought that the only thing worth hiring a person for is their ability to give talks, making that decision based on just one talk would be a very bad idea).

    To reiterate: I'm not telling you what criteria you should or shouldn't use to hire people; I'm saying that whatever those criteria are, you should set about applying them in a way that is as quantitative as possible and minimizes human bias as much as possible. If you really, truly, believe that a relatively unreliable single-item measure of talk performance should be a very large component of the decision to hire someone you have to work next for 30 years, then by all means, say things like "their talk was so boring I fell asleep, so we shouldn't hire them". My guess though is that most search committees don't actually mean to assign such large weight to idiosyncratic personal interactions and impressions, but--being human--they simply fail to realize that that's what their doing when they go about business as usual rather than taking steps to do things in a more objective way. I'm willing to bet good money that if the next time you sat in a search committee you followed some of the simple steps I suggested above (most notably, circulating rating sheets to everyone, and aggregating the results before having hallway discussions about the candidates), you would be surprised at how big a difference there is between one or two influential people's "clear" subjective impression of a candidate, and what the numbers aggregated over many observations actually say.

  • proflikesubstance says:

    T - you keep saying that the talk doesn't matter and I keep telling you why it does. I've made it clear that whether or not you make the ranking of certain criteria, such as "organized future directions" or "feasibility for funding in this area" numeric, as you suggest, it is no more objective. I have never been on a search committee where the preferred candidate of anyone did not have a clear and interesting future directions talk. You seem to think that making a ranking sheet solves everything and I am contending that I can't imagine you have been involved in a faculty search if you believe that.

    Contrary to your assertion, faculty positions are not won or lost on whether one chews with their mouth open at dinner. Research trajectory comes first, followed by not being insufferable or exhibiting characteristics that people find objectionable (e.g. sexism, racism, etc.). Beyond that, it is usually self-interest of the faculty involved in making the decision - typically with collaboration or increased capacity in a particular area in mind.

    At issue is having several people with diverging interests involved in making the decision and that can't be quantified on your sheets. Or at least, those self-interests will shine through in the exact same nonobjective way they would otherwise. This is the "randomness" that people refer to, and there's nothing random about it. It only seems that way to people not involved in the decision making process.

  • Frosty says:

    I usually just lurk here, but I cannot resist chiming in to say I would bet cash money that t has recently read Daniel Kahneman's 'Thinking, Fast and Slow'

  • Dan says:

    One thing that I have found helpful is being very specific about what information we would like in the application. This gives the department maximal information, and helps to level the playing field between well-coached and poorly-coached candidates. (I'm not interested in hiring based on coaching.)

    The first place I saw this was at Princeton's EEB department, where they ask for a "a vision statement, no longer than 2 pages, that outlines one or more major unsolved problems in their field and how they plan to address them." This speaks to the field-changing visionaries that I imagine they are looking for.

    At my SLAC, we explicitly ask people to keep the research statement to two pages and discuss how they will involve undergraduate students in the work. We don't have the expertise to evaluate the fine details of the proposed work (If we are hiring a cell biologist, it is probably because we don't have a cell biologist on the faculty) but we need to know how they will get the done without grad students. You can ask for similarly specific things in a teaching statement.

    If there is information that could be provided in writing but isn't always included in the current format, then ask for it.

  • t says:

    Well, I think we're probably at an impasse, so I'll leave my last (long) response here and let you have the last word.

    I think a parallel from a different domain might help clarify the issue. There is an enormous literature on actuarial vs. human prediction of criminal recidivism in the justice system. Virtually without exception, it turns out that simple equations (and often literally just a summing up of factors thought to predict recidivism) substantially outperform the predictions of parole officers with years or decades of experience on the job. And yet there are very few jurisdictions at any level that allow computers to make decisions about who to release and who to keep in jail. The typical arguments against actuarial prediction (when anyone actually even bothers to formalize them) are quite similar to the objections you're raising in this thread--e.g., that simple regression equations and quantitative ratings can't possibly capture the complexity and nuance that humans see. And yet, the data are unequivocal: mechanical predictions are much better, and if we replaced human parole boards with computer programs wholesale, we would save enormous amounts of money, prevent many unnecessary deaths, and greatly simplify the process.

    Now, I suspect that if you, as a scientist, took the time to read this literature, and considered the overwhelming non-response to it on the part of the legal system, you might get a bit angry. Because it really is a travesty, from a human cost perspective, that we continue to use intuitive judgments to make life-and-death decisions in domains where we know, unequivocally, that there is a much better way to do things. And yet the fact that actuarial prediction is never actually taken seriously in domains where many lives hang on the line becomes understandable as soon as you observe your own reaction to the same suggestion in a domain that applies to you--in this case, hiring your future colleagues. You have a very similar knee-jerk response and unwillingness to seriously consider the possibility that, yes, simple equations can in fact outperform a group of PhDs deliberating very carefully for many hours over the matter.

    The facts are simple, PLS: in every domain that's been studied to date that has the same properties (i.e., the need to consider and integrate many different pieces of information; the lack of immediate and explicit feedback; etc.), humans judgment simply, to put it gently, sucks. This is certainly true in the domain of personnel hiring, which actually shows some of the most pronounced biases in favor of mechanical predictions--and that's when the people in question are HR "experts" with years on the job, not just a committee full of people who do this particular task with no prior experience and at best once every year or two. I'd encourage you to read some of this literature with an open mind; for example, here's a nice paper on the difficulty in getting HR departments to use empirically validated hiring techniques and abandon unstructured interviews (not so different from ours!) despite a world of evidence that they'd be much better off. And here's a classic article by Paul Meehl discussing basically the same issue in the context of clinical diagnoses. I'm happy to provide many, many others--there are hundreds if not thousands of articles to basically the same effect, and very few on the other side of the issue.

    So, unless you dispute that characterization, we have two ways to interpret what's going on here. One possibility is that in fact, yes, the faculty hiring process really is somehow unique, and is not subject to the same principles that seem to govern every other domain like this (including manager hiring, clinical diagnosis, criminal recidivism, academic performance, and so on). The implication would be that somehow you and four of your colleagues sitting together in a room are uniquely able to outperform, based on essentially just your qualitative judgment--with no decision aids or appeal to quantitative prediction--an actuarial approach that has about 80 years of solid support with virtually no counterexamples. Is this conceivable? I suppose so. Is it likely? Not at all.

    The other possibility to consider is that maybe you're kidding yourself. Please consider the possibility, however painful, that there's not much difference between your knee-jerk response that the quality of prospective faculty "can't be quantified on your sheets", and the common objection clinical psychologists raise to actuarial prediction when they claim that it's just ridiculous to think that a simple prediction equation could capture the depth and complexity of a patient's behavior. Or when an HR director says "the unstructured interview is the best tool, because that's when you really get a sense of what a person is like and how they're going to perform". In every domain where this comes up, the experts in question stubbornly resist the notion that they--with all their intelligence, knowledge, and many years of training--could be outdone by a dumb equation with numbers blindly plugged in. And in every domain we've looked at, they are, empirically, wrong.

    That's the background evidence we need to keep in mind when discussing this particular context. And I think that data is incompatible with many if not most of the points you make:

    T - you keep saying that the talk doesn't matter and I keep telling you why it does.

    On re-reading what I've said, I don't see where I said it doesn't matter at all. I see where I said it's nowhere near as important as people think, and I stand by that, because there's a wealth of empirical evidence demonstrating that people consistently overweight personal impressions to the detriment--often near-total--of other indicators. The examples you and others have given throughout this thread provide ample evidence of that. For example, you suggested that it's important to have a colleague who sounds excited when talking research over beer, but it's not at all clear that you have the ability to make general characterological assessments on the basis of one occasion, and in any case it's patently obvious if you look around that there are many exceptional scientists who are kind of quiet and never sound excited about what they do over beer.

    I've made it clear that whether or not you make the ranking of certain criteria, such as "organized future directions" or "feasibility for funding in this area" numeric, as you suggest, it is no more objective.

    It's unquestionably more objective. In several ways. First, you make the ratings right after you interact with the candidate, while your memory is fresh; that pretty much ensures they're more reliable than retrieving your memory of the event days or weeks later, when all sorts of biases are bound to creep in (e.g., primacy and recency effects). Second, the rating that I turned in doesn't change just because someone on the search committee is particularly aggressive or persuasive when we all meet. If I wrote down the number 6, I wrote down the number 6. It doesn't change later just because I don't want to rock the boat in the presence of my more obnoxious colleagues. Third, and most importantly, having numerical ratings makes it possible to easily aggregate over the entire department's responses, not just the search committee's, which is pretty much guaranteed to give more reliable results (because each individual estimate is very noisy). That's just basic psychometrics.

    I have never been on a search committee where the preferred candidate of anyone did not have a clear and interesting future directions talk.

    You're basically making my case for me here. If this is really true, it should be a big clue that search committees' intuitive criteria are not well-calibrated, because it's crazy to think that there are no job candidates out there who do exceptional science and have highly productive records demonstrating their capacity to conduct systematic research, but who for any number of reasons might fail to give a good "future directions" talk (as subjectively assessed by the committee, mind you). An actuarial prediction would never in a million years weight a factor like this to the detriment of all else, but what you're basically saying here is that if you don't have a good future directions talk, you're toast. To me this sounds just like a parole officer saying "if a prisoner can't give you a compelling account of how they're going to direct their life in a law-abiding way for the next five years after we release them, they can't possibly be safely released." It may feel comforting to think that, because it gives you the illusion of certainty about the future, but it's almost certainly very wrong. Life is just not that simple.

    You seem to think that making a ranking sheet solves everything and I am contending that I can't imagine you have been involved in a faculty search if you believe that.

    No, I think nothing solves everything, because making these kinds of predictions is an inherently unreliable enterprise, and there is no amount of information that will allow you to make slam dunk judgments consistently. However, I do think that there are better and worse ways to go about the process, and I think an enormous literature supports me in saying that actuarial decisions are the right way to go in this context.

    And for the record, I have been on a hiring committee (once, as a student representative), and everything I saw was consistent with both the case I'm making here and the kinds of arguments you're raising. But it actually shouldn't matter, because when you say "you couldn't possibly understand because you haven't been there," you're making the exact argument that clinicians raise when people suggest using actuarial diagnoses ("it's absurd to think that an equation could do better; clearly you've never seen a patient!"), even though the data speaks for itself.

    Contrary to your assertion, faculty positions are not won or lost on whether one chews with their mouth open at dinner.

    Well, so far in this thread, you and other people have implied that jobs might be won or lost (and in some cases, irrevocably lost with no chance of repair) because a candidate is (a) not excited enough over beer; (b) too monotonic when talking; (c) too disorganized when talking; (d) overly confident; (e) works in a field populated by serious competitors; (f) can't think on the spot when asked tough questions; and there are probably others I've missed. My point is that while these behaviors surely carry some minimal information, they're much closer to "chews with mouth open" territory as you realize.

    Research trajectory comes first, followed by not being insufferable or exhibiting characteristics that people find objectionable (e.g. sexism, racism, etc.). Beyond that, it is usually self-interest of the faculty involved in making the decision - typically with collaboration or increased capacity in a particular area in mind.

    Again, you are implicitly assuming that you can determine with reasonably accuracy what someone's research trajectory is going to be over a career based on a thirty-minute chat or an hour-long talk. But if you actually stop to think about all the cases of exceptional scientists who you've seen given a single terrible talk (just one!), or bad ones who've given a compelling one, surely you'll see that this is a very bad idea.

    Since you like anecdotes, let me give you one. When I was a grad student, an acquaintance of mine at a different school got an interview at my PhD institution. The candidate emailed me a couple of weeks ahead to ask what I thought she should talk about in her talk. I said that I didn't really know and described what I'd heard they were looking for in general terms. This was a person who had two relatively distinct lines of research (but both active), and so could give basically two different job talks. So my acquiantance shows up and gives a talk on one line of research, presumably because she thought that was what the committee wanted to hear. Then, I later hear from the search chair (who was my advisor at the time, and someone I have enormous respect for) that the search committee "had hoped the candidate would talk about [other line of research], but they blew it". Now, if you're dispassionate about it, I think this is just a silly conclusion to draw: the candidate's CV clearly indicates that they're still actively working in the other research area, and they told people as much in one-on-one meetings when they were asked. But no. The talk wasn't what the search committee wanted to hear, so that candidate was gone--bad research 'fit'. And despite what you may want to believe, this kind of thing isn't an isolated occurrence; it happens all the time, because what candidates say in their talks and in person is, your intuitions notwithstanding, not a direct window into who they are and what they will do 5 years down the road. It's actually a very poor and indirect gauge that is clouded by all sorts of irrelevant factors, ranging from strategic decisions that seemed quite reasonable but turned out badly, all the way through the fact that the candidate is distracted by a stain on their pants they missed and hope no one else notices.

    At issue is having several people with diverging interests involved in making the decision and that can't be quantified on your sheets. Or at least, those self-interests will shine through in the exact same nonobjective way they would otherwise. This is the "randomness" that people refer to, and there's nothing random about it. It only seems that way to people not involved in the decision making process.

    This is patently false. If you take actuarial prediction to its limit (and I'm not saying one has to do that), there is no room for "diverging interests" to have anything but a very small effect. If you construe the job of the search committee as being to come up with a prediction equation--and thereafter it's just a matter of plugging in numbers and picking the top candidate--then there's precious little room for anyone to exert their ego or biases once the candidates arrive. If we collect quantitative ratings from the 15 - 20 people who meet with each candidate, even if I want to torpedo everyone but candidate B, the worst I can do is very slightly bias the numbers. That's a world away from me sitting down in a room with three or four people and trying to argue them into submission based on my qualitative impression and clever way with words.

    And even if you don't trust just the numbers (though, again, the literature suggests you should), and still insist on arguing about it, it surely still helps to have quantitative data to turn to. For instance, when one committee member says "oh, that guy works on Organic Herbicide Removal, and no one in the department would really be able to collaborate with him", someone else can simply pull out the aggregate ratings from all the faculty who met with the candidate and indicate that, in fact,that candidate got the highest "I could see myself collaborating with this person" ratings.

    The bottom line is that you're basically arguing from your own personal intuition and anecdotal experience against what is a voluminous and very well-established literature in I/O psychology and judgment and decision-making. Is it conceivable that faculty hiring committees exist in a unique bubble of their own, and are not susceptible to all of the cognitive biases and limitations that affect just about every other kind of decision-making process like this? I suppose so. But a priori, it seems extremely unlikely, and I don't think you make a very good case for it by making exactly the same arguments that people raise in other domains where we demonstrably know that actuarial predictions are much better than the 'experts'. And those are domains where people go to school for years, and then work full-time for years, making exactly the same kinds of decisions that are at issue. Here we're talking about people who have no background in personnel hiring, do it once every year or two in the best of cases--and yet are somehow supposed to have a level of insight and ability that doctors, lawyers, judges, academic counselors, and clinical psychologists don't have. I don't know; it doesn't seem so plausible.

  • SearchChair says:

    t,

    The point in these issues is not that a single one will kill your chance, but they generally cluster together. I do stand by the fact that you have to give a coherent, logical, throught provoking job talk that summarizes your area of work and where you are going next. If you can't give a clear explanation of your work and get to the damn point under time, then you wont get a job in my department. Do some of the old farts give bad talks? Absolutely. But they have been doing this forever. As a postdoc newbie, you need to at least start out driven, clear, concise, and to the point. Once you are a PI for 20 years you can do whatever you want.

    If you haven't noticed, its a different time now than 20 years ago. A lot different qualifications to get hired and even harder to get money. I think our criteria are pretty clear. And even then, everyone gets a vote. There have not been any issues with consensus on any candidates since I have been a search chair.

    SC

Leave a Reply


× nine = 63