Why Interviews Are Flawed

A ‘Pragmatic Contract’ For Criticism
A common framework for political criticism seems to be to pin it up against an ideology. However, politics is the art of the real. It is therefor incoherent to provide criticism that isn’t grounded in the system as it currently exists. Holding a criticism of some social reality to this standard is an essential ‘pragmatic contract’.

The Interview As It Currently Exists
“hiring people – whether as employees or as third-party contractors – generally means sacrificing today’s staff capacity for future staff capacity, and is always a risky (though essential) long-term investment with a difficult-to-forecast payoff.” (Holden Karnofsky)

The Flaw: interviews are risky because the information gathered is not adequate to correlate with future performance
Of course, this only holds to the extent that the position is not well-defined or easy-to-measure. However, in general, the limited information gives rise to an intuitive prediction that doesn’t regress towards the mean, and is therefor biased. Consider this example:

“Suppose that I predict for each golfer in a tournament that his score on day 2 will be the same as his score on day 1. This prediction does not allow for regression to the mean: the golfers who fared well on day 1 will on average do less well on day 2, and those who did poorly will mostly improve. When they are eventually compared to actual outcomes, non-regressive predictions will be found to be biased. They are on average overly optimistic for those who did best on the first day and overly pessimistic for those who had a bad start. The predictions are as extreme as the evidence.”

The pitfall of prediction: We substitute the ‘question of prediction’ for the ‘question of evaluation’
“People are asked for a prediction but they substitute an evaluation of the evidence, without noticing that the question they answer is not the one they were asked. This process is guaranteed to generate predictions that are systematically biased; they completely ignore regression to the mean.”

‘What You See Is All There Is’ (WYSIATI) is a ‘trait’ of System 1 where your associative memory quickly and automatically constructs the best possible story from the information available.

WYSIATI holds a lot of influence in our creation of an answer to the ‘questioning of evidence.’  The confidence of this answer “is determined by the coherence of the best story you can tell from the evidence at hand.” This has little regard for the quality or accuracy of the gathered evidence, and therefor also of its predictive quality.

An Example Analysis Of An Interview
“A department is about to hire a young professor and wants to choose the one whose prospects for scientific productivity are the best. The search committee has narrowed down the choice to two candidates:

1) Kim recently completed her graduate work. Her recommendations are spectacular and she gave a brilliant talk and impressed everyone in her interviews. She has no substantial track record of scientific productivity

2) Jane has held a postdoctoral position for three years. She has been very productive and her research record is excellent, but her talk and interviews were less sparkling than Kim’s

The intuitive choice favors Kim, because she left a stronger impression, and WYSIATI. But it is also the case that there is much less information about Kim than about Jane. We are back to the law of small numbers. In effect, you have a smaller sample of information from Kim than from Jane, and extreme outcomes are much more likely to be observed in small samples. There is more luck in the outcomes of small samples, and you should therefore regress your prediction more deeply toward the mean in your prediction of Kim’s future performance. When you allow for the fact that Kim is likely to regress more than Jane, you might end up selecting Jane although you were less impressed by her. In the context of academic choices, I would vote for Jane, but it would be a struggle to overcome my intuitive impression that Kim is more promising. Following our intuitions is more natural, and somehow more pleasant, than acting against them.”

tangent: “You can readily imagine similar problems in different contexts, such as a venture capitalist choosing between investments in two start-ups and that operate in different markets. One start-up has a product for which demand can be estimated with fair precision. The other candidate is more exciting and intuitively promising, but its prospects are less certain. Whether the best guess about the prospects of the second start-up is still superior when the uncertainty is factored in is a question that deserves careful consideration”

(unmarked quotes are from Thinking, Fast and Slow by Daniel Kahneman)

Culture as a counter-point
Creating a company’s culture is an intuitive process (and doesn’t seem to be considered in ‘academic choices’). Therefor, making decisions based on this might result in a less academically effective choice, but its benefit is that you will be surrounded more by people who foster a thriving culture. This is very valuable when creating an environment that cultivates innovation, and in creating a safe space of mutual respect.

Even though google collects all the data it can to make their hiring decision, it still cannot find a correlation that is an accurate predictor of future performance of a new employee. However, general mental ability (GMA) is said to be  a strong predictor of performance for a wide range of jobs. The correlation between GMA and job performance ranges from between .56 and .85 and GMA increases in predictive power if the job is more complex.

Alternatives to Interviews:
1) after a successful, project-based interview process, accept someone on a 1-3 month trial period (aka internship, fellowship)
2) staff-up weekend: like start-up weekend, but for a job. went to one this past week. Its implementation was awful. But it seems like an alternative idea.
3) loosely related to staff-up weekend, is Paul Graham’s view of Hiring is Obsolete. In it, he talks about start-ups as an improved way to initialize employment: work with a team to build something users want, and then make a company in the space pay market value for both your team and your product.

In addition, one could look at what the interviewee forgoes in order to apply for the opportunity. What would they do if they don’t get the job? This opportunity cost is estimated by the value of the alternatives (salary range, competitiveness, social benefits)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s