What do interview questions actually measure?

The other day, I happened to be working in the back of a room where a career panel ended up being hosted. You know, one of those things where experienced industry folks gave advice to people who were just starting their careers. I mostly try to avoid these because they tend to be all anecdote and no data, but since I was already there, I stuck around.

In the first part of the career panel, someone from [redacted prestigious company] described how they got their job. They failed the first few interviews they did, but once they got the hang of the kinds of questions that are asked in interviews, they did well in their interviews and got a number of job offers.

In the second part, someone asked the person from [redacted prestigious company] about what they look for when interviewing, and the panelist described the exact same kinds of questions that tripped them up and then non-ironically described how these questions give you great insight into the motivation and competence of the candidates you’re interviewing.

When the panelist was answering interview questions, it appears that the only things these questioned measured was whether or not they knew how to play the game and were in practice at the game. But for some reason, when the panelist is asking interview questions, they think that the questions reveal deep information about interview candidates.

This is not the first time I’ve seen this set of opinions paired together. It’s not as common as the “I’m good at this style of question and I’m smart, therefore this style of question tells you how smart someone is” pairing, but I can at least understand the latter line of reasoning, even if I disagree. I don’t understand why the former pair is so common, though. While it’s technically true that the two opinions aren’t necessarily contradictory -- maybe the panelist and others sharing the opinions think that they’re a special case and that most candidates are not a special case. However, the ubiquity of the opinion that you should save the places you most want a job at for last because you’re likely to mess up your first few interviews sure makes it seem like the panelist wasn’t a special case but was the common case.