How Effective are Technical Interviews?
Pulse, the startup I work at, is looking to hire a few more developers, and so I’ve recently been thinking a lot about the hiring process and my own experiences with interviews.
The worst interview I ever had was a phone screen for an internship, during which two interviewers spewed Java trivia one-liners at me on their speakerphone. At first I felt pretty bad because I wasn’t that experienced in Java at that point, and felt that I had done pretty poorly in the interview. I later reasoned that I wouldn’t want to intern at a company that didn’t probe more deeply into how I tackled problems anyways. That is the extreme end of technical interviews, but I’ve always wondered how effective technical interviews are at selecting the right people for the job. My suspicion is that people who do well in interviews are generally fine, but that technical interviews may weed out candidates that don’t interview well but would actually do a good job.
I think of technical interviews as the kind where you go into a room with an interviewer and solve problems on the whiteboard for an hour. Rinse and repeat 4-8 times in one day, depending on the company and type of job. For those of you who feel calm and collected and confident during such interviews, I admire you. During the first few interviews I had for internships, I remember being so nervous that I couldn’t really think straight. My mind would sort of blank, and then think “oh shit, I need to solve this problem, think think think.” Mentally telling yourself to think while some guy stares at you is pretty freaking stressful, and it’s definitely not too conducive to problem-solving.
Stress issues aside, do technical interview problems map directly to on-the-job work? Many technical interviews are very theory-heavy, muddling the divide between Computer Science and Software Engineering. I don’t claim to be any sort of expert on interviews and hiring, but it seems to me that a typical interview also isn’t a great indication of how well candidates will work in a team, their coding style, how passionate they are about the company, and their work ethic.
A few months ago, I had some really positive experiences with hiring processes that I believe give candidates more opportunities to showcase their skills, while giving companies more useful and relevant feedback.
The first one that really stood out to me was at a local startup. First, I met with someone from the company for coffee to learn a bit more about the company. I was then given a coding challenge in the form of written specs for a small Android app, which I coded up in a few hours and sent back to them. They looked it over and brought me in for interviews, during which they asked me about my general design for the Android app. I was also asked more conventional technical questions and coded up solutions on the whiteboard, but it was well-complemented by the coding challenge. As a candidate, I felt that this gave me a good opportunity to showcase some of the skills that would not necessarily come across in an on-site interview, such as good coding style and design.
The other really positive hiring process I went through was at Pulse, where I ended up. I met with the engineering team (3 people) for lunch one day, and then worked onsite on some small projects for the next week. I was able to jump right into the codebase and show the team what I could do (I was also paid for the week), and when I accepted a full-time offer, I already knew how exciting and fulfilling my new job would be. A friend recently commented that finding the right startup is just like finding the right relationship. A “trial period” (maybe analogous to dating?) gives both the candidate and the existing team the opportunity to test the waters and see if the relationship is a good fit.
Both these types of processes allow the candidate to write some not-too-trivial code at his or her own pace. For software engineers who are already experienced in the job-specific platform, their expertise will shine through. For smart people who are less experienced, they have the opportunity to show that they can learn quickly and get stuff done, without having to worry about getting tripped up on syntax in an on-site interview. With the growing number of startups and companies hiring developers in Silicon Valley, and the dearth of available candidates with the “required years of experience,” this type of hiring might be an effective way to hire “outside of the box.”
I haven’t had too much interviewing experience (on either end), so I’m not sure how uncommon these hiring strategies are. For larger companies, they may not even be logistically possible. Has anyone else gone through a process that was different from the more conventional technical interviews? For those of you who have more experience interviewing candidates, how effective are the hour-long technical interviews?
Related posts:
- Why I Left Google
- Thoughts about School and Work
- How to deter developers with your craigslist ad
- My Experiences as a Female Software Engineer
- Outside of Silicon Valley
Comments (61)
Except for Pulse, I think your experience is typical. I’ve conducted a lot of interviews and have not found them very useful. I remember one interview with someone who interviewed very poorly. I was definitely not hiring him. However, I talked to someone I respected who had worked with the candidate and told me to hire him no matter how badly he’d interviewed. I did, and he turned out to be a very good developer. I’ve also hired people that were outstanding in the interview and didn’t work out at all. Most people fall in between those extremes, but I think deciding based on a lottery would have been just as effective as the interviews.
At one startup where I worked the VP of Engineering only hired based on recommendations of people he trusted. He said that he had completely given up on interviews.
The process used by Pulse sounds very good. I hope more companies try it.
Hiring through referrals is something I didn’t mention, but seems to be the best way of all. At Pulse, we are trying to figure out how to make our process more scalable, especially as the team grows.
I’m glad you were able to hire the good developer even though he did not do so well at the interview. Some interview philosophies, such as the one espoused by Joel Spolsky, would not have given the candidate a chance to prove himself.
In my last position, I interviewed many people. I did ask them to code at the whiteboard, but if I needed specific syntax, I provided a language reference manual. Most of the time, I did not ask for specific syntax; rather, I allowed them to use whatever language they felt comfortable with (as long as they could explain the language constructs to me), even pseudocode. Also, programming was neither a dealbreaker nor the vast majority of the interview. Other issues covered included having the candidates explain some of the design issues in a current or recent project, and asking if they participated in open-source projects and/or standards bodies such as the IETF.
Mostly, what I look for in a candidate is someone who can learn on their own and take initiative. In my previous job, in addition to doing software engineering, we did tier-2 support, testing, documentation, and training, so good communication skills were just as important as the ability to program.
There are also ample studies that show that stress reduces creativity and ability to think quickly, so why we think someone’s ability to be creative and think quickly will shine through in a job interview is beyond me. I personally believe interviewing is a skill, like any other, it generally takes practice, of course most people can’t (or don’t want to) practice it that often, and when they need the skill the most is when they are under the most stress. In essence what a standard technical interview shows you is how familiar someone is with the material you present them. For the most part there is a (relatively) small corpus of possible question topics and it isn’t unreasonable to be fairly familiar, at least at a high level, with most of the possible question areas. This will make you look very good in the interview as you appear to be able to quickly (and under stress) produce the ‘right’ answers; however, I don’t think this maps very well to a real world job environment.
Ryan
I’ve done a lot of interviewing and have been interviewed myself a few times so I think I have a good amount of experience in the process. There are a lot of factors involved in technical interviews which I think a lot of people miss.
*Tech interviews are usually only an hour, so the questions have to be short – this often leads to more academic questions, and questions not directly related to the work done at the company.
*Interviews only tell you so much. You won’t really know much about a candidate until they’ve been working with you for awhile. As such, many technical interviews are more about weeding out bad candidates than finding good ones.
*People embellish, cheat and lie. Just because a resume is impressive doesn’t mean the candidate is impressive. Just because they worked on a project doesn’t mean they contributed anything worthwhile to it. Most people are honest but will still bend the truth sometimes. It’s a lot harder to do that in person.
I like the idea of having a candidate work with the company for a week, but that’s not realistic for a majority of cases. Not all companies are going to be able to do this, and neither are all candidates. If a candidate already has a job, asking them to take a week off is asking a lot. I also like the idea of a “homework” assignment. However it’s possible for the candidate to cheat. It’s also asking a lot of the candidate who may have a job and a family life.
These two approaches heavily favor college students and recent grads.
It’s far from a perfect system. However it’s reasonable (or so it seems) and is easy for companies to implement.
> As such, many technical interviews
> are more about weeding out bad candidates
> than finding good ones.
Exactly. If you are lucky enough to hire through referrals, you don’t understand the scale of a problem. But if you need to hire 50 developers, you quickly run out of your connections and you have to start hiring off the market. And this is where the hell unleashes — huge majority of candidates who show up are not worth a minute of your time and a penny of a salary. Huge majority, like 90%. You then use each and every way possible to sift out as much chaff as possible, as early as possible. We use phone screening, we used written coding assignments, now switched to on-line programming tests (on Codility — really cool little service), and we manage to reduce the number of heads by a factor of 10x before we meet people for interviews (technical and non-technical). We still don’t hire all of them, but at least we have some assurance that they are worth talking to. We would be nowhere if we attempted to talk to everybody who applies.
I am in period of looking for an opportunity in startup, had done several phone interviews. Yeah, that’s painful, I admitted that I am not that good at interview, I felt every time I ends up like an idiot at all, after that, when I thought about what questions they asked, it is obvious that I shall do different, but why I could not ?
Sometime there is a kind of chemistry between interviewer and interviewee. Upon heared voice of other side, you kind of starts to image what other side looks like, and whether you like (to work/talk with) him/her …
When the voice from other side is just mono tone, I ends up very nervous, and start to wonder around …
When I felt other side is in a rush to ask tech questions, I also felt very uncomfortable …
Maybe I am just too pick or I don’t know …
I’ve had this experience as well. If I feel as if the interviewer is on “my side” I do a lot better. A few words of encouragement can go a long way in interviews.
As an interviewer, this is exactly the approach I take. I run through a long (45 minute) multi-part problem solving question. I am the candidate’s best friend. I am their textbook and compiler. Everyone gets the question right in the end. It’s HOW they solve the problem that determines whether I want to hire them, not IF they solve the problem.
For me, when I did not know the answer, I later looked up the answer. I failed a lot of interviews, but I ended up learning a lot of new concepts and ideas, which I would not have known about.
I interviewed recently with a large company in Seattle. Seven interviewers during the day, no breaks apart from lunch. I’m a very sharp guy, an experienced developer, have a lot to offer any employee, and aced my CS degree 20 years ago. But this interview left me feeling pretty hopeless about this style of technical interview process. “Think, think, think” is exactly how I experienced it. I’ve tried to think of times in my professional career when “think, think, think”, and the ability cope with that sort of pressure has actually been useful. I can only think of a few deployments, which don’t happen like they used to with the sort of practices I use these days. I didn’t get the job and wasn’t offered any explanation, and perhaps I wasn’t the right candidate. But I can’t help thinking that look, I did all those exams way back and aced my degree and did well under the pressure of whole days of exams, and exactly how do you think throwing me back in that situation is even remotely relevant to the job in hand? The idea of working for a few days in the company sounds utterly sensible, but I realize may not be possible for most employers. Now, anyway. Perhaps Pulse could share their experience, and how they manage to make it work for them.
Hey jean,
Thanks for the post =). Curious as to whether the small projects you worked for on pulse during that one week were projects that were specifically tailored towards these trials for new developers, or were they deeply integrated projects that just happened to fit the criteria?
The first day I chose between two projects. Both were small and self-contained enough that I didn’t have to have a deep comprehensive understanding of the project code, but I did have to work closely with the existing codebase. There are always little features that you want to implement but keep getting passed up for more important features =)
Great ideas here. At this point all that I hope to get from an interview process is that the candidate seems to want to work hard and has bothered to do some basic background research on the company and the job.
That in itself is a useful screen for folks who are not so good… I have talked to people who literally didn’t know what government agency he was interviewing for and people applying for programming positions who told me they hate programming and intend to do something else.
Unfortunately, it’s hard to be creative when it comes to hiring at larger organizations.
Another thing we are trying is to ask candidates to download the app (if they haven’t already) and play around with it and give us feedback, what they would change, what they might like to work on, etc. That can be pretty indicative of if they are excited about the company.
Be careful with this, if a developer sees a scary poorly defined product he may end up “running for the hills” than continuing. I went to an interview once and in the phone interview had mentioned I wanted to get away from some of the small company practices of pressing development… got to the site and was given a tour and I was ready to “run like crazy”. They were everything and more of what I didn’t want!
Why would this be a problem? If someone isn’t going to be happy working at your company; it’s best that they find out before being tied down.
Having a potential hire come work for you is great if that hire is able to do so (fresh out of school or unemployed), but how does this work for someone you are trying to pull from another job?
Interesting!! your comment: “The worst interview I ever had was a phone screen for an internship, during which two interviewers spewed Java trivia one-liners at me on their speakerphone. At first I felt pretty bad because I wasn’t that experienced in Java at that point, and felt that I had done pretty poorly in the interview. ” That’s almost exactly my experience with a government job here in Washington, I won’t mention any names. I to have been to many interviews and have also interviewed many people. 5, 10 & 15 years ago, I could always ace my interviews, 1 person, 3 or 6, did not matter, I always went in with confidence, answered all their questions and was able to ace any algorithm on the white board. But this last year I started interviewing again and it has absolutely changed dramatically. Each interview I have been to and severl 30 min phone screens have been very uncomfortable, with the exception of my most recent one at Microsoft. The interviewer was smart enough to know that my 20 years of programming was plenty of expertise and he clearly noted my ambition and passion for programming. In addition, I was extremely interested in the product as I spent several days researching it prior to the interview. This is very important as some of you have mentioned here. I was hired for the job with a quick 30 minute interview and only one algorithm on the whiteboard. During the 1st week I literally finished about 3 weeks worth of work. So to have a programmer work for a week to see if their going to be a good fit is a good idea. The issue here is that un-skilled interviewers who spend a lot or a little time preparing, or attempting to ask only questions about coding that they know is not going to let them know the programmers enormous amount of technilogical know how. This is where the company loses out on the potential of hiring a Great programmer and contributing employee. As an interviewer, don’t be so narrow mided, find a method that allows you to acknowledge the programmers large skill set, not just the few coding questions you put together and had to study yourself in order to conduct the interview. As a programmer, it is absolutely important that you are passionate about the position and programming. The interviewer will naturally pick up on this if their paying attention.
Having just gone through the gamut of looking for a new job after making a major location change (yea i know. should of done that first
this strikes home. I’ve been a developer for longer than most here and have done well, least the companies I worked for made money from my work… The first thing I had to deal with was phone screenings… I treated them like in person interviews until I realized they were asking those college exam type questions that have little to do with actual work experiences. Then I started playing their game and had all kinds of reference material on the screen when the call came. The best interviews were in person more or less a how you would deal with this and that. Plus how well you would be with the team. The experiences you had I would of enjoyed and felt would of expressed my true capabilities, rather than the dog and pony shows that I dealt with.. Oh yea.. I did accept a position 
I’m a .NET and C++ developer My worst interview involved being handed a pen and paper test and asked to provide syntax specific code examples that proved my knowledge of the language. Most of the questions involved obscure language quirks (such as write an example of operator overloading, or write code examples of the 3 types of constructor chaining, and many other things I might have only used once or twice in my entire career), all I discovered from this is that I am dependent on my development environment and (with its wonderful code completion), and anything more precise than pseudo code is difficult when using pen and paper.
The two best technical interviews I have ever had involved being asked if I could provide a copy of some code or projects I had worked on in my own time, after which I was asked if I could come in and talk through my code and answer questions regarding my approach and design with the lead developer.
As a final stage one of the firms provided me with a small project specification and asked if I could complete it in the next few days, which I didn’t’ mind doing as I was able to complete it in my own time and my own computer (with my own settings), this was much less stressful and allowed me to think clearly.
I ended up taking accepting a job from one of these companies over two other offers I had received at the time.
I later found out that their reasoning for this type of technical test was to:
1) see if I could code
2) see if I really enjoyed programming eg. people who see programming as a 9 till 5 job wouldn’t code as a hobby (outside of work) and either couldn’t provide them with sample project or provided something that had been cobbled together out of urgency.
@Jesse W – I can relate…
I don’t find any mode of selection or interview technique I have experienced so far effective for choosing a new employee, let alone a programmer.
They might as well just make it a lucky dip or lottery prize. At least it’s more obvious that they are taking a risk that way. I have heard employers say that they just throw away half the applications at random, without looking at them, if they get too many of them. Recruitment agencies mostly look for keywords in the CV and often don’t even talk to the applicants.
As the article mentions, the skills required to be a great programmer often do not show in an interview. I sometimes think I should take my laptop along to interviews, since as an experienced developer, I feel naked and more stressed without it. When you have worked in front of one for 20 odd years, it’s difficult to think without it. Some of us get to the point where conscious thinking is not even required – it just flows…
Personally I think technical interviews are a waste of time. If you have interviewed alot as I have you should know that everyone asks basically the same questions. Perfect example: What is the difference between an inner join and outer join? LOL! I know you have been asked that before! That being said I believe in hiring people on a contract to hire (3 months max) basis. This give me the time to see them in action. I much rather look at the person’s resume for inconsistancies and perform a personality interview to see if they are the right fit for the team and if they are, move forward. Technical skills are just 35% of what makes a person a good developer. EVERYONE SHOULD GO READ THE BOOK – PEOPLEWARE
Although the idea of a trial period sounds good, it only works for people who don’t already have jobs or other good prospects. Most people can’t take an offer of a one-week trial period unless they’re unemployed.
I’ve found the “take home project” to be the best balance – anyone who is interested in a job can take a few hours over a weekend to work out a small sample app.
Yeah for a small company looking for its first employees a trial period is nice if a possibility, but not very scalable as the company grows.
People seem to be concerned about “cheating” on take home projects, but you can ask them to describe their design in the interview. Also weeds out people who are not really that interested in the company and don’t want to spend the extra time on the project.
Some Good points here but i am not sure how this would fit in for finding out experienced guys as they would be working somewhere by the time of interview, so working for a week would be almost impossible.
I concur with a prior post that one should be invited to BRING source code (or relative proof of experience) and review that in detail during the interview. As for me, I’ve never been consistently perfect and quick when put on the spot, especially given that nerves / adrenaline are already peaking which (for me) can adversely affect certain areas (archived vs current) of memory.
Aside from that, the last technical interview I attended included topics with which I was very familiar … but also inluded topics that were outside of my experience and/or expertise. Regardless, I employed honesty and stated that – while I have not yet explored that area of technology – I was not intimidated by it and would welcome the opportunity to learn something new … adding that I was highly adaptable and capable. The percentage of familiar vs unfamiliar was 70:30.
I landed the job and, within my first year, grew tremendously while using my past experience to enhance my overall team. It’s been a great experience on both sides, according to my recent performance review.
I’m reading this as I prepare to go on my first technical interview in 20 + years. I fear questions designed to test my knowledge of syntax. I find that with today’s tools your knowledge of syntax doesn’t have to be perfect. If you forget a semicolon, the IDE reminds you. Code completion takes care of the other things you previously had to know from memory.
I’m hoping the interviewer is astute enough to ask questions that would uncover how much I enjoy programming. How stimulating I find problem solving. How much I enjoy collaborating with my colleagues.
I’ve had both good and bad technical interviews. The worst was an interview where they wanted me to show them how I would use algebra to determine the times at which the minute and hour hands on an analog clock would exactly overlap while two managers watched me like hawks. Brain-freeze. In my best technical interview I was asked to go through a short (200 or so line) program for a simple linked list application, fix all the bugs, and get it working. They set me down at a computer in a conference room, told me I had two hours, and walked away. It only took about an hour, and I was later offered the job.
There are three things I notice that are practiced yet quite irrelevant to finding a good candidate for a programming job.
1. Asking technical questions that are irrelevant to coding.
Those college questions are as useful as a whole in the head.
I am taking classes currently in a java II class. I have a big
handicap with these classes, my previous experience writing
working code without any theory knowledge. Since I know that
the code will work regardless of the stated theory the theory
is irrelevant to me and I have the hardest time reading and
retaining that information.
2. Information about the company. I am turning 50 this year. And
the years of work experience tells me that your company info
is mostly hype. Lies that the company will not be living up to no
matter how long I work for it. This is certainly cynical, but then,
I have yet to work for a firm that lived up to it’s own hype.
3. Being passionate about programming. Seriously? How do you
know if the person is truly passionate about programming or
just putting on a show for the interview?
I have been in many interview situations where I got excited about
the job, only to find out that the person was lying to me and that
that the job was not worth getting all excited about. So the only
way I will get excited about the job and your company is if you
give me a job. Even then, I will do so cautiously.
Question: How many of you asking those theory questions are developing new programming languages? None? While there
are some basis for theory such as OOP in which you develop
classes as the structure of the program, the majority of
programming theory is not needed unless you are actually
developing a language. So why would you ask those questions
of potential employees that do not need them.
By the way, thanks for the mentioning Codility website. Maybe
I can learn something from them.
Maybe I’ll even check out Pulse. Sounds like a company that’s as
compulsive in doing things the right way like I am.
Forgive my incoherent rant…I loved your article. While I think your hopes and experience are more for the startup domain I do think the personal touch and getting to know the candidate as a personal fit for the team as well as things like style are very important and not heavily flushed out in “typical” interview processes. I’ve been doing technical hiring to staff my teams for over 3 years now and other teams have come to me to do their hiring as well. I’m computer scientist, the super dorky kind that can still go out and party and hold a conversation, so I try to bring that to my interviews to show i’m not some bigwig looking to knock them down a few pegs. While I still do the aforementioned conventional “corporate” hour long interview I suppliment it with a few phone calls first or a brief conversation about who they are and just get them talking. I find if people are talking and i’m just listening the pressure is usually removed. This is also a good way to read people if you’re good at that kind of stuff. People end up giving away a lot of clues as to what type of person they are if they’re just comfortable and ranting away. Once the candidates comfortable I begin the technical portion. Usually you can tell right away if a persons just a buzzword freak and throws around words they’ve only heard from another person (say an architect or sr/lead engineer). Or you’ll find a person who can elobrate on what they did with said “buzzword technology”. To figure out how people think and to see if they’re honest I’ll usually throw out a related question to something on their resume but something a little more obscure to see if they’ll say they don’t actually know something or see if they give me an educated guess. Rarely do I put people to the whiteboard (i think i’ve done this once out of the almost 100 people i’ve interviewed so far). I do think a lot of people take the “microsoft” approach to interviews…I know i’ll prolly get flammed for calling it the microsoft approach but they have some really intense interviews and books on how to interview (might be worth looking into). I find that if the problems you ask people to solve are fun and relevant but don’t neccesairly have an exact yes/no right/wrong answer it will challenge them enough to get them to think but also not make them freak out. The fermi questions “how many ping pong balls can fit in the empire state building”.
I didn’t see any comments that mention the source of the problems with technical interviews.
The problem is that developers are the ones doing the interview.
The skill of a developer is expressed in delivering systems. Despite claims of being able to “communicate” that is a side skill of which just being adequate or even being below average might still might be sufficient.
Thus the very people responsible for such interviews are using a skill that they are certainly unlikely to be above average in. And yet judgements are often made based specifically on the perceived social interactions between the interviewee and interviewer.
Additionally explicit skill tests are almost universally never ‘tested’ before the actual interview. Few developers find it acceptable to deliver a system into production with absolutely no testing yet find no problem coming up with adhoc questions a few minutes before an interview that they claim will adequately judge the technical competence of someone they have never met.
This is further complicated by standard probabilities in that many developers will in fact have below average communication skills and many developers will have below average development skills. And are really unlikely to be above average in a broad range of technologies.
And lest you think this is an indictment of interviewees then note that it applies to the interviewers as well. Thus one can end up as an interviewee trying to defend a correct answer with an interviewer that with an interviewer that doesn’t know the correct answer. Or even worse doesn’t understand the question.
I have found that many developers who rely on technical questions take offense at my position because they “know” that their questions lead to better candidates. However I have never seen any evidence of that. Conversely at the places where it did seem like above average candidates were the norm the best reason I found was that it was due to the above average skills of those working in human resource departments. Which doesn’t surprise me.
I totally flubbed a Google phone screen interview. Partially cause they hadn’t told me what to expect (they’d contacted me about the job, I hadn’t applied). I did much better with YouTube (who were picking up the google leavings I guess) having then prepared. And then in the 5 hours of interviews at the YouTube offices as well. It was grueling and stressful, and I felt one of the interviewers was actively hostile.
I did find that I got questions that the interviewer actually didn’t know the answers too. They were just kind of riffing of things I said, but didn’t necessarily differentiate for me what they were expecting.
One guy (the actively hostile one) questioned the code I’d written up on the board and whether I needed to do something. I thought I did, though I was too nervous to be strong about it, and I went back home and proved it. I’m not sure they ever saw that (going through the recruiter).
I wasn’t hired partially ’cause of that guy (but also for some other legitimate reasons that they were screening for).
I’m now going to be doing some interviewing of my own, and am considering what I might do. It seems like it might need some balancing.
I just had the same situation! They called me and set up phone interview. Then they were 1 hour late in calling due to being in a conference that ran long. I was speaking with a recruiter about a particular job which was very similar to my current job. The benefit of this one was it was work from home and more flexible and challenging. Anyway, I was completely surprised by the questions asked and totally flubbed ‘why do you want to leave your current company?’ At end of interview recruiter who seemed somewhat hostile throughout asked if I maybe was more interested in a positon x or y (both of which are way below my level—I’m a senior manager and by end of interview I was being offered a mid-level job).
Agreed, i myself never found any relation ship between whatever was asked in my past inteviews and the work that i was offered after joining in all my previous and curent organizations.
And also the hypothetical pressure put on interviwee often causes his real character to stay hidden.
No selection method will be perfect. By the time you are ready to invite a candidate for interview you have two questions in mind
1. Can they (learn to) do the job. The CV says yes. You need to verify that.
2. Will they fit in the team? Will the team like them.
I was once asked to take a programming test because the client had had big problems with outsourcing to India and finding programmers that could talk a great job. So I understand why some places want a coding test especially for inexperienced people.
If they have a few years experience however you do need only check they really worked for the companies they claim to have worked for to get an answer to (1) that is probably true, but you still need to check a few points.
Question (2) probably needs a long interview. And will not be perfect. An ex-colleague of mine is excellent technically but was told by their manager that nobody wanted to work with them.
Great article Jean! If you don’t mind I’m going to quote you here- “My suspicion is that people who do well in interviews are generally fine, but that technical interviews may weed out candidates that don’t interview well but would actually do a good job.” That is so true. It’s also heartening to read the comments. I think a lot of interviewees suffer from these very stressful tech interviews, with no follow-up. I also did a trial period with my current job, that I am very happy at. It’s good for me to know, in my insecure moments, that they could have bailed then, and also for me, if I ran into something really unsettling and over my head, that I could have left, too. You also did a great job writing up this thorny problem. Having been someone who has also tried to articulate it, it’s a hard job to do.
thanks! yeah it’s an interesting topic, and I think people realize there is a problem with the status quo, but maybe don’t know what’s the best way to change it…
Interesting article. The sort of practical assessments and tests that you mentioned are, IMHO, much better than pen-and-paper coding or talking alone at identifying people with the right skills. Giving candidates take-home projects that you later review with them in an open debrief, and having ‘trial starts’ (where the candidate works within the company on a trial basis on small projects that they get paid for) are really good ways to see how a potential team member handles real work, using the actual tools that they’ll have access to on the job (including the internet, MSDN, a full IDE with Intellisense, etc). My only slight criticism of the ‘trial start’ approach is that there aren’t too many of the most marketable developers (who’ll most often currently be in work) that will have the time to devote to it, so a ‘trial day’ might be more suitable than the week you had at Pulse for most.
Yeah, the full week trial is not scalable on the company side, and often not a possibility on the candidate side. We have been trying a day trial, as that gives both sides a good idea of what full-time work would be like.
“oh shit, I need to solve this problem, think think think.” Mentally telling yourself to think while some guy stares at you is pretty freaking stressful, and it’s definitely not too conducive to problem-solving.
THIS WAS ME!
On my 3rd interview, I still thought that, but my mind then started to churn the butter in my head and tried to solve the problem at hand.
Pingback: Why The New Guy Can’t Code | MakeNoise : MakeNoise
Pingback: Why The New Guy Can’t Code | User Manual Guide
Pingback: Technology Global » Why The New Guy Can’t Code
Great article and comments. I found this from http://ecarmi.org/writing/google-internship/ which also has some great comments.
jschell is right, developers give interviews. I used to work for a well known dot.com and was asked to do interviews. The problem was, I was a great tech but awful with people. Don’t get me wrong, I was fine with the team… but am simply not a people person. Not to mention, I didn’t really have time to prepare.
For what it’s worth, interns and college grads… When I finally talked our senior guys into hiring a young person with no experience, that person ended up doing more work that two of my colleagues combined. I’d like to think we didn’t just get lucky. If your young and you have the desire to kick ass with technology, do and it will get noticed. And if it’s not noticed (financially), find another job. In my opinion, it’s all about attitude and thats what I ended up looking for in candidates.
Only a couple of commenters mentioned indirectly that you as a candidate are also interviewing the employer. Keep that in mind. Everyone in the working world might want to check out the book: The No Asshole Rule
Good article and comments.
I had worked for good startups, and could get great candidates who had explored and earned great opportunities later.
At Softoria India, it was a startup with non-IT industrialist I needed helping hand, while most of the techincal stuff to move on was known to me, and I had full support to pick up the right deserving candiate.
My first thoughts were to get a person who can have a good relation and can be trust worthy for long term, gaining techincal knowledge was the secondary as the person can pickup if he is good learner.
I got my friend in, who had been good with aptitude, attitude and loved to learn most complex quickly, he was not good in communication, and slow learner for the new technology, I had to struggle a lot to keep the person as management was never satisfied.
But in 8months time, the guy picked up quite well and was the most trustworthy and promising and risk taker, I didnt had to bother about how is doing, was the most precious employee and helped me picking the right candidates, training and motivating them.
I tried my best to get the candidates who showed the risk taking and tried to solve the problems with there limited knowledge, I used to give them the puzzle like what could be the keywords he puts in so that google gets the right result in top five.
What are the options to help the remote client if his server is having issues, and you dont have time to go and fix it.
At Cybertech India also I had free hand in picking up the candidates, and here we had the product ready, had to pickup the candidate who can follow the instruction and put his own inputs to get the fix. Now the right candidate was to have good understanding.
Most of the interviews I faced were generic kind, and if you sit on internet two days before you get the kind of company and interviews they have.
We should strategize the interviews based on the requirement and long term relations rather than for the current projects in hand, and thats what most companies look in.
Person gains confidence with more interviews, the most important I found was, the best result can be obtained in 4hours work only.
We should check for problem solving techniques, understanding the problems, knowledge depth of what they had been working on, how they use the free time, awareness of other topics, you need free and frank candidate.
Pingback: Why the New Guy Can’t Code « The Job Corner
I once did a phone interview for a minor software company,
The interviewer asked me if I had used their software (I had) and what I thought of it, I answered honestly that I had used it and that I believed it needed some substantial improivements. There was a dead silence, and it was obvious that the interviewer had lost interest.
I also worked at a place the decided to go electronic for paperwork, issuing engineers with barcod scanners with purpose written software to “sp[eed up” the paperwork process, unfortunately they never actaully asked us how to implement it with the result that paperwork was slowed down by 30% due to the poor layout and inability to replicate (when doign paper worlk for 100 identical systems on one order scanning in sereial numbers and asset numbers obviously has to be done, it was just plain ridiculous that we also had to scan in the sku code each time, and the codes for anything we had installed each time), The old system we used we just set up a template, input sku codes etc once, replicated it 99 times and the filled in the blanks. But then taht was pretty typical of the way things were done there, we were still writing it all out by hand in 2009 . . . . . .and this was a Major IT company
Hi, I’m surprised that more or less the whole thread finds interviews worthless. I was sort of co-founder of a startup that’s now part of a big company, so I’ve seen the whole thing from the interviewer’s side only. I haven’t been involved in recruiting for some years now, but back when we were a startup here’s what we did, and I think it was effective.
* Interviews were 3-4 hours long. Anything less than 2 hours felt like a waste of time.
* We asked candidates to describe a recent project they were involved in, in detail, including what it was for and why they did it that way. A surprising number had trouble expressing the purpose of their project.
* We asked just enough basic questions to verify that the candidates could tell their stacks from their pointers. Just a reality check.
* We asked thinking questions about a real (well, fancy) problem that neither side had an answer for. Example: How would you go about simulating the motion of a galaxy with reasonable accuracy and efficiency. How would you track the position of a flying probe based on a camera that looks down at the ground. We wanted to hear things like hierarchical decomposition for the particle system, either discrete feature or signal-based approaches to tracking, etc.
* We asked an in-depth question about a common programming concept like “what is inheritance for”. We wanted to hear about interfaces and maybe about extension, but definitely not about birds or penguins.
By the end of the interview both the candidate and the two interviewers were very tired, and either did or didn’t want to work with each other. We may have lost a few good candidates, notably Chinese ones who didn’t respond well to the format for some reason, but we didn’t hire any jokers.
And here are the main qualities that we were actually looking for in the candidates:
* That they were sane and not aggressive or otherwise difficult people.
* That they were not afraid of any problem being “too hard” and that they would attack it in some reasonably productive way.
* That they cared enough about the details of things to produce good work.
Jean, my experience is that technical interviews _can be_ very effective, if done right. I’ve used several successful approaches – all have already been mentioned. I measure success by our satisfaction with the technical capabilities of the people we hire. Over the last 4.5 years, we’ve hired about 150 developers and 70 testers. We’ve had about 97% satisfaction.
A lot has already been discussed. One thing that I believe is very important when designing technical interviews, is to understand what traits and skills will be selected for a given approach. Likewise, consider what will be filtered out. Every approach will favor certain people and excessively challenge others.
Next, be sure those that are favored are they types you want to hire. The technical interview bias must match those things your organization values. Otherwise you will hire the wrong people. Understand that every approach will generate false negatives: some candidates will fail that would have made good employees. Characterize that group, and determine if you can live with the false negatives your technical interviews generate.
Finally, constantly monitor and reflect on the quality and effectiveness of all parts of your hiring process.
You might find my new book, “Agile Hiring” interesting. Among many other things, I describe how to define and execute successful technical interviews.
I agree with a lot of what you’ve said, but I wouldn’t want to do the Pulse-style process, because unless you’re pretty sure going into that trial period, it monopolizes the candidate’s time too much. People aren’t just interviewing with one company. In my last job search, I sometimes had multiple interviews in the same week. If you have that trial period and it doesn’t work out, then the candidate has lost a valuable week of time that they could have been using to interview elsewhere or apply to more jobs.
At the place where I work now, in addition to the in-person interview (which was focused more on talking about projects then on writing whiteboard code), I had to do some real coding at home on my own time. It certainly ate some time but it wasn’t full-time. I probably spent 10-12 hours on it over the course of 3 days. I was able to keep on doing other job-hunting stuff, in case it didn’t work out.
Hi Jessie,
I agree the process I went through is not scaleable, and often not possible. Now we are doing a combination of phone calls, in-person interviews and mini projects either in office or during their own time, so that sounds fairly similar to your process.
A very insightful discussion! I’ve been with my company now for 6 years and just recently decided to test the waters and see what else is out there. I’ve been through 3 interviews so far and quite frankly it’s been a bit of a shock.
In my experience these “interviews” have been anything but. Instead I’m faced with a battery of tests. Oral, written and coded, mostly based on syntax and theory. I get the odd question thrown in at the end like “tell me about a project you were involved in and what your contribution was to it”. But it’s almost as though it’s asked to fill the get-to-know-ya quota.
After my third round of testing at one company I was offered the job. I declined. I just felt after the interview process it wasn’t the place for me.
Pingback: How Effective are Technical Interviews? | Stanley Kwok
Pingback: Why The New Guy Can't Code | TechCrunch
Technical Interviews are required to analyse a particular person’s ability.But my approach is that technical rounds can be written instead of Oral as during wriiten a person can relax and then think and write the answers.This will avoid the stress they are facing when the interviewer expects the answer immediately and stares at us due to which me may forget even the known concept by getting tensed unnecessarily.
Pingback: Redesigning the Technical Hiring Process | [Jean Hsu]
Pingback: 技术面试能发挥多大效用呢？ - 博客 - 伯乐在线
Hi Jean,
I somehow stumbled across your blog this afternoon and read this post. I think it’s a great piece and extremely well written. I am currently a senior in the CS program, and it’s been life changing. I definitely grew up over the past 3 years more than I ever thought I could.
I just wanted to let you know that you voiced many of the feelings and experiences that I have felt, and I think it’s awesome you are writing about it.
Keep up the awesome work, you will continue to be an inspiration!
Cheers,
Natalie Swope
Thanks Natalie! It’s been an incredibly fulfilling and challenging career so far. Best of luck to you!