The Secret to Creating the Best Job Screening Test
Posted on April 22, 2020 by P.K. Maric
The average corporate job posting attracts 250 candidates. How do you quickly figure out who to interview? The traditional method is to eliminate most candidates based on their resume, but this is a bad idea.
Most people don’t know how to write a good resume, even if they do have the relevant knowledge and experience required for the job. More importantly, those in charge of reviewing resumes are also terrible at it, and their judgments are about as accurate as flipping a coin. Even if they were able to accurately evaluate resumes, there is another problem — resumes don’t predict job performance well.
Test Basic Skills First — Before the Interview
We recommend pre-interview tests as the first step in the hiring process. These tests should be short, with only 2-3 questions, and should eliminate unsuitable candidates who don’t have the core skills required for the job. The questions should be so simple that even beginners can solve them because, despite how easy they are, most candidates will still fail.
For example, many people who apply to programming jobs can’t program at all, even people with Computer Science degrees! That’s why it’s common to use the FizzBuzz question in programming tests. It’s a simple task that asks candidates to write a piece of code that prints numbers from 1 to 100. The program should print “Fizz” for multiples of 3 and “Buzz” for multiples of 5; for multiples of both 3 and 5, the program should print “FizzBuzz.” Most programmers can do this in a couple of minutes, yet many people that apply to programming jobs can’t do it at all.
You can use FizzBuzz if you want to, but any simple task will do. Like this one:
Fix the bugs. … public static double Average(int a, int b) { return a + b / 2; } …
You’ll be surprised how many candidates fail such basic tasks. Still, it will quickly narrow down your selection and make further screening and interviews easier.
Test Job-Relevant Skills, and Nothing Else
After this initial pre-selection test of essential skills, you can create another test that evaluates skills in more detail so that you can compare candidates. You should only test job-relevant skills. Forget brain teasers or anything else that isn’t directly related to the job. Google was notorious for using brain teasers such as “How many piano tuners are there in the entire world?” They discontinued this practice when they realized that questions like these are utterly useless.
Use practical tasks that represent the kind of work a candidate would do on the job. Work-sample tasks like these have the highest validity when it comes to predicting job performance. But don’t fall into the trap of thinking that theoretical or anecdotal questions work this way because they don’t. They’re time-consuming and ineffective. If you’re hiring a customer service representative, would you ask them to tell you about a time they had to deal with an angry customer? Or would you give them an example of a message from an angry customer and have them draft a reply? The second option is much better because you can directly see how they apply their skills to a sample of actual work:
You can try answering this question here.
If you choose the former option and ask a candidate to tell you about their experience with an angry customer, they’ll often make themselves look better than they are. That doesn’t mean anecdotal questions are useless, but they’re better suited to interviews, not screening tests.
There is also a difference between what a candidate knows and what a candidate can do. Someone may recall everything they learned at university, but it doesn’t matter if they’re not able to apply that knowledge at work. A job screening test should measure performance by focusing on the practical application of knowledge, or Level 3 and above in Bloom’s Taxonomy.
Almost all questions on TestDome are work-sample tasks. For example, we ask candidates to implement a function in Python, create test cases for QA, or make a spreadsheet formula. Questions like these test if a candidate will be good at work by giving them a sample of the actual work to do. It’s a simple and effective idea, and you should keep it in mind when creating custom questions.
Set Appropriate Time Limits
Don’t make your test too long. All questions on TestDome have a time limit. You can increase the time for each question up to 60 minutes if needed, and add an extra 200% time buffer on top of that. But you can’t remove the time limit. Time limits are important because candidates need to be able to complete job tasks within a reasonable time frame. Candidates are also more likely to complete a test if it doesn’t take too long. An initial pre-screening test should take roughly 10-20 minutes, while a more in-depth skill test can take about 45-120 minutes.
Most candidates don’t like long assessments that take up too much of their day. They will likely refuse it if they see that it takes 6 hours, or if you give them a project that takes over a day to complete. For some reason, many companies like these kinds of lengthy assessments, especially when it comes to programming. They likely feel that it’s a better measure of skill, but it just measures how much bullshit candidates are willing to put up with. Many will see it as a company asking them to do work for free, and some even argue that they should be paid for their time if a screening project takes more than 4 hours.
The shorter your test is, the better. 2 or 3 questions per skill should be enough to evaluate how good a candidate is in that specific area, and your test shouldn’t take more than 2 hours. If you think that an extended test is necessary, then explain to your candidates why it has to be so long. But don’t be surprised if some of them reject it.
A Note on Validation Studies
Some customers ask if we’ve done a validation study, as defined by The U.S. Equal Employment Opportunity Commission (EEOC). We haven’t, because a validation study needs to be done against a job description and cover the whole test. We don’t control either of those things. We provide a question library from which employers choose a set of questions they want to use. Even our premade tests can be modified to suit your needs. We also provide functionality for employers to create custom questions.
The reason for EEOC’s regulations is to prohibit employers from using tests that discriminate against people based on race, religion, sex, disability, or national origin. The way we fight that is by focusing on work-sample tasks that have high predictive validity. This way, it’s easy to see who succeeds or fails based on their own merits, regardless of which group they belong to.
To make sure this works, we have a quality assurance process in place for our questions. Apart from the author, at least two independent subject matter experts (with interviewing experience) have to review the question to ensure that it’s testing an essential skill. After that, we present each question in a test setting to several candidates with experience in the relevant field. We release the question if a reasonable number of them solve it completely. Once released, we monitor score statistics and feedback and take appropriate actions where needed.
Conclusion
The goal of a screening test is to save time for both you and your candidates and to increase the quality of candidates you invite to the interview. Whether you are choosing from a set of questions or creating your own, the tasks should be relevant to the job. The tests should be reasonably short, and you should send them out as early as possible in your hiring process. This will help you find better candidates faster, but it will also give your candidates a fair shot at landing the job.
