ChatGPT, cheating protection, and the new Interviews product
Posted on August 9, 2023 by Zeljko Svedic
The past seven months were crazy. There were many TestDome changes and many customer questions that we will explain and answer in this blog post. Let’s start:
ChatGPT
On Nov 30th 2022, an innocent, one-liner tweet by an author who didn’t bother capitalizing sentences turned the IT world upside down. It announced ChatGPT:
People were frantic. The Internet was full of reactions like “Developer jobs are going away!” and “No more juniors needed!” At TestDome, we were enthusiastic and started imagining an AI-enabled testing future. We got many ideas: prompt engineer test, AI-based adaptive testing, solving tests together with an AI chatbot, etc.
But, after the dust settled, we realized that things would not change so fast. ChatGPT and Copilot are great at coding simple problems if given detailed instructions and if there is a human programmer to fix all the edge cases. Well, creating detailed instructions and fixing edge cases is what programmers do most of the time, anyway. Programmer jobs are staying but changing and becoming more productive when writing boilerplate code. We realized that most of our customers don’t want completely new testing; instead, many TestDome customers want cheating protection to ensure candidates solve the question themselves.
You can now enjoy all our work from the last seven months in the form of advanced cheating protection and the new Interviews product.
Cheating protection
Use multiple methods in combination to prevent cheating:
Screen proctoring. When you enable test proctoring now, the candidate’s webcam and screen will both be recorded. Snapshots are taken every 30 seconds so you can quickly review them (e.g. 40-minute test has 80 snapshots like the one below). Don’t worry; candidates don’t know when snapshots are taken; they see proctoring streams constantly.
Webcam proctoring. This old feature helps in a new way. Screen proctoring detects AI tools on the main monitor, so candidates may try to use a smartphone to OCR question text and put it into an AI tool. The webcam will show if a candidate takes a photo with another device.
Copy/paste protection. Question text copying was blocked before, but now you will also get a notification if a candidate used their browser’s DevTools to override copy/paste protection (example below). Preventing copying is important because long questions take too much time to retype in an AI tool. Copy/paste protection is always enabled.
AI-resistant questions. Many questions have a new “AI-resistant” tag. That means that even if candidates manage to use an AI tool (despite proctoring) and somehow copy question text, ChatGPT will not be able to get a passing score. We automatically check all premium questions each week and mark them as AI-resistant if ChatGPT Plus could not score more than 35%. This is an additional layer of protection, so sorting questions by Recommended on the new library page shows AI-resistant questions first. Our content team is focused on expanding the number of AI-resistant questions.
Questions requiring higher-level thinking. Since a programmer’s job is slowly changing, we are changing our questions. New questions are often longer and harder, both because they require higher-level thinking and because there is more demand for seniors. New TestDome questions often don’t spell requirements but let the candidate deduce what they are from desired behavior. For example, we created questions where the task is specified as an animated GIF (see the question below) or an image (see the Book Sales question). To find such questions, filter questions by the “New” tag.
New Interviews product
TestDome Interviews is the simplest way to replace coding on whiteboards with remote interviews. TestDome customers voted it the most requested feature (see the Code Pair / Live Coding for real-time coding interviews feature request). It was more than a year in the making because it was hard to integrate audio, video, chat, and real-time collaboration on shared code. We also selected 500+ questions (out of 1300+ premium questions) that make sense in 1-on-1 coding interviews. Although TestDome Interviews is a new product, it is integrated with Screening and uses the same candidate invite credits. We think the Interview UI is both functional and beautiful; check it out:
As a side effect, TestDome Interviews also prevents cheating, AI or non-AI. You can see what the candidate is doing in real-time, ask them questions, and if they paste code from somewhere else, you will see a block of code appearing from nowhere.
TestDome Interviews is still in the polishing phase; feel free to report any issues at support@testdome.com. Every time you buy a pack, most of that money goes to the salaries of TestDome employees. You tell us what to use your money for by giving us feedback.
Let us know what you think!
