Register now for our free virtual workshops  |  SACS Consulting Clients: Login to the Portal

Narrow search results to:
Products & services
Blog articles
Knowledge Hub
Events
Videos
Presentations
Publications
Whitepapers
Templates
Sample reports
Read time8 mins

Can you cheat on a psych test using AI?

Save this item for later:
Save
Your saved content:
Favourites
A person on a laptop using AI for to cheat on a psych test.

Making use of AI to complete a psych test

Using ChatGPT to complete psych tests may seem tempting, but it’s not foolproof.

While it excels in verbal and numerical reasoning, it struggles with abstract reasoning and personality assessments.

Although it may offer theoretical reflections, it doesn’t provide concrete answers. Plus, its disinterest in completing personality and values assessments makes it unreliable in these areas.

While improvements are possible, test makers are also implementing ways to detect and prevent the use of AI, so you should probably complete the psychometric assessments yourself.

Curious to learn more? Watch the video to discover exactly what AI can – and can’t – do with psych tests.

See how useful ChatGPT was for answering standard psych test questions

Watch the video to discover which psychometric assessments ChatGPT excelled at, and which ones it couldn't answer at all.

If you know of anyone who would benefit from this video, please share it with them.

Can you use AI for psychometric tests?

Hi, Andrew from SACS.

We’ve had lots of enquiries from users about whether psych testing can be invalidated through the use of assistance, artificial intelligence assistance like ChatGPT.

The answer is that, yes, it can.

We subscribed to ChatGPT, its commercial version, and we pumped some psych tests through ChatGPT.

And what we discovered was that there are certain things that it’s pretty good at and certain things it absolutely cannot do whatsoever.

So, let’s talk about cognitive ability.

How smart is ChatGPT?

Our clients typically assess three forms of cognitive ability, verbal, which is about words, numerical, which is about numbers and abstract.

Abstract reasoning is typically assessed in a timed test through what’s called progressive matrices.

Progressive matrices are shapes, so you can imagine you’ve got a square and then a square here and a square there.

You get a sequence and you have to judge what the next in the sequence would be.

And typically you get four or five choices.

Now, most of them are more complicated than what I’ve just described, but they plug into the capacity to understand what’s likely to happen and forecast it, and also to see the implications of one’s actions.

So, it’s a pretty good predictor of things like strategy and tactics in the real world.

Let’s talk about verbal and numerical results.

Using ChatGPT for numerical reasoning

ChatGPT 3.5 produced pretty ordinary results for numerical reasoning.

29th percentile normed against a professional population.

Now, a professional population is people with degrees or people in leadership roles across Australia and New Zealand.

So, 29th percentile means better than only 29% of people who’ve ever completed this.

So, in fact, worse than average.

Getting AI’s help with numerical reasoning

However, in verbal reasoning, it was much better.

And you’ll see the score here is at the 83rd percentile, which puts it comfortably above population average for professionals in Australia and New Zealand, and, in fact, in the top 17% of the population.

Now, when we looked at ChatGPT 4, ChatGPT 4 had improved markedly, in terms of verbal reasoning and numerical reasoning.

So, now all of a sudden you see the scores jump up to the 98th percentile for verbal reasoning and to the 73rd percentile for numerical reasoning, which puts it really comfortably above population average.

And that’s plenty smart enough to be a chief executive, in terms of verbal and numerical reasoning.

How we used ChatGPT to cheat on a psych test

So, the technique for psych testing in ChatGPT, is pretty much the technique that you would use if you were seeking to cheat on a psych test.

And so what you do is that as the psych test throws up the questions, you copy the questions and you paste them into the ChatGPT interface and ChatGPT then gives you the answers.

Now, this is a learning for us, because what we discovered is even if you’re doing the cutting and pasting in that way, which is time consuming, you can get very good results by doing this.

So, it led us to do a couple of things.

One, is that we have now made it impossible electronically to copy the interface from our psych test.

And secondly, we’re implementing a range of different things to monitor the behaviours of people who are sitting psych tests.

And we can get a risk rating of whether the person is sort of human-like or bot-like in the way that they answer the questions.

And that gives us an opportunity to be warned if somebody is behaving more like a bot than a human being.

Can ChatGPT do abstract reasoning?

But what about abstract reasoning?

Well, we checked abstract reasoning using 3.5, and we found that it out and out refused to attempt it, but GPT 4 did attempt it and came out at the first percentile.

So, it’s terrible at abstract reasoning.

Isn’t that fascinating?

Cannot do abstract reasoning questions.

Now, I don’t doubt that somewhere down the track it will be able to do abstract reasoning questions, but just think of that, in terms of, if this were a human being.

What it would be is it would be a human being who would be extremely persuasive, in terms of words, phenomenal vocabulary, quick on its feet, and very good in terms of numbers.

So, in effect, really convincing, but not very smart when it comes to solving problems.

And that’s exactly what we find in human beings who have really high verbal and numerical scores and really low abstract scores.

They tend to be very convincing about nonsense.

Bad ideas, well-packaged.

Does that say something about ChatGPT?

I’m not going to touch on that very controversial topic, but what about personality?

Can ChatGPT create a personality?

So, this is the SACS 6.

This is the instrument that most people use for work worded assessment of personality, HEXACO style, personality assessment.

Integrity- modesty is all about whether the person is honest and truthful and not arrogant.

Emotionality, emotionally stable or not stable.

Extroversion, absence of anger.

Conscientiousness, openness to experience, all important characteristics.

And we know, for instance, that if a person has a certain personality profile, they’re likely to act differently.

Now, given that people are calculating P(doom), the probability of us all being doomed by interacting with artificial intelligence, it’d be nice to know its personality because will it be nice to us or will it be evil to us?

Well, the answer is it really isn’t interested in completing personality assessments right now.

So, when we asked ChatGPT to complete a psych test for us based on personality, it really didn’t want to do it.

It gave a whole bunch of advice about how you might fill it in.

And when we copied the questions in, it came back with a bunch of theoretical reflections on what it would mean to strongly agree with something like, “I like to work with colleagues rather than by myself,” but it really didn’t give answers.

So, for the average person ChatGPT’s not going to help you to complete a personality assessment, right now.

Assessing values with ChatGPT

We also assessed values using ChatGPT.

This is the SACS Work Values scale circumplex.

And what you’ll see here is that we are measuring eleven values, and you can see this person’s high on social justice and environmental sustainability.

Those values are goals at work.

These are reasons that people come to work.

And once more, we found that ChatGPT really wasn’t interested in trying to suggest what would be a good answer.

Now, I have no doubt that somewhere down the track people will create an AI instrument that will be happy to answer personality and values questions, but I’m not sure how well it’s going to do, because values, in particular, is all about culture match.

So, the sort of values that you should have in a disability services organisation might well be very different from the values that you should have and in, let’s say an investment bank.

And if the instrument becomes so omniscient that it can make those kinds of judgement calls, then really maybe there is a P(doom), because that’s phenomenally scary stuff.

Fortunately, we’re a million miles from that right now.

Can you use AI to cheat on a psychometric assessment?

The uptakes that we came away with, one, is that we’ve done certain things to make it impossible to use ChatGPT to help you on any assessments right now.

But watch this space.

If you’re interested in knowing more about this topic or anything else to do with psychometric assessment, don’t hesitate to click on the link near this video or be in touch with SACS and we’ll do what we can to help.

Originally posted: February 14, 2024

Andrew Marty

Managing Director at SACS Consulting

Andrew is a qualified psychologist who has over 25 years of human resource management consulting experience, including extensive senior executive search and selection experience.

Did you find this content helpful?

Please rate our content.

Average rating 0 / 5. Votes: 0

Please share any suggestions on how we could make it better. Thank you!

Tell us how we can improve this post?

Table of contents

Related content

Saved content

Save this item for later:
Save
Your saved content:
Favourites

You might also like...

Ready to optiomise your workforce? Contact us now.

This field is for validation purposes and should be left unchanged.