• News |  
• Articles |  
• TYDKYDK that AI knows about you

How well can an AI chatbot measure your personality?

A new study on Artificial Intelligence, or AI, led by Psychological Sciences Professor Jinyan Fan finds that an AI chatbot can infer personality traits as well as or better than traditional self-report personality measures. In “How well can an AI chatbot infer personality? Examining psychometric properties of machine inferred personality scores,” published in the “Journal of Applied Psychology,” Fan and researchers from four other universities used an AI firm’s cognitive AI chatbot to infer personality and examine psychometric properties—reliability and validity—of machine-inferred personality scores. Fan's work was also recently featured in I/O at Work, a website that helps to bridge the gap between I/O research and its application in the HR world (and beyond).

What did this study find? Why is it important?

The overall conclusion is an AI chatbot can infer personality pretty well based on a series of evidence. For instance, to verify or to validate a psychological assessment, we look closely at the nature of psychometric properties of machine scores. All that evidence suggests yes, the machine score seems to demonstrate reasonably good reliability and validity, based on well-established criteria in psychometrics.

The big takeaway is the traditional assessment has some downsides. For instance, it's easy to fake. If you're applying for jobs and your potential employer asks you to complete a personality test as part of the application process, because your motivation to get a job offer is very high, you try to put the best foot forward. If you do, that damages the validity of traditional assessment. The AI approach is not easy to fake because it’s less transparent than the traditional approach. You don’t know which features are important, so it’s very difficult for lay people to figure that out.

In the AI approach, you just go through a virtual interview. So, we have some confidence it’s more difficult to fake than traditional assessment. It’s a two-way communication back and forth, the chatbot asks you a few questions and may even make small talk and show certain levels of active listening. It feels more like talking to somebody naturally through two-way communication, so it’s a better experience.

What are the real-world implications of these findings?

I can only speculate that a little bit because I'm in the field of human resource management. I study psychology, how to apply knowledge of psychology to the workplace. One such area is human resource management or talent assessment. You can select people based on their machine scores instead of having them going through the traditional assessment. Or, you can have a group of current employees chat with the AI chatbot, figure out their personality scores, which areas they need to improve on, and you can then send them to different training programs. That's one major area of application.

Another application is in marketing. If the chatbot can chat with a client for a few minutes, maybe the algorithm can infer that specific client's personality and then the chatbot can have a better way of dealing with clients’ needs, and that will make the marketing effort more efficient. These are just a couple of examples.

How will this change future research on personality?

I think you will see more and more research focusing on this machine learning-based approach to personality assessment. In other words, my prediction is over time, AI-based personality assessment will dominate the traditional assessment, particularly if we can further establish the psychometric property of the machine scores. In other words, if machine scores can replace self-reported personality scores, then suddenly, the major implication is we don't need traditional assessment anymore. That's a dangerous idea. It's not here yet but moving forward, that could be a real possibility.

What does this study implicate about the future of AI?

It's just another application of AI in a different field. You can see AI is being applied to many, many different fields, even biology, sports games, et cetera. And now it comes to human resource management. I would predict you will see more and more AI in our daily life, and it will replace some of the traditional working methods or working people, but also it will create new opportunities for people who specialize in data science and things like that. So, I can see that coming.

About Jinyan Fan:

Jinyan Fan is a professor in the Department of Psychological Sciences in the College of Liberal Arts at Auburn University. His research expertise includes artificial intelligence, newcomer orientation and socialization, cross-cultural adjustment and training, personnel selection and adult social self-efficacy. Fan’s work has appeared in the “Journal of Applied Psychology,” “Journal of Management,” “Journal of Organizational Behavior” and more.

Transcript

Speaker 1:

Who doesn't love a good personality test? If we're talking about old reliable Myers-Briggs, I'm an ESTJ. The new kid on the block, the Enneagram says that I'm a 3 wing 2. And the DiSC assessment says I'm a captain. Whatever that means. Personality tests can provide some great insights. I think those three tests I mentioned are pretty spot on for me. But the thing that each test has in common is that the person being tested is self-reporting their characteristics.

Dr. Jinyan Fan:

I mean, in a traditional way, usually you talk about self-report questionnaires, right?

Speaker 1:

That's Dr. Jinyan Fan, a professor in industrial and organizational psychology at Auburn University.

Dr. Jinyan Fan:

You're presented many, many statements. You look at each statement and then decide whether this is like me or this is not like me, strongly agree, strongly disagree. You do those things. And for those kind of traditional assessment, we know items are developed specifically to measure different personality dimensions. That's why we know we have the confidence, yes, these items measure these different personality dimensions.

Speaker 1:

When Dr. Fan reconnected with a friend from college, he learned about a new tool in personality testing, one that was almost too good to believe.

Dr. Jinyan Fan:

One of the co-founders of Juji company, we were undergraduate school buddies. We check on each other what's going on. And he told me he went to IBM and then finally he started in his own company with another person. And they specialize in developing AI based chatbot system. And they have even have a model that they can infer a person's personality based on chatbot thing.

I was very surprised because I study psychology, I study personality. So I was like, how do you know? Machine scores. In other words, the scores, personality scores inferred by your algorithm actually measures personality. Do you have any evidence? Well, he said, "Not much. And we believe they measure personalities." I said, "Okay, let's find out. Why not?" We do a study trying to examine the psychological properties of machine scores.

Speaker 1:

Dr. Fan wasn't sold on the idea, so he did what any good researcher would. He created a study to find out if an AI engine could predict personality.

Dr. Jinyan Fan:

So for any machine learning or AI approach studies, you have two major stages. The first stage is called model training. We get over a thousand Auburn students undergraduate. They have done two things. First, they have complete a self-report, traditional personality test, IPIP-300. At the same time, they also chat with the chatbot, Juji's chatbot for 30 minutes. Now, we use natural language processing techniques to extract many different language features and use this as a predictor to predict self-reported personality scores. The result is a bunch of model parameters. Okay? Then that's first stage.

The second stage is we also have these people completed these two tests and also they allow us the permission, give us the permission so that we can get these students' GPA from the registrar. They also nominate some of their friends, and we contact the friends and have them complete a peer-rated college adjustment thing. And then we have two sets of personality scores. One set is self-report, basically traditional assessment scores. Another set is machine scores. We compare which scores can have a higher predictive power to predicting GPA and the peer-rated college adjustment.

It turns out both self-report scores and machine scores can predict these important two outcomes. In addition, in some of the analysis, machine scores can go beyond and above what self-report scores can predict, which we call incremental validity. And that's important because that means machine scores bringing to the table something, some new insights that the traditional assessment does not have.

Speaker 1:

Those results were recently published in the Journal of Applied Psychology.

Dr. Jinyan Fan:

I think you will see more and more research focusing on this machine learning based approach to personality assessment. In other words, my prediction is over the time AI based personality assessment would dominate the traditional assessment, particularly if we can establish a psychological property.

Speaker 1:

Dr. Fan says one of the biggest benefits of using AI to predict personality is how difficult it is to fake.

Dr. Jinyan Fan:

The traditional assessment has some downsides. It's easy to fake. If you're applying for jobs and your potential employer will ask you to complete a personality test as part of the application processes, because your motivation to get job offer is very high, you try to put the best foot forward. We call it faking. You can faking good. If you're successfully in faking good, that damage the validity of traditional assessment. So the traditional assessment becomes useless because everybody almost have high scores across all these dimensions. And it's difficult to differentiate very good candidate and not so good candidates.

Whereas the AI approach, it's not easy to fake because in a traditional approach, items are very transparent. So they ask you this question very explicitly and you know how to fake good. But in AI approach, you just chat. You just going through virtual interviews with the AI chatbot. You don't know which language features are extracted. You don't know among many, many language features which are important, which are significant predictors. I mean, those parameters are used to infer to score personalities. So it's very difficult for lay people to figure that out. And so we have the confidence it's difficult to fake.

And plus, another thing is the traditional assessment, usually you have to complete over 200 items. Pretty tedious in one sitting situation. Pretty tedious experience. Whereas you chat with a chatbot, you go through a virtual interview, it's two-way communication back and forth. Chatbot ask you a few questions and you respond, and chatbot might follow it up. Sometimes they may even make some comments, small talks. So it feels more natural. A two-way, communication, talking to somebody. So that's a better experience.

Speaker 1:

As Dr. Fan looks to the future, he sees potential for many different use cases for this AI tool.

Dr. Jinyan Fan:

I can only speculate that a little bit because I'm in the field of human resource management. Right? I study psychology, how I apply knowledge of psychology to workplace. So one area is human resource management. Our talent assessment, talent management. So you can select people based on their machine scores.

Again, instead of having them going through the traditional assessment, you send them for a chatbot, chatbot virtual interview, and you can get their machine inferred personality scores, and then you can rank order them, make selection decisions. Sometimes you can have a group of current employees chat with the chatbot, figure out their personality scores, and figure out which areas they need to improve on, and you can send them to different training programs. So that's one major area of application.

Another application is sometimes in the marketing areas, if the chatbot can chat with a client for a few, I don't know, 10 minutes, maybe the algorithm can infer that specific client's personality, and then the chatbot can have a better way of how to deal with how to address this type of personilty client's needs, and that will make the marketing effort more efficient. These are couple of [inaudible 00:07:58].

Speaker 1:

The full report on Dr. Fan's latest work is available now from the Journal of a Applied Psychology. Thanks for joining us for another episode of the Things You Didn't Know You Didn't Know podcast. Make sure to subscribe on whatever platform you listen to podcasts and give us a five-star review. It helps us out more than you know. We'll see you next time on the Things You Didn't Know You Didn't Know podcast.

Speaker 3:

Things You Didn't Know You Didn't Know is a production of the College of Liberal Arts at Auburn University. To learn more about our faculty's life-changing research, visit cla.auburn.edu.

Tags: Psychological Sciences Research