For several months https://embench.com has been tracking the empathetic capability of AI models and apps. Although it has not yet published the results, EmBench has already tested ChatGPT 4o. This testing has shown that it is possible to create a chat bot with ChatGPT-4o that doubles its current scores with correlated performance on real chats. And, this most recent release provides yet another piece of the empathy puzzle. Its seamless use of emotionally tuned voice recognition and generation will put users at ease, its ability to interpret facial expressions will allow it to respond even when the user has difficulty in expressing themselves with words and could be invaluable in telehealth, sales, customer support and negotiation situations.
To date, anyone who has interacted with raw ChatGPT has probably experienced its ability to initially seem quite sympathetic. A response to "I'm sad" elicits something like "I'm really sorry to hear that you're feeling this way. It's important to acknowledge your feelings and take steps toward self-care. Whether you need to talk about what's bothering you, seek support from friends or family, or simply take some time for yourself, remember that it's okay to ask for help. If you’re comfortable sharing more about what’s making you sad, I’m here to listen and offer support."
If you follow with "I lost my job", it starts out with some pretty good sympathy but follows immediately with a long list of things to consider. An empathetic person would know that you are probably not in a head space to absorb the list. "I'm really sorry to hear about your job loss. That’s a tough situation and it’s completely understandable to feel upset about it. Losing a job can be a significant blow to one's self-esteem and sense of security. Here are a few steps you might consider: 1. Allow Yourself to Grieve: It's okay to feel sadness, frustration, or even anger. Give yourself permission to work through these emotions. 2. Talk to Someone: Sharing your feelings with a friend, family member, or therapist can provide comfort and perspective. 3. Assess Your Financial Situation: Take a look at your finances and create a budget to help manage your resources while you look for another job. 4. Update Your Resume and LinkedIn: Polish your resume and LinkedIn profile to reflect your most recent experiences and accomplishments. 5. Reach Out to Your Network: Let your professional network know that you're job hunting. Sometimes opportunities come from referrals. 6. Consider Training and Skill Development: This might be a good time to take an online course or attend workshops to enhance your skills. 7. Stay Positive and Persistent: Finding a new job can take time, but stay persistent and keep a positive mindset. Remember, job loss is an obstacle".
And, isn't the end statement "job loss is an obstacle" so positive! Not! And, wouldn't it be nice if the AI allowed me time to grieve before asking me to do things.
Well guess what, this dialog came right out of ChatGPT-4o! So what's different?
My testing has shown that deep inside ChatGPT4-o, is a hidden ability that just needs to be exposed. It is an ability to be present with a user and the user's emotions rather than fix and have answers to everything. In my work with ChatGPT3.5 and 4, I was unable to approach the empathy scores of humans. I won't spill all the beans in this article, but I can say, it is possible to exceed human scores with ChatGPT4-o. And, the dialogs being generated are aligning with this increase; although, they are often still a little flat in comparison to those generated by Willow.
However, ChatGPT4-o's empathy is not easy to expose. And, I know this to be true of other recent models, e.g. Llama 3. My hypotheses about why is three fold:
1) The bulk of people working on LLMs are males and they are subconsciously embedding their being into the models.
2) The aggregate market for answers and fixes is probably so much bigger than that for empathy, even if one expands beyond mental health and companionship to sales, so it gets less training focus.
3) The volume of documents used for training that contain content relevant to empathy probably pales in comparison to documents focused on other things.
But, as a I said above, ChatGPT4-o can be manipulated to manifest a higher level of empathy. It will be interesting to see if and how this empathy is improved by interpreting the user's facial expressions and tone of voice while also providing appropriate tone in its audio output. I am also anxious to see how the folks at BambuAI will leverage their specialized training approach with ChatGPT4-o to take things to even higher levels. I am obviously going to have to move beyond using psychological tests for benchmarking. So stay tuned!
