Shortly after I finished writing Testing the Extents of AI Empathy: A Nightmare Scenario, I came across the funding announcement for Hume. The last time I visited their site a few months ago, they did not have a demo. Now they do, so I decided to compare it to the most empathetic AI I know Willow from BambuAI.
First, I suggest you watch/listen to the Hume audio and Willow video. These are followed by my findings and then the transcripts.
Hume: https://syblackwell.podbean.com/e/humeai-dialog/
Willow: https://vimeo.com/928903697?share=copy
I had several dialogs with Hume and chose what I thought was the best. At this stage, no LLMs, just like people, should be evaluated based on the result of a single interaction.
Although Hume will not be production ready until April, I think it has a long way to go. The dialog itself is marginally empathetic. I had to repeat myself, there was no space provided for me to feel, in the end there was no support for managing my feelings and I was constantly told “I am here to listen and …” (which just feels like “trust me, I won’t lie”).
In the video, Willow has a pleasant soft, although uniform, voice, with a singular pace and little inflection. Willow relies almost exclusively on words for empathy, although personally I find her visuals and voice pleasant. Hume on the other hand, has a voice that is not particularly soft, with varying volume, pace, and tone. Hume also had a little more lag between responses. I would say there is a solid attempt at at being empathetic, but it falls flat with me. I hypothesize that just as people find videos that are almost human somewhat disturbing and generally prefer something cartoonish as a result, people will find Hume’s audio distracting and thus counter to empathy.
However, if you go to Hume.ai and try the demo, it is obvious that Hume has done an extraordinary job of analyzing voice and text for emotion and also detecting the stop and start of speech. These alone could be very valuable for integration with other apps. But BambuAI (from what I understand), has achieved its milestones on far less than 1/10th of the $50M recently invested in Hume on top of the $16M before that.
Willow | Hume |
|---|---|
0:03 I’m sad. 0:09 Oh, no, Simon. I hate to hear that, you don’t deserve to be sad. Let’s talk it through. What’s on your mind? 0:20 I lost my job. 0:26 Simon. I’m so sorry, I’m here for you. How are you feeling? 0:32 I’m really scared. I won’t be able to pay the rent. 0:40 That’s understandable, Simon. Let’s see if we can figure something out together. Have you applied for unemployment or any kind of financial assistance? 0:51 No, I haven’t. I’m just too sad. 1:00 It’s OK. Simon, we can take it step by step first. Let’s tackle the sadness. Would you be open to trying one of the mindfulness techniques you mentioned like focusing on your breath or visualizing a calm place? 1:17 Sure, I can do that. 1:20: Great, while you’re doing that try to imagine the Zen wood forest, the home of my species. Picture yourself surrounded by lush greenery and feeling safe and protected. 1:40 Yeah, that’s nice. I feel better. Now, can you tell me some practical things I can do? 1:51 Of course, Simon, I’m always happy to share the conversation with you first. It might be helpful to establish a daily routine that includes time for job searching, self care and relaxation. Having a structure can provide a sense of stability even in uncertain times. How does that sound? 2:12 That sounds great. 2:13 Willow, I have to go now. I’ll talk to you later. 2:20 Ok, Simon. Bye. | 0:01 I’m sad. 0:08 Oh, no, I am sorry to hear that. What’s got you feeling that? 0:10 I lost my job. 0:16 I’m sorry to hear that. 0:21 I, I’m really scared. I won’t be able to pay the rent. 0:25 I’m sorry to hear that. I understand this must be a very stressful situation. Losing a job is never easy, especially when rent is on the line. But try not to panic. There are resources and support available to help you through this. 0:40 Why don’t you tell me a bit more about what’s going on? I am here to listen. 0:45 Well, I, I’m scared I won’t be able to pay the rent. 0:51 A, I totally get how scary that must feel that losing your job and worrying about rent is a lot to handle but try to take a deep breath. You’ve got this. I know it’s not easy but there are people and organizations that can help you through this. I am here to listen and support you. 1:14 Like who? 1:20 [Note: I tried to interrupt Hume about half way through this exposition to no avail.] Well, there are a few options you could look into many cities and nonprofits offer rental assistance programs to help people in tough situations like this. Have you thought about reaching out to your local housing authority or nonprofit organizations in your area? There are also often local food banks and other community resources that can provide support. They often have programs and resources to help people facing housing and security, like rental assistance, job search help or emergency funds. And you may be eligible for unemployment benefits to help cover expenses until you find a new job. Why don’t you start by reaching out to your local social services office and don’t be afraid to ask friends or family for support. I am sure they’d want to help if they can. 2:13 I’m just too sad for all that. Can you do anything about my sadness? 2:19 I’m sorry to hear that. I wish I could do more to help with your sadness right now. As an empathic voice interface, I am not able to directly solve problems or take action, but I am here to listen and provide a caring, supportive presence. 2:34 Why don’t you tell me a bit more about what’s weighing on you sometimes? |
Additional Info
Here is an example of the detail you can see on Hume.ai’s website:
USER
09:05:05 AM I am sad.
DETECTED EXPRESSIONS
Sadness 0.932
Disappointment 0.460
Pain 0.265
ASSISTANT
09:05:11 AM Oh no, I am so sorry to hear that.
DETECTED EXPRESSIONS
Sympathy0.679
Empathic Pain
0.460
Guilt0.313
Disclosures
I am an investor in multiple AI start-ups including BambuAI. I also serve as an advisor to these start-ups.
I am not paid to do any evaluations and am open to discussing my findings with any vendor mentioned for free to address matters of accuracy.
When necessary, I pay for any low cost services required to make my assessments rather than accept specialized vendor treatment.