Emotional AI: More harm than good?

Emotional AI: More harm than good? | GZERO World

Generative AI mimics human-generated text, images, and video, and it's got huge implications for geopolitics, economics, and security. But that's not all - emotionally intelligent AI is on the rise.

And sometimes the results are ugly. Take the mental health nonprofit, KOKO, which used an AI chatbot to support counselors advising 4,000 people who were seeking counseling. The catch: The patients didn't know that a bot was generating the advice they were receiving. While users initially rated the bot-generated responses highly, the therapy lost its effectiveness once the patients were informed that they'd be talking to a fancy calculator.

The real question is: When does emotionally intelligent AI cross the line into emotionally manipulative territory?

This is not just a concern for virtual therapists -- politics could be impacted. And who knows, maybe even your favorite TV host will use generative AI to convince you to keep watching. Now there's an idea.

More from GZERO Media

Ukraine's President Volodymyr Zelenskiy gives a press conference in Kyiv, Ukraine, on Feb. 19, 2025.
TETIANA DZHAFAROVA/Pool via REUTERS

The war of words between US President Donald Trump and Volodymyr Zelensky has hit a new low, with Trump labeling the Ukrainian president a “dictator” who “has done a terrible job.”

German conservative CDU candidate for chancellor Friedrich Merz attends a campaign event in Vechta, Germany, on Feb. 19, 2025.

REUTERS/Carmen Jaspersen

The CDU/CSU is very likely to win, making Friedrich Merz the country’s new chancellor. But he’s likely to lead a coalition government with a weak mandate, in part because he has vowed to reject any cooperation with the AfD.

A Chinese People's Liberation Army Navy Harbin Z-9 helicopter sits on CNS Yulin during a display of warships ahead of an exhibition at Changi Naval Base in Singapore on May 18, 2015.

REUTERS/Edgar Su

A Chinese naval helicopter flew nearly 10 feet from a Philippine patrol plane on Tuesday over a contested reef in the South China Sea, escalating tensions with Manila and Washington in the airspace over international waterways Beijing claims as its own.