How can we make human-computer interaction more collaborative, useful, and lifelike? Look to psychology

If you sit down to have a conversation with someone, you begin to build the foundation of your communication before either of you open your mouths. As soon as two people become aware of each other they begin to form mental models, attempting to understand each other by ascribing mental states to them based on their behavior, their words, and your own previous, similar interactions with others. The better your mental models of one another, the more easily you’ll be able to understand the other person’s motivations and perspective, and the more effectively you’ll be able to communicate.

This interaction, according to Khoury College of Computer Sciences professor Stacy Marsella, is the essence of the psychological concept called theory of mind.

Stacy Marsella. Photo by Liz Linder.

An interdisciplinary professor jointly appointed to Northeastern’s Khoury College of Computer Sciences and the Department of Psychology in the College of Science, Marsella is deeply fascinated by the human mind. His research interests run the gamut from psychology to disaster relief to robotics, and over the course of many years a key question has consistently made its way to the forefront: What role can theory of mind play in creating useful human-computer interactions?

Early experiments with AI and theory of mind
Initially, Marsella’s background actually wasn’t in computer science at all—as a young researcher, he was focused on psychology and economics. He first became interested in AI as a way to create better economic models, more rooted in human psychology. But Marsella has always loved to build things, and when he discovered the potential AI had to build detailed psychological tools he was hooked.

For many years, he researched ways to build narrative-based training simulations driven by AI that used theory of mind. A narrative-based training situation takes an emotionally charged situation — for instance, a doctor informing a patient of their terminal illness — and allows that doctor to practice their approach and delivery ahead of time. The simulations can be a little like video games, in which the dialogue options you select impact how your conversation partner responds. As all video game players know, though, selecting dialogue options to advance a selected storyline bears little resemblance to real life, in which each interlocutor will have unique biases, emotions, needs, and ways of understanding the world that affect the unfolding narrative.

What role can theory of mind play in creating better training simulations? Here’s where Marsella’s work comes in. Giving that virtual conversation partner a complete mental model gives the trainee something to attempt to understand. By first crowdsourcing a vast array of potential scenarios and then using AI to weave them into a dense web of potential avenues of interaction, Marsella and his team created simulations that could go in any number of ways—but always in a way that’s consistent with the specific virtual conversation partner’s personality, beliefs, desires, and emotions.

“[The simulated conversation partners are] acting in very autonomous and novel ways, essentially, which forces the person interacting with them to think about, what are their motivations? What are they trying to achieve?” explained Marsella. “It was an attempt to not only give the agent a model of the human, but to force the human to develop a model of the agent.”

Although this research has since concluded, it laid much of the groundwork for Marsella’s other, more recent projects. If theory of mind can successfully create better virtual conversation partners, where else can it be applied to human-computer interactions?

Robots, earthquakes, and disaster relief
Robots are massively useful disaster relief tools — but at least for now, tools are all they are. Though the work environments are dangerous, humans are still needed to ‘supervise’ the robots. Marsella believes that they have the potential to become true collaborators in our disaster relief efforts though, and that theory of mind may be one of the keys to unlocking that potential. The Robot Human Interaction Project is run in conjunction with assistant professor Chris Amato, whose focus is on the robotic side of the interaction, while Marsella focuses on the human side.

In order to develop a theory of mind model, the team is first studying humans in simulated rescue environments to create models of how people are likely to act. “Then, using those models to inform the robots, the team is able to help their robots develop a theory of mind in regards to their human counterparts. This facilitates a level of collaboration that’s otherwise impossible.

“Effective teamwork usually assumes that people have an understanding, and an expectation about, what everyone else is doing in the team. If that breaks, if these entities don’t have that, then they can’t be responsive to each other,” Marsella explained. “To be adaptive as a team and respond to uncertainty, you need to have an ability to predict the other entity, and predict that other entity under changing conditions. That, essentially, is what theory of mind is giving.”

If a novel situation arises — say, a wall in a building where a robot is conducting search-and-rescue operations has collapsed — the robot without a theory of mind would likely need to either stop and await new instructions, or risk injuring itself and its collaborators. The robot recognizes that the wall is down, and the human collaborator might know as well, but without a theory of mind, the robot is unable to know if the human knows the wall is down (or if the human knows that the robot knows that they know, etc.). If the robot was to react autonomously — for instance, by digging through the wall — without an understanding of the human’s knowledge, goals, and likely reactions, it could negatively impact them both — for instance, when the wall collapsed because people were still walking on top of it, unaware that the robot had proceeded with the rescue.

“In order to get the robots to really work well with people, they need to understand people better than they do now. You can see this with your robot vacuum cleaner, or your other robots like Alexa…they are very, very limited in what they can do, what they expect, and how they can interact with you,” explained Amato. “I think having things like theory of mind are necessary in order to have the robots get to the next level, and be able to think about why the human is saying this, what they’re going to do as a consequence, and how the robot’s actions will influence the people’s thoughts of them.”

“That is essentially where human-AI interaction is going,” added Marsella. “This is something that’s evolving in the whole AI community, this notion of human-AI interaction where there is much more of a balance. There’s a transparency between the robot and the human in the sense that they have models of each other.”

Theory of mind on a superstorm scale
Marsella is also working on another project that combines disaster relief and theory of mind: a program to understand and simulate if and when people evacuate in the face of an oncoming hurricane. The project accounts for factors like what kind of official warnings have been issued, people’s likely emotional states, and the time before the storm makes landfall to model when people flee — and what factors might make them more likely to flee, more quickly.

“There’s an interesting close connection between theory of mind and emotion, because a lot of emotions are social emotions. Things like anger, or shame, or guilt — these are emotions that we form because of the mental models we have (of) ourselves, and how they relate to other people in the environment,” said Marsella. In this way, one’s individual beliefs about the minds of themselves and others, and the emotions they feel about those beliefs, help to build the framework they would use to make decisions.

In progress since 2017, the project is now to a stage where the model is ready to get into the hands of policymakers. Marsella hopes it can be used to hone evacuation messaging to ensure the maximum number of people make it to safety.

“If you have a model that makes predictions about whether or not people are going to evacuate, and what are the reasons they’re not evacuating, then you can actually systematically play with that model and see, how do we convince them to evacuate?,” said Marsella. Local leaders could use the model to run simulations with different messages to predict what techniques would effectively motivate people to evacuate.

Marsella’s work displays how computer science is, at its heart, an intensely human discipline. In order to make robots and virtual agents who can communicate smoothly with real people, researchers like Marsella and Amato need to understand the parts of our minds that are difficult to communicate. To create a model maximizing the number of people who will evacuate in the face of a coming storm, researchers and policymakers need to be able to understand and empathize with the reasons they might want to stay. Time and time again, interdisciplinary research brings an element of humanity to computer science that just makes it click.

Amato was able to neatly sum up what it is that Marsella and his theory of mind-based approach uniquely bring to his research.

“It’s [work that is] not doable by someone that doesn’t know about humans. How do you get robots to work with humans, if you don’t know about humans?”

Subscribe to the Khoury College newsletter