David Liu, the CEO of Sonde Health, joins Being Patient Live Talks to explain the science behind voice biomarkers and their potential for the early identification of cognitive impairment.
Brain health researchers are exploring new ways to leverage cutting-edge technology in the diagnosis of diseases like Alzheimer’s. One emerging field of study is that of vocal biomarkers: the idea that how the way our voices sound can reveal symptoms of depression, respiratory diseases, even cognitive decline.
Some companies are even developing technology that enables personal smart devices to identify vocal biomarkers — and, perhaps one day in the future, to aid in early diagnosis or even disease prevention. Sonde Health is one of them.
“These illnesses or diseases will impact the physiology of speech, and so we analyze how you sound,” David Liu, CEO of Sonde Health, explains, “We break it down to a very atomic level and then begin to derive insight into what’s actually happening with your body.”
Liu sat down with Being Patient video reporter Mark Niu on Live Talks to explain this emerging field and tell us a bit more about the science behind voice biomarkers and their potential for the early identification of cognitive impairment.
Read or watch the full conversation to learn more about vocal biomarkers and their potential for early detection of cognitive decline.
Being Patient: What are vocal biomarkers?
David Liu: Vocal biomarkers can be defined as a signature feature. These are signals that are buried within the audio of your voice that you can analyze, based upon research, to see if different physiological changes are happening within the body.
What we do know is that when there are chronic illnesses and diseases that start taking effect in the body, the hundred or so different body parts that enable you to speak, your tongue, your throat, and your vocal cords, are invariably impacted by that illness. That can be mental health, depression, [or] anxiety; it certainly is probably more obvious when you talk about respiratory illness like asthma or COPD, and even cognitive impairment.
These illnesses or diseases will impact the physiology of speech, and so we analyze how you sound. We break it down to a very atomic level, and then begin to derive insight into what’s actually happening with your body.
Our use cases for what we do is really in the health and wellness space. We’re very much similar [to] what Apple Health does Samsung Health or, or even Google Fit, to be able to give you an objective measure of the signals that are coming from not your wrist or your finger, with wearables, but from your voice.
Being Patient: How does the technology work in identifying the different biomarkers?
Liu: There are, I think, literally thousands of vocal features that are made up or inside of your voice. We have been studying voice for many, many years. That’s why audio signal processing expertise is so critical, to be able to understand different types of vocal features, combinations of those features such as prosody features, jitter, or shimmer. These are the roughness and smoothness of your voice.
How often or how lengthy are the pauses in your speech? How fast are you speaking? We have a proprietary feature that measures the time distance and we timestamp it, between when air is pushed out of your mouth, and when audible sound is recorded or sensed. That time period is different for different people who have different health conditions.
Again, we’re providing, through the analysis of these features, a relative risk level reading of potentially a person having symptoms of these health conditions. It is not to be used as a diagnostic device. This is not to replace a clinician and their clinical knowledge, but what it does is this can be put into the hands of billions of people because we all carry devices, such as smartphones.
“This is not to replace a clinician and their clinical knowledge, but what it does is this can be put into the hands of billions of people because we all carry devices, such as smartphones.”
Being Patient: Is a smartphone what is primarily used to detect these vocal biomarkers?
Liu: Today, that is our primary use case because our SDK can be embedded in any device with a microphone. We started with smartphones because, of course, they are with us most hours of the day, but we are now expanding into devices such as televisions, other home appliances, and even automobiles.
Being Patient: How does voice collection and analysis work with a smartphone?
Liu: That’s a great question. Just to be very clear, we do have an app that we use more for marketing. Of course, to show people so that they can experience what vocal biomarker health readings look like, or [what] health tracking looks like or feels like.
But we’re not a direct-to-consumer company, and we do not market a direct-to-consumer app. That’s not our business. Our business is to put the SDK and our APIs into our customers’ products, which already have massive user bases and loyal users. Whether it be an app or a device, that’s how one would see us.
For example, GN Group is a very large company out of Copenhagen. They are the owners of ReSound hearing aids, as well as Jabra headphones and earbuds. They are going to integrate our technology into their products so that, for instance, older adults who wear hearing aids will be able to track their mental fitness as part of wearing hearing aids.
They are also interested in and are working with us to develop mild cognitive impairment local biomarkers so that we can get ahead of detecting symptoms of MCI or what we call cognitive fitness— that’s under development right now.
Being Patient: It sounds like you’re collecting these voice samples almost passively when people are just going about their daily lives and not thinking about it. Is that right?
Liu: It’s absolutely passive, and the thing that differentiates Sonde is that we don’t require you to buy a ring, we don’t require you to buy a $400 watch— we don’t require you to do anything other than the devices and apps that you are already using today.
We’ve been very deliberate about our strategy to work with third parties to put our technology into their products that are being used everyday around the world, especially in countries that don’t have high speed internet like we do, or Korea does, for example.
You want to be able to reach those people too, so that they have equal access to this health tracking capability through voice. So, that’s a big part of how we work, we have to work with these large customers in this end consumer OEM devices space, like I said, phones, appliances, and even automobiles.
We’re working with large organizations such as the military and Air Force, which have large populations that they want to see do well, perform well, and stay healthy. I’m going to repeat this over and over again, but very much in the health tracking business.
If you can get ahead of understanding the trends of your own health and then be able to look at what has happened in the past, [you will] be able to do a little bit of analysis. This is what has happened with all of us and using the Apple Watch or any of these other fitness trackers. We begin to pay attention, we begin to look at trends, we begin to understand when our resting heart rate isn’t what it should be.
We begin to put two and two together and say, “Oh, yeah, actually, I had a tough couple of days last week because of XY and Z.” We are seeing the same type of thing when we play it back for our users, they can understand that their mental fitness was a bit lower.
At the end of the day, a day full of meetings or a day with family, we are now seeing some of those things. When you start putting together mental fitness tracking from Sonde with physical, traditional fitness tracking data like from Apple Watch, you now have a full 360-degree view of how you’re doing and, most importantly, what to do about it.
Most people do know what to do. If I didn’t get a lot of sleep and my mental fitness is a little bit low, you know what to do, or maybe taking a walk two or three times a week could improve. These are all well-known facts that have been studied exercise, sleep, diet, and even social interactions.
Checking in with somebody, talking to somebody can improve your mental health. So, we’re very much in the business of getting to people early so they can stay mentally fit and well. We’re not in the business of diagnosing for disease. We want to catch this much, much earlier. There’s a much bigger population around the world that can benefit from this. If we’re successful at Sonde, we prevent people from ever entering the healthcare system.
“You want to be able to reach those people too, so that they have equal access to this health tracking capability through voice.”
Being Patient: How can vocal biomarker identifying technology detect or aid in identifying cognitive decline?
Liu: This is the newest product that we’re working on now. I’m really glad you’re asking about it. It has not been released. We are in the later stages of developing what we’re calling a cognitive fitness or a cognitive impairment vocal biomarker that will sit alongside what we’re doing in mental fitness for depression and anxiety and respiratory fitness for asthma and COPD.
So, those two products are already out there, and we’re working with large enterprise customers to deploy those vocal biomarkers; the newest one, cognitive impairment, is incredibly exciting. I don’t have to go through all the statistics with you, but we are aging as a species all around the world. To be able to understand much earlier symptoms of cognitive impairment can make all the difference in how you treat a person.
I have my mom, who is 84 this year, and I’ve definitely noticed that there are things that are happening with her that, from a cognitive impairment standpoint, are different. She has not been diagnosed with dementia, but mild cognitive impairment is a precursor to dementia. So, it’s imperative now, for us to be able to recognize some of these signs earlier.
Maybe we want to get her into an independent living situation where nursing is available without having to call somebody in. These things can be done much, much sooner and so that you don’t have some sort of disastrous event happen. This is repeating itself, this story over and over again, not only in this country but in so many countries around the world.
Being able to have those signals sooner so that family, loved ones, and the network that is around this person can take much more proactive action can make all the difference in the quality of life of that person, for the remainder of their life.
I would be remiss to not mention that cognitive fitness impacts us all. Have you ever woken up in the morning and not had your, well for me, two cups of coffee? I will tell you that if I don’t have that, and maybe even a workout, I’m not thinking as clearly—simply put. I’m not firing on all cylinders.
That doesn’t mean that I have [early-onset] dementia or MCI, mild cognitive impairment. It just means that at that moment in time, the facility that I have, with my cognitive capabilities, is not at its optimal level. So, if we can provide people with a quick view into that, and so that you can do what you want to be able to do so you can prepare for that meeting or that interview, or that performance, whatever it is— it really impacts all of us.
“To be able to understand much earlier symptoms of cognitive impairment can make all the difference in how you treat a person.”
Being Patient: Do you foresee a day when this technology will help detect different types of dementia?
Liu: Never say never, I guess, is my answer to that; I don’t know, is quite honestly the truth. We’re going to take it as far as we can. We have no less than four clinical studies, validation, and research studies that we are running in this mild cognitive impairment field with four different clinical organizations—all at the top of their field in this space.
So, I do believe we’re going to push it as far as we can. Do we need to get it to a point where it is a digital diagnostic for mild cognitive impairment or dementia? I don’t think so, but I never say never. There could be this incredibly valuable use case and, of course, more business and market opportunities for us to do it. If it makes sense, we will go do that.
What I can tell you is that what we’ve seen for mental fitness, which is to give you an indication, a signal, a relative risk or, if you will, of having symptoms of depression or anxiety, it’s a much bigger market—we can have a much bigger impact on the world and on a lot more people, if we can catch that earlier.
What that means is, get it into the devices and activities that everyday people are doing today so that they can monitor their mental fitness as easily and as routinely as they monitor their physical fitness. How many steps? How’s my resting heart rate? What’s my blood oxygen level? These are things we’ve grown accustomed to over the past ten years because of these pretty incredible wearables.
We know that if we do simple things like exercise and eating better, these metrics will improve. It is my hope that it’s the same corollary [that] we will use for cognitive impairment. If we can first get to a lot of people, get them aware, have them be aware of these things, so that they can begin to change lifestyle and behavior.
These large chronic diseases, including brain diseases, maybe not dementia, but a lot of chronic diseases that now dominate the healthcare systems of every country in the world, in terms of cost in terms of how [many] resources we devote to treating chronic disease, it’s well known that if you can change lifestyle and behavior, you can begin to prevent or even reverse a lot of these chronic diseases.
Our hope is that if people have objective information, and data that’s very easily consumed or that is a part of what they’re already doing. Do not ask them to do something extra in their life. We’re all busy. Fathers, mothers, husbands, wives, brothers, sisters, and, of course, coworkers—we’re very busy people. If you can do all of that and deliver this information, people will, over time, understand it. It is my hope, and we’ve seen it in our clinical study, that they’ll change [their] behavior.
Being Patient: Is there a way to get a baseline analysis from this vocal biomarker technology to help track decline?
Liu: We actually love using the term baseline internally because if we could, we would always want to get that because of our models, our machine learning models. We do deploy them based upon population data, and what we always say is that it’s really great.
You can use Sonde without having a prior voice analysis with Sonde, but it works way better, on an individual level, if we can establish, you know, ten readings over a period of a couple of weeks because everyone’s different. Our voices are different. Not just the sound of it, but the way we speak. Things get much, much better when you establish a baseline.
Being Patient: In terms of data collection, as it’s done passively through devices, you have to have concerns about privacy. Can you tell us how you’re approaching that?
Liu: How can you not? I’ve never used Alexa; I never will. I think we’ve seen some of the issues that have gone on there. But the business model there is different; they’re trying to sell stuff. For us, it’s very simple. I come from a background in both healthcare and education and understand just how valuable [and] how precious health data is. Voice is health data.
So, we haven’t just checked the boxes on having the right privacy policy, we’ve evolved our product so that we’re putting consumer and user data protection at the forefront. One very big example of this is that we now have an SDK that we deploy to our customers’ products that keep voice analysis recordings local to that device.
No voice recordings will be sent up into the cloud, Sonde’s cloud, for analysis, and then scores sent back. That’s typically how a lot of these apps work. We made the determination that not only will the performance and the experience be much faster and quicker if we can keep things on the edge so that all the analysis and data is local to the device, but, from a privacy perspective and a data security perspective, is way better. So, we really nip that right in the bud.
Of course, it’s still something that everyone has to be cognizant of, and we care a lot about that. We do take a bit of a hit in terms of not getting every last bit of data to look at because we made the choice to keep it on the device, but it’s well worth it because we can use our vast resourcing on the research and development side working with clinical organizations, to further our models, to further our product, but not use consumer data to do that.
“I come from a background in both healthcare and education and understand just how valuable [and] how precious health data is. Voice is health data.”
Being Patient: How much of a voice sample do you typically need for an analysis?
Liu: 30 seconds, and it doesn’t have to be continuous. We do, at this point, analyze three second snippets at a time. If, for some reason, you get cut off, maybe you end your speech at ten seconds, then we’ll hold that then seconds on the device. Then, once you speak again, we’ll start analyzing your voice, and then every 32nd interval, that’s enough time for us to be able to provide a score.
Being Patient: Tell me more about that score and how it works if you’re on a smartphone.
Liu: The score is a relative risk score. Again, it’s not diagnostic. We are not telling you [that] you have a 98 percent chance of having depression, that is not what Sonde does, to be very clear, or you have a 92 percent chance of having mild cognitive impairment, that is not what Sonde does. What we do is provide you a relative risk score of having potential symptoms of these diseases, of these illnesses.
That’s very important because we know that there are symptoms of depression and anxiety that come out of the voice that could be caused by other things. We’re very careful. To give you a very simple-to-understand scoring paradigm, you either have an excellent score that’s 80 to 100, a good score, excellent means that you’re doing great, good means you’re fine, from 71 to 80, and then anything below 70, we just say “Pay attention.”
Now, “Pay attention” doesn’t mean that you definitely have anything going on in terms of depression or anxiety, for example, or mental fitness. I would expect us to have something similar for cognitive impairment when we get there. But it is telling us that your voice at that moment, just like your wrist or your fingers telling that wearable, that something is happening that is depressing your vocal feature scores and, therefore, your overall mental fitness score.
We are not telling you that it is because of depression, but we need to tell you that yes, your speech rate has gone down, your energy level from, say, your speech has gone down, and therefore, your overall aggregate score is lower, and put you in the “Pay attention” range. That’s all we say, and then we have you reflect. In our clinical study, that has been incredibly valuable because people now, for the first time, can reflect back on their mental fitness over the past week, day, week, or month, and begin to put things together for themselves.
[It’s] so much like what Apple Watch does with resting heart rate. My resting heart rate typically is around 60. Will I get some readings at 75 or 77? Yes, that’s happened to me. Does Apple tell me that I have cardiovascular disease or some issue? No, but they’ll tell me that there is something you know that your wrist is giving you a signal at that moment. The same type of thing here: we’re going to tell you that your voice is telling us something at that moment that you may need to pay attention to.
“We’re going to tell you that your voice is telling us something at that moment that you may need to pay attention to.”
Being Patient: What’s next for research on digital voice biomarkers? I mean, you’ve alluded to some of the technologies that you’ve got going on, but what do you see next?
Liu: We’re doing some pretty big spaces right now; mental fitness and mental health are massive. You can’t go two days without reading a new article about how this is impacting the world. I think there’s a new study out saying that mental health now should be designated at the top of the list of all of the largest and most impactful diseases around the world, including cancer, diabetes, and others. So, that’s how big that is.
What is next right now is that we’re the only company in the world that has two clinically validated and published vocal biomarker products, one in mental health and one in respiratory health. We are very busy when working with our customers. Again, these are very large Fortune 500, Fortune 1000 companies around the world, not just in the United States, but in Asia and in Europe, to deploy this so that we can begin to give people the access and the information that they need to begin to improve their mental health and respiratory health.
We’re working very hard to get the mild cognitive impairment or cognitive fitness products built and launched later this year and validated. That’s very exciting for us as well. I would say this more that we are in the business of providing health insight and data, and health tracking around health conditions that had been previously very difficult to track.
We’re not going into any of the areas that Apple Health, Google Fit, or any of these others, or Aura—they can duke it out in the traditional fitness biomarker space. We’re right now predominantly looking at biomarkers and from voice that give us an indication about your health from the shoulders up. That’s really never been done before, even by Apple.
So, we’re very fortunate to have worked on this for so long. I think we’re one of the most experienced and we do also have probably the deepest, most diverse voice data set because we’ve been researching this for so long. We will continue to look at new health conditions where we can provide wellness tracking, so that people can again, get ahead of these chronic diseases or if they’re being treated for chronic disease, get on that path sooner to recovery.
Because chronic disease does not typically happen overnight, it develops over a number of years, so can we not get ahead of this? I call it a runaway train. The chronic disease epidemic that we’re seeing because it’s a huge not only human impact around the world but it’s going to be a massive economic impact for every country.
Katy Koop is a writer and theater artist based in Raleigh, NC.
