Smartwatches and health trackers like Fitbits can measure your sleep, heart rate, and step count. Now, scientists are developing apps that might be able to measure brain health — and potentially cognitive decline — by listening to the way you speak.
About 89 million Americans use smartwatches and other fitness trackers to measure their fitness levels, sleep, and weight. Soon, there will be apps that measure your brain health over time.
A smartphone application could provide insights into brain health by analyzing how someone speaks throughout the day. Researchers also believe they could tap into digital voice biomarkers to help them spot subtle changes in brain health and screen for the earliest signs of cognitive decline.
To speed up the search, in October 2023, the Alzheimer’s Drug Discovery Foundation’s Diagnostic Accelerator launched the first longitudinal, international study of vocal changes associated with Alzheimer’s disease, as part of its $100 million investment in finding new biomarkers for the disease.
Changes in brain health can be detected through speech
There are two different approaches to measuring brain health through speech: focusing on what people say and how they say it. For example, when someone is stressed or anxious, it can cause the muscles in the throat to tense up affecting the tone, pitch, or other qualities of speech.
A number of companies are already putting this concept to the test.
Barcelona-based startup Accexible is developing screening tools for clinicians that could screen for mild cognitive impairment by analyzing 30 to 60 seconds of speech. It is currently being tested in more than 10,000 patients across Spain, Colombia, Mexico, the UK, and the U.S. across 17 different clinical trials.
“Vocal biomarkers can be a very powerful tool as an early detection prescreening tool,” Carla Zaldua Aguirre, the company’s CEO told Being Patient over email.
She added that the product has 89-percent sensitivity, meaning it can accurately tell whether someone has mild cognitive impairment correctly nearly nine out of 10 times.
Vocal biomarkers of Alzheimer’s, ALS and other neurodegenerative diseases
A 2023 study found that some digital biomarkers related to word use, grammar, and speech structure may correlate to beta-amyloid pathology — a build-up of protein plaques that appear in brains with Alzheimer’s. Like physical plaque build-up, these vocal biomarkers could appear before other obvious signs of cognitive decline.
Multiple companies are already using this technology.
Aural Analytics develops apps that help monitor brain health and cognition by analyzing a few snippets of speech. In 2022, the company registered the Speech Vitals computerized cognitive assessment, which can now be used to monitor changes in cognitive health over time.
In 2023, the company received an FDA breakthrough designation last year for their Speech Vitals – ALS tool to monitor patients’ health with amyotrophic lateral sclerosis (ALS), a neurodegenerative disease that causes muscle wasting. The company is also working to develop tools to help them diagnose diseases like ALS.
Modality.AI is also testing this technology for ALS, alongside Alzheimer’s and Parkinson’s disease. They have developed an app-based cognitive assessment that monitors and analyzes speech and facial patterns. The technology is being tested across multiple clinical trials to see how well it can detect many different diseases.
Meanwhile, Winterlight Labs developed a tablet-based speech assessment to monitor and screen for cognitive impairment and Alzheimer’s disease and measure whether someone is responding to a new treatment.
Liam Kaufman, CEO, told Being Patient that the app takes short snippets of speech, transcribes the words, and then analyzes hundreds of features of the speech as well as the way it sounds.
“Currently the technology is being used in clinical trials for biotech and it’s also being used retrospectively to analyze recordings from phase 2 and phase 3 trials,” he said. “It will be part of a screening assessment or primarily used to look at response to therapy.”
Are vocal biomarkers useful?
While researchers are investigating these biomarkers in a clinical setting, some companies are also developing mental fitness tracking technology based on vocal biomarkers.
According to David Liu, CEO of Sonde Health, told Being Patient that these new apps will hit the market within the next five years. The company is developing software that integrates into various health apps or smart devices to help people monitor and track their brain health.
Kaufman thinks that these apps can present some challenges. “From our own research, and talking to others in the area, the signal associated with depression and anxiety certainly exists but it’s more subtle and variable than say Alzheimer’s or schizophrenia,” he said. He isn’t sure that the average person would be interested in having this data at their fingertips.
Liu thinks that mental fitness tracking apps could be helpful in the same way that an Apple Watch or Fitbit might help someone track their health.
The goal of having this data handy isn’t to provide you with a scary diagnosis. Even the health and fitness trackers on the market aren’t going to tell you that you have heart disease. Instead, like existing trackers, they provide a snapshot of your brain health to make you more aware of how you’re feeling “so that you may take some proactive action,” Liu said.
In other cases, some of the applications being developed might become useful screening tools for doctors and clinicians, making it easier to identify which patients might be in the early stages of cognitive decline.
Ultimately, the jury is still out on whether having these biomarkers handy will lead to improved health outcomes for otherwise healthy people.
Making sure the algorithm works well for non-white people
Many new AI or machine learning health applications are developed with white people in mind. Fitness trackers, skin-scanning dermatology apps, and even AI software like ChatGPT aren’t as accurate for people who are Black, Hispanic, or otherwise have darker skin tones.
Kaufman said that Winterlight Labs has collected data from over a dozen countries including in North Africa and the Middle East to make sure their test is robust. “We’ve been able to replicate findings across populations,” he added.
Liu explained that Sonde Health was also conscious of these issues. As a result, the company has collected data from people across ten countries in Africa, Europe, Asia, and North America speaking many different languages. These voice clips are then given to clinicians to be scored based on specific vocal features that used to train the algorithm. This makes it very clear what the algorithm is measuring and can help prevent other biases from sneaking into the assessment process.
Right now, they have an application that can get enough information from 30 seconds of speech to score features of anxiety or depression from someone’s voice. And they make it clear that scoring high isn’t diagnostic for having any disease.
“It just means that your voice at this moment in time is giving us an indication that there may be symptoms of these conditions,” Liu said.
Privacy and security concerns
While some companies are developing digital voice biomarker tests that will be available only to physicians, trackers like those developed by Sonde Health may become widely available.
With existing fitness and wellness trackers, there are already concerns over the amount of data collected from users and how the company shares or sells the data.
After Roe v Wade was overturned, there was concern from many users of period tracking apps that the data could be used by law enforcement to prosecute individuals receiving abortions. Many fitness apps are also not very well secured and can leak location data about its users. Other apps can also sell fitness data to third-party advertisers.
To address some of these concerns, Sonde Health doesn’t actually store user data in its database. “We can process your voice without having to record it, and score it and deliver those insights to the device,” Liu said.
In addition, Sonde Health is currently working with GN Group in Copenhagen, which develops hearing aids to help develop a vocal biomarker for mild cognitive impairment. Hearing loss is a major risk factor for developing dementia, but addressing it early with the right hearing aids could reverse the cognitive impairment.
The company is also partnering with Massachusetts General Hospital to see how well their technology can track the cognitive health of 50 adults with frontotemporal dementia over time and whether this information could enhance patient care.
That was a very interesting and well-presented blog. Many thanks in sharing current research projects on this topic.
Thank you for being here, Barbara!