Just speak your mind, Nature

To most people, speaking does not appear as a challenging task. However, it is one of the most complex activities that humans perform. It temporally requires precise control of multiple muscles. Due its complex nature, multiple neurological diseases, such as stroke, amyotrophic lateral sclerosis or other neurological disorders, can result in loss of the ability. However, a paper in Nature by Anumanchipalli et al this week has shown that maybe this loss can be reversed, not through restoring the muscle control, but through an approach in which spoken sentences are produced from brain signals using deep-learning methods.

The researchers worked with five volunteers who were undergoing a procedure termed intracranial monitoring, in which electrodes are used to monitor brain activity as part of a treatment for epilepsy. The authors used a technique called high-density electrocorticography to track the activity of areas of the brain that control speech and articulator movement as the volunteers spoke several hundred sentences. To reconstruct speech, rather than transforming brain signals directly into audio signals, Anumanchipalli et al. used a two-stage decoding approach in which they first transformed neural signals into representations of movements of the vocal-tract articulators, and then transformed the decoded movements into spoken sentences. Both of these transformations used recurrent neural networks — a type of artificial neural network that is particularly effective at processing and transforming data that have a complex temporal structure.

Previous studies have attempted to convert brain signal directly into speech but have failed to remove acoustic distortion due to limited sample size. In this study, researchers consulted with existing knowledge of correlations between brain signal and vocal-tract muscle movement, and then tried to correlate muscle movements with speech. With previous knowledge, decoding requires fewer data to produce satisfying results. 

Given these breakthroughs, many challenges still remain on the path to a clinically viable speech BCI. The intelligibility of the reconstructed speech was still much lower than that of natural speech. Whether the BCI can be further improved by collecting larger data sets and continuing to develop the underlying computational approaches remains to be seen. Future data collection is also hindered by our limited ability to acquire human brain signal with deeper coverage and higher resolution. Also, this method cannot be applied to people who cannot speak, since previous abilities to coordinate muscles are required. 

These compelling proof-of-concept demonstrations of speech synthesis in individuals who cannot speak, combined with the rapid progress of BCIs in people with upper-limb paralysis, argue that clinical studies involving people with speech impairments should be strongly considered. With continued progress, we can hope that individuals with speech impairments will regain the ability to freely speak their minds and reconnect with the world around them.

Material Properties of a School of Fish, Science News

Physicists have long noted striking similarities between the movements of flocks of animals and the behavior of atoms and molecules. For example, mackerel in a school tend to swim in the same direction, aligning their bodies to their neighbors much as iron atoms align their spins to make the metal magnetic. Similarly, a murmuration of starlings wheeling across the sky looks much like fluid droplets as they flow, stretch, and swirl in response to some unseen stirring (perhaps the wind). Such collective behavior arises not because of some grand design, but because each individual moves in response to the animals next to it, which is similar to the behavior of molecules inside a bulk material. Now, one physicist has gone further and devised a way to measure the springiness and “temperature” of a school of fish.
James Puckett, a physicist at Gettysburg College in Pennsylvania, and his students designed an experiment to measure the springiness of a school of rummy-nose tetras, a freshwater tropical fish that have no social hierarchy (eliminate social interactions) and avoid light. Placed in a thin layer of water, the fish congregate in a square shadow in the center of the tank once Pokhrel shone a light from above. Using computer controls, Pokhrel then split the shadow and moved the two halves apart. In response, the school of fish stretched out initially but then suddenly snapped back. Pokhrel tracked the motion of individual fish during the process and found out that they behaved exactly like a simple spring--the rate at which the fish accelerate toward the center increased in proportion with their distance from it. From the data, he then extracted the rate at which acceleration increases with distance—the spring constant—and found that a school of rummy-nose tetra is extremely elastic: stretched at a given amount and it pulls back with only about one–ten-thousandth the force of a rubber band.
Taking another step further, Julia Giannini, a graduate student at Syracuse University in New York—who until recently worked with Puckett—reported a way to measure a thermodynamics property of a school of tetras: temperature. In an ordinary material, temperature is a measure of the average energy of the constituent atoms or molecules. In the case of a school of fish, it is analogous to the average kinetic energy. Using the tank, she confined the tetras in a circular shadow about 30 centimeters in radius and then shrank the circle at different speeds, causing the fish to crowd together. Using the individual fishes’ speeds, she calculated a quantity analogous to pressure. Just as with a volume of gas, the pressure increased in proportion to the density (PV=nRT). And the constant of proportionality, which depends on the speed at which the circle shrinks, then plays the role of temperature, Giannini told the meeting. A fast-shrinking circle, for example, leads to “hotter” fish. Thus, the school of tetras acts like a gas with a constant, well-defined temperature.
Though it is unclear how far we could push the material analogy, it would be fascinating if we could eventually describe the dynamics of a school of fish or flock of birds using its macroscopic “material” properties, without tracking the individual animals.
How is a school of fish like a rubber band?
Flight of the Starlings, National Geographic

Is That My Egg? Philosophical Transactions of the Royal Society B

Some species of birds do not incubate their own eggs. Instead, they leave their eggs in the nests of other birds and leave the burden of chick care to the foster parents. Apparently, the foster parents are not happy about this situation and would not want to waste their precious energy on raising other birds’ children.  Thus, the poor foster parents have to develop a mechanism to selectively kick out eggs in their own nests. Previously, researchers had assumed that foster parents such as mockingbirds reject cowbird eggs that don't look like their own, in pattern and color. But the new study finds it’s not that simple.
To get a better sense of how mockingbirds decide which eggs to boot, evolutionary ecologist Daniel Hanley at Long Island University in Brookville, New York, and colleagues painted 70 3D-printed eggs a range of colors and put spots on half of them. They distributed these eggs among 85 mockingbird nests and checked several days later to see which eggs were still there. The result was that spots on the eggs played a larger role than colors. Mockingbirds have blue spotted eggs and in the experiment they removed unspotted brown eggs—a “wrong” color and pattern—90% of the time. But the birds were less sure when the egg had spots and they removed brown eggs with spots just 60% of the time.
In the same issue of Philosophical Transactions of the Royal Society B, where the previous study was published, Mary Caswell Stoddard, an evolutionary biologist at Princeton University, and her colleagues recorded when 122 tawny-flanked prinias (Prinia subflava) rejected foreign eggs from their nests and suggested a different method that birds kicked out eggs based on their placement and orientation in the nest. “The exact placement is very hard to mimic,” Cotter points out, making it possible for prinias to use that information when they are not sure whether an egg is theirs.
This phenomenon could be a showcase of the co-evolution process. After parasites evolve spots as a consistent part of the egg’s disguise, the foster parent evolves to use more brain power so it can remember more details about the spotting and hence become more discriminating.

Quote of the week: Deaf people communicate at the speed of light by lip reading, while normal people communicate at the speed of sound by talking.
ISP Sci. Rev. 17 (2019)
Editor: Rossoneri Jing, Shiwei Wang
Integrated Science Program
Northwestern University

This email was sent to <<Email Address>>
why did I get this?    unsubscribe from this list    update subscription preferences
ISP Science Review · 616 Noyes St · Evanston, IL 60208-0860 · USA

Email Marketing Powered by Mailchimp