1 A blind woman sits in front of a bowl of mostly green apples. Whenasked to find the single red one, she immediately takes it from the bowl.
It seems like magic, but it is not: the woman is using an app—calledEyeMusic—that turns visual information into sound. 2 So how does it work? Users wearheadphones and hold their smartphone in front of them. The phone’s camera scansthe environment, and the visual information is translated into a sequence ofnotes. Notes played first correspond to things on the left, and notes playedlater on the right. Higher notes signal that something is toward the top of thescene, lower notes toward the bottom. Color is shown using different types ofinstruments. For example, a human voice represents white. 3 Lead developer Amir Amedi explains that using the app, people who have been blind since birth can learn to read letters andnumbers, understand what they are seeing in front of them, and recognize facialexpressions and body postures.
“The general concept is that you don’t needto teach each object individually,” Amedi says, “you teach the principles—justlike the brain understands the principles of dots and lines and how to combinethem.” Translating the visual world4 AI, artificial intelligence, isbeginning to make a powerful difference in blind people’s lives. Seeing AI—anapp developed by Microsoft—uses the smartphone camera to turn the visual worldinto an auditory world. Unlike EyeMusic, however, Seeing AI translates theworld into actual spoken language. Users point the camera at an object and theapp will tell them what it is.
When pointed at a person, it can describe theirgender, approximate age, and even who they are and how they are feeling. It canrecognize and read texts, and can even describe an entire scene—identifyingthat a young woman is throwing a ball in a park, for example. 5Saquib Shaikh is a Microsoft software engineer who has been working on the app.He lost his sight at the age of seven.
“Years ago this was sciencefiction,” he says. “I never thought it would be something you can actually do,but artificial intelligence is improving at an ever faster rate and I am excitedto see where this can go.” In the near future, users will be able to use the appwearing smart glasses, rather than via a smartphone. 6 Developed in California, Aipoly Vision isanother app designed to help the blind and visually impaired. Similar to SeeingAI, it turns visual data into spoken text. Users point their smartphone at anobject and the app will say what it can see. If it’s dark, the app willautomatically turn on the smartphone’s flashlight.
It can also identify colors—aparticularly useful feature for people who are color blind. The Aipoly appworks in real time, and has the advantage of not needing an Internet connection. 7 Aipoly is not just successful withpeople who are visually impaired. The developers of the app were surprised tolearn that a lot of people in Japan who had no problems with their sight weredownloading and using the app. The reason? They found that it was a great wayto improve their English. 8 Inventive apps such as Seeing AI,EyeMusic, and Aipoly Vision are greatly empowering blind and visually impairedpeople.
As Microsoft CEO Satya Nadella says, they enable them “to feel moreincluded and connected to the world around them.” But this is just thebeginning. From AI-assisted smart glasses to autonomous cars that speak,technology offers the promise of significantly improving the way people can experiencethe world.