Facebook is using AI to tag your pictures to help blind people

The feature, called "automatic alternative text”, uses image recognition technology developed through machine-learning to identify the objects pictured.

It’s not perfect yet, with sample captions from Facebook, shared by The Verge, reading more like a laundry list than a descriptive overview: "Image may contain: two people, smiling, sunglasses, sky, outdoor, water” reads the caption on one picture of a couple taking a photograph on a beach, while "image may contain: pizza, food” is the caption for a picture of a tasty-looking pepperoni and olive pizza.

But the technology, which is launching first on iOS, will do a great deal towards making the Facebook app more usable for blind and visually impaired visitors. On iOS, the most popular accessibility feature for such users is called Voiceover, which reads screen elements out loud. But without some sort of caption, images can’t have any more description than "a picture”, or the filename being read out, which leaves Facebook’s content – heavily dominated by images – largely inaccessible.

In March, Twitter announced its own attempt to solve the problem, rolling out a feature for its apps that lets users manually add captions to images they upload. Because people are better than machines at describing images (for now), the captions are likely to be more accurate and descriptive, but the burden of entering a caption for every image means that few actually have a description appended.

That’s not helped by the fact that Twitter’s feature is off by default, buried in the accessibility settings menu, alongside other settings aimed at people who need accessibility help, not those who want to provide the help.

Facebook’s approach could overcome those hurdles, but the company faces problems of its own. Inaccuracy is the big one: not only do machines still get things wrong, sometimes they get them horribly wrong. A similar system rolled out by Flickr last year managed to tag an image of Auschwitz with "sport” and a picture of a black man with "ape”.

Facebook hopes to dodge those bullets by only applying captions when the system is "80% sure”, and applying no caption at all in sensitive situations.

(theguardian.com)