Facebook’s AI now enables users to search for images by content

Image: Shutterstock
Ingrid Fadelli

Could this mean we are approaching the death of tags? That’s worth a Like

Tags have been our way of accessing specific content for a while now, but a new Facebook feature might be revolutionizing how we group and search for pictures online.

Facebook has updated its search feature, integrating it with its AI computer vision platform, Lumos. This means that Facebook users will now be able to search for photos on the social media platform using words that describe the content of the picture they are looking for. At a stroke, we’ll be freed from the daily grind of tagging!

For instance, you could search for “pictures of cats” and get posts by your friends that match this description as a result. The new feature should show you these results higher up, and then also include other similar pictures that you might be interested in viewing.

This new search system uses computer vision to browse through a vast amount of data and spot the most relevant pictures, even if they are not tagged with content-related words.

It scans pictures using AI features such as object recognition and analyzing other information embedded in the image.

In addition to checking for pictures with similar visual characteristics, Facebook’s AI can use other contextual clues from the image, such as captions or comments.

Facebook’s image recognition feature is similar to tools that have already been integrated into iOS 10 and Google Photos

In a post about the new feature, Joaquin Quinonero Candela, Director of Applied Machine Learning at Facebook, wrote: “Whether an image was discoverable was dependent on whether it was sufficiently tagged or had the right caption — until now. That’s changing because we’ve pushed computer vision to the next stage with the goal of understanding images at the pixel level. This helps our systems do things like recognize what’s in an image, what type of scene it is, if it’s a well-known landmark, and so on. This, in turn, helps us better describe photos for the visually impaired and provide better search results for posts with images and videos.”

Lumos was originally used to improve the experience of visually impaired users’ on Facebook. It is a computer vision platform that was built on top of Facebook’s general-purpose platform FBLearner Flow, a system designed as a basis for machine learning applications on the social media site.

Facebook is also working on applying a technology called AAT (Automatic Alt Text) that can describe the content of images to visually impaired individuals.

Classifiers, such as “people dancing” or “people riding a horse”, would describe pictures to visually impaired users, who would therefore be offered an entirely new way of accessing the social media site.

Feature set to prove useful when applied to all users’ searches

“Using Facebook’s automatic image classifiers, just like the ones used in the AAT example, you can imagine a scenario where someone could search through all the photos that his or her friends have shared to look for a particular one based on the image content instead of relying on tags or surrounding text,” said Candela.

This might be the beginning of a real revolution in the way we browse for things online, and is a perfect example of how AI tools such as machine learning and computer vision are making huge steps forward. Indeed, these strides are taking the tech way beyond simply screening out stray nipples.

Candela said: “More than 200 visual models have been trained and deployed on Lumos by dozens of teams, for purposes such as objectionable-content detection, spam fighting, and automatic image captioning. The applications are wide-reaching, with everyone from our Connectivity Labs to Search to the Accessibility team using the technology.”

The photo search update is now available for Facebook users in the US, both on mobile and web. In future, there is probably no reason why it cannot also be extended to video searches, changing entirely the way in which we search for visual and/or audio content.

And now for the bad news

Anyone who routinely rips off other people’s images for their own blogs or publishing ventures, is about to find their wings painfully clipped.

Until now, only the huge stock houses like Shutterstock, Getty, and Dreamstime have had the wherewithal to track down and beat up on the freeloaders who steal their copyrighted works.

However, once this technology reaches the level of ubiquity that it certainly will (because alltechnologies do), the day cannot be far away when smaller, less well-resourced designers, creatives, and publishers will be able to hunt down and remand due recompense from the bad guys.

It might even spawn a new breed of attorney to help them do just that. We could call them hot-shot lawyers. Or something.

Start typing and press Enter to search