AI Sketch-Based Object Detection Tool Could Revolutionize Tumor Detection

An artist's sketch is brimming with individual cues that words simply cannot express fully, reminding one of the old adage 'a picture is worth a thousand words'. While emerging artificial intelligence (AI) systems can create images based on basic descriptive words, they often fail to express the unique artistic expression or specific match required by the user. Now, a novel sketch-based tool enables artists to guide the AI through sketches, enabling it to locate the precise object while ignoring irrelevant ones. This could be particularly beneficial in medical scenarios, such as identifying aggressive tumors.

Researchers at the University of Surrey (Surrey, UK) have developed an innovative sketch-based object detection tool. This unique tool enables users to sketch an object, which the AI then uses as a reference to search within an image for a matching object, disregarding other, more general options. The researchers provide the example of the tool being used to search for a specific zebra in an image filled with zebras, guided solely by a sketch of one zebra eating. The AI tool considers visual cues such as posture and structure but bases its decisions primarily on the exact specifications provided by the user's sketch. This ability to teach machine learning tools to identify specific objects in a particular image and discard others is a breakthrough that could lead to significant improvements in cancer detection, say the researchers.

Image: AI that uses sketches to detect objects within an image could boost tumor detection (Photo courtesy of Freepik)

“The ability for AI to detect objects based on individual amateur sketches introduces a significant leap in harnessing human creativity in Computer Vision,” said Professor Yi-Zhe Song, who is leading this research at the University of Surrey's Institute for People-Centred AI. “It allows humans to interact with AI from a whole different perspective, no longer letting AI dictate the decisions, but asking it to behave exactly as instructed, keeping necessary human intervention.”

Related Links:
University of Surrey