Unlocking Trust in Visual-Textual Search

In the world of computer vision and language processing, there's a big challenge: how to make sure that when a computer tries to match pictures with text, it does so in a trustworthy way. This is called visual-textual retrieval. The problem is that current methods don't always know when they're making a mistake. They just rank things based on how similar they seem, but they can't say how sure they are about their choices. To tackle this issue, a new approach called Trust-Consistent Learning (TCL) has been introduced. This framework is designed to make visual-textual retrieval more reliable. It does this in a few key ways. First, it looks at the evidence for matching visuals and text to figure out how uncertain the match is. This helps the system understand when it might be making a mistake. Second, TCL uses a consistency module to make sure that the system's judgments are reliable. This module checks that the opinions of the system when it's looking at pictures to find text, and vice versa, are in agreement. This consistency is crucial for making the system both reliable and accurate. To test how well TCL works, it was put through extensive experiments on six well-known datasets. These datasets include a variety of images and text pairs, covering different scenarios and complexities. The results showed that TCL performs better than existing methods, proving its superiority and generalizability. But the testing didn't stop at performance metrics. Qualitative experiments were also conducted to provide deeper insights into how TCL works. These experiments helped verify the reliability and interoperability of the framework, showing that it can be trusted to handle different types of visual and textual data effectively. The creators of TCL have made the code available to the public. This allows other researchers and developers to use, test, and build upon the framework, fostering further innovation in the field of visual-textual retrieval.