Evaluating and complementing vision-to-language technology for people who are blind with conversational crowdsourcing

Salisbury, Elliot, Kamar, Ece and Morris, Meredith Ringel (2018) Evaluating and complementing vision-to-language technology for people who are blind with conversational crowdsourcing. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018. vol. 2018-July, International Joint Conferences on Artificial Intelligence. pp. 5349-5353 . (doi:10.24963/ijcai.2018/751).

Abstract

We study how real-time crowdsourcing can be used both for evaluating the value provided by existing automated approaches and for enabling workflows that provide scalable and useful alt text to blind users. We show that the shortcomings of existing AI image captioning systems frequently hinder a user's understanding of an image they cannot see to a degree that even clarifying conversations with sighted assistants cannot correct. Based on analysis of clarifying conversations collected from our studies, we design experiences that can effectively assist users in a scalable way without the need for real-time interaction. Our results provide lessons and guidelines that the designers of future AI captioning systems can use to improve labeling of social media imagery for blind users.

This record has no associated files available for download.

Contributors

Download statistics