Retrieval Augmentation as a Shortcut to the Training Data

Text generation has been shown to benefit from retrieval augmentation, wherein information retrieved from a source external to the generation model is used to condition its predictions. The approach is widely used in knowledge-intensive tasks where facts are retrieved from a knowledge base and used to inform, for example, the answer to a factual question. My interest lies in an alternative use of retrieval augmentation as a way to provide the generation model with access to training data for a given task. I will present a recent work on retrieval-augmented image captioning, in which we show that model size can be greatly reduced when the training data is available through retrieval and therefore need not be fully “learned”.

What’s more, we find that the model can be readily transferred across domains by replacing the source of retrieval with new data and thus shifting the distribution of the generated process augmented with retrieval. I intend to continue this work along two dimensions, one focusing on the intersection of retrieval augmentation and in-context learning, and the other, on controllable image captioning for the generation of language learning materials.