Memory representation and retrieval in neuroscience and AI 

Memory retrieval occurs at different levels in the brain – from the object level (spellings of words,  association between name of color and color, arithmetic, sound of different keys etc.), embedding level (remembering the smell of complex aromas) to circuit level (performing algorithmic tasks, appropriate motor response for a stimulus etc.). Currently, retrieval systems in AI such as RAG retrieve by fine tuning a model (circuit) to pick relevant tokens (objects retrieved via embedding similarity) and add it to context. However, explicitly parametrizing retrieval via combinatorial object identity or spatial proximity is not explored, and we hypothesize that there are gains to be made by creating systems that can parametrize and retrieve at each level. Here, we explore two approaches to retrieve, one at the circuit level that samples from domain fine-tuned LLMs via a router inspired by the human thalamus. A second approach builds on distributed memory instead of distributed compute to perform image retrieval using semantic similarity and component proximity over an image’s components. Together, we show that small models with access to curated structured databases that are trained on retrieval can outperform large general purpose models.