NLP for Long, Structured Documents

A great part of the world’s knowledge is stored in long, structured documents such as scientific articles or legal texts. Language models (LMs) can help humans in handling these documents, for example by answering questions about documents or summarizing them. However, this potential has not been fully explored. First, the rich structure of documents composed of sections, subsections, etc, is often ignored. And second, due to hallucination, users need to verify LM responses, which can be hard for long documents. In this talk, I will present research that addresses these gaps: Regarding the use of document structure, we conducted probing experiments to assess the representation of document structure in long document transformers. Our data shows that the models learn to represent document structure during pre-training, and this representation can be enhanced via additional inputs. To improve verifiability, we investigated optimal approaches to attribution, i.e. the generation of responses with evidence from the document. We found that large models (e.g. GPT-4) are well able to cite from their inputs, while smaller models benefit from post-hoc evidence retrieval. The talk concludes with an outlook on promising directions for future work.

Post Talk Link:  Click Here

Passcode: Umwvvr#1