REALM
Overview
The REALM model was proposed in REALM: Retrieval-Augmented Language Model Pre-Training by Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang. It's a
retrieval-augmented language model that firstly retrieves documents from a textual knowledge corpus and then
utilizes retrieved documents to process question answering tasks.
The abstract from the paper is the following:
Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks
such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network,
requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we
augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend
over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the
first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language
modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We
demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the
challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both
explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous
methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as
interpretability and modularity.
This model was contributed by qqaatw. The original code can be found
here.
RealmConfig
[[autodoc]] RealmConfig
RealmTokenizer
[[autodoc]] RealmTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary
    - batch_encode_candidates
RealmTokenizerFast
[[autodoc]] RealmTokenizerFast
    - batch_encode_candidates
RealmRetriever
[[autodoc]] RealmRetriever
RealmEmbedder
[[autodoc]] RealmEmbedder
    - forward
RealmScorer
[[autodoc]] RealmScorer
    - forward
RealmKnowledgeAugEncoder
[[autodoc]] RealmKnowledgeAugEncoder
    - forward
RealmReader
[[autodoc]] RealmReader
    - forward
RealmForOpenQA
[[autodoc]] RealmForOpenQA
    - block_embedding_to
    - forward