before_revision: Bidirectional Encoder Representations from Transformers (BERT)models for medical specialties , such as BioBERT and clinicalBERT, have significantly improved in performing biomedical text mining tasks and have enabled extracting valuable information from biomedical literature ; however, only English speakers benefit due to the significant scarcity of high-quality medical documents, such as PubMed, in each language. Therefore, we propose a method to train a high-performance BERT model using a small corpus . We introduce the method to train a BERT model on a small medical corpus both in English and in Japanese , and we present the evaluation of each model in terms of the biomedical language understanding evaluation (BLUE) benchmark and the medical document classification task in Japanese, respectively. After confirming their satisfactory performances, we applied our method to develop a modelcomparable to the publicly available models. OuBioBERT, short for Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University, achieved the best score in terms of the BLUE benchmark . The total score is 1.1 points above that of BioBERT and 0.3 points above that of the ablated model trained without our proposed method. This proposed technique is an effective approach to develop localized medical BERT models and to enhance domain-specific models in the biomedical domain .
after_revision: Pre-training large-scale neural language models on raw texts has made a significant contribution to improving transfer learning in natural language processing (NLP). With the introduction of transformer-based language models , such as bidirectional encoder representations from transformers (BERT), the performance of information extraction from a free text by NLP has significantly improved for both the general domain and medical domain ; however, it is difficult to train specific BERT models that perform well for domains in which there are few publicly available databases of high quality and large size. We hypothesized that this problem can be addressed by up-sampling a domain-specific corpus and using it for pre-training with a larger corpus in a balanced manner. Our proposed method consists of a single intervention with one option: simultaneous pre-training after up-sampling and amplified vocabulary. We conducted three experiments and evaluated the resulting products. We confirmed that our Japanese medical BERT outperformed conventional baselines and the other BERT models in terms of the medical document classification task and that our English BERT pre-trained using both the general and medical-domain corpora performed sufficiently well for practical use in terms of the biomedical language understanding evaluation (BLUE) benchmark . Moreover, our enhanced biomedical BERT model, in which clinical notes were not used during pre-training, showed that both the clinical and biomedical scores of the BLUE benchmark were 0.3 points above that of the ablation model trained without our proposed method. Well-balanced pre-training by up-sampling instances derived from a corpus appropriate for the target task allows us to construct a high-performance BERT model .
edit_actions: R
edit_actions: Bidirectional Encoder Representations from Transformers (BERT)models for medical specialties
edit_actions: Pre-training large-scale neural language models on raw texts has made a significant contribution to improving transfer learning in natural language processing (NLP). With the introduction of transformer-based language models
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: BioBERT and clinicalBERT, have significantly improved in performing biomedical text mining tasks and have enabled extracting valuable information from biomedical literature
edit_actions: bidirectional encoder representations from transformers (BERT), the performance of information extraction from a free text by NLP has significantly improved for both the general domain and medical domain
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: only English speakers benefit due to the significant scarcity of high-quality medical documents, such as PubMed, in each language. Therefore, we propose a method to train a high-performance BERT model using a small corpus . We introduce the method to train a BERT model on a small medical corpus both in English and in Japanese , and we present the evaluation of each model
edit_actions: it is difficult to train specific BERT models that perform well for domains in which there are few publicly available databases of high quality and large size. We hypothesized that this problem can be addressed by up-sampling a domain-specific corpus and using it for pre-training with a larger corpus in a balanced manner. Our proposed method consists of a single intervention with one option: simultaneous pre-training after up-sampling and amplified vocabulary. We conducted three experiments and evaluated the resulting products. We confirmed that our Japanese medical BERT outperformed conventional baselines and the other BERT models in terms of the medical document classification task and that our English BERT pre-trained using both the general and medical-domain corpora performed sufficiently well for practical use
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: and the medical document classification task in Japanese, respectively. After confirming their satisfactory performances, we applied our method to develop a modelcomparable to the publicly available models. OuBioBERT, short for Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University, achieved the best score in terms
edit_actions: . Moreover, our enhanced biomedical BERT model, in which clinical notes were not used during pre-training, showed that both the clinical and biomedical scores
edit_actions: coherence
edit_actions: coherence clarity coherence
edit_actions: R
edit_actions: . The total score is 1.1 points above that of BioBERT and
edit_actions: were
edit_actions: clarity
edit_actions: clarity clarity coherence
edit_actions: R
edit_actions: ablated
edit_actions: ablation
edit_actions: fluency
edit_actions: fluency style fluency
edit_actions: R
edit_actions: This proposed technique is an effective approach to develop localized medical BERT models and to enhance domain-specific models in the biomedical domain
edit_actions: Well-balanced pre-training by up-sampling instances derived from a corpus appropriate for the target task allows us to construct a high-performance BERT model
edit_actions: clarity
edit_actions: clarity style clarity