before_revision: Bidirectional Encoder Representations from Transformers (BERT) models for biomedical specialties such as BioBERT and clinicalBERT  have significantly improved in biomedical text-mining tasks and enabled us to extract valuable information from biomedical literature . However, we benefitted only in English because of the significant scarcity of high-quality medical documents, such as PubMed, in each language. Therefore, we propose a method that realizes a high-performance BERT model by using a small corpus. We introduce the method to train a BERT model on a small medical corpus both in English and Japanese, respectively, and then we evaluate each of them in terms of the biomedical language understanding evaluation (BLUE) benchmark and the medical-document-classification task in Japanese, respectively. After confirming their satisfactory performances, we apply our method to develop a model that outperforms the pre-existing models. Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University (ouBioBERT) achieves the best scores on 7 of the 10 datasets in terms of the BLUE benchmark. The total score is 1.0 points above that of BioBERT  .
after_revision: Bidirectional Encoder Representations from Transformers (BERT) models for medical specialties, such as BioBERT and clinicalBERT , have significantly improved in performing biomedical text mining tasks and have enabled extracting valuable information from biomedical literature ; however, only English speakers benefit due to the significant scarcity of high-quality medical documents, such as PubMed, in each language. Therefore, we propose a method to train a high-performance BERT model using a small corpus. We introduce the method to train a BERT model on a small medical corpus both in English and in Japanese, and we present the evaluation of each model in terms of the biomedical language understanding evaluation (BLUE) benchmark and the medical document classification task in Japanese, respectively. After confirming their satisfactory performances, we applied our method to develop a model comparable to the publicly available models. OuBioBERT, short for Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University , achieved the best score in terms of the BLUE benchmark. The total score is 1.1 points above that of BioBERT and 0.3 points above that of the ablated model trained without our proposed method. This proposed technique is an effective approach to develop localized medical BERT models and to enhance domain-specific models in the biomedical domain .
edit_actions: R
edit_actions: biomedical specialties
edit_actions: medical specialties,
edit_actions: style
edit_actions: style style style
edit_actions: A
edit_actions: ,
edit_actions: fluency
edit_actions: fluency fluency fluency
edit_actions: R
edit_actions: biomedical text-mining tasks and enabled us to extract
edit_actions: performing biomedical text mining tasks and have enabled extracting
edit_actions: clarity
edit_actions: clarity style clarity
edit_actions: R
edit_actions: . However, we benefitted only in English because of
edit_actions: ; however, only English speakers benefit due to
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: R
edit_actions: that realizes
edit_actions: to train
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: D
edit_actions: by
edit_actions: coherence
edit_actions: coherence clarity coherence
edit_actions: R
edit_actions: Japanese, respectively, and then we evaluate each of them
edit_actions: in Japanese, and we present the evaluation of each model
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: medical-document-classification
edit_actions: medical document classification
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: R
edit_actions: apply
edit_actions: applied
edit_actions: fluency
edit_actions: fluency fluency fluency
edit_actions: R
edit_actions: that outperforms the pre-existing models.
edit_actions: comparable to the publicly available models. OuBioBERT, short for
edit_actions: clarity
edit_actions: clarity meaning-changed clarity
edit_actions: R
edit_actions: (ouBioBERT) achieves the best scores on 7 of the 10 datasets
edit_actions: , achieved the best score
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: 1.0
edit_actions: 1.1
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: A
edit_actions: and 0.3 points above that of the ablated model trained without our proposed method. This proposed technique is an effective approach to develop localized medical BERT models and to enhance domain-specific models in the biomedical domain
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed