| RoCBert | |
| Overview | |
| The RoCBert model was proposed in RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining by HuiSu, WeiweiShi, XiaoyuShen, XiaoZhou, TuoJi, JiaruiFang, JieZhou. | |
| It's a pretrained Chinese language model that is robust under various forms of adversarial attacks. | |
| The abstract from the paper is the following: | |
| Large-scale pretrained language models have achieved SOTA results on NLP tasks. However, they have been shown | |
| vulnerable to adversarial attacks especially for logographic languages like Chinese. In this work, we propose | |
| ROCBERT: a pretrained Chinese Bert that is robust to various forms of adversarial attacks like word perturbation, | |
| synonyms, typos, etc. It is pretrained with the contrastive learning objective which maximizes the label consistency | |
| under different synthesized adversarial examples. The model takes as input multimodal information including the | |
| semantic, phonetic and visual features. We show all these features are important to the model robustness since the | |
| attack can be performed in all the three forms. Across 5 Chinese NLU tasks, ROCBERT outperforms strong baselines under | |
| three blackbox adversarial algorithms without sacrificing the performance on clean testset. It also performs the best | |
| in the toxic content detection task under human-made attacks. | |
| This model was contributed by weiweishi. | |
| Resources | |
| Text classification task guide | |
| Token classification task guide | |
| Question answering task guide | |
| Causal language modeling task guide | |
| Masked language modeling task guide | |
| Multiple choice task guide | |
| RoCBertConfig | |
| [[autodoc]] RoCBertConfig | |
| - all | |
| RoCBertTokenizer | |
| [[autodoc]] RoCBertTokenizer | |
| - build_inputs_with_special_tokens | |
| - get_special_tokens_mask | |
| - create_token_type_ids_from_sequences | |
| - save_vocabulary | |
| RoCBertModel | |
| [[autodoc]] RoCBertModel | |
| - forward | |
| RoCBertForPreTraining | |
| [[autodoc]] RoCBertForPreTraining | |
| - forward | |
| RoCBertForCausalLM | |
| [[autodoc]] RoCBertForCausalLM | |
| - forward | |
| RoCBertForMaskedLM | |
| [[autodoc]] RoCBertForMaskedLM | |
| - forward | |
| RoCBertForSequenceClassification | |
| [[autodoc]] transformers.RoCBertForSequenceClassification | |
| - forward | |
| RoCBertForMultipleChoice | |
| [[autodoc]] transformers.RoCBertForMultipleChoice | |
| - forward | |
| RoCBertForTokenClassification | |
| [[autodoc]] transformers.RoCBertForTokenClassification | |
| - forward | |
| RoCBertForQuestionAnswering | |
| [[autodoc]] RoCBertForQuestionAnswering | |
| - forward |