license: cc-by-nc-4.0
datasets:
- tatsu-lab/alpaca
language:
- en
Eluwa: A Conversational LoRA for Facebook's OPT 2.7b Architecture
Eluwa is a fine-tuned Low-Rank Adapter (LoRA) model for Facebook's OPT 2.7b. It is trained on the Stanford Alpaca dataset. The idea was that OPT 2.7 was too curt (and frankly, a bit of an asshole) for a model of its size, and that we could finetune it like Alpaca did to Llama.
This repository contains the Eluwa 2.7b 2 epoch model, which represents a significant improvements in question-answering ability compared to the default OPT 2.7b model. Below are the results of Vicuna-style testing: 80 questions in various categories, with the responses rated by GPT-4.
| Model | OPT 2.7b base | Eluwa 2.7b 1000 iter | Eluwa 2.7b 2 epoch |
|---|---|---|---|
| Generic | 22 | 44 | 57 |
| Knowledge | 35 | 60 | 72 |
| Roleplay | 29 | 38 | 58 |
| Common sense | 20 | 48 | 50 |
| Fermi | 4 | 28 | 23 |
| Counterfactual | 5 | 24 | 23 |
| Coding | 2 | 7 | 7 |
| Math | 0 | 3 | 3 |
| Writing | 8 | 19 | 19 |
| Total | 125 | 271 | 312 |
(A sheet of questions, answers and GPT's reviews are also included in this repo).
Because of its small size, Eluwa can be used as research into conversational models with older and slower hardware. To load it in a UI like oobabooga, download the model's .bin and .json files, put them in a folder inside the /loras folder, and load it with the OPT 2.7b model.