sgzsh269/fingpt-sentiment_llama2-13b_merged

Model Info

• This model is a merge of the following models:

  • Base Model: NousResearch/Llama-2-13b-hf
  • LoRA Model: FinGPT/fingpt-sentiment_llama2-13b_lora

• Primary Usecase: Financial Sentiment Analysis

Model Merging Method

The base model was dequantized before merging to retain the performance of base model + LoRA adapter. Check the below resources for more details:

• https://medium.com/@bnjmn_marie/dont-merge-your-lora-adapter-into-a-4-bit-llm-65b6da287997
• https://gist.github.com/ChrisHayduk/1a53463331f52dca205e55982baf9930

import torch
import os
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM
from peft import PeftModel

base_model_name = "NousResearch/Llama-2-13b-hf" 
peft_model_name = "FinGPT/fingpt-sentiment_llama2-13b_lora"
output_path = "merged_model"

model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto",
    load_in_8bit=True
)

# Dequantize base model weights to match precision of LoRA weights
model.dequantize()

model = PeftModel.from_pretrained(model, peft_model_name)

merged_model = model.merge_and_unload()

merged_model.save_pretrained(output_path)

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
tokenizer.save_pretrained(output_path)

Inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

device = "cuda" if torch.cuda.is_available() else "cpu"

model_path = "sgzsh269/fingpt-sentiment_llama2-13b_merged"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map=device,
    torch_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(model_path)
tokenizer.pad_token = tokenizer.eos_token

model.eval()

prompts = [
'''Instruction: What is the sentiment of this news? Please choose an answer from {negative/neutral/positive}
Input: FINANCING OF ASPOCOMP 'S GROWTH Aspocomp is aggressively pursuing its growth strategy by increasingly focusing on technologically more demanding HDI printed circuit boards PCBs .
Answer: ''',
'''Instruction: What is the sentiment of this news? Please choose an answer from {negative/neutral/positive}
Input: According to Gran , the company has no plans to move all production to Russia , although that is where the company is growing .
Answer: ''',
'''Instruction: What is the sentiment of this news? Please choose an answer from {negative/neutral/positive}
Input: A tinyurl link takes users to a scamming site promising that users can earn thousands of dollars by becoming a Google ( NASDAQ : GOOG ) Cash advertiser .
Answer: ''',
]

model_inputs = tokenizer(prompts, return_tensors='pt', padding=True).to(device)

with torch.no_grad():
    generated_ids = model.generate(**model_inputs, max_length=512)

    output_ids = generated_ids[:, model_inputs.input_ids.shape[1]:].tolist()

    output_texts = tokenizer.batch_decode(output_ids, skip_special_tokens=True)

    print(output_texts) # Expected: ['positive', 'neutral', 'negative']