---
library_name: transformers
language:
- en
metrics:
- rouge
base_model:
- meta-llama/Llama-3.2-3B-Instruct
---

# Model Details
LoRA finetuned checkpoint of a meta-llama/Llama-3.2-3B-Instruct base model. This model can be loaded on an M3 Macbook Air with 16GB unified memory.

### Model Description
<!-- Provide a longer summary of what this model is. -->
This model assists users with searching for research papers. It assists in creating a query that is compatible with a search API.
The model is finetuned to output structured markdown corresponding to the user query. This makes it possible to parse the output and construct a query for a search API.

### Model Sources

- **Repository:** https://github.com/shaikh58/llm-paper-retriever
- **Developed by:** Mustafa Shaikh
- **Language(s) (NLP):** English
- **License:** MIT
- **Finetuned from model:** meta-llama/Llama-3.2-3B-Instruct

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
This model is intended to be used with the MCP server released in the repository linked above. It is complete with search functionality and is integrated with Cursor.


## How to Get Started with the Model

If you wish to use the model directly, rather than through Cursor, you can use the code below to load it.

```
from transformers import AutoModelForCausalLM
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    trust_remote_code=True,
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    "Shaikh58/llama-3.2-3b-instruct-lora-arxiv-query"
)
```

## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

| Input query | Label |
|------------|------------------------------|
| "Find recent papers on transformer architectures in NLP published since 2023 with at least 100 citations" | ```"## QUERY PARAMETERS\n\n- **Topic**: NLP\n\n## CONSTRAINTS\n\n- **Citations**: (>=, 100)\n- **Keyword**: transformers\n- **Year**: (>=, 2023)\n\n## OPTIONS\n\n- **Limit**: 10\n- **Sort By**: relevance\n- **Sort Order**: descending"``` | 

During training, the input query is also augmented with a system prompt (not shown) to guide the model to output structured markdown.

### Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
LoRA finetuned on 50,000 synthetically generated training data points.


#### Training Hyperparameters

- **Training regime:**
- fp16 mixed precision <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
- LoRA: r = 16, alpha = 32, dropout = 0.05


## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

Same format as training data.


#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

The model was evaluated with the rouge metric. This is because the expected output is known in advance.

### Results
Several versions of the model were evaluated, each with a different number of trianing samples used during fine tuning. The plots show that finetuning with as low as 1000 samples leads to a major improvement in model performance. 
Empirically, we see that the model trained on 50,000 samples performs better in production, even though the rouge score is similar to models trained on less data. This is because the rouge score does not penalize minor differences to the expected output. However, 
minor differences can lead to very different parsing of the output and query result. 

<p float="left">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/670d6b862d412a30df9a5d0b/obQRVjgwML7y6JEjnd5rJ.png" width="300" >
  <img src="https://cdn-uploads.huggingface.co/production/uploads/670d6b862d412a30df9a5d0b/2RTl-3qNYv-DjCVvtTBlN.png" width="300" /> 
</p>