SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: sentence-transformers/paraphrase-mpnet-base-v2
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 5 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
Product inquiry	'The Amazon Echo is unable to establish a stable internet connection. It keeps disconnecting intermittently. How can I troubleshoot this network problem?\n\nOn Windows 8 and higher, please check in "Network settings". You This problem started occurring after the recent software update. I haven't made any other changes to the device.' "I'm having an issue with the Asus ROG. Please assist.\n\nThank you.\n\nThe same guy who is already selling my house and is being charged with assault, theft, and battery in my home, also has I've noticed that the issue occurs consistently when I use a specific feature or application on my Asus ROG." 'I'm having an issue with the Lenovo ThinkPad. Please assist. :) " " "My Product Purchases":[ " ", ", "",, ""," ", ", ", ", ", ", ", I've tried clearing the cache and data for the Lenovo ThinkPad app, but the issue persists.'
Billing inquiry	"I'm having an issue with the Amazon Kindle. Please assist. I've recently updated the firmware of my Amazon Kindle, and the issue started happening afterward. Could it be related to the update?" "There seems to be a hardware problem with my Sony Xperia. The screen is flickering, and I'm unable to use it. What should I do?\n\nI had no idea that the device was behaving this way and started running I need assistance as soon as possible because it's affecting my work and productivity." "There seems to be a hardware problem with my Garmin Forerunner. The screen is flickering, and I'm unable to use it. What should I do?I will start a repair. What should I do with it? I've performed a factory reset on my Garmin Forerunner, hoping it would resolve the problem, but it didn't help."
Refund request	"I'm having an issue with the Nikon D. Please assist.\n\nPlease enable JavaScript to view the comments powered by Disqus.\n\n\nDisqus I'm concerned about the security of my Nikon D and would like to ensure that my data is safe." "I'm having an issue with the Samsung Soundbar. Please assist. Please help me determine if it's warranted.\n\nIf it is, please let me know and I'll do my best to get as much of it out before This problem started occurring after the recent software update. I haven't made any other changes to the device." "I'm having an issue with the Microsoft Surface. Please assist. I'm worried that the issue might be hardware-related and might require repair or replacement."
Cancellation request	"There seems to be a hardware problem with my Garmin Forerunner. The screen is flickering, and I'm unable to use it. What should I do?\n\nWhat I should do:\n\nTo revert to normal behavior I've tried using different cables, adapters, or peripherals with my Garmin Forerunner, but the issue persists." "I'm having an issue with the Lenovo ThinkPad. Please assist.\n\nMy code does not show anything about this line, which is why I deleted it here.\n\nIf you are still in pain, ask the maintainers I'm experiencing this issue on multiple devices of the same model, so it seems to be a widespread problem." "I'm having an issue with the Garmin Forerunner. Please assist. I've recently updated the firmware of my Garmin Forerunner, and the issue started happening afterward. Could it be related to the update?"
Technical issue	"I'm having an issue with the iPhone. Please assist. I appreciate your help and help to me. This product uses cookies to make this website possible. Thank you for your feedback. I'm unable to find the option to perform the desired action in the iPhone. Could you please guide me through the steps?" "I'm having an issue with the Adobe Photoshop. Please assist. If your product is purchased with the Adobe Photoshop, the Adobe Photoshop will be refunded.\n\n7) Pay it I've performed a factory reset on my Adobe Photoshop, hoping it would resolve the problem, but it didn't help." "I'm having an issue with the GoPro Hero. Please assist.\n\nIt looks like an auto-complete can be found on your phone's dashboard and in most cases. But, this thing has so many questions. The problem I need assistance as soon as possible because it's affecting my work and productivity."

Evaluation

Metrics

Label	Accuracy
all	0.1

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("neha-duggirala/customer-support-classifier_setfit-trainer")
# Run inference
preds = model("I'm having an issue with the LG OLED. Please assist.

If you have a issue with this, simply contact me on the forums. I've recently updated the firmware of my LG OLED, and the issue started happening afterward. Could it be related to the update?")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	24	46.3	60

Label	Training Sample Count
Billing inquiry	8
Cancellation request	8
Product inquiry	8
Refund request	8
Technical issue	8

Training Hyperparameters

batch_size: (5, 5)
num_epochs: (5, 5)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 5
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0125	1	0.3545	-
0.625	50	0.2266	-
1.25	100	0.2179	-
1.875	150	0.1296	-
2.5	200	0.0492	-
3.125	250	0.0478	-
3.75	300	0.028	-
4.375	350	0.0147	-
5.0	400	0.011	-

Framework Versions

Python: 3.10.12
SetFit: 1.1.1
Sentence Transformers: 3.3.1
Transformers: 4.47.0
PyTorch: 2.5.1+cu121
Datasets: 3.3.1
Tokenizers: 0.21.0

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

neha-duggirala
/

customer-support-classifier_setfit-trainer