jill/LLMsForDisinformationAnalysis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Re-allow multithreading on service. Add results table

Browse Source
This commit is contained in:
William Jeynes
2026-03-24 18:29:40 +00:00
parent 80bc151379
commit 624d45bc53
4 changed files with 31 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
supporting/RAGAS_Service/README.md
+13
View File
@@ -0,0 +1,13 @@
| Model | % Correct | % Valid taken forward|Used in ensemble|Link
|------------------------------------------------------------|-----------|----------------------|----------------|-
| Original | 53.22 | 61.72 |
| Original (RAGAS) | 56.01 | 57.73 |
| Roberta (base) | 75 | 70 |
| Roberta (Generated Data) | 76 | 71 |
| Roberta (Generated Data + Back Translation) | 74 | 71 |
| Roberta (Generated Data + Back Translation + Thresholding) | 77 | 90 |Y|[Here](https://huggingface.co/WillJeynes/LLMsForDisinformationAnalysis)
| Distilled Roberta | 72.73 | 69.57 |
| Flan | 79.17 | 85.71 |Y|[Here](https://huggingface.co/WillJeynes/LLMsForDisinformationAnalysis-Flan)
| Simple Regression Model | 74.77 | 85.71 |Y|[Here](https://huggingface.co/WillJeynes/LLMsForDisinformationAnalysis-Regression)
| Ensemble Model (weighted confidence score sum) | 84.21 | 83.33 |
| Ensemble Model (majority voting) | 80.2 | 95.12 |
supporting/RAGAS_Service/ensemble_service.py
+5 -5
View File
@@ -102,6 +102,11 @@ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
flan_model.to(device)
flan_model.eval()
label_token_ids = {
label: flan_tokenizer(label, add_special_tokens=False).input_ids[0]
for label in LABEL_TO_INT.keys()
}
def format_prompt(text: str) -> str:
return (
@@ -204,11 +209,6 @@ def evaluate(req: EvalRequest):
skip_special_tokens=True
)
label_token_ids = {
label: flan_tokenizer(label, add_special_tokens=False).input_ids[0]
for label in LABEL_TO_INT.keys()
}
label_logits = torch.tensor(
[logits[0, tid].item() for tid in label_token_ids.values()]
)