# AI models for identifying trigger events in disinformation analysis

## Abstract
Disinformation on the internet has become a significant and growing challenge, driven by the sheer volume and speed of content generation across digital platforms. 

While prior work has thoroughly investigated tasks such as automatic disinformation detection, especially using large language models, comparatively little attention has been paid to applying these techniques to generative analyses of claims.  In particular, the retrieval of structured contextual background information that explains why disinformation spreads remains underexplored.
 
This paper introduces a dataset of trigger events, defined as actions or statements that influence the creation, spread or believability of disinformation claims. 
The dataset is created using large language models as few-shot retrievers and subsequently validated through an ensemble classifier to ensure consistency.

We find that we can successfully retrieve a large dataset of trigger events with high accuracy, and that the dataset proves useful in applications that can be useful in the real-world scenarios of debunking and prebunking. 


[Project Writeup](/literature/WillJeynes-Dissertation.pdf)

[Project Presentation](https://jillweynes.github.io/LLMsForDisinformationPrediction-GraphVizBuilt/presentation)

## Repository Information

This repository contains the code for main dataset generation, as a [langchain](https://www.langchain.com/) project. 

![diagram](literature/design.png)

In **green** is claim normalisation based on [this paper](https://arxiv.org/abs/2508.17402)

In **red** is the main AI processing, based on a simple tool invocation setup

![tool invocation](literature/prompting.png)

Finally, in **yellow** is an ensemble model to quantify the usefullness of the AI generated responses, compared against a human-labelled dataset

## Generated Dataset Link

A dataset of this pipeline ran against 2000 input claims from [StopFake](https://www.stopfake.org/en/main/) and [ScienceFeedback](https://science.feedback.org/) is available on [HuggingFace](https://huggingface.co/datasets/WillJeynes/LLMsForDisinformationAnalysis-Dataset)


## More information
A Graph-Based Dataset Visualisation tool is available [online](https://jillweynes.github.io/LLMsForDisinformationPrediction-GraphVizBuilt/), to show the potential usability of the content.

This and other usage experiments can be seen on the [sister repository](https://git.host.jeynes.uk/jill/LLMsForDisinformationPrediction)



# This repository:


## Classifier Refinement
To see more detail about the ensembkle classifier, see [this folder](/supporting/RAGAS_Service/)

## Agent Refinement
To see more detail about experiments to improve the output quality, see [this folder](/agent/)


## Repository Structure
```
├── run.sh                          # Bash script to run project elements from one place
├── data/                           # Holder from project data
|   ├── blocked.jsonl               # Web search results blocked by the Iffy list
|   ├── error.log                   # Log file containing critical exceptions
|   ├── claims.json                 # Retreived claims from dbkf fetcher
|   ├── dev-eng.csv
|   ├── train-eng.csv               # Normalized disinformation claims in CSV format from CLAN
|   ├── Iffy.json                   # Iffy dataset of disinformation domains
|   ├── input.jsonl                 # Response in cleaned format to give as context to agent
|   ├── ranked.jsonl                # Cleaned trigger event response from scorer frontend
|   └── results.jsonl               # Output from wrapper script, read and modified by scorer
├── agent/                          # Code for main project pipeline
|   ├── agent.ts                    # Graph definition file
|   ├── conditionals/               # Conditional translations
|   ├── prompts/                    # System promps, plus replacement code
|   ├── tools/                      # Internal and LLM facing tools
|   └── utils/                      # Logger
└── supporting/                     
    ├── dbkf/                       # Tool to download claims from DBKF for use in wrapper
    ├── RAGAS_Service               # Small python API to make RAGAS metrics available in the TS projects (required to run pipeline)
    ├── scorer                      # Frontend for labelling data, plus associated analysis
    └── Wrapper                     # Bulk run pipeline on pre-downloaded claims
```
