Disinformation on the internet has become a significant and growing challenge, driven by the sheer volume and speed of content generation across digital platforms.

While prior work has thoroughly investigated tasks such as automatic disinformation detection, especially using large language models, comparatively little attention has been paid to applying these techniques to generative analyses of claims. In particular, the retrieval of structured contextual background information that explains why disinformation spreads remains underexplored.

This paper introduces a dataset of trigger events, defined as actions or statements that influence the creation, spread or believability of disinformation claims. The dataset is created using large language models as few-shot retrievers and subsequently validated through an ensemble classifier to ensure consistency.

We find that we can successfully retrieve a large dataset of trigger events with high accuracy, and that the dataset proves useful in applications that can be useful in the real-world scenarios of debunking and prebunking.

Project Writeup

Project Presentation

Repository Information

This repository contains the code for main dataset generation, as a langchain project.

In green is claim normalisation based on this paper

In red is the main AI processing, based on a simple tool invocation setup

Finally, in yellow is an ensemble model to quantify the usefullness of the AI generated responses, compared against a human-labelled dataset

Generated Dataset Link

A dataset of this pipeline ran against 2000 input claims from StopFake and ScienceFeedback is available on HuggingFace

More information

A Graph-Based Dataset Visualisation tool is available online, to show the potential usability of the content.

This and other usage experiments can be seen on the sister repository

This repository:

Classifier Refinement

To see more detail about the ensembkle classifier, see this folder

Agent Refinement

To see more detail about experiments to improve the output quality, see this folder

Repository Structure

├── run.sh                          # Bash script to run project elements from one place
├── data/                           # Holder from project data
|   ├── blocked.jsonl               # Web search results blocked by the Iffy list
|   ├── error.log                   # Log file containing critical exceptions
|   ├── claims.json                 # Retreived claims from dbkf fetcher
|   ├── dev-eng.csv
|   ├── train-eng.csv               # Normalized disinformation claims in CSV format from CLAN
|   ├── Iffy.json                   # Iffy dataset of disinformation domains
|   ├── input.jsonl                 # Response in cleaned format to give as context to agent
|   ├── ranked.jsonl                # Cleaned trigger event response from scorer frontend
|   └── results.jsonl               # Output from wrapper script, read and modified by scorer
├── agent/                          # Code for main project pipeline
|   ├── agent.ts                    # Graph definition file
|   ├── conditionals/               # Conditional translations
|   ├── prompts/                    # System promps, plus replacement code
|   ├── tools/                      # Internal and LLM facing tools
|   └── utils/                      # Logger
└── supporting/                     
    ├── dbkf/                       # Tool to download claims from DBKF for use in wrapper
    ├── RAGAS_Service               # Small python API to make RAGAS metrics available in the TS projects (required to run pipeline)
    ├── scorer                      # Frontend for labelling data, plus associated analysis
    └── Wrapper                     # Bulk run pipeline on pre-downloaded claims