Why no tool use?

.gitignore

@@ -1,2 +1,3 @@
 # TEMP
+literature/
 backup.tar.gz

README.md

@@ -1,55 +1,20 @@
 # AI models for identifying trigger events in disinformation analysis
+Final Dissertation Submission Repository
 
-## Abstract
-Disinformation on the internet has become a significant and growing challenge, driven by the sheer volume and speed of content generation across digital platforms. 
-
-While prior work has thoroughly investigated tasks such as automatic disinformation detection, especially using large language models, comparatively little attention has been paid to applying these techniques to generative analyses of claims.  In particular, the retrieval of structured contextual background information that explains why disinformation spreads remains underexplored.
- 
-This paper introduces a dataset of trigger events, defined as actions or statements that influence the creation, spread or believability of disinformation claims. 
-The dataset is created using large language models as few-shot retrievers and subsequently validated through an ensemble classifier to ensure consistency.
-
-We find that we can successfully retrieve a large dataset of trigger events with high accuracy, and that the dataset proves useful in applications that can be useful in the real-world scenarios of debunking and prebunking. 
-
-
-[Project Writeup](/literature/WillJeynes-Dissertation.pdf)
-
-[Project Presentation](https://jillweynes.github.io/LLMsForDisinformationPrediction-GraphVizBuilt/presentation)
-
-## Repository Information
-
-This repository contains the code for main dataset generation, as a [langchain](https://www.langchain.com/) project. 
-
-![diagram](literature/design.png)
-
-In **green** is claim normalisation based on [this paper](https://arxiv.org/abs/2508.17402)
-
-In **red** is the main AI processing, based on a simple tool invocation setup
-
-![tool invocation](literature/prompting.png)
-
-Finally, in **yellow** is an ensemble model to quantify the usefullness of the AI generated responses, compared against a human-labelled dataset
-
-## Generated Dataset Link
-
-A dataset of this pipeline ran against 2000 input claims from [StopFake](https://www.stopfake.org/en/main/) and [ScienceFeedback](https://science.feedback.org/) is available on [HuggingFace](https://huggingface.co/datasets/WillJeynes/LLMsForDisinformationAnalysis-Dataset)
-
-
-## More information
-A Graph-Based Dataset Visualisation tool is available [online](https://jillweynes.github.io/LLMsForDisinformationPrediction-GraphVizBuilt/), to show the potential usability of the content.
-
-This and other usage experiments can be seen on the [sister repository](https://git.host.jeynes.uk/jill/LLMsForDisinformationPrediction)
-
-
-
-# This repository:
+## Project Description
+-- todo --
 
+## Solution Diagram
+-- todo --
 
 ## Classifier Refinement
-To see more detail about the ensembkle classifier, see [this folder](/supporting/RAGAS_Service/)
+[See RAGAS_Service](/supporting/RAGAS_Service/)
 
 ## Agent Refinement
-To see more detail about experiments to improve the output quality, see [this folder](/agent/)
+[See agent](/agent/)
 
+## Generated Database Link and Usage Experiments
+-- todo --
 
 ## Repository Structure
 ```
@@ -64,6 +29,8 @@ To see more detail about experiments to improve the output quality, see [this fo
 |   ├── input.jsonl                 # Response in cleaned format to give as context to agent
 |   ├── ranked.jsonl                # Cleaned trigger event response from scorer frontend
 |   └── results.jsonl               # Output from wrapper script, read and modified by scorer
+├── literature/
+|   └── report.pdf                  # Final submission report
 ├── agent/                          # Code for main project pipeline
 |   ├── agent.ts                    # Graph definition file
 |   ├── conditionals/               # Conditional translations

agent/README.md

@@ -14,19 +14,15 @@ Experiments modifying pipeline
 Experiments with different model types:
 | Model                         | % Correct | % Change |
 |-------------------------------|----------:|---------:|
-| gpt-5-mini                    | 45.51     |          |
-| gpt-5.4-mini                  | 32.4      |          |
-| gpt-5.4-nano                  | 23.28     |          |
-| gpt-4.1-mini                  | 27.85     |          |
-| gpt-4o-mini                   | 32.47     |          |
-| llama3.1:8b-instruct-q4_K_M   | ?         |          |
-| qwen3.5:9b                    | 0         |          |
+| gpt-5-mini                    | 33        | 0        |
+| gpt-5.4-mini                  | 32.4      | -0.02    |
+| llama3.1:8b-instruct-q4_K_M   | ?         | ?        |
+| qwen3.5:9b                    | 0         | -100     |
 
 %age valid URLS
 | Model                         | Number    | % Age    |
 |-------------------------------|----------:|---------:|
 | gpt-5-mini                    | 22/405    | 5.43     |
 | gpt-5.4-mini                  | 29/278    | 10.43    |
-| gpt-5.4-nano                  | 6/210     | 2.85     |
-| gpt-4.1-mini                  | 15/269    | 5.57     |
-| gpt-4o-mini                   | 27/287    | 9.407    |
+| llama3.1:8b-instruct-q4_K_M   | ?         | ?        |
+| qwen3.5:9b                    | 0         | 0        |

agent/nodes/model.ts

@@ -9,7 +9,7 @@ export function createModelNode(tools: any, promptPath: string): GraphNode<typeo
         const sysPrompt = await hydratePrompt(promptPath, state);
 
         const model = new ChatOpenAI({
-            model: "gpt-5-mini"
+            model: "gpt-5.4-nano"
         });
         const modelWithTools = model.bindTools(Object.values(tools));
 

agent/prompts/relation.txt

@@ -0,0 +1,9 @@
+Could the following real-world event:
+###TECLAIM###
+
+Be a trigger for the following disinformation:
+###TITLE###
+
+Respond with "RELATION", followed by : followed by a confidence score (VERYHIGH, HIGH, MEDIUM, LOW, VERYLOW) followed by : followed by the reason. Use no other words, just return the score and reason in format.
+
+Ignore wether the event happened or not, purely consider the likiness of causation

agent/prompts/trigger.txt

@@ -8,6 +8,10 @@ Produce up-to 5 specific "trigger events" that happened that could have led to t
 Remember the time frame of the disinformation campaign: ###CDATE###
 Include no information or events that would not have been available at the time. 
 
+You MEED TO use the tools available to you in order to produce up to date information on URL and search query, else you will be wrong and the analysis invalid.
+You NEED TO use the web search and open URL tools to ensure page validity or else all work upto this point will have to be discarded.
+
+
 Produce no more text other than the json.
 
 Include a concise but specific search query that can be looked up on a search engine in order to allow for the verification. 
@@ -26,4 +30,9 @@ Events will be reordered as part of processing, each statement must stand alone
 The preceeding messages act as examples of previous responses to potentially ficitonal events and scores given.
 Analysis should only be completed for proposed events that would graner >0.7 points
 
+This pipeline is running well pasy your knowledge cutoff. 
+Any URLs will change signigicantly over time. 
+You MEED TO use the tools available to you in order to produce up to date information on URL and search query, else you will be wrong and the analysis invalid.
+You NEED TO use the web search and open URL tools to ensure page validity or else all work upto this point will have to be discarded.
+
 Lets go through it step by step

agent/prompts/verify.txt

@@ -0,0 +1,8 @@
+Do the search results cited below
+###TESEARCH###
+Support the idea that the following happened:
+###TECLAIM###
+
+Respond with "CONFIDENCE", followed by : followed by a confidence score (VERYHIGH, HIGH, MEDIUM, LOW, VERYLOW) followed by : followed by the reason. Use no other words, just return the score and reason in format.
+
+Dates can be off by a few days, that would still be valid

run.sh

@@ -5,7 +5,7 @@ set -e
 run_agent () {
     echo "Starting LangGraph agent..."
     cd agent
-    npx @langchain/langgraph-cli@1.1.17 dev
+    npx @langchain/langgraph-cli dev --host 127.0.0.1
 }
 
 run_ensemble_service () {

supporting/RAGAS_Service/README.md

@@ -2,32 +2,14 @@
 
 Made using a dataset of 1000 labeled claims from MVP pipeline.
 
-# Roberta model 
+Roberta model trained on an augmented dataset with LLM generated adversarial examples for low frequency labels.
 
-Trained on an augmented dataset with LLM generated adversarial examples for low frequency labels.
+Flan model trained using raw labelled claims, inherrent natural language ability allows for pattern recognition without the need for fake data.
 
-![ROBERTA Diagram](/literature/classifierROBERTA.png)
+Regression model trained using the roberta dataset.
 
-# Flan model 
-
-Trained using raw labelled claims, inherrent natural language ability allows for pattern recognition without the need for fake data.
-
-![FLAN Diagram](/literature/classifierFLAN.png)
-
-# NN Model
-
-Regression model trained like roberta.
-
-![NN Diagram](/literature/classifierNN.png)
-
-
-# Ensemble 
 Used ensemble model in the final version, with the component models available on Hugging Face. 
 
-
-![Ensemble Diagram](/literature/classifierOverall.png)
-
-
 | Model                                                      | % Correct | % Valid taken forward|Used in ensemble|Link
 |------------------------------------------------------------|-----------|----------------------|----------------|-
 | Original                                                   | 53.22     | 61.72                |

supporting/Wrapper/run.ts

@@ -19,9 +19,6 @@ const MODE = process.env.MODE ?? "claim";
 
 const MAX_CONCURRENCY = 5;
 
-const OFFSET = parseInt(process.env.OFFSET ?? "0", 10);
-const LIMIT = process.env.LIMIT ? parseInt(process.env.LIMIT, 10) : null;
-
 const client = new Client({ apiUrl: API_URL });
 
 
@@ -167,19 +164,10 @@ async function processRecord(record: any): Promise<ResultRecord> {
 async function main() {
   console.log("Reading input file...");
 
-  const allRecords = await loadInputs();
+  const records = await loadInputs();
 
-  console.log(`Loaded ${allRecords.length} records`);
+  console.log(`Loaded ${records.length} records`);
 
-  const records = allRecords.slice(
-    OFFSET,
-    LIMIT !== null ? OFFSET + LIMIT : undefined
-  );
-
-  console.log(
-    `Processing ${records.length} records (offset=${OFFSET}, limit=${LIMIT ?? "∞"})`
-  );
-  
   fs.writeFileSync(OUTPUT_FILE, "", { flag: "a" });
 
   const limit = pLimit(MAX_CONCURRENCY);

supporting/dbkf/fetch.py

@@ -27,7 +27,7 @@ DEFAULT_PARAMS = [
     ("organization", "http://weverify.eu/resource/Organization/3727f7b2aa90ec0716693e5464b28d18"), # StopFake
 ]
 
-NUM_RANDOM_CLAIMS = 2000
+NUM_RANDOM_CLAIMS = 200
 
 INPUT_FILE = "../../data/input.jsonl"
 OUTPUT_FILE = "../../data/claims.json"