K-Monitor’s news database for corruption and public funds has been an important tool for nearly two decades, enabling users to search articles about corruption cases and problematic public spending in Hungary. Until now, the processing of these articles has been carried out by dedicated volunteers who have meticulously monitored and tagged the content from various Hungarian news portals following a strict methodology. This database, which now contains over 50,000 articles, has been a crucial resource in training our new, AI-based press monitoring system. While we will continue to rely on our volunteer community, the development of the automated interface has made their work much easier.

What makes the K-Monitor database unique is its consistent tagging system, which helps users quickly find information related to specific public fund or corruption stories, whether it concerns individuals, institutions, topics, or locations. Manual data entry is time-consuming, so it became necessary to develop an automated solution—this demand fueled our experiment to enhance the process with a language model-based development.

The first operational version of the new system retrieves news articles from the RSS feeds of online newspapers, extracting content from the web pages. Our algorithms then standardize the extracted data, perform some error filtering, and finally, a BERT language model, trained for classification, decides whether the article fits the methodology of the news database. If it does, the article is presented for approval to our monitoring volunteers.

Currently, we monitor nearly 100 Hungarian online news sources, with our system reviewing around 3,000 articles daily.

Automation doesn't stop at identifying articles and classifying them according to our methodology; we have also focused on recognizing individuals, institutions, and locations mentioned within the articles. The algorithm identifies named entities in the text and is (to some extent) able to recognize whether they are active participants in the described events or merely passive characters or narrators of the story. If a person or institution appears in a relevant context, the new system suggests tagging the entity for our monitors who record the article. This solution greatly enhances the accuracy and efficiency of the article tagging process.

During development, we faced challenges, particularly in fine-tuning the language models to the peculiarities of the Hungarian language, improving the accuracy of automatic context recognition, and detecting articles that have been copied or come from news agencies. Beyond the successful developments so far, there is still much potential for further improvement of the system. Future plans include expanding the role of artificial intelligence to recognize tags describing the circumstances of the articles and files grouping cases, storing data about the authors, and grouping similar articles.

Automation is a long-term project that not only makes the work of our volunteers easier but also ensures more reliable and comprehensive content for users. Feel free to contact us at info@k-monitor.hu if you have any suggestions or you’re working on a similar project.

The 2024 development, in addition to the work of K-Monitor's staff, was also supported by pro bono IT assistance from Spreadmonitor. We are incredibly grateful to them and to our volunteers and team members who have worked on the database.