PDF Page 1
How public datasets constrain the development of diversity-aware news recommender systems, and what law could do about it.
MAX VAN DRUNEN, Institute for Information Law, University of Amsterdam, the Netherlands SANNE VRIJENHOEK, Institute for Information Law. University of Amsterdam, the Netherlands
News recommender systems increasingly determine what news individuals see online. Over the past decade, researchers have
extensively critiqued recommender systems that prioritise news based on user engagement. To offer an alternative, researchers have
analysed how recommender systems could support the media’s ability to fulfil its role in democratic society by recommending news
based on editorial values, particularly diversity. However, there continues to be a large gap between normative theory on how news
recommender systems should incorporate diversity, and technical literature that designs such systems. We argue that to realise
diversity-aware recommender systems in practice, it is crucial to pay attention to the datasets that are needed to train modern news
recommenders. We aim to make two main contributions. First, we identify the information a dataset must include to enable the
development of the diversity-aware news recommender systems proposed in normative literature. Based on this analysis, we assess
the limitations of currently available public datasets, and show what potential they do have to expand research into diversity-aware
recommender systems. Second, we analyse why and how European law and policy can be used to provide researchers with structural
access to the data they need to develop diversity-aware news recommender systems.
Additional Key Words and Phrases: Datasets, News Recommender Systems, Diversity, Law
1 Introduction
News recommender systems play an important role in determining what news (if any) individuals see online by ordering
information from most to least relevant [34, 42, 57]. Researchers have extensively criticised the common industry
practice of measuring relevance through clicks, arguing this leads media organisations to recommend news articles
based on what is most engaging rather than based on the editorial values that should guide the way the way the
media informs the public. [4, 16]. Over the past decade, researchers have therefore proposed alternative approaches
to news recommendation that would support the media’s democratic function of informing the public, particularly
by allowing media organisations to make more diverse recommendations that expose each reader to perspectives
and topics that are new to them. Researchers have analysed extensively how diversity-aware recommender systems
should be conceptualised [4, 35, 58, 71, 72], and what technical tools are needed to measure and embed diversity in
recommender systems [2, 31, 43, 70, 75].
However, a significant gap remains between the diversity-aware news recommender systems proposed in normative
literature, and the kinds that have been realized in technical literature. For example, scholars in law and journalism
studies call for recommender systems that allow readers to become experts in their chosen field by showing them
new perspectives on a few topics that interest them, or that foster tolerance by showing readers political viewpoints
with which they disagree but which do not lead them to see other societal groups as enemies [35, 41, 58]. In contrast,
technical literature most often implements diversity as an intra-list similarity function, to check whether the items in
the recommendation are not too similar to each other [7, 55]. While work that designs news recommender systems
based on more normative conceptualisations of diversity exists, it remains highly limited and regularly emphasises it is
not yet suitable for deployment. Much work has yet to be done before industry-level recommender systems can truly
account for normative interpretations of diversity [75].
Authors’ Contact Information: Max van Drunen, m.z.vandrunen@uva.nl, Institute for Information Law, University of Amsterdam, Amsterdam, the Netherlands; Sanne Vrijenhoek, s.vrijenhoek@uva.nl, Institute for Information Law. University of Amsterdam, Amsterdam, the Netherlands.
1
arXiv:2510.05952v1 [cs.IR] 7 Oct 2025
PDF Page 2
2 van Drunen and Vrijenhoek
In this paper, we argue that the public datasets on which researchers rely to create and evaluate news recommender
systems are a key factor limiting the development of diversity-aware recommender systems. We show that while these
datasets do have some untapped potential to further research into diversity-aware recommender systems, they lack
much of the data needed to create the kinds of recommender systems proposed based on normative theory. Instead,
they primarily facilitate the development of recommender systems that optimise for readers’ short-term preferences.
Further, we argue that this constraint cannot be fully addressed with the publication of another single dataset, as the
data needed for diverse recommendations varies significantly across time and regions. We therefore also explore the
European law and policy tools available to secure structural access to the data needed to develop diversity-aware news
recommender systems.
With this, we aim to answer the following research question: how do public datasets constrain the development of
diversity-aware news recommender systems, and how can European law and policy be used to overcome this constraint?
We focus on European law because of its recent advances in data access law and policy, its role for active governmental
intervention to support media pluralism, and its wide geographic scope (allowing it to, potentially, support the creation
of public datasets in multiple countries). We include the four most-used public datasets in our analysis, but focus in
particular on the Microsoft News Dataset (MIND). Due to its size, its publication in 2022 was a significant development
for news recommendation research, and our research indicates that it is by far the most used dataset to develop new
recommender systems.
Section 2 analyses how public datasets are used to develop news recommender systems, and how they can impact
the recommender systems used by media organisations by influencing the research that develops and evaluates the
performance of new recommender systems. Section 3 shows what data would have to be included in a dataset to
develop the kinds of diversity-aware recommender systems imagined in legal and journalism studies literature. Section
4 evaluates how the data included in the public datasets that are currently most used in news recommendation research
limits the development of diversity-aware recommender systems. Section 5 analyses why European law and policy
should play a role to address the lack of datasets needed to build better news recommender systems, and how existing
European laws and policies intended to expand access to data can do so. Throughout, we aim to make the paper
accessible for computer science, legal, and journalism scholars researching how more diverse news recommender
systems can be realised. The analysis may therefore at times be overly simplistic for one specific audience.
2 From datasets to news recommendations
Training a recommender system involves optimising it to predict a quantitatively measurable target; most often, whether
a user will click on an item or not [80]. A recommender system is trained to make these predictions by identifying
patterns between item characteristics and user characteristics that are associated with the click. These patterns are
subtle. For example, a click on an article about the Olympic Games may indicate the user is interested in the Olympic
Games, the politics surrounding it, or a specific team, sport, or athlete. The broader the scope of content a system needs
to cover, the harder it is to find the patterns that predict user behaviour. Furthermore, the way readers interact with a
system is highly connected to how content is shown to them, meaning that slight changes in the user interface can
lead to very different engagement patterns [3]. To capture these nuances, recommender systems are trained on large
datasets.
Data at this scale is hard to come by: one needs to have a recommender system running in order to generate a dataset
needed to train recommender systems. Media organisations resolve this chicken-and-egg situation by using simpler
recommender systems to generate their own datasets, rather than relying on pulic datasets directly. This is due to the
PDF Page 3
How public datasets constrain the development of diversity-aware news recommender systems, and what law could do about it. 3
aforementioned impact that the user interface has on the way users interact with a recommender system: using a public
dataset such as MIND (which was generated with microsoftnews.msn.com) to train a recommender system that will
run on the BBC will mean the BBC’s recommender system is optimised to identify patterns that differ from those in the
context in which it is deployed.
Researchers, however, rarely have direct access to the datasets needed to develop news recommender systems. Even
if they do, relying on a proprietary dataset limits the reproducibility of their results [16]. As a result, researchers and
practitioners that create and test new recommender system algorithms generally rely on publicly available datasets.
The research they produce on how recommender systems can be built, and which approaches are most successful, is in
turn a valuable resource for media organisations, as it saves them from developing their recommender systems from
scratch [19, 79].
One concrete example of how this works in practice can be seen in work by Einarsson, Helles & Lomborg, which
studied the effect of news recommender systems deployed by Ekstra Bladet during the 2022 Danish election [23]. Ekstra
Bladet had two recommender systems in production, both of which were trained on Ekstra Bladet’s own data. However,
the algorithms Ekstra Bladet used to train its recommender systems were developed and evaluated in the research
literature, using public datasets (NRMS and Matrix Factorization [54, 77]. By analogy, Ekstra Bladet used a recipe
(researchers’ recommender system algorithm) that was developed and evaluated using one set of ingredients (public
datasets) to create a specific dish (Ekstra Bladet’s recommender system) using their own ingredients (Ekstra Bladet’s
own dataset).
In short, the research media organisations rely on to determine how to design recommender systems and which
algorithms are most successful heavily relies on public datasets. Public datasets can shape this research in two main
ways. Firstly, the data included in a dataset constrains what recommender systems developed with the dataset can be
designed to do. ‘Simply’ training a recommender system that predicts what news users will click will already require
a large dataset of news articles and users’ interactions with those articles from which these patterns can be derived.
Developing a recommender system that balances user preferences with normative values such as diversity requires a
richer dataset, which also includes data necessary to assess how the recommendation relates to those values [8, 65]. For
example, training a recommender to expose users to a diverse set of viewpoints requires a training dataset that contains
data about the viewpoints expressed in its articles, or an effective proxy for this data.
Second, a dataset can become a benchmark. Indeed, MIND was created with the explicit purpose “to serve as a
benchmark dataset for news recommendation and to facilitate the research in news recommendation" [50]. Benchmark
datasets fulfil an important function in research: they serve as a common reference point by allowing the performance
of new recommendation algorithms to be compared to that of existing algorithms tested on the same dataset. However,
they also make it more likely that research is done on the data structures present in the benchmark dataset. A good
example of the practical implications of this dynamic is the MovieLens dataset, published in 2005 and still a widely used
dataset for recommender system research. MovieLens contained information on users rating a movie on a scale of 1 to
5, which meant that consequently, a lot of research was carried out on predicting scores on an ordinal scale [30].
3 Conceptualising diversity in news recommender systems
3.1 Data as a constraint on diversity-aware news recommender systems
Scholars from law and journalism studies have proposed many different ways in which recommendations can be
made more diverse, including not only accounting for the viewpoints but also the topics, styles, and formats of the
PDF Page 4
4 van Drunen and Vrijenhoek
recommended content, as well as the characteristics of the users to whom content is recommended. Indeed, there
arguably is not one right way to approach diversity. As the need for diverse news is often ultimately grounded in
democratic theory, the type of diversity a recommender system should promote depends on the kind of democratic
system (e.g., liberal, deliberative, or agonistic democracy) it is used to support [35]. Our goal here is not to advocate
for one specific approach to diversity. Nor do we argue a dataset should necessarily be suitable to realise all types of
diversity we will outline below. For example, the need for diversity may be outweighed by the privacy concerns of
collecting detailed data on the news consumption of marginalised groups. At this stage, we only aim to identify the
characteristics of recommendations that are important from normative perspectives on diversity, so we can evaluate
which aspects of diversity can and cannot be incorporated in news recommender system research with the data included
in currently popular public datasets.
3.2 What data is needed to make diverse recommendations?
The diversity of the topics of recommended articles plays a dominant role in the literature on diversity-aware news
recommender systems. Many authors highlight that by accounting for what users have seen in the past, recommender
systems could expose them to topics that are new to them, and thereby broaden their horizons or alert them to topics
affecting other societal groups [29, 66, 72]. Alternatively, recommender systems can show a user extensive information
on a few topics to create deeply informed expert citizens, or show individuals news on the topics they prefer to engage
with to support their autonomy [35, 38]. Between these extremes sit views that argue for a more mixed approach. A
mix of entertaining and hard news could make diversity-aware recommender systems pleasurable to use, while a mix
of political and non-political content could give individuals a fuller picture of society [5, 29, 35].
Viewpoint diversity is often addressed in conjunction with topic diversity, and concerns the diversity of perspectives
on a particular topic to which users should be exposed [28]. In general, authors advocate for showing users a wide
diversity of perspectives within the same topic, either to better inform them, or increase tolerance and social cohesion
[4, 35, 46, 72]. The different viewpoints to which users should be exposed are usually ideological in nature; several
authors explore how users should be exposed to perspectives from across the political spectrum [4, 31, 72] or to the
perspectives of other societal (e.g., ethnic, linguistic, or marginalised) groups to increase tolerance [1, 46, 58]. An
important but rarely explicit assumption in this context, is that the diversity of perspectives to which a recommender
system should expose users may differ depending on the political perspectives or societal groups that exist in the society
in which the system is deployed [76].
The style of news plays an important role in discussions on the type of public debate diversity-aware recommenders
should foster. Deliberative approaches that see media as creating space for an informed and rational debate imply the
recommended content should be impartial and promote active discourse [35]. Recommender systems that seek to foster
tolerance may recommend positive news about ideological opponents, or respectful discussions between opponents
[29]. Finally, from an agonistic perspective the goal of a recommender should be to facilitate conflict while ensuring the
different sides do not perceive each other as enemies [58].
The format in which news is recommended matters to recommender systems that respond to users’ consumption
preferences. If a recommender system is used to serve a wide cross-section of society, it is important to ensure news is
recommended in a format (such as video, audio, or text) with which different users can engage [13, 35, 65]. The need for
different formats is also implied in some of the approaches to diversity mentioned above. For example, a recommender
system that aims to enable users to become experts in a specific topic requires larger background articles and deep
dives for users that have a lot of pre-existing knowledge on the topic, and simple explainers for users that do not.
PDF Page 5
How public datasets constrain the development of diversity-aware news recommender systems, and what law could do about it. 5
Finally, few authors pay explicit attention to the characteristics of the users to whom content is recommended. One
exception is Vermeulen, who emphasises the need to balance diverse exposure with user choice [72]. Additionally,
detailed information about the user is required to provide the diverse recommendations described above. For example,
a recommender system must be able to assess what news a user knows already in order to give them a new perspective
or more deeply inform them. From a fundamental rights perspective, moreover, the need for diversity is also grounded
in the right to receive information, and more specifically the need to ensure that “everyone has access to a diverse
range of journalistic content” [14, 24]. In that context the Council of Europe has for example referred to the need to
ensure diverse news can be accessed despite socio-economic barriers and income level and that minorities, disabled
persons, and disadvantaged and local communities are able to access news [13, 14]. Similarly, writing on recommender
systems’ impact on epistemic welfare, [38] emphasise the importance of ensuring a recommender system is able to
“ensure that many and diverse users can encounter and engage with epistemically valuable content". Overall, if the
goal of a recommender system is to meet the information needs of all members of society, it is also important for the
recommender system to be able to effectively recommend news to different societal groups, even if they have different
consumption patterns.
3.3 Subconclusion
Many of the features that would be needed to make diverse recommendations are hard to identify, and require
additional data processing [74, 75]. Think, for example, of identifying the opinions reflected in an article text so a
recommender system can show a user different perspectives on the same topic. There are extensive lines of research
into identifying different viewpoints algorithmically, usually referred to as opinion mining or viewpoint detection.
However, neither currently have off-the-shelf solutions [56]. Alternatively, researchers or journalists could manually
annotate the viewpoints included in an article [48]. However, manual annotation at the scale needed for training data
for recommender systems is a time-consuming and costly endeavour. This problem is exacerbated by the conceptual
unclarity of diversity, as labelling by non-experts is likely to yield inconsistent results. Despite, and arguably especially
given the difficulty of identifying the features relevant to diversity, it is important that these aspects are included in
public datasets, if they are to facilitate research developing diversity-aware recommender systems.
4 Data as a constraint on diversity-aware news recommender systems
In this section we analyse which datasets are used in recent research developing and testing news recommendation
models, the characteristics of those datasets, and how they can(not) facilitate the different aspects of diversity outlined
on page 3.
4.1 Datasets used in current news recommendation research
To assess which public datasets are used for news recommendation research, we queried the Scopus search engine for
papers with the term “news recommend*" in the title, abstract or keywords, that were published between January 2022
and December 2024. This returned 383 results. We filtered out conference proceedings, papers that only mention news
recommendation as a potential field of application, that are behind a paywall, or not written in English. This left us
with 314 papers. Out of these 314 papers, 46 are written from a social science perspective, whereas the vast majority
(268) is technical. The majority of technical papers (171) implemented and tested a news recommendation model.
Figure 1 provides an overview of the datasets used in these papers. It lists all datasets that are cited more than 5 times,
and groups the others into the ‘Other’ and ‘Proprietary’ categories. The ‘Other’ category (35 papers) mostly consists of
PDF Page 6
6 van Drunen and Vrijenhoek
Fig. 1. Counts of the datasets used in papers on developing news recommender systems between January 1st, 2022 and December 31, 2024. Excludes datasets cited 5 times or fewer.
datasets such as MovieLens, which is a standard benchmark dataset for (non-news specific) recommendation tasks, and
task-specific datasets such as FakeNewsNet for fake news detection. In addition, the ‘Proprietary’ category (32 papers)
covers datasets that are not shared with the academic community. Such datasets are always used for online experiments
that test recommender systems on real users, but without publication of the data these studies are not reproducible.
There are four news-specific public datasets that are frequently used to implement and test news recommendation
models: MIND [78](used 98 times), Adressa [27](used 31 times), EB-NeRD [40](published for the 2024 RecSys challenge,
used 9 times), and Globo[18] (used 6 times). All four datasets contain (the IDs of) the articles people clicked during a
certain time period on a news website: the fully algorithmic aggregator MSN News for MIND, Norwegian Adresseavisen
for Adressa, the Brazilian G1 news portal for Globo, and Danish tabloid Ekstra Bladet for EB-NeRD. Table 1 provides an
overview of the data contained in each of the datasets.
MIND is the most prominent dataset, accounting for 57% of all papers published on developing news recommendation
models. The publication of MIND in 2020 was a significant development for news recommender system research. While
at first glance its scale seems comparable to that of Adressa, the interactions in MIND take place in a much shorter time
period. This means that there are more interactions per published news article. The articles in MIND are written in
English, which increases result interpretability for researchers. Most importantly, and contrary to Adressa and Globo,
MIND also contains an ‘impression log’ with articles that users saw but did not click. In addition to this raw data,
Microsoft published a package of news recommender algorithms that could be trained on the dataset through open
source code. Microsoft also launched a competition, challenging developers to create algorithms that would outperform
their own. Performance was determined by the recommender system’s ability to correctly rank a list of candidate items
based on the likelihood a user clicks on them.
PDF Page 7
How public datasets constrain the development of diversity-aware news recommender systems, and what law could do about it. 7
MIND Adressa Globo EB-NeRD #interactions 15 million 23 million 3 million 37 million #users 1 million 3 million 314.000 1 million #articles 160.000 48.000 46.000 125.000 time period covered Oct 12 - Nov 22, 2019 Jan 1 - Mar 31, 2017 Oct 1 - Oct 16, 2017 Apr 27 - June 8, 2023 language English Norwegian Portuguese Danish organization type aggregator local news national news national tabloid article metadata category, subCategory, title, abstract, URL, named entities in title, named entities in abstract, embeddings
title, category, URL, word count, time published, keywords, author, entities
category, time published, publisher, word count, embeddings
title, subtitle, body, category, subcategory, time published, premium or open, IDs of corresponding images, article type, topic, views, reads, sentiment, entities, embeddings click metadata user ID, timestamp, clicked articles, non-clicked articles
timestamp, session start/stop, read time, referrer
user ID, article ID, timestamp, session, referrer
user ID, article ID, session ID, articles viewed, articles clicked, time stamp, read time, scroll percentage, device type user metadata history city, region, country, os, device
region, country, os, device history, subscription, gender, post code, age Table 1. The context and data included in the four primary datasets used for training news recommender systems. Most of the users on news websites are not logged in - unless IP address is considered it is difficult to determine the number of unique users a dataset encompasses. The URLs to the full articles in MIND and Adressa are expired (https://github.com/msnews/msnews.github.io/issues/22).
4.2 The constraints of current datasets from a diversity perspective
The datasets described above are important resources in news recommender system research and development - it would
be very challenging for researchers to develop (reproducible) recommendation models without them. However, as we
argue below, the characteristics of these datasets (specifically the kind of metadata, content and audiences they contain)
pose significant limitations to the development of diversity-aware recommender systems, and make it comparatively
easier to develop recommender systems optimised for engagement. Our analysis includes all datasets but focuses in
more detail on MIND, as our research indicates this is the most frequently used public dataset in news recommender
system research.
4.2.1 Metadata. Most datasets contain only very limited information about the articles that have been recommended.
None of the datasets contain readily available information on the styles and formats of the articles, or the perspectives
expressed in them. This makes it considerably more difficult to use these datasets to develop recommender systems that
incorporate these aspects of diversity. As mentioned in the previous section, it is possible to obtain information on for
example the viewpoint expressed in an article by mining the relevant content characteristics from the text of article.
However, this process (already difficult in itself) is complicated by the fact that, with the exception of EB-NeRD, the
datasets do not contain the full text of the articles. Because of licensing issues, Globo has not published any additional
information about their articles; content is only represented through an ID, category code (e.g,.‘614’) and embeddings
created based on all the texts. MIND and Adressa only contain an URL, not the actual content of news articles. Before
researchers begin the difficult work of mining characteristics from articles’ content, they must therefore verify whether
they are legally allowed to scrape the content, build the tools to do so, and ascertain which URLs still function (which,
in May 2025, is not the case in either of the datasets).
Adressa, EB-NeRD and MIND all do contain metadata on an article’s topic, category, and title; Adressa also has
information on the author of an article. This information could be used to develop recommender systems that recommend
diverse topics or categories of articles to users. However, the generic nature of the metadata that is available on the
PDF Page 8
8 van Drunen and Vrijenhoek
articles’ topics challenges the development of such recommender systems. A recommender system can only identify
patterns in data that is available; as such, the granularity with which article categories are chosen significantly influences
the types of patterns that can be identified. For example, all political news in MIND is put under the umbrella category
‘politics’. As such, a recommender system developed with MIND could not determine a preference for the Democratic
over the Republican Party, let alone a preference in non-American politics, unless other metadata such as the title
contains sufficient data to infer such specific information.
4.2.2 Content. The content in the datasets is another significant limitation on the datasets’ suitability for the de-
velopment of diversity-aware recommender systems. To train a recommender system to recommend diverse topics
or perspectives on matters of public interest (whether to promote social cohesion, create expert citizens, or another
objective of diversity-aware news recommenders), the recommended articles must actually include diverse content and
perspectives. However, only 30% of the MIND’s dataset consists of articles categorised as news, with the rest fitting into
categories such as ‘lifestyle’, ‘sports’, and ‘food’ [73]. Similar distributions can be found in EB-NeRD, where only 28.000
of the 125.000 articles belong to the ‘news’ category. Adressa contains roughly 60% news, but its categories are mostly
focused on the geographic area the article covers, rather than its content (‘nyheter|nordtrondelag’). Furthermore, the
subcategories MIND uses for news indicate it is focused on the US context, with ‘newsus’ representing 47% of all news
items, ‘newspolitics’ 17%, and ‘newsworld’ 8% [73].
This is potentially problematic; while the datasets contain large amounts of articles, they contain relatively few news
articles, which are likely to include the kinds of diverse ideological perspectives and address the kinds of public interest
issues that the diversity-aware recommender systems proposed in normative literature take into account. This issue is
exacerbated by the popularity of MIND and its focus on US news, as a recommender system trained to recommend
diverse political perspectives and topics in the US two-party system would be optimised to identify consumption
patterns that differ from countries with different political and societal groups [59]. This would in turn challenge the
ability of recommender systems to facilitate the access of individuals outside the US to content that reflects the diversity
of political outlooks or societal groups in the society in which they live. This issue can be partially addressed by use of
the other datasets, which reflect the Brazilian, Norwegian and Danish political systems. However, the popularity of
MIND indicates this solution has not yet been widely adopted.
Finally, it should be noted that EB-NeRD has some unique potential for the development of style- and format-aware
recommender systems. Unlike the other datasets it contains additional information like content type (video, gallery, etc.),
accompanying images, and estimated sentiment score. It could therefore be used to research recommender systems that
respond to the format consumption preferences of different users (e.g., deep-dives or simple explainers) and affective
styles (e.g., deliberative discussion or humanisation of political opponents). Conversely, the lack of such data in MIND,
as well as the fact that MIND only contains text articles, make it more difficult to use this dataset to take the style and
format of news into account when developing diversity-aware news recommenders.
4.2.3 Audience. There are distinct differences in the information each of the datasets includes about its users. The
only information available on MIND’s users is when they accessed the system, which items they have interacted with
in the past, which items were presented to them in a specific session, and on which items they have clicked. This is
commendable from a privacy perspective. However, it makes it impossible to assess for which audiences recommender
systems trained on MIND are optimised. Different societal groups (e.g., age, ethnic, or socio-economic background) are
likely to have different consumption patterns [69]. If a particular user’s or group’s consumption pattern falls outside the
patterns for which a recommender system is optimised, the recommender system will be less able to determine what
PDF Page 9
How public datasets constrain the development of diversity-aware news recommender systems, and what law could do about it. 9
news they should be recommended. It is likely that MSN users are not fully representative of the different audiences for
whom recommender systems are developed, if only because of MIND’s focus on English-language US news. However,
as MIND does not contain even aggregate data on the characteristics of its users, it is impossible to know exactly in
what ways and for which groups MIND’s audience is unrepresentative. The other datasets do contain information on
the geographic location a click happened: Globo provides the region, Adressa the city, and EB-NeRD the postcode.
This data makes it possible to assess how a recommender system performs for local communities in each country.
However, it is not possible to assess how a recommender system performs for different socio-economic, disadvantaged,
or minority groups. EB-NeRD is a partial exception to this, as postcode could be combined with other data to infer (for
example) socio-economic status and the dataset sometimes contains information on users’ gender (7%) and age group
(3%).
The limited number of days in each of the datasets poses another limitation. MIND and EB-NeRD only contain
interactions of a period of 6 weeks, versus two weeks for Globo and three months for Adressa. Moreover, as users of
news websites are often not logged in (6.7% in EB-NeRD), it is difficult or impossible to track changing behavior over
time. For example, in MIND only 20% of visits in the dataset are from users that have accessed the system five or more
times, which can be explained by the fact that user IDs reset after 24 hours. As such, little of the data that is necessary
to reliably observe long term changes in people’s reading behavior is available. This limits research on diversity-aware
recommender systems that aim to foster long term changes in the audience, for example to promote tolerance, create
expert citizens, or stimulate self-development.
Furthermore, all datasets encompass a time period where, electorally, nothing significant happened. It is therefore
still impossible to gauge how a model would behave during democratically important moments such as elections. Most
of the time, this is a deliberate design choice. On the challenge website EB explicitly states that "[t]his timeframe was
selected to avoid major events, e.g., holidays or elections, that could trigger atypical behavior at Ekstra Bladet." However
Einarsson, Helles & Lomborg, in a prior direct collaboration project with Ekstra Bladet, noted that during the 2019
Danish election the use of news recommender system increased the amount of soft news readers consumed [23]. If
diverse recommendation algorithms are to be researched and developed, they also (perhaps even especially) need to be
robust towards exceptional behavior patterns.
4.3 Implications for research
Taken together, the data included in the currently most-used datasets are most suitable to develop recommender systems
that prioritise news based on users’ immediate, short term content preferences. Much of the data needed to develop the
kinds of diversity-aware recommender systems proposed in legal and journalism studies literature based on democratic
principles is lacking from public datasets. In particular, the recommendation of diverse perspectives, formats, and styles
is highly challenging with current datasets, as the relevant content characteristics are either not included as metadata
or lacking from the recommended articles themselves. The development of recommender systems that aim to foster
long term changes (for example by promoting tolerance or deeply informing individuals about specific topics they
select) or are robust enough to perform well during different societally significant events (such as elections or wars) is
similarly challenging, due to the short time period covered by the datasets. To a lesser extent, optimising recommender
systems to perform well for different societal groups is difficult with current public datasets, as data on the individuals
to whom news is recommended is often lacking from the datasets.
The lack of data relevant to diverse recommendations also shapes the way datasets can be used to evaluate the
performance of recommender systems. This is particularly important for benchmark datasets that are to serve as a
PDF Page 10
10 van Drunen and Vrijenhoek
common reference point against which the performance of all recommender systems is measured. In the context of
MIND, the success of recommender system algorithms is judged by how well they predict the way 1 million MSN News
users in 2019 interacted with the English-language mix of hard, entertainment, and sports news on that site. This is not
a neutral measure of success. Because no data is available on the characteristics of these MSN users, it is not clear how
they are (un)representative of other groups. However, for example, a recommendation algorithm that is perfectly able
to inform expert and non-expert users in the Netherlands by providing them news at the level of detail they are able to
understand may not perform well on MIND, due to the limited diversity of its content and likely focus on a US audience.
The datasets’ suitability for creating recommender systems that optimise for engagement is exacerbated by the
competitions set out by the companies that have published them. Competitions are a common way for companies to
increase researchers’ use of their dataset, and allow them to set out the specific performance indicator they would like
researchers to optimise for [44]. In the case of MIND, Microsoft set out a competition tasking researchers to create
a recommender system that ranks “articles according to the personal interest of [the] user" [53]. The 2024 RecSys
challenge, which focused on news recommendation and relied on EB-NeRD, did aim to address “both the technical and
normative challenges inherent in the design of effective and responsible recommender systems". However, the main
evaluation metric used to determine the winner of the competition focused on engagement: the odds that a randomly
chosen clicked article is ranked higher than a randomly chosen non-clicked article (referred to in computer science
literature as the ‘area under the curve’ [77]). While participants were encouraged to also include beyond-accuracy
objectives in their challenge submission, participants were not evaluated on these metrics, and challenge submissions
exclusively maximised accuracy. One of the challenge participants ([32]) noted that, through the design of the challenge
the potential to optimise on diversity was limited, and that therefore the challenge was “a missed opportunity to move
beyond ‘leaderboard-chasing culture’".
5 Ensuring access to training data: the role of European law and policy
5.1 The legal case for public support for public datasets
If one were to construct a dataset that is better able to enable the development of diverse news recommender systems
proposed in legal and journalism studies literature based on democratic principles, it would include 1) a focus on news
relevant to public debate, 2) granular metadata on the content characteristics (topic, viewpoint, format, and style) that a
diversity-aware recommender may need to take into account, 3) a large number of frequently returning users over a
longer period of time, and 4) with sufficient metadata on these users and the societal groups they represent. Table 1
provides a more complete overview of the data needed to develop diverse recommender systems, based on the analysis
in sections 3 and 4.
It is unlikely any single dataset will contain all of this data. As noted in section 3, different data is required depending
on the specific kind of diversity a recommender system is expected to promote. More importantly, however, the the
viewpoints and topics present in the public debate, as well as the societal groups whose perspectives are recommended
and for whom recommender systems have to be optimised are different in each society and at different times. While the
publication of another dataset with the characteristics above would already be a significant improvement, it would
only facilitate the development of diversity-aware recommender systems that optimise their recommendations for the
specific society and the specific time in which the dataset was created. To enable the development of recommender
systems that work for multiple different societies, multiple datasets are needed.
PDF Page 11
How public datasets constrain the development of diversity-aware news recommender systems, and what law could do about it. 11
Aspect of diversity Data to be included Topics News on the political topics about which citizens need to be informed in their democratic society; topics affecting other societal groups; topics (e.g., entertainment, fashion, climate) with which individuals want to engage. Perspectives The range of ideological and political perspectives in society; perspectives of societal groups (e.g., ethnic, linguistic, or marginalised) groups. Style Impartial and promoting active discourse; Positive and respectful about societal opponents; critical about opponents while not framing them as enemies. Format Video/audio/text to meet the consumption preferences of different news users; deep dives and simple explainers depending on users’ pre-existing knowledge about a topic. Users Different groups (e.g., socio-economic groups, minorities, disabled persons, and disadvantaged and local communities) that need to be able to be informed in democratic society; data on user characteristics relevant to the above, such as the formats they prefer to consume, political leaning, topics they have engaged with. Table 2. Overview of data to be included in datasets to facilitate the development of diversity-aware recommender systems proposed in legal and journalism studies literature
There is little incentive for any particular media organisation to invest in the creation of a high quality dataset others
could use to research diversity-aware recommender systems, much less the creation of multiple datasets that reflect the
consumption patterns of different societies at different times. Nor is facilitating the development of diversity-aware
recommender systems arguably the societal role of a commercial media or technology organisation. However, states do
have an interest in ensuring that the different media organisations in their society can provide diverse perspectives.
Under European fundamental rights law, this is moreover a legal obligation. Article 10 of the European Convention on
Human Rights has long imposed a positive obligation on states to be the “ultimate guarantor” of diversity, meaning
states must not only refrain from censorship, but also actively ensure that media organisations can provide (and the
public can receive) diverse perspectives representing different societal groups [47, 64, 67].
States have traditionally fulfilled this obligation by ensuring the existence of different media organisations through
subsidies or media concentration laws, or by creating public service media organisations. However, as scholars have
argued extensively over the past decade, individuals’ access to diverse information not only relies on the existence of
different media organisations, but also on the design of technologies such as recommender systems that determine
to what information individuals are exposed [6, 13, 33]. So far, legal researchers and policymakers have primarily
addressed the media’s use of technology by emphasising the media’s responsibility to use technologies in line with
editorial values, as well as the limits freedom of expression imposes on any regulation of the media’s use of technology
[15, 36, 51, 71]. However, both to ensure media organisations can actually live up to these responsibilities and to fulfill
their own obligation to guarantee diversity in the media system, it is also important to consider how states can create
the conditions media organisations need to use technologies like recommender systems responsibly. In particular, as
we have argued in this article, the existence of a high quality dataset is an important condition for organisations to
ultimately be able to recommend diverse news to their audiences.
Law and policy that supports the creation of public datasets also serves another purpose, namely to safeguard the
media’s independence from large technology companies. A large strand of journalism studies research has assessed how
media organisations rely on large technology companies for technologies used to gather, produce, and distribute news,
including news recommender systems [17, 25, 39, 52, 62], as well as the computing power, research, and indeed training
data needed to develop new technologies [60–62]. This reliance challenges media organisations’ ability to independently
PDF Page 12
12 van Drunen and Vrijenhoek
determine how the technologies they use inside newsrooms serve editorial values. Instead, they increasingly adopt the
logics of the large technology companies on which they rely [17, 45, 52, 60, 68]. As our research shows, over half the
research into the development of new news recommender systems relies since 2022 relies on a dataset published by
Microsoft (MIND), and the datasets currently available are primarily suitable for developing recommender systems that
optimise for engagement. As such, public support for the creation of datasets that facilitate the creation of recommender
systems that promote different conceptualisations of diversity would lessen MIND’s influence on the way media
organisations can recommend news to their audiences.
5.2 Access to public datasets through European data access law and policy
EU law has significantly expanded its data access obligations in recent years. However, none of the new obligations
ensure access to data that could be used to develop diversity-aware recommender systems. The Data Governance Act
requires public sector bodies to facilitate the re-use of data, but article 3(2) exempts public service broadcasters and
cultural institutions from this obligation. The Data Act’s data access obligations are limited to the Internet of Things
(Chapter II) or public authorities (Chapter V). Finally, article 40 of the Digital Services Act does grant researchers access
to data necessary to assess the effects of large platforms’ recommender systems on diversity. However, this data access
right is limited to the data that is necessary to understand platforms’ impact and risk mitigation measures. While it
could be used to evaluate the diversity of the datasets used to train platforms’ recommender systems along the criteria
laid out on page 4, it therefore likely does not allow researchers to develop alternative recommender systems.
EU law and policy is also beginning to facilitate voluntary data sharing. Data spaces, which allow organisations to
share data in accordance with common standards, play a particularly important role in the EU’s push to strengthen
digital media and the European technology industry more broadly [9, 10]. The Commission has in 2022 set out a
€8.000.000 grant for the development of a ‘media data space’ that would provide organisations with content, metadata,
and audience data to develop “data services matching European values, in particular ethics, equality and diversity”,
according [11]. A pilot project and the European Broadcasting Union (an organisation representing Europe’s public
service media) both foresee that this media data space could provide access to data necessary to develop recommender
systems [12, 21, 22].
In large part, however, the focus on data spaces in European (media) data policy aims to address a different issue than
the one posed by the lack of data needed to develop diversity-aware recommender systems. Data spaces are useful to
increase access to data currently held by separate organisations. The same is true of other mechanisms in the Data and
Data Governance Act facilitating data sharing, such as data intermediaries or data altruism. However, the differences in
audience behaviour caused by (for example) different recommender systems’ user interfaces, algorithms, and goals,
make it highly challenging to identify meaningful patterns between recommendations and user behaviour in such an
aggregated dataset. Data spaces may therefore be most suitable as an overarching framework through which datasets
for recommender system development can be made available. For example, they can make it easier to test recommender
systems on multiple datasets by setting common standards on data structure, and provide means for other actors to
enrich datasets by adding metadata concerning, for example, viewpoint diversity [12, 21, 49]. They can also expand
access to the data necessary to understand audience preferences and behaviours on a more general level [22]. However,
all this would not address the core issue identified in this article, namely the lack of a dataset (or datasets) with which
different types of recommender systems promoting different kinds of diversity can be developed.
Though EU law does not currently ensure access to such a dataset, it also does not pre-empt either governmental
or media initiatives to create it. In particular, public service media organisations are arguably well positioned to do
PDF Page 13
How public datasets constrain the development of diversity-aware news recommender systems, and what law could do about it. 13
so. Public service media organisations already have access to a diverse library of content that can be recommended,
and several already operate recommender systems that can be used to generate datasets. Moreover, in contrast to
Microsoft and the commercial media organisations that have so published the datasets used in this article, public service
media organisations have a specific societal role and sometimes even legal obligation to promote citizens’ ability to
access diverse information [20, 71]. Public service media organisations have traditionally fulfilled this role by providing
diverse content themselves. Increasingly, they have also experimented with using recommender systems to better serve
their own audiences, and in this process been forced to navigate the difficulties of incorporating diversity in their
recommender systems [37, 63].
However, what remains lacking, and what public service media organisations could begin to provide, is support for
the more foundational research that enables media organisations throughout the media system to automate editorial
decision-making in line with their editorial values. European law can only play a limited role in facilitating such
support; the regulation of public service media remains primarily a national competence. However, organisations such
as the European Broadcasting Union do already operate projects that aim to facilitate cooperation between public
service media developing diverse recommender systems [26, 37]. Such approaches could be expanded to also generate
publicly accessible datasets needed for researchers to create new kinds of diverse recommender systems. Alternatively,
individual public service media organisations could publish datasets on their own initiative. In both cases, the data
outlined in table 2 may serve as a useful starting point for public service media organisations looking to publish the
data needed to enable the creation of a wide variety of diversity-aware news recommender systems [65].
6 Conclusion
In this paper we have analysed how the lack of suitable public datasets constrains the development of diversity-aware
recommender systems. We have analysed what data is needed to develop the kinds of diversity-aware recommender
systems proposed in legal and journalism studies literature, and to what extent this data is available in the currently
most used public datasets. We have argued current datasets are primarily useful to optimise for short-term engagement,
and to a much lesser extent for the recommendation of diverse topics. The development of diverse recommender systems
that take the formats, styles, and viewpoints of articles or the characteristics of different audiences into account is
particularly limited by current datasets. Furthermore, we argued why European law should address this limitation, how
existing data access measures fail to do so, and how public service media playing a more active role in technological
development could provide a way forward.
In the short term, our analysis indicates there is potential to better use the currently available datasets to develop
diversity-aware recommender systems. The majority of research in the past 3 years has relied on MIND; complementing
the use of this dataset with EB-NeRD, Adressa, and Globo would allow research to optimise the performance of
recommender systems not only for audiences that resemble 2019 US MSN news users, but also audiences from (specific
regions in) Brazil, Denmark, and Norway. In addition, due to its inclusion of article text and wider variety of content,
EB-NeRD partially allows for the development of diversity aware recommender systems that take the formats, styles,
and viewpoints of articles into account by inferring the relevant characteristics from the text. However, the lack of
metadata on (for example) the viewpoints represented in the articles still makes this challenging. Moreover, the limited
spectrum of news content, formats, and timeframes in all datasets continue to limit the development of diversity aware
recommender systems.
In the long term legal and technical research should therefore also expand its focus from assessing how the media
should use technologies such as recommender systems responsibly, to analysing the conditions that need to be in
PDF Page 14
14 van Drunen and Vrijenhoek
place for researchers to develop and media organisations to deploy technologies that align with editorial values. At
present, the public datasets used to develop diversity-aware news recommender systems are supplied by commercial
technology companies (Microsoft) or commercial media companies (Adresseavisen, Ekstra Bladet, and G1). Providing
the datasets needed to develop the kinds of diversity-aware recommender systems proposed in normative theory is
not these companies’ societal role, nor do they (given their financial goals) have a strong incentive to do so. Instead,
to enable media organisations to enact the responsible use they call for, as well as to live up to their own obligation
under European fundamental rights law to safeguard a diverse media system, states should take a more active role in
providing the data needed to develop diversity-aware news recommender systems. Collaboration between researchers
and public service media organisations has an important role to play here, as these organisations have the diverse
content and domain knowledge needed to generate the necessary data, as well as the societal role to promote access to
diverse news.
Acknowledgments
We would like to thank Natali Helberger and Laura Hollink for their feedback on earlier drafts of this article. All errors
are ours.
Funding Statement. This research was supported by funding for the AI, Media and Democracy Lab (grant nr.
NWA.1332.20.009) from the Dutch Research Council.
Data availability statement. The four datasets studied in this article are available at https://msnews.github.io/ (MIND),
https://recsys.eb.dk/dataset/ (EB-NeRD), https://www.kaggle.com/datasets/gspmoreira/news-portal-user-interactions-
by-globocom (Globo), and https://reclab.idi.ntnu.no/dataset/ (Adressa). Restrictions apply to their use.
Competing Interests. Max van Drunen co-authored a chapter in a book projected funded by Microsoft while writing
this article. Further information is available at https://www.ivir.nl/projects/news-and-media-law-in-europe/
Ethical Standards. The research meets all ethical guidelines, including adherence to the legal requirements of the
study country.
References
[1] Mariella Bastian, Mykola Makhortykh, and Tom Dobber. News personalization for peace: how algorithmic recommendations can impact conflict
coverage. International Journal of Conflict Management, 30(3):309–328, jun 24 2019. doi: 10.1108/IJCMA-02-2019-0032. ZSCC: 0000001. [2] Christine Bauer, Chandni Bagchi, Olusanmi A. Hundogan, and Karin van Es. Where are the Values? A Systematic Literature Review on News
Recommender Systems. ACM Transactions on Recommender Systems, mar 28 2024. doi: 10.1145/3654805. URL https://dl.acm.org/doi/10.1145/3654805. Just Accepted. [3] Joeran Beel and Haley Dixon. The ‘unreasonable’ effectiveness of graphical user interfaces for recommender systems. In Adjunct Proceedings of
the 29th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ’21, page 22–28, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383677. doi: 10.1145/3450614.3461682. URL https://doi.org/10.1145/3450614.3461682. [4] Abraham Bernstein, Claes De Vreese, Natali Helberger, Wolfgang Schulz, Katharina A Zweig, C Baden, Michael A Beam, MP Hauer, L Heitz,
P Jürgens, Christian Katzenbach, B Kille, Beata Klimkiewicz, Wiebke Loosen, Judith Möller, G Radanovic, G Shani, Nava Tintarev, S Tolmeijer, Wouter van Atteveldt, Sanne Vrijenhoek, and T Züger. Diversity, Fairness, and Data-Driven Personalization in (News) Recommender System. Technical report, Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2020. URL https://drops.dagstuhl.de/opus/volltexte/2020/11986/. [5] Balázs Bodó. Selling News to Audiences – A Qualitative Inquiry into the Emerging Logics of Algorithmic News Personalization in European Quality
News Media. Digital Journalism, 7(8):1054–1075, sep 14 2019. ISSN 2167-0811. doi: 10.1080/21670811.2019.1624185. ZSCC: 0000005. [6] Elda Brogi, Roberta Carlini, Iva Nenadić, Pier Luigi Parcu, and Mario Viola de Azevedo Cunha. EU and media policy: conceptualising media
pluralism in the era of online platforms. The experience of the Media Pluralism Monitor, pages 16–31. Edward Elgar Publishing, sep 21 2021. ISBN
PDF Page 15
How public datasets constrain the development of diversity-aware news recommender systems, and what law could do about it. 15
978-1-78643-933-8. URL https://www.elgaronline.com/display/edcoll/9781786439321/9781786439321.00007.xml. section: Research Handbook on EU Media Law and Policy. [7] Pablo Castells, Neil Hurley, and Saul Vargas. Novelty and diversity in recommender systems. In Recommender systems handbook, pages 603–646.
Springer, 2021. [8] Jin Yao Chin, Yile Chen, and Gao Cong. The datasets dilemma: How much do we really know about recommendation datasets? In Proceedings of the
Fifteenth ACM International Conference on Web Search and Data Mining, pages 141–149, 2022. [9] Commission. Europe’s Media in the Digital Decade: An Action Plan to Support Recovery and Transformation. Technical Report COM(2020) 784
final, Commission, dec 3 2020. URL https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52020DC0784. [10] Commission. Commission Staff Working Document on Common European Data Spaces. Technical Report SWD(2022) 45 final, Brussels, feb 23 2022.
URL https://digital-strategy.ec.europa.eu/en/library/staff-working-document-data-spaces. [11] Commission. Digital Europe Programme (DIGITAL) Call for proposals Cloud Data and TEF (DIGITAL-2022-CLOUD-AI-03). Technical report, sep 15
2022. URL https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/digital/wp-call/2022/call-fiche_digital-2022-cloud-ai-03_en.pdf. [12] Commission. Pilot Project - Digital European Platform of Quality Content Providers (2nd Phase). Technical Report CNECT/2021/OP/0002, 2022.
URL https://europemedialab.eu/wp-content/uploads/2022/10/Media-Data-Space_Interim-Report_Summary_final.pdf. [13] Council of Europe. Recommendation of the Committee of Ministers to member States on media pluralism and transparency of media owner-
ship. Technical Report CM/Rec(2018)1, Strasbourg, 2018. URL https://search.coe.int/cm/Pages/result_details.aspx?ObjectId=0900001680790e13. CM/Rec(2018)1. [14] Council of Europe. Recommendation CM/Rec(2022)4 of the Committee of Ministers to member States on promoting a favourable environment
for quality journalism in the digital age. Technical report, Strasbourg, 2022. URL https://www.coe.int/en/web/freedom-expression/committeeof-ministers-adopted-texts/-/asset_publisher/aDXmrol0vvsU/content/recommendation-cm-rec-2022-4-of-the-committee-of-ministers-tomember-states-on-promoting-a-favourable-environment-for-quality-journalism-in-the-digita. [15] Council of Europe. Guidelines on the Responsible Implementation of Artificial Intelligence Systems in Journalism. Technical report, Steering Com-
mittee on Media and Information Society, Strasbourg, 2023. URL https://rm.coe.int/cdmsi-2023-014-guidelines-on-the-responsible-implementationof-artific/1680adb4c6. [16] Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are We Really Making Much Progress? A Worrying Analysis of Recent
Neural Recommendation Approaches. In Proceedings of the 13th ACM Conference on Recommender Systems, pages 101–109, sep 10 2019. doi: 10.1145/3298689.3347058. URL http://arxiv.org/abs/1907.06902. arXiv:1907.06902 [cs]. [17] Mathias-Felipe de Lima-Santos, Allen Munoriyarwa, Adeola Abdulateef Elega, and Charis Papaevangelou. Google News Initiative’s Influence
on Technological Media Innovation in Africa and the Middle East. Media and Communication, 11(2), apr 28 2023. ISSN 2183-2439. URL https://www.cogitatiopress.com/mediaandcommunication/article/view/6400. [18] Gabriel de Souza Pereira Moreira, Felipe Ferreira, and Adilson Marques da Cunha. News Session-Based Recommendations using Deep Neural
Networks. In Proceedings of the 3rd Workshop on Deep Learning for Recommender Systems, DLRS 2018, pages 15–23, New York, NY, USA, oct 6 2018. Association for Computing Machinery. ISBN 978-1-4503-6617-5. doi: 10.1145/3270323.3270328. URL https://dl.acm.org/doi/10.1145/3270323.3270328. [Online; accessed 2025-01-22]. [19] Tomás Dodds, Valeria Reséndez, Gerret von Nordheim, Theo Araujo, and Judith Moeller. Collaborative Coding Cultures: How Journalists Use
GitHub as a Trading Zone. Digital Journalism, 0(0):1–22, 2024. ISSN 2167-0811. doi: 10.1080/21670811.2024.2342468. publisher: Routledge _eprint: https://doi.org/10.1080/21670811.2024.2342468. [20] Karen Donders. Public Service Media in Europe: Law, Theory and Practice. Routledge, London, jun 10 2021. ISBN 978-1-351-10556-9. DOI:
10.4324/9781351105569. [21] EBU. Dataspace for Cultural and Creative Industries. oct 20 2022. URL https://tech.ebu.ch/publications/dataspace-for-cultural-and-creative-industries. [22] EBU. Personalisation and recommendation ecosystem developed by broadcasters for broadcasters. Technical report, EBU, Geneva, 2023. URL
https://docs.peach.ebu.io/. [23] Árni Már Einarsson, R. Helles, and S. Lomborg. Algorithmic agenda-setting: the subtle effects of news recommender systems on political agendas in
the Danish 2022 general election. Information, Communication & Society, 0(0):1–21, 2024. ISSN 1369-118X. doi: 10.1080/1369118X.2024.2334411. publisher: Routledge _eprint: https://doi.org/10.1080/1369118X.2024.2334411. [24] Sarah Eskens, Natali Helberger, and Judith Möller. Challenged by news personalisation: five perspectives on the right to receive information.
Journal of Media Law, 9(2):259–284, jul 3 2017. ISSN 1757-7632. doi: 10.1080/17577632.2017.1387353. ZSCC: 0000017. [25] Alexander Fanta and Ingo Dachwitz. Google, the media patron. How the digital giant ensnares journalism. Technical report, Otto Brenner
Foundation, nov 24 2020. URL https://osf.io/3qbp9. DOI: 10.31235/osf.io/3qbp9. [26] Graziadio, Marco and Weström, Pär . The european data market study 2024-2026: Story on the common european media data space. Technical
report, European Commission, Brussels, 2024. URL https://ec.europa.eu/newsroom/repository/document/2024-31/EDM_20242026_StoryMedia_ Data_Space_final_Qfq92zEU7HzdPZzSQFFBnxX14vI_107670.pdf. [27] Jon Atle Gulla, Lemei Zhang, Peng Liu, Özlem Özgöbek, and Xiaomeng Su. The Adressa dataset for news recommendation. In Proceedings of the
International Conference on Web Intelligence, pages 1042–1048, Leipzig Germany, aug 23 2017. ACM. ISBN 978-1-4503-4951-2. doi: 10.1145/3106426. 3109436. URL https://dl.acm.org/doi/10.1145/3106426.3109436. [Online; accessed 2024-04-29].
PDF Page 16
16 van Drunen and Vrijenhoek
[28] Rishav Hada, Amir Ebrahimi Fard, Sarah Shugars, Federico Bianchi, Patricia Rossini, Dirk Hovy, Rebekah Tromble, and Nava Tintarev. Beyond
digital "echo chambers": The role of viewpoint diversity in political discussion. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pages 33–41, 2023. [29] Jaron Harambam, Natali Helberger, and Joris van Hoboken. Democratizing algorithmic news recommenders: how to materialize voice in a
technologically saturated media ecosystem. Philosophical Transactions of the Royal Society A, 376(2133):1–21, 2018. doi: 10.1098/rsta.2018.0088. [30] F. Maxwell Harper and Joseph A. Konstan. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems, 5(4):
19:1–19:19, dec 22 2015. ISSN 2160-6455. doi: 10.1145/2827872. [31] Lucien Heitz, Juliane A. Lischka, Alena Birrer, Bibek Paudel, Suzanne Tolmeijer, Laura Laugwitz, and Abraham Bernstein. Benefits of Diverse News
Recommendations for Democracy: A User Study. Digital Journalism, 0(0):1–21, feb 8 2022. ISSN 2167-0811. doi: 10.1080/21670811.2021.2021804. publisher: Routledge _eprint: https://doi.org/10.1080/21670811.2021.2021804. [32] Lucien Heitz, Sanne Vrijenhoek, and Oana Inel. Recommendations for the recommenders: Reflections on prioritizing diversity in the recsys
challenge. In Proceedings of the Recommender Systems Challenge 2024, pages 22–26. 2024. [33] Natali Helberger. Exposure Diversity as a Policy Goal. Journal of Media Law, 4(1):65–92, 2012. ISSN 17577632. doi: 10.5235/175776312802483880. [34] Natali Helberger. Policy Implications From Algorithmic Profiling and the Changing Relationship Between Newsreaders and the Media. Javnost -
The Public, 23(2):188–203, apr 2 2016. ISSN 1318-3222. doi: 10.1080/13183222.2016.1162989. ZSCC: 0000009. [35] Natali Helberger. On the Democratic Role of News Recommenders. Digital Journalism, 7(8):993–1012, sep 14 2019. ISSN 2167-0811. doi:
10.1080/21670811.2019.1623700. ZSCC: 0000011. [36] Natali Helberger, Max Zeno van Drunen, Sarah Eskens, Mariella Bastian, and Judith Möller. A freedom of expression perspective on AI in the media.
European Journal of Law and Technology, 11(3), dec 30 2020. ISSN 2042-115X. URL https://ejlt.org/index.php/ejlt/article/view/752. [Online; accessed 2021-02-23]. [37] Jockum Hildén. The Public Service Approach to Recommender Systems: Filtering to Cultivate. Television & New Media, 23(7):777–796, nov 1 2022.
ISSN 1527-4764. doi: 10.1177/15274764211020106. publisher: SAGE Publications. [38] Aaron Hyzen, Hilde van den Bulck, Manuel Puppis, Michelle Kulig, and Steve Paulussen. Epistemic welfare and algorithmic recommender systems:
overcoming the epistemic crisis in the digitalized public sphere. Communication Theory, 0(00):1–12, 2025. ISSN 1468-2885. doi: 10.1093/ct/qtaf018. [39] Lisa Merete Kristensen and Jannie Møller Hartley. The Infrastructure of News: Negotiating Infrastructural Capture and Autonomy in Data-Driven
News Distribution. Media and Communication, 11(2):307–318, jun 28 2023. ISSN 2183-2439. doi: 10.17645/mac.v11i2.6388. [40] Johannes Kruse, Kasper Lindskow, Saikishore Kalloori, Marco Polignano, Claudio Pomo, Abhishek Srivastava, Anshuk Uppal, Michael Riis
Andersen, and Jes Frellsen. Eb-NeRD a large-scale dataset for news recommendation. In Proceedings of the Recommender Systems Challenge 2024, RecSysChallenge ’24, pages 1–11, New York, NY, USA, oct 14 2024. Association for Computing Machinery. ISBN 9798400711275. doi: 10.1145/3687151.3687152. URL https://dl.acm.org/doi/10.1145/3687151.3687152. [Online; accessed 2025-01-22]. [41] Bibo Lin and Seth C. Lewis. The One Thing Journalistic AI Just Might Do for Democracy. Digital Journalism, 0(0):1–23, jun 14 2022. ISSN 2167-0811.
doi: 10.1080/21670811.2022.2084131. publisher: Routledge _eprint: https://doi.org/10.1080/21670811.2022.2084131. [42] Felicia Loecherbach, Judith Möller, Damian Trilling, and Wouter van Atteveldt. The Unified Framework of Media Diversity: A Systematic Literature
Review. Digital Journalism, 8(5):1–38, 2020. ISSN 2167-0811, 2167-082X. doi: 10.1080/21670811.2020.1764374. [43] Feng Lu, Anca Dumitrache, and David Graus. Beyond Optimizing for Clicks: Incorporating Editorial Values in News Recommendation. In Proceedings
of the 28th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ’20, pages 145–153, New York, NY, USA, jul 13 2020. Association for Computing Machinery. ISBN 978-1-4503-6861-2. doi: 10.1145/3340631.3394864. URL https://dl.acm.org/doi/10.1145/3340631.3394864. [Online; accessed 2025-01-22]. [44] Dieuwertje Luitse. Platform power in AI: The evolution of cloud infrastructures in the political economy of artificial intelligence. Internet Policy
Review, 13(2), jun 26 2024. ISSN 2197-6775. URL https://policyreview.info/articles/analysis/platform-power-ai-evolution-cloud-infrastructures. [Online; accessed 2024-07-01]. [45] Dieuwertje Luitse, Tobias Blanke, and Thomas Poell. Ai competitions as infrastructures of power in medical imaging. Information, Communication
& Society, mar 29 2024. ISSN 1369-118X. URL https://www.tandfonline.com/doi/abs/10.1080/1369118X.2024.2334393. publisher: Routledge. [46] Mykola Makhortykh and Mariella Bastian. Personalizing the war: Perspectives for the adoption of news recommendation algorithms in the media
coverage of the conflict in Eastern Ukraine. Media, War & Conflict, pages 1–21, feb 19 2020. ISSN 1750-6352. doi: 10.1177/1750635220906254. ZSCC: 0000000 publisher: SAGE Publications. [47] ECtHR Manole and Others v. Moldova. Manole and Others v. Moldova. https://hudoc.echr.coe.int/eng?i=001-94075, sep 17 2009. URL https:
//hudoc.echr.coe.int/eng?i=001-94075. 13936/02. [48] Laura Mascarell, Tatyana Ruzsics, Christian Schneebeli, Philippe Schlattner, Luca Campanella, Severin Klingler, and Cristina Kadar. Stance Detection
in German News Articles. In Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER), pages 66–77, Dominican Republic, 11 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.fever-1.8. URL https://aclanthology.org/2021.fever-1.8. [49] Ariadne Matas, Antoine Isaac, Esther Huyer, Fred Saunderson, Julia Fallon, Kerstin Arnold, Marie-Véronique Leroi, Paul Keller, and Valentine
Charles. Data Governance for the common European data space for cultural heritage: Strategy and Plan. Technical report, Europeana, 2023. URL https://pro.europeana.eu/files/Europeana_Professional/Publications/Data_Governance_Strategy_and_Plan.pdf. [50] Microsoft. Microsoft news recommendation. https://learn.microsoft.com/en-us/azure/open-datasets/dataset-microsoft-news?tabs=azureml-
opendatasets. Accessed: 2025-10-01.
PDF Page 17
How public datasets constrain the development of diversity-aware news recommender systems, and what law could do about it. 17
[51] Ofcom. Ofcom’s strategic approach to AI 2024/25. Technical report, London, 2024. URL https://www.ofcom.org.uk/siteassets/resources/documents/
consultations/category-2-6-weeks/273274---ofcoms-proposed-plan-of-work-for-2024-25/associated-documents/ofcoms-strategic-approach-toai.pdf?v=321367. [52] Charis Papaevangelou. Funding Intermediaries: Google and Facebook’s Strategy to Capture Journalism. Digital Journalism, 0(0):1–22, jan 13 2023.
ISSN 2167-0811. doi: 10.1080/21670811.2022.2155206. publisher: Routledge _eprint: https://doi.org/10.1080/21670811.2022.2155206. [53] Ying Qiao, Jiun-Hung Chen, Winnie Wu, Fangzhao Wu, Chuhan Wu, Tao Qi, Yingwei Yi, Ling Luo, and Xing Xie. Mind News Recommendation
Competition. https://competitions.codalab.org/competitions/24122#learn_the_details-task, jul 20 2020. URL https://competitions.codalab.org/ competitions/24122#learn_the_details-task. [54] Shaina Raza and Chen Ding. News recommender system: a review of recent progress, challenges, and opportunities. Artificial Intelligence Review,
pages 1–52, jul 21 2021. ISSN 0269-2821. doi: 10.1007/s10462-021-10043-x. PMID: 34305252 PMCID: PMC8294232. [55] Shaina Raza and Chen Ding. News recommender system: a review of recent progress, challenges, and opportunities. Artificial Intelligence Review,
pages 1–52, 2022. [56] Myrthe Reuver, Nicolas Mattis, Marijn Sax, Suzan Verberne, Nava Tintarev, Natali Helberger, Judith Moeller, Sanne Vrijenhoek, Antske Fokkens, and
Wouter van Atteveldt. Are we human, or are we users? The role of natural language processing in human-centric news recommenders that nudge users to diverse content. In Proceedings of the 1st Workshop on NLP for Positive Impact, pages 47–59, Online, 8 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.nlp4posimpact-1.6. URL https://aclanthology.org/2021.nlp4posimpact-1.6. [57] Francesco Ricci, Lior Rokach, and Bracha Shapira, editors. Recommender Systems Handbook. Springer US, Boston, MA, 2015. ISBN 978-1-4899-7636-9.
URL https://link.springer.com/10.1007/978-1-4899-7637-6. DOI: 10.1007/978-1-4899-7637-6. [58] Marijn Sax. Algorithmic News Diversity and Democratic Theory: Adding Agonism to the Mix. Digital Journalism, 10(10):1650–1670, nov 26 2022.
ISSN 2167-0811. doi: 10.1080/21670811.2022.2114919. publisher: Routledge _eprint: https://doi.org/10.1080/21670811.2022.2114919. [59] Karthik Shivaram, Ping Liu, Matthew Shapiro, Mustafa Bilgic, and Aron Culotta. Reducing Cross-Topic Political Homogenization in Content-Based
News Recommendation. In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys ’22, pages 220–228, New York, NY, USA, sep 13 2022. Association for Computing Machinery. ISBN 978-1-4503-9278-5. doi: 10.1145/3523227.3546782. URL https://dl.acm.org/doi/10.1145/3523227. 3546782. [60] Felix M. Simon. Uneasy Bedfellows: Ai in the News, Platform Companies and the Issue of Journalistic Autonomy. Digital Journalism, 0(0):1–23, may
18 2022. ISSN 2167-0811. doi: 10.1080/21670811.2022.2063150. publisher: Routledge _eprint: https://doi.org/10.1080/21670811.2022.2063150. [61] Felix M. Simon. Escape Me If You Can: How AI Reshapes News Organisations’ Dependency on Platform Companies. Digital Journalism, pages 1–22,
nov 29 2023. ISSN 2167-0811, 2167-082X. doi: 10.1080/21670811.2023.2287464. [62] Felix M. Simon. Artificial Intelligence in the News: How AI Retools, Rationalizes, and Reshapes Journalism and the Public Arena. Technical report,
2024. URL https://academiccommons.columbia.edu/doi/10.7916/ncm5-3v06. publisher: [object Object]. [63] Jannick Kirk Sørensen. Public Service Media, Diversity and Algorithmic Recommendation. In CEUR Workshop Proceedings, page 7. Aalborg
University, 2019. [64] Robert Spano. The concept of media pluralism under the European Convention on Human Rights – substantive principles and procedural
safeguards. Journal of Media Law, 15(2):168–178, jul 3 2023. ISSN 1757-7632. doi: 10.1080/17577632.2024.2305419. publisher: Routledge _eprint: https://doi.org/10.1080/17577632.2024.2305419. [65] Jonathan Stray. Editorial Values for News Recommenders: Translating Principles to Engineering, pages 151–165. Routledge, 2023. [66] Emily Sullivan, Dimitrios Bountouridis, Jaron Harambam, Shabnam Najafian, Felicia Loecherbach, Mykola Makhortykh, Domokos Kelen, Daricia
Wilkinson, David Graus, and Nava Tintarev. Reading News with a Purpose: Explaining User Profiles for Self-Actualization. page 5, 2019. [67] ECtHR NIT S.R.L. v. the Republic of Moldova. Nit S.R.L. v. the Republic of Moldova. https://hudoc.echr.coe.int/eng?i=001-216872, apr 5 2022. URL
https://hudoc.echr.coe.int/eng?i=001-216872. [68] Max Zeno van Drunen. Editorial independence in an automated media system. Internet Policy Review, 10(3), sep 13 2021. ISSN 2197-6775. URL
https://policyreview.info/articles/analysis/editorial-independence-automated-media-system. [Online; accessed 2021-10-07]. [69] Hanne Vandenbroucke, Ulysse Maes, Lien Michiels, and Annelien Smets. Not one news recommender to fit them all: How different recommender
strategies serve various user segments. In 19th ACM Conference on Recommender Systems Proceedings. ACM, September 2025. [70] Saúl Vargas and Pablo Castells. Rank and relevance in novelty and diversity metrics for recommender systems. In Proceedings of the fifth ACM
conference on Recommender systems, RecSys ’11, pages 109–116, New York, NY, USA, oct 23 2011. Association for Computing Machinery. ISBN 978-1-4503-0683-6. doi: 10.1145/2043932.2043955. URL https://dl.acm.org/doi/10.1145/2043932.2043955. [71] Judith Vermeulen. Access Diversity Through Online News Media and Public Service Algorithms: An Analysis of News Recommendation in Light of
Article 10 ECHR, pages 269–287. Palgrave Global Media Policy and Business. Springer International Publishing, Cham, 2022. ISBN 978-3-030-87086-7. URL https://doi.org/10.1007/978-3-030-87086-7_14. DOI: 10.1007/978-3-030-87086-7_14. [72] Judith Vermeulen. To Nudge or Not to Nudge: News Recommendation as a Tool to Achieve Online Media Pluralism. Digital Journalism, 10(10):1671–
1690, nov 26 2022. ISSN 2167-0811. doi: 10.1080/21670811.2022.2026796. publisher: Routledge _eprint: https://doi.org/10.1080/21670811.2022.2026796. [73] Sanne Vrijenhoek. Do you mind? reflections on the mind dataset for research on diversity in news recommendations. In International Workshop on
Algorithmic Bias in Search and Recommendation, pages 147–154. Springer, 2023. [74] Sanne Vrijenhoek, Mesut Kaya, Nadia Metoui, Judith Möller, Daan Odijk, and Natali Helberger. Recommenders with a mission: assessing
diversity in news recommendations. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval. ACM, 2021. URL
PDF Page 18
18 van Drunen and Vrijenhoek
http://arxiv.org/abs/2012.10185. arXiv: 2012.10185. [75] Sanne Vrijenhoek, Gabriel Bénédict, Mateo Gutierrez Granada, Daan Odijk, and Maarten De Rijke. Radio–rank-aware divergence metrics to measure
normative diversity in news recommendations. In Proceedings of the 16th ACM conference on recommender systems, pages 208–219, 2022. [76] Sanne Vrijenhoek, Savvina Daniil, Jorden Sandel, and Laura Hollink. Diversity of what? on the different conceptualizations of diversity in
recommender systems. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 573–584, 2024. [77] Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. Neural News Recommendation with Multi-Head Self-Attention.
In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6389–6394, Hong Kong, China, 11 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1671. URL https://aclanthology.org/D19-1671. [Online; accessed 2024-04-29]. [78] Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou.
Mind: A Large-scale Dataset for News Recommendation. jul 1 2020. URL https://www.microsoft.com/en-us/research/publication/mind-a-largescale-dataset-for-news-recommendation/. [79] Rodrigo Zamith. Open-Source Repositories as Trust-Building Journalism Infrastructure: Examining the Use of GitHub by News Outlets to Promote
Transparency, Innovation, and Collaboration. Digital Journalism, 0(0):1–22, 2023. ISSN 2167-0811. doi: 10.1080/21670811.2023.2202873. publisher: Routledge _eprint: https://doi.org/10.1080/21670811.2023.2202873. [80] Eva Zangerle and Christine Bauer. Evaluating recommender systems: Survey and framework. ACM Comput. Surv., 55(8), December 2022. ISSN
0360-0300. doi: 10.1145/3556536. URL https://doi.org/10.1145/3556536.