Wikidata:WikiProject Source Reliability

WikiProject source reliability, aka the Credibility Ratings + Assessments Project, is an effort to identify and aggregate online sources of assessments of the reliability and credibility of sources. These assessments include estimates of bias, verifiability of claims, level of editorial oversight or peer review, and expertise in specific topics. Assessments may be of individual documents, or aggregate assessments of authors or domains.

Types of assessments include:

• Compilations of self-assessments. Examples include the TRANSPOSE project for collating self-reported peer-review practices of journals).
• Evaluations by groups created specifically to evaluate source reliability. Examples include Media Bias Fact Check, other fact checking sites, and sites like Ad Fontes Media that produce visuals referenced by others.
• Evaluations by communities of practice, as a by-product of their work reviewing or sourcing information to others. Examples include Perennial Sources lists on various language Wikipedias, topical Reliable Sources lists from individual WikiProjects, and newsrooms that publish their internal measures of source reliability.
• Compilations of secondary assessments, including the above. Examples include Iffy.news.

Goals

Short-term

• Add source credibility ratings on Wikidata (including topical focus on news outlets and vaccines)
  • the catalog of sources,
  • for each catalog entry, a timestamped snapshots of its reliability assessments,
  • adding assessments to a WD entity for the source [create it if it doesn't exist yet]
• Curate master Wikipedian-source reliability lists (as versioned tables w/ explicit schema)
  • Federating WD (specialist sets of tabular data)
  • citeable (as archival soure w/ identifier)
  • explicit schemas (+ maps between them)
  • versioned (schemas + datasets)
  • provenanced (chains of derivatives for composite data)

Mid-term

• Model source credibility as a schema using new or existing properties.
• Catalog a range of sources for such ratings
• Set up versioned, archival backups of the snapshots

Long-term

• Integration into future shared citations / wikicite efforts

Tasks

• Propose properties (or identify existing properties) for: assessment of factual-reporting, assessment of credibility, assessment of political bias

  assessed source reliability - could this be a single property w/ qualifiers for "type of reliability" and also for "according to whom"?

• Propose properties for Permanent IDs for these sources

  Compare: MBFC ID, AllSides ID, PolitiFact ID. Get sources to create identifiers for entities if they don't have one.

• Add these properties to entities in the datasets below (CU, Iffy, &c)
• Create entities for news associations that recognize other orgs (State + national press associations)

  Relate publications to the press association to which they belong (instance of: member?).

• Define a shared schema : data model
• Figure out how to add "hijacked journal" as "sourcing circumstances" option, or otherwise make it an accessible qualifier. (Example)
• ...

Interested participants

1. Ocaasi (talk) 18:43, 7 March 2023 (UTC)
2. Sj (talk) 18:49, 7 March 2023 (UTC)
3. Harej (talk) 18:49, 7 March 2023 (UTC)
4. SuperHamster (talk) 18:51, 7 March 2023 (UTC)
5. Hearvox (talk) 20:48, 10 April 2023 (UTC)
6. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits
7. So9q (talk) 14:45, 29 April 2023 (UTC)
8. Daniel Mietchen (talk)
9. AmandaSLawrence (talk) 12:06, 6 September 2023 (UTC)
10. Nikola Tulechki (talk) 16:45, 13 March 2024 (UTC)

Data model

Derived from "perennial sources" data model:

• url → official website (P856)
• title → title (P1476)
• type → instance of (P31): relevant class item
  • fake news website (Q27881073)) : e.g. Disclose.tv (Q115099512) – an instance of (P31) this
  • evaluation (Q1379672) : evaluation (of merit, scope, worth, significance)
  • peer review (Q215028)
  • assessment (Q851176) : evaluation (of a fact, an object or a person)
  • media criticism (Q1753773)
  • source critic (Q56417579) : scientist? evaluating the quality of sources
  • Intelligence source and information reliability (Q6043974) : field-specific
  • systematic review (Q1504425) : including of reliability (ref'ed by WikiFactMine)
• Commons quality assessment (P6731)
• sourcing circumstances (P1480) : qualification of the truth or accuracy of a source: circa (Q5727902), near (Q21818619), presumably (Q18122778), etc. clustered with aliases: accuracy, reliability, confidence, precision, certainty, validity, qualitative valuation
• assessment outcome (P9259)

Property proposals

• assessment → proposed properties for "assessed source reliability" and "assessed source bias"
• Reference:
  • discussionForum → published in (P1433): e.g. Wikipedia:Reliable sources/Noticeboard (Q11425125)
  • discussionLastDate → point in time (P585)
  • discussionLink → reference URL (P854)
• verifiability of property
• Peer review process for journals (to revisit, now that there's a PID)

see also: Proposed schema for reusable source data

Indexes

• Media Bias Fact Check [MBFC] (mainly news)
  • Measures: Accuracy, right-left bias (US scale)
• Iffy.news (incorporates MBFC)
  • Measure: Iffyness
  • Thoughts on incorporating into WD: notes
• All Sides
• Ad Fontes rankings by source (mainly news; slow site)
  • Measures: Reliability + news value, Right-Left bias (US scale),
• NewsQ[1] (mainly news)
  • Measures: identification, journalistic content, journalistic behavior, diversity, external assessment, external affiliation, traffic + engagement
• Beall's List (journals)
  • Measures: tags: predatory journals, hijacked journals

Source lists

General sources lists

• Deprecated sources
• Good free online resources

Perennial sources

wide range of source sites, including journals + popular websites

• EN
• FA
• ZH
• RU
• EL
• FR
• LT
• PT
• NE
• TR
• VI
• SV

Topical sources lists

1. Africa
2. Airfract engines
3. Albums
4. Anime and manga
5. Arena Football League
6. Basketball
7. Beauty pageants
8. Birds
9. Board games
10. Christian music
11. College football
12. Comics
13. Conservatism
14. COVID-19
15. Dogs
16. D&D
17. Fantasy novels
18. Film
19. Historic Places
20. Korea
21. Latin music
22. LDS and Mormonism
23. Math
24. Motorsports
25. Nigeria
26. Oregon
27. Pakistan (Insight_3)
28. Record charts
29. Science fiction
30. Timeline tracer
31. Venezuela
32. Video games (I)
33. Video games (II)
34. Webcomics

Current compilations of reliability indicators

• w:User talk:Headbomb/unreliable.js (Headbomb) –
• CiteUnseen (SuperHamster) – categorized domains (json)
• w:Wikipedia:New page patrol source guide (Novem Linguae, Rosguill, &c.)

  w:User:Novem Linguae/Scripts/CiteHighlighter/AllSourcesExceptNPPSG – (NL)

External sources:

• NewsQ - had an online API; soon a public dataset
• Media Bias/Fact Check - 2.4k entries

See also

Related projects

• Wikidata:WikiProject Policy Reports (AU/NZ policy reports, reliability measured by the APO)
• WikiFactMine

Related policy

• Wikidata:Verifiability

Research into credibility and reliability assessments

• Why people trust WP: credibility assessment strategies used by readers (Forte 2022)
• News Credibility in the eyes of different assessors (2019)
• Reliability index for Twitter handles (2018)
• Web credibility assessment (2015)
• Credibilit assessment of FB pagews
• An information reliability index for medical websites
• Developing a unifying framework of credibility assessment (2008)

In the broader context of citations

• The invisible citation commons
• Shared Citations proposal