Overview of the DagPap22 Shared Task on Detecting Automatically Generated Scientific Papers

Yury Kashnitsky , Drahomira Herrmannova , Anita de Waard , Georgios Tsatsaronis , Catriona Fennell , Cyril Labbé (1, 2)

1 LIG - Laboratoire d'Informatique de Grenoble
2 SIGMA - Systèmes d’Information - inGénierie et Modélisation Adaptables

This paper provides an overview of the 2022 COLING Scholarly Document Processing workshop shared task on the detection of automatically generated scientific papers. We frame the detection problem as a binary classification task: given an excerpt of text, label it as either human-written or machine-generated. We shared a dataset containing excerpts from human-written papers as well as artificially generated content and suspicious documents collected by Elsevier publishing and editorial teams. As a test set, the participants were provided with a 5x larger corpus of openly accessible human-written as well as generated papers from the same scientific domains of documents. The shared task saw 180 submissions across 14 participating teams and resulted in two published technical reports. We discuss our findings from the shared task in this overview paper.

Domains: Computer Science [cs]; Computation and Language [cs.CL]

Yury Kashnitsky, Drahomira Herrmannova, Anita de Waard, Georgios Tsatsaronis, Catriona Fennell, et al.. Overview of the DagPap22 Shared Task on Detecting Automatically Generated Scientific Papers. Third Workshop on Scholarly Document Processing, Oct 2022, Gyeongju, South Korea. ⟨hal-03828597⟩