This article by Jürgen Schmidhuber, addresses a series of alleged misattributions in the field of AI, particularly in relation to deep learning. Schmidhuber argues that three recipients of the 2018 ACM Turing Award for deep learning (unnamed in the summary but identifiable as Yoshua Bengio, Geoffrey Hinton, and Yann LeCun) have frequently republished methods and concepts originally developed by his team without proper attribution.
Some examples:
Generative Adversarial Networks (GANs): Schmidhuber asserts that his work on gradient-based neural networks in 1990 is effectively the same as GANs, which were later popularized by Bengio and his team in 2014, without proper citation of his earlier work.
Vanishing Gradient Problem: He claims that Sepp Hochreiter, his student, first analyzed this problem in 1991, while Bengio published a similar analysis in 1994 without citing Hochreiter.
Metalearning: Schmidhuber alleges that he initiated research in this area in his 1987 thesis, whereas Bengio claimed to have begun earlier, citing his 1991 paper.
Soft Attention and Transformers: Schmidhuber discusses his early work in 1991 on what he describes as the first Transformer variant, which he claims influenced the later development of attention-based Transformers that are central to models like ChatGPT.
Gated Recurrent Units (GRUs): He contends that Bengio’s 2014 work on GRUs failed to credit his team’s earlier work on similar concepts in LSTM architectures.
Auto-regressive Neural Networks for Density Estimation: Schmidhuber suggests that his team developed similar models in 1995, while Bengio claimed to have introduced them in 1999.
Time Scale Hierarchy in Neural Nets: Schmidhuber’s work dating back to 1991-93 is claimed to be foundational in this area, with Bengio later presenting similar ideas in 1995.
Schmidhuber also raises concerns about the broader impact of such misattributions on scientific integrity and proposes the creation of a “pillory list” to hold accountable papers that replicate earlier work without proper citation.
Additionally, this post by Tomas Mikolov mentions issues involving the Seq2Seq model and Google researchers including Ilya Sutskever. Discussion here.
While I acknowledge the difficulties in discerning objective truth on these matters, I find it disheartening that the scientific field, which strives for the advancement of knowledge, can sometimes become ensnared in egotism and personal rivalries.