Huawei, DeepSeek linked to pirated books by Han Kang in AI training
Chinese AI companies Huawei and DeepSeek have been linked to training their AI models with datasets reportedly containing pirated books by Korean authors, including Nobel laureate Han Kang. The connections emerge from analyses of company technical reports and independent research findings.
Huawei's Pangu Sigma AI model reportedly utilized "The Pile," a large dataset documented to include "Books3," a collection of pirated e-books. Analysis of "Books3" on GitHub indicates it contains English translations of at least 19 books by 10 Korean authors, such as Han Kang's "The Vegetarian." While this establishes a reported pathway from unauthorized content to a dataset used by Huawei, the company's technical report does not specify whether its model trained on all these particular novels.
DeepSeek's DeepSeek-VL model reportedly sourced 860,000 English-language and 180,000 Chinese-language e-books from "Anna's Archive," a website known for providing access to pirated books. Anna's Archive lists works by prominent Korean authors. DeepSeek stated it selected materials for training rather than downloading every book available, leaving the direct inclusion of Korean works in its training data unconfirmed.
However, researchers from institutions including Columbia University and Carnegie Mellon University presented indirect evidence concerning DeepSeek. Their paper, updated in September, indicates DeepSeek 3.1 reproduced a 19-word sequence matching Han Kang's "The Vegetarian" after undergoing fine-tuning. Although these findings have not yet undergone peer review, the researchers interpreted this outcome as indirect evidence of the AI model having been pretrained on the novel, illustrating how model output can suggest dataset content.
Share this article
Related reading
6 stories
Anthropic’s AI model accessed US voter data, says Axios
We reported on Anthropic's AI model accessing US voter data, another instance of AI training data controversy.
Open-Source AI Models Are Catching up to Closed Ones on Accounting Work, DualEntry Benchmark Finds
The debate over open-source versus closed AI models often includes discussions around training data ethics.
For Sale: A Rare 1971 Isuzu Bellett 1800 GT – The First Japanese “GT”
What happens when AI starts self-replicating?
Troubleshoot etcd Clusters Easily with Ivory
