Amid growing controversy over American AI firm Anthropic’s unauthorized downloading of digital books by South Korean authors, it was confirmed that Chinese AI companies also scraped works by Han Kang and Hwang Sok-yong.
Huawei and the ‘The Pile’ Dataset Connection
According to arXiv, a site publishing the latest AI research, Chinese IT and AI firm Huawei released a technical report for its AI model ‘Pangu Sigma’ in March 2023. The document details how Huawei collected data for model training, explicitly stating that English texts were sourced from datasets like ‘The Pile,’ amounting to 800 GB.
‘The Pile’ is a dataset combining books, research papers, and web documents. Documents describing ‘The Pile’ show that it includes Books3, a collection of pirated e-books that Anthropic also downloaded. This issue led Anthropic to settle with copyright holders for $3,000 per book.
A review of the Books3 work list reveals at least 19 English-translated e-books by Korean authors, including Han Kang’s ‘The Vegetarian’ and ‘Human Acts,’ Hwang Sok-yong’s ‘The Prisoner,’ and Shin Kyung-sook’s ‘Please Look After Mom.’ The creators of ‘The Pile’ noted that digital books were included because they are invaluable for long-context modeling and consistent story generation.
DeepSeek and ‘Anna’s Archive’
China’s leading AI firm, DeepSeek, was more specific about its data sources. Its March 2024 technical report for ‘DeepSeek-VL’ notes the use of 860,000 English and 180,000 Chinese e-books secured from ‘Anna’s Archive.’
‘Anna’s Archive’ is a notorious shadow library containing English translations of prominent Korean authors, as well as Chinese translations like Kim Ae-ran’s ‘Flight of Dreams.’ While it is unconfirmed if DeepSeek downloaded these specific Korean titles, indirect evidence exists.
A paper by researchers from Columbia University and Carnegie Mellon University revealed that DeepSeek 3.1 produced 19 consecutive words of Han Kang’s ‘The Vegetarian’ verbatim after a fine-tuning process, suggesting prior pre-training on the novel, though the study remains unreviewed.
A Widespread Industry Practice
The use of pirated e-books is not limited to Anthropic or Chinese companies. Lawsuits have revealed that OpenAI and Meta also scraped digital books without authorization.
According to KAIST AI Professor Joo Jae-κ±Έ, utilizing pirated books for AI training was an open secret in the industry. Driven by intense performance competition, companies often relied on public datasets containing pirated materials without fully vetting legal risks.
