AI researchers release Talkie, a 13-billion-parameter language model trained exclusively on English-language texts published before 1931, and show it can solve simple Python problems despite its historical knowledge cutoff.

A trio of AI researchers has released Talkie, a 13-billion-parameter language model trained solely on digital scans of English-language books, newspapers, periodicals, scientific journals, patents, and case law published before the end of 1930. The team describes it as the largest vintage language model they are aware of.

When tested against a model with identical architecture trained on modern data, Talkie generated some correct solutions to Python programming test problems, but only simple ones: 'All correct solutions generated by the vintage models are simple one-line programs (such as adding two inputs), or small modifications to in-context example programs,' the team said.

The researchers plan to scale Talkie significantly and estimate they can grow the corpus to well over a trillion tokens of historical text, which they say should be sufficient to create a GPT-3.5 level model—similar in capability to the original ChatGPT—by summer. The team is also working to address issues including OCR (optical character recognition) noise in the training data and temporal leakage, where the model incorrectly identifies information from after 1930.