AI2 Introduces Infini-gram for Text Provenance

The Allen Institute for AI (AI2) has developed Infini-gram, a vast n-gram language model. This model indexes 1.4 trillion tokens from 125 million books, encompassing both publicly available and copyrighted works. Its primary function is to determine whether a given text sequence appears verbatim within its indexed corpus. The system can identify if text segments, up to 13-grams, are present in its dataset. This capability allows Infini-gram to trace text generation directly to source materials.

Infini-gram offers a method to analyse the originality of AI-generated content. When AI models produce text, they sometimes output sequences identical to their training data. This can occur even if the original text is copyrighted. Infini-gram detects these exact matches, providing evidence of direct reproduction. The system’s architecture enables rapid lookup, completing a search in milliseconds. This efficiency allows for large-scale analysis of AI outputs.

The development of Infini-gram addresses growing concerns about intellectual property in the age of generative AI. By identifying copied phrases, the tool can inform discussions on copyright infringement. It provides a mechanism to examine claims of originality for AI-produced content. The model’s underlying data includes a wide array of published literature, making it a comprehensive reference. Future applications could extend to academic integrity checks and broader content verification.

AI2 intends for Infini-gram to be a foundational resource. They have made the lookup tool publicly accessible. This allows researchers and legal professionals to investigate text origins. The system does not assess copyright status directly. Instead, it provides data points for such assessments. How will the publishing industry leverage this new tracing capability?