MUMBAI, India, Nov. 2 -- Intellectual Property India has published a patent application (202611108834 A) filed by Indian Institute Of Technology, Jammu on September 10, 2026, for Data Curation System And Method For Multilingual Large Language Model Corpus Preparation.
Inventors include Kumar, Sanjeev; Kumar, Paidisetty Rohan; and Kaloge, Mayur Vinayak.
The application for the patent was published on October 30, 2026, under issue no. 44/2026.
Abstract: The present disclosure relates to a data curation system (100) and a method (700) for preparing multilingual corpus records for a Large Language Model (LLM). The data curation system (100) includes a corpus input interface (108), worker computing nodes (102), and an output interface (110). Each worker computing node (102) includes processor(s) (106) and a memory (104). The processor(s) (106) assign language buckets based on language votes and a dominant-script determination, generate exact duplicate keys, fuzzy-duplicate candidate signatures, and similarity bucket identifiers in a single processing pass using vector-processing instructions, and identify exact duplicate document records, fuzzy duplicate document records, and semantic duplicate document records. Semantic duplicate identification uses compact binary similarity representations generated from embedding vectors and bit-level distance comparison. The output interface (110) outputs curated language-bucketed corpus records to a curated corpus record destination (114).
Disclaimer: Curated by HT Syndication.
