MUMBAI, India, July 30 -- Intellectual Property India has published a patent application (202641088506 A) filed by Neelambika Basavaraj Hiremath; Dayananda Sagar Academy Of Technology And Management; Ms. Lakshmi S R; Abhishek Y; Vikas Gowda H B; Vikram S; and Sanjay G V on July 20, 2026, for Twinly- Dataset Similarity Detection Using Metadata Management Tools And Version Control.
Inventors include Neelambika Basavaraj Hiremath; Dayananda Sagar Academy Of Technology And Management; Ms. Lakshmi S R; Abhishek Y; Vikas Gowda H B; Vikram S; and Sanjay G V.
The application for the patent was published on July 24, 2026, under issue no. 30/2026.
Abstract: Dataset similarity detection and metadata governance have become critical challenges in modern Big Data environments due to the rapid growth of heterogeneous datasets across organizations. Existing duplicate detection systems mainly rely on checksum hashing, filename comparison, or manual metadata analysis, which are unable to accurately identify semantic duplicates, structural similarities, partially overlapping datasets, or redundant submissions stored in different formats. These traditional approaches also suffer from high false-positive rates, poor scalability, lack of semantic understanding, and limited redundancy optimization capabilities. The present invention introduces Twinly – Dataset Similarity Detection Using Metadata Management Tools and Version Control, an intelligent hybrid similarity analysis platform designed to detect exact duplicates, semantic duplicates, structural similarities, and overlapping datasets across heterogeneous file repositories. The proposed framework combines SHA256 checksum hashing, schema similarity analysis, semantic embedding techniques using SentenceTransformers, cosine similarity, statistical content overlap analysis, K-Nearest Neighbors (KNN) classification, and K-Means clustering into a unified intelligent similarity pipeline. The platform supports multiple dataset formats including CSV, XLSX, JSON, TSV, and Parquet, and converts all heterogeneous datasets into a unified dataframe representation for scalable analysis and metadata-driven governance. Experimental evaluation demonstrated that the proposed system achieved a duplicate detection accuracy of 91%, semantic similarity accuracy of 84%, and overall classification accuracy of 87% while reducing false positives by 72% compared to traditional checksum-only approaches. The invention successfully generated meaningful similarity clusters, overlap analysis reports, redundancy optimization metrics, and interactive visualization dashboards for large dataset repositories. The proposed system provides an efficient, scalable, and enterprise-ready solution for intelligent dataset governance, redundancy elimination, metadata intelligence, and storage optimization in modern Big Data ecosystems.
Disclaimer: Curated by HT Syndication.
