MUMBAI, India, Oct. 12 -- Intellectual Property India has published a patent application (202641118755 A) filed by Easwari Engineering College; and Srm Institute Of Science And Technology, Ramapuram Campus on October 05, 2026, for Cross-Session Semantic Key-Value Cache Scheduling System And Method For Large Language Model Inference.
Inventors include Kamalesh B; Jagan S; and Dr. J. Priya.
The application for the patent was published on October 09, 2026, under issue no. 41/2026.
Abstract: The present invention relates to a system and method for cross-session semantic reuse of Key-Value cache data structures in large language model inference. In particular, incoming text prompts from different user sessions are converted into dense vector representations and clustered into semantic groups according to their semantic similarity. An intelligent scheduling controller prioritizes semantic groups by performing a utility estimation based on factors such as access frequency, temporal distance, recomputation cost, size of the Key-Value cache data structure, and popularity trend. Based on the utility estimation, the scheduling controller then migrates Key-Value cache data structures between different levels of a memory hierarchy including a GPU memory level, a host memory level, and a secondary memory level to enable cache-assisted inference. This approach allows for improved memory utilization, reduced redundant computations, lower latency, and better support for concurrent large language model inference from different user sessions while maintaining execution semantics.
Disclaimer: Curated by HT Syndication.
