corporate

English

arxiv_labs

You're looking at domain 'arxiv.org' on the non-profit oriented generic top-level domain (.org) while skipping any subdomain. The registrable portion 'arxiv' spans 5 characters holding two vowels versus 3 consonants. Segmentation suggests two words: ar, xiv. The median word length lands at 2.5 characters. Most frequently, 'ar' shows up in Breton. Usage also turns up in Latvian and Lithuanian contexts.

query

searchtype

source

html/55/15/5515b727-4c84-4dfe-bfc8-03a1bcf2e473.html.gz

inline; filename=cornell-reduced-white-SMALL.svg

inline; filename=arxiv-logo-one-color-white.svg

inline; filename=arxiv-logomark-small-white.svg

inline; filename=member_acknowledgement.js

https://doi.org/10.48550/arXiv.2501.00663

Which authors of this paper are endorsers?

viewport

msapplication-TileColor

theme-color

Abstract page for arXiv paper 2501.00663: Titans: Learning to Memorize at Test Time

description

Titans: Learning to Memorize at Test Time

/static/browse/0.3.4/images/arxiv-logo-fb.png

Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context. We present a new neural long-term memory module that learns to memorize historical context and helps attention to attend to the current context while utilizing long past information. We show that this neural memory has the advantage of fast parallelizable training while maintaining a fast inference. From a memory perspective, we argue that attention due to its limited context but accurate dependency modeling performs as a short-term memory, while neural memory due to its ability to memorize the data, acts as a long-term, more persistent, memory. Based on these two modules, we introduce a new family of architectures, called Titans, and present three variants to address how one can effectively incorporate memory into this architecture. Our experimental results on language modeling, common-sense reasoning, genomics, and time series tasks show that Titans are more effective than Transformers and recent modern linear recurrent models. They further can effectively scale to larger than 2M context window size with higher accuracy in needle-in-haystack tasks compared to baselines.

twitter:site

twitter:card

twitter:title

Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size...

twitter:description

https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png

twitter:image

twitter:image:alt

citation_title

citation_author

citation_date

citation_online_date

citation_pdf_url

citation_arxiv_id

citation_abstract

HSTS header missing - vulnerable to protocol downgrade attacks

CSP header missing - vulnerable to XSS attacks

child-src https:  chrome-untrusted://new-tab-page/ chrome-untrusted://ntp-microsoft-auth/;connect-src chrome://resources chrome://theme 'self';default-src 'self';font-src chrome://resources 'self';img-src chrome://resources chrome://theme chrome://image chrome://favicon2 chrome://app-icon chrome://extension-icon chrome://fileicon blob: data: 'self';object-src 'none';require-trusted-types-for 'script';script-src chrome://resources chrome://webui-test 'self';style-src chrome://resources chrome://theme 'self' 'unsafe-inline';trusted-types parse-html-subset sanitize-inner-html static-types webui-test-script webui-test-html polymer-html-literal polymer-template-event-attribute-policy lit-html-desktop;frame-ancestors 'none';

child-src 'none';object-src 'none';require-trusted-types-for 'script';script-src chrome://resources 'self';trusted-types;worker-src blob: 'self';frame-ancestors 'none';

child-src 'none';object-src 'none';require-trusted-types-for 'script';script-src chrome://resources 'self';trusted-types;frame-ancestors 'none';

Jeffrey Neace

Network Solutions, LLC

arxiv.org

Abuse Account

Fastly RIR Administrator

Fastly Network Operations

Fastly, Inc.

SKYCA-3

MathJax.Hub.Config({
  messageStyle: 'none',
  extensions: ['tex2jax.js'],
  jax: ['input/TeX','output/HTML-CSS'],
  tex2jax: {
    inlineMath: [['$','$']],
    processEscapes: true,
    ignoreClass: 

Script

Open-Graph-Protocol

Google Cloud

Google Cloud Trace

jsDelivr

jQuery CDN

jQuery

[2501.00663] Titans: Learning to Memorize at Test Time

⚠️ High Risk - https://arxiv.org/abs/2501.00663

Requests	IP Address	Location	AS Autonomous System
16	151.101.67.42	San Francisco, California, United States	AS54113FASTLY
1	151.101.195.42	San Francisco, California, United States	AS54113FASTLY
1	151.101.3.42	San Francisco, California, United States	AS54113FASTLY
1	104.16.175.226	United States	AS13335CLOUDFLARENET
1	151.101.131.42	San Francisco, California, United States	AS54113FASTLY
1	151.101.2.137	San Francisco, California, United States	AS54113FASTLY
1	104.16.174.226	United States	AS13335CLOUDFLARENET
1	151.101.130.137	San Francisco, California, United States	AS54113FASTLY
1	151.101.66.137	San Francisco, California, United States	AS54113FASTLY
1	2a04:4e42:200::649	United States	AS54113FASTLY
31	16	-	-

Security Scan Report: arxiv.org

Summary

AI Security Verdict

Safe Website

Safety Factors

Details

Screenshot

Page Load Overview

Language Analysis

Primary Language

Detection Details

Website Classification

Primary Category

All Detected Categories

Detected Features

Domain & IP Information

Detected Technologies6

Content Similarity HashesFor malware variant detection

TLSH (Trend Micro Locality Sensitive Hash)

ssdeep (Context Triggered Piecewise Hashing)

sdhash (Similarity Digest Hashing)

Image Hashes

Perceptual Hashes

Other Hashes

Scan History