Entity Tracking

Code reasoning is treated as partly entity-tracking-like because it involves variables, objects, functions, scopes, and update dependencies. Causal entity-tracking attention tends to concentrate most mass in local block-diagonal neighborhoods. Standard dense self-attention is costly for long-sequence entity tracking because it scales at least quadratically with sequence length. Entity tracking requires maintaining latent entity and attribute states over long sequences.