How Transformers Learn Causal Structure with Gradient Descent

Watch a 38-minute lecture from Princeton University's Jason Lee at the Simons Institute exploring how transformer models learn and encode causal relationships through gradient descent training. Delve into the mechanics of self-attention mechanisms that enable transformers to effectively model sequences by transferring information between different sequence parts. Examine a specific in-context learning task designed to demonstrate how transformers learn latent causal structure, supported by mathematical proofs showing that gradient descent on a simplified two-layer transformer successfully encodes causal graphs in its first attention layer. Learn how the gradient of the attention matrix captures mutual information between tokens, and through the data processing inequality, reveals edges in latent causal graphs. Understand how these principles apply to special cases like in-context Markov chains where transformers develop induction heads, with experimental validation showing transformers can recover various causal structures through training.