Technical AI Safety Conference

Jesse Hoogland

フーグランド・ジェス

Timaeus

Jesse Hoogland is the executive director of Timaeus. In addition to managing the organization, he conducts research on singular learning theory and developmental interpretability. He was previously a research assistant in David Krueger’s lab, where he worked with Lauro Langosco and Xander Davies on unifying grokking and double descent.

The Structure and Development of Neural Networks

Friday, April 6th, 10:30–11:00

We show that in-context learning emerges in transformers in discrete developmental stages, when they are trained on either language modeling or linear regression tasks. We introduce two methods for detecting the milestones that separate these stages, by probing the geometry of the population loss in both parameter space and function space. We study the stages revealed by these new methods using a range of behavioral and structural metrics to establish their validity.