Causality has captivated philosophers for centuries, not merely to grapple with the exact relationship between a cause and an effect, but also because it underpins so many other concepts of interest. In this talk, we will start by exploring how causality is linked to several fundamental issues in AI safety, including incentives, misspecification, generalisation, deception, and corrigibility. Next, we will posit that agency – central to many AGI threat models – is itself a causal concept. We will then bridge our theoretical foundations with practical applications by showcasing our approaches towards developing ‘agency detectors’.