Aleksandar Petrov
ペトロフ・アレクサンダー
University of Oxford
Aleksandar Petrov is a PhD student in the Autonomous Intelligent Machines and Systems CDT at the University of Oxford supervised by Philip Torr and Adel Bibi. His research interests lie in safe and reliable machine learning, such as autonomous systems and language models, focusing on methods to evaluate and verify their performance. He is also interested in the ethical implications of deploying AI-based systems as well as their regulation and governance. Prior to Oxford, he studied for an MSc at ETH Zürich focusing on applied category theory. His thesis was on Compositional Computational Systems.
Universal Approximation via Prefix-tuning a Single Transformer Attention Head
Saturday, April 6th, 15:30–16:00
Despite the widespread adoption of prompting, prompt tuning and prefix-tuning of transformer models, our theoretical understanding of these fine-tuning methods remains limited. A key question is whether one can arbitrarily modify the behavior of pretrained model by prompting or prefixtuning it. Formally, whether prompting and prefixtuning a pretrained model can universally approximate sequence-to-sequence functions. This paper answers in the affirmative and demonstrates that much smaller pretrained models than previously thought can be universal approximators when prefixed. In fact, the attention mechanism is uniquely suited for universal approximation with prefixtuning a single attention head being sufficient to approximate any continuous function. Moreover, any sequence-to-sequence function can be approximated by prefixing a transformer with depth linear in the sequence length. Beyond these densitytype results, we also offer Jackson-type bounds on the length of the prefix needed to approximate a function to a desired precision.
Technical AI Safety Conference 