PhD defence by Zhan Su

Follow this link to participate on Zoom

Title

Information Propagation in Modular Language Modeling and Web Tracking

Abstract

Information propagation is the process through which data are transmitted within a system. The growth of large-scale web datasets has led to explosive growth in information, triggering new research areas such as large language models (Brown et al., 2020a) or digital surveillance (Westerlund et al., 2021). In the realm of language modeling, this thesis studies information propagation in modular language modeling, where a subset of training parameters are treated as modules. Each module can be individually trained using a domain-specific dataset, leading to the creation of a module uniquely tailored to each domain. Following this, a routing function determines which module should be activated, and an aggregation function is then employed to integrate the outputs of the active modules. This study provides an in-depth analysis of information propagation in modular language modeling, examining three key aspects: modules, routing, and aggregation. Firstly, a novel module that incorporates tensor product representation is introduced, making a significant advancement in parameter efficiency by considerably reducing the number of parameters needed in the trained model. Secondly, the investigation explores a variety of routing functions in the context of multi-task learning, focusing on few-shot and zero-shot scenarios. thirdly, a new aggregation method is presented, designed from the ground up based on the newly proposed modules. Finally, we take steps to create modular language models by building and reusing a library of modules, paving the way for efficient and flexible utilization of language models across a wide array of tasks. In the domain of digital surveillance, the research also delves into information propagation in web tracking, with a particular focus on the evolution of web tracking practices over the past decade. A comprehensive historical analysis of third-party web tracking practices is conducted by utilizing the Wayback Machine. This approach not only sheds light on the technical advancements in web tracking but also maps out the changing landscape of digital surveillance.

Supervisors

Principal Supervisor Jakob Grue Simonsen

Assessment Committee

Associate Professor Ken Friis Larsen, Computer Science
Professor Nishanth Sastry, University of Surrey
Principal Research Associate Ivan Vulic, University of Cambridge

Leader of defense: Professor Fritz Henglein, Computer Science

For an electronic copy of the thesis, please visit the PhD Programme page.