TY - JOUR

T1 - Cross-Subject EEG Feedback for Implicit Image Generation

AU - Torre-Ortiz, Carlos de la

AU - Spape, Michiel M.

AU - Ravaja, Niklas

AU - Ruotsalo, Tuukka

N1 - Publisher Copyright: Authors

PY - 2024

Y1 - 2024

N2 - Generative models are powerful tools for producing novel information by learning from example data. However, the current approaches require explicit manual input to steer generative models to match human goals. Furthermore, how these models would integrate implicit, diverse feedback and goals of multiple users remains largely unexplored. Here, we present a first-of-its-kind system that produces novel images of faces by inferring human goals directly from cross-subject brain signals while study subjects are looking at example images. We report on an experiment where brain responses to images of faces were recorded using electroencephalography in 30 subjects, focusing on specific salient visual features (VFs). Preferences toward VFs were decoded from subjects’ brain responses and used as implicit feedback for a generative adversarial network (GAN), which generated new images of faces. The results from a follow-up user study evaluating the presence of the target salient VFs show that the images generated from brain feedback represent the goal of the study subjects and are comparable to images generated with manual feedback. The methodology provides a stepping stone toward humans-in-the-loop image generation.

AB - Generative models are powerful tools for producing novel information by learning from example data. However, the current approaches require explicit manual input to steer generative models to match human goals. Furthermore, how these models would integrate implicit, diverse feedback and goals of multiple users remains largely unexplored. Here, we present a first-of-its-kind system that produces novel images of faces by inferring human goals directly from cross-subject brain signals while study subjects are looking at example images. We report on an experiment where brain responses to images of faces were recorded using electroencephalography in 30 subjects, focusing on specific salient visual features (VFs). Preferences toward VFs were decoded from subjects’ brain responses and used as implicit feedback for a generative adversarial network (GAN), which generated new images of faces. The results from a follow-up user study evaluating the presence of the target salient VFs show that the images generated from brain feedback represent the goal of the study subjects and are comparable to images generated with manual feedback. The methodology provides a stepping stone toward humans-in-the-loop image generation.

KW - Brain modeling

KW - Brain–computer interfaces

KW - Electroencephalography

KW - electroencephalography (EEG)

KW - Faces

KW - generative models

KW - image generation

KW - Image synthesis

KW - Manuals

KW - Task analysis

KW - Visualization

UR - http://www.scopus.com/inward/record.url?scp=85196722868&partnerID=8YFLogxK

U2 - 10.1109/TCYB.2024.3406159

DO - 10.1109/TCYB.2024.3406159

M3 - Journal article

C2 - 38889044

AN - SCOPUS:85196722868

SP - 1

EP - 0

JO - IEEE Transactions on Cybernetics

JF - IEEE Transactions on Cybernetics

SN - 2168-2267

ER -