Question on Scene-Level Control in Your Framework

[itsnickchiu]

Thank you for your excellent work!

I understand that your work focuses on manipulating attributes of specific subjects. However, I’m curious that would it be possible to control global aspects of the scene within the same framework, such as weather, background, or overall environment?

[stefan-baumann]

Thank you for this interesting question!

In preliminary experiments, which included unconstrained backpropagation into all embedding tokens for single image-caption pairs (see Sec. 3.1 in the v1 of the paper on arxiv), I saw that the token embedding can also capture some more global changes (general perspective for the car example, more realistic -> stylized photo for the portrait example):

I never looked deeply into this, but I assume that a large part of the global changes captured in this setting are learned by the EOS token(s), as they seem like the most sensible candidate for this.

If you are interested in looking into this, the EOS token is definitely where I would start - learning one delta that is broadcast over the EOS tokens in a targeted manner for a target attribute in a similar setting like we do for subject-targeted attributes.

Best,
Stefan