>I think both you and the author of the article are making the same mistake here.
Maybe?
>If you have a nonlinear function f on a vector space, there's no reason why an orthogonal basis for that space will give a better parameterization than a nonorthogonal basis.
I don't think I made that claim. Here's all I'm saying: To whatever degree the features of interest are linearized in the latent space (and there's really no guarantee that they are), we don't have any guarantee that those linear features are orthogonal to one another, so tuning the latent representation along one feature will also impact others.
> (For example, take f(x,y) = (x-y,y). Then f(x,0)=(x,0) and f(y,y)=(0,y), so "correlated" input directions (1,0) and (1,1) are mapped to "independent" or orthogonal outputs.)
That's true, but remember that the nonlinear mapping is from our latent space (spanned by uniformly random 512-element input vectors) to pixel space. We really don't care about linear algebra in pixel space. I have zero expectation that we would preserve orthogonality from latent to pixel space.
I don't think any part of the GAN objective requires that these interesting features actually be linearized in the latent space (obviously they are not in pixel space), but the approach is to use a GLM to find the latent vectors that best fit the features anyway. Whether or not the vectors you identify with the GLM really retain their semantic meaning through the latent space, they're also clearly not orthogonal, so changing the latent representation along one dimension also changes others.
Maybe?
>If you have a nonlinear function f on a vector space, there's no reason why an orthogonal basis for that space will give a better parameterization than a nonorthogonal basis.
I don't think I made that claim. Here's all I'm saying: To whatever degree the features of interest are linearized in the latent space (and there's really no guarantee that they are), we don't have any guarantee that those linear features are orthogonal to one another, so tuning the latent representation along one feature will also impact others.
> (For example, take f(x,y) = (x-y,y). Then f(x,0)=(x,0) and f(y,y)=(0,y), so "correlated" input directions (1,0) and (1,1) are mapped to "independent" or orthogonal outputs.)
That's true, but remember that the nonlinear mapping is from our latent space (spanned by uniformly random 512-element input vectors) to pixel space. We really don't care about linear algebra in pixel space. I have zero expectation that we would preserve orthogonality from latent to pixel space.
I don't think any part of the GAN objective requires that these interesting features actually be linearized in the latent space (obviously they are not in pixel space), but the approach is to use a GLM to find the latent vectors that best fit the features anyway. Whether or not the vectors you identify with the GLM really retain their semantic meaning through the latent space, they're also clearly not orthogonal, so changing the latent representation along one dimension also changes others.