Added relativistic discriminator loss used in ESRGAN paper.

[bnb32]

This changes the previous disc loss calc (and adversarial loss) to use the relativistic versions described in the ESRGAN paper. The symmetry enables us to remove the adversarial loss function, and instead just swap the arguments in the new disc loss calc, and also seems worth doing since this improved on the SRGAN framework.

I'm also finding this to result in much stabler training, btw.

[grantbuster]

@bnb32 Starting review - to clarify, you only implemented the feature "Relativistic average GAN (RaGAN) [20], which
learns to judge “whether one image is more realistic than the other” rather than “whether one image is real or fake”" from the Wang paper, right? Seems like Wang did a lot of different things and I'm trying to track what to pay attention to.

Also, does this completely revise the previous disc method or is it an option?

[bnb32]

@grantbuster Yeah, the RaGAN disc loss is what I added. This changes the previous method. It allows us to use disc_loss(true, gen) for the discriminator and disc_loss(gen, true) for adversarial loss, instead of two different methods for these, so I removed the previous function for adversarial loss.

[grantbuster]

Minor suggestion but LGTM

[grantbuster]

This seems really inefficient to run all loss calculations and then only output one of them. Why dont we wrap the actual loss calculations in the if statement?

[bnb32]

Good call. We've had this inefficient setup for a while.

[grantbuster]

yeah no blame just reading our old code and thinking well that could have been done better haha

[bnb32]

@grantbuster Actually, we need to compute the disc loss every batch to track whether to train it or not. We can skip the gen loss calcs though.

[grantbuster]

Gotcha, makes sense. Maybe note this in-line comment so future-us remembers haha

[bnb32]

@grantbuster Nvm, how's this solution? d122ace

[grantbuster]

LGTM

[grantbuster]

Suggested change

	loss_gen_content, loss_gen_content_details = (
	self.calc_loss_gen_content(hi_res_true, hi_res_gen)
	)
	loss_gen_advers = self.calc_loss_disc(
	disc_out_true=disc_out_gen, disc_out_gen=disc_out_true
	)
	loss_gen = loss_gen_content + weight_gen_advers * loss_gen_advers
	loss_disc = self.calc_loss_disc(
	disc_out_true=disc_out_true, disc_out_gen=disc_out_gen
	)
	loss = None
	if train_gen:
	loss = loss_gen
	elif train_disc:
	loss = loss_disc
	loss_details = {
	'loss_gen': loss_gen,
	'loss_gen_content': loss_gen_content,
	'loss_gen_advers': loss_gen_advers,
	'loss_disc': loss_disc,
	}
	loss_details = {}
	if train_gen:
	loss_gen_content, loss_gen_content_details = (
	self.calc_loss_gen_content(hi_res_true, hi_res_gen)
	)
	loss_gen_advers = self.calc_loss_disc(
	disc_out_true=disc_out_gen, disc_out_gen=disc_out_true
	)
	loss = loss_gen_content + weight_gen_advers * loss_gen_advers
	loss_details['loss_gen'] = loss
	loss_details['loss_gen_content'] = loss_gen_content,
	loss_details['loss_gen_advers'] = loss_gen_advers,
	elif train_disc:
	loss = self.calc_loss_disc(
	disc_out_true=disc_out_true, disc_out_gen=disc_out_gen
	)
	loss_details['loss_disc'] = loss

[grantbuster]

You might note that in our current TF version 2.15 this actually just results in scrambled outputs.

[grantbuster]

(which is very hard to detect)