text: [Overview]In this paper, the authors proposed a novel model called MemoryGAN, which integrates memory network with GAN. As claimed by the authors, MemoryGAN is aimed at addressing two problems of GAN training: 1) difficult to model the structural discontinuity between disparate classes in the latent space; 2) catastrophic forgetting problem during the training of discriminator about the past synthesized samples by the generator. It exploits the life-long memory network and adapts it to GAN. It consists of two parts, discriminative memory network (DMN) and Memory Conditional Generative Network (MCGN). DMN is used for discriminating input samples by integrating the memory learnt in the memory network, and MCGN is used for generating images based on random vector and the sampled memory from the memory network. In the experiments, the authors evaluated memoryGAN on three datasets, CIFAR-10, affine-MNIST and Fashion-MNIST, and demonstrated the superiority to previous models. Through ablation study, the authors further showed the effects of separate components in memoryGAN. [Strengths] 1. This paper is well-written. All modules in the proposed model and the experiments were explained clearly. I enjoyed much to read the paper. 2. The paper presents a novel method called MemoryGAN for GAN training. To address the two infamous problems mentioned in the paper, the authors proposed to integrate a memory network into GAN. Through memory network, MemoryGAN can explicitly learn the data distribution of real images and fake images. I think this is a very promising and meaningful extension to the original GAN. 3. With MemoryGAN, the authors achieved best Inception Score on CIFAR-10. By ablation study, the authors demonstrated each part of the model helps to improve the final performance. [Comments] My comments are mainly about the experiment part: 1. In Table 2, the authors show the Inception Score of images generated by DCGAN at the last row. On CIFAR-10, it is ~5.35. As the authors mentioned, removing EM, MCGCN and Memory will result in a conventional DCGAN. However, as far as I know, DCGAN could achieve > 6.5 Inception Score in general. I am wondering what makes such a big difference between the reported numbers in this paper and other papers? 2. In the experiments, the authors set N = 16,384, and M = 512, and z is with dimension 16. I did not understand why the memory size is such large. Take CIFAR-10 as the example, its training set contains 50k images. Using such a large memory size, each memory slot will merely count for several samples. Is a large memory size necessary to make MemoryGAN work? If not, the authors should also show ablated study on the effect of different memory size; If it is true, please explain why is that. Also, the authors should mention the training time compared with DCGAN. Updating memory with such a large size seems very time-consuming. 3. Still on the memory size in this model. I am curious about the results if the size is decreased to the same or comparable number of image categories in the training set. As the author claimed, if the memory network could learn to cluster training data into different category, we should be able to see some interesting results by sampling the keys and generate categoric images. 4. The paper should be compared with InfoGAN (Chen et al. 2016), and the authors should explain the differences between two models in the related work. Similar to MemoryGAN, InfoGAN also did not need any data annotations, but could learn the latent code flexibly. [Summary] This paper proposed a new model called MemoryGAN for image generation. It combined memory network with GAN, and achieved state-of-art performance on CIFAR-10. The arguments that MemoryGAN could solve the two infamous problem make sense. As I mentioned above, I did not understand why the authors used such large memory size. More explanations and experiments  should be conducted to justify this setting. Overall, I think MemoryGAN opened a new direction of GAN and worth to further explore.