text: Quality The method description, particularly about reference ambiguity, I found difficult to follow. The experiments and analysis look solid, although it would be nice to see experiments on more challenging natural image datasets. Clarity “In general this is not possible… “ - you are saying it is not possible to learn an encoder that recovers disentangled factors of variation? But that seems to be one of the main goals of the paper. It is not clear at all what is meant here or what the key problem is, which detracts from the paper’s motivation. What is the purpose of R_v and R_c in eq 2? Why can these not be collapsed into the encoders N_v and N_c? What does “different common factor” mean? What is f_c in proof of proposition 1? Previously f (no subscript) was referred to as a rendering engine. T(v,c) ~ p_v and c ~ p_c are said to be independent. But T(v,c) is explicitly defined in terms of c (equation 6). So which is correct? Overall the argument seems plausible - pairs of images in which a single factor of variation changes have a reference ambiguity - but the details are unclear. Originality The model is very similar to Mathieu et al, although using image pairs rather than category labels directly. The idea of weakly-supervised disentangling has also been explored in many other papers, e.g. “Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis”, Yang et al. The description of reference ambiguity seems new and potentially valuable, but I did not find it easy to follow. Significance Disentangling factors of variation with weak supervision is an important problem, and this paper makes a modest advance in terms of the model and potentially in terms of the theory. The analysis in figure 3 I found particularly interesting - illustrating that the encoder embedding dimension can have a drastic effect on the shortcut problem. Overall I think this can be a significant contribution if the exposition can be improved. Pros- Proposed method allows disentangling two factors of variation given a training set of image pairs with one factor of variation matching and the other non-matching. - A challenge inherent to weakly supervised disentangling called reference ambiguity is described. Cons- Only two factors of variation are studied, and the datasets are fairly simple. - The method description and the description of reference ambiguity are unclear.