text: This paper suggests a simple yet effective approach for learning with weak supervision.  This learning scenario involves two datasets, one with clean data (i.e., labeled by the true function) and one with noisy data, collected using a weak source of supervision.   The suggested approach assumes a teacher and student networks,  and builds the final representation incrementally, by taking into account the "fidelity" of the weak label when training the student at the final step.  The fidelity score is given by the teacher, after being trained over the clean data,  and it's used to build a cost-sensitive loss function for the students.  The suggested method seems to work well on several document classification tasks.  Overall, I liked the paper.   I would like the authors to consider the following questions - - Over the last 10 years or so, many different frameworks for learning with weak supervision were suggested (e.g., indirect supervision, distant supervision, response-based, constraint-based, to name a few).   First, I'd suggest acknowledging these works and discussing the differences to your work.  Second - Is your approach applicable to these frameworks?   It would be an interesting to compare to one of those methods  (e.g., distant supervision for relation extraction using a knowledge base), and see if by incorporating fidelity score, results improve.  - Can this approach be applied to semi-supervised learning?  Is there a reason to assume the fidelity scores computed by the teacher would not improve the student in a self-training framework? - The paper emphasizes that the teacher uses the student's initial representation, when trained over the clean data.   Is it clear that this step in needed?  Can you add an additional variant of your framework when the fidelity score are  computed by the teacher when trained from scratch?  using different architecture than the student?