text: This work addresses the scenario of fine-tuning a pre-trained network for new data/tasks and empirically studies various regularization techniques. Overall, the evaluation concludes with recommending that all layers of a network whose weights are directly transferred during fine-tuning should be regularized against the initial net with an L2 penalty during further training. Relationship to prior work: Regularizing a target model against a source model is not a new idea. The authors miss key connections to A-SVM [1] and PMT-SVM [2] -- two proposed transfer learning models applied to SVM weights, but otherwise very much the same as the proposed solution in this paper. Though the study here may offer new insights for deep nets, it is critical to mention prior work which also does analysis of these regularization techniques. Significance: As the majority of visual recognition problems are currently solved using variants of fine-tuning, if the findings reported in this paper generalize, then it could present a simple new regularization which improves the training of new models. The change is both conceptually simple and easy to implement so could be quickly integrated by many people. Clarity and Questions: The purpose of the paper is clear, however, some questions remain unanswered. 1) How is the regularization weight of 0.01 chosen? This is likely a critical parameter. In an experimental paper, I would expect to see a plot of performance for at least one experiment as this regularization weighting parameter is varied. 2) How does the use of L2 regularization on the last layer effect the regularization choice of other layers? What happens if you use no regularization on the last layer? L1 regularization? 3) Figure 1 is difficult to read. Please at least label the test sets on each sub-graph. 4) There seems to be some issue with the freezing experiment in Figure 2. Why does performance of L2 regularization improve as you freeze more and more layers, but is outperformed by un-freezing all. 5) Figure 3 and the discussion of linear dependence with the original model in general seems does not add much to the paper. It is clear that regularizing against the source model weights instead of 0 should result in final weights that are more similar to the initial source weights. I would rather the authors use this space to provide a deeper analysis of why this property should help performance. 6) Initializing with a source model offers a strong starting point so full from scratch learning isn’t necessary -- meaning fewer examples are needed for the continued learning (fine-tuning) phase. In a similar line of reasoning, does regularizing against the source further reduce the number of labeled points needed for fine-tuning? Can you recover L2 fine-tuning performance with fewer examples when you use L2-SP? [1] J. Yang, R. Yan, and A. Hauptmann. Adapting svm classifiers to data with shifted distributions. In ICDM Workshops, 2007. [2]  Y. Aytar and A. Zisserman. Tabula rasa: Model transfer for object category detection. In Proc. ICCV, 2011.