Impact statement: (as required by NeurIPS 2020 CFP) Continual learning is a key desiderata for Artificial General Intelligence (AGI). Hence, this line of research has the benefits as well as the pitfalls of any other research effort geared in this direction. In particular, our work can help deliver impact on making smarter AI products and services, which can learn and update themselves on-the-fly when newer tasks and domains are encountered, without forgetting previously acquired knowledge. This is a necessity in any large-scale deployments of machine learning and computer vision, including in social media, e-commerce, surveillance, e- governance, etc - each of which have newer settings, tasks or domains added continually over time. Any negative effect of our work, such as legal and ethical concerns, are not unique to this work - to the best of our knowledge, but are shared with any other new development in machine learning, in general.