Miloš Prágr presents Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence

On 2020-01-14 11:00:00 at G205, Karlovo náměstí 13, Praha 2
Reading group on the work "Riemannian Walk for Incremental Learning:
Understanding Forgetting and Intransigence" (ECCV 2018) by Arslan Chaudhry,
Puneet K. Dokania, Thalaiyasingam Ajanthan, Philip H. S. Torr presented by
Miloš Prágr.

Paper abstract:
Incremental learning (IL) has received a lot of attention recently, however,
the literature lacks a precise problem definition, proper evaluation settings,
and metrics tailored specifically for the IL problem. One of the main
objectives of this work is to fill these gaps so as to provide a common ground
for better understanding of IL. The main challenge for an IL algorithm is to
update the classifier whilst preserving existing knowledge. We observe that, in
addition to forgetting, a known issue while preserving knowledge, IL also
suffers from a problem we call intransigence, inability of a model to update
knowledge. We introduce two metrics to quantify forgetting and intransigence
that allow us to understand, analyse, and gain better insights into the
behaviour of IL algorithms. We present RWalk, a generalization of EWC++ (our
efficient version of EWC [Kirkpatrick2016EWC]) and Path Integral
[Zenke2017Continual] with a theoretically grounded KL-divergence based
perspective. We provide a thorough analysis of various IL algorithms on MNIST
and CIFAR-100 datasets. In these experiments, RWalk obtains superior results in
terms of accuracy, and also provides a better trade-off between forgetting and

The reading group will additionally include background from prior work

[Kirkpatrick2016EWC] Kirkpatrick, J., Pascanu, R., Rabinowitz, N.C., Veness,
J., Desjardins, G., Rusu, A.A., Milan,K., Quan, J., Ramalho, T.,
Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran,D., Hadsell, R.:
Overcoming catastrophic forgetting in neural networks. Proceedings of the
National Academy of Sciences of the United States of America (PNAS) (2016.
[Zenke2017Continual] Zenke, F., Poole, B., Ganguli, S.: Continual learning
through synaptic intelligence. In: ICML (2017)

Paper URL:

Instructions for participants: The reading group studies the literature in the
field of pattern recognition and computer vision. At each meeting one or more
papers are prepared for presentation by a single person, the presenter. The
meetings are open to anyone, disregarding their background. It is assumed that
everyone attending the reading group has, at least briefly, read the paper –
not necessarily understanding everything. Attendants should preferably send
questions about the unclear parts to the speaker at least one day in advance.
During the presentation we aim to have a fruitful discussion, a critical
analysis of the paper, as well as brainstorming for creative extensions.

See the page of reading groups
Responsible person: Petr Pošík