Detail of the student project

Topic:Learning Good Descriptors to Compress
Department:Katedra kybernetiky
Supervisor:Torsten Sattler, Dr. rer. nat.
Announce as:Diplomová práce, Semestrální projekt
Description:Matching local features between images or matching features between an image and a 3D scene model are fundamental tasks in many computer vision pipelines, including Structure-from-Motion and visual localization, with applications in robotics (self-driving cars, etc.) and augmented and virtual reality. Typically, matching is done by comparing feature descriptors and performing nearest neighbor search in descriptor space.
In order to scale to large scenes, e.g., for city-scale reconstruction or world-scale visual localization, it is necessary to compress the descriptors in order to take up as little space as possible while still maintaining a good matching performance [1]. A typical example for such compression approaches is product quantization [2], which splits a descriptor into multiple parts and quantizes each part to be stored using just a few bits / a byte. While modern local features are typically trained to be descriminative [3] (meaning that descriptors belonging to the same feature are very similar and descriptors belonging to different features are very different), the ability to effectively compress the descriptors without loosing descriminative power (too much) is not taken into account. Building on insights from [4], the goal of this project is to train feature descriptors to be both descriminative and compressable by jointly optimizing objectives for both properties during training.
If successful, this project could lead to a publication in one of the main computer vision conferences (CVPR, ECCV, ICCV).
Bibliography:[1] Lynen et al., Large-scale, real-time visual–inertial localization revisited, IJRR 2020
[2] Jegou et al., Product quantization for nearest neighbor search, TPAMI 2010
[3] Mishchuk et al., Working hard to know your neighbor's margins: Local descriptor learning loss, NeurIPS 2017
[4] Sablayrolles et al., Spreading vectors for similarity search, ICLR 2019
Responsible person: Petr Pošík