October 2020
tl;dr: Plug-in TSM module to enable 2D conv nets for efficient video understanding.
The idea seems to be inspired by Shift as conv but naive implementation did not work, both in efficiency and performance. It uses efficient 2D conv operations and reaches good performance as compared to 3D conv.
It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. Roughly same latency as 2D CNN baseline. The only cost is 1/4 of the feature maps of each residual block need to be saved. For ResNet-50, we only have to cache 0.9 MB data. TSM only uses 2D convolution which is highly optimized for hardwares.
TSM is orthoganal to non-local, and boosts all 2D conv backbones. It leads to double digit improvement in some challenging datasets that has strong temporal relationship.
It seems to be better than convLSTM CVPR 2015 oral.