Fine-Grained Image (细粒度图像) – Papers, Codes and Datasets
Table of contents
-
Introduction
-
Survey papers
-
Benchmark datasets
-
Fine-grained image recognition
-
Fine-grained recognition by localization-classification subnetworks
-
Fine-grained recognition by end-to-end feature encoding
-
Fine-grained recognition with external information
-
Fine-grained recognition with web data / auxiliary data
-
Fine-grained recognition with multi-modality data
-
Fine-grained recognition with humans in the loop
-
-
-
Fine-grained image retrieval
-
Unsupervised with pre-trained models
-
Supervised with metric learning
-
-
Fine-grained image generation
-
Generating from fine-grained image distributions
-
Generating from text descriptions
-
-
Future directions of FGIA
-
Automatic fine-grained models
-
Fine-grained few shot learning
-
Fine-grained hashing
-
FGIA within more realistic settings
-
-
Leaderboard
1. Introduction
This homepage lists some representative papers/codes/datasets all about deep learning based fine-grained image, including fine-grained image recognition, fine-grained image retrieval, fine-grained image generation, etc. If you have any questions, please feel free to leave message.
2. Survey papers
A Survey on Deep Learning-based Fine-Grained Object Classification and Semantic Segmentation
Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan. International Journal of Automation and Computing, 2017.
3. Benchmark datasets
Summary of popular fine-grained image datasets. Note that ‘‘BBox’’ indicates whether this dataset provides object bounding box supervisions. ‘‘Part anno.’’ means providing the key part localizations. ‘‘HRCHY’’ corresponds to hierarchical labels. ‘‘ATR’’ represents the attribute labels (e.g., wing color, male, female, etc). ‘‘Texts’’ indicates whether fine-grained text descriptions of images are supplied.
Dataset name | Year | Meta-class | Images | Categories | BBox | Part anno. | HRCHY | ATR | Texts |
Oxford flower | 2008 | Flowers | 8,189 | 102 | |||||
CUB200 | 2011 | Birds | 11,788 | 200 | |||||
Stanford Dog | 2011 | Dogs | 20,580 | 120 | |||||
Stanford Car | 2013 | Cars | 16,185 | 196 | |||||
FGVC Aircraft | 2013 | Aircrafts | 10,000 | 100 | |||||
Birdsnap | 2014 | Birds | 49,829 | 500 | |||||
NABirds | 2015 | Birds | 48,562 | 555 | |||||
DeepFashion | 2016 | Clothes | 800,000 | 1,050 | |||||
Fru92 | 2017 | Fruits | 69,614 | 92 | |||||
Veg200 | 2017 | Vegetable | 91,117 | 200 | |||||
iNat2017 | 2017 | Plants & Animals | 859,000 | 5,089 | |||||
RPC | 2019 | Retail products | 83,739 | 200 |
4. Fine-grained image recognition
Fine-grained recognition by localization-classification subnetworks
-
Part-based R-CNNs for Fine-Grained Category Detection
Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. ECCV, 2014. [Code]
-
Interaction Part Mining: A Mid-Level Approach for Fine-Grained Action Recognition
Yang Zhou, Bingbing Ni, Richang Hong, Meng Wang, and Qi Tian. CVPR, 2015.
-
Fine-Grained Recognition without Part Annotations
Jonathan Krause, Hailin Jin, Jianchao Yang, and Li Fei-Fei. CVPR, 2015. [Code]
-
The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classification
Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. CVPR, 2015.
-
Deep LAC: Deep Localization, Alignment and Classification for Fine-grained Recognition
Di Lin, Xiaoyong Shen, Cewu Lu, and Jiaya Jia. CVPR, 2015.
-
Spatial Transformer Networks
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. NIPS, 2015. [Code]
-
Part-Stacked CNN for Fine-Grained Visual Categorization
Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang. CVPR, 2016.
-
Mining Discriminative Triplets of Patches for Fine-Grained Classification
Yaming Wang, Jonghyun Choi, Vlad I. Morariu, and Larry S. Davis. CVPR, 2016.
-
SPDA-CNN: Unifying Semantic Part Detection and Abstraction for Fine-grained Recognition
Han Zhang, Tao Xu, Mohamed Elhoseiny, Xiaolei Huang, Shaoting Zhang, Ahmed Elgammal, and Dimitris Metaxas. CVPR, 2016.
-
Picking Deep Filter Responses for Fine-grained Image Recognition
Xiaopeng Zhang, Hongkai, Xiong, Wengang Zhou, Weiyao Lin, and Qi Tian. CVPR, 2016.
-
Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-Grained Image Recognition
Jianlong Fu, Heliang Zheng, and Tao Mei. CVPR, 2017.
-
Fine-Grained Recognition as HSnet Search for Informative Image Parts
Michael Lam, Behrooz Mahasseni, and Sinisa Todorovic. CVPR, 2017.
-
Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition
Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo. ICCV, 2017. [Code]
-
Weakly Supervised Learning of Part Selection Model with Spatial Constraints for Fine-Grained Image Classification
Xiangteng He, and Yuxin Peng. AAAI, 2017.
-
Localizing by Describing: Attribute-Guided Attention Localization for Fine-Grained Recognition
Xiao Liu, Jiang Wang, Shilei Wen, Errui Ding, and Yuanqing Lin. AAAI, 2017.
-
Learning to Navigate for Fine-grained Classification
Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, Jun Gao, and Liwei Wang. ECCV, 2018. [Code]
-
Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition
Ming Sun, Yuchen Yuan, Feng Zhou, and Errui Ding. ECCV, 2018. [Code]
-
Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification From the Bottom Up
Weifeng Ge, Xiangru Lin, and Yizhou Yu. CVPR, 2019.
Fine-grained recognition by end-to-end feature encoding
-
Hyper-Class Augmented and Regularized Deep Learning for Fine-Grained Image Classification
Saining Xie, Tianbao Yang, Xiaoyu Wang, and Yuanqing Lin. CVPR, 2015.
-
Subset Feature Learning for Fine-Grained Category Classification
ZongYuan Ge, Christopher McCool, Conrad Sanderson, and Peter Corke. CVPR, 2015.
-
Bilinear CNN Models for Fine-grained Visual Recognition
Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. ICCV, 2015. [Code]
-
Multiple Granularity Descriptors for Fine-Grained Categorization
Dequan Wang, Zhiqiang Shen, Jie Shao, Wei Zhang, Xiangyang Xue, and Zheng Zhang. ICCV, 2015.
-
Compact Bilinear Pooling
Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. CVPR, 2016. [Code]
-
Fine-Grained Image Classification by Exploring Bipartite-Graph Labels
Feng Zhou, and Yuanqing Lin. CVPR, 2016. [Project Page]
-
Kernel Pooling for Convolutional Neural Networks
Yin Cui, Feng Zhou, Jiang Wang, Xiao Liu, Yuanqing Lin, and Serge Belongie. CVPR, 2017.
-
Low-rank Bilinear Pooling for Fine-Grained Classification
Shu Kong, and Charless Fowlkes. CVPR, 2017. [Code]
-
Higher-order Integration of Hierarchical Convolutional Activations for Fine-Grained Visual Categorization
Sijia Cai, Wangmeng Zuo, and Lei Zhang. ICCV, 2017. [Code]
-
Learning a Discriminative Filter Bank within a CNN for Fine-grained Recognition
Yaming Wang, Vlad I. Morariu, and Larry S. Davis. CVPR, 2018. [Code]
-
Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization
Peihua Li, Jiangtao Xie, Qilong Wang, and Zilin Gao. CVPR, 2018. [Code]
-
Maximum-Entropy Fine Grained Classification
Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. NIPS, 2018.
-
Pairwise Confusion for Fine-Grained Visual Classification
Abhimanyu Dubey, Otkrist Gupta, Pei Guo, Ramesh Raskar, Ryan Farrell, and Nikhil Naik. ECCV, 2018. [Code]
-
DeepKSPD: Learning Kernel-matrix-based SPD Representation for Fine-Grained Image Recognition
Melih Engin, Lei Wang, Luping Zhou, and Xinwang Liu. ECCV, 2018.
-
Hierarchical Bilinear Pooling for Fine-Grained Visual Recognition
Chaojian Yu, Xinyi Zhao, Qi Zheng, Peng Zhang, and Xinge You. ECCV, 2018. [Code]
-
Grassmann Pooling as Compact Homogeneous Bilinear Pooling for Fine-Grained Visual Classification
Xing Wei, Yue Zhang, Yihong Gong, Jiawei Zhang, and Nanning Zheng. ECCV, 2018.
-
Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-Grained Image Recognition
Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. CVPR, 2019. [Code]
-
Destruction and Construction Learning for Fine-grained Image Recognition
Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. CVPR, 2019. [Code]
Fine-grained recognition with external information
Fine-grained recognition with web data / auxiliary data
-
Augmenting Strong Supervision Using Web Data for Fine-Grained Categorization
Zhe Xu, Shaoli Huang, Ya Zhang, and Dacheng Tao. ICCV, 2015.
-
Webly Supervised Learning Meets Zero-shot Learning: A Hybrid Approach for Fine-grained Classification
Li Niu, Ashok Veeraraghavan, and Vshu Sabbarwal. CVPR, 2018.
-
Fine-Grained Visual Categorization using Meta-Learning Optimization with Sample Selection of Auxiliary Data
Yabin Zhang, Hui Tang, and Kai Jia. ECCV, 2018. [Code]
-
Learning from Web Data using Adversarial Discriminative Neural Networks for Fine-Grained Classification
Xiaoxiao Sun, Liyi Chen, and Jufeng Yang. AAAI, 2019.
Fine-grained recognition with multi-modality data
-
Fine-Grained Image Classification via Combining Vision and Language
Xiangteng He, and Yuxin Peng. CVPR, 2017.
-
Audio Visual Attribute Discovery for Fine-Grained Object Recognition
Hua Zhang, Xiaochun Cao, and Rui Wang. AAAI, 2018.
-
Fine-grained Image Classification by Visual-Semantic Embedding
Huapeng Xu, Guilin Qi, Jingjing Li, Meng Wang, Kang Xu, and Huan Gao. IJCAI, 2018.
-
Knowledge-Embedded Representation Learning for Fine-Grained Image Recognition
Tianshui Chen, Liang Lin, Riquan Chen, Yang Wu, and Xiannan Luo. IJCAI, 2018.
Fine-grained recognition with humans in the loop
-
Fine-grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loop
Yin Cui, Feng Zhou, Yuanqing Lin, and Serge Belongie. CVPR, 2016.
5. Fine-grained image retrieval
Unsupervised with pre-trained models
-
Selective Convolutional Descriptor Aggregation for Fine-Grained Image Retrieval
Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou. TIP, 2017. [Project Page]
Supervised with metric learning
-
Centralized Ranking Loss with Weakly Supervised Localization for Fine-Grained Object Retrieval
Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, Feiyue Huang, and Yanhua Yang. IJCAI, 2018.
-
Towards Optimal Fine Grained Retrieval via Decorrelated Centralized Loss with Normalize-Scale layer
Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Baochang Zhang, Yongjian Wu, and Feiyue Huang. AAAI, 2019.
6. Fine-grained image generation
Generating from fine-grained image distributions
-
CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training
Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. ICCV, 2017. [Code]
-
FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. CVPR, 2019. [Code]
Generating from text descriptions
-
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. CVPR, 2018. [Code]
7. Future directions of FGIA
Fine-grained few shot learning
-
Piecewise classifier mappings: Learning fine-grained learners for novel categories with few examples
Xiu-Shen Wei, Peng Wang, Lingqiao Liu, Chunhua Shen, and Jianxin Wu. TIP, 2019.
FGIA within more realistic settings
-
Fine-grained Recognition in the Wild: A Multi-Task Domain Adaptation Approach
Timnit Geru, Judy Hoffman, and Li Fei-Fei. ICCV, 2017.
-
The iNaturalist Species Classification and Detection Dataset
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. CVPR 2018.
-
RPC: A Large-Scale Retail Product Checkout Dataset
Xiu-Shen Wei, Quan Cui, Lei Yang, Peng Wang, and Lingqiao Liu. arXiv: 1901.07249, 2019. [Project Page]
8. Leaderboard
The section is being continually updated. Since CUB200-2011 is the most popularly used fine-grained dataset, we list the fine-grained recognition leaderboard by treating it as the test bed.
Method | Publication | BBox | Part | External information | Base model | Image resolution | Accuracy |
PB R-CNN | ECCV 2014 | Alex-Net | 224x224 | 73.9% | |||
MaxEnt | NIPS 2018 | GoogLeNet | TBD | 74.4% | |||
PB R-CNN | ECCV 2014 | Alex-Net | 224x224 | 76.4% | |||
PS-CNN | CVPR 2016 | CaffeNet | 454x454 | 76.6% | |||
MaxEnt | NIPS 2018 | VGG-16 | TBD | 77.0% | |||
Mask-CNN | PR 2018 | Alex-Net | 448x448 | 78.6% | |||
PC | ECCV 2018 | ResNet-50 | TBD | 80.2% | |||
DeepLAC | CVPR 2015 | Alex-Net | 227x227 | 80.3% | |||
MaxEnt | NIPS 2018 | ResNet-50 | TBD | 80.4% | |||
Triplet-A | CVPR 2016 | Manual labour | GoogLeNet | TBD | 80.7% | ||
Multi-grained | ICCV 2015 | WordNet etc. | VGG-19 | 224x224 | 81.7% | ||
Krause et al. | CVPR 2015 | CaffeNet | TBD | 82.0% | |||
Multi-grained | ICCV 2015 | WordNet etc. | VGG-19 | 224x224 | 83.0% | ||
TS | CVPR 2016 | VGGD+VGGM | 448x448 | 84.0% | |||
Bilinear CNN | ICCV 2015 | VGGD+VGGM | 448x448 | 84.1% | |||
STN | NIPS 2015 | GoogLeNet+BN | 448x448 | 84.1% | |||
LRBP | CVPR 2017 | VGG-16 | 224x224 | 84.2% | |||
PDFS | CVPR 2016 | VGG-16 | TBD | 84.5% | |||
Xu et al. | ICCV 2015 | Web data | CaffeNet | 224x224 | 84.6% | ||
Cai et al. | ICCV 2017 | VGG-16 | 448x448 | 85.3% | |||
RA-CNN | CVPR 2017 | VGG-19 | 448x448 | 85.3% | |||
MaxEnt | NIPS 2018 | Bilinear CNN | TBD | 85.3% | |||
PC | ECCV 2018 | Bilinear CNN | TBD | 85.6% | |||
CVL | CVPR 2017 | Texts | VGG | TBD | 85.6% | ||
Mask-CNN | PR 2018 | VGG-16 | 448x448 | 85.7% | |||
GP-256 | ECCV 2018 | VGG-16 | 448x448 | 85.8% | |||
KP | CVPR 2017 | VGG-16 | 224x224 | 86.2% | |||
T-CNN | IJCAI 2018 | ResNet | 224x224 | 86.2% | |||
MA-CNN | ICCV 2017 | VGG-19 | 448x448 | 86.5% | |||
MaxEnt | NIPS 2018 | DenseNet-161 | TBD | 86.5% | |||
DeepKSPD | ECCV 2018 | VGG-19 | 448x448 | 86.5% | |||
OSME+MAMC | ECCV 2018 | ResNet-101 | 448x448 | 86.5% | |||
StackDRL | IJCAI 2018 | VGG-19 | 224x224 | 86.6% | |||
DFL-CNN | CVPR 2018 | VGG-16 | 448x448 | 86.7% | |||
PC | ECCV 2018 | DenseNet-161 | TBD | 86.9% | |||
KERL | IJCAI 2018 | Attributes | VGG-16 | 224x224 | 87.0% | ||
HBP | ECCV 2018 | VGG-16 | 448x448 | 87.1% | |||
Mask-CNN | PR 2018 | ResNet-50 | 448x448 | 87.3% | |||
DFL-CNN | CVPR 2018 | ResNet-50 | 448x448 | 87.4% | |||
NTS-Net | ECCV 2018 | ResNet-50 | 448x448 | 87.5% | |||
HSnet | CVPR 2017 | GoogLeNet+BN | TBD | 87.5% | |||
MetaFGNet | ECCV 2018 | Auxiliary data | ResNet-34 | TBD | 87.6% | ||
CrossX | ICCV 2019 | ResNet-50 | 448x448 | 87.7% | |||
DCL | CVPR 2019 | ResNet-50 | 448x448 | 87.8% | |||
TASN | CVPR 2019 | ResNet-50 | 448x448 | 87.9% | |||
Ge et al. | CVPR 2019 | GoogLeNet+BN | Shorter side is 800 px | 90.4% |