基于神威高性能计算机的大规模有噪声图像训练系统

刘 锐 ( 北京航空航天大学计算机学院 )

王 锐 ( 北京航空航天大学计算机学院 )

周彧聪 ( 北京航空航天大学计算机学院 )

刘 轶 ( 北京航空航天大学计算机学院 )

https://doi.org/10.37155/2717-5170-0402-43

Abstract

在深度学习应用中,大量且准确标注的训练数据是保证模型准确性的关键因素之一。在图像识别应用 中,为减少人工标注工作量,常常采用从互联网抓取大量图像并自动标注的方法。这带来两方面的问题:首先,引入 的错误标注数据将影响整体训练效果,其次,图像数量众多导致训练所需计算量庞大。本文在神威高性能计算机上设 计实现了一种基于Caffe的大规模有噪声图像训练系统,该系统采用多进程并行和数据预取方法以充分发挥申威众核处 理器和神威高性能计算机的性能优势,进而提升神经网络模型的训练速度,同时采用一种校准训练方法以降低噪声数 据给模型训练带来的不利影响。实验数据表明,该系统可在保证训练效果的前提下,大幅减少训练所需时间,同时具 有良好的性能可扩展性。

Keywords

图像识别;高性能计算;分布式训练;噪声标签

Full Text

PDF

References

[1] Li J, Socher R, Hoi S C H. DivideMix: Learning with
Noisy Labels as Semi-supervised Learning[C]//International
Conference on Learning Representations. 2019.
[2] Vinyals O, Blundell C, Lillicrap T, et al. Matching
networks for
one shot learning[C]//Advances in neural information
processing systems, 2016, pages 3630–3638.
[3] Chen T, Kornblith S, Norouzi M, et al. A simple
framework for contrastive learning of visual representations[C]//
International conference on machine learning. PMLR, 2020:
1597-1607.
[4] Xiao T, Xia T, Yang Y, et al. Learning from massive
noisy labeled data for image classification[C]//Proceedings
of the IEEE conference on computer vision and pattern
recognition. 2015: 2691-2699.
[5] Krizhevsky A, Sutskever I, Hinton G E. Imagenet
classification with deep convolutional neural networks[C]//
Advances in neural information processing systems, 2012, 25.
[6] Murphy J. Deep learning benchmarks of NVIDIA
tesla P100 PCIe, tesla K80, and tesla M40 GPUs[J]. 2017.
[7] Jia Y, Shelhamer E, Donahue J, et al. Caffe:
Convolutional architecture for fast feature embedding[ C]
/ / Proceedings ofthe 22nd ACM international conference on
Multimedia, 2014: 675-678.
[8] Li L, Fang J, Fu H, et al. swcaffe: A parallel
framework for accelerating deep learning applications on
sunway taihulight[C]//IEEE International Conference on
Cluster Computing (CLUSTER). IEEE, 2018: 413-422.
[9] Liu R, Liu Y, Wang R, et al. Mutual calibration
training: Training deep neural networks with noisy
labels using dual-models[J]. Computer Vision and Image
Understanding, 2021, 212: 103277.
[10] C. Zhang, S. Bengio, M. Hardt, B. Recht, O.
Vinyals, Understanding deep learning requires rethinking
generalization, arXiv preprint arXiv:1611.03530 (2016).
[11] Algan G, Ulusoy I. Image classification with
deep learning in the presence of noisy labels: A survey[J].
Knowledge-Based Systems, 2021, 215: 106771.
[12] Yi K, Wu J. Probabilistic end-to-end noise
correction for learning with noisy labels[C]//Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern
Recognition. 2019: 7017-7025.
[13] Jindal I, Nokleby M, Chen X. Learning
d e e p n e t w o r k s f r o m n o i s y l a b e l s w i t h d r o p o u t
regularization[C]//2016 IEEE 16th International Conference
on Data Mining (ICDM). IEEE, 2016: 967-972.
[14] Patrini G, Rozza A, Krishna Menon A, et al. Making
deep neural networks robust to label noise: A loss correction
approach[C]//Proceedings of the IEEE conference on
computer vision and pattern recognition. 2017: 1944-1952.
[15] Hinton G, Vinyals O, Dean J. Distilling the
knowledge in a neural network (2015)[J]. arXiv preprint
arXiv:1503.02531, 2015, 2.
[16] Li Y, Yang J, Song Y, et al. Learning from
noisy labels with distillation[C]//Proceedings of the IEEE
International Conference on Computer Vision. 2017: 1910-
1918.
[17] Tanaka D, Ikami D, Yamasaki T, et al. Joint
optimization framework for learning with noisy labels[C]//
Proceedings of the IEEE conference on computer vision andpattern recognition. 2018: 5552-5560.
[18] W. Li, L. Wang, W. Li, E. et al, Webvision database:
Visual learning and understanding from web data, arXiv
preprint arXiv:1708.02862 (2017).
[19] Fu H, Liao J, Yang J, et al. The Sunway TaihuLight
supercomputer: system and applications[J]. Science China
Information Sciences, 2016, 59(7): 1-16.
[20] Dean J, Corrado G, Monga R, et al. Large scale
distributed deep networks[C]//Advances in neural information
processing systems, 2012, 25.
[21] Raina R, Madhavan A, Ng A Y. Large-scale
deep unsupervised learning using graphics processors[C]//
Proceedings of the 26th annual international conference on
machine learning. 2009: 873-880.
[22] Lopes N, Ribeiro B. GPUMLib: An efficient opensource GPU machine learning library[J]. International Journal
of Computer Information Systems and Industrial Management
Applications, 2011, 3: 355-362.
[23] Gibiansky, A. Bringing hpc techniques to deep
learning. Technical report, Baidu Research, Tech. Rep., 2017,
http://research.baidu.com/bringing-hpc-techniques-deeplearning/. Bingjing Zhang TESTS & CERTIFICATIONS IBM
Certified Database Associate-DB2 Universal Database (2017)
[24] Lee K H, He X, Zhang L, et al. Cleannet: Transfer
learning for scalable image classifier training with label
noise[C]//Proceedings of the IEEE conference on computer
vision and pattern recognition. 2018: 5447-5456.

Copyright © 2022 刘 锐,王 锐,周彧聪,刘 轶 Creative Commons License Publishing time:2022-04-29
This work is licensed under a Creative Commons Attribution 4.0 International License