Marques, Jose and Falcao, Gabriel and Alexandre, Luís A. (2018) Distributed Learning of CNNs on Heterogeneous CPU/GPU Architectures. Applied Artificial Intelligence, 32 (9-10). pp. 822-844. ISSN 0883-9514
Distributed Learning of CNNs on Heterogeneous CPU GPU Architectures.pdf - Published Version
Download (2MB)
Abstract
The convolutional neural networks (CNNs) have proven to be powerful classification tools in tasks that range from check reading to medical diagnosis, reaching close to human perception, and in some cases surpassing it. However, the problems to solve are becoming larger and more complex, which translates to larger CNNs, leading to longer training times that not even the adoption of Graphics Processing Units (GPUs) could keep up to. This problem is partially solved by using more processing units and distributed training methods that are offered by several frameworks dedicated to neural network training, such as Caffe, Torch, or TensorFlow. However, these techniques do not take full advantage of the possible parallelization offered by CNNs and the cooperative use of heterogeneous devices with different processing capabilities, clock speeds, memory size, among others. This paper presents a new method for the parallel training of CNNs where only the convolutional layer is distributed. The paper analyzes the influence of network size, bandwidth, batch size, number of devices, including their processing capabilities, and other parameters. Results show that this technique is capable of diminishing the training time without affecting the classification performance for both CPUs and GPUs. For the CIFAR-10 dataset, using a CNN with two convolutional layers, and 500
and 1500
kernels, respectively, best speedups achieve 3.28×
using four CPUs and 2.45×
with three GPUs. Larger datasets will certainly require more than 60
-90
% of processing time calculating convolutions, and speedups will tend to increase accordingly.
Item Type: | Article |
---|---|
Subjects: | Archive Paper Guardians > Computer Science |
Depositing User: | Unnamed user with email support@archive.paperguardians.com |
Date Deposited: | 27 Jun 2023 07:04 |
Last Modified: | 06 Dec 2023 04:31 |
URI: | http://archives.articleproms.com/id/eprint/1364 |