DROPOUTS were inserted after both of the last two layers. fully connected, just after nonlinearity; this was The original approach of the authors who proposed this type of layer¹². One of the most recent research shows. Of the advantages obtainable in using this type of layer even after first convolutions, always after nonlinearities, although with some much lower \(p\) values, around \(0.1-0.2\)¹³. The purpose of this type of layer is to regularize the network, forcing it to adapt to randomly broken connections with probability \(p\) at each forward step (and then only during training). This causes that at each training step the layer on to which the dropout is applied is seen and treated as having a Number of nodes and a different configuration than the pitch previous. The end result is to approximate, or better, simulate, the training of multiple networks with different architectures, in parallel.
In addition, a BATCH NORMALIZATION layer was inserted after the first fully connected layer to help the convergence and stability of the network, through re-centering and re-scaling the layers of input¹⁴. It was believed that this practice reduced the shift Of the internal covariance of the parameters, a problem related to to network initialization, but more recent studies show. How the reason for the increase in performance is not due to this normalization effect¹⁵. In a publication recently, it is shown that using a clipping technique of the gradient and through some tricks on the tuning of certain hyperparameters is made marginal the need for batch normalization¹⁶.

Experiments

Four different trainings were performed:

• 1-2) ResNet-18:

  • Optimizer: Adam and SGD; best model chosen.
  • Learning rate scheduled from \(1 × 10^{-3}\) to \(8 × 10^{-5}\)
  • Loss: Cross Entropy

• 3) Custom net 1:

  • Optimizer: Adam
  • Learning rate fixed at \(1 × 10^{-3}\)
  • Loss: Cross Entropy

• 4) Custom net 2:

  • Optimizer: Adam
  • Learning rate scheduled from \(1 × 10^{-3}\) to \(8 × 10^{-5}\)
  • Loss: Cross Entropy

All nets were trained for a total of 30 epochs, and this for a twofold reason: while the training time began to become almost prohibitive for a larger number of eras, On the other hand, it was seen that even as the eras (and the decrease in the learning rate in cases (1) and (3)) no benefit on the test accuracy metric. It may also have been implemented an early stopping technique, but it was deemed useful to let the training for all 30 eras complete the same scheduled, to still verify that they are not in a situation particular. In addition, better results could have been found by Due to the continuous decrease in the learning rate.

Custom network results Let us then analyze the results of this graph: as we can well see, after about 10-15 epochs the test accuracy is already that Ultimately, both for the classifier with learning rate scheduler and for the one without, and is around a 79% for the network with scheduler versus 77 percent for the network without. The difference is small, but remains detectable, however, for all training eras of the network, thus demonstrating the usefulness of this approach To optimization with progressive decrease in learning rate. Another thing to take into account is that the accuracy on the test set in both cases goes up to 99%, so completely in overfitting(?) regarding the test set. Transfer learning results with ResNet-18 Regarding the results obtained with the second architecture, and that is, the one that uses ResNet-18 as a feature extractor, the results Unfortunately, they leave much to be desired. Although even in this case a decreasing learning rate was used, the results of

accuracy almost immediately reach their maximum: accuracy on the test set is close to 48% at most, and that on the train set reaches a more stable value around 50 percent. Trying to investigate the problem, I noticed that the network was pre-trained on the dataset ImageNet^{10 17}. The latter consists of. millions of images, divided into more than 20000 categories. The theory I have so formulated is that the relatively shallow depth of the network may have caused a bottleneck problem in the power of representability of the network itself with respect to such a large amount of information, but it is only a hypothesis.
Let us now look at confusion matrix: for the network trained with architecture custom we will consider only the model that makes use of learning rate progressive, as it reports better results overall.

As we see from Fig. 3, classes are identified with a confidence more than good; given their low resolution, it is Interesting but predictable to see how the classes that come most confused with each other are “cat” with “dog,” “airplane” with “bird,” and “ship” (probably because of the often mostly blue background), and “automobile” with “truck,” also probably due to some common visual features among them, such as wheels or doors.

We see how in the example shown in Fig. 4, not all images are significantly different from each other from the visual point of view; some of them can easily be classified incorrectly even by humans¹⁸. This explains the false positives outside the diagonal of this matrix.

In the latter case, as we expected given the results of Top-1 accuracy in Fig. 2, the classifier struggles much more to Correctly identify the classes to which the various images. One notices the same confusions discussed above but much more pronounced, and outliers can also be seen: while there we might as well wait for some images of low “cat” resolution can be mistaken for images of “dog,” certainly we do not expect these “cat” images to be identified as “frog “ with a frequency that is half that of the classifications correct. On computational complexity I think it is important to mention the issue of the resources of calculation: the custom convolutional network seen in this project was Definitely not on par with the depth and performance of other networks convolutional at the top of competitions such as ILSVRC¹⁹. As we can see from Fig. 5, not necessarily more layers, and thus of parameters, automatically indicates better performance, as it could only mean more problems during the optimization. In any case, as the depth of the network increases, it will can undoubtedly achieve a greater representative capacity, something that I have not been able to experience as even just adding an additional convolutional layer (and thus bringing the total to four) already manifested the physical limitations of the computer in my possession, despite owning a graphics card with 2GB of memory dedicated video. With the configuration just seen, the use of memory is around 1.3-1.5 GB during the training phase, with a batch size of 64 images. By reducing this hyper-parameter you can reduce memory use, although, against my expectations, in rather marginally.

Also related to the ILSVRC, we can see from Fig. 6 as with ResNet there Has been a “depth revolution” of convolutional networks:

the number of layers has exploded, and the classification performance is doubled. The real revolution, however, is that all this has not brought also to an explosion of the parameters, and thus the size and complexity of the networks, but as we can see from Fig. 5 the number of parameters and the operations required for forward remained more than contained, or even decreased in some cases.

Finally, we visually explore some of the feature maps that have been extracted from the dataset. Also called “activation maps,” these indicate. The activation response of a given filter to the passage of An image in the convolutional network. In Fig. 7 we can see the responses of feature map number 6 in the various layers, and how the edge of the plane’s wings is made more and more pronounced.

In Fig. 8, we can see how the 11th feature map succeeds in removing some of the Background elements as not informative for identification of the dog class, and to maintain a high response on the contours Of the dog’s body and on the muzzle.

Conclusion

From the results, we can therefore conclude that even a network from the shallow architecture such as the custom one discussed can achieve more than satisfactory accuracy despite its simple nature. In addition, the increase in complexity and computational resources have quickly detected an obstacle to experimentation, failing to To go beyond just 3 convolutional layers.
As for the network obtained from pre-trained ResNet, unfortunately. performance is disappointing. As discussed earlier, the weights of the Not fully connected layers of this network are derived from a training on a different dataset from the one used for the rest of the experiment, namely ImageNet¹⁰. Only the last dense layer was retrained to fit the task; however, this approach has not proved productive. Nevertheless, this absolutely does not mean that the transfer technique is not useful learning; instead, it means that it should be used with more care, as e.g. with a fine tuning of the parameters instead of keeping them frozen; in other words, starting from a minimum, instead of remaining There you go toward a better minimum through multiple steps of optimization.
Finally, it is worth mentioning how the classes that put the most in difficulties the network are, of course, those that are most visually close, such as e.g. cat and dog, car and truck, and for a matter of common background, even airplane is confused a significant number of times with images of the bird and ship classes; an example is that shown in Fig. 4.
Incremental work on these results would be very easy to realize conceptually, and could include increasing the layers convolutional, the removal of one of the fully connected layers, and a Better fine tuning of parameters in the case of reuse of transfer learning. In practice, this would require resources additional hardware, given what was discussed above.

---

1. The CIFAR-10 dataset
   Alex Krizhevsky ↩︎

2. 80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
   W. T. Freeman and R. Fergus and A. Torralba ↩︎

3. convolutional neural network image classification - Google Scholar
    ↩︎

4. Very Deep Convolutional Networks for Large-Scale Image Recognition
   Karen Simonyan and Andrew Zisserman ↩︎

5. Imagenet classification with deep convolutional neural networks
   Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E ↩︎

6. Going Deeper with Convolutions
   Christian Szegedy and Wei Liu and Yangqing Jia and Pierre Sermanet and Scott Reed and Dragomir Anguelov and Dumitru Erhan and Vincent Vanhoucke and Andrew Rabinovich ↩︎

7. Deep Residual Learning for Image Recognition
   Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun ↩︎

8. Identity Mappings in Deep Residual Networks
   Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun ↩︎

9. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
   Yanping Huang and Youlong Cheng and Ankur Bapna and Orhan Firat and Mia Xu Chen and Dehao Chen and HyoukJoong Lee and Jiquan Ngiam and Quoc V. Le and Yonghui Wu and Zhifeng Chen ↩︎

10. Torchvision Models
    The Torchvision library authors and community ↩︎ ↩︎ ↩︎

11. Pattern Recognition and Machine Learning (Information Science and Statistics)
    Bishop, Christopher M. ↩︎

12. Improving neural networks by preventing co-adaptation of feature detectors
    Geoffrey E. Hinton and Nitish Srivastava and Alex Krizhevsky and Ilya Sutskever and Ruslan R. Salakhutdinov ↩︎

13. Analysis on the Dropout Effect in Convolutional Neural Networks
    Park, Sungheon and Kwak, Nojun ↩︎

14. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
    Sergey Ioffe and Christian Szegedy ↩︎

15. How Does Batch Normalization Help Optimization?
    Shibani Santurkar and Dimitris Tsipras and Andrew Ilyas and Aleksander Madry ↩︎

16. High-Performance Large-Scale Image Recognition Without Normalization
    Andrew Brock and Soham De and Samuel L. Smith and Karen Simonyan ↩︎

17. Imagenet: A large-scale hierarchical image database
    Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li ↩︎

18. After asking her to classify the pictures of the figure, a friend of mine mistakenly labeled the last two pictures as “cat” and “dog” respectively, thus reversing the classes.
     ↩︎

19. ImageNet Large Scale Visual Recognition Challenge
    Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and Alexander C. Berg and Li Fei-Fei ↩︎