Performance Analysis of Neural Networks With Backpropagation on Binary and Multi-Class Data Classification

Abdul Tahir; Irdam  Irdam; Sirama   Sirama

Authors

Abdul Tahir Akademi Teknik Soroako
Irdam Akademi Teknik Soroako
Sirama Sirama Akademi Teknik Soroako

Keywords:

Neurons, Classification, Optimization, Accuracy, Regularization

Abstract

Neural networks represent a widely adopted paradigm within the domain of machine learning, employed for a multitude of classification endeavors, encompassing image recognition and natural language processing. This investigation seeks to elucidate the influence of varying neuron quantities in hidden layers on the efficacy of neural networks in both binary and multi-class classification endeavors. The research utilizes a dataset procured from images depicting characters and digits, which were transformed into binary format via a thresholding methodology. The neural network architectures comprise one and two hidden layers, which are trained employing the backpropagation algorithm in conjunction with the Adam optimizer. The evaluation of the models is conducted through metrics such as accuracy, loss curves, and confusion matrices. Findings reveal that the configuration featuring two hidden layers with 40 sampai 99 neurons achieves the pinnacle accuracy of 99.64 percent alongside optimal loss stability. Furthermore, models incorporating a single hidden layer exhibited commendable accuracy, thereby indicating that a reduced number of neurons can proficiently encapsulate data complexity in less demanding tasks. This research underscores the criticality of selecting suitable neural network configurations contingent upon data complexity and classification objectives, while advocating for further investigation into regularization strategies to enhance performance.

References

A. Di Piazza, M. C. Di Piazza, G. La Tona, and M. Luna, “An artificial neural network-based forecasting model of energy-related time series for electrical grid management,” Math Comput Simul, vol. 184, pp. 294-305, Jun. 2021, doi: 10.1016/j.matcom.2020.05.010.

E. Ayyildiz, M. Erdogan, and A. Taskin, “Forecasting COVID-19 recovered cases with Artificial Neural Networks to enable designing an effective blood supply chain,” Comput Biol Med, vol. 139, p. 105029, Dec. 2021, doi: 10.1016/j.compbiomed.2021.105029.

S. R. Dubey, S. K. Singh, and B. B. Chaudhuri, “Activation functions in deep learning: A comprehensive survey and benchmark,” Neurocomputing, vol. 503, pp. 92-108, Sep. 2022, doi: 10.1016/j.neucom.2022.06.111.

Y. Luo, H.-H. Tseng, S. Cui, L. Wei, R. K. Ten Haken, and I. El Naqa, “Balancing accuracy and interpretability of machine learning approaches for radiation treatment outcomes modeling,” BJR|Open, vol. 1, no. 1, Jul. 2019, doi: 10.1259/bjro.20190021.

M. V. Narkhede, P. P. Bartakke, and M. S. Sutaone, “A review on weight initialization strategies for neural networks,” Artif Intell Rev, vol. 55, no. 1, pp. 291–322, Jan. 2022, doi: 10.1007/s10462-021-10033-z.

R. Abdulkadirov, P. Lyakhov, and N. Nagornov, “Survey of Optimization Algorithms in Modern Neural Networks,” Mathematics, vol. 11, no. 11, p. 2466, May 2023, doi: 10.3390/math11112466.

M. Dampfhoffer, T. Mesquida, A. Valentian, and L. Anghel, “Backpropagation-Based Learning Techniques for Deep Spiking Neural Networks: A Survey,” IEEE Trans Neural Netw Learn Syst, vol. 35, no. 9, pp. 11906-11921, Sep. 2024, doi: 10.1109/TNNLS.2023.3263008.

S. Chakravarty, M. H. Tanveer, R. C. Voicu, M. Banerjee, and G. Mahdi, “Backpropagation Techniques in SNN and Application in Image Segmentation,” in SoutheastCon 2024, IEEE, Mar. 2024, pp. 1475-1481. doi: 10.1109/SoutheastCon52093.2024.10500286.

T. Liu, C. Zhang, and D. Li, “Reducing Overfitting In Deep Neural Networks By Intra-class Decorrelation,” in 2023 International Seminar on Computer Science and Engineering Technology (SCSET), IEEE, Apr. 2023, pp. 200–203. doi: 10.1109/SCSET58950.2023.00052.

S.-H. Lyu, L. Wang, and Z.-H. Zhou, “Improving generalization of deep neural networks by leveraging margin distribution,” Neural Networks, vol. 151, pp. 48-60, Jul. 2022, doi: 10.1016/j.neunet.2022.03.019.

C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning (still) requires rethinking generalization,” Commun ACM, vol. 64, no. 3, pp. 107-115, Mar. 2021, doi: 10.1145/3446776.

Q. Chen, W. Hao, and J. He, “A weight initialization based on the linear product structure for neural networks,” Appl Math Comput, vol. 415, p. 126722, Feb. 2022, doi: 10.1016/j.amc.2021.126722.

Shem L. Gonzales, “Enhancing Image Classification Performance: A Comparative Analysis of Optimization Algorithms,” International Journal of Advanced Research in Science, Communication and Technology, pp. 641-645, Jun. 2023, doi: 10.48175/IJARSCT-11918.

H. T. Sihotang, M. Albert, F. Riandari, and L. Rendell, “Efficient optimization algorithms for various machine learning tasks, including classification, regression, and clustering,” Idea: Future Research, vol. 1, no. 1, pp. 14-24, Jan. 2023, doi: 10.35335/idea.v1i1.3.

D. Elhani, A. C. Megherbi, A. Zitouni, F. Dornaika, S. Sbaa, and A. Taleb-Ahmed, “Optimizing convolutional neural networks architecture using a modified particle swarm optimization for image classification,” Expert Syst Appl, vol. 229, p. 120411, Nov. 2023, doi: 10.1016/j.eswa.2023.120411.

L. Abualigah, K. H. Almotairi, and M. A. Elaziz, “Multilevel thresholding image segmentation using meta-heuristic optimization algorithms: comparative analysis, open challenges and new trends,” Applied Intelligence, vol. 53, no. 10, pp. 11654-11704, May 2023, doi: 10.1007/s10489-022-04064-4.

Y. Tian, “Artificial Intelligence Image Recognition Method Based on Convolutional Neural Network Algorithm,” IEEE Access, vol. 8, pp. 125731-125744, 2020, doi: 10.1109/ACCESS.2020.3006097.

F. E. Sadeq and Z. T. M. Al-Ta’i, “Comparison Between Face and Gait Human Recognition Using Enhanced Convolutional Neural Network,” Journal of Applied Engineering and Technological Science (JAETS), vol. 5, no. 1, pp. 18-30, Dec. 2023, doi: 10.37385/jaets.v5i1.2806.

D. AL Kafaf, N. N. Thamir, and S. S. AL-Hadithy, “Malaria Disease Prediction Based on Convolutional Neural Networks,” Journal of Applied Engineering and Technological Science (JAETS), vol. 5, no. 2, pp. 1165-1181, Jun. 2024, doi: 10.37385/jaets.v5i2.3947.

A. Purnomo and H. Tjandrasa, “IMPROVED DEEP LEARNING ARCHITECTURE WITH BATCH NORMALIZATION FOR EEG SIGNAL PROCESSING,” JUTI: Jurnal Ilmiah Teknologi Informasi, vol. 19, no. 1, p. 19, Jan. 2021, doi: 10.12962/j24068535.v19i1.a1023.

A. Akhgar, D. Toghraie, N. Sina, and M. Afrand, “Developing dissimilar artificial neural networks (ANNs) to prediction the thermal conductivity of MWCNT-TiO2/Water-ethylene glycol hybrid nanofluid,” Powder Technol, vol. 355, pp. 602–610, Oct. 2019, doi: 10.1016/j.powtec.2019.07.086.

C. Singh, “Machine Learning in Pattern Recognition,” European Journal of Engineering and Technology Research, vol. 8, no. 2, pp. 63-68, Apr. 2023, doi: 10.24018/ejeng.2023.8.2.3025.

M. Dialameh, A. Hamzeh, H. Rahmani, S. Dialameh, and H. J. Kwon, “DL-Reg: A deep learning regularization technique using linear regression,” Expert Syst Appl, vol. 247, p. 123182, Aug. 2024, doi: 10.1016/j.eswa.2024.123182.

P. Freire, E. Manuylovich, J. E. Prilepsky, and S. K. Turitsyn, “Artificial neural networks for photonic applications-from algorithms to implementation: tutorial,” Adv Opt Photonics, vol. 15, no. 3, p. 739, Sep. 2023, doi: 10.1364/AOP.484119.

A. Shafiq, A. Batur Colak, T. Naz Sindhu, S. Ahmad Lone, A. Alsubie, and F. Jarad, “Comparative study of artificial neural network versus parametric method in COVID-19 data analysis,” Results Phys, vol. 38, p. 105613, Jul. 2022, doi: 10.1016/j.rinp.2022.105613.

Z. Zhou, C. Qiu, and Y. Zhang, “A comparative analysis of linear regression, neural networks and random forest regression for predicting air ozone employing soft sensor models,” Sci Rep, vol. 13, no. 1, p. 22420, Dec. 2023, doi: 10.1038/s41598-023-49899-0.