DISCUSSION
In evaluating the performance of our Convolutional Neural Network (CNN) model, two critical factors emerge as potential contributors to its suboptimal results: data imbalance and model architecture. Before delving into these specific aspects, it is essential to understand their overarching impact on the model’s functionality. Both data imbalance and model architecture play pivotal roles in determining the efficacy and accuracy of machine learning models, particularly in complex applications such as image recognition or medical diagnosis. The interplay of these factors can significantly influence how well a model learns, generalizes, and performs on new, unseen data. In the following sections, we will explore in detail why data imbalance and model architecture are major factors influencing our CNN model’s performance and how they manifest in the results observed.
1. DATA IMBALANCE AND CLASS WEIGHT
In assessing the performance of the Convolutional Neural Network (CNN) model, data imbalance emerges as a potential key factor contributing to suboptimal results [1]. This possibility is particularly evident when examining the confusion matrix (Figure 2), which reveals a high number of False Negatives (102) compared to True Positives (75) [2]. Such a discrepancy suggests an increased rate of false negatives and reduced sensitivity in detecting abnormal cases, a critical concern in the context of medical imaging analysis.
Furthermore, the recall rate, lower than desired, reflects this diminished sensitivity. While the model’s accuracy on the test set stands at about 48.49%, this figure may be misleading due to the imbalance in the dataset [3]. With a greater number of normal cases (1660 normal subtracted mammograms) compared to abnormal cases (1324 abnormal subtracted mammograms), a model predominantly predicting ‘normal’ results could attain an ostensibly decent accuracy. This scenario masks the model’s ineffectiveness in accurately identifying abnormal cases, a crucial capability for reliable medical diagnosis.
Additionally, the fluctuating and generally high loss values observed during both training and validation phases, along with a moderate F1 score of 0.476, point to challenges in effectively evaluating and fine-tuning the model [4]. These metrics suggest that the model may not be capturing the complexity of the data adequately, leading to difficulties in accurately assessing its performance, particularly for the minority class, which in this case are the abnormal instances.
Moreover, the significant epoch-to-epoch variations in loss and accuracy, and particularly the large swings in validation loss, highlight potential instability in the training process [3]. For instance, the validation loss exhibited a wide range from values like 106.2919 to 5.2842. Such variability could indicate convergence issues, possibly stemming from the model’s complexity, learning rate, or other hyperparameters not being optimally configured for the specific characteristics of the dataset.
In conclusion, the data imbalance in the dataset appears to be a significant factor affecting the CNN model’s ability to perform optimally. This imbalance skews the model’s predictive accuracy, especially in detecting less frequent but clinically significant abnormal cases. Addressing this imbalance, possibly through techniques such as data augmentation or re-sampling, along with optimizing model hyperparameters, could be crucial steps towards enhancing the model’s performance and reliability.
2. MODEL ARCHITECTURE AND COMPLEXITY
Figures In the realm of machine learning, particularly in the context of Convolutional Neural Network (CNN) models, the architecture and complexity of the model play crucial roles in determining its effectiveness and efficiency. For our CNN model, it appears that its architecture and complexity may be key contributors to the observed suboptimal results.
Deep and complex models, characterized by a multitude of parameters, are inherently susceptible to overfitting. This tendency is especially pronounced when the available training data lacks diversity or is limited in quantity. Overfitting is a phenomenon where a model, rather than generalizing from observed patterns, learns the training data too thoroughly, encompassing its noise and outliers [4]. This issue manifests as impressive performance on training data but deteriorates significantly when the model encounters unseen data, as evident in the validation and test sets.
In our model’s case, signs of overfitting are noticeable. Referring to Figures 3 and 4, there is a discernible discrepancy in performance, with the model exhibiting high accuracy or low loss on the training set but faltering significantly on the validation and test sets. This gap between training and validation performance is a classic indicator of overfitting [5].
Moreover, the complexity of the model might impede its ability to generalize to new, unseen data. This limitation arises from the model’s excessive focus on the specifics of the training data, thereby hampering its predictive accuracy on data beyond its training set [6]. One of the tell-tale signs of this issue is the large fluctuations in validation loss, which in our model ranged from 106.2919 to 5.2842, as shown in Figure 3. Such variability suggests that the model, owing to its complexity, struggles to find a stable and generalizable pattern.
Additionally, the unusually high loss values observed during training and, more critically, during validation phases further corroborate this point. For instance, the model registered loss values of 106.29 in the first epoch and escalated to 170.52 by the fourth epoch (Figure 3). These figures imply that the model is grappling with convergence issues, potentially attributable to its inherent complexity [6].
In light of these observations, it becomes apparent that the architecture and complexity of our CNN model are likely contributing to its suboptimal performance. Addressing these aspects by possibly simplifying the model or enhancing its ability to generalize could lead to more reliable and robust performance, particularly when dealing with diverse and previously unseen data sets.
FUTURE WORK RECOMMENDATIONS
Looking ahead, there are several strategic approaches that can be undertaken to enhance the performance and reliability of our model. Firstly, efforts should be made to expand our dataset. If feasible, collecting a larger volume of data can significantly improve the model’s learning and predictive capabilities [7]. To this end, consider forming collaborations with other organizations or institutions to access more diverse and comprehensive datasets. Such partnerships can provide a broader range of data, encompassing varied scenarios and conditions, which is crucial for the development of a robust model.
Additionally, consider adjusting and refining the model architecture. This task may require the use of high-end computing resources to effectively handle the complexities and computational demands of advanced models. Utilizing more powerful computing systems can enable more sophisticated model architectures and potentially lead to enhanced performance.
Lastly, it is crucial to engage with medical professionals in the development process. Partnering with experts in the relevant medical fields can provide invaluable insights into the clinical relevance of the model’s predictions. Their feedback can guide the fine-tuning of the model to ensure it meets the practical requirements and nuances of medical diagnostics. Such collaboration not only improves the model’s accuracy but also its applicability and trustworthiness in real-world clinical settings.
LIMITATIONS
This report acknowledges several key limitations that have impacted the development and evaluation of the Convolutional Neural Network (CNN) model:
1. TIME CONSTRAINT
One of the primary limitations faced during this project was the time constraint. Constructing a sophisticated CNN model within a three-month timeframe proved to be quite challenging. This time limitation restricted the scope of exploratory research and iterative development that is typically necessary for fine-tuning such models. In machine learning, especially with complex models like CNNs, an extended period of development allows for comprehensive testing and refinement, which was constrained in this case.
2. DATA AVAILABILITY
The effectiveness of a CNN model is heavily reliant on the quantity and quality of the training data. In this project, the availability of data was limited, which posed a significant challenge. Having access to a larger and more diverse dataset could have substantially enhanced the model’s learning capacity and its ability to generalize from the training data to real-world scenarios [7]. The limited dataset not only affected the training process but also constrained the model’s ability to validate and test across a wider range of data inputs.
3. COMPUTATIONAL RESOURCES
The computational capacity of the equipment used, in this case, a personal laptop, also presented a limitation. The laptop’s inability to process high-resolution images efficiently and to construct deeper and more complex CNN models or to implement transfer learning techniques restricted the potential depth and sophistication of the model. Advanced machine learning models often require high-end computational resources for optimal performance, especially when handling large datasets and complex architectures. The lack of such resources in this project was a significant hindrance to achieving a more advanced and accurate model.
CONCLUSION
This report has presented a comprehensive analysis of the Convolutional Neural Network (CNN) model, developed over a period of 50 epochs, with a focus on key performance metrics such as loss, accuracy, precision, recall, and the ROC curve. Through this evaluation, the model demonstrated notable fluctuations in these metrics, highlighting areas for improvement.
The analysis identified data imbalance and model architecture as the primary factors contributing to the model’s suboptimal performance. Data imbalance, evidenced by a higher rate of false negatives and lower recall, suggests challenges in the model’s ability to accurately detect abnormal cases, crucial in medical diagnostic applications. The fluctuating and high loss values, along with a moderate F1 score, further point to difficulties in model evaluation and fine-tuning, particularly for minority classes.
Moreover, the model’s architecture and complexity have been found to potentially contribute to overfitting, as indicated by the disparity in performance between training and unseen data. This issue, along with large swings in validation loss and unusually high loss values, suggests that the model struggles with convergence and generalization, likely due to its complexity.
The report also acknowledges key limitations that have impacted the project, including time constraints, data availability, and computational resources. These limitations have undeniably influenced the model’s development and potential efficacy.
In conclusion, while the CNN model shows promise, its current performance necessitates careful re-evaluation and strategic modifications. Addressing data imbalance and optimizing model architecture are essential steps towards enhancing the model’s accuracy and generalizability. Future efforts should also focus on expanding the dataset, leveraging high-end computational resources, and incorporating feedback from medical professionals to ensure the model’s relevance and applicability in clinical settings. Despite the challenges faced, the insights gained from this analysis provide valuable guidance for future enhancements and applications of the CNN model in complex domains such as medical image analysis.
ACKNOWLEDGMENT
We extend our sincere gratitude to Khaled et al. for their invaluable contribution to the field of breast cancer research. We specifically want to acknowledge the use of their dataset [8]. This dataset has been instrumental in the development and evaluation of our Convolutional Neural Network (CNN) model. The accessibility of such comprehensive and detailed data has significantly enriched our research, enabling us to undertake a thorough and robust analysis. Their work represents a substantial contribution to the advancement of medical imaging and cancer detection, and we are grateful for their efforts in facilitating this important field of study.
- Luque, A., Carrasco, A., Martín, A., & de las Heras, A. (2019). The impact of class imbalance in classification performance metrics based on the binary confusion matrix. Pattern Recognition, 91, 216–231. https://doi.org/10.1016/j.patcog.2019.02.023
- Thabtah, F., Hammoud, S., Kamalov, F., & Gonsalves, A. (2020). Data imbalance in classification: Experimental evaluation. Information Sciences, 513, 429–441. https://doi.org/10.1016/j.ins.2019.11.004
- Jeni, L. A., Cohn, J. F., & De La Torre, F. (2013). Facing Imbalanced Data–Recommendations for the Use of Performance Metrics. 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. https://doi.org/10.1109/acii.2013.47
- Garbin, C., Zhu, X., & Marques, O. (2020). Dropout vs. batch normalization: an empirical study of their impact to deep learning. Multimedia Tools and Applications, 79(19-20), 12777–12815. https://doi.org/10.1007/s11042-019-08453-9
- Pham, H. N. A., & Triantaphyllou, E. (2008). The Impact of Overfitting and Overgeneralization on the Classification Accuracy in Data Mining. Soft Computing for Knowledge Discovery and Data Mining, 391–431. https://doi.org/10.1007/978-0-387-69935-6_16
- Guberman, N. (2016). On Complex Valued Convolutional Neural Networks. ArXiv Preprint. arxiv:1602.09046
- Luo, C., Li, X., Wang, L., He, J., Li, D., & Zhou, J. (2018, November 1). How Does the Data set Affect CNN-based Image Classification Performance? IEEE Xplore. https://doi.org/10.1109/ICSAI.2018.8599448
- Khaled, R., Helal, M., Alfarghal, O., Mokhtar, O., Elkorany, A., El Kassas, H., & Fahmy, A. (2021). Categorized Digital Database for Low energy and Subtracted Contrast Enhanced Spectral Mammography images [Dataset]. The Cancer Imaging Archive. https://doi.org/10.7937/29kw-ae92
