What should be accuracy of deep learning model while changing the SNR from [-20,-10,0,10,20,30]?

I am using the following link to generate the dataset on different SNR. I am not good in signal domain.

https://www.mathworks.com/help/deeplearning/ug/modulation-classification-with-deep-learning.html

I have read the literature that shows when the we increase the SNR the accurancy should increase as well. but in my case there's opposite. i have trained the Resnet-50 model on different SNR from [-20,-10,0,10,20,30] and then test the model on saparate SNR. My accuracy is below

SNR Accuracy

SNR -20dB 99.99

SNR -10dB 99.84

SNR 0dB 99.93

SNR 10dB 99.97

SNR 20dB 99.70

SNR 30dB 97.46

should i wonder about it? could the problem is my dataset?

Please assist

Answer 1

[Vaibhav]

Hi John

It is my understanding that you are facing issues with the decrease in accuracy as the SNR increases during the testing of Resnet-50 on specific SNR values.

Higher signal-to-noise ratios (SNR) generally correlate with increased model accuracy, but a decline in accuracy at very high SNR values is not uncommon. This could be attributed to potential overfitting, where the model becomes less adaptable to new data.

In this scenario, the observed decrease in Resnet-50 accuracy at high SNR values may stem from overfitting or issues related to dataset quality.

To investigate further, consider the following steps:

1. Validate the Dataset: Ensure the dataset is balanced and representative. Consider augmentation for increased diversity.
2. Model Regularization: Apply dropout and weight decay to prevent overfitting and enhance generalization.
3. Hyperparameter Tuning: Experiment with adjusting the learning rate, batch size, and number of epochs to optimize accuracy.
4. Diverse Metrics: Use precision, recall, or F1 score beyond accuracy, especially for imbalanced classes.
5. Testing Set Diversity: Ensure a varied testing set with representative SNR levels. Uneven distribution can impact accuracy.

You can refer to the following MathWorks documentation links to know more about "regularization" and "overfitting":

https://www.mathworks.com/discovery/regularization.html?s_tid=srchtitle_site_search_1_regularization

https://www.mathworks.com/discovery/overfitting.html?s_tid=srchtitle_site_search_1_overfitting

Hope this helps!