Deep Learning Image Detection - Help needed deciphering machine learning loss and accuracy graph and finding solutions to fix model

I have an imbalanced dataset from Google OpenImages of 6 classes

Train (starfish=439; Dolphin = 890; Turtle = 1362; Fish = 6216; Jellyfish = 733; Shellfish = 1141 )

Validation (starfish=20; Dolphin = 61; Turtle = 6; Fish = 370; Jellyfish = 38; Shellfish = 42 )

Test (starfish=72; Dolphin = 139; Turtle = 12; Fish = 1028; Jellyfish = 105; Shellfish = 115 )

I have 4 different PyVision pre-trained models that I am using. I am using data augmentation and have tried adjusting the weights manually to compensate for the imbalance. I finally got my models to run on Google Colab 100 epochs, batch=8.

optimizer = torch.optim.SGD(params, lr=0.05, momentum=0.9, weight_decay=0.0005)

lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)

These are my graphs from the training of the pretrained model on my image set.

torchvision.models.detection.fasterrcnn_mobilenet_v3_large_320_fpn enter image description here

My Interpretation:

  • Loss: not training and not learning
  • Accuracies: Underrepresented

torchvision.models.detection.fasterrcnn_mobilenet_v3_large_fpn enter image description here

My Interpretation:

  • Loss: not training and not learning
  • Accuracies: Underrepresented

torchvision.models.detection.fasterrcnn_resnet50_fpn

TBD - assume similar graph - model running

torchvision.models.detection.FasterRCNN

TBD - assume similar graph - model running

QUESTIONS:

Are my interpretations correct? What more does this say and what other graph would you recommend that would tell me what to do to fix this?

What would you do to fix this?

Would you create extra images to pad the uneven classes? i.e. for the Train dataset, make all the classes equal the same amount as the largest class. Then run it again?

I can't think of anything else. Using pretrained models is supposed to make things better and overcome imbalances, as well as data augmentation, and designating the weights for the imbalanced classes.
I used weight = [9.36, 1.48, 12.87, 13.98, 21.18, 86.83] for classes = ['Jellyfish', 'Fish', 'Dolphin', 'Starfish', 'Shellfish', 'Turtle'].

Did I do that wrong?

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum