Hey there,

Let us have a beautiful story.

This parable is told of a farmer who owned an old mule. The mule fell into the farmer’s well. The farmer heard the mule praying or whatever mules do when they fall into wells.

After carefully assessing the situation, the farmer sympathized with the mule, but decided that neither the mule nor the well was worth the trouble of saving. Instead, he called his neighbors together, told them what had happened, and enlisted them to help haul dirt to bury the old mule in the well and put him out of his misery.

Initially the old mule was hysterical! But as the farmer and his neighbors continued shoveling and the dirt hit his back, a thought struck him. It suddenly dawned on him that every time a shovel load of dirt landed on his back, HE WOULD SHAKE IT OFF AND STEP UP!

This he did, blow after blow. “Shake it off and step up… shake it off and step up… shake it off and step up!” He repeated to encourage himself. No matter how painful the blows, or how distressing the situation seemed, the old mule fought panic and just kept right on SHAKING IT OFF AND STEPPING UP!

It wasn’t long before the old mule, battered and exhausted, stepped triumphantly over the wall of that well! What seemed like it would bury him actually helped him … all because of the manner in which he handled his adversity.

THAT’S LIFE! If we face our problems and respond to them positively, and refuse to give in to panic, bitterness, or self-pity.

Main Content From here

In this lesson, we'll learn about dropout, which is another technique that can also help us avoid overfitting.

• We learned that during training and neural network adjusts its weights and biases to minimize the loss function.
• One problem that can sometimes occur during training is that one part of the network ends up with very large weights, while other parts end up with very small weights.

This results in the parts having the larger weights playing a major role in the training process, while the other parts with smaller weights won't play much of a role and they don't get trained very much.

One way to avoid this from happening, is to perform dropout.

• Dropout consists of randomly turning off some of the neurons in the network during training.
• By turning off some neurons during training, this forces the other neurons to pick up the slack and to take a more active part in the training.
• So as we go through the epics during training, we'll randomly turn off some neurons.

For example, let's suppose in the first epoch we choose not to use these two neurons.

So we do our feedforward and our backpropagation passes without using these neurons. The second epoc we choose not to use these three neurons.

Again, we do feed forward and back-propagation passes without using those neurons, and then the third epic which is not to use these two neurons.

Again, we do feed forward and back-propagation passes without using the neurons and so on. By training our neural network in this way, we can avoid overfitting.

• You can say that the network becomes more resistant, because it cannot rely on all of the neurons being there to solve the problem.
• Consequently, other neurons will have to step up and also be able to do the job.
• In practice, we specify the probability that each neuron gets dropped at each training epic.
• Notice, that since we specify a probability, some neurons may get turned off more than others and some may never get turned off at all.
• However, this is not a problem, because we're doing it many many times over.
• Therefore on average, each neuron will get the same chance of being turned off.

Other Techniques to Prevent Overfitting

In this lesson we saw three different techniques to prevent overfitting:

• Early Stopping: In this method, we track the loss on the validation set during the training phase and use it to determine when to stop training such that the model is accurate but not overfitting.
• Image Augmentation: Artificially boosting the number of images in our training set by applying random image transformations to the existing images in the training set.
• Dropout: Removing a random selection of a fixed number of neurons in a neural network during training. However, these are not the only techniques available to prevent overfitting.

You can read more about these and other techniques in the link below: Memorizing is not learning! — 6 tricks to prevent overfitting in machine learning

Summary In this lesson we learned how Convolutional Neural Networks work with color images and saw various techniques that we can use to avoid overfitting . The main key points of this lesson are:

CNNs with RGB Images of Different Sizes:

• Resizing: When working with images of different sizes, you must resize all the images to the same size so that they can be fed into a CNN.
• Color Images: Computers interpret color images as 3D arrays.
• RGB Image: Color image composed of 3 color channels: Red, Green, and Blue.
• Convolutions: When working with RGB images we convolve each color channel with its own convolutional filter. Convolutions on each color channel are performed in the same way as with grayscale images, i.e. by performing element-wise multiplication of the convolutional filter (kernel) and a section of the input array. The result of each convolution is added up together with a bias value to get the convoluted output.
• Max Pooling: When working with RGB images we perform max pooling on each color channel using the same window size and stride. Max pooling on each color channel is performed in the same way as with grayscale images, i.e. by selecting the max value in each window.
• Validation Set: We use a validation set to check how the model is doing during the training phase. Validation sets can be used to perform Early Stopping to prevent overfitting and can also be used to help us compare different models and choose the best one.

Methods to Prevent Overfitting:

• Early Stopping: In this method, we track the loss on the validation set during the training phase and use it to determine when to stop training such that the model is accurate but not overfitting.
• Image Augmentation: Artificially boosting the number of images in our training set by applying random image transformations to the existing images in the training set.
• Dropout: Removing a random selection of a fixed number of neurons in a neural network during training.