0.5 Results

Results

In actually constructing our network, we cycled through each of the activation functions and cost functions finding the combinations that resulted in the greatest accuracy. We found using ReLU with a log-likelihood cost function in conjunction with a softmax output layer gave us the best results, so that is the structure we used in our network. Another component we played with was the learning rate. Each cost function created a different gradient, so we had to be careful in how quickly we traversed that gradient. When we noted a network that did not learn well we concluded that we were likely caught in a local minimum of the gradient and we had to lower the learning rate in our subsequent networks to combat this.

Another interesting takeaway was the observation that many of the kernels produced by the convolutional layers resemble prototypical image processing filters like edge detectors in various orientations. This observation is particularly interesting because it leads to additional discussion about whether it would be possible to manipulate these kernels somehow.