0.1 Optimizing logistic regression for a particle physics application  (Page 3/4)

Most of the wide variety of classifiers performed within an order of magnitude of each other, except for one called “Hyperpipes" which performed two orders of magnitude better than any of the others. This struck Dr. Padley as very strange as he had never heard of the Hyperpipes algorithm before.

Current work - elec 631 fall 2011

One interesting aspect of the behavior of the Hyperpipes algorithm on the dataset was the very low true positive rate: less than 10% of the t-tbar events were correctly classified as such. However, the false positive rate was much, much lower – this is what accounted for its good performance. At this point the author became suspicious that the default parameters for the WEKA machine learning algorithms were poorly suited to this application and that some of the other algorithms could also demonstrate greatly improved performance with a little tuning. Since logistic regression is relatively simple, and since the author now understands a little about how the algorithm works from the on-line Stanford course, the decision was made to try and tune it to this application.

Cost matrix

The first stage of tuning involved adjusting the cost matrix. This followed from the observation that when training a classifier, we really want to penalize false positives much more harshly than false negatives. Sacrificing up to 50% of our true positives is acceptable if it leads to a very large decrease in background false positive events and thus an improved signal-to-noise ratio. Keeping the true positive rate relatively high is still important as t-tbar events are rare enough that keeping enough for statistical significance is still a problem, as we shall see a bit later.

A variety of cost matrices were created and the logistic regression classifier in WEKA was trained on a training data set with each one of them. The resulting classification models were then tested on a cross-validation data set. Both the training data set and the cross-validation set had 10,000 of each type of event as before. The results are summarized in Table TODO. Note that as expected, increasing the cost of false positives relative to other kinds of errors greatly improved the signal to noise ratio with an acceptable decrease in the true positive rate. Note that the cost matrices are properly scaled by WEKA: only the relative costs of the different types of errors represented by the cost matrix matter. Interestingly, eliminating the costs associated with misclassifying W background events as QCD background events and vice-versa seems to have no effect on the signal-to-noise ratio.

Choosing a cost matrix which penalizes each type of false positive 300 times more than other types of errors seemed to give good results without leading to diminishing returns in the form of decreased true positives. Penalizing false positives caused by QCD events more than W events didn't improve the results on the large test set.

Ratio of signal to background in training set

Dr. Devika Subramanian of the Rice Computer Science department suggested that classifiers need to be trained on data that has each type of event to be classified in a ratio that is roughly the same as that in the real test data. This suggestion indicated that the previous strategy of training classifiers with an equal ratio of each type of event was unsound. Unfortunately, generating training and test sets with the correct ratio of event types is quite difficult because of the huge backgrounds that must be generated, in this case several million QCD events for every t-tbar event. Generating enough background for 100 t-tbar events would take a modern cpu core over 1000 days to compute enough background.