3.6 Possible improvements and additional projects

Main faults and possible improvements

While this program is fairly effective, it is not perfect. Below is a detailed list of all the problems with each of the four major blocks of code, along with methods for possibly improving these faults.

Problems with detecting note onsets

• If a song is played fast enough, note hits will be missed, especially if they are very close to another note hit.
• There is no way of telling when a note is released. Thus, if there were rests in the middle of a song, it would likely be viewed as the note immediately before it being held for the length of the rest. Attempting to use the negative peaks of the filtered song to find note offsets might fix this problem, but note offsets are much less defined than the onsets.
• This only detects when a note was played, not what note was played. So if one note is being held for a long time, while shorter notes are being played, the long note appears identical to that note being played multiple times.
• Sometimes the location of the note hit is not precisely where the note hit occurred. By looking at Figure 4 in the Detecting Note Onsets , you can see that at times, the note depression happens just before the sharp rise, rather than on it.

Problems with finding frequencies

• Pianos differ widely with their frequency spectra. Our thresholds were tailored to give good results for a certain piano, but songs played on different pianos did not work nearly as well.
• The harmony and bass lines of a piano piece are usually played much softer than the melody. Therefore, they don't show up in the frequency domain very strongly and are often below the threshold we have set. Our algorithm does an excellent job of picking out the notes in the melody, though.
• Notes can only be detected when the starts of the notes coincide. If a note is being held with faster notes being played on top of it, this algorithm will determine that the note being held is played repeatedly with every note on top of it. It is very difficult to look at the frequency spectrum to fix this.
• Sometimes, the previous note played will still be resonating into the current note. The code will usually pick that up as another note that is being played. Using a Hamming window or another non-rectangular window could fix this problem.
• Piano harmonics are not perfect integer multiples of the fundamental frequency due to small deviations from the ideal. Furthermore, the higher notes of a piano are tuned with the high harmonics of the lower notes, instead of being tuned to the mathematically determined frequency. This results in several missed harmonics.

Problems with note decoding

• If a piano is tuned differently than expected, some notes may decode wrong. Analyzing the piece as a whole to see if frequencies are always just higher or just lower than expected may help allow for the tuning of that individual piano.
• Only a musical system based on 12 tones can be used with this implementation. Arrangements could be made to allow this program to deal with alternate systems of music.
• Our program currently has no way of finding the key of the piece. If such an algorithm were to be created, this is the most likely place to implement it.

Problems with finding note lengths

• This method assumes that the song is played at a constant tempo. It does not account for any kind of speeding up or slowing down. One way to address this is to look through the song for a noticeable change in the note duration distribution with respect to time. If one is found, then the song can be divided into two parts and this method can be used separately on each part.
• This method relies on a lengthy song to accurately determine the tempo and note lengths. Very short pieces or segments of pieces are likely to yield an incorrect tempo.
• This assumes that the most common note is the quarter note, or the beat. This may not be correct for different time signatures, and analysis of which notes carry the emphasis might yield a clue as to the actual note lengths and tempo, as well as the time signature.
• The output of this function is a matrix representation of sheet music. It could actually generate standard sheet music with some sort of program designed to print it.

Additional projects

One additional method of approaching this problem that we looked at but did not pursue was to look at the frequency content of the recording first, followed by the time content. This method is discussed below.

Start by taking the song, and run a band-pass filter through it that is centered around a known note frequency, such as 440 Hz. Make sure this band pass filter has a narrow enough band that it does not pass any note frequency other than the center frequency of the band. When we attempted this, it left the majority of the song at around 0, while the portion of the song in which 440 Hz was played contained the expected sharp note hit and decay. At this point it is very easy to find when the note was first played and somewhat easier to find when the note was released. Repeating this process for all 88 possible notes should yield a very accurate description of when each note was played.

The downsides of the method above are numerous. The most obvious one is that this method would take 88 times longer than our original method. Considering that our original method takes 13 seconds for a 5 second recording, the new method would take almost 20 minutes for the same recording. A full length song (around 4 minutes) would take 16 hours. The next issue is the creation of these band-pass filters. For very low frequencies, the difference in frequency between adjacent notes is very small. For the lowest frequency, 27.5 Hz, the nearest note to it is 29.14 Hz. To make a filter with enough frequency resolution to distinguish between these two frequencies requires a filter length on the order of two-thirds of a second long. Most notes are much shorter than this, considering that a quarter note at 120 beats per minute is only half a second long. This would likely cause the shape of the filtered signal to be very different from the expected sharp onset and exponential decay. In addition, this method does little to aid the problem of determining which notes are harmonics and which are actual notes played.

An alteration of this method would include filtering a note and all of its harmonics at the same time. This would reduce the amount of time needed to run the algorithm significantly. However, it does not help with the necessity of very long filters; in some cases, the filter may actually need to be longer. An additional issue this raises is that multiple filters will have the same pass-bands. For example, a frequency such as 1318.5 Hz (E6) is a harmonic of both 440 Hz (A4) and 659 Hz (E5), but A4 and E5 are not harmonics of each other. Thus, if A4 were played, it might also pick up E5 being played faintly as a result.