Train a (vector form) : [ 0 1 0 1 0 0 0 0 0 1 0 ] Train b (vector form) : [0 1 0 1 0 0 0 0 1 0 0 ]For the first four or so elements things look good. The trains are intuitively similar and they appear to have similar vectors. But the rub occurs at elements 9 and 10. Train B points along direction 9 one whole unit, and train ten points along direction ten a whole unit and yet, intuitively, the two spike trains are very similar around these bins. A better illustration can be had if we restrict our vectors to just elements 9, 10. Then:
Train a (vector form) : [0 1] Train b (vector form) : [1 0]The Pythagorean distance between these two vectors is just
sqrt( (0-1)^2 + (1-0)^2 ) = sqrt(2). But if we just nudge the spike in train b forward a little in time (making in incremental change in the nature of train b), as soon as we push over the bin boundary, the vectors become:
Train a (vector form): [0 1] Train b (vector form): [0 1]And the distance between them goes to, obviously, zero. In other words, a small change in our data produces a large change in the vectors representing our data. This is not a desirable behavior. Particularly troubling is that if we want a "higher resolution" view of our data, we naturally make smaller bins, which just makes this problem worse. The moral: not everything is a vector and so the Pythagorean Distance is not appropriate to characterize "differences" between all types of things. This should surprise us at all. What, after all, is the vector distance between the words "cat" and "cattle"? Its obvious they are similar words (at least textually) but there is no obvious way to turn them into vectors. There are a variety of types of spaces of objects with fewer properties than vector spaces, and it makes sense to use them. Putting some piece of data into a vector space, for example, requires that addition of the data be defined in some way. It isn't even sensible to ask what "cat" + "cattle" would be in terms of another vector of characters. Maybe one day I will write some about the information theoretic ways distances can be calculated between data which don't behave well as vectors. Until then, don't go applying the Pythagorean Theorem all willy nilly.