2.2 Machine learning lecture 3 course notes  (Page 8/12)

Rather than applying SVMs using the original input attributes $x$ , we may instead want to learn using some features $\Phi \left(x\right)$ . To do so, we simply need to go over our previous algorithm, and replace $x$ everywhere in it with $\Phi \left(x\right)$ .

Since the algorithm can be written entirely in terms of the inner products $⟨x,z⟩$ , this means that we would replace all those inner products with $⟨\Phi \left(x\right),\Phi \left(z\right)⟩$ . Specificically, given a feature mapping $\Phi$ , we define the corresponding Kernel to be

Then, everywhere we previously had $⟨x,z⟩$ in our algorithm, we could simply replace it with $K(x,z)$ , and our algorithm would now be learning using the features $\Phi$ .

Now, given $\Phi$ , we could easily compute $K(x,z)$ by finding $\Phi \left(x\right)$ and $\Phi \left(z\right)$ and taking their inner product. But what's more interesting is that often, $K(x,z)$ may be very inexpensive to calculate, even though $\Phi \left(x\right)$ itself may be very expensive to calculate (perhaps because it is an extremely high dimensional vector). In such settings, by using in our algorithman efficient way to calculate $K(x,z)$ , we can get SVMs to learn in the high dimensional feature space space given by $\Phi$ , but without ever having to explicitly find or represent vectors $\Phi \left(x\right)$ .

Let's see an example. Suppose $x,z\in {\mathbb{R}}^n$ , and consider

We can also write this as

Thus, we see that $K(x,z)=\Phi {\left(x\right)}^T\Phi \left(z\right)$ , where the feature mapping $\Phi$ is given (shown here for the case of $n=3$ ) by

Note that whereas calculating the high-dimensional $\Phi \left(x\right)$ requires $O\left(n^2\right)$ time, finding $K(x,z)$ takes only $O\left(n\right)$ time—linear in the dimension of the input attributes.

For a related kernel, also consider

(Check this yourself.) This corresponds to the feature mapping (again shown for $n=3$ )

and the parameter $c$ controls the relative weighting between the $x_i$ (first order) and the $x_i{x}_j$ (second order) terms.

More broadly, the kernel $K(x,z)={(x^{T}z+c)}^d$ corresponds to a feature mapping to an $\left({}_d^{n+d}\right)$ feature space, corresponding of all monomials of the form $x_{i_1}{x}_{i_2}...x_{i_k}$ that are up to order $d$ . However, despite working in this $O\left(n^d\right)$ -dimensional space, computing $K(x,z)$ still takes only $O\left(n\right)$ time, and hence we never need to explicitly represent feature vectors in this very high dimensionalfeature space.

Now, let's talk about a slightly different view of kernels. Intuitively, (and there are things wrong with this intuition, but nevermind),if $\Phi \left(x\right)$ and $\Phi \left(z\right)$ are close together, then we might expect $K(x,z)=\Phi {\left(x\right)}^T\Phi \left(z\right)$ to be large. Conversely, if $\Phi \left(x\right)$ and $\Phi \left(z\right)$ are far apart—say nearly orthogonal to each other—then $K(x,z)=\Phi {\left(x\right)}^T\Phi \left(z\right)$ will be small. So, we can think of $K(x,z)$ as some measurement of how similar are $\Phi \left(x\right)$ and $\Phi \left(z\right)$ , or of how similar are $x$ and $z$ .

Given this intuition, suppose that for some learning problem that you're working on, you've come up with some function $K(x,z)$ that you think might be a reasonable measure of how similar $x$ and $z$ are. For instance, perhaps you chose