0.1 A matrix times a vector

A matrix times a vector

In this chapter we consider the first problem posed in the introduction

where the matrix $\mathbf{A}$ and vector $\mathbf{x}$ are given and we want to interpret and give structure to the calculation of the vector $\mathbf{b}$ . Equation [link] has a variety of special cases. The matrix $\mathbf{A}$ may be square or may be rectangular. It may have full column or row rank or it may not. It may be symmetric or orthogonalor non-singular or many other characteristics which would be interesting properties as an operator. If we view the vectors as signals and thematrix as an operator or processor, there are two interesting interpretations.

• The operation [link] is a change of basis or coordinates for a fixed signal. The signal stays the same, the basis (or frame) changes.
• The operation [link] alters the characteristics of the signal (processes it) but within a fixed basis system. The basis stays the same, the signalchanges.

An example of the first would be the discrete Fourier transform (DFT) where one calculates frequency components of a signal which arecoordinates in a frequency space for a given signal. The definition of the DFT from [link] can be written as a matrix-vector operation by $\mathbf{c}=\mathbf{Wx}$ which, for $w=e^{-j2\pi /N}$ and $N=4$ , is

An example of the second might be convolution where you are processing or filtering asignal and staying in the same space or coordinate system.

A particularly powerful sequence of operations is to first change the basis for a signal, then process the signal in this new basis, and finally returnto the original basis. For example, the discrete Fourier transform (DFT) of a signal is taken followed by setting some of the Fourier coefficients tozero followed by taking the inverse DFT.

Another application of [link] is made in linear regression where the input signals are rows of $\mathbf{A}$ and the unknown weights of the hypothesisare in $\mathbf{x}$ and the outputs are the elements of $\mathbf{b}$ .

Change of basis

Consider the two views:

1. The operation given in [link] can be viewed as $\mathbf{x}$ being a set of weights so that $\mathbf{b}$ is a weighted sum of the columns of $\mathbf{A}$ . In other words, $\mathbf{b}$ will lie in the space spanned by the columns of $\mathbf{A}$ at a location determined by $\mathbf{x}$ . This view is a composition of a signal from a set of weights as in [link] and [link] below. If the vector ${\mathbf{a}}_{\mathbf{i}}$ is the $i^{th}$ column of $\mathbf{A}$ , it is illustrated by
   $$\mathbf{A}\mathbf{x}=x_1\left[\begin{array}{c}⋮\\ {\mathbf{a}}_{\mathbf{1}}\\ ⋮\end{array}\right]+x_2\left[\begin{array}{c}⋮\\ {\mathbf{a}}_{\mathbf{2}}\\ ⋮\end{array}\right]+x_3\left[\begin{array}{c}⋮\\ {\mathbf{a}}_{\mathbf{3}}\\ ⋮\end{array}\right]=\mathbf{b}.$$
2. An alternative view has $\mathbf{x}$ being a signal vector and with $\mathbf{b}$ being a vector whose entries are inner products of $\mathbf{x}$ and the rows of A . In other words, the elements of $\mathbf{b}$ are the projection coefficients of $\mathbf{x}$ onto the coordinates given by the rows of A . The multiplication of a signal by this operator decomposes the signal and gives thecoefficients of the decomposition. If ${\overline{\mathbf{a}}}_{\mathbf{j}}$ is the $j^{th}$ row of $\mathbf{A}$ we have:
   $$b_1=\left[\begin{array}{c}\cdots {\overline{\mathbf{a}}}_{\mathbf{1}}\cdots \end{array}\right]\left[\begin{array}{c}⋮\\ \mathbf{x}\\ ⋮\end{array}\right]b_2=\left[\begin{array}{c}\cdots {\overline{\mathbf{a}}}_{\mathbf{2}}\cdots \end{array}\right]\left[\begin{array}{c}⋮\\ \mathbf{x}\\ ⋮\end{array}\right]etc.$$
   Regression can be posed from this view with the input signal being the rows of A .