chapter_2.6.lyx

#LyX 2.0 created this file. For more info see http://www.lyx.org/
\lyxformat 413
\begin_document
\begin_header
\textclass scrbook
\begin_preamble
\setcounter{chapter}{2}
\setcounter{section}{5}
\usepackage{graphicx}
\usepackage{pict2e}
\usepackage{graphpap}
\usepackage{color}
\usepackage{bm}
\end_preamble
\use_default_options false
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding auto
\fontencoding global
\font_roman default
\font_sans default
\font_typewriter default
\font_default_family default
\use_non_tex_fonts false
\font_sc false
\font_osf false
\font_sf_scale 100
\font_tt_scale 100

\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref false
\papersize a4paper
\use_geometry false
\use_amsmath 1
\use_esint 1
\use_mhchem 1
\use_mathdots 1
\cite_engine basic
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\use_refstyle 0
\index Index
\shortcut idx
\color #008000
\end_index
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header

\begin_body

\begin_layout Section
Principal Component Analysis (PCA)
\end_layout

\begin_layout Paragraph
Example: PCA on chromatic pixels
\end_layout

\begin_layout Standard
The first plot of Figure 
\begin_inset CommandInset ref
LatexCommand ref
reference "fig:natimg"

\end_inset

 shows the histograms of the red (R), green (G) and blue (B) channel of
 a natural image.
 We can see that the value are spread out over the whole dynamic range 
\begin_inset Formula $[0,1]$
\end_inset

.
 The other plot in Figure 
\begin_inset CommandInset ref
LatexCommand ref
reference "fig:natimg"

\end_inset

 shows the RGB values as points in 3D.
 While the histograms suggested that the whole dynamic range of the cube
 
\begin_inset Formula $[0,1]³$
\end_inset

 should be covered, we can see that this is definitely not the case.
 In fact, the RGB values are correlated and cover only a part of the color
 cube.
 The main axis of correlation is vaguely 
\begin_inset Formula $(1,1,1)^{\top}$
\end_inset

, which corresponds to the luminance of the three pixels.
 
\end_layout

\begin_layout Standard
\align right
\begin_inset Formula $\Diamond$
\end_inset


\end_layout

\begin_layout Standard
Image we wanted to encode each pixel value with a single neuron, i.e.
 we wanted to find an axis along which we would get most of the information
 about the three RGB values.
 This means we want to find a direction 
\begin_inset Formula $\mathbf{v}\in\mathbb{R}³$
\end_inset

 such that the projection 
\begin_inset Formula $v_{i}=\langle\mathbf{v},\mathbf{x}_{i}\rangle$
\end_inset

 of pixel values 
\begin_inset Formula $\mathbf{x}_{i}=(x_{i}^{red},x_{i}^{green},x_{i}^{blue})^{\top}$
\end_inset

 catches most of the information about all our pixels 
\begin_inset Formula $\mathbf{x}_{i}$
\end_inset

 in a Euclidean norm sense 
\begin_inset Formula 
\begin{eqnarray}
\mathbf{v} & = & \mbox{argmin}_{\mathbf{v}}\frac{1}{m}\sum_{i=1}^{m}||\mathbf{x}_{i}-\langle\mathbf{v},\mathbf{x}_{i}\rangle\mathbf{v}||².\label{eq:pcaObj}
\end{eqnarray}

\end_inset

Which direction should we take? Before we compute the direction from equation
 (
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:pcaObj"

\end_inset

), let us quickly spent some thought on what we would expect.
 Intuitively, most information about a color pixel is preserved when we
 convert it into a single gray value.
 However, this would mean that the axis, we are searching for, is exactly
 the axis 
\begin_inset Formula $(1,1,1)^{\top}$
\end_inset

 along which the color values have the largest variance.
 We will see in the following that this is indeed the case.
 
\end_layout

\begin_layout Standard
Before we compute our direction 
\begin_inset Formula $\mathbf{v}$
\end_inset

, we need to transform equation (
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:pcaObj"

\end_inset

) a little bit
\begin_inset Formula 
\begin{eqnarray*}
 &  & \frac{1}{m}\sum_{i=1}^{m}||\mathbf{x}_{i}-\langle\mathbf{v},\mathbf{x}_{i}\rangle\mathbf{v}||²\\
 & = & \frac{1}{m}\sum_{i=1}^{m}\langle\mathbf{x}_{i}-\langle\mathbf{v},\mathbf{x}_{i}\rangle\mathbf{v},\mathbf{x}_{i}-\langle\mathbf{v},\mathbf{x}_{i}\rangle\mathbf{v}\rangle\\
 & = & \frac{1}{m}\sum_{i=1}^{m}\left(\mathbf{x}_{i}^{\top}\mathbf{x}_{i}-2\langle\mathbf{v},\mathbf{x}_{i}\rangle^{2}+\langle\mathbf{v},\mathbf{x}_{i}\rangle^{2}\mathbf{v}^{\top}\mathbf{v}\right).
\end{eqnarray*}

\end_inset


\end_layout

\begin_layout Standard
\begin_inset Float figure
placement H
wide false
sideways false
status collapsed

\begin_layout Plain Layout
\align center
\begin_inset Graphics
	filename figs/orgImg.eps
	width 5.5cm

\end_inset


\begin_inset Graphics
	filename figs/ColorHistograms.eps
	width 5cm

\end_inset


\end_layout

\begin_layout Plain Layout
\align center
\begin_inset Graphics
	filename figs/Scatterplot.eps
	width 7cm

\end_inset


\end_layout

\begin_layout Plain Layout
\begin_inset Caption

\begin_layout Plain Layout
\begin_inset CommandInset label
LatexCommand label
name "fig:natimg"

\end_inset


\series bold
Left Top:
\series default
 Original natural image.
 
\series bold
Right Top:
\series default
 Histograms over the red, green and blue channel of a natural image.

\series bold
 Center Bottom:
\series default
Scatter plot of color pixel values in RGB space.
 The gray points on the side of the cube are the marginal scatter plots,
 i.e.
 the projections on the respective coordinate plane.
 
\end_layout

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Standard
Since we are only interested in a direction, we can assume choose our vector
 
\begin_inset Formula $\mathbf{v}$
\end_inset

 to have length one 
\begin_inset Formula $||\mathbf{v}||=1$
\end_inset

.
 In fact we already implicitly assumed this when we wrote the reconstruction
 of 
\begin_inset Formula $\mathbf{x}_{i}$
\end_inset

 through 
\begin_inset Formula $\mathbf{v}$
\end_inset

 as 
\begin_inset Formula $\langle\mathbf{v},\mathbf{x}_{i}\rangle\mathbf{v}$
\end_inset

.
 With 
\begin_inset Formula $||\mathbf{v}||=1$
\end_inset

, the equation becomes 
\begin_inset Formula 
\begin{eqnarray*}
 &  & \frac{1}{m}\sum_{i=1}^{m}\left(\mathbf{x}_{i}^{\top}\mathbf{x}_{i}-2\langle\mathbf{v},\mathbf{x}_{i}\rangle^{2}+\langle\mathbf{v},\mathbf{x}_{i}\rangle^{2}\mathbf{v}^{\top}\mathbf{v}\right)\\
 & = & \frac{1}{m}\sum_{i=1}^{m}\left(\mathbf{x}_{i}^{\top}\mathbf{x}_{i}-2\langle\mathbf{v},\mathbf{x}_{i}\rangle^{2}+\langle\mathbf{v},\mathbf{x}_{i}\rangle^{2}\right)\\
 & = & \frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}^{\top}\mathbf{x}_{i}-\frac{1}{m}\sum_{i=1}^{m}\langle\mathbf{v},\mathbf{x}_{i}\rangle^{2}.
\end{eqnarray*}

\end_inset

Note that if the 
\begin_inset Formula $\mathbf{x}_{i}$
\end_inset

 had mean zero, i.e.
 
\begin_inset Formula $\frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}=0$
\end_inset

, then 
\begin_inset Formula $\frac{1}{m}\sum_{i=1}^{m}\langle\mathbf{v},\mathbf{x}_{i}\rangle^{2}$
\end_inset

 would be the variance of the projections of the 
\begin_inset Formula $\mathbf{x}_{i}$
\end_inset

 onto our direction 
\begin_inset Formula $\mathbf{v}$
\end_inset

.
 We can further rewrite this term into 
\begin_inset Formula 
\begin{eqnarray*}
\frac{1}{m}\sum_{i=1}^{m}\langle\mathbf{v},\mathbf{x}_{i}\rangle^{2} & = & \frac{1}{m}\sum_{i=1}^{m}\langle\mathbf{v},\mathbf{x}_{i}\rangle\langle\mathbf{x}_{i},\mathbf{v}\rangle\\
 & = & \frac{1}{m}\sum_{i=1}^{m}\mathbf{v}^{\top}\mathbf{x}_{i}\cdot\mathbf{x}_{i}^{\top}\mathbf{v}\\
 & = & \mathbf{v}^{\top}\left(\frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}\cdot\mathbf{x}_{i}^{\top}\right)\mathbf{v}.
\end{eqnarray*}

\end_inset

Again, for mean zero 
\begin_inset Formula $\mathbf{x}_{i}$
\end_inset

, 
\begin_inset Formula $\frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}\cdot\mathbf{x}_{i}^{\top}$
\end_inset

 would be the covariance matrix of the 
\series bold

\begin_inset Formula $\mathbf{x}_{i}$
\end_inset


\series default
.
\end_layout

\begin_layout Standard
Now, let us turn to the question how we compute 
\begin_inset Formula $\mathbf{v}$
\end_inset

.
 After all our transformations we are left with the problem 
\begin_inset Formula 
\begin{eqnarray*}
\mbox{minimize}_{\mathbf{v}} &  & \frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}^{\top}\mathbf{x}_{i}-\mathbf{v}^{\top}\left(\frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}\cdot\mathbf{x}_{i}^{\top}\right)\mathbf{v}\\
 &  & \mbox{s.t. }||\mathbf{v}||²=1.
\end{eqnarray*}

\end_inset

Note that the first term 
\begin_inset Formula $\frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}^{\top}\mathbf{x}_{i}$
\end_inset

 does not depend on 
\begin_inset Formula $\mathbf{v}$
\end_inset

 at all, so we can drop it from the optimization.
 Furthermore, note that minimizing 
\begin_inset Formula $-\mathbf{v}^{\top}\left(\frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}\cdot\mathbf{x}_{i}^{\top}\right)\mathbf{v}$
\end_inset

 is the same as maximizing 
\begin_inset Formula $\mathbf{v}^{\top}\left(\frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}\cdot\mathbf{x}_{i}^{\top}\right)\mathbf{v}$
\end_inset

.
 Denoting 
\begin_inset Formula $\mathbf{C}=\frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}\cdot\mathbf{x}_{i}^{\top}$
\end_inset

 we are therefore left with the final version of the problem 
\begin_inset Formula 
\begin{eqnarray*}
\mbox{maximize}_{\mathbf{v}} &  & \mathbf{v}^{\top}\mathbf{C}\mathbf{v}\\
 &  & \mbox{s.t. }||\mathbf{v}||²=1.
\end{eqnarray*}

\end_inset

The straightforward way for computing 
\begin_inset Formula $\mathbf{v}$
\end_inset

 would be to compute the derivative, set it to zero and solve for 
\begin_inset Formula $\mathbf{v}$
\end_inset

.
 However that does not tell us how we can deal with the constraint 
\begin_inset Formula $||\mathbf{v}||=1$
\end_inset

.
 In order to take it into account we need a method called 
\emph on
Lagrange multipliers
\begin_inset Index idx
status collapsed

\begin_layout Plain Layout
Lagrange multipliers
\end_layout

\end_inset

.
 
\emph default
The Lagrange multiplier method is a way of solving exactly those problems
 above.
 We will not formally introduce the method here, but only give a rough intuition
 for the idea behind it.
 
\end_layout

\begin_layout Standard
Remember that the gradient of a function always points into the direction
 of maximal ascent.
 Looking at the constraint 
\begin_inset Formula $||\mathbf{v}||²=1$
\end_inset

 or, equivalently, 
\begin_inset Formula $||\mathbf{v}||²-1=0$
\end_inset

 we can see that it describes a contour line of the function 
\begin_inset Formula $||\mathbf{v}||²-1$
\end_inset

.
 Intuitively, since function is constant along the contour line, the gradient
 of 
\begin_inset Formula $||\mathbf{v}||²-1$
\end_inset

 will be orthogonal to that contour line.
 Optimizing the function under the constraint now means searching for the
 optimum along the contour line.
 
\end_layout

\begin_layout Standard
We now state the condition for optimality and then spent some though on
 why that makes sense.
 The condition for optimality under the constraint can be written as 
\begin_inset Formula 
\begin{eqnarray*}
\nabla\left(\mathbf{v}^{\top}\mathbf{C}\mathbf{v}\right) & = & \lambda\nabla(||\mathbf{v}||²-1).
\end{eqnarray*}

\end_inset

The condition means that at the optimum, the gradient of the constraint
 must have the same orientation as the gradient of the function.
 Why does that make sense? First, imagine that the optimum of 
\begin_inset Formula $\mathbf{v}^{\top}\mathbf{C}\mathbf{v}$
\end_inset

 without lies on the contour line.
 Then, the gradient 
\begin_inset Formula $\nabla\left(\mathbf{v}^{\top}\mathbf{C}\mathbf{v}\right)$
\end_inset

 is zero and we simply set 
\begin_inset Formula $\lambda=0$
\end_inset

 and the condition is fulfilled.
 Now, assume that the optimum of 
\begin_inset Formula $\mathbf{v}^{\top}\mathbf{C}\mathbf{v}$
\end_inset

 is not on the contour line.
 Then, at some point, the only way to improve the function 
\begin_inset Formula $\mathbf{v}^{\top}\mathbf{C}\mathbf{v}$
\end_inset

 is to leave the constraint line, which is equivalent of saying that 
\begin_inset Formula $\nabla\left(\mathbf{v}^{\top}\mathbf{C}\mathbf{v}\right)$
\end_inset

 is orthogonal to the constraint line.
 However, we noted before that 
\begin_inset Formula $\nabla(||\mathbf{v}||²-1)$
\end_inset

 is always orthogonal to the constraint line.
 This means that the two gradients have the same orientation and the condition
 is fulfilled as well.
 
\end_layout

\begin_layout Standard
Finally, note that we can write the condition for optimality as
\begin_inset Formula 
\begin{eqnarray*}
\mbox{maximize}_{\mathbf{v}} &  & \mathbf{v}^{\top}\mathbf{C}\mathbf{v}-\lambda\left(||\mathbf{v}||²-1\right)
\end{eqnarray*}

\end_inset


\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\noun off
\color none
 because taking the derivative of 
\begin_inset Formula $\mathbf{v}^{\top}\mathbf{C}\mathbf{v}-\lambda\left(||\mathbf{v}||²-1\right)$
\end_inset

 and setting it to zero exactly yields 
\begin_inset Formula $\nabla\left(\mathbf{v}^{\top}\mathbf{C}\mathbf{v}\right)=\lambda\nabla(||\mathbf{v}||²-1).$
\end_inset

 The nice feature of the optimization problem above is now, that it does
 not have constraints and we can simply solve it by setting the derivative
 to zero and solve for 
\begin_inset Formula $\mathbf{v}$
\end_inset

.
 In theory we also needed to compute the derivative with respect to lambda
 and solve for it.
 However, in this case there is a simple way.
 
\end_layout

\begin_layout Standard
The derivative with respect to 
\begin_inset Formula $\mathbf{v}$
\end_inset

 is given by
\begin_inset Formula 
\begin{eqnarray*}
\frac{\partial}{\partial\mathbf{v}}\left(\mathbf{v}^{\top}\mathbf{C}\mathbf{v}-\lambda\left(||\mathbf{v}||²-1\right)\right) & = & \frac{\partial}{\partial\mathbf{v}}\left(\mathbf{v}^{\top}\mathbf{C}\mathbf{v}-\lambda\mathbf{v}^{\top}\mathbf{v}-\lambda\right)\\
 & = & \frac{\partial}{\partial\mathbf{v}}\mathbf{v}^{\top}\mathbf{C}\mathbf{v}-\lambda\frac{\partial}{\partial\mathbf{v}}\mathbf{v}^{\top}\mathbf{v}\\
 & = & 2\mathbf{Cv}-2\lambda\mathbf{v}.
\end{eqnarray*}

\end_inset

Setting it to zero yields 
\begin_inset Formula 
\begin{eqnarray*}
2\mathbf{Cv}-2\lambda\mathbf{v} & = & 0\\
\Leftrightarrow\mathbf{Cv} & = & \lambda\mathbf{v}.
\end{eqnarray*}

\end_inset

This is exactly the condition for 
\begin_inset Formula $\mathbf{v}$
\end_inset

 being an eigenvector corresponding to the largest eigenvalue 
\begin_inset Formula $\lambda$
\end_inset

 of 
\begin_inset Formula $\mathbf{C}$
\end_inset

.
 Luckily there are many algorithms that carry out this computation for us.
 For example, we can use the command 
\family typewriter
eig 
\family default
in matlab.
 If we assume again that the mean of the 
\begin_inset Formula $\mathbf{x}_{i}$
\end_inset

 is zero, we can even compute the variance of 
\begin_inset Formula $\langle\mathbf{x}_{i},\mathbf{v}\rangle$
\end_inset

:
\begin_inset Formula 
\begin{eqnarray*}
\frac{1}{m}\sum_{i=1}^{m}\langle\mathbf{x}_{i},\mathbf{v}\rangle^{2} & = & \mathbf{v}^{\top}\left(\frac{1}{m}\sum_{i=1}^{m}\mathbf{x}_{i}\cdot\mathbf{x}_{i}^{\top}\right)\mathbf{v}\\
 & = & \mathbf{v}^{\top}\mathbf{C}\mathbf{v}\\
 & = & \lambda\mathbf{v}^{\top}\mathbf{v}\\
 & = & \lambda.
\end{eqnarray*}

\end_inset

Since 
\begin_inset Formula $\lambda$
\end_inset

 is the maximal eigenvalue of 
\begin_inset Formula $\mathbf{C}$
\end_inset

, for mean zero signals finding the most informative 
\begin_inset Formula $\mathbf{v}$
\end_inset

 is the same as finding the direction of maximal variance.
 This is exactly what 
\emph on

\begin_inset Index idx
status collapsed

\begin_layout Plain Layout
Principle Component Analysis (PCA)
\end_layout

\end_inset

Principle Component Analysis (PCA)
\emph default
 does.
 If the mean of the 
\begin_inset Formula $\mathbf{x}_{i}$
\end_inset

 is not zero, then, in general, 
\begin_inset Formula $\mathbf{v}$
\end_inset

 will not be the direction of maximal variance but also point in the direction
 of the mean.
 That is the reason why the data is usually centered, i.e.
 the mean is subtracted, before computing PCA.
 
\end_layout

\begin_layout Standard
In its complete version, the PCA algorithm usually computes all eigenvectors
 instead of simply the largest one.
 As we saw in the last chapter about eigenvalues, symmetric matrices like
 
\begin_inset Formula $\mathbf{C}$
\end_inset

 have eigenvectors that are mutually orthogonal to each other.
 In terms of PCA this means that after computing the direction of the largest
 variance, we compute the direction of largest variance which is orthogonal
 to the first one and so on.
 
\end_layout

\begin_layout Paragraph
Example: PCA on chromatic pixels (con'd)
\end_layout

\begin_layout Standard
Figure 
\begin_inset CommandInset ref
LatexCommand ref
reference "fig:PCAimg"

\end_inset

 shows the centered pixels and the pixels that have been transformed into
 the PCA basis.
 For visualization we also transformed the RGB color planes along with the
 data.
 Looking at the right plot of Figure 
\begin_inset CommandInset ref
LatexCommand ref
reference "fig:PCAimg"

\end_inset

 shows that the first principle component (now the 
\begin_inset Formula $x$
\end_inset

-axis) indeed corresponds to the luminance axis.
 Furthermore, we can observe that the second and third principle component
 (now the 
\begin_inset Formula $y$
\end_inset

- and the 
\begin_inset Formula $z$
\end_inset

-axis, respectively) roughly corresponds to the blue-yellow and the red-green
 channel, respectively.
 This is not an coincidence.
 In fact, this finding is very stable of many natural images.
 
\end_layout

\begin_layout Standard
\begin_inset Float figure
placement H
wide false
sideways false
status open

\begin_layout Plain Layout
\begin_inset Graphics
	filename figs/Mean0Scatter.eps
	width 6cm

\end_inset


\begin_inset Graphics
	filename figs/PCAbase.eps
	width 6cm

\end_inset


\end_layout

\begin_layout Plain Layout
\begin_inset Caption

\begin_layout Plain Layout
\begin_inset CommandInset label
LatexCommand label
name "fig:PCAimg"

\end_inset


\series bold
Left:
\series default
 Scatter plot of centered chromatic natural image pixels.
 
\series bold
Right:
\series default
 Scatter plot of color pixel values in PCA space.
 We can see that the 
\begin_inset Formula $y$
\end_inset

-axis in the PCA basis roughly corresponds to the blue-yellow channel whereas
 the 
\begin_inset Formula $z$
\end_inset

-axis roughly corresponds to the red-green channel.
 Both channels are also found in the early visual system.
 
\begin_inset Newline newline
\end_inset

In both plots the color planes have been shifted with the pixels.
\end_layout

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Standard
Interestingly, the same color opponencies are also found in the early visual
 system.
 What could be the advantage of those channels? PCA can give as an answer
 to that.
 Assume that 
\begin_inset Formula $\mathbf{C}$
\end_inset

 is the covariance matrix (i.e.
 the 
\begin_inset Formula $\mathbf{x}_{i}$
\end_inset

 have mean zero).
 Remember from the last chapter, that we can write a symmetric matrix in
 terms of its eigenbasis as follows 
\begin_inset Formula $\mathbf{C}=\mathbf{V}\Lambda\mathbf{V}^{\top}$
\end_inset

, where the columns of 
\begin_inset Formula $\mathbf{V}$
\end_inset

 correspond to the eigenvectors and 
\begin_inset Formula $\Lambda$
\end_inset

 is a diagonal matrix that contains the eigenvalues.
 Now, we can ask the question how 
\begin_inset Formula $\mathbf{C}$
\end_inset

 looks like in the basis 
\begin_inset Formula $\mathbf{V}$
\end_inset

.
 A quick computation shows that the transformation that has to be applied
 is 
\begin_inset Formula $\mathbf{C}\mapsto\mathbf{V}^{\top}\mathbf{C}\mathbf{V}$
\end_inset

.
 However, since 
\begin_inset Formula $\mathbf{C}=\mathbf{V}\Lambda\mathbf{V}^{\top}$
\end_inset

 and 
\begin_inset Formula $\mathbf{V}^{\top}\mathbf{V}=\mathbf{V}\mathbf{V}^{\top}=\mathbf{I}$
\end_inset

 we know the answer: In the PCA basis 
\begin_inset Formula $\mathbf{V}^{\top}\mathbf{C}\mathbf{V}=\Lambda$
\end_inset

, i.e.
 the covariance matrix is diagonal.
 This means that all off-diagonal terms are zero and, therefore, the signals
 are uncorrelated.
 How could the visual system profit from uncorrelated signals? 
\end_layout

\begin_layout Standard
\begin_inset Float figure
placement H
wide false
sideways false
status open

\begin_layout Plain Layout
\align center
\begin_inset Graphics
	filename figs/pc1.eps
	width 4cm

\end_inset


\begin_inset Graphics
	filename figs/pc2.eps
	width 4cm

\end_inset


\begin_inset Graphics
	filename figs/pc3.eps
	width 4cm

\end_inset


\end_layout

\begin_layout Plain Layout
\begin_inset Caption

\begin_layout Plain Layout
\begin_inset CommandInset label
LatexCommand label
name "fig:PCs"

\end_inset

Image only represented by the first (left), second (middle) and third (right)
 principal component in color space.
\end_layout

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Standard
There are basically two reasons.
 Imagine, we use three neurons to signal the value of each color pixel,
 either on neuron for each RGB channel or one neuron for each principle
 component.
 Now we make two reasonable assumption about our neurons: Firstly, our neurons
 have limited expressive power.
 This means that they can only signal a finite number of pixel values.
 Secondly, our neuron has limited capacity.
 This means it can only transmit a limited amount of information in a given
 time window.
 For both cases, signalling in the PCA basis is favorable for the neuron.
 Why is that the case? We already saw, that the histograms in the RGB space
 covered the full dynamic range 
\begin_inset Formula $[0,1]$
\end_inset

.
 A neuron, which has limited expressive power, must distribute the pixel
 values, that it wants to transmit over the whole dynamic range for all
 three color channels in RGB space.
 This means that the resolution is very coarse.
 In the PCA basis however, we can see from 
\begin_inset CommandInset ref
LatexCommand ref
reference "fig:PCAimg"

\end_inset

 that the histograms for the blue-yellow and the red-green channel will
 be much more compressed.
 This means that the neurons responsible for those channels, can spend their
 finite amount of states on that range, thereby obtaining a finer resolution.
 
\end_layout

\begin_layout Standard
The PCA basis is also more favorable in terms of capacity.
 If a neuron can only transmit a finite amount of information in a given
 time window, it is not favorable if the three different signals are correlated
 or, in other words, redundant (ignoring noise issues for the moment).
 The reason is simply that by transmitting two correlated signals, each
 signal also transmits some information about the other signal.
 Since our neurons can only signal a limited amount of information in a
 given time window, the total amount of information that can be transmitted
 becomes less when the signals are correlated.
 However, as we have already seen above, signals are uncorrelated in the
 PCA basis.
 This means that signalling in that representation is also favorable in
 terms of capacity.
 
\end_layout

\begin_layout Standard
\align right
\begin_inset Formula $\Diamond$
\end_inset


\end_layout

\end_body
\end_document