> For the complete documentation index, see [llms.txt](https://huang-jason.gitbook.io/deep/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://huang-jason.gitbook.io/deep/handsonml-8.-dimension-reduction.md).

# HandsonML 8. Dimension Reduction

## **1.** Problems of millions of features

1. slow training 
2. easy overfitting

#### 1.1. Solution = dimension reduction

1. bad: information loss
2. bad: complex pipeline
3. good: speed up training
4. good: data visualization (2D image or clustering)

{% hint style="info" %}
No guarantee to filter noise, unnecessary details & better result.
{% endhint %}

#### 1.2. DR techniques

{% hint style="info" %}

1. Projection
2. Manifold learning
3. PCA
4. Maintain distance (LLE, tSNE)
   {% endhint %}

#### 1.3. High dimension v.s. low dimension

|                           | Low dimension  | High dimension  |
| ------------------------- | -------------- | --------------- |
| on border                 | 0.4% on border | 99.9% on border |
| distance between 2 points | 0.52 for 2-D   | 408 for 1M-D    |

{% hint style="info" %}
Large dimension dataset: 

(1) sparse: instances are far away from each other

(2) risk of overfitting.
{% endhint %}

## 2. Projection

{% hint style="info" %}
**(1)** Training instances lie on lower-dimension subspace of the high-dimension space. 

(2) Problem: could squash data when distribution is twisted.
{% endhint %}

{% tabs %}
{% tab title="projection" %}
![](https://4199572293-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lq83Pz1Ot3xU1lv1xHK%2F-LwV1I8k-iqYYX6ddZeD%2F-LwV1dUqg4_iMD54fwa4%2Fimage.png?alt=media\&token=70ec3764-3e8e-4336-b782-2f2fdec57182)
{% endtab %}

{% tab title="twisted data" %}
![](https://4199572293-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lq83Pz1Ot3xU1lv1xHK%2F-LwV1f6ZMN8sweEPJsnZ%2F-LwV1qFHR6KmbGhlNHUf%2Fimage.png?alt=media\&token=f889d1d2-dfd1-4c12-936e-34eabc2376f8)
{% endtab %}
{% endtabs %}

## 3. Manifold Learning

you need to understand data distribution first

## 4. PCA (principle component analysis)

{% hint style="info" %}
(1) Identify hyper-plane that lies closest to the data. 

(2) Preserve **variance** as much as possible
{% endhint %}

#### 4.1. Preserving maximum variance -> lose less information

find the axis which preserve the maximum variance (C1 & C2 in this case)

PC = C1(maximum variance) & C2(largest remaining variance)

![](https://4199572293-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lq83Pz1Ot3xU1lv1xHK%2F-LwV2QJn_R4Y93N0Y_l3%2F-LwV338do5mnIEnxp80x%2Fimage.png?alt=media\&token=dce6c2db-cb03-4e3e-a27b-01e8453a5a42)

#### 4.2. SVD, single value decomposition

how to find PC? => (SVD) single value decomposition

`X` is the training set.

$$
X = U \cdot  \Sigma \cdot  V^{T}
$$

$$
V = \left \[ PC\_{1},  PC\_{2}, PC\_{3}, \cdots  \right ]
$$

Projecting the training set down to d-dimension.

`Wd` contains first `d` column of `V`.

$$
X\_{d\_proj} = X \cdot W\_{d}
$$

#### 4.3. Choosing the right number of dimensions

1. for data visualization = select 2 or 3
2. summation of variance >= 95%

#### 4.4. Reconstructing error

mean squared distance between the original data & the reconstructed data

$$
X\_{reconstructed} = X\_{d\_proj} \cdot W\_{d}^{T}
$$

#### 4.5. Other PCA

randomized PCA: reduce computation complexity

incremental PCA: avoiding feeding the whole training set, but feeding by mini-batches.

Kernel PCA: map instances into a very high-dim (feature space)  `linear-> nonlinear`

LLE (Local linear embedding): (1) measuring each training instance linearly relates to its closest neighbors (2) **looking for a low-dimensional where local relationships are best preserved** (3) good at unrolling twisted manifolds (4) poor scaling

t-SNE (t-Distributed Stochastic Neighbor Embedding): **keep similar instances close and dissimilar instances apart**.

#### 4.6. How to choose hyper-params for Unsupervised learning (e.g. kPCA)

1. Unsupervised learning = no obvious performance measure 
2. Unsupervised learning = the input for another  supervised learning

{% hint style="info" %}

1. Grid search to select params that lead to the best performance for the supervised task.
2. Lowest reconstruction error.
   {% endhint %}