Questions tagged [principal-component-analysis]
Principal component analysis (PCA) is a linear dimensionality reduction technique. It reduces a multivariate dataset to a smaller set of constructed variables preserving as much information (as much variance) as possible. These variables, called principal components, are linear combinations of the input variables.
143 questions
0
votes
3
answers
112
views
What is variance in context of projection of a vector?
I looked at the rules, I think it's not wrong to ask people to just explain something to you. (I hope the post won't get flagged). My knowledge level is as much as high-schooler. But I've been ...
0
votes
1
answer
71
views
Do top eigenvectors maximise both Tr$(P\Sigma)$ and Tr$(P\Sigma P\Sigma)$ for orthogonal projection matrices P?
Let $P \in \mathbb{R}^{d \times d}$ be an rank $p < d$ orthogonal projection matrix given by $P = VV^T$, $V \in \mathbb{R}^{d \times p}$. Do we have that
$$
\underset{P^2 = P = P^T,\; \text{rank}(P)...
- 71
0
votes
1
answer
153
views
Why is it necessary to center data before performing PCA? [closed]
I'm trying to get a deeper understanding of Principal Component Analysis (PCA), and I keep coming across the point that we must center the data around zero before determining the principal components.
...
- 11
0
votes
0
answers
46
views
Bounding sine of angle between PCA estimators using Davis Kahan
I am currently working on the following problem:
Assume that we observe n random vectors $X_1, . . . , X_n$ that are
independent copies of a random vector $X \sim N (0, \Sigma)$ where $\Sigma$ is a $d\...
- 57
3
votes
1
answer
124
views
Why are the following equivalent formulations of the PCA optimization problem correct?
In these notes, the author formulates the PCA problem as follows. Given a matrix of data $X$ the PCA problem is:
$$\text{min}_Y \|Y - X\|_F \qquad \text{rank}(Y) = k$$
This reads to me as obviously ...
- 8,711
0
votes
1
answer
65
views
Reversing PCA Transformation After Linear Combination of PCA Subspaces
I have two data matrices $ X_1 $ and $ X_2 $ with sizes $n \times d $, and I apply PCA to them resulting in:
$$
Z_1 = W_1 X_1
$$
$$
Z_2 = W_2 X_2
$$
Then, I apply a linear transformation in the PCA ...
- 351
0
votes
0
answers
47
views
What is exactly a principal component in PCA
Given a data matrix $X\in\mathbb{R}^{n \times p}$
The PCA algorithm is computing the covariance matrix $C$ of the centered data matrix $\bar{X}$ and then diagonalizing it
$$C=VLV^{T}$$
Are the "...
- 33
1
vote
1
answer
46
views
General question on principal component analysis subject to by an incorrect mean
Suppose a probability density function over $\mathbb R^d$.
Let its expectation be $\boldsymbol{\mu}$, and some erronous estimate of it $\tilde{\boldsymbol{\mu}} = \boldsymbol{\mu} + \boldsymbol{\...
- 11
1
vote
0
answers
33
views
Approximate a linear combination by one of fewer vectors
Given a $d\times n$ real matrix $M$ and a “weights” vector $w\in \mathbb{R}^n$ such that $\sum w_i =1$ and $0\leq w_i\leq 1$ for all $i$, I'd like to find, for some fixed $m<n$, some $d\times m$ ...
- 716
0
votes
1
answer
52
views
PCA analog when points are SPD matrices and congruence instead of linear projection
Let's consider the following formulation of PCA. We have a set of points $\{x_1, ..., x_m\}$ in $\mathbb{R}^n$, and a function $L_{\mathbf{V}}: \mathbb{R}^n \to \mathbb{R}^k$, with $k < n$, where $...
- 340
1
vote
1
answer
120
views
Prove that PCA decomposition captures all information in a factor model
Assume a data matrix $X \in \mathbb{R}^{N \times p_X}$. Let it have some exact lower dimensional factor representation $X = A F$, where $F \in \mathbb{R}^{N \times p_F}$ and $p_F < p_X$. Let the ...
- 1,678
1
vote
0
answers
61
views
Proving Optimal Loss Analogous to PCA
Suppose we have a data distribution on $x ∈ R^d$. Suppose $E_x[x] = 0$, and let $Σ = E_x[xx^T]$ be the covariance of x. Let $Σ = USU^T$ be the spectral decomposition of Σ, with U orthonormal and S ...
- 11
2
votes
1
answer
100
views
minimizing $| x x^T - A |^2$ for a covariance matrix $A$
For a given covariance matrix $A$ (so $A$ is symmetrical and positive semidefinite), I want to find a vector $x$ such that $| x x^T - A |^2$ is minimized. Here $|M|^2$ just means the sum of squares ...
- 1,916
1
vote
1
answer
118
views
PCA Reconstruction Properties
Let $X \in \mathbb{R}^{n \times d}$ be our data matrix where $n$ is the number of examples and $d$ is the feature dimension. Applying PCA to $X$, we get a low-dimensional representation $A \in \mathbb{...
- 295
1
vote
1
answer
72
views
Intuition behind Principal Component Analysis: linear combinations of original dimensions vs the PC direction as linear function of y and x
So in this example here the Principal Components (PC) are:
$$
\begin{align}
PC_1 (Size) &= 0.707 * Height + 0.707 * Diameter \quad (1)\\
PC_2 (Shape) &= 0.707 * Height - 0.707 * Diameter \...
- 33
1
vote
0
answers
47
views
Question about a linear algebra detail of Kernel PCA
As is shown in this question kernel pca eigenproblem and many other refernce materials about kernel PCA. They all point out that the solution of $K^2a_j=\lambda_jnKa_j$ and $Ka_j=\lambda_jna_j$ only ...
- 168
0
votes
0
answers
43
views
PCA: derivation of successive principal axes
Let $\Sigma$ be the covariance matrix. The first principal axis of PCA, $u_1$, can be found by solving the optimization problem:
$$\max_{u_{1}}=u^T_1\Sigma u_1$$
subject to the constraint $u^T_1u_1=1$....
- 63
1
vote
1
answer
294
views
Does PCA always find the best-fitting plane?
Here, the best-fitting plane is the plane that minimizes the sum of squared perpendicular distance from the data points to the plane. In other words, the best-fitting plane is the solution to the ...
- 13
1
vote
0
answers
68
views
Confusion regarding the geometrical meaning of singular values in SVD
I am trying to visualize in MATLAB the relationship between the singular value decomposition (SVD) of a matrix of points. To simplify the problem, I am working in 2D and I am considering an ellipse ...
- 121
0
votes
1
answer
54
views
the first principal component’s variance
The expression comes from The Elements of Statistical Learningp66(3.49)
Related part about this is shown in below figure.
Where Using SVD, $X$ has the form: $X=UDV^T$. And $D$ is a diagonal matrix, ...
- 13
2
votes
0
answers
560
views
Proof of Eckart-Young Theorem (Mathematics for Machine Learning, Deisenroth)
I am trying to understand a proof of the Eckart-Young Theorem (source in title). Let me add the definition and proof they provided, afterward I'll say what I do not understand.
Proof:
Some points I ...
- 69
1
vote
1
answer
484
views
Why do the "important" eigenvectors of a graph Laplacian have small-magnitude eigenvalues?
In spectral clustering, one computes sample-sample similarities, then from this computes a graph Laplacian matrix. (Typically, one uses the symmetrically normalized Laplacian matrix, but the pattern I'...
- 271
0
votes
1
answer
181
views
Suppose that the variance-covariance matrix of a $p$-dimensional random vector $X$ is
Suppose that the variance-covariance matrix of a $p$ -dimensional random vector $X$ is $\Sigma=\sigma_{ij}$ for all $i,j=1, 2, ...,p$. Show that the coefficients of the first principal component have ...
- 67
0
votes
0
answers
83
views
Order of the vectors in principal component analysis
For a given matrix $A$, the Principal Component Analysis (PCA) is done by finding the eigenvalues/eigenvectors of the covariance matrix associated with $A$. However, the entries of the covariance ...
- 535
0
votes
0
answers
58
views
Why the number of non-zero singular values is $n-1$ for $X\in \mathbb{R}^{n \times m}$ when $n\ll m$
There's a statement in the paper Phatak (1997)
In most spectroscopic problems, the number of non-zero singular values is $n-1$ for $X\in \mathbb{R}^{n \times m}$ when $n\ll m$. Consequently, at most $...
- 380
0
votes
0
answers
273
views
Proof that PCA is equivalent to MDS when using Euclidean distances
As I was watching a video explaining how MDS works, the narrator mentioned that PCA is equivalent to MDS when Euclidean distances are used. I got confused as to how that's the case.
My guess is that ...
- 121
1
vote
0
answers
29
views
Relation of principal component analysis between a matrix and its transopose
With a matrix $X_{n\times p}$ ($n>p$), we perform a principal component analysis:
$T_{n\times p}=X_{n\times p}W_{p\times p}$
where $W$ is the loadings matrix while $T$ is the scores matrix for $X$. ...
- 153
1
vote
1
answer
123
views
Can SVM be special case of PCA?
Let $X$ and $Y$ two linearly separable finite subsets of a $K$-dimensional real vector space $V$ with orthonormal basis $A = \{a_1,\ldots, a_K\}$. The covariance matrix $\Sigma_A$ of the set $X \cup Y$...
0
votes
0
answers
125
views
The coskewness and cokurtosis of uncorrelated standardized random vector
I was conducting the Karhunen-Loeve (K-L) Expansion for a random vector.
Based on the KL expansion, I transformed the original random vector into a standardized random vector $\boldsymbol{X}=[X_1,X_2,\...
- 63
1
vote
1
answer
1k
views
Why the singular values in SVD are always hierarchical/descending?
Please, I'm trying to understand why singular values (SV) are always hierarchical/descending. At the beginning of my studies, I thought that the hierarchy of sigmas ($ \sigma_1 \geq \sigma_2 \geq ... \...
- 19
0
votes
1
answer
62
views
Find line with respect to which the moment of inertia is minimized
Consider a function $F(x,y,z): \mathbb{R}^3 \mapsto \mathbb{R}^+$ (i.e., $F(x,y,z) > 0 ~~\forall~ x,y,z$) and consider a set of points in the (3D) space, $\{p_1, p_2, \cdots , p_N\}$.
The problem ...
- 144
0
votes
0
answers
199
views
Eigenvectors, Singular Vectors, and Excel
First time asker here.
First off, I know I should be doing this in R or Python. I will. For now I'm reading a textbook, using simple examples and Excel to try to learn the concepts of linear algebra. ...
- 13
2
votes
0
answers
52
views
How to find two small matrices $M_1$ and $M_2$ such that $M_1 M_2 A \approx M A$?
If we have a matrix $M$ and we want to find its least squares approximation as the product of two smaller (as in less rows or columns) matrices $M_1M_2$ of a given size, we can simply run SVD and pick ...
- 21
1
vote
1
answer
158
views
Clarifying the constraints used in deriving the Principal Components of PCA
In studying principal components analysis, I am confused by one point.
For a set of $N$ (zero-centered) data points of dimension $m$, projected to a dimension $k < m$, we want a set of vectors of ...
- 3,977
2
votes
0
answers
192
views
Understanding the solution to the varimax rotation problem
I'd like to preface this post by saying that this is my first post on stack exchange, so if there is anything to improve, be it redaction or just the structuring of posts, I'm more than willing to ...
1
vote
1
answer
53
views
Can you use EVD, SVD, PCA to solve least squares? (Intuitive Understanding)
I am trying to fully understand/demystify EVD, SVD and PCA. Am I right to assume all of these are tools/methods to solve least squares (not only but for CG)? If I am not wrong, even there are multiple ...
- 111
2
votes
0
answers
34
views
strategies for looking at the phase space of a system with 6 dimensions
I have a system of odes where the state vector has 6 elements. The system is a population biology model, where I am tracking the evolution of some competing species over time.
Now I was trying to ...
- 2,705
1
vote
0
answers
615
views
Computation of exponential and logarithmic maps on Riemann manifolds
In my computational problem, I have a Riemann submanifold $S^{1000}$ embedded in $\mathbb{R}^{300000}$. I can numerically compute the induced metric tensor and the Jacobian. I have no analytical ...
- 11
1
vote
1
answer
494
views
Proof of the variance of one-dimensional projections
In Bishop's Pattern Recognition And Machine Learning book, Chapter 12,
Suppose $X$ is an uncentered data matrix and $\bar{x}=\frac{1}{m}\sum_ix_i$ is the sample mean of the columns of $X$.
For the ...
- 303
0
votes
1
answer
323
views
Where in PCA does the non-uniqueness of eigenvectors come from?
I tried comparing sklearn.decomposition.KernelPCA with a linear kernel to sklearn.decomposition.PCA on the same data set and got different eigenvectors. My understanding is that these should be ...
- 1,946
0
votes
1
answer
176
views
Proof of "The sum of squared distances from the points to the line is a minimum"
I'm reading "Introduction to linear algebra" of Gillbert Strang.
PCA by SVD section.
Text says that the sum of squared distances from the points to the line is a minimum and author is trying ...
- 61
3
votes
1
answer
119
views
quasi-PCA reconstruction of the matrix by orthogonal basis
let's say I have a "data" matrix $X$ of $N$ rows and $p$ cols with $N \gg p$. Now PCA with $L$ components can be formulated as $$X_L = argmin_{Y:rank(Y) = L} ||X- Y||^2_F $$, where Y is ...
3
votes
1
answer
380
views
How to compute principal components for a curvature found given XYZ points?
I have a certain XYZ set of points that make up an object. I chose a random point and make the nearest radius analysis and find the neighbors. From these neighbors, I get the green pointcloud curve ...
- 83
0
votes
1
answer
2k
views
projection of the data along the 1st k principal components
I'm a final year maths undergrad doing a course in multivariate data analysis, but I'm really struggling with the linear algebra. In particular the “projection of the data along the 1st k principal ...
- 39
1
vote
0
answers
39
views
PCA factor $\frac{1}{n-1}$
I've seen multiple PCA derivations where the $\frac{1}{n}$ (for variance) or $\frac{1}{n-1}$ (for sample variance) is just omitted, e.g. here. I see that they are proportional to each other but is ...
- 333
2
votes
0
answers
35
views
How to learn data points by minimizing a loss function given their pairwise distance matrix?
Suppose $x_i \in\mathbb{R}^2$ for $i=1,2,...9$ are unknown. I'm given the pair-wise distance matrix between these points $D$ which is a $9*9$ symmetric matrix. I want to learn these data points by ...
- 2,633
1
vote
2
answers
2k
views
What does singular value decomposition of covariance matrix represent?
I am reading the paper "Understanding dimensional collapse in contrastive self-supervised learning." The authors identified a dimensional collapse phenomenon:
i.e. some dimension of ...
- 11
1
vote
1
answer
105
views
How to prove some insights regarding a new pca coordinate system
I've a question regarding pca variant.
Let $X ∈ \Bbb R^{D×n}$ be a data matrix, $\{u_i\}^{d}i=1$ be the $d$ principal components of $X$, and where $μ ∈ \Bbb R^d$ is the sample mean vector and $1_n ∈ \...
- 19
0
votes
0
answers
57
views
Distance metric that is insensitive to correlated variables
I'm trying to find a suitable pairwise distance metric where the addition of correlated vectors results in (essentially) no change in the distance.
Specifically, consider a set of $k$ vectors each of ...
- 275
2
votes
1
answer
293
views
How do you measure the 'explained variance' of arbitrary linear embeddings?
Question Elaboration:
When I say 'linear embeddings' I mean a lower-dimensional representation of variables resulting from an arbitrary linear transformation. And when I say 'explained variance' I ...
- 387