memory pca vs pcacov

Dear all,

I'm running a pca on a large matrix (33*500,000) and with pcacov I get a memory error, but pca gives me no trouble. Could anyone explain this to me? Is the matrix somehow being reduced before computing the covariance matrix in pca?

Thanks!

Best

Hans

Answer 1

[Hiro Yoshino]

In PCA, your matrix (p x q) will be once converted into the variance-covariance matrix (q x q).

This would reqiure huge memory comsumption. Meanwhile, pcacov accepts a variance-covariance matrix as an input and, therefore the argument (input) should be a square matrix though.

As for big data anaysis, you may want to use tall array - this can be a solution.

tall array link

Answer 2

[Hans van der Horn]

Dear Hiro,

Thanks for your answer. If I compute cov(my matrix) to get the input for pcacov I run out of memory. However, if I run the pca function on my matrix. Why does the pca.m not get into memory problems?

Best

Hans

[Hans van der Horn]

*sorry, something was missing 'However, when I run the pca function on my matrix, it goes well. Why does pca.m not get into memory problems?'

Thanks

[Hiro Yoshino]

I do not believe pcacov works with your matrix in the first place since the shape of your matrix is unacceptable. I do not know what the error will be like.

Answer 3

[Hans van der Horn]

Dear Hiro,

The problem is that by constructing the covariance matrix Matlab runs out of memory. Why does the pca.m function not give memory problems (when I of course enter the original matrix)?

thanks!

[Hiro Yoshino]

I got your point now!!

OK, actually to avoid memory problem, pca normally takes a different approach to calculate eigen vectors - SVD. This is not a direct method and produces some by-product. This is a well-known fact - you may find the this in your text book too, I'm sure.

Also, the documentation for PCA reads it uses SVD.

[Hans van der Horn]

thanks! It's clear to me now:) I will also read the documentation for PCA (I thought pcacov also used svd..).