How can I calculate very large matrix and store the values?
3 views (last 30 days)
Show older comments
Suppose I have 1 million-by-1 million matrix.
I will read later about creating very large matrix at here http://blogs.mathworks.com/loren/2011/10/14/new-mat-file-functionality-in-r2011b/
Now here's the problem.
I have 1 million points (with their Euclidean coordinate). I want to calculate distances between them, for example distance between point 1 and 1, 1 and 2, 1 and 3, and so on; distance between point 2 and 1, 2 and 2, 2 and 3, and so on; distance between point 3 and 1, 3 and 2, 3 and 3, 3 and 4 , and so on.
I want to store those calculations in that very large matrix.
Is mapreduce, or any other methods, can do the calculations and store them? If yes, how can I do that?
Any help will be greatly appreciate.
Thank you very much.
0 Comments
Answers (1)
Hugo
on 10 Apr 2015
I guess one option would be to reduce the size of your data using clustering such as K-means or Gaussian mixtures, and replace all points in a cluster with the centre of that cluster. You will end up with a reduced distance matrix and a map between cluster centres and data points. Of course, doing so will incur in an error, which can be decreased by increasing the number of clusters, at the cost of increasing the computational expenses.
0 Comments
See Also
Categories
Find more on Statistics and Machine Learning Toolbox in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!