is it faster to load a big file (198MB) once and pass it onto a function in a parfor loop or let the function in parfor loop to load it every iteration?

7 views (last 30 days)
Dear All,
is it faster to load a big .mat file (198MB) once and pass it onto a function in a parfor loop or let the function in parfor loop to load it every iteration? My hunch is that to load it once (which take about 10min) and let it pass to the function. But I am not sure. Also, are there faster ways to load .mat file? Currently, I am simply using the "load" function.
Another question is that will the script run faster if I put all the codes in a single m file as compared to separating them into different functions (also m files). The readability will improve but I am not sure if the computation efficiency is significantly impacted. Thanks! best, Gene
  1 Comment
Jason Ross
Jason Ross on 7 Feb 2013
Edited: Jason Ross on 7 Feb 2013
Are you running everything locally on one machine, or are your workers distributed among several machines? Do you plan on running this on a cluster anytime soon? Is there a shared file system if you are planning on doing so?

Sign in to comment.

Answers (2)

Jason Ross
Jason Ross on 7 Feb 2013
You might find Edric's FEX submission worth reviewing:

Dan K
Dan K on 7 Feb 2013
A few things...
1 it should not take minutes to load a file that's only 200MB, so you may want to look at how you're loading it.
2. Yes, it is faster to load once then pass it around, since RAM is faster than a HD.
3. It doesn't matter if it's one file or many, but.... You'll get much more benefit by converting your script into a function

Categories

Find more on Parallel for-Loops (parfor) in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!