I'm running 64-bit MATLAB R2010b on a MacBook Pro, OS 10.7. I have some large data files that I'd like to store on a non-AFS server, and I was wondering if it's possible to load the variables stored in the files directly into MATLAB without copying the files from the server to my local disk. If the server were AFS, this would be simple, since mounting the server then makes it accessible from my local file system. However, the server is not mounted via AFS and must be SSH'ed into. I have set up SSH keys, so assume that I am not required to enter a password to access the files, only the remote file name, in the format "user@hostname : remote/path.mat".
I know that I could scp the files directly to my local disk via the "unix" command, but I'd like to load the variables in the .mat file directly into the workspace without copying the large file to my disk. Furthermore, the server itself does not have MATLAB installed, so I cannot obviate the problem by running MATLAB on the remote machine.
Does anyone know how this might be implemented in MATLAB, preferably within a function? Or would it be much less of a hassle to just bite the bullet and store the files on my local disk?
I vote for storing the files locally. This is straight and easy and the (time) costs for transferring the data through the net cannot be avoided at all.
If you are talking about TerraBytes of data and really want/need to avoid the duplication, store them on an AFS server instead. This will be less expensive than constructing a smart MEX-function, which loads the data transparently using the HFS5 structure of the MAT files.