Select Numerous Files from Folder Based on Values in Matrix

Hi all!

I have one big folder with thousands of files containing data for each day of each year from 1980 to 2000. The name of each .nc file includes the date on which data was recorded and looks something like this ('northward_winds_Np.19910705.nc4', which means this is the data file corresponding to July 5th, 1991). However, I am currently trying to find a more practical way to select only certain files from my big folder using the values in a matrix. That is to say, my matrix has various columns representing the specific month, day, and year on which a certain event happened. As such, I would like to select only the files on which this event happened from the original folder and move the selected files into a smaller folder to further analyze the files as a group. Any suggestions? I greatly appreciate any type of feedback and/or guidance.

Thanks in advance!

Best,

M.

Answer 1

[Sahil Jain]

Hi Michelle. Since I do not know the exact structure of your data, I'll provide a few pointers which you can use to solve your problem. I'm assuming that every file is named "northward_winds_Np.YYYYMMDD.nc4" and that the matrix has a separate column indicating whether the event of interest occured on that date or not.

To get the list of files on which the event occured, the following "for" loop will work. The matrix has been named "data" and the first, second and third column are assumed to contain the year, month and day values respectively.

relevant_files = [];
for i=1:length(data)
    if % condition to check if event occured
        year = num2str(data(i,1));
        
        if numel(num2str(data(i,2)))==1
            month = strcat("0",num2str(data(i,2)));
        else
            month = num2str(data(i,2));
        end
        
        if numel(num2str(data(i,3)))==1
            day = strcat("0",num2str(data(i,3)));
        else
            day = num2str(data(i,3));
        end
        
        file_name = strcat("northward_winds_Np.",year,month,day,".nc4");
        relevant_files = [relevant_files; file_name];
    end
end

Once you have the list of all the files you need, you can use the "copyfile" function to copy these files to a different folder.

[Michelle De Luna]

Sahil, thank you for your response! It was very helpful. I've gotten to the point where I can successfully create the "relevant_files" list, but I am unsure of how to open the items in this list using the "copyfile" function. When I use the following code, an error message appears:

mkdir myFolder
copyfile relevant_files myfolder

I then try something a little different by pulling out one file from my group of selected files and naming it "file", which doesn't seem to work either...

file = ('northward_winds_100_Np.19800602.nc4.nc4')
mkdir myFolder
copyfile file myfolder

However, when I write out the actual file name, the file does indeed appear in the subfolder titled "myFolder"...

mkdir myFolder
copyfile northward_winds_100_Np.19800602.nc4.nc4 myFolder

Since writing out the individual name of each of my selected files would be impractical, do you have any suggestions moving forward?

[Sahil Jain]

In the code snippet I provided in the answer, could you try adding the following lines of code below it?

for i=1:length(relevant_files)
    status = copyfile(relevant_files(i), "myFolder");
end