Download all daily data (.nc) directly from the command window

The script below collects data for a specific latitude and longitude of precipitation for all days in January 2000 at this link: https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/ access /daily/0.25deg/2000/01/

However, my goal is not just to pull data from specific locations and turn it into .xlsx for just one month.

I want to download every day of the year directly from this link without specifying latitude and longitude and without having to do the month-by-month process: https://www.ncei.noaa.gov/data/cmorph-high-resolution -global-precipitation-estimates /access /daily /0.25 degrees/

% Download the CPC data used in the script below
year_start = 20000101; % first year to download
year_stop = 20000131; % last year to download
nb_of_years = year_stop - year_start +1;
httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/daily/0.25deg/2000/01/";
filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name
LG=[0.125];  % longitude (NB range is 0 : 360°)
LT=[-55.625];     % latitude
% export results to excel
filename_export='PRP_CMORPH.xlsx';
%% main loop
for ci = 1:nb_of_years
clear Cmorph
filename = strcat("CMORPH_V1.0_ADJ_0.25deg-DLY_00Z_", num2str(year_start+ci-1), ".nc"); % create filename according to list above
disp(strcat(filename, "  is processed"));
dataUrl = strcat(httpsUrl, filename);
data = webread(dataUrl);
filename_out = websave(filename_tmp,dataUrl); % save downloaded data
%     ncdisp(filename_tmp,'/','min');      % commented, just to avoid filling the command window
precip=ncread(filename_tmp,'cmorph');  % Dimensions: lon,lat,time
long=ncread(filename_tmp,'lon');
lat=ncread(filename_tmp,'lat');
time=ncread(filename_tmp,'time');
for i=1:3
LGG=find(long==LG(i));
LTT=find(lat==LT(i));
if isempty(LGG)||isempty(LTT)
disp('LOCATION NOT FOUND IN THIS NetCDF FILE')
break
end
% Step 5 :Convert 3 dimensional array to column vector array for one station
Precip(:,i)=precip(LGG,LTT,:);  % Unrecognized function or variable 'precip'.
X(i)={'Precipitation(mm/day)'};% label for variable
end
latt2=[LG ; LT];
time=double(time);
AA=time/24+datenum('1900-01-01 00:00:0.0');
[yy,mm,dd,~,~,~] = datevec(AA);
date1=[yy mm dd];
DD={'Year','Month','Day'};
sheet=ci;  % NB : every year on a new sheet - same file
C={'LOCATION: Cities','','','Longitude';'DATE FROM JAN-DEZ 2021','','','Latitude'};
writecell(C,filename_export,"Sheet",sheet,"Range",'A1');
writecell(DD,filename_export,"Sheet",sheet,"Range",'A3');
writematrix(latt2,filename_export,"Sheet",sheet,"Range",'E1');
writecell(X,filename_export,"Sheet",sheet,"Range",'E3');
writematrix(date1,filename_export,"Sheet",sheet,"Range",'A4');
writematrix(Precip,filename_export,"Sheet",sheet,"Range",'E4');
end

Answer 1

[Mathieu NOE]

hello Augusto

this code will loop over all specified monthes (it determines automatically how many days are in the current month)

remains now what you want to do will the daily data ...

% Download the CPC data used in the script below
which_year = 2000; % year to download

httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/daily/0.25deg/";
% filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name


%% main loop
mm = (1:12); % all monthes in year
% mm = [1 6]; % process only some specific monthes in year

for ci = 1:numel(mm) % loop over monthes
    
    days_per_month = datenum(which_year, mm(ci)+1, 1) - datenum(which_year, mm(ci), 1); % compute how many days this month/year

    for ck = 1:days_per_month % loop over days
        
       ymd_str = strcat(sprintf('%04d', which_year), sprintf('%02d', mm(ci)), sprintf('%02d', ck)); % year month day string
       Url = strcat(httpsUrl, sprintf('%04d', which_year), "/", sprintf('%02d', mm(ci)),"/");
       filename = strcat("CMORPH_V1.0_ADJ_0.25deg-DLY_00Z_", ymd_str ,".nc"); % create filename according to list above
       disp(strcat(filename, "  is processed"));
        dataUrl = strcat(Url, filename);
        data = webread(dataUrl);
        filename_tmp=strcat("data_", ymd_str ,".nc"); % temporary filename (for loop); data will be always stored under this temp name
        filename_out = websave(filename_tmp,dataUrl); % save downloaded data
        
        %     ncdisp(filename_tmp,'/','min');      % commented, just to avoid filling the command window
        precip=ncread(filename_tmp,'cmorph');  % Dimensions: lon,lat,time
        long=ncread(filename_tmp,'lon');
        lat=ncread(filename_tmp,'lat');
        time=ncread(filename_tmp,'time');
    
    end
    
end

[Augusto Gabriel da Costa Pereira]

this loop is only for the year 2000. if i want to loop only one script from 2000 to 2005 how do i do that?

[Augusto Gabriel da Costa Pereira]

I tried the code below but I can't download the data between the years 2000 to 2020.

%% This script downloads PRP data from the CMORPH reanalysis product


% Download the CMORPH data used in the script below
which_year = 20000101; % year start 
which_year2= 20201231; % year end
nb_of_years= which_year2 - which_year +1;
httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/daily/0.25deg/";

%% main loop
mm = (1:12); % every month of the year
% mm = [1 6]; % process only some specific monthes in year
for ci = 1:numel(mm) % loop over months 
    
    days_per_month = datenum(nb_of_years, mm(ci)+1, 1) - datenum(nb_of_years, mm(ci), 1); 
    
    for cl = 1:nb_of_years  % error here?
        filename = strcat("CMORPH_V1.0_ADJ_0.25deg-DLY_00Z_", num2str(year_start+cl-1), ".nc")
        
    for ck = 1:days_per_month 
        
       ymd_str = strcat(sprintf('%04d', nb_of_years), sprintf('%02d', mm(ci)), sprintf('%02d', ck)); % ano,mes e dia com string
       Url = strcat(httpsUrl, sprintf('%04d', nb_of_years), "/", sprintf('%02d', mm(ci)),"/");
       filename = strcat("CMORPH_V1.0_ADJ_0.25deg-DLY_00Z_", ymd_str ,".nc"); % crie o nome do arquivo de acordo com a lista acima
       disp(strcat(filename, "  is processed"));
        dataUrl = strcat(Url, filename);
        data = webread(dataUrl);
        filename_tmp=strcat("data_", ymd_str ,".nc");
        filename_out = websave(filename_tmp,dataUrl); 
             
        precip=ncread(filename_tmp,'cmorph');  
        long=ncread(filename_tmp,'lon');
        lat=ncread(filename_tmp,'lat');
        time=ncread(filename_tmp,'time');
    
    end
    
end

[Walter Roberson]

which_year = 20000101; % year start 
which_year2= 20201231; % year end
nb_of_years= which_year2 - which_year +1

nb_of_years = 201131

That is not the number of years you are dealing with !!

I recommend that you switch everything to use datetime arrays

which_year = datetime(20000101, 'ConvertFrom', 'yyyyMMdd')
which_year2 = datetime(20201231, 'ConvertFrom', 'yyyyMMdd')
nb_of_years = year(which_year2+1) - year(which_year)

[Augusto Gabriel da Costa Pereira]

still does not work, the problem may be more serious. as you can see, in Mathieu's script it is only to download the year 2000, I want to download the data range from 2000 to 2005.

[Walter Roberson]

What is your rewritten code that uses datetime and not datenum?

[Augusto Gabriel da Costa Pereira]

my only option is the one above, if you test the script on your computer, it will solve my problem

[Walter Roberson]

Suppose that I run the same code on my computer and it turns out to work (unlikely), then how does that solve your problem?

Suppose that I run the same code on my computer and it does not work (more likely) then how does that solve your problem?

If your only option is to use exactly the same code you posted, and that code does not do what you need, then you have a serious metaphysical challenge. If the only possibility is to keep doing the same thing as you are already doing but you need the outcome to be different, then you are asking that someone changes the universe to be such that what you are already doing gives a different outcome.

I would suggest to you that what is above is not your only option. Besides such possibilities as rewriting the code to use datetime, chances are that you also have options such as becoming a monk, or working on an oil rig, or becoming a fisher.

[Augusto Gabriel da Costa Pereira]

That is not the problem. The code that Mathieu solved is only for the year 2000 and you failed to understand that I want to solve this problem by transforming the script to the range 2000 to 2005. You suggested solutions don't work, I have tried several ways.

I could easily at each year end run a script for another year (i.e., when you finish downloading 2000, change the year to 2001, and so on), however, this becomes too manual, I want something more automatic, without having to make these manual changes.

Can you understand the question? It is something so simple, but I could not solve this simple problem.

[Walter Roberson]

I did not fail to understand that you want to solve the problem by transforming the script to the range 2000 to 2005. I understand that completely.

Suppose that you saw me doing a minor construction project using thin nails, and I was complaining that the work kept falling apart because the nails kept coming out. Would you give instructions on how to nail at a variety of angles to make it slightly harder for the nails to come out with every passing breeze -- or would you advise me to use screws and anchors ?

Your program is falling apart because of the way the code was written to handle dates. I am not going to advise you how to hack it to get through one particular issue: your program would be much more stable if you switched to using datetime facilities in your code, replacing all of the date handling logic with the newer facilities for datetime and duration objects.

[Mathieu NOE]

@Augusto Gabriel da Costa Pereira

Augusto

simple extension of my previous code to multiple years case (one more for loop)

ok, nothing fancy, I have to admit...

all the best

clc
clearvars

% Download the CPC data used in the script below
which_years = (2000:2005); % years to download

httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/daily/0.25deg/";
% filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name


%% main loop
mm = (1:12); % all monthes in year
% mm = [1 6]; % process only some specific monthes in year

for cc = 1:numel(which_years) % loop over years
    
    this_year = which_years(cc);

    for ci = 1:numel(mm) % loop over monthes

        days_per_month = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % compute how many days this month/year

        for ck = 1:days_per_month % loop over days

           ymd_str = strcat(sprintf('%04d', this_year), sprintf('%02d', mm(ci)), sprintf('%02d', ck)); % year month day string
           Url = strcat(httpsUrl, sprintf('%04d', this_year), "/", sprintf('%02d', mm(ci)),"/");
           filename = strcat("CMORPH_V1.0_ADJ_0.25deg-DLY_00Z_", ymd_str ,".nc"); % create filename according to list above
           disp(strcat(filename, "  is processed"));
           
            dataUrl = strcat(Url, filename);
            data = webread(dataUrl);
            filename_tmp=strcat("data_", ymd_str ,".nc"); % temporary filename (for loop); data will be always stored under this temp name
            filename_out = websave(filename_tmp,dataUrl); % save downloaded data

            %     ncdisp(filename_tmp,'/','min');      % commented, just to avoid filling the command window
            precip=ncread(filename_tmp,'cmorph');  % Dimensions: lon,lat,time
            long=ncread(filename_tmp,'lon');
            lat=ncread(filename_tmp,'lat');
            time=ncread(filename_tmp,'time');

        end

    end 

end

[Augusto Gabriel da Costa Pereira]

thank you Mathieu

Your answer was much more accurate.

[Mathieu NOE]

My pleasure !