Initial setup (same as what you have):
myToken = "NpMoDLCgEVBMZTdPFQHQUOwllVQNHYjz";
s1="locationid=FIPS:";
FIPS_counter=51; %%%%% enter FIPS of any state and find out the number of elligible weather stations%%%%%
s2=num2str(FIPS_counter);
location=strcat(s1,s2);
myURL = "https://www.ncei.noaa.gov/cdo-web/api/v2/stations?"+location+"&startdate=2000-01-01&enddate=2021-12-31";
opt = weboptions('KeyName','token','KeyValue',myToken);
Data=webread(myURL,opt);
Number_of_station=Data.metadata.resultset.count
To get the IDs at "each stage of the for loop" (note that this works even if Number_of_station is <= 1000):
F = ceil(Number_of_station/1000);
IDs = cell(1,F);
for offset=0:F-1
offsetString = sprintf("&offset=%d",offset*1000);
myURL = "https://www.ncei.noaa.gov/cdo-web/api/v2/stations?locationid=FIPS:37&startdate=2000-01-01&enddate=2021-12-31&limit=1000" + offsetString;
opt = weboptions('KeyName','token','KeyValue',myToken);
data(offset+1)=webread(myURL,opt);
IDs{offset+1} = cellfun(@(x)x.id,data(offset+1).results,'UniformOutput',false);
end
IDs
IDs{:}
And maybe you want to combine all the IDs into a single cell array:
IDs = vertcat(IDs{:});
And maybe you want to remove the extra IDs that were retrieved as a result of having to get 1000 at a time:
IDs(Number_of_station+1:end) = []
Or maybe you don't have to get 1000 at a time (this may have been what you were going for already):
F = ceil(Number_of_station/1000);
R = rem(Number_of_station,1000);
limits = 1000*ones(1,F);
limits(F) = R;
IDs = cell(1,F);
for offset=0:F-1
offsetString = sprintf("&offset=%d",offset*1000);
limitString = sprintf("&limit=%d",limits(offset+1));
myURL = "https://www.ncei.noaa.gov/cdo-web/api/v2/stations?locationid=FIPS:37&startdate=2000-01-01&enddate=2021-12-31" + limitString + offsetString;
opt = weboptions('KeyName','token','KeyValue',myToken);
data(offset+1)=webread(myURL,opt);
IDs{offset+1} = cellfun(@(x)x.id,data(offset+1).results,'UniformOutput',false);
end
IDs = vertcat(IDs{:})
But if you need the rest of the results besides just the IDs, maybe this is better:
F = ceil(Number_of_station/1000);
R = rem(Number_of_station,1000);
limits = 1000*ones(1,F);
limits(F) = R;
results = cell(Number_of_station,1);
for offset=0:F-1
offsetString = sprintf("&offset=%d",offset*1000);
limitString = sprintf("&limit=%d",limits(offset+1));
myURL = "https://www.ncei.noaa.gov/cdo-web/api/v2/stations?locationid=FIPS:37&startdate=2000-01-01&enddate=2021-12-31" + limitString + offsetString;
opt = weboptions('KeyName','token','KeyValue',myToken);
data(offset+1)=webread(myURL,opt);
results(offset*1000+(1:limits(offset+1))) = data(offset+1).results;
end
results
results{1} % see what the 1st one looks like
And you can get the IDs from that like this:
IDs = cellfun(@(x)x.id,results,'UniformOutput',false)
Remind ourselves what each element of the 'data' struct array looks like:
data(1)
data(1).metadata
data(1).metadata.resultset
If you don't need the 'metadata' field of the 'data' struct array, then you don't need the 'data' struct array at all (since you already have all the 'results' fields):
F = ceil(Number_of_station/1000);
R = rem(Number_of_station,1000);
limits = 1000*ones(1,F);
limits(F) = R;
results = cell(Number_of_station,1);
for offset=0:F-1
offsetString = sprintf("&offset=%d",offset*1000);
limitString = sprintf("&limit=%d",limits(offset+1));
myURL = "https://www.ncei.noaa.gov/cdo-web/api/v2/stations?locationid=FIPS:37&startdate=2000-01-01&enddate=2021-12-31" + limitString + offsetString;
opt = weboptions('KeyName','token','KeyValue',myToken);
results(offset*1000+(1:limits(offset+1))) = getfield(webread(myURL,opt),'results');
end
results
