hourly data to daily- when time steps start at 00:00 and end on 23:00 each day (how to consider another 1 hour between 23:00 to 23:59 which presents on the next day)

I have one year data (3d array, name is tp) (lat*lon*time) based on hourly time steps from a netcdf file, I want to convert this hourly data to daily. the time steps of each days are:

0:00
1:00
2:00
3:00
4:00
5:00
6:00
7:00
8:00
9:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00

I want to have one value for each day. as you can see time range of 23:00 to 23:59 presents in next day, so I think I should have a code that continue to one hour ahead of each day (like 25 hours)

please help me.

here is ncdisp() I want tp daily (49*41*365)

>> ncdisp(filename)
Source:
           C:\Users\Behzad\Desktop\download.nc
Format:
           64bit
Global Attributes:
           Conventions = 'CF-1.6'
           history     = '2019-11-01 07:36:15 GMT by grib_to_netcdf-2.14.0: /opt/ecmwf/eccodes/bin/grib_to_netcdf -o /cache/data6/adaptor.mars.internal-1572593007.3569295-19224-27-449cad76-bcd6-4cfa-9767-8a3c1219c0bb.nc /cache/tmp/449cad76-bcd6-4cfa-9767-8a3c1219c0bb-adaptor.mars.internal-1572593007.35751-19224-4-tmp.grib'
Dimensions:
           longitude = 49
           latitude  = 41
           time      = 8760
Variables:
    longitude
           Size:       49x1
           Dimensions: longitude
           Datatype:   single
           Attributes:
                       units     = 'degrees_east'
                       long_name = 'longitude'
    latitude 
           Size:       41x1
           Dimensions: latitude
           Datatype:   single
           Attributes:
                       units     = 'degrees_north'
                       long_name = 'latitude'
    time     
           Size:       8760x1
           Dimensions: time
           Datatype:   int32
           Attributes:
                       units     = 'hours since 1900-01-01 00:00:00.0'
                       long_name = 'time'
                       calendar  = 'gregorian'
    tp       
           Size:       49x41x8760
           Dimensions: longitude,latitude,time
           Datatype:   int16
           Attributes:
                       scale_factor  = 3.0792e-07
                       add_offset    = 0.010089
                       _FillValue    = -32767
                       missing_value = -32767
                       units         = 'm'
                       long_name     = 'Total precipitation'

Answer 1

[Guillaume]

Nothing is attached. Anyway, the simplest way to do what you want is to store your data in a timetable. Then you simply retime the timetable:

tt = timetable(???); %don't know what your data is
ttdaily = retime(tt, 'daily', 'mean'); %you haven't said what aggregation method you want to use

[BN]

Dear Guillaume,

I was editing the question and write some information.

my variable is a 3d array (lat*lon*time) (see tp in ncdisp)

I want to summation hourly data to daily.

please click below to download my .nc file from my google drive (size is 6 MB):

https://drive.google.com/open?id=1qm5AGj5zRC3ifD1_V-ne2nDT1ch_Khik

when I type this code below:

filename='download.nc'; %Name of netcdf file
ncdisp(filename)       %show summary of nc
lat = ncread(filename,'latitude'); %reading latitude     
lon = ncread(filename,'longitude'); %reading longitude 
precip = ncread(filename,'tp'); %reading the main variable
t = ncread(filename,'time'); %read time
tt = timetable(precip); %% ERROR

the error is:

Error using timetable (line 424)
Provide a datetime or duration row times vector, a duration or
calendarDuration time step, or a positive numeric sample rate to
create timetable.

thank you

[BN]

I have hourly data from ECMWF ERA5 for each day in a specific year. I want to convert that data from hourly to daily. Copernicus has a Python code for this here https://confluence.ecmwf.int/display/CKB/ERA5%3A+How+to+calculate+daily+total+precipitation

I want to know what is the matlab code to do this? I was upload the netcdf file in my google drive here:

https://drive.google.com/open?id=1qm5AGj5zRC3ifD1_V-ne2nDT1ch_Khik

time steps of each day are:

0:00
1:00
2:00
3:00
4:00
5:00
6:00
7:00
8:00
9:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00

Notice to cover total precipitation for 1st January 2017 for example, we need two days of data:

1st January 2017 time = 01 - 23 will give you total precipitation data to cover 00 - 23 UTC for 1st January 2017

2nd January 2017 time = 00 will give you total precipitation data to cover 23 - 24 UTC for 1st January 2017

here is ncdisp():

>> ncdisp(filename)
Source:
           C:\Users\Behzad\Desktop\download.nc
Format:
           64bit
Global Attributes:
           Conventions = 'CF-1.6'
           history     = '2019-11-01 07:36:15 GMT by grib_to_netcdf-2.14.0: /opt/ecmwf/eccodes/bin/grib_to_netcdf -o /cache/data6/adaptor.mars.internal-1572593007.3569295-19224-27-449cad76-bcd6-4cfa-9767-8a3c1219c0bb.nc /cache/tmp/449cad76-bcd6-4cfa-9767-8a3c1219c0bb-adaptor.mars.internal-1572593007.35751-19224-4-tmp.grib'
Dimensions:
           longitude = 49
           latitude  = 41
           time      = 8760
Variables:
    longitude
           Size:       49x1
           Dimensions: longitude
           Datatype:   single
           Attributes:
                       units     = 'degrees_east'
                       long_name = 'longitude'
    latitude 
           Size:       41x1
           Dimensions: latitude
           Datatype:   single
           Attributes:
                       units     = 'degrees_north'
                       long_name = 'latitude'
    time     
           Size:       8760x1
           Dimensions: time
           Datatype:   int32
           Attributes:
                       units     = 'hours since 1900-01-01 00:00:00.0'
                       long_name = 'time'
                       calendar  = 'gregorian'
    tp       
           Size:       49x41x8760
           Dimensions: longitude,latitude,time
           Datatype:   int16
           Attributes:
                       scale_factor  = 3.0792e-07
                       add_offset    = 0.010089
                       _FillValue    = -32767
                       missing_value = -32767
                       units         = 'm'
                       long_name     = 'Total precipitation'

tp is my variable which have 3 dimensions (lon*lat*time) = 49*41*8760

I want it in the 49*41*365 for a non-leap year.

The result should be the daily values for the whole year.

I don't know how to do it, please help me.

thank you

[Guillaume]

Well, yes you need to pass the dates when you construct the timetable. However, now that I can see your data, a timetable is not going to be so useful. A table might be.

You still haven't said how you want to aggregate the data over a full day. I'm assuming you want the average for the rest of this answer.

Currently your data is made of 24 measurements per day and you always have a full day worth of measurement, so you could just take the average over 24 consecutive measurements. That's very simple:

daily_precip = squeeze(mean(reshape(precip, size(precip, 1), size(precip, 2), 24, []), 3));

Done!

However, if you could have irregular measurements or non-full days you'd have to use a different approach.

First you'd have to convert your time into a datetime type:

dt = datetime(double(t)*3600, 'ConvertFrom', 'epochtime', 'Epoch', '01-Jan-1900');

Then you could use discretize and splitapply

[group, actualday] = discretize(dt, 'day');
daily_precip = splitapply(@(pages) {mean(precip(:, :, pages), 3)}, (1:size(precip, 3))', group);
daily_precip = cat(3, dail_precip{:});

or you could store all your data in a table

[llat, llon] = ndgrid(lon, lat, 1:size(precip, 3));
hourly_precip = table(repelem(dt, numel(lat)*numel(lon)), llat(:), llon(:), precip(:), 'VariableNames', {'Date', 'Lat', 'Lon', 'Precipitation'});

and then use groupsummary to make it daily

daily_precip = groupsummary(hourly_precip, {'Date', 'Lat', 'Lon'}, {'day', 'none', 'none'}, 'mean')

[BN]

Thank you so much for your help. I try your code but in the following stage:

[group, actualday] = discretize(dt, 'day');
daily_precip = splitapply(@(pages) mean(precip(:, :, pages), 3), (1:size(precip, 3))', group);
daily_precip = cat(3, dail_precip{:});

it's give me this error:

Error using splitapply (line 132)
The function '@(pages)mean(precip(:,:,pages),3)' returned a non-scalar
value when applied to the 1st group of data.

To compute nonscalar values for each group, create an anonymous
function to return each value in a scalar cell:

	@(pages){mean(precip(:,:,pages),3)} 
>> 

------------------

then I was try this code:

[llat, llon] = ndgrid(lon, lat, 1:size(precip, 3));
hourly_precip = table(dt, llat(:), llon(:), precip(:), 'VariableNames', {'Date', 'Lat', 'Lon', 'Precipitation'});

but it gaves me this error too:

Error using table (line 231)
All input variables must have the same number of rows. 

I want to sum hourly data for calculating daily data.

I'm really new on this issue if you can help me I would be so thankful.

[Steven Lord]

Guillaume, in your code to create a datetime:

dt = datetime(double(t)*3600, 'ConvertFrom', 'epochtime', ...
    'Epoch', '01-Jan-1900');

An approach that's closer to how the data is described in the output of ncdisp may be a little easier to understand.

% 'hours since 1900-01-01 00:00:00.0'
d = hours(t) + datetime(1900, 1, 1);

[Guillaume]

All the bugs have been fixed (hopefully!). Indeed Steven's approach to constructing the datetime vector is much simpler than mine.

[BN]

A million thanks to you! it's run correctly :)