How to create separate data for daylight hours and nighttime hours in MATLAB?

Dear all,

I have hourly surface temperature data from the model output. The data is a 3D matrix with a size of 303×185×1176 (lon×lat×time). Now, I have to create two matrixes from the existing data. One matrix will be for only daylight hours (6.30 AM-6.30 PM) and another matrix will be for only nighttime hours (6.30 PM-5.30 AM). Since the timestep of the model output is in UTC, I have to add 5.30 with the model hours to get the IST (Indian standard time). The model hours starts from 00:00 UTC on frst simulation day and ends to 00:00 UTC on last simulation day.Thus, in my case, for the first day, daylight hours start from the timesteps 1 to 13, and nighttime hours start from the timesteps 13 to 24. So, we have to add 24 hours to the above timestep ranges to get the daytime and nighttime hours for the next day. Similarly, for the day after the next day, we have to add 48 hours with the first timestep ranges for getting the daytime and nighttime hours, and so on. I tried with the following MATLAB code:

num_hours_per_day = 24;
num_days = size(ST_data, 3) / num_hours_per_day; 
daytime_start = 1; % Initial daytime start timestep
nighttime_start = 13; % Initial nighttime start timestep
num_hours = size(ST_data, 3);
% Initialize daytime and nighttime matrices
daytime_matrix = [];
nighttime_matrix = [];
% Loop through each day
for day = 1:num_days
    % Calculate indices for daytime and nighttime hours
    daytime_indices = daytime_start:daytime_start+12; % Daytime hours (01:00 to 13:00)
    nighttime_indices = nighttime_start:nighttime_start+11; % Nighttime hours (14:00 to 24:00)
    
    % Update daytime and nighttime matrices
    daytime_matrix(:,:,daytime_indices) = ST_data(:,:,daytime_indices);
    nighttime_matrix(:,:,nighttime_indices) = ST_data(:,:,nighttime_indices);
    
    % Update start timesteps for the next day
    daytime_start = daytime_start + 24;
    nighttime_start = nighttime_start + 24;
end

But, I think that my code is incorrect. Because, after running the code, the size of daytime_matrix is 303×185×1165 and nighttime_matrix is 303×185×1176. If I am separating the total hours into daytime and nighttime, then the size of the third dimension of each matrix should be almost half of the total timestep. So, can anyone please help me to create the correct MATLAB code according to the above calculation? Any help will be highly appreciated. Thank you for your time and consideration.

With regards,

Ankan

Answer 1

[Star Strider]

It would be best to begin by importing your data to a table using the readtable function. The date and time values (whatever they are) should be automatically imported as a datetime array. Setting the original timezone as UTC and then converting it to your local time is straightforward, as are the logical operations necessary to separate daytime and nighttime. If necessary, you can also convert your data to a timetable using the table2timetable function. Other possibilities, for example using the retime function are then possible as well.

[ANKAN SARKAR]

Dear Star Strider,

Thank you for your response. My surface temperature data is in a netcdf format, not in a table format. So, what to do then? Do I have to use array2timetable function instead of table2timetable function for converting my matrix into a table? Can you please guide me? I am really looking forward to hearing from you.

With regards,

Ankan

[Dyuman Joshi]

"My surface temperature data is in a netcdf format, not in a table format. So, what to do then? "

See - cdfinfo and cdfread

[Star Strider]

As always, my pleasure!

It was not obvious from your posted code that the files were CDF files. As @Dyuman Joshi mentioned, you need to import them with cdfread.

See that documentation for the correct way to import the dates and times as datetime arrays. There are examples of that in the cdfread documentation, and the various name-value pair options are defined in the Description section.

[ANKAN SARKAR]

Dear all,

My data is netcdf format ('.nc'). cdfread don't works for this file.

>> data = cdfread(fname);

Error using cdfread

The file format is netCDF, not CDF. See "help netcdf".

To read netcdf file, I imported data using ncread.

My question is that is there any need to convert the matrix file (which appears in the workspace after reading using 'ncread') to a timetable? Or else will it not work? Can you please explain it a little bit more? That will be very helpful.

Thanks and regards,

Ankan

[Dyuman Joshi]

Sorry, that was an oversight from my side, the appropriate function here (as you found out) is ncread.

Could you please share your data (or a sample of it), so that we have a better idea how your data is stored/strucutred? Use the paperclip button to attach.

[Star Strider]

My apologies with respect to cdfread and not ncread. I was following the lead suggested by @Dyuman Joshi and did not look more closely.

I do not have your data, so it is difficult for me to comment on the matrix. I would convert it to a table using the array2table function, and then if you need to, convert it to a timetable. (If you only want to segment the daytime and nighttime variables, it may not be necessary to use a timetable.) It may be necessary to treat the date and time column separately, convert it to a datetime array, and use addvars to add it as the first variable in the table, since I am not certain if array2table will import a datetime array as well.

As a general rule, table arrays are extremely useful, so I definitely would convert the array to a table.

[ANKAN SARKAR]

Dear all,

Thank you all for your continious responses. I have attached a sample data (.mat files) for you since only files of less than 5MB are allowed to upload here. However, I think it will also help you to understand the structure of my data. Please find the attachments. And, it will be helpful if you can provide any sample MATLAB code using these files, so that I can run in my MATLAB and understand better of what you are saying. Otherwise I am not able to understand properly that how I can create two sepearte matrixes among which one will consist of only day hours and other one will contain only data for night hours. I am really looking forward to hearing from you.

Thanks in advance.

With regards,

Ankan

[Star Strider]

It took a while to get this to work. In order to test it, I finally created my own separate ste of variables, and it works with them. I cannot determine if it works correctly with your data set, since the time vector is not long enough.

I cannot get the time logic get the logic to work with fractional hours (06:30 and 18:30). The isbetween function does not like to work with them, and I cannot figure out how to combine the hour and minute part of the time to create the logic to work with them. For one day this would be relatively straightforward, however foir several days it would require looping through each day separately. So, I ended up going with integer hours.

Try this —

load('ST_data.mat')
load('time.mat')

% whos
ST = ST_data;
TUCT = datetime(time, 'ConvertFrom','posix', 'TimeZone','UCT')

Warning: 'UCT' specifies a time zone with a fixed offset from UTC, +00:00. This zone does not follow daylight saving time, and so may give unexpected results. See the datetime.TimeZone property for details about specifying time zones.

TUCT = 72x1 datetime array
   07-Jan-1970 22:23:13
   07-Jan-1970 22:23:14
   07-Jan-1970 22:23:15
   07-Jan-1970 22:23:16
   07-Jan-1970 22:23:17
   07-Jan-1970 22:23:18
   07-Jan-1970 22:23:19
   07-Jan-1970 22:23:20
   07-Jan-1970 22:23:21
   07-Jan-1970 22:23:22
   07-Jan-1970 22:23:23
   07-Jan-1970 22:23:24
   07-Jan-1970 22:23:25
   07-Jan-1970 22:23:26
   07-Jan-1970 22:23:27
   07-Jan-1970 22:23:28
   07-Jan-1970 22:23:29
   07-Jan-1970 22:23:30
   07-Jan-1970 22:23:31
   07-Jan-1970 22:23:32
   07-Jan-1970 22:23:33
   07-Jan-1970 22:23:34
   07-Jan-1970 22:23:35
   07-Jan-1970 22:23:36
   07-Jan-1970 22:23:37
   07-Jan-1970 22:23:38
   07-Jan-1970 22:23:39
   07-Jan-1970 22:23:40
   07-Jan-1970 22:23:41
   07-Jan-1970 22:23:42

[s,f] = bounds(TUCT)

s = datetime
   07-Jan-1970 22:23:13

f = datetime
   07-Jan-1970 22:24:24

TIST = TUCT;
TZ = '+05:30';                                  % IST Offset
TIST.TimeZone = TZ                              % Set This For Your Time Zone

TIST = 72x1 datetime array
   08-Jan-1970 03:53:13
   08-Jan-1970 03:53:14
   08-Jan-1970 03:53:15
   08-Jan-1970 03:53:16
   08-Jan-1970 03:53:17
   08-Jan-1970 03:53:18
   08-Jan-1970 03:53:19
   08-Jan-1970 03:53:20
   08-Jan-1970 03:53:21
   08-Jan-1970 03:53:22
   08-Jan-1970 03:53:23
   08-Jan-1970 03:53:24
   08-Jan-1970 03:53:25
   08-Jan-1970 03:53:26
   08-Jan-1970 03:53:27
   08-Jan-1970 03:53:28
   08-Jan-1970 03:53:29
   08-Jan-1970 03:53:30
   08-Jan-1970 03:53:31
   08-Jan-1970 03:53:32
   08-Jan-1970 03:53:33
   08-Jan-1970 03:53:34
   08-Jan-1970 03:53:35
   08-Jan-1970 03:53:36
   08-Jan-1970 03:53:37
   08-Jan-1970 03:53:38
   08-Jan-1970 03:53:39
   08-Jan-1970 03:53:40
   08-Jan-1970 03:53:41
   08-Jan-1970 03:53:42

[start,finish] = bounds(TIST)

start = datetime
   08-Jan-1970 03:53:13

finish = datetime
   08-Jan-1970 03:54:24

Daytime = hour(TIST)>=6 & hour(TIST)<=18;
Check = [numel(Daytime)  nnz(Daytime)]

Check = 1x2

72 0

Excerpt = squeeze(ST_data(1,2,:));

figure
plot(TIST(Daytime), Excerpt(Daytime), '-r', 'DisplayName','Daytime')
hold on
plot(TIST(~Daytime), Excerpt(~Daytime), '-b', 'DisplayName','Nighttime')
hold off
grid
xlabel('Time')
ylabel('Amplitude')
legend('Location','best')

%=========================================================================
% For Demonstration & Testing Purposes Only — Not Part Of Functioning Code

tv = datetime('07-Jan-1970 00:00') + minutes(0:15:4500).';      % Create Data  To Check Logic
tv.TimeZone = TZ;

dv = randn(size(tv));

% Daytime = hour(tv)>=6 & minute(tv)>=30 & hour(tv)<=18 & minute(tv)<=30;
% Daytime = hour(tv)>=6.5 & hour(tv)<=18.5;
Daytime = hour(tv)>=6 & hour(tv)<=18;

figure
plot(tv(Daytime), dv(Daytime), '-r', 'DisplayName','Daytime')
hold on
plot(tv(~Daytime), dv(~Daytime), '-b', 'DisplayName','Nighttime')
hold off
grid
xlabel('Time')
ylabel('Amplitude')
legend('Location','best')

% xline(datetime('07-Jan-1970 06:30','InputFormat','dd-MMM-yyyy HH:mm'), '--k')
% xline(datetime('07-Jan-1970 18:30','InputFormat','dd-MMM-yyyy HH:mm'), '--k')

I also cannot get xline to work with datetime arguments. I thought that worked at one point, however I cannot make it work here. It just throws a warning.

.

Answer 2

[Steven Lord]

Since the timestep of the model output is in UTC, I have to add 5.30 with the model hours to get the IST (Indian standard time).

Rather than trying to do the date and time arithmetic yourself, you probably should specify the TimeZone when you construct the datetime array from your data and then change that property as necessary.

N = datetime('now', TimeZone = 'America/New_York', Format="dd-MMM-yyyy hh:mm:ss a Z")

N = datetime
   21-Mar-2024 01:31:40 PM -0400

N.TimeZone = 'UTC'

N = datetime
   21-Mar-2024 05:31:40 PM +0000

I see two time zone names in the time zone database with a UTC offset of 5.5 hours, so you could use one of those.

data = timezones;
data(data.UTCOffset == 5.5, :)

ans = 2x4 table
          Name           Area    UTCOffset    DSTOffset
    _________________    ____    _________    _________

    {'Asia/Calcutta'}    Asia       5.5           0    
    {'Asia/Colombo' }    Asia       5.5           0    

Then you can extract parts of your data like the timeofday.

timeSinceMidnight = timeofday(N)

timeSinceMidnight = duration
   17:31:40

You could also use isbetween.

isDuringWorkingHours = isbetween(timeSinceMidnight, hours(9), hours(17)) % 9 AM - 5 PM

isDuringWorkingHours = logical
   0

But is it working hours in New York?

N.TimeZone = 'America/New_York'

N = datetime
   21-Mar-2024 01:31:40 PM -0400

timeSinceMidnight = timeofday(N)

timeSinceMidnight = duration
   13:31:40

isDuringWorkingHours = isbetween(timeSinceMidnight, hours(9), hours(17)) % 9 AM - 5 PM

isDuringWorkingHours = logical
   1

As I type this it's 1:31 PM so yes, it's normal business working hours.