How to filter out dates within a datetime list?

Hi! I have two timetable containing hourly information from 2005 to 2019. The first one NOAA.mat is a 131472x1 timetable and the second one BUOY.mat is a 131274x1 timetable.

As you can see, there are some observation lost in the BUOY file. Can anyone please give me an idea on how to match the NOAA data with the BUOY data so they have the similar size? (i.e., I want to delete the extra observations from the NOAA data while keeping the rest of the time perfectly aligned.)

Any feedback will be greatly appreciated!!

[Adam Danz]

Both tables have datetime values that are not shared between both tables. Is the goal to eliminate all non-shared datetime values so that both tables have the same time stamps?

[Ashfaq Ahmed]

Hi! I wanted to prepare the NOAA data in a way that it matches the best with the BUOY timetable. Eliminating the non shared datetimes will be no problem!

Answer 1

[Adam Danz]

Here are the steps you can take.

1. NOAA contains dates and durations in separate variables. Combine them into 1 datetime vector using

dt=dateshift(NOAA.N_Time,'start','day') + NOAA.N_Hr;

If you want, you can format it to see the full datetime values using

dt.Format='dd-MMM-uuu HH:mm:ss';

2. To find matching datetime values between the two vectors and to update the tables to only include rows with matching time stamps,

[~, NOAAidx, BUOYidx] = intersect(dt, BUOY.B_Time);

NOAA = NOAA(NOAAidx,:);

BUOY = BUOY(BUOYidx,:);

3. Verify sizes

size(NOAA) % 107630 x 1

size(BUOY) % 107630 x 1

Are the time stamps the same?

isequal(BUOY.B_Time, dateshift(NOAA.N_Time,'start','day') + NOAA.N_Hr)

ans = true

[Ashfaq Ahmed]

Amazing!! But how could you check "NOAA contains dates and durations in separate variables"?

[Adam Danz]

Not sure what you mean. Are you asking how to check if a table has a datetime column and a duration column?

[Ashfaq Ahmed]

No. I was asking how did you check that the NOAA time and the BUOY time was that different? Because I thought there might be only ~200 data points missing. But it seems that more than 30,000 points did not match one another.

[Adam Danz]

The intersect function finds matching values between two vectors but the values have to be an exact match so if one datetime value is 0.001 seconds before or after another datetime value, it won't be a match.

If the number of non-matches is surprising, I urge you to explore your data, look at the values that were not matches and undestand why they are not matches. Maybe you want to change the algorithm (e.g. match dates and ignore time) or maybe your expectations were off.