How do I filter data based on time?

I have decades of hourly meteorological data that I want to investigate for the months October-April every year (essentially a long winter). I figured out how to filter only those dates, but I also want to filter the data itself for those dates. At this point, I have a shortened dateArray, but my other variables are still the entire year. How do I associate the variables with a particular time?

I tried using the timeseries class, but it did not like that I was using datetime. Also, the timeseries class does not have "month," which would make it difficult for me to filter by season.

date1 = dataArray{:, 1};
windDir = dataArray{:, 2};
windSpeed = dataArray{:, 3};
ceilingHeight = dataArray{:, 4};
visibility = dataArray{:, 5};
temp = dataArray{:, 6};
dewpoint = dataArray{:, 7};
SLP = dataArray{:, 8};

formatIn = 'yyyymmddHHMM'; 

dateArray = datetime(datevec(num2str(date1),formatIn),'InputFormat','yyyymmddHHMMSS','TimeZone','America/Los_Angeles','Format','d-MMM-y HH:mm:ss Z');
   %converted my numerical date to a string to use datevec to break it up into 6 vectors to be read in by datetime 

dateMonth = month(dateArray);
   %identify month to filter by season

dateSpring=(dateArray(dateMonth <=4));
   %create a variable that is all the dates for Jan-April
dateFall=(dateArray(dateMonth >=10));
   %create a variable that is all the dates for Oct-Dec
dateWinter=union(dateSpring,dateFall);
   %combine Jan-April and Oct-Dec

Answer 1

[Star Strider]

Without your data, it’s not possible to write specific code. You need the row numbers that correspond to your chosen dates. One approach that I would use first is the ismember function. If all goes well, the ‘RowIdx’ vector will have the row indices of your chosen dates, and you can use them to select your other data.

Hypothetical code:

Lia = ismember(dateArray, date1);
RowIdx = find(Lia);

This is obviously UNTESTED CODE. I’m assuming that the dates in ‘dateArray’ have the same format as those in ‘date1’. They have to have the same format in order for this approach to work.

[Ellyn Gray]

Hi Star Strider! Thank you for your reply. I have included an example of what the data looks like in the first few lines. I will check out the ismember function right away.

'dateArray' looks different from 'date1'. 'date1' format was 201501010000, which did not go over well with Matlab. I used 'dateArray' to change it to '1-Jan-2015 00:00:00 -0800' with datetime function. Then using datetime, I filtered the data for the months I want, but I don't know how to still have these times connected to the corresponding wind speed, visibility, etc

date  windDir  windSpeed  ceilingHeight  visibility   temp  dewpoint  SLP
201501010000  10  16  NaN  10  56  4  1021.2
201501010053  360  14  722  10  54  4  1021.2
201501010153  10  21  722  10  53  4  1021
201501010253  10  24  722  10  53  1  1020.9
201501010353  10  11  722  10  50  6  1021.4
201501010453  350  6  722  10  50  8  1021.4
201501010553  360  16  722  10  50  6  1020.9
201501010600  360  16  NaN  10  50  6  1020.9

[Star Strider]

If the dates and times are in exactly the same format in ‘dateArray’ and ‘date1’, the ismember function (as I implemented it) should have no problem returning the row indices of the dates you selected. After that, the ‘RowIdx’ variable plugs in to your cell array indices, for example:

windDirWinter = windDir{RowIdx};

or:

windDirWinter = windDir(RowIdx);

or something similar for the others as well. This creates new variables for all of your existing ones, corresponding to the ismember results and ‘dateWinter’. They should match the corresponding dates, although I would check a few of them to be certain the indexing is correct.

[Ellyn Gray]

Oh this worked very well! I attached what I did below. The RowIdx now has the row # of all the data I want. However, I'm little confused how to take that number and match it with the other vectors. I'm going to start looking around for something. Not sure if this should be obvious.

[winterFilter] = ismember(dateArray, dateWinter);
RowIdx = find(winterFilter);

[Star Strider]

Great! Without your data, I’ve been doing the best I can.

Nothing was attached. (I’ve been waiting for it to appear.)

To create all the individual variables as equal-length column vectors, do the same operation with each of the vectors:

windDirWinter = windDir(RowIdx);
windSpeedWinter = windSpeed(RowIdx);

and similarly for all the others.

If your data are all numeric (specifically double-precision variables, including the dates that you can create as datevec date vectors for example), you can simply concatenate them into a single numeric matrix, since they are all column vectors of the same length:

winterMtx = [dateWinter, windDirWinter, windSpeedWinter, ... , SLPWinter];

If they are not all numeric ( i.e. different data types ), do the same concatenation but as a cell array instead:

winterCell = {dateWinter, windDirWinter, windSpeedWinter, ... , SLPWinter};

It might not be all that obvious if you haven’t much experience with MATLAB matrix operations, but it is straightforward.

[Ellyn Gray]

Thank you so much, Star Strider!!! This worked splendidly. I attached the data (sorry I didn't know that was common practice) but your recommendation worked very well. I can't tell you how much I appreciate it. I hope I can pass it along to someone else when I am no longer a matlab-newbie.

[Star Strider]

My pleasure!

You’re always welcome to share your expertise here. We were all newbies once, and I learned much by reading other Answers and solving interesting problems here that I might not otherwise have encountered in my own research areas.

I notice your data are from Redding Municipal Airport. I never flew into there myself, so I’m curious to know if you’re also a Private Pilot, since a lot of the information you’re analysing is of interest principally to pilots. (I’m ASEL/IRA here!)

[Ellyn Gray]

I'm actually a graduate student doing atmospheric science research. We tend to use a lot of aviation data, because there are such long, publicly available records with high spatial resolution. We're all quite grateful to the aviation community for them!

Thank you for your tips about checking out other questions. I will definitely do that.