Best way to organize a database of simulation log files to be searchable

6 views (last 30 days)
Over the past few years I've developed MATLAB code for simulating the motion of groups of autonomous agents (e.g. robots) employing a variety of feedback control laws (for research purposes). When I run the simulator for a given set of parameters, my simulator generates a "simresultsFinal.mat" log file which contains all of the pertinent data generated by the simulation (e.g. vectors of position data, etc.) as well as a couple of other .mat log files.
My question is if there is an efficient way to organize my database of .mat log files so that they would be searchable based on the parameters used for that particular simulation. My current method is to auto-generate a (Windows) folder with a folder name that includes the values of certain key parameters and then save the corresponding log files in that particular folder. If I later attempt to run a simulation with the same parameters, my code searches to determine if a folder with that name already exists on the path. This feels like a very inefficient and non-scalable way to implement my "database" organization, and it is not able to capture all of the pertinent parameter data because there is a limit to how long a Windows folder name can be. Any ideas on a better way to do this?
  4 Comments
dpb
dpb on 4 Jul 2018
Looks like reasonable approach. I'm not sure how much of an issue the lookup will be if the table size gets really long. You might consider if you could derive a hashing algorithm so that you had a single value to compare as a field.
The other thing that might make sense is to use the categorical data type for fields that are not truly numeric such as your ParamB above example. Even ParamA is a candidate if the levels of factors is limited to some set over a range (like in a designed experiment, say) rather than random over the range of all possible doubles.
Kevin Galloway
Kevin Galloway on 5 Jul 2018
Thanks, that's some helpful feedback. I'll code it all up and see how well it works, and if the lookup time starts to get unwieldy, I'll try some of those approaches.

Sign in to comment.

Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!