create a parquet file with struct inside it
22 views (last 30 days)
Show older comments
Guys, I'm trying to create a parquet file which has to contain different informations. The problem I'm facing is parquet let me write just infos taken from tables, instead I want to store inside it a struct.
I try to explain better. For example I have a blf file which I have to read with dedicated command (blfread) and then insert the result of blfread inside parquet. Whats I want is having this blfread file as a struct contained in the parquet file, not a table of thousand rows and lot of columns.
From what I know it's something I might achieve using sqlite to create a proper database, but I received infos about using parquet database to manage these situations because it should be more efficient.
Is it really something which can be done or I just have to use sqlite? Final goal is having as result a file database of one row, where every column must contain different infos. One will cotain blf compressed, another will contain infos i might write in a table, etc..
0 Comments
Answers (1)
Kautuk Raj
on 14 Jun 2023
Parquet is a columnar storage format that is optimized for reading and writing large amounts of structured data efficiently. While it is possible to store a struct in a Parquet file, it may not be the most efficient or recommended way to store data.
If you want to store a struct in a Parquet file, you will need to convert the struct to a table or a record batch before writing it to the file. You can convert a struct to a table using the struct2table function in MATLAB, and then write the table to a Parquet file using parquetwrite function.
However, if you have different types of data to store (e.g., binary data, text data, and structured data), it is usually better to store them in separate columns or tables within the Parquet file. You can then use SQL queries to join the tables together or extract the data you need.
Alternatively, as you mentioned, you can use a relational database like SQLite to store your data in a more flexible and scalable way. SQLite allows you to store structured data in tables as well as unstructured data like binary blobs, and you can use SQL queries to retrieve the data in any format you need.
3 Comments
Kautuk Raj
on 19 Jun 2023
Edited: Kautuk Raj
on 21 Jun 2023
You can create a table with columns to store the BLF data as a binary blob (BLOB data type), as well as other columns for the related information. You can achieve this using Python's sqlite3 library or SQLite in MATLAB.
See Also
Categories
Find more on Database Toolbox in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!