Problem with processing large data set

1 view (last 30 days)
I want to train a BiLSTM-CNN (Bidirectional Long Short-Term Memory Convolutional Neural Network) with 203457 timeseries. I am trying to store them in a table with one line and 203457 columns, each cell containing an array with 100,000 values, but my server freezes whenever I run my code. I don’t get any error messages so I am not sure what the problem is and how to fix it. Do you have any idea why this happens and how to solve this?
Thank you!

Answers (1)

Eric Sofen
Eric Sofen on 14 Dec 2020
I'm not sure of the BiLSTM-CNN input requirements, but for starters you're probably better off storing those timeseries as rows in a table rather than variables (columns). You've got some 20 billion total values, so you may be hitting memory limits. You may want to look into using a tall variable for out-of-memory computation.
  1 Comment
Andrei-Ieronim Constantinescu
Thank you very much for the response. Meanwhile, I found a forum suggesting that a fileDatastore is good for processing MAT files and big data due to the fact that it is a type of datastore. I will try both options and update here wich one is better.

Sign in to comment.

Categories

Find more on Sequence and Numeric Feature Data Workflows in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!