HDF5 Insert an Array Field into Compound Dataset

Long story in short, I am trying to save a structure as compound dataset

below long pasted code is a simplified demo code, most fields in my struct are numbers, however, there is one field (see FieldArray) that is an array

function h5ex_t_cmpdWArray
fileName       = 'h5ex_t_cmpd.h5';
DATASET        = 'DS1';
DIM0           = 5;
ADIM0          = 10;

dims = DIM0;

wdata.FieldA   =int32([1153 ; 1184 ; 1027  ;    1313]);
wdata.FieldB    ={'Exterior (static)', 'Intake',...
    'Intake manifold', 'Exhaust manifold'};
wdata.FieldC =[53.23; 55.12; 130.55; 1252.89];
wdata.FieldD    =[24.57; 22.95;  31.23;   84.11];

% NEW: 4x10 
wdata.FieldArray = [randn(1,ADIM0);randn(1,ADIM0);randn(1,ADIM0);randn(1,ADIM0)];

%
%% Create a new file using the default properties.
%
file = H5F.create (fileName, 'H5F_ACC_TRUNC',...
    'H5P_DEFAULT', 'H5P_DEFAULT');

%
%Create the required data types
%
intType   =H5T.copy('H5T_NATIVE_INT');
sz(1)     =H5T.get_size(intType);
strType   = H5T.copy ('H5T_C_S1');
H5T.set_size (strType, 'H5T_VARIABLE');
sz(2)     =H5T.get_size(strType);
doubleType=H5T.copy('H5T_NATIVE_DOUBLE');
sz(3)     =H5T.get_size(doubleType);
doubleType=H5T.copy('H5T_NATIVE_DOUBLE');
sz(4)     =H5T.get_size(doubleType);

% array size
base_type_id = H5T.copy('H5T_NATIVE_DOUBLE');
dimsArr = size(wdata.FieldArray);
arrayType = H5T.array_create(base_type_id, fliplr(dimsArr));
type_id = H5T.copy(base_type_id);
H5T.set_size(type_id, 16);
type_size = H5T.get_size(type_id);
%foo = H5T.get_size(arrayType);
sz(5) = type_size;

%
% Computer the offsets to each field. The first offset is always zero.
%
offset(1)=0;
offset(2:5)=cumsum(sz(1:4));

%
% Create the compound datatype for memory.
%
memtype = H5T.create ('H5T_COMPOUND', sum(sz));
H5T.insert (memtype,...
    'FieldA',offset(1),intType);
H5T.insert (memtype,...
    'FieldB',offset(2), strType);
H5T.insert (memtype,...
    'FieldC',offset(3), doubleType);
H5T.insert (memtype,...
    'FieldD',offset(4), doubleType);
H5T.insert (memtype,...
    'FieldArray',offset(5), type_id);
%
% Create the compound datatype for the file.  Because the standard
% types we are using for the file may have different sizes than
% the corresponding native types, we must manually calculate the
% offset of each member.
%
filetype = H5T.create ('H5T_COMPOUND', sum(sz));
H5T.insert (filetype, 'FieldA', offset(1),intType);
H5T.insert (filetype, 'FieldB', offset(2), strType);
H5T.insert (filetype, 'FieldC',offset(3), doubleType);
H5T.insert (filetype, 'FieldD',offset(4), doubleType);

H5T.insert (filetype, 'FieldArray',offset(5), doubleType);


%
% Create dataspace.  Setting maximum size to [] sets the maximum
% size to be the current size.
%
space = H5S.create_simple (1,fliplr( dims), []);

%
% Create the dataset and write the compound data to it.
%
dset = H5D.create (file, DATASET, filetype, space, 'H5P_DEFAULT');
H5D.write (dset, memtype, 'H5S_ALL', 'H5S_ALL', 'H5P_DEFAULT', wdata);

%
% Close and release resources.
%
H5D.close (dset);
H5S.close (space);
H5T.close (filetype);
H5F.close (file);

The code errored out for line:

H5D.write (dset, memtype, 'H5S_ALL', 'H5S_ALL', 'H5P_DEFAULT', wdata);

detail error see

Error using hdf5lib2
Attempted to transfer too many values to or from the library buffer.
Error in H5D.write (line 100)
H5ML.hdf5lib2('H5Dwrite', varargin{:});
Error in h5ex_t_cmpdWArray (line 112)
H5D.write (dset, memtype, 'H5S_ALL', 'H5S_ALL', 'H5P_DEFAULT', wdata);

There are few questions I am confused about

1. how to insert array into compound datatype
2. how to write

Appreciate a lot

Answer 1

[Aman]

Hi Kate,

I see that you are trying to create a compound dataset, but while doing so, you are facing issues while writing it to an HDF5 file.

This issue arises when you allocate an inappropriate amount of dataspace. Please refer to the below example code where we have allocated two dataspaces for "sampleData" while each "sampleData" sample is of size one.

fileName       = 'h5ex_t_cmpd.h5';
dataset        = 'DS1';
sampleData(1).temperature = 11.32;
sampleData(2).temperature = 42.11;
file = H5F.create (fileName, 'H5F_ACC_TRUNC', 'H5P_DEFAULT', 'H5P_DEFAULT');
memtype = H5T.create ('H5T_COMPOUND', H5ML.sizeof (sampleData(1)));
H5T.insert (memtype, 'temperature',H5ML.hoffset (sampleData(1), 'temperature'),'H5T_NATIVE_DOUBLE');
filetype = H5T.create ('H5T_COMPOUND', 8 );
H5T.insert (filetype, 'temperature', 0 ,'H5T_NATIVE_DOUBLE');
space = H5S.create_simple (1,2, []);
dset = H5D.create (file, dataset, filetype, space, 'H5P_DEFAULT');
H5D.write (dset, memtype, 'H5S_ALL', 'H5S_ALL', 'H5P_DEFAULT', sampleData);
H5D.close (dset);
H5S.close (space);
H5T.close (filetype);
H5F.close (file);

In order to rectify this issue, you either need to make the size of each sample of "sampleData" two or allocate only one dataspace. In order to allocate only one dataspace in the below manner,.

space = H5S.create_simple (1,1, []);

I hope it helps!