Why is the maximum variable size allowed by the program is exceeded when using Matlab Coder but not when running code directly?

Dear all,

I have encountered an issue with using the Matlab Coder package. The Matlab code that I am trying to convert to C code works perfectly in the Matlab environment. However, when I try to convert it to C code using the Coder package I encounter the following error message:

Maximum variable size allowed by the program is exceeded.

I find this strange as the code works fine when run it as Matlab code. I have already set up Matlab to use the maximum amount of RAM available on my computer (16GB). Can anybody help me with this issue?

Thank you all in advance!

This is the code snippet where the error occurs. The final line of code is the part giving the issue.

Q0  = sparse(nbk*nbb*nbh*nbm*nbr,nbk*nbb*nbh*nbm*nbr);
for i=1:nbk
    for j = 1:nbb
        for h = 1:nbh
            for m =1:nbm
                for r_old = 1:nbr
                    Q0(i+nbk*(j-1)+nbk*nbb*(h-1)+nbk*nbb*nbh*(m-1)+nbk*nbb*nbh*nbm*(r_old-1),k_index(i,j,h,m,r_old,z)+nbk*(b_index(i,j,h,m,r_old,z)-1)+nbk*nbb*(h_index(i,j,h,m,r_old,z)-1)+nbk*nbb*nbh*(m_index(i,j,h,m,r_old,z)-1)+nbk*nbb*nbh*nbm*(r_index(i,j,h,m,r_old,z)-1)) = 1;
                end     
            end
        end
    end
end
Q((z-1)*nbk*nbb*nbh*nbm*nbr+1:z*nbk*nbb*nbh*nbm*nbr,:) = kron(PI(z,:),Q0);
end
As = sparse(speye(nbk*nbz*nbb*nbh*nbm*nbr)-beta*Q);

[Steven Lord]

Where do the variables nbk, nbz, nbb, nbh, nbm, and nbr come from? Are they hard-coded in your code, are they passed into your function as input arguments, are they computed from input arguments?

What are typical values for those variables? If they're all 2 then your sparse matrices aren't going to be that large; if they're all 100 or 1000 they're going to be much, much larger.

What sizes are beta and Q?

What's the density of beta*Q? Is it possible that while speye is sparsely populated that enough of the off-diagonal elements of beta*Q are non-zero so the result is not sparsely populated? Consider, as an extreme example:

A = ones(10);
B = sparse(A);
whos A B

  Name       Size            Bytes  Class     Attributes

  A         10x10              800  double              
  B         10x10             1688  double    sparse    

B is stored as a sparse matrix but it's not sparsely populated since every element is non-zero. So the sparse storage actually consumes more memory than the full storage due to having to store the indices.

[Steven Hoekstra]

nbk, nbz, nbb, nbh, nbm, and nbr are all constants in the range from 2-10. These are fixed during the programme.

beta is a number usually between 0.9 and 0.99.

Q is a sparse Markovian transition matrix with nbz doubles on each row. It is created by taking the Kronecker product of a dense Markovian transition matrix PI with another transition matrix Q0 which has the density of an identity matrix ( one nonzero entry per row).

I have found a strange fix for the problem. If I replace the line which fills in matrix Q with the following line,

Q((z-1)*nbk*nbb*nbh*nbm*nbr+1:z*nbk*nbb*nbh*nbm*nbr,:) = beta*kron(PI(z,:),Q0);

and replace the line that defines As with the following line

As = sparse(speye(nbk*nbz*nbb*nbh*nbm*nbr)-Q);

the problem is solved. The workings of the algorithm do not change.

However, I still do not understand why this "fix" works. I would like to change the situation back to the old situation as this way of solving the problem changes the definition of my variables and obfusciates the meaning of some steps.

The complete code that works is pasted below.

Q0  = sparse(nbk*nbb*nbh*nbm*nbr,nbk*nbb*nbh*nbm*nbr);
for i=1:nbk
    for j = 1:nbb
        for h = 1:nbh
            for m =1:nbm
                for r_old = 1:nbr
                    Q0(i+nbk*(j-1)+nbk*nbb*(h-1)+nbk*nbb*nbh*(m-1)+nbk*nbb*nbh*nbm*(r_old-1),k_index(i,j,h,m,r_old,z)+nbk*(b_index(i,j,h,m,r_old,z)-1)+nbk*nbb*(h_index(i,j,h,m,r_old,z)-1)+nbk*nbb*nbh*(m_index(i,j,h,m,r_old,z)-1)+nbk*nbb*nbh*nbm*(r_index(i,j,h,m,r_old,z)-1)) = 1;
                end     
            end
        end
    end
end
Q((z-1)*nbk*nbb*nbh*nbm*nbr+1:z*nbk*nbb*nbh*nbm*nbr,:) = beta*kron(PI(z,:),Q0);
end
As = sparse(speye(nbk*nbz*nbb*nbh*nbm*nbr)-Q);

Answer 1

[Gowthami]

Hello Steven,

MATLAB will attempt to create a matrix with a large number of elements, as long as that number of elements is less than the maximum number of elements allowed in a matrix.

You are attempting to create a matrix with more elements than the maximum number of elements allowed in MATLAB.

If the matrix you are attempting to create has relatively few nonzero elements, you may be able to create it as a sparse matrix. You can use the SPARSE function and the other sparse matrix manipulation functions to create and manage this matrix. For more information on SPARSE function, type the following command in command window for a list of the sparse matrix manipulation functions.

help sparfun

However, that the limitation on the maximum number of elements still exists, but now it only applies to the nonzero elements of the sparse matrix. If your matrix is not sparse, however, you will need to break it into sections with a number of elements less than the maximum returned by the COMPUTER function.

You can determine this maximum using the COMPUTER function in the following way -

[str, maxsize] = computer;

[Steven Hoekstra]

The amount of elements in the matrix is less than the maximum amount of elements that are allowed in Matlab. Moreover, the code runs when run as Matlab code without any erros. However the code cannot be converted to C code as then Matlab coder gives the error. It is the conversion from Matlab code into C code which gives the error.

When reordering the assignment of the variables the error disappears hence it cannot be the size of the matrix. Additionally, even when the matrix is reduced to a small matrix the error keeps occuring.