mxCreateSharedDataCopy no longer supported in R2018a

2 views (last 30 days)

This is an error message I got when compiling my C-mex file

Error using mex Function mxCreateSharedDataCopy not supported. Either switch to using the more efficient MATLAB Data API available in C++ MEX Applications, or rerun MEX with the -R2017b command line option to use the R2017b API.

Can't see anything remotely equivalent in MATLAB Data API. This function is very important to me to make faster MEX code. What is the alternative if we want to use R2018a (interleaved complex) and create shared data copy ?

  2 Comments
James Tursa
James Tursa on 19 Apr 2018
Edited: James Tursa on 19 Apr 2018

I won't be able to update any of my most important mex code to run under R2018a without this function! This is bad ... very bad ...

Bruno Luong
Bruno Luong on 19 Apr 2018
I request TMW to reintroduce this function in the API.

Sign in to comment.

Answers (4)

Ned Charles
Ned Charles on 9 May 2018
We’d like to clarify that MEX-files built in R2018a using default options to the MEX command will not see a build error if they use the undocumented API mxCreateSharedDataCopy. Such MEX files work in both R2018a and older releases (and future releases). It is not possible to build MEX-files that both opt into the new interleaved complex API and use the undocumented mxCreateSharedDataCopy. MEX-Files that opt into Interleaved Complex only work in R2018a and future releases.
It is possible to build MEX files that both use interleaved complex data and have fully documented support for copy-on-write behavior. Arrays created with the C++ MATLAB Data API automatically use copy-on-write without any special handling and knowledge needed by the programmer. Such MEX files will work in R2018a and future releases. See https://www.mathworks.com/help/matlab/matlab_external/c-mex-functions.html for more information on C++ MEX. To Bruno’s question, not only can a C++ MEX function share data with input arguments, but copies of arrays created inside the MEX file will also use copy-on-write. See https://www.mathworks.com/help/matlab/matlab_external/copy-cpp-api-matlab-arrays.html.
We have not documented or supported copy-on-write in MEX-Files because we believe our internal C API would place too high a burden on the MEX-File author. Without the abstraction and implementation-hiding provided by C++, the C API would require cooperation among all the code using the arrays being shared so that copies are made before writing to the array data. With the C++ API, it is possible to fully support optimization without a MEX-File author needing to take any kind of special action to get correct behavior. MathWorks encourages the documented C++ API because it provides the performance of copy-on-write without the hassles of managing it and the risks of getting that management wrong which can lead to bugs that may be difficult to detect, diagnose, and fix.
  1 Comment
James Tursa
James Tursa on 19 Feb 2020
Edited: James Tursa on 24 Feb 2020
"... With the C++ API, it is possible to fully support optimization without a MEX-File author needing to take any kind of special action to get correct behavior ..."
Unless the correct behavior is to avoid large data copies happening in the background without direct control from the user.
As a mex programmer, I don't want that happening without being able to control when it happens. Unless inspection methods are provided so that the programmer can detect when a variable is shared (isaSharedDataCopy, isaReferenceCopy, etc.) so that downstream algorithms can avoid large deep data copies, this new C++ interface has very limited use for me ... maybe even no use at all. I need to be able to know when an action I am about to take will or will not result in a large deep data copy. As I currently read the doc, I don't see any methods that allow me to do this. I know TMW has put a lot of thought and effort into this new C++ interface, but you still have left many of us out in the cold until you provide, at a very minimum, these inspection methods.

Sign in to comment.


Jan
Jan on 20 Apr 2018
Edited: Jan on 20 Apr 2018
[More a comment than an answer:]
mxCreateSharedDataCopy was not documented. There might be an undocumented method in the 2018 Data API also.
It would be nice, if MathWorks provides an exhaustive documentation concerning the inplace access of array in the MEX level. I'm waiting for this since 1999.
  3 Comments
Jan
Jan on 20 Apr 2018
I do not want my answer to be accepted also, because the best answer will be written by MathWork's support team, preferably containing a link to the updated documentation.
This answer might be more accurate: "The drop of support shows, why this feature has not been documented yet. Replacing it by another undocumented method implies the danger of additional incompatibilities in the future. Therefore currently only deep data copies are reliable." But this would not match the needs of programmers.
Thanks, Bruno, for this important question. I hope that a discussions demonstrates the need of a documented method for shared data copies.
James Tursa
James Tursa on 27 Apr 2018
Edited: James Tursa on 27 Apr 2018
So, it is notable that the error message is not this
"... unresolved external symbol ..."
But it is this
"... mxCreateSharedDataCopy not supported ..."
So it is not the case that the function isn't necessarily there and the linker couldn't find it. What appears to be happening is that the mex command itself is stopping the build because it sees you are trying to link with this particular function.
The creation of shared data copies functionality is still present in MATLAB (easily verified by a few simple tests at the command line with "format debug"), and in fact the mxCreateSharedDataCopy function still looks like it is present in the libmx file, but the mex command is specifically preventing you from linking to it. So this appears to be a deliberate decision by TMW, not an inadvertent omission. This may be the result of the fact that the mxArray header was changed (e.g. the pi pointer was removed) and TMW not wanting to write a new function that is R2018a mxArray compatible, or perhaps TMW is trying to steer you towards their C++ DATA API. Either way this leaves the C programmers out in the cold with no alternative API functions to use.
As a side note, the undocumented mxCreateReference API function was actually deleted from the libmx file back in R2014a. This was very useful in mex functions when dealing with cell and struct variables.

Sign in to comment.


Cris Luengo
Cris Luengo on 2 May 2018
Edited: Cris Luengo on 9 May 2018
Note that the new C++ interface (introduced in R2018a) has documented ways of creating shared data copies:
I think, going forward, we all need to move to the C++ interface starting with R2018a. Up to version R2017b we use the C interface with separate real and imaginary components, and starting with R2018a we use the C++ interface with interleaved complex values.
I have not gotten started with this new interface yet, but it seems to simplify use significantly. I'm actually looking forward to try it out! :)
  1 Comment
Bruno Luong
Bruno Luong on 4 May 2018
This article does seem to be related to the question, it merely state the rule to create a new copy when data is modified (because MATLAB can have another instant of variable that share the data with the input).
The question I ask is to be able to create a new instant of mxArray that share data with an existing variable, a mechanism TMW keeps for their internal uses. Pity. Many of my MEX codes are now become suddenly inefficient or obsolete.

Sign in to comment.


Bruno Luong
Bruno Luong on 22 Sep 2018
Edited: Bruno Luong on 28 Oct 2020
An alternative solution would be to hack mxArray internal structure
/* File "mxInternals_R2018B.h" */
/* Hacked internal MXARRAY DEFINITION */
/* Matlab version: R2018b */
#define NumericalMask 0x00000080
#define ComplexMask 0x00000800
#define SparseMask 0x00000010
typedef struct mxArray_tag mxArray_tag;
struct mxArray_tag {
mxArray_tag *B_Xlink;
mxClassID classID;
unsigned int unknown1;
mxArray_tag *F_Xlink;
mwSize ndims;
unsigned int refcount; /* Number of mxArray* objects that point to this structure */
unsigned int Mask; /* See some of the mask values defined above */
union {
mwSize rows; /* Valid if the array has <= 2 dimensions */
mwSize *pSize; /* vector of size, in heap, Valid if the array has > 2 dimensions */
} size;
mwSize cols; /* If the array has > 2 dimensions, this is actually numel(X) / size(X,1) */
union {
struct {
void *Data;
unsigned int unknown0;
unsigned int unknown1;
unsigned int unknown2;
unsigned int unknown3;
unsigned int unknown4;
unsigned int unknown5;
} number_array;
struct {
void * Data;
mwIndex * pRowIndex;
mwIndex * pColFirstElement; /* pColFirstElement[j] is the index of the first entry in column j (see mxSetJc). The number of non-sparse elements is pColFirstElement[cols] */
mwSize nzmax; /* The number of non zero elements allocated for the sparse array */
} sparse_array;
struct {
mxArray_tag ** Data;
unsigned int unknown1;
unsigned int unknown2;
unsigned int unknown3;
unsigned int unknown4;
unsigned int unknown5;
unsigned int unknown6;
unsigned int unknown7;
} cell_array;
struct {
mxArray_tag ** Data;
void * pFieldNames; /* N.B. I have no idea how field names are stored */
unsigned int unknown1;
unsigned int unknown2;
unsigned int unknown3;
unsigned int unknown4;
unsigned int unknown5;
} struct_array;
} data;
unsigned int unknown3;
unsigned int unknown4;
unsigned int unknown5;
unsigned int unknown6;
unsigned int unknown7;
};
Then create shared mxArray like this (more complicated data can be handle along this line):
/*
* B = mxCreateSharedMatrix2018(A)
*
* INPUT: A is double matrix real or complex, (i.e., dim=2, not checked)
*
* Create an array B (mxArray differents than A) but shares data pointer with A
*
* >> mex -R2018a mxCreateSharedMatrix2018.c
*/
#include "mex.h"
#include "mxInternals_R2018B.h"
#define A_IN prhs[0]
#define A_OUT plhs[0]
void mexFunction( int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
size_t m, n;
mxComplexity ComplexFlag;
mxArray_tag *Ain, *Aout;
if (nrhs < 1) mexErrMsgTxt("mxCreateSharedMatrix2018: one input required.");
ComplexFlag = mxIsComplex(A_IN) ? mxCOMPLEX : mxREAL;
m = mxGetM(A_IN);
n = mxGetN(A_IN);
A_OUT = mxCreateDoubleMatrix(0,0,ComplexFlag);
mxSetM(A_OUT,m);
mxSetN(A_OUT,n);
Ain = (mxArray_tag*)A_IN;
Aout = (mxArray_tag*)A_OUT;
/* Update cross-link, insert Aout as back of Ain */
Aout->B_Xlink = Ain->B_Xlink;
Aout->F_Xlink = Ain;
if (Aout->B_Xlink) Aout->B_Xlink->F_Xlink = Aout;
else Aout->B_Xlink = Ain;
if (Ain->F_Xlink==NULL) Ain->F_Xlink = Aout;
Ain->B_Xlink = Aout;
/* Shared data */
Aout->data.number_array.Data = Ain->data.number_array.Data;
/* It seems I do not need to update refcount of Ain and Aout */
}
  9 Comments
Duc Le
Duc Le on 30 Dec 2022
Hi @Bruno Luong @James Tursa - are you still working on this? I've just had a bit of time to play around and it looks like the mxArray structure doesn't seem to have changed much since 2020.
It looks like that instead of the cross-links, there is now just a pointer to a counter. All shared arrays have the same pointer, and it seems that when an array is destroyed the counter is decremented, and the memory is freed when the counter gets to zero...
The following code seems to work ok with numeric arrays in R2022b.
/* create_shared.c */
#include "mex.h"
#include "matrix.h"
#include <stdint.h>
struct mxArray_header_2020a { // 96 bytes long
int64_t *refcount; // Pointer to the number of shared copies
void *unknown1; // Seems to be zero
int64_t ClassID; // https://mathworks.com/help/matlab/apiref/mxclassid.html
int64_t flags; // ???
union {
int64_t M; // Row size for 2D matrices, or
int64_t *dims; // Pointer to dims array for nD > 2 arrays
} Mdims;
union {
int64_t N; // Column size for 2D matrices, or
int64_t ndims; // Number of dimemsions for nD > 2 arrays
} Nndim;
void *unknown_addr1; // Something related to structs and cells
void *pr; // Pointer to the data
void *unknown_addr2; // ? these address meaning seem to change between
void *unknown_addr3; // ? R2020a, R2021b and R2022b.
void *unknown2; // Seems to be zero
void *unknown3; // Seems to be zero
};
void mexFunction( int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
if (nrhs < 1)
mexErrMsgTxt("One input required.");
mxComplexity ComplexFlag = mxIsComplex(prhs[0]) ? mxCOMPLEX : mxREAL;
void* arr = (void*) prhs[0];
struct mxArray_header_2020a* mx = (struct mxArray_header_2020a*)arr;
if (mx->ClassID < 3 || mx->ClassID > 15)
mexErrMsgTxt("Can only create shared numeric arrays");
mwSize dims[1] = {1};
plhs[0] = mxCreateNumericArray(1, dims, mx->ClassID, ComplexFlag);
/* Now comes the hacking */
void* arro = (void*) plhs[0];
struct mxArray_header_2020a* mxo = (struct mxArray_header_2020a*)arro;
mxo->pr = mx->pr;
mxo->flags = mx->flags;
mxo->Mdims = mx->Mdims;
mxo->Nndim = mx->Nndim;
if (mx->refcount == NULL) {
/* No refcount pointer defined, create our own... memory leak? */
int64_t* rc = malloc(sizeof(int64_t));
*rc = 1;
mx->refcount = rc;
}
mxo->refcount = mx->refcount;
mxo->refcount[0]++;
/* printf("refcount = %i\n", mxo->refcount[0]); */
}

Sign in to comment.

Categories

Find more on Write C Functions Callable from MATLAB (MEX Files) in Help Center and File Exchange

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!