How do we know that MATLABs algorithms are working properly?

As MATLAB is closed source, it is impossible to look at what the software is doing unter the hood. So there is no guarantee that every subtle bug is fixed when a new feature is introduced. I was just wondering if there is some public rewiewing or testing procedure or how correctness of an algorithm is provided. Sorry for being that suspicious :)

Answer 1

[Roger Stafford]

Yes, there are many people, including myself, who would welcome this kind of transparency in all such computing systems! In my opinion the same goes for the logic built in to all computers. Such crucially important matters really ought to be made known to the public. How do we know for sure that there aren't errors in them, as in fact has actually happened in the past with an Intel Pentium chip? See

 http://www.emery.com/1e/pentium.htm

[Eli Duenisch]

thanks for the link, very nice story

Answer 2

[Jan]

Although Matlab is closed source, many functions rely on well known code. Examples:

• Matlab suffered from the same SPRINTF/DTOA bug as other libs also, see http://www.mathworks.com/matlabcentral/newsreader/view_thread/269939 and http://seclists.org/fulldisclosure/2010/Jan/124.
• ACOS is calculated by FDLIBM, as explained in the documentation.
• Core functions are subject to bugs repeatedly, e.g. SUM has been multi-threaded in 2009a with accumulating the partial results in the order of the finishing threads - and this means randomly. Therefore the results could differ from run to run due to rounding errors.
• The JIT causes troubles, see e.g. Answers: Setting breakpoints changes answers
• Older versions of Matlab used an instable implementation of LOG10. The source code was not published (as far as I know), but the problem was discussed exhaustively, see e.g. http://www.cs.berkeley.edu/~wkahan/LOG10HAF.TXT

When such bugs appear in the published releases, the testing team has obviously overseen them. The corresponding unit-tests must be missing or incomplete. This is a great pity, but cannot be avoided, because bugs are an inherent feature of large complex systems.

The Matlab users will find such bugs and report it either directly to the technical support, or they post it here in the forum.

The list of published bugs (see e.g. the bugs FAQ) follow this policy:

 Published bug reports are for bugs that have been reported in this system that may be of interest to customers, based on general use of our products.

As far as I understand this, there are also known bugs, which are not published, because TMW thinks, that they are not in the interest of the customers. Unfortunately the customers do not agree from time to time and it is hard to define, what the "general use of [the] products" exactly cover. In all of these cases (5 times for me personally) the TMW team has been very active in clearing the problem and finding workarounds. In addition this forum and CSSM help frequently users to find out, why an assumed bug is an expected feature: While bugs are the nature of software, misunderstandings are the nature of human users.

[Eli Duenisch]

Thank you for providing this - it clearifies for me how bug fixing is done for MATLAB. TMW publishes the known (and interesting to the customer) bugs here http://www.mathworks.de/support/bugreports/

Answer 3

[James Tursa]

Well, ultimately how do you know that any language (C, Fortran, Python, MATLAB, etc) is compiling/running your code correctly? You trust that the vendors are doing their best to create a quality bug-free product, you become active in the related forums to keep aprised of known bugs and workarounds, and you do your own testing.

[Daniel Shub]

MATLAB is a little harder to trust since it is closed source, TMW only chose to enter a subset of existing bugs in the public bug tracking system, and only a subset of the changes are documented in the release notes.

[Jan]

The code Matlab's ACOS calculation is public. But reading the source will not allow to estimate the quality of the results. Even if the method to construct the constants and the code for the exhaustive testing is published also, the term "quality of the results" is not perfectly defined: Is it a bug, when the last significant digits differs from the correct value, when the runtime is extremely fast?

I took ACOS as example, because I've struggled with it: I've examined the problems of calculating the angle between two vectors. Using ATAN2 is more stable, but ACOS is faster. I've measured the differences between both methods and to my surprise I've found not only the expected deviation between ATAN2 and ACOS, but also between an ACOS implementation as M and as MEX file, which used the compilers ACOS command. Then I've used the C-code of ACOS from FDLIBM, and there was still a difference to the M-code. But why?

Then I've experimented with different compilers: LCC2.4 (shipped with Matlab), LCC3.8 (from the net), OpenWatcom, Borland, MSVC 32/64 with and without SSE. And all compilers produced different results for the same code! The instability of ACOS amplified tiny differences. But which of the results is "correct" then?!

At this point I gave up to search for the truth and decided to avoid any dependencies to any assumed exact results: I cannot prove in a unit-test, if the results are correct. But I can define and control the required accuracy: For the calculation of the relative attitude of two 3D-bodies, whose positions are measured with an accuracy of 0.1mm, 1e-6rad for the angle between two vectors are enough for the motion analysis.

But of course, when I want determine the position of the moon by triangulation with an accuracy of centimeters, more sophisticated controls of the underlying trigonometric functions are demanded.

My conclusion: Open or closed source is no argument for the degree of trustworthiness. You need to test individually, if the combination of libraries, compilers, Matlab- and OS-versions satisfies the required accuracy of an application. Even the term "bug" has a relative character then and the decision of TMW, if a "bug may be of interest to customers" is doubtful: How could they know, if my programs are affected?