Assuming you really want this to be faster, not an increase in the time required, you can just do this:
If you have the current releae of MATLAB, thus R2017a (or later), then you could have done it like this:
That should be slightly faster for larger matrices, saving some multiplies by zero. For 4x4 matrices, in fact the version that uses diag seems fastest.
A = rand(100);
C = rand(100,1);
B = rand(100);
For 1000x1000 matrices, the difference is more dramatic.
A = rand(1000);
C = rand(1000,1);
B = rand(1000);
Even these computations are O(N^3) in time. But the constant in front is pretty small.
Oh, you state your code was exponentially increasing with size. That is wrong. While you need to learn to preallocate matrices like out, since if you do not, it will be slow. The matrix multiplies that you wrote as loops are O(N^3) in time. But what was worse is that if you do not preallocate the out array, the computation time will grow as O(N^4), since the size of the array will grow as N^2, but the time to dynamically reallocate the memory at each step grows as the square of that.
Regardless, O(N^4) is NOT exponential growth, merely polynomial growth. And you can cut all of that dramatically, merely by learning to use array operations instead of loops.