There is no mathematical definition of "one angle between 2 3D-bodies". There is more than 1 degeree of freedom. An angle between 2 axes is possible. What is "alpha" and "beta" in acos(cos(alpha)*cos(betha)) ? Why do you assume, that the formula "does not work" for small angles? How are the Euler angles defined in your case?
The ACOS is instable near to 0, but you get deviations in the result by up to SQRT(EPS), about 1-e8. This is no explanation for your observation, that you get +-5 deg, while you expect +-60 deg.
Of course you need the vectors to determine the angle between them. There is no magic abbreviation to determine angles between axes based on Euler angles without computations.
If your Euler Angles are defined in the order XYZ, the Z-axis points to:
M.Z = [sin(E2), -cos(E2).*sin(E1), cos(E2).*cos(E1)];
where E1, E2, E3 are the corresponding Euler angles. For other orders, other formulas are needed. There are 12 different sets of orders, so please post, which one you are using.
After the two Z axes have been calculated, this is a stable method to determine the angle: