Your problem looks like you find the EM field with 3 layers and 2 interfaces. The coefficients A, B, C, D gives the strength of different field components, they are linearly related because EM is linear. In order to solve with without getting trivial solution
You must for example fix one of them arbitrary, e.g.
That left you with 4 linear equations and 3 unknowns. Then in order such system to have solution, you must have the linear system to have dependent equation, meaning determinant of the matrix is 0.
Write it down this probably gives the equation you looks for.
I don't have symbolic tbx to do this kind of calculation to confirm, it would be possible to carry out numerically me think.