If you really want help with this problem, put together a small test suite that gives some inputs and some sample outputs (that you tabulate by hand) and wait for someone to produce a MATLAB function to do what you want.
This is an interesting problem. Acting as an administrator, I cleaned it up and added some tests.
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!