any hints on how to reduce the size of this thing?
Hi Anton, here's a hint: You're chopping down P then taking the difference between itself and its circshift. This is very close to just a simple diff() command. Circshift is a useful function, but not quite where you've got it ;)
Think of it this way:
A diff() will get vectors from one point to the next. If any neighbouring vectors point in the exact same direction, they are fair game to remove. More specifically, whenever the direction of the line *changes*, you've found a vertex you want to keep. How can you detect that?
Test  Status  Code Input and Output 

1  Pass 
%% Edge case: no vertices
P = zeros(0,2);
P2 = zeros(0,2);
assert(isequal(simplify_polygon(P), P2));

2  Pass 
%% Edge case: one vertex
P = [1 1];
P2 = [1 1];
assert(isequal(simplify_polygon(P), P2));

3  Pass 
%% Edge case: three vertices (a single line segment)
P = [...
1 1
1 2
1 1 ];
P2 = [...
1 1
1 2
1 1];
assert(isequal(simplify_polygon(P), P2));

4  Pass 
%% Single line segment with multiple vertices
P = [ ...
1 1
2 1
3 1
4 1
5 1
4 1
3 1
2 1
1 1];
P2 = [ ...
1 1
5 1
1 1];
assert(isequal(simplify_polygon(P), P2));

5  Pass 
%% Single line segment, different spacing
P = [ ...
1 1
2 1
4 1
5 1
1 1];
P2 = [ ...
1 1
5 1
1 1];
assert(isequal(simplify_polygon(P), P2));

6  Pass 
%% Rectangle
P = [ ...
1 1
2 1
3 1
4 1
4 2
4 3
3 3
2 3
1 3
1 2
1 1];
P2 = [ ...
1 1
4 1
4 3
1 3
1 1];
assert(isequal(simplify_polygon(P), P2));

7  Pass 
%% Two rectangles separated by line segment
P = [ ...
1 2
1 1
2 1
2 2
1 2
1 3
1 4
1 5
2 5
2 4
1 4
1 3
1 2];
P2 = [ ...
1 1
2 1
2 2
1 2
1 5
2 5
2 4
1 4
1 1];
assert(isequal(simplify_polygon(P), P2));

8  Pass 
%% Nonsimple polygon (figure eight)
P = [ ...
1 1
2 2
3 3
1 3
2 2
3 1
1 1];
P2 = [ ...
1 1
3 3
1 3
3 1
1 1];
assert(isequal(simplify_polygon(P), P2));

9  Pass 
%%
P = [ ...
1 1
2 2
3 3
4 4
5 5
5 4
6 3
8 1
7 1
1 1];
P2 = [ ...
1 1
5 5
5 4
8 1
1 1];
assert(isequal(simplify_polygon(P), P2));

10  Pass 
%% Circle; no points should be removed
theta = linspace(0,2*pi,200);
theta(end) = 0;
x = 20*cos(theta);
y = 20*sin(theta);
P = [x', y'];
P2 = P;
assert(isequal(simplify_polygon(P), P2));

11  Pass 
%% Starting vertex can be removed
P = [ ...
2 1
3 1
3 2
3 3
2 3
1 3
1 2
1 1
2 1];
P2 = [ ...
3 1
3 3
1 3
1 1
3 1];
assert(isequal(simplify_polygon(P), P2));

2572 Solvers
Project Euler: Problem 1, Multiples of 3 and 5
1074 Solvers
Back to basics 22  Rotate a matrix
685 Solvers
Sum of diagonal of a square matrix
1175 Solvers
64 Solvers