any hints on how to reduce the size of this thing?
Hi Anton, here's a hint: You're chopping down P then taking the difference between itself and its circshift. This is very close to just a simple diff() command. Circshift is a useful function, but not quite where you've got it ;)
Think of it this way:
A diff() will get vectors from one point to the next. If any neighbouring vectors point in the exact same direction, they are fair game to remove. More specifically, whenever the direction of the line *changes*, you've found a vertex you want to keep. How can you detect that?
Test  Status  Code Input and Output 

1  Pass 
%% Edge case: no vertices
P = zeros(0,2);
P2 = zeros(0,2);
assert(isequal(simplify_polygon(P), P2));

2  Pass 
%% Edge case: one vertex
P = [1 1];
P2 = [1 1];
assert(isequal(simplify_polygon(P), P2));

3  Pass 
%% Edge case: three vertices (a single line segment)
P = [...
1 1
1 2
1 1 ];
P2 = [...
1 1
1 2
1 1];
assert(isequal(simplify_polygon(P), P2));

4  Pass 
%% Single line segment with multiple vertices
P = [ ...
1 1
2 1
3 1
4 1
5 1
4 1
3 1
2 1
1 1];
P2 = [ ...
1 1
5 1
1 1];
assert(isequal(simplify_polygon(P), P2));

5  Pass 
%% Single line segment, different spacing
P = [ ...
1 1
2 1
4 1
5 1
1 1];
P2 = [ ...
1 1
5 1
1 1];
assert(isequal(simplify_polygon(P), P2));

6  Pass 
%% Rectangle
P = [ ...
1 1
2 1
3 1
4 1
4 2
4 3
3 3
2 3
1 3
1 2
1 1];
P2 = [ ...
1 1
4 1
4 3
1 3
1 1];
assert(isequal(simplify_polygon(P), P2));

7  Pass 
%% Two rectangles separated by line segment
P = [ ...
1 2
1 1
2 1
2 2
1 2
1 3
1 4
1 5
2 5
2 4
1 4
1 3
1 2];
P2 = [ ...
1 1
2 1
2 2
1 2
1 5
2 5
2 4
1 4
1 1];
assert(isequal(simplify_polygon(P), P2));

8  Pass 
%% Nonsimple polygon (figure eight)
P = [ ...
1 1
2 2
3 3
1 3
2 2
3 1
1 1];
P2 = [ ...
1 1
3 3
1 3
3 1
1 1];
assert(isequal(simplify_polygon(P), P2));

9  Pass 
%%
P = [ ...
1 1
2 2
3 3
4 4
5 5
5 4
6 3
8 1
7 1
1 1];
P2 = [ ...
1 1
5 5
5 4
8 1
1 1];
assert(isequal(simplify_polygon(P), P2));

10  Pass 
%% Circle; no points should be removed
theta = linspace(0,2*pi,200);
theta(end) = 0;
x = 20*cos(theta);
y = 20*sin(theta);
P = [x', y'];
P2 = P;
assert(isequal(simplify_polygon(P), P2));

11  Pass 
%% Starting vertex can be removed
P = [ ...
2 1
3 1
3 2
3 3
2 3
1 3
1 2
1 1
2 1];
P2 = [ ...
3 1
3 3
1 3
1 1
3 1];
assert(isequal(simplify_polygon(P), P2));

Find all elements less than 0 or greater than 10 and replace them with NaN
11630 Solvers
Make an Ndimensional Multiplication Table
106 Solvers
Back to basics 17  white space
209 Solvers
357 Solvers
198 Solvers