I just solved the problem by writing the function below
function outputimg = removeBorder(inputimg, size)
outputimg = zeros(size,size,3);
grayscale = rgb2gray(inputimg);
%Find upper left element of the foreground
[x,y] = find(grayscale~=204,1);
outputimg = inputimg(x:x+size-1,y:y+size-1,:);
Observing that the grays region (border) pixel intensity is always 204, basically, it detects the first element position that is not equal to 204, that will be the upper left point of the square. Then, since we know the square size of efficient region, we can finally crop the matrix to remove all the border.
The ONLY drawback of the code that may fail is that the foreground cannot contain pixel value that is exactly 204 where in my case, it is acceptable. I also tried to count border size in each side, but found the size is not balanced for some reason, not sure why.
Thank you for @ Benoit_11's help.