See attached manual convolution.
Yes, it will be very slow. This is because you need to access every single element in the entire window every time the window slides over. Optimized methods realize that, for large windows, the middle (N-2) columns don't change. Only the first column and the last column change. So when you're summing up the products of the matrix and the kernel, you only need to add in the weighted sum of the new column (that just entered the window) and the weighted sum of the last column (which just left the window).
Another method checks to see if the kernels are separable, like an x function times a y function. If it is, you can use the method of separable kernels to just do two 2-D convolutions with 1-D kernels, instead of one 2-D convolution with a 2-D kernel. And that saves a lot of time.
So there are a number of steps you can take to optimize it, each one speeding it up, until you arrive at a highly optimized algorithm that's about as fast as it can be, like I'm sure is already built into MATLAB.