Can we impose geometry constraints on fisheye lens calibration and rectification?

Assuming that the distorted lines generated by fisheye projection should be straight after rectification, can we use deep neural network to impose explicit geometry constraints onto processes of the fisheye lens calibration and the distorted image rectification?

[cr]

Can it not be done with lens calibration?

[cui]

In cases where calibration is not easy/unknown, such as when there is only one distorted image and the a priori linear geometric relationship in the image is known and can be corrected without the need to obtain an internal camera intrinsic matrix.

For example, how do you obtain a corrected image based on this a priori information alone when the wall edge and parking space lines away from the car perimeter should be straight?