Calibrate Monocular Vision Cameras With Fisheye Lens ==================================================== Fisheye Calibration Basics -------------------------- A fisheye camera is a `pinhole camera `_ equipped with a `fisheye lens `_. Normally, pictures taken by a fisheye camera are extremely distorted. Background theories about fisheye calibration is a kind of complicated. Two sources have been provided in the following for further investigation. - `Matlab Fisheye Calibration Basics `_: This official Matlab Fisheye Calibration implements `Davide Scaramuzza `_'s paper `A Toolbox for Easily Calibrating Omnidirectional Cameras `_ . - `OpenCV's Fisheye camera model `_: According to the `StackOverFlow's Talk `_, current `OpenCV's Fisheye camera model `_ implements `Jean-Yves Bouguet `_'s `Camera Calibration Toolbox for Matlab `_, probably heavily inspired by Juho Kannala and Sami S. Brandt's paper `A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses `_, which is the **UNIQUE** paper related to **fisheye** provided on `Jean-Yves Bouguet's A few links related to camera calibration `_ . Anyway, since we carry out our test based on `OpenCV `_, we'll cite foluma deducted on `OpenCV's Fisheye camera model `_. After the pinhole camera projection, let's denote: .. math:: &a = x/z, b = y/z \\ &r^2 = a^2 + b^2 \\ &\theta = atan(r) In such, fisheye distortion is defined as: .. math:: \theta_d = \theta(1 + k_1\theta^2 + k_2\theta^4 + k_3\theta^6 + k_4\theta^8) ,where angle :math:`\theta` can be of any value. The distorted point coordinates are: .. math:: x' = (\theta_d/r)a \\ y' = (\theta_d/r)b Finally, the pixel coordinates are: .. math:: &u = f_x(x' + ay') + c_x \\ &v = f_yy' + c_y Our target of calibrating a fisheye camera is just to calculate the 4 parameters: :math:`k_1, k_2, k_3, k_4` . Demonstrations -------------- Preparation ^^^^^^^^^^^ Again, our demos of fisheye calibration are based on the `chessboard pattern <./02_calibrate_monocular_vision_cameras_with_narrow_angle_lens.html#classical-black-white-chessboard>`_ and the `cirle-grid pattern <./02_calibrate_monocular_vision_cameras_with_narrow_angle_lens.html#asymmetrical-circle-pattern>`_ respectively, but with a different camera: `JeVois Smart Machine Vision Camera `_ with its `120 degree fisheye lens `_ . .. code-block:: bash ➜ ~ lsusb ...... Bus 001 Device 010: ID 1d6b:0102 Linux Foundation EEM Gadget ...... JeVois Streaming By Python OpenCV ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Code Snippet: jevois_streaming.py """"""""""""""""""""""""""""""""""" .. literalinclude:: ../resource/chapter2/code/jevois_streaming.py :language: python3 :emphasize-lines: 47-50 :linenos: Demo 1: Calibration Based On Classical Black-white Chessboard ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Code Snippet: chessboard_fisheye.py """"""""""""""""""""""""""""""""""" .. literalinclude:: ../resource/chapter2/code/chessboard_fisheye.py :language: python3 :emphasize-lines: 74,99-110 :linenos: Intermediate Images: Chessboard """"""""""""""""""""""""""""""" .. list-table:: * - .. image:: ../resource/chapter2/images/JeVois/chessboard/00.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/00.jpg :alt: 00.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/01.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/01.jpg :alt: 01.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/02.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/02.jpg :alt: 02.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/03.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/03.jpg :alt: 03.jpg * - .. image:: ../resource/chapter2/images/JeVois/chessboard/04.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/04.jpg :alt: 04.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/05.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/05.jpg :alt: 05.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/06.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/06.jpg :alt: 06.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/07.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/07.jpg :alt: 07.jpg * - .. image:: ../resource/chapter2/images/JeVois/chessboard/08.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/08.jpg :alt: 08.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/09.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/09.jpg :alt: 09.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/10.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/10.jpg :alt: 10.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/11.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/11.jpg :alt: 11.jpg * - .. image:: ../resource/chapter2/images/JeVois/chessboard/12.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/12.jpg :alt: 12.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/13.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/13.jpg :alt: 13.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/14.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/14.jpg :alt: 14.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/15.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/15.jpg :alt: 15.jpg * - .. image:: ../resource/chapter2/images/JeVois/chessboard/16.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/16.jpg :alt: 16.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/17.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/17.jpg :alt: 17.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/18.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/18.jpg :alt: 18.jpg - .. image:: ../resource/chapter2/images/JeVois/chessboard/19.jpg :align: center :target: ../resource/chapter2/images/JeVois/chessboard/19.jpg :alt: 19.jpg Results: calibration_chessboard_fisheye.yml """"""""""""""""""""""""""""""""""""""""""" .. literalinclude:: ../resource/chapter2/code/calibration_chessboard_fisheye.yml :language: xml :linenos: Clearly, .. math:: &\begin{pmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix} =\\ &\begin{pmatrix} 3.7163812469674616e+02 & 0. & 3.0622492925810030e+02s \\ 0. & 3.7067642862628497e+02 & 2.2683018293363298e+02 \\ 0. & 0. & 1. \end{pmatrix} .. math:: \begin{pmatrix} k_1 & k_2 & k_3 & k_4 \end{pmatrix} = \begin{pmatrix} 9.3050265697576837e-02 \\ -1.7065824578177742e+00 \\ 6.5970560553632547e+00 \\ -8.1535329116759439e+00 \end{pmatrix}^T Demo 2: Calibration Based On Asymmetrical Circle Pattern ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Code Snippet: circle_grid_fisheye.py """""""""""""""""""""""""""""""""""" .. literalinclude:: ../resource/chapter2/code/circle_grid_fisheye.py :language: python3 :emphasize-lines: 148,153,184-195 :linenos: Intermediate Images: Circle Grid """""""""""""""""""""""""""""""" .. list-table:: * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/00.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/00.jpg :alt: 00.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/01.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/01.jpg :alt: 01.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/02.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/02.jpg :alt: 02.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/03.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/03.jpg :alt: 03.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/04.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/04.jpg :alt: 04.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/05.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/05.jpg :alt: 05.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/06.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/06.jpg :alt: 06.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/07.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/07.jpg :alt: 07.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/08.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/08.jpg :alt: 08.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/09.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/09.jpg :alt: 09.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/10.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/10.jpg :alt: 10.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/11.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/11.jpg :alt: 11.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/12.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/12.jpg :alt: 12.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/13.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/13.jpg :alt: 13.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/14.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/14.jpg :alt: 14.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/15.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/15.jpg :alt: 15.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/16.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/16.jpg :alt: 16.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/17.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/17.jpg :alt: 17.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/18.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/18.jpg :alt: 18.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/19.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/19.jpg :alt: 19.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/20.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/20.jpg :alt: 20.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/21.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/21.jpg :alt: 21.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/22.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/22.jpg :alt: 22.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/23.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/23.jpg :alt: 23.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/24.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/24.jpg :alt: 24.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/25.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/25.jpg :alt: 25.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/26.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/26.jpg :alt: 26.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/27.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/27.jpg :alt: 27.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/28.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/28.jpg :alt: 28.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/29.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/29.jpg :alt: 29.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/30.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/30.jpg :alt: 30.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/31.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/31.jpg :alt: 31.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/32.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/32.jpg :alt: 32.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/33.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/33.jpg :alt: 33.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/34.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/34.jpg :alt: 34.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/35.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/35.jpg :alt: 35.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/36.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/36.jpg :alt: 36.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/37.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/37.jpg :alt: 37.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/38.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/38.jpg :alt: 38.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/39.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/39.jpg :alt: 39.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/40.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/40.jpg :alt: 40.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/41.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/41.jpg :alt: 41.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/42.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/42.jpg :alt: 42.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/43.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/43.jpg :alt: 43.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/44.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/44.jpg :alt: 44.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/45.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/45.jpg :alt: 45.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/46.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/46.jpg :alt: 46.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/47.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/47.jpg :alt: 47.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/48.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/48.jpg :alt: 48.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/49.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/49.jpg :alt: 49.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/50.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/50.jpg :alt: 50.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/51.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/51.jpg :alt: 51.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/52.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/52.jpg :alt: 52.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/53.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/53.jpg :alt: 53.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/54.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/54.jpg :alt: 54.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/55.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/55.jpg :alt: 55.jpg * - .. image:: ../resource/chapter2/images/JeVois/circlegrid/56.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/56.jpg :alt: 56.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/57.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/57.jpg :alt: 57.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/58.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/58.jpg :alt: 58.jpg - .. image:: ../resource/chapter2/images/JeVois/circlegrid/59.jpg :align: center :target: ../resource/chapter2/images/JeVois/circlegrid/59.jpg :alt: 59.jpg Results: calibration_circle_grid_fisheye.yml """""""""""""""""""""""""""""""""""""""""""" .. literalinclude:: ../resource/chapter2/code/calibration_circle_grid_fisheye.yml :language: xml :linenos: Clearly, .. math:: &\begin{pmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix} =\\ &\begin{pmatrix} 4.3351108701524601e+02 & 0. & 2.6920617462414498e+02 \\ 0. & 4.3197528630519076e+02 & 1.9692952627450200e+02 \\ 0. & 0. & 1. \end{pmatrix} .. math:: \begin{pmatrix} k_1 & k_2 & k_3 & k_4 \end{pmatrix} = \begin{pmatrix} 2.8597131872136111e+00 \\ -1.8511466426164975e+01 \\ 4.4720394622220461e+03 \\ -3.8778158662574569e+03 \end{pmatrix}^T **Clearly**, result ``calibration_chessboard_fisheye.yml`` from `Demo 1 <./03_calibrate_monocular_vision_cameras_with_fisheye_lens.html#demo-1-calibration-based-on-classical-black-white-chessboard>`_ and result ``calibration_circle_grid_fisheye.yml`` from `Demo 2 <./03_calibrate_monocular_vision_cameras_with_fisheye_lens.html#demo-2-calibration-based-on-asymmetrical-circle-pattern>`_ are of some difference. Highlights ^^^^^^^^^^ - Fisheye Calibration * `Code snippet in Demo 1 for Fisheye Calibration <./03_calibrate_monocular_vision_cameras_with_fisheye_lens.html#code-snippet-chessboard-fisheye-py>`_ * `Code snippet in Demo 2 for Fisheye Calibration <./03_calibrate_monocular_vision_cameras_with_fisheye_lens.html#code-snippet-circle-grid-fisheye-py>`_ are different from - Narrow-angle Calbration * `Code snippet in Demo 1 for Narrow-Angle Calibration <./02_calibrate_monocular_vision_cameras_with_narrow_angle_lens.html#code-snippet-chessboard-py>`_ * `Code snippet in Demo 2 for Narrow-Angle Calibration <./02_calibrate_monocular_vision_cameras_with_narrow_angle_lens.html#code-snippet-circle-grid-py>`_ mainly in 2 aspects: - ``cv2.calibrateCamera`` -> ``cv2.fisheye.calibrate`` - input data objpoints ``2 dimension`` -> ``3 dimension`` Assignments ----------- Try to calibrate `JeVois Smart Machine Vision Camera `_ with ``cv2.calibrateCamera`` and ``cv2.fisheye.calibrate`` respectively, and tell differences between **undistorted** images from respective generated calibration files.