[SOLVED] Where does Linux call a camera driver for image delivery

Issue

The Linux driver for OV5640 camera provides a limited functionality.

static struct i2c_driver ov5640_i2c_driver = {
    .driver = {
        .name  = "ov5640",
        .of_match_table = ov5640_dt_ids,
    },
    .id_table = ov5640_id,
    .probe    = ov5640_probe,
    .remove   = ov5640_remove,
};

These functionalities are name, probe and remove.

I am not very familiar with the drivers but I cannot find any part of the driver that renders the pixel so I can store them to an array. I also cannot find any virtual memory address for reading the pixels.

Which part of the code does the captured image data transfer happen in? I am looking for two spots: one in the driver and one in the higher level which calls the driver.

Solution

Short answer

You can capture the video frame from next places:

1. Figure out where is your IPU driver and sniff data there.
2. Write user-space application using V4L2 API to capture the frame.

Second option is preferred and easier.

Detailed answer

There are three drivers in play here:

1. Camera driver: implements the control path over I2C (e.g. starting the streaming)
   Example: "ovti,ov5640"
2. CSI Receiver driver (CSI-2 host controller driver): basically it's a PHY, which converts CSI-2 data format into some internal data format for IPU.
   Example: "fsl,imx6-mipi-csi2"
3. IPU (Image Processing Unit) driver: implements a lot of features, e.g. Image DMA, video capturing, etc.
   Example: "fsl,imx6q-ipu"

• capturing subsystem driver can be separated from IPU driver.
  Example: "fsl,imx-capture-subsystem"

Sometimes CSI RX and IPU drivers are combined into single IP-core (like Video Input Port in TI AM57xx chips). Sometimes video capturing can be implemented in separate driver (although it's part of IPU). That's just a matter of platform architecture you are working on, but the main idea is the same.

Take a look at this picture:

So if you are looking for a place to hook the data, it should be in one of next places:

1. In your IPU driver: driver is probably doing the DMA transfer to RAM, feeding that RAM location to V4L2. That's where you can try to sniff the image data. As you didn't reference your platform, you'll need to find that driver yourself. Look inside of drivers/media/platform/ and in drivers/staging/media/, it should be somewhere there. Also, explore your device tree file, the ov5640 node should reference CSI/IPU driver in port node.
2. In your user-space app: V4L2 user space interface allows you to capture the video frame; see this example for details.

You can try to find your capture driver either by grepping kernel source tree by some known V4L2 constant, e.g.:

$ grep -lIr V4L2_BUF_TYPE_VIDEO_CAPTURE drivers/media/platform/* drivers/staging/media/*

or try to find that out from your device tree file. For example, on iMX6 platform all related nodes look like this:

ov5640: camera@3c {
    compatible = "ovti,ov5640";
    port {
        ov5640_to_mipi_csi2: endpoint { remote-endpoint = <&mipi_csi2_in>; };
    };
};

mipi_csi: mipi@21e0000 {
    compatible = "fsl,imx6-mipi-csi2";
    port@0 {
        mipi_csi2_in: endpoint { remote-endpoint = <&ov5640_to_mipi_csi2>; };
    };
};

ipu1: ipu@2800000 {
    compatible = "fsl,imx6q-ipu";
    ipu1_csi0: port@0 { };
};

capture-subsystem {
    compatible = "fsl,imx-capture-subsystem";
    ports = <&ipu1_csi0>;
};

But I would suggest you to try and capture the video frame from user-space app, like it's done here, as doing so in kernel driver could be difficult task. Also, in user-space app it would be easier to save captured frame to jpg file, like it's done here.

iMX6 resources

There is a lot of documentation for iMX6 w.r.t. camera out there:

• Kernel iMX6 V4L2 documentation: link
• iMX6 TRM (Technical Reference Manual): all docs, direct link
• iMX Linux Reference Manual: link
• Application Notes like this
• Slides like this: HW Accelerated Video Streaming with V4L2 and OpenGL on iMX6

Answered By - Sam Protsenko

Answer Checked By - David Marino (WPSolving Volunteer)