4.3. Image Capture and Streaming

This section provides an overview of functionality available on NVIDIA Jetson and a few examples of basic use-cases.

4.3.1. V4L2

V4L2 (Video4Linux2) is a framework within the Linux kernel which supports camera streaming.

4.3.1.1. v4l-utils

v4l-utils is a package which provides various userspace utilities for interacting with V4L2 drivers. Typically, it is installed by installing d3-jetson-util, but it can also be installed separately:

sudo apt update
sudo apt install v4l-utils

To start a camera stream, run:

v4l2-ctl -d <position> --stream-mmap
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps

Note

<position> refers to the number of the video node. For example, /dev/video0 would be position 0.

To set camera controls, run:

v4l2-ctl -d <position> \
     -c sensor_mode=<mode number> \
     -c bypass_mode=0 \              # Override Argus' control
     --set-fmt-video=width=<frame width>,height=<frame height>,pixelformat=<pixel format>

Setting camera controls can be combined with streaming. For example: v4l2-ctl -c sensor_mode=2 --stream-mmap. For a full list of controls, run v4l2-ctl -d <position> --all.

Note

Most camera drivers do not support changing frame_rate directly. It is typically controlled by sensor_mode.

4.3.1.2. Custom Applications

The Linux kernel provides documentation for developing custom V4L2 applications here.

4.3.2. libargus

Argus is NVIDIA Jetson’s Image Signal Processor (ISP). In contrast to V4L2 which provides raw image data direct from the camera, Argus processes the raw image data by applying various algorithms like color correction, noise reduction, white balance, and more.

Note

Argus only supports cameras that output raw Bayer data. Some cameras output already-processed data like D3 Embedded’s ISX031. Argus does not support these cameras.

4.3.2.1. Sample Applications

We provide a script to install and build NVIDIA’s Argus sample applications, including argus_camera. Simply run sudo d3-build-argus-camera and follow the instructions in the terminal. After the script completes, you can run argus_camera in a terminal window to play around with the cameras and view the image data after being processed by the Argus ISP.

Note

Some sample applications like argus_camera require a display.

For a full list of Argus sample applications, run ls /usr/local/bin/ | grep argus in a terminal window.

4.3.2.2. GStreamer

NVIDIA provides various GStreamer elements for interacting with Argus. Before using them, they must be installed. Run:

sudo apt update
sudo apt install nvidia-l4t-gstreamer

An example GStreamer pipeline using nvarguscamerasrc with D3 Embedded’s IMX390:

# HDR (sensor-mode=3, use sensor-mode=0 for non-HDR)
gst-launch-1.0 -v nvarguscamerasrc sensor-id=0 sensor-mode=3 \
   ! 'video/x-raw(memory:NVMM),format=NV12,width=1936,height=1096,framerate=30/1' \
   ! nvvidconv \
   ! 'video/x-raw,format=I420' \
   ! queue ! xvimagesink

See also

NVIDIA Accelerated GStreamer Guide

4.3.2.3. Custom Applications

Custom libargus applications can be created to capture and process images and video from cameras. We recommend reviewing the Argus sample applications’ source code which comes with the nvidia-l4t-jetson-multimedia-api apt package along with the NVIDIA Jetson Linux Developer Guide. After installing the package, you can find the sources at /usr/src/jetson_multimedia_api/.

4.3.3. Deterministic Camera Nodes and IDs

When you boot a system, the kernel will assign /dev/videoN nodes (used by V4L2) incrementally on a ‘first-probed, first-served’ basis. The order in which drivers probe is not deterministic, meaning the camera at /dev/video0 on first boot may not be the same one after a reboot.

D3 Embedded provides a method on most of its carrier boards/interface cards which can be used to deterministically identify which camera is occupying a /dev/videoN node. For each /dev/videoN node, there will be a corresponding symlink located at /dev/d3-videoM. While the N is random, the M is always the physical port number on the board.

Since Argus does not use /dev/videoN nodes, a slightly modified method has to be used. Internally, Argus assigns sensor IDs based on the alphanumerical order of camera module node names in the device tree (module0, module1, …, module15). For example, if physical ports 0, 1, 2, and 11 are enabled, the respective sensor IDs will be 0, 1, 3, 2 because 11 comes before 2 when alphanumerically sorted. In practice, this can be used by running a directory listing of /dev/, filtering for d3-video, and sorting alphanumerically. For example:

ls -1 /dev | grep d3-video | tr '\n' ' ' | sed 's/d3-video//g'
# Output:    0 1 11 2
# Sensor ID: 0 1 2  3

The sensor ID is the physical port’s zero-based index in the list.