• like the USB / Ethernet cable being too fragile
  We needed to connect a USB camera that was under our bot to the main compute within the bot and the only entry was through a penetrator only designed for thinner ethernet wires and we also mostly had lots of high quality ethernet tether so we had to frankenstein a USB -> Ethernet -> USB setup. This worked on a good day but it was also very fragile
  
• the USB 2.0 cable being re-soldered so many times
  it has to be re-twisted now and then to properly transmit data or the bandwidth drops and the framerate goes to like 3 or 5 from an expected 60 and if you didn't know about the cable issue you'd not figure it out
  The insulation had also been cut so the strands didnt have anything holding them together :)
  

How do digital cameras work

Sensor / Bayer Filter

So the camera is made up of a grid of photo cells, these cells can only measure luminance, as in how bright the light hitting it is, this grid is then read either row by row (CCA) or each cell can be read individually (CMOS), Most cameras today follow the CMOS architecture since it easier and hence cheaper to manufacture.

But as you can imagine, a grid of luminence sensors isnt exactly sufficient to percieve colour so this is where colour filters come in. Unlike the human eye which has cones and rods for colour & luminence respectivley, Camera sensors need some way to filter the luminence they recieve to a particular colour spectrum, This is usually done with a bayer filter.

...
&{/} {
	clk_csi0_imx219_fixed: csi0-imx219-xclk {
		compatible = "fixed-clock";
		#clock-cells = <0>;
		clock-frequency = <24000000>;
	};
};

&mcu_gpio0 {
    status = "okay";
};

&main_i2c2 {
	status = "okay";
	pinctrl-names = "default";
	pinctrl-0 = <&main_i2c2_pins_default>;
	clock-frequency = <400000>;

	#address-cells = <1>;
	#size-cells = <0>;

	imx219_0: sensor&#64;10 {
		compatible = "sony,imx219";
		reg = <0x10>;

		clocks = <&clk_csi0_imx219_fixed>;
		clock-names = "xclk";

		pinctrl-names = "default";
		pinctrl-0 = <&csi0_gpio_pins_default>;

		reset-gpios = <&mcu_gpio0 15 GPIO_ACTIVE_HIGH>;

		port {
			csi2_cam0: endpoint {
				remote-endpoint = <&csi2rx0_in_sensor>;
				link-frequencies = /bits/ 64 <456000000>;
				clock-lanes = <0>;
...	

Video4Linux (USB / Standards compliant cameras)

So this is the linux subsystem that is in charge of all video device aquistion, enumeration and handling on linux, It is what lets you configure the camera properties like exposure, auto focus etc and for this article most importantly it lets you query all the devices and shows you all the different streams it supports, a stream is just a valid configuration of FORMAT, RESOLUTION & FPS that the device currently supports and is capable of publishing.

V4l2 Documentation

So for example my laptop webcam supports the following:

$ v4l2-ctl --all
Driver Info:
        Driver name      : uvcvideo
        Card type        : Integrated Camera: Integrated C
        Bus info         : usb-0000:05:00.3-3
        Driver version   : 6.8.12
        Capabilities     : 0x84a00001
                Video Capture
                Metadata Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Media Driver Info:

                                                                                        
                            Bandwidth (log scale)                                       
                                                                                        
     10000 𜶖𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜴆𜶙                      
           ▐                                                     ▐  1080p 30fps 𜴡𜴮🮅𜴮🮅𜴍  
           ▐                                                 ▂𜺣  ▐  1080p 60fps 𜴡𜴮🮅𜴮🮅𜴍  
           ▐                                                 𜵨▌  ▐   720p 30fps 𜴡𜴮🮅𜴮🮅𜴍  
           ▐                                   𜵭𜵈  𜶖▆   ▐█  𜷇𜵨▌𜵡▖▐   720p 60fps 𜴡𜴮🮅𜴮🮅𜴍  
      1000 ▐𜺨                        𜺠▂   𜶖𜵈  ▂𜵨▌ 𜺠𜷕█  ▗▟█▂𜺣𜷈𜵨▌𜵨▌▐                      
           ▐                         ▐█  ▗▟▌▂𜺣𜷈𜵨▌𜵨▐▐█▐█▐▐█𜷈▌𜷈𜵨𜷊𜵨▌▐                      
           ▐                        𜶖𜷥█▗▄▐▐▌𜷈▌𜷈𜵨▙𜵨▐▐▛▟█▐▐𜵊𜷈▌𜷈𜵨𜷈𜵨▌▐                      
           ▐                        ▐▐█▐█▐▐𜵮𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                      
           ▐                        ▐▐▐▐█▐▐𜵨𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                      
Mbps   100 ▐𜺨                    𜵨▌ ▐▐▐▐█▐▐𜵨𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                      
           ▐                    ▄𜵨𜷀▂▐▐▐▐█▐▐𜵨𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                     M
           ▐                    𜷈𜵨▐█▐▐▐▐█▐▐𜵨𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                      
           ▐                    𜷈𜵨▐█▐▐▐▐█▐▐𜵨𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                      
        10 ▐𜺣 𜷊▌                𜷈𜵨▐█▐▐▐▐█▐▐𜵨𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                      
           ▐ 𜵭𜷈▌𜷇𜵈 ▆        𜷊▌  𜷈𜵨▐█▐▐▐▐█▐▐𜵨𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                      
           ▐ 𜵨𜷈𜷀𜷈▌▂█𜺠▂ ▐█  𜵡𜷈▌▂𜺣𜷈𜵨▐█▐▐▐▐█▐▐𜵨𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                      
           ▐ 𜵨𜷈𜵨𜷈▌𜷈█▐█▗▟█▗▄𜵨𜷈▌𜷈▌𜷈𜵨▐█▐▐▐▐█▐▐𜵨𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                      
           ▐ 𜵨𜷈𜵨𜷈▌𜷈𜵊𜷥█▐▐█▐█𜵨𜷈𜵮𜷈▌𜷈𜵨▐█▐▐▐▐█▐▐𜵨𜷈▌𜷈𜵨𜷈𜵨▐▐▐▐█▐▐▐𜷈▌𜷈𜵨𜷈𜵨▌▐                      
         1 ▐▂𜷠𜷈𜷠𜷈𜷀𜷈𜷕𜷀🮅𜷕𜷀𜷕𜷀🮅𜷠𜷈𜷠𜷈𜷀𜷈𜷠𜷀🮅𜷕𜷀𜷕𜷀🮅𜷕𜷀𜷠𜷈𜷀𜷈𜷠𜷈𜷠𜷕𜷀𜷕𜷀🮅𜷕𜷀𜷕𜷈𜷀𜷈𜷠𜷈𜷠𜷀𜷕                      
            H.264H.265 AV1 VP9 MJPNV12/IYUV42YUV44RAW10RAW1RGB24                        
                                                                                        

        Model            : Integrated Camera: Integrated C
        Serial           :
        Bus info         : usb-0000:05:00.3-3
        Media version    : 6.8.12
        Hardware revision: 0x00002510 (9488)
        Driver version   : 6.8.12
Interface Info:
        ID               : 0x03000002
        Type             : V4L Video
Entity Info:
        ID               : 0x00000001 (1)
        Name             : Integrated Camera: Integrated C
        Function         : V4L2 I/O
        Flags            : default
        Pad 0x01000007   : 0: Sink
          Link 0x02000013: from remote pad 0x100000a of entity 'Extension 4' (Video Pixel Formatter): Data, Enabled, Immutable
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
        Width/Height      : 1280/720
        Pixel Format      : 'MJPG' (Motion-JPEG)
        Field             : None
        Bytes per Line    : 0
        Size Image        : 1843200
        Colorspace        : sRGB
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: Rec. 709
        Quantization      : Default (maps to Full Range)
        Flags             :
Crop Capability Video Capture:
        Bounds      : Left 0, Top 0, Width 1280, Height 720
        Default     : Left 0, Top 0, Width 1280, Height 720
        Pixel Aspect: 1/1
Selection Video Capture: crop_default, Left 0, Top 0, Width 1280, Height 720, Flags:
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 1280, Height 720, Flags:
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 30.000 (30/1)
        Read buffers     : 0

User Controls

                     brightness 0x00980900 (int)    : min=0 max=255 step=1 default=128 value=128
                       contrast 0x00980901 (int)    : min=0 max=100 step=1 default=32 value=32
                     saturation 0x00980902 (int)    : min=0 max=100 step=1 default=64 value=64
                            hue 0x00980903 (int)    : min=-180 max=180 step=1 default=0 value=0
        white_balance_automatic 0x0098090c (bool)   : default=1 value=1
                          gamma 0x00980910 (int)    : min=90 max=150 step=1 default=120 value=120
           power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=1 value=1 (50 Hz)
                                0: Disabled
                                1: 50 Hz
                                2: 60 Hz
      white_balance_temperature 0x0098091a (int)    : min=2800 max=6500 step=1 default=4600 value=4600 flags=inactive
                      sharpness 0x0098091b (int)    : min=0 max=7 step=1 default=3 value=3
         backlight_compensation 0x0098091c (int)    : min=0 max=2 step=1 default=1 value=1

Camera Controls

                  auto_exposure 0x009a0901 (menu)   : min=0 max=3 default=3 value=3 (Aperture Priority Mode)
                                1: Manual Mode
                                3: Aperture Priority Mode
         exposure_time_absolute 0x009a0902 (int)    : min=2 max=1250 step=1 default=156 value=156 flags=inactive
     exposure_dynamic_framerate 0x009a0903 (bool)   : default=0 value=1
                        privacy 0x009a0910 (bool)   : default=0 value=0 flags=read-only

On device image pub sub

Once you have obtained your image in memory in some known format (typically RGB, NV12 or YUV422) there will typically be multiple applications that need to use it, and they might all want it in a different format, resolution & framerate etc.

So your publishing system must be able to easily handle creating multiple streams and providing scripts data where it is wanted (eg: ML scripts will typically want a resized image of RGB format in the GPU)

The solution you would most easily reach for is ROS since that is what you have been using for all your other sensors & data. But images & large data in general is a place where ROS really does not shine. In my experience it will reliably transfer frames but a 30FPS is typically a 15-20FPS frame stream after going through ROS.

By default ROS transfers image raw over the network, a 720p RGB image becomes 1280 * 720 * 3 + 56 = ~2.7MB bytes (the 56 is the frame header)

For 60fps this demands a bandwidth of 2.8 * 60 / 1000 = ~165MB/s = ~1.33Gbps purely for a single image stream & typically you will have multiple streams based on the number of nodes working on it and possible streams of multiple cameras and different resolutions, formats etc.

This becomes better with image_transport but this isn't directly supported in rclpy so almost always you end up with the raw stream in my experience.
(There is support added but not everyone architects their scripts with it / around it) - and then I have found it to be kind of a stop-gap solution that doesn't fully solve the problem every time.

I wish I could show this with benchmarks but its something that happens when the system is under load of other things (like DAQ, control loops etc) and typically embedded systems less apable from your typical laptop. I should capture a pcap of a system while it is running.

Typically also in image pipelines you want to avoid copies and ROS does about 3-4 copies per image. Obviously all these latencies compound with larger images and higher framerates.

This framerate that fluctuates under load is something I have been chasing a solutions for my entire time in Dreadnought and these are the solutions I have found that work

iceoryx (shared memory, zero copy)

Even just avoiding sending the image over the network and keeping it on the same device and trying to avoid copies can make a large difference, You can also _technically_ do this with the ROS2 RMW backends CycloneDDS & FastDDS but I have not that work well for me.

So I have mostly been using iceoryx as a separate pub-sub system alongside ROS for larger data (like laser profiler batches) with a config like this:

[general]
version = 1

[[segment]]

    # ── Small pools (iceoryx internal: service discovery, heartbeats, routing) ──
    [[segment.mempool]]
    size  = 128
    count = 1000

    [[segment.mempool]]
    size  = 1024
    count = 500

    [[segment.mempool]]
    size  = 16384
    count = 200

    # ── camera payload pool ─────────────────────────────────────────────
    # Covers 1440×1080 BayerRG8 (1.5 MB + iceoryx overhead). Increase size to
    # ~16 MiB if running 2448×2048 RGB8 cameras; reduce count if RAM is tight.
    [[segment.mempool]]
    size  = 2097152     # 2 MiB
    count = 300         # 300 × 2 MiB = 600 MB	

motion-jpeg (mjpeg)

When trying to display the stream you either need a custom viewer script (ROS has this with rviz & image_view) or the format needs to be one of mjpeg or rtsp.

Even when trying to display your live video stream in the browser or any kind of front end your best bet without streaming to the cloud is mjpeg or WebRTC - if you are streaming to the cloud then you can transcode and display it however you live, typically I think HLS is used.

But displaying with webrtc is a whole _process_ in of itself, since typically the flow looks something like

class MjpegStreamer {
public:
  MjpegStreamer(int port, int quality)
      : port_(port), quality_(quality), running_(false), listenFd_(-1),
        currentFrameId_(0) {}

  ~MjpegStreamer() { stop(); }

  void start() {
    if (running_.exchange(true))
      return;
    acceptThread_ = std::thread([this] { acceptLoop(); });
  }

  void stop() {
    if (!running_.exchange(false))
      return;
    if (listenFd_ >= 0) {
      ::shutdown(listenFd_, SHUT_RDWR);
    }
    frameCv_.notify_all();
    if (acceptThread_.joinable()) {
      acceptThread_.join();
    }
    if (listenFd_ >= 0) {
      ::close(listenFd_);
      listenFd_ = -1;
    }
  }

  int getPort() const { return port_; }

  void pushFrame(const cv::Mat &bgr) {
    if (bgr.empty()) {
      return;
    }

    std::vector<uint8_t> jpegBuf;
    std::vector<int> params = {cv::IMWRITE_JPEG_QUALITY, quality_};
    cv::imencode(".jpg", bgr, jpegBuf, params);

    {
      std::lock_guard<std::mutex> lk(frameMtx_);
      frame_ = std::move(jpegBuf);
      currentFrameId_++;
    }
    frameCv_.notify_all();
  }

private:
  static constexpr const char *kHttpHdr =
      "HTTP/1.1 200 OK\r\n"
      "Content-Type: multipart/x-mixed-replace; boundary=mjpeg_boundary\r\n"
      "Cache-Control: no-cache\r\n"
      "Connection: close\r\n"
      "\r\n";

  void acceptLoop() {
    listenFd_ = ::socket(AF_INET, SOCK_STREAM, 0);
    if (listenFd_ < 0) {
      return;
    }
    int opt = 1;
    ::setsockopt(listenFd_, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
    sockaddr_in addr{};
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = INADDR_ANY;
    addr.sin_port = htons(static_cast<uint16_t>(port_));
    if (::bind(listenFd_, reinterpret_cast<sockaddr *>(&addr), sizeof(addr)) 
        0) {
      ::close(listenFd_);
      listenFd_ = -1;
      return;
    }
    ::listen(listenFd_, 8);

    while (running_.load()) {
      sockaddr_in ca{};
      socklen_t cl = sizeof(ca);
      int fd = ::accept(listenFd_, reinterpret_cast<sockaddr *>(&ca), &cl);
      if (fd < 0) {
        break;
      }
      char rbuf[2048];
      int received = ::recv(fd, rbuf, sizeof(rbuf) - 1, 0);
      if (received > 0) {
        std::thread([this, fd] { serveClient(fd); }).detach();
      } else {
        ::close(fd);
      }
    }
  }

  void serveClient(int fd) {
    if (::send(fd, kHttpHdr, strlen(kHttpHdr), MSG_NOSIGNAL) < 0) {
      ::close(fd);
      return;
    }
    uint64_t lastSentId = 0;
    while (running_.load()) {
      std::vector<uint8_t> jpeg;
      {
        std::unique_lock<std::mutex> lk(frameMtx_);
        frameCv_.wait_for(lk, std::chrono::seconds(2), [this, lastSentId] {
          return currentFrameId_ > lastSentId || !running_.load();
        });
        if (!running_.load())
          break;
        if (currentFrameId_ <= lastSentId)
          continue;
        jpeg = frame_;
        lastSentId = currentFrameId_;
      }
      std::string hdr = "--mjpeg_boundary\r\n"
                        "Content-Type: image/jpeg\r\n"
                        "Content-Length: " +
                        std::to_string(jpeg.size()) +
                        "\r\n"
                        "\r\n";
      if (::send(fd, hdr.data(), hdr.size(), MSG_NOSIGNAL) < 0)
        break;
      if (::send(fd, jpeg.data(), jpeg.size(), MSG_NOSIGNAL) < 0)
        break;
      if (::send(fd, "\r\n", 2, MSG_NOSIGNAL) < 0)
        break;
    }
    ::close(fd);
  }

  int port_;
  int quality_;
  std::atomic<bool> running_;
  int listenFd_;
  std::thread acceptThread_;
  std::mutex frameMtx_;
  std::condition_variable frameCv_;
  std::vector<uint8_t> frame_;
  uint64_t currentFrameId_;
};	

--mjpeg_boundary\r\n
Content-Type: image/jpeg\r\n
Content-Length: 48213\r\n
\r\n
[raw JPEG bytes]\r\n

rtsp

Mjpeg is good enough for many usecases but if you want to minimize bandwidth and not have to worry about latency of each individual copy you have to look into video codecs like H264, These leverage the fact that between 2 individual frames not much content actually changes, so you can store just the _delta_ of what exactly changed between 2 full frames and using these deltas and the keyframes you can fully reconstruct the original video.

This is a better explanation of how these modern codecs work, this is for h265 but the same general ideas persist for h264, av1 etc.

Typically the video encoding you choose really depends on what exactly your hardware supports, if you are on Intel you can use the qsv family of encoders, nvidia nvenc and AMD amf, ffmpeg hwaccel guide - There are also generic accelerated encoders that use va-api, dx11, dx12 or vulkan for acceleration but I find that using the device specific api is reliably better.

On my laptop this is available by default (use this script to find for your hardware):

  Tool        Vendor / Backend             Codec   Encoder / Element
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ffmpeg      NVIDIA                       H.264   h264_nvenc
  ffmpeg      NVIDIA                       HEVC    hevc_nvenc
  ffmpeg      Intel/AMD (VAAPI)            H.264   h264_vaapi
  ffmpeg      Intel/AMD (VAAPI)            HEVC    hevc_vaapi
  gstreamer   NVIDIA (nvcodec)             H.264   nvcudah264enc
  gstreamer   NVIDIA (nvcodec)             H.264   nvautogpuh264enc
  gstreamer   NVIDIA (nvcodec)             HEVC    nvcudah265enc
  gstreamer   NVIDIA (nvcodec)             HEVC    nvautogpuh265enc

export OPENCV_FFMPEG_CAPTURE_OPTIONS="fifo_size;500000|overrun_nonfatal;1|fflags;nobuffer|flags;low_delay|framedrop;1|vf;setpts=0"