Robotic Arm System Design

Initial Conditions

Jetson AGX Orin		Datasheet
Livox Mid 360 Lidar		Datasheet
ECon Systems STURDeCAM25		Datasheet
Universal Robots UR7e		Datasheet

Data Pipeline

Vision

Point Cloud

Similar flow only, using NITROS to enable zero copy transfer across our stack.

Notes

• Modern Isaac ROS for the Jetson host, recommends Nvidia SIPL for the camera but doesn't seem to be stable / reliable for the Orin, using older version of Isaac ROS (v3.3)
  
• If using ROS, a 1080p raw image at 65 f/s is:
  1920 × 1080 × 3 × 65 bytes/s ⇒ 404.352 Megabytes/s
  This data is copied multiple times during serialization & deserialization.
  This eats up memory bandwidth.
  
• The TensorRT inference node can automatically use the NITROS image handles w/o any copies, removing need for copy from host to device.
  
• Can run inference on the Jetson's DLA to offload processing from the GPU.
  
• In the LIDAR driver, could use remote DMA if available to directly write the received sensor data to the Orin's memory (accessible by the CPU & GPU).
  
• Can test with using Cyclone DDS with shared memory to reduce latency in the pipeline, especially between 2 regular ROS nodes.
  

System Architecture / Synchronization

• Orin Time Sync | NVIDIA Docs
  
• Generic Timestamp Engine — Jetson Linux Developer Guide documentation
  

Clock Synchronization Architecture

UR7e Motor Controller (RTDE)

The UR7e uses its own clock that I can't figure out how to synchronize, so what I'd compromise with is taking the timestamp from RTDE & using that to convert time from the robot to PTP time domain & vice versa, kind of like this:

t1 = getTime
ur = RTDETime
t2 = getTime

# Assume RTDE timestamp taken at (t1 + t2) / 2 = e

offset = (e / 1e9) - ur
          └── convert from ns to s

Camera Hardware Synchronization

Finding camera time offset

t1 = SystemTime()
tc = getCameraFrame().ts # → Get the next available camera frame
t2 = SystemTime()

offset = ((t1 + t2) / 2) - tc	

ts = offset / 1e3 ; nTs = 1e3

while (true) {
    emitPulse(FSYNC_PIN);
    period += ts * (nTs--);
    sleep(period);
}	

Compute Resource Monitoring

System Health Struct

struct SystemHealth {

    // ── Timing & Synchronization
    int64_t   lidar_ptp_offset_ns;     // from ptp4l — target: < 500'000 ns
    int64_t   camera_tsc_corr_error_ns;// from correlated_timestamp_driver
    int64_t   ur_clock_drift_ppm;      // from URClockTracker
    uint64_t  last_trigger_jitter_ns;  // actual vs intended FSYNC fire time
    uint32_t  camera_frames_dropped;   // rolling 1s window
    uint32_t  lidar_packets_missed;    // rolling 1s window

    // ── AI Inference
    float     inference_latency_ms;    // latest TensorRT forward pass
    float     inference_latency_p99_ms;// 99th percentile over 10s window
    uint32_t  trajectory_buffer_depth;// waypoints ahead of current time
                                       // < 3 = AI struggling
    uint64_t  last_ai_trajectory_ns;   // age of most recent trajectory
    bool      ai_timeout;              // set if > 80ms since last trajectory

    // ── Control Loop
    float     control_loop_jitter_us;  // 500Hz loop period deviation
    uint32_t  rtde_write_failures;     // failed RTDE packets (rolling 1s)
    bool      deceleration_active;     // true = AI timeout engaged

    // ── Thermal & Power
    float     gpu_temp_celsius;
    float     cpu_temp_celsius;
    float     gpu_power_watts;
    uint32_t  gpu_throttle_events;    // non-zero = thermal throttle occurred

    // ── Robot Safety
    bool      estop_engaged;
    float     tcp_velocity_m_s;        // must remain < safety_limit
    float     joint_torque_max_nm;     // highest torque across all joints
    bool      protective_stop_active;  // UR internal safety violation

    // ── System-Level
    float     cpu_load_1s[12];         // per-core load
    uint64_t  core2_max_latency_ns;    // worst RT wakeup latency on isolcpus
    uint32_t  watchdog_misses;         // health monitor heartbeat misses

    // ── Derived Status
    enum class Status { GREEN, YELLOW, RED, ESTOP } overall_status;
};	

Action Matrix

Core Control Logic

Trajectory Estimation Pseudocode

MAX_WAYPOINTS = 64
front = []
back = []
f_c = 0
b_c = 0

atomic<bool> new_trajectory_ready = false;
atomic<uint64> last_ai_update_ns = 0;


def pus(waypts):
    b_c = min(len(waypts), MAX_WPTS)
    back = [...waypts[0:b_c]]
    last_ai_update_ns = now();
    new_trajectory_ready = True;


# Called by controller loop
def maybe_swap():
    if (new_trajectory_ready == False):
        swap(front, back);
        f_c = b_c


def interpolate_at(ts):
    for i in range(0, f_c):
        if front[i].ts <= ts && front[i+1].ts >= ts:
            α = (ts - front[i].ts) / (front[i+1].ts - front[i].ts)
            return lerp(front[i], front[i+1], α)

    return front[f_c+1]	

Core Allocation

• GPU (CUDA Cores): TensorRT inference graph, point cloud voxelization
  
• GPU (DLA Core 0): Primary TensorRT engine (routed via Isaac ROS)
  
• GPU (DLA Core 1): Fallback / secondary model
  
• Kernel boot params: isolcpus=2 nohz_full=2 rcu_nocbs=2
  
• RT thread startup: mlockall(MCL_CURRENT | MCL_FUTURE) + stack prefault
  

Robot Arm Interface

Actions

Services

Safety Handshake