ROS2 Packages on NVIDIA Jetson
Ease of use and deployment have made the NVIDIA Jetson platform a logical choice for developers, researchers, and manufacturers building and deploying robots.
ROS2 Package for Accelerated AprilTags
AprilTags is a common fiducial tag that makes it easy to derive its 6DOF pose of in a scene with only a monocular camera. These tags are used in a variety of applications in robotics such as object tracking and visual localization. The faster one can detect the pose of a tag, the faster the closed loop can be for more responsive behaviors. Native ROS2 package wraps NVIDIA's GPU-accelerated AprilTag detector for fast detection of the 36h11 tag family published as a standard TF topic.
The underlying NVAprilTag library is still in development, so please validate its performance for your use cases. Feedback is appreciated.
ROS2 node uses the NVIDIA GPU-accelerated AprilTags library to detect AprilTags in images and publish their poses, ids, and additional metadata. This has been tested on ROS2 (Foxy) and should run on x86_64 and aarch64 (Jetson hardware). It is modeled after and comparable to the ROS2 node for CPU AprilTags detection
For more information on the Isaac GEM this node is based off of, see the Isaac SDK 2020.2 documentation
For more information on AprilTags themselves, the paper and the reference CPU implementation: please check UMICH Apriltag
ROS2 Package for Human Hand Pose Estimation
ros2_trt_pose_hand package is implemented based on
trt_pose_hand, which implements a real-time hand pose estimation and gesture classification using TensorRT.
Here are the key features of the
- Hand Pose message with 21 key-points
- Hand Pose detection image message
std_msgsfor gesture classification with 6 classes [fist, pan, stop, fine, peace, no hand]
- Visualization markers
- Launch file for RViz
ROS2 Package for Human Pose Estimation
ros2_trt_pose package is implemented based on
trt_pose, which enables pose estimation on the Jetson platform. The repository provides two trained models for pose estimation using resnet18 and densenet121. To understand human pose, pretrained models infer 17 body parts based on the categories from the COCO dataset.
Here are the key features of the
pose_msgssuch as count of person and
person_id. For each
person_id, it publishes 17 body parts.
- Provides launch file for easy usage and visualizations on Rviz2:
- Contains a Jetson-based Docker image for easy install and usage
ROS2 Package for PyTorch and NVIDIA TensorRT
There are two packages for classification and detection using PyTorch, each with their corresponding TRT versions implemented. These four packages are a good starting point for roboticists using ROS 2 to get started with deep learning using PyTorch.
TensorRT has been integrated into the packages with the help of torch2trt for accelerated inference. It generates a runtime engine which is optimized according to the architecture of the network and the deployment device.
The main features of the packages are as follows:
- For classification, you can select from various ImageNet pretrained models, including Resnet18, AlexNet, SqueezeNet, and Resnet50.
- For detection, MobileNetV1-based SSD is currently supported, trained on the COCO dataset.
- The TRT packages provide a significant speedup in carrying out inference relative to the PyTorch models performing inference directly on the GPU.
- The inference results are published in the form of
- On running the node, a window is also shown with the inference results visualized.
- A Jetson-based Docker image and launch file is provided for ease of use
ROS 2 Package for DeepStream SDK
The DeepStream SDK delivers a complete streaming analytics toolkit to build end-to-end AI-based solutions using multi-sensor processing, video, and image understanding. It offers support for popular object detection and segmentation models such as state of the art SSD, YOLO, FasterRCNN, and MaskRCNN.
NVIDIA provide ROS 2 nodes that perform two inference tasks based on the DeepStream Python Apps project as follows:
- Object detection: Four classes of objects are detected: Vehicle, Person, RoadSign, and TwoWheeler.
- Attribute classification: Three types of attributes are classified for objects of class Vehicle: Color, Make, and Type.
These publisher nodes take single or multiple video streams as input from camera or file. They perform inference and publish results of detection and classification to different topics. We also provide sample ROS 2 subscriber nodes that subscribe to these topics and display results in
vision_msgs format. Each inference task also spawns a visualization window with bounding boxes and labels around detected objects. Additional inference tasks and custom models can be integrated with the DeepStream pipeline provided in this project.
ROS2 Package for Jetson Stats
ros2_jetson_stats package is a community build package that monitors and controls your Jetson device. It can run on your terminal and provides a Python package for easy integration in Python scripts. Take advantage of the
ros2_jetson_stats library and build ROS 2 diagnostic messages and services.
ros2_jetson_stats package features the following ROS 2 diagnostic messages:
- GPU/CPU usage percentage
- EMC/SWAP/Memory status (% usage)
- Power and temperature of SoC
You can now control the following through the ROS 2 command line:
- Fan (Mode and Speed)
- Power model (nvpmodel)
jetson_clocksYou can also provide a parameter to set the frequency of reading diagnostic messages.
Deep Learning Nodes for ROS/ROS2
This repo contains deep learning inference nodes and camera/video streaming nodes for ROS/ROS2 with support for Jetson Nano/TX1/TX2/Xavier NX/AGX Xavier and TensorRT.
The nodes use the image recognition, object detection, and semantic segmentation DNN's from the
jetson-inference library and NVIDIA Hello AI World tutorial, which come with several built-in pretrained networks for classification, detection, and segmentation and the ability to load customized user-trained models.
The camera/video streaming nodes support the following input/output interfaces:
- MIPI CSI cameras
- V4L2 cameras
- RTP / RTSP
- Videos & Images
- Image sequences
- OpenGL windows
ROS Melodic and ROS2 Eloquent are supported, and the latest version of JetPack is recommended.
This work is licensed under a Creative Commons Attribution 4.0 International License