Update to v0.9 release

- Add bindings for image data and tensor metadata.
- Add sample apps for image data usage, custom inference output parsing, USB camera source and RTSP output.

Author: nv-zhliu

.python-app-pipeline.png

@@ -1,12 +1,13 @@
 # Frequently Asked Questions and Troubleshooting Guide
 
-[Application fails to work with mp4 stream](#faq0)  
-[Ctrl-C does not stop the app during engine file generation](#faq1)  
-[Application fails to create gst elements](#faq2)  
-[GStreamer debugging](#faq3)  
-[Application stuck with no playback](#faq4)  
-[Error on setting string field](#faq5)  
-[Pipeline unable to perform at real time](#faq6)  
+* [Application fails to work with mp4 stream](#faq0)  
+* [Ctrl-C does not stop the app during engine file generation](#faq1)  
+* [Application fails to create gst elements](#faq2)  
+* [GStreamer debugging](#faq3)  
+* [Application stuck with no playback](#faq4)  
+* [Error on setting string field](#faq5)  
+* [Pipeline unable to perform at real time](#faq6)  
+* [Triton container problems with multi-GPU setup](#faq7)
 
 <a name="faq0"></a>
 ### Application fails to work with mp4 stream
@@ -93,3 +94,9 @@ The application appears to be stuck without any playback activity.
      (info.get_buffer()) from traversing the pipeline until user return.  
   b) loops inside probe() callback could be costly in python.  
 
+<a name="faq7"></a>
+### Triton container problems with multi-GPU setup
+The Triton Inference Server plugin currently only supports single-GPU usage.
+When running the docker, please specify  
+`--gpus device=<GPU ID>`  
+e.g.: `--gpus device=0`

FAQ.md

@@ -8,14 +8,16 @@ This guide provides resources for DeepStream application development in Python.
 * [Running Sample Applications](#run_samples)
 * [Pipeline Construction](#pipeline_construction)
 * [MetaData Access](#metadata_access)
+* [Image Data Access](#imagedata_access)
+* [Custom Inference Output Parsing](apps/deepstream-ssd-parser/custom_parser_guide.md)
 * [FAQ and Troubleshooting](FAQ.md)
 
 <a name="prereqs"></a>
 ## Prerequisites
 
 * Ubuntu 18.04
-* [DeepStream SDK 4.0.1](https://developer.nvidia.com/deepstream-download) or later
-* Python 3
+* [DeepStream SDK 5.0 Developer Preview](https://developer.nvidia.com/deepstream-download) or later
+* Python 3.6
 * [Gst Python](https://gstreamer.freedesktop.org/modules/gst-python.html) v1.14.5
 
 Gst python should be already installed on Jetson.  
@@ -36,11 +38,11 @@ If missing, install with the following steps:
 <a name="run_samples"></a>
 ## Running Sample Applications
 
-Download the release package and unpack it under DS 4.0.1 installation:  
-```tar xf ds_pybind_0.5.tbz2 -C <DeepStream 4.0.1 ROOT>/sources```  
+Download the release package and unpack it under DS 5.0 installation:  
+```tar xf ds_pybind_v0.9.tbz2 -C <DeepStream 5.0 ROOT>/sources```  
 
 This will create the following directory:  
-```<DeepStream 4.0.1 ROOT>/sources/python```  
+```<DeepStream 5.0 ROOT>/sources/python```  
 
 The Python apps and bindings are under the "python" directory.  
 Go into each app directory and follow instructions in the README.  
@@ -56,7 +58,7 @@ See [sample applications](apps/) main functions for pipeline construction exampl
 <a name="metadata_access"></a>
 ## MetaData Access
 
-DeepStream MetaData contains inference results and other information used in analytics. The MetaData is attached to the Gst Buffer received by each pipeline component. The metadata format is described in detail in the [SDK MetaData documentation](https://docs.nvidia.com/metropolis/deepstream/plugin-manual/index.html#page/DeepStream_Plugin_Manual%2Fdeepstream_plugin_metadata.03.1.html) and [API Guide](https://docs.nvidia.com/metropolis/deepstream/dev-guide/DeepStream_Development_Guide/baggage/group__metadata__structures.html).  
+DeepStream MetaData contains inference results and other information used in analytics. The MetaData is attached to the Gst Buffer received by each pipeline component. The metadata format is described in detail in the [SDK MetaData documentation](https://docs.nvidia.com/metropolis/deepstream/plugin-manual/index.html#page/DeepStream_Plugin_Manual%2Fdeepstream_plugin_metadata.03.1.html) and [API Guide](https://docs.nvidia.com/metropolis/deepstream/python-api/index.html).  
 
 The SDK MetaData library is developed in C/C++. Python bindings provide access to the MetaData from Python applications. The bindings are provided in a compiled module, available for x86_64 and Jetson platforms. Find them in the release package with the following layout:
 ```
@@ -118,10 +120,12 @@ This will cause a memory buffer to be allocated, and the string "TYPE" will be c
 This memory is owned by the C code and will be freed later. To free the buffer in Python code, use:  
 ```pyds.free_buffer(obj.type)```  
 
+NOTE: NvOSD_TextParams.display_text string now gets freed automatically when a new string is assigned.
+
 ##### Reading String Fields
 Directly reading a string field returns C address of the field in the form of an int, e.g.:  
 ```python
-obj = pyds.glist_get_nvds_vehicle_object(data);
+obj = pyds.NvDsVehicleObject.cast(data);
 print(obj.type)
 ```
 This will print an int representing the address of obj.type in C (which is a char*).  
@@ -133,25 +137,38 @@ print(pyds.get_string(obj.type))
 
 #### Casting
 
-Some MetaData instances are stored in GList form. To access the data in a GList node, the data field needs to be cast to the appropriate structure. This casting is done via binding functions:  
+Some MetaData instances are stored in GList form. To access the data in a GList node, the data field needs to be cast to the appropriate structure. This casting is done via cast() member function for the target type:
+```python
+NvDsBatchMeta.cast
+NvDsFrameMeta.cast
+NvDsObjectMeta.cast
+NvDsUserMeta.cast
+NvDsClassifierMeta.cast
+NvDsDisplayMeta.cast
+NvDsLabelInfo.cast
+NvDsEventMsgMeta.cast
+NvDsVehicleObject.cast
+NvDsPersonObject.cast
+```
+
+In version v0.5, standalone cast functions were provided. Those are now deprecated and superseded by the cast() functions above:
 ```python
 glist_get_nvds_batch_meta
 glist_get_nvds_frame_meta
 glist_get_nvds_object_meta
-glist_get_nvds_user_met
+glist_get_nvds_user_meta
 glist_get_nvds_classifier_meta
 glist_get_nvds_display_meta
 glist_get_nvds_label_info
 glist_get_nvds_event_msg_meta
-glist_get_nvds_event_msg_meta
 glist_get_nvds_vehicle_object
 glist_get_nvds_person_object
 ```
 
 Example:
 ```python
 l_frame = batch_meta.frame_meta_list
-frame_meta = pyds.glist_get_nvds_frame_meta(l_frame.data)
+frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
 ```
 
 #### Callback Function Registration
@@ -193,3 +210,9 @@ These are performend on each object in deepstream_test_4.py, causing the aggrega
 
 This function populates the input buffer with a timestamp generated according to RFC3339:  
 ```%Y-%m-%dT%H:%M:%S.nnnZ\0```
+
+<a name="imagedata_access"></a>
+## Image Data Access
+
+Decoded images are accessible as NumPy arrays via the `get_nvds_buf_surface` function. This function is documented in the [API Guide](https://docs.nvidia.com/metropolis/deepstream/5.0/python-api/index.html).
+Please see the [deepstream-imagedata-multistream](apps/deepstream-imagedata-multistream) sample application for an example of image data usage.

HOWTO.md

@@ -2,8 +2,8 @@
 
 This repository contains Python bindings and sample applications for the [DeepStream SDK](https://developer.nvidia.com/deepstream-sdk).  
 
-The bindings and apps are currently in *Alpha* at [v0.5](../..//releases/tag/v0.5-alpha). API changes are expected in future releases.  
-SDK version supported: 4.0.1  
+The bindings and apps are currently in *Alpha* at v0.9. The API is maturing but changes are still expected.  
+SDK version supported: 5.0 Developer Preview  
 
 Download the latest release package complete with bindings and sample applications from the [release section](../../releases).  
 
@@ -19,11 +19,11 @@ DeepStream pipelines can be constructed using Gst Python, the GStreamer framewor
 Python bindings are provided in the form of a compiled module. Download these bindings [here](https://developer.nvidia.com/deepstream-download#python_bindings). This module is generated using [Pybind11](https://github.com/pybind/pybind11).  
 
 <p align="center">
-<img src=".python-app-pipeline.png" alt="bindings pipeline" height="400px"/>
+<img src=".python-app-pipeline.png" alt="bindings pipeline" height="600px"/>
 </p>
 
 These bindings support a Python interface to the MetaData structures and functions. Usage of this interface is documented in the [HOW-TO Guide](HOWTO.md) and demonstrated in the sample applications.  
-The current bindings are limited to the [NvDsBatchMeta](https://docs.nvidia.com/metropolis/deepstream/plugin-manual/index.html#page/DeepStream_Plugin_Manual%2Fdeepstream_plugin_metadata.03.2.html%23) hierarchy. Image data access is currently not included.  
+This release adds bindings for decoded image buffers (NvBufSurface) as well as inference output tensors (NvDsInferTensorMeta).
 
 <a name="sample_applications"></a>
 ## Sample Applications
@@ -37,13 +37,15 @@ To run the sample applications or write your own, please consult the [HOW-TO Gui
 <img src=".test3-app.png" alt="deepstream python app screenshot" height="400px"/>
 </p>
 
-We currently provide four sample applications:
+We currently provide the following sample applications:
 * [deepstream-test1](apps/deepstream-test1) -- 4-class object detection pipeline
 * [deepstream-test2](apps/deepstream-test2) -- 4-class object detection, tracking and attribute classification pipeline
 * [deepstream-test3](apps/deepstream-test3) -- multi-stream pipeline performing 4-class object detection
 * [deepstream-test4](apps/deepstream-test4) -- msgbroker for sending analytics results to the cloud
-
-These are Python versions of the first four test applications included in the DeepStream SDK.  
+* [deepstream-imagedata-multistream](apps/deepstream-imagedata-multistream) -- multi-stream pipeline with access to image buffers
+* [deepstream-ssd-parser](apps/deepstream-ssd-parser) -- SSD model inference via Triton server with output parsing in Python
+* [deepstream-test1-usbcam](apps/deepstream-test1-usbcam) -- deepstream-test1 pipelien with USB camera input
+* [deepstream-test1-rtsp-out](apps/deepstream-test1-rtsp-out) -- deepstream-test1 pipeline with RTSP output
 
 Detailed application information is provided in each application's subdirectory under [apps](apps).  
 

README.md

@@ -1,5 +1,5 @@
 ################################################################################
-# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2019-2020, NVIDIA CORPORATION. All rights reserved.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a
 # copy of this software and associated documentation files (the "Software"),
@@ -25,8 +25,8 @@ DeepStream SDK Python Bindings
 ================================================================================
 Setup pre-requisites:
 - Ubuntu 18.04
-- DeepStream SDK 4.0.1
-- Python 3
+- NVIDIA DeepStream SDK 5.0 Developer Preview
+- Python 3.6
 - Gst-python
 
 --------------------------------------------------------------------------------
@@ -42,43 +42,39 @@ The DeepStream Python package includes:
       |- jetson
             |- pyds.so
 2. DeepStream test apps in Python
-   Four test apps are available:
-   deepstream-test1/deepstream_test_1.py
-   deepstream-test2/deepstream_test_2.py
-   deepstream-test3/deepstream_test_3.py
-   deepstream-test3/deepstream_test_4.py
+   The following test apps are available:
+   deepstream-test1
+   deepstream-test2
+   deepstream-test3
+   deepstream-test4
+   deepstream-imagedata-multistream
+   deepstream-ssd-parser
+   deepstream-test1-rtsp-out
+   deepstream-test1-usbcam
 
 --------------------------------------------------------------------------------
 Installing Pre-requisites:
 --------------------------------------------------------------------------------
 
-DeepStream SDK 4.0.1
+DeepStream SDK 5.0 Developer Preview
 --------------------
 Download and install from https://developer.nvidia.com/deepstream-download
 
-Python 3
+Python 3.6
 ----------
 Should be already installed with Ubuntu 18.04
 
 Gst-python
 ----------
 Should be already installed on Jetson
 If missing, install with the following steps:
-   $ sudo apt-get install python-gi-dev
-   $ export GST_LIBS="-lgstreamer-1.0 -lgobject-2.0 -lglib-2.0"
-   $ export GST_CFLAGS="-pthread -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include"
-   $ git clone https://github.com/GStreamer/gst-python.git
-   $ cd gst-python
-   $ git checkout 1a8f48a
-   $ ./autogen.sh PYTHON=python3
-   $ ./configure PYTHON=python3
-   $ make
-   $ sudo make install
+$ sudo apt update
+$ sudo apt install python3-gi python3-dev python3-gst-1.0 -y
 
 --------------------------------------------------------------------------------
 Running the samples
 --------------------------------------------------------------------------------
-The apps are configured to work from inside the DeepStream SDK 4.0.1 installation.
+The apps are configured to work from inside the DeepStream SDK 5.0 installation.
 Untar the contents into <DeepStream install root>/sources/.
 
 Note that the apps import the pyds module from this relative path:
@@ -90,9 +86,30 @@ Example: running test1 app:
 $ cd deepstream-test1
 $ python3 deepstream_test_1.py <input .h264 file>
 
+--------------------------------------------------------------------------------
+Running the samples inside DeepStream SDK docker
+--------------------------------------------------------------------------------
+The general steps are:
+1. Pull the DeepStream SDK docker of choice following the latest DeepStream
+   Release Notes at https://developer.nvidia.com/deepstream-sdk for more info.
+   Note that the deepstream-ssd-parser app requires the Triton docker on x86_64.
+2. Run the docker with Python Bindings mapped using the following option:
+   -v <path to this python bindings directory>:/opt/nvidia/deepstream/deepstream-5.0/sources/python
+3. Inside the container, install packages required by all samples:
+   $ sudo apt update
+   $ sudo apt install python3-gi python3-dev python3-gst-1.0 -y
+4. Optionally install additional dependencies required by specific samples.
+   See README in app's directory for such requirements.
+   $ sudo apt install python3-opencv
+   $ sudo apt install python3-numpy
+   $ sudo apt install libgstrtspserver-1.0-0 gstreamer1.0-rtsp
+   $ sudo apt install libgirepository1.0-dev
+   $ sudo apt install gobject-introspection gir1.2-gst-rtsp-server-1.0
+5. Run sample apps following directions in each app's README.
+
 --------------------------------------------------------------------------------
 Notes:
 --------------------------------------------------------------------------------
 As with DeepStream SDK, if the application runs into errors, cannot create gst elements, 
 try again after removing gstreamer cache
-   rm ${HOME}/.cache/gstreamer-1.0/registry.x86_64.bin
+   rm ${HOME}/.cache/gstreamer-1.0/*

apps/README

@@ -1,3 +1,25 @@
+################################################################################
+# Copyright (c) 2019-2020, NVIDIA CORPORATION. All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+# DEALINGS IN THE SOFTWARE.
+################################################################################
+
 import time
 start_time=time.time()
 frame_count=0

apps/common/FPS.py

@@ -1,3 +1,25 @@
+################################################################################
+# Copyright (c) 2019-2020, NVIDIA CORPORATION. All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+# DEALINGS IN THE SOFTWARE.
+################################################################################
+
 import gi
 import sys
 gi.require_version('Gst', '1.0')

apps/common/bus_call.py

@@ -1,14 +1,31 @@
+################################################################################
+# Copyright (c) 2019-2020, NVIDIA CORPORATION. All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+# DEALINGS IN THE SOFTWARE.
+################################################################################
+
 import platform
 import sys
+
+
 def is_aarch64():
-    machine = platform.uname()[4]
-    if machine in ['aarch64']:
-        return True
-    else:
-        return False
+    return platform.uname()[4] == 'aarch64'
 
-if is_aarch64():
-    sys.path.append('../../bindings/jetson')
-else:
-    sys.path.append('../../bindings/x86_64')   
 
+sys.path.append('../../bindings/' + ('jetson' if is_aarch64() else 'x86_64'))