This tutorial aims at giving a brief introduction to the GStreamer 1.0 multimedia framework using its Python bindings.
<sec:meta>
A GStreamer application is built as a directed, acyclic graph. In the figures these graphs are illustrated with the convention:
- right solid line rectangles indicate basic GStreamer elements
- rounded solid line rectangles to indicate GStreamer bins (and pipelines) subclasses of elements.
- rounded dashed line rectangles to indicate pads
<sec:intro>
This tutorial is meant to be a quick way to get to know more about GStreamer but it’ll take some time to write it though because we don’t know it ourselves … yet. We’re usually using GNU/Linux and GTK in the examples but we try to keep the GUI code to an absolute minimum so it should not get in the way. Just remember that GStreamer depends heavily on Glib so you must make sure that the Glib Mainloop is running if you want to catch events on the bus. We take for granted that you are at least a fairly descent Python coder. For problems related to the Python language we redirect you over to Online Python docs.
There are also some example coding distributed with the PyGST source which you may browse at the gst-python git repository. Reference documents for GStreamer and the rest of the ecosystem it relies on are available at lazka’s GitHub site. The main GStreamer site has Reference Manual, FAQ, Applications Development Manual and Plugin Writer’s Guide. This tutorial targets the GStreamer 1.0 API which all v1.x releases should follow. The Novacut project has a guide to porting Python applications from the prior 0.1 API to 1.0. Finally, GStreamer provides the GstSDK documentation which includes substantial C programming tutorials.
As you may see this tutorial is far from done and we are always looking for new people to join this project. If you want to write a chapter, fix the examples, provide alternatives or otherwise improve this document please do so. It is suggested to clone the repository on GitHub and issue a pull request. Note, the original source of this document was here but is now dead.
<subsec:cli>
Before getting started with Python some of the command line interface (CLI) programs that come with GStreamer are explored. Besides being generally useful they can help you find and try out what you need in a very fast and convenient way without writing a bit of code. With =gst-inspect= you can track highlevel elements which are shipped with the various plugins packages.
man gst-inspect-1.0If you are looking for an element but you don’t know its name you can use it with grep. Getting the elements that handles ie mp3 is done like this:
gst-inspect-1.0 | grep mp3 | sort | head -3flump3dec: flump3dec: Fluendo MP3 Decoder (liboil build) lame: lamemp3enc: L.A.M.E. mp3 encoder libav: avdec_mp3adufloat: libav ADU (Application Data Unit) MP3 (MPEG audio layer 3) decoder
The =playbin= element is an autoplugger which usually plays anything you throw at it, if you have the appropriate plugins installed.
gst-inspect-1.0 playbin > gst-inspect-playbin.txtBrowse example output here.
You can also run pipelines directly in a terminal with =gst-launch=:
man gst-launch-1.0For playing a file with playbin:
gst-launch-1.0 playbin \
uri=http://docs.gstreamer.com/media/sintel_trailer-480p.webmIt’s also possible to link elements together with “!”:
gst-launch-1.0 audiotestsrc ! alsasinkYou may also make different streams in the pipeline:
gst-launch-1.0 audiotestsrc ! alsasink videotestsrc ! xvimagesinkOr, you can make a single frame JPEG
gst-launch-1.0 videotestsrc num-buffers=1 ! jpegenc ! \
filesink location=videotestsrc-frame.jpg
If you are using the “name” property you may use the same element more than once. Just put a “.” after its name, eg with oggmux here.
gst-launch-1.0 audiotestsrc ! vorbisenc ! oggmux name=mux ! \
filesink location=file.ogg videotestsrc ! theoraenc ! mux.In the next chapter we will show you more examples with Playbin.
<sec:playbin>
The =playbin= element was exercised from the command line in section subsec:cli and in this section it will be used from Python. It is a high-level, automatic audio and video player. You create a playbin object with:
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst
Gst.init(None)
# ...
my_playbin = Gst.ElementFactory.make("playbin", None)
assert my_playbin
print my_playbin<__main__.GstPlayBin object at 0x7fd8e88e6aa0 (GstPlayBin at 0x1c0cf00)>
To get information about a playbin run:
gst-inspect-0.10 playbinThis figure shows how playbin is built internally. The “optional stuff” are things that could be platform specific or things that you may set with properties.
/--------------------------------------------------------------------\
| +----------------+ /---------------\ |
| | | | | |
| +->-+ optional stuff +->-+ autoaudiosink +----->- Audio Output
| /----------------\ | | | | | |
| | +--+ +----------------+ \---------------/ |
uri ----->-+ uridecodebin | |
| | +--+ +----------------+ /---------------\ |
| \----------------/ | | | | | |
| +->-+ optional stuff +->-+ autovideosink +----->- Video Output
| | | | | |
| playbin +----------------+ \---------------/ |
\--------------------------------------------------------------------/
The ”uri” property should take any possible protocol supported by your GStreamer plugins. One nice feature is that you may switch the sinks out for your own bins as shown below. Playbin always tries to set up the best possible pipeline for your specific environment so if you don’t need any special features that are not implemented in playbin, it should in most cases just work “out of the box”. Ok, time for a few examples.
<subsec:playbinaudio>
This first example is just a simple audio player, insert a file with absolute path and it’ll play. It’s code is listed below. You can run it like:
python playbin-example-audio.pyIt will open a small window with a text entry. Enter the full path to some audio file and click “Start”.
playbin-example-audio.py<subsec:playbinvideo>
A playbin plugs both audio and video streams automagically and the videosink has been switched out to a fakesink element which is GStreamer’s answer to directing output to /dev/null. If you want to enable video playback just comment out the following lines:
fakesink = Gst.ElementFactory.make("fakesink", "fakesink")
self.player.set_property("video-sink", fakesink)If you want to show the video output in a specified window you’ll have to use the =enable_sync_message_emission()= method on the bus. Here is an example with the video window embedded in the program.
playbin-example-video.pyAnd just to make things a little more complicated you can switch the playbin’s video sink to a {{{gstclass(Bin)}}} with a {{{gstclass(GhostPad)}}} on it. Here’s an example with a =timeoverlay=.
bin = Gst.Bin.new("my-bin")
timeoverlay = Gst.ElementFactory.make("timeoverlay")
bin.add(timeoverlay)
pad = timeoverlay.get_static_pad("video_sink")
ghostpad = Gst.GhostPad.new("sink", pad)
bin.add_pad(ghostpad)
videosink = Gst.ElementFactory.make("autovideosink")
bin.add(videosink)
timeoverlay.link(videosink)
self.player.set_property("video-sink", bin)Add that code to the example above and you’ll get a timeoverlay too. We’ll talk more about “ghost pads” later.
Here now adds a CLI example which plays music. It can be run it with:
python cliplayer.py /path/to/file1.mp3 /path/to/file2.oggA playbin implements a {{{gstclass(Pipeline)}}} element and that’s what the next chapter is going to tell you more about.
<sec:pipeline>
A {{{gstclass(Pipeline)}}} is a top-level bin with its own bus and clock. If your program only contains one bin-like object, this is what you’re looking for. You create a pipeline object with:
my_pipeline = Gst.Pipeline.new("my-pipeline")A pipeline is a “container” where you can put other objects and when everything is in place and the file to play is specified you set the pipeline’s state to =Gst.State.PLAYING= and there should be multimedia coming out of it.
The first example here starts with the audio player from section sec:playbin and switches the playbin for the mad decoding pipeline that is capable of handling MP3 streams. Before coding it in Python it can be tested using gst-launch.
gst-launch-1.0 filesrc location=file.mp3 ! mad ! audioconvert ! alsasinkConceptually this pipeline looks like:
Example Gst.Pipeline
/---------------------------------------------------------------------\
| +---------+ +-------+ +--------------+ +----------+ |
| | | | | | | | | |
file.mp3--->-| filesrc +-->--+ mad +-->--+ audioconvert +-->--+ alsasink +---->-Audio Output
| | | | | | | | | |
| +---------+ +-------+ +--------------+ +----------+ |
\---------------------------------------------------------------------/
Done in Python this would look like pipeline-example.py:
The next example is playing MPEG2 videos. Some demuxers, such as mpegdemux, uses dynamic pads which are created at runtime and therefor you can’t link between the demuxer and the next element in the pipeline before the pad has been created at runtime. Watch out for the demuxer_callback() method below.
Gst.Pipeline
+----------------------------------------------------------------------------------------------------------+
| |
| +-------+ +-------+ +----------------+ +---------------+ |
| | | | | | | | | |
| +->-+ queue +->-+ mad +->-+ audioconvert +->-+ autoaudiosink +--------->-Audio Output
| +---------+ +-----------+ | | | | | | | | | |
| | | | +->-+ +-------+ +-------+ +----------------+ +---------------+ |
file.mpg--->-| filesrc |->-| mpegdemux | |
| | | | +->-+ +-------+ +----------+ +------------------+ +---------------+ |
| +---------+ +-----------+ | | | | | | | | | |
| +->-+ queue +->-+ mpeg2dec +->-+ ffmpegcolorspace +->-+ autovideosink +---->-Video Output
| | | | | | | | | |
| +-------+ +----------+ +------------------+ +---------------+ |
| |
+----------------------------------------------------------------------------------------------------------+
pipeline-branch-example.py
The elements in a pipeline connects to each other with pads and that’s what the next chapter will tell you more about.
<sec:srcsinkpad>
As their names imply, a src is an object that is “sending” data and a sink is an object that is “receiving” data. These objects connect to each other with pads. Pads could be either src or sink. Most elements have both src and sink pads. For example, the mad MP3 decoder element looks something like the figure below:
mad element
+---------------------------------------------------------+
| |
| /------------\ +----------------+ /-----------\ |
| : : | | : : |
->---+ pad (sink) +-->--+ internal stuff +-->--+ pad (src) +-->-
| : : | | : : |
| \------------/ +----------------+ \-----------/ |
| |
+---------------------------------------------------------+
And as always if you want to know more about highlevel elements gst-inspect is your friend:
gst-inspect-1.0 madIn particular, the inheritance diagram shows that mad is an element:
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstAudioDecoder
+----GstMad
There are many different ways to link elements together. In pipeline-example.py we used the Gst.Pipeline.add() and the .link() method of the produced elements. You can also make a completely ready-to-go pipeline with the parse_launch() function. The many .add() calls in that example can be rewritten as:
mp3_pipeline = Gst.parse_launch("filesrc name=source ! mad name=decoder ! " \
"audioconvert name=conv ! alsasink name=sink")The micro-language used in this function call is that of the gst-launch command line program.
When you do manually link pads with the .link() method make sure that you link a src-pad to a sink-pad. No rule though without exceptions. A {{{gstclass(GhostPad)}}} should be linked to a pad of the same kind as itself. We have already showed how a ghost pad works in the addition to example 2.2. A =Gst.Bin= can’t link to other objects if you don’t link a =Gst.GhostPad= to an element inside the bin. The playbin-example-video.py example in section sec:playbin should look something like this:
Gst.Bin
/---------------------------------------------------------------------------- - -
|
| timeoverlay
| +----------------------------------------------------------+
| /----------\ | /------------\ +---------------------+ /-----------\ |
| : | | : | | | : | |
->---+ ghostpad |->---+ pad (sink) |->-| internal stuff here |->-+ pad (src) +-->--
| : | | : | | | : | |
| \----------/ | \------------/ +---------------------+ \-----------/ |
| +----------------------------------------------------------+
|
\---------------------------------------------------------------------------- - -
And the ghostpad above should be created as type “sink”!!!
Some pads are not always available and are only created when they are in use. Such pads are called “dynamical pads”. The next example will show how to use dynamically created pads with an =oggdemux=. The link between the demuxer and the decoder is created with the demuxer_callback() method, which is called whenever a pad is created in the demuxer using the ”pad-added” signal.
Now after reading through these four chapters you could need a break. Happy hacking and stay tuned for more interesting chapters to come.
<sec:seeking>
Seeking in GStreamer is done with the =seek()= and =seek_simple()= methods of {{{gstclass(Element)}}}. To be able to seek you will also need to tell GStreamer what kind of seek it should do. In the following example we will use a =TIME= value (of {{{gstenum(Format)}}} enum) format constant which will, as you may guess, request a time seek. We will also use the =query_duration()= and =query_position()= methods to get the file length and how long the file has currently played. GStreamer uses nanoseconds by default so you have to adjust to that.
In this next example we take the Vorbis-Player from example 4.1 and update it with some more stuff so it’s able to seek and show duration and position.
seeking-example.py<sec:caps>
Capabilities, {{{gststruct(Caps)}}}, is a container where you may store information that you may pass on to a {{{gstclass(PadTemplate)}}}. When you set the pipeline state to either playing or paused the elements pads negotiates what caps to use for the stream. Now the following pipeline works perfectly:
gst-launch-1.0 videotestsrc ! video/x-raw, width=320, height=240 ! \
xvimagesinkBut if you try to switch out the =xvimagesink= for an =ximagesink= you will notice that it wouldn’t work. That’s because ximagesink can’t handle video/x-raw so you must put in an element BEFORE in the pipeline that does.
gst-launch-1.0 videotestsrc ! video/x-raw, width=320, height=240 ! \
videoconvert ! ximagesinkAnd as ximagesink does not support hardware scaling you have to throw in a videoscale element too if you want software scaling.
gst-launch-1.0 videotestsrc ! video/x-raw, width=320, height=240 ! \
videoscale ! videoconvert ! ximagesinkTo put the above examples in code you have to put the caps in a capsfilter element.
capabilities-example.pyA frequently asked question is how to find out what resolution a file has and one way to do it is to check the caps on a decodebin element in paused state.
capabilities-resolution-example.pyIn the next example we will use the playbin from section subsec:playbinvideo and switch its video-sink out for our own homemade bin filled with some elements. Now, let’s say that you run a tv-station and you want to have your logo in the top right corner of the screen. For that you can use a textoverlay but for the fonts to be the exact same size on the screen no matter what kind of resolution the source has you have to specify a width so everything is scaled according to that.
<sec:vidmix>
The videomixer element makes it possible to mix different video streams together. Here is a CLI example:
gst-launch-1.0 filesrc location=tvlogo.png ! pngdec ! alphacolor ! \
videoconvert ! videobox border-alpha=0 alpha=0.5 top=-20 left=-200 ! \
videomixer name=mix ! videoconvert ! autovideosink videotestsrc ! \
video/x-raw, width=320, height=240 ! mix.Fixme: this fails with:
Setting pipeline to PAUSED ... Pipeline is PREROLLING ... ERROR: from element /GstPipeline:pipeline0/GstFileSrc:filesrc0: Internal data flow error. Additional debug info: gstbasesrc.c(2865): gst_base_src_loop (): /GstPipeline:pipeline0/GstFileSrc:filesrc0: streaming task paused, reason not-negotiated (-4) ERROR: pipeline doesn't want to preroll. Setting pipeline to NULL ... Freeing pipeline ...
You have to make a 
image (100x100 px) to be able to run it. With the videobox element you can move the image around and add more alpha channels.
In the next example we take the now working Mpeg2-Player from section sec:pipeline and add the elements shown above.
videomixer-example.pyFixme: This fails with:
Error: Internal data flow error. gstbasesrc.c(2865): gst_base_src_loop (): /GstPipeline:player/GstFileSrc:png-source: streaming task paused, reason not-negotiated (-4)
<sec:webcam>
Remember to peal the tape off your web cam lens before testing this.
webcam-example.py