GAMBAS BOOK 3.19.1

The Gambas components gb.media and gb.media.form provide a variety of classes and control elements that are based on the free MultiMedia framework GStreamer. GStreamer is licensed under LGPL and provides libraries for processing media streams and thus forms the basis for media players, for example.

To be able to work with the components gb.media and gb.media.form, it is essential to familiarise yourself with the GStreamer concept. The following source is recommended:

GStreamer Webseite: https://gstreamer.freedesktop.org/

As a first step, you should familiarise yourself with the basic structure of GStreamer pipelines, which form the basis for every media application as a function chain. Gambas builds such GStreamer pipelines with an identical structure and uses a Gambas-specific syntax.

The following GStreamer elements are available as basic elements for pipelines:

Figure 23.9.0.1.1: Base elements

These base elements can represent various GStreamer plug-ins with different functions. You can find an overview of the plug-ins on the GStreamer website: https://gstreamer.freedesktop.org/documentation/plugins_doc.html?gi-language=c .

Elements are provided with at least one input or one output and are connected to another element via these. In such a connection, the input of an element serves as a sink for the connected element, which is why the input is referred to as a 'sink' and the output of an element as a 'source'. In GStreamer terminology, the inputs and outputs of the elements correspond to the so-called 'pads'. In many cases, elements are equipped with so-called properties, which you can use to parameterise the element - here for the 'filesrc' element with its 'location' property:

$ gst-launch-1.0 filesrc location=mp.mp3 ! audioconvert ! autoaudiosink

A GStreamer pipeline represents a complete media function and is made up of elements connected in series. GStreamer pipelines can also branch out, for example to handle the audio and video signal separately or to output an audio signal via one branch and record it in a file via the other.

Here is an example of a pipeline for playing an ogg file:

Figure 23.9.0.1.2: Pipeline

To be able to map a GStreamer pipeline in Gambas, knowledge of the GStreamer syntax is essential. It is even recommended to try out a pipeline as a CLI command before implementing it in Gambas.

Here is a simple example of a pipeline in the form of a GStreamer CLI command that generates a 440Hz sine wave test tone that is output to the PC's loudspeaker:

$ gst-launch-1.0 audiotestsrc ! audioconvert ! autoaudiosink

gst-launch-1.0 is the GStreamer command line tool for executing pipelines, the playback of which can be cancelled at any time with Ctrl-C. All other expressions represent elements that are separated from each other by an exclamation mark and are built up as a function chain from left to right and represent a pipeline as a whole. Here you can find information about the gst-launch-1.0 tool and a list of useful examples: https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html?gi-language=c.

The pipeline consists of the following elements connected in series:

• audiotestsrc (audio test source)
• audioconvert (converts the audio signal into different formats)
• autoaudiosink (offers an automatically determined audio output)

Figure 23.9.0.1.3: Generator pipeline

The structure of the above pipeline can be transferred to the Gambas syntax in the next step:

Private pl As MediaPipeline
Private src As MediaControl
Private cnv As MediaControl
Private snk As MediaControl
 
Public Sub Form_Open()
 
'-- Generate Pipeline
    pl = New MediaPipeline
 
'-- Generate MediaControls  (GStreamer Elements)
    src = New MediaControl(pl, "audiotestsrc")
    cnv = New MediaControl(pl, "audioconvert")
    snk = New MediaControl(pl, "autoaudiosink")
 
'-- Link MediaControls (GStreamer Elements)
    src.LinkTo(cnv)
    cnv.LinkTo(snk)
 
'-- Start/play pipeline
    pl.Play()
 
End

In a further step, you could, for example, specify a frequency other than the 440Hz standard frequency. The CLI command to output a 1000Hz tone would then look like this:

gst-launch-1.0 audiotestsrc freq=1000 ! audioconvert ! autoaudiosink

The Gambas programme above would only need to be extended by one line in which the 'freq' property is set:

Private pl As MediaPipeline
Private src As MediaControl
Private cnv As MediaControl
Private snk As MediaControl
 
Public Sub Form_Open()
 
'-- Generate Pipeline
    pl = New MediaPipeline
 
'-- Generate MediaControls  (GStreamer Elements)
    src = New MediaControl(pl, "audiotestsrc")
    src["freq"] = 1000
    cnv = New MediaControl(pl, "audioconvert")
    snk = New MediaControl(pl, "autoaudiosink")
 
'-- Link MediaControls (GStreamer Elements)
    src.LinkTo(cnv)
    cnv.LinkTo(snk)
 
'-- Start/play pipeline
    pl.Play()
 
End

The second programme could be extended to a practical tone generator - with adjustable frequency, different waveform and variable output level.

Please note that data types declared with 'Dim' are only valid locally and have a lifetime for the period in which the programme is in the sub-routine. Of course, this also applies to all declarations that affect a pipeline. This means that the above example with locally declared MediaPipelines and MediaControls would not work because all variables would be destroyed immediately after the Play() method is executed and the sub-routine is exited. As you usually want to operate pipelines for longer periods of time, make sure that you declare all variables of a pipeline at least with 'Private' - i.e. with validity at module level or at class level.

As of Gamba version 3.19, it is sufficient if the control element with the highest level, such as MediaPipeline or MediaControl, is declared with a validity at module/class level.

Since the selection of necessary GStreamer elements for a pipeline is often not easy and requires a lot of background knowledge in individual cases, it is advisable to find examples of GStreamer pipelines on the web or to fall back on already proven pipelines.

To be able to play videos in a Gambas GUI control, you must also define a control for the display in addition to a sink. For example, you can use a form, the controls MediaView (gb.media.form), DrawingArea or a PictureBox.

Example: First execute the following CLI command line to see what is to be implemented:

gst-launch-1.0 videotestsrc ! xvimagesink

The implementation in Gambas looks like this, for example, whereby the form FMain is used here to display the GStreamer test image:

Private pl As MediaPipeline
Private src As MediaControl
Private snk As MediaControl
 
Public Sub Form_Open()
 
  pl = New MediaPipeline As "PipeEvents"
  src = New MediaControl(pl, "videotestsrc")
  snk = New MediaControl(pl, "xvimagesink")
 
  snk.SetWindow(Me)
 
  src.LinkTo(snk)
 
  pl.Play()
 
End

In some cases, it is necessary to split a signal. For example, you can use one branch to output it to the speaker while the other saves the signal to a file and records it. The GStreamer framework provides the so-called tee element for such purposes. A pipeline with a tee element looks like this:

Figure 23.9.0.1.4: Pipeline for listening to and recording web radio streams

The above pipeline plays a web radio station on the loudspeaker and records the audio signal in parallel in a file in MP3 format. The corresponding GStreamer CLI command follows and can be tried out in the console.

gst-launch-1.0 uridecodebin uri="http://icecast.ndr.de/ndr/ndrinfo/schleswigholstein/mp3/128/stream.mp3" \
! audioconvert ! tee name=radio ! queue ! autoaudiosink radio. ! queue \
! lamemp3enc target=bitrate bitrate=128 cbr=true ! filesink location="/home/username/output.mp3"

From Gambas 3.19 an implementation in Gambas could look like this:

    Private pl As MediaPipeline
'-- Declarations for source
    Private src As MediaControl
    Private tee As MediaControl
    Private cnv As MediaControl
'-- Declarations for branch 1
    Private que1 As MediaControl
    Private snk1 As MediaControl
'-- Declarations for branch 2
    Private que2 As MediaControl
    Private enc2 As MediaControl
    Private snk2 As MediaControl
 
Public Sub Form_Open()
 
    pl = New MediaPipeline
'-- Source
    src = New MediaControl(pl, "uridecodebin")
    src["uri"] = "http://icecast.ndr.de/ndr/ndrinfo/schleswigholstein/mp3/128/stream.mp3"
    cnv = New MediaControl(pl, "audioconvert")
    tee = New MediaControl(pl, "tee")
'-- Branch 1
    que1 = New MediaControl(pl, "queue")
    snk1 = New MediaControl(pl, "autoaudiosink")
'-- Branch 2
    que2 = New MediaControl(pl, "queue")
    enc2 = New MediaControl(pl, "lamemp3enc")
    enc2["target"] = "bitrate"
    enc2["bitrate"] = 128
    enc2["cbr"] = True
    snk2 = New MediaControl(pl, "filesink")
    snk2["location"] = User.Home & "/radio_rec.mp3"
 
'-- Link source elements
    src.LinkLaterTo(cnv)
    cnv.LinkTo(tee)
 
'-- Link to TEE and link branch 1 elements
    tee.LinkTo(que1)
    que1.LinkTo(snk1)
 
'-- Link to TEE and link branch 2 elements
    tee.LinkTo(que2)
    que2.LinkTo(enc2)
    enc2.LinkTo(snk2)
 
    pl.Play()
 
End
 
Public Sub Form_Close()
 
  pl.Stop()
  pl.Close()
 
End

In all earlier Gambas versions < 3.19 you will notice that the mp3 file is emptied after the programme has finished. This is due to a bug for which you can use the following workarounds:

• Set the 'Location' property of the 'filesink' element to zero before ending the programme or
• write the audio data to a temporary file and save it under a different name before exiting the programme.

Please note that GStreamer pipelines work with buffers, which may have to be terminated properly in order to obtain a valid recording file. It may therefore be necessary to take further measures that are not covered here.

Some GStreamer pipelines generate events in which messages/information are delivered. These events can also be intercepted and processed in Gambas. The Gambas MediaPipeline provides the 'Message' and 'Tag' events for this purpose. A corresponding declaration looks like this, for example:

hPipeLine = New MediaPipeline As "PipeEvents"

The events can then be intercepted and processed in corresponding event handling routines. For messages, for example, it looks like this

Public Sub PipeEvents_Event(Message As MediaMessage)
  Print "Message.Name: "; Message.Name
End

The names of the messages correspond to the types described on the GStreamer website: https://gstreamer.freedesktop.org/documentation/additional/design/messages.html?gi-language=c .

To intercept media tags such as the current music title/artist in the web radio stream, the following event handling routine can be used as a basis:

Public Sub PipeEvents_Tag(TagList As MediaTagList)
  For Each s As String In Taglist.Tags
      Print "Tag: "; s
  Next
End