转载：Advanced Concepts of Gstreamer Plugin

转自：https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/index.html?gi-language=c

https://blog.csdn.net/techx/article/details/44134911

Request and Sometimes pads

There's pads that are only being created in some cases, or only if the application requests the pad. The first is called a sometimes; the second is called a request pad. The availability of a pad (always, sometimes or request) can be seen in a pad's template.

Sometimes pads

A “sometimes” pad is a pad that is created under certain conditions, but not in all cases. This mostly depends on stream content: demuxers will generally parse the stream header, decide what elementary (video, audio, subtitle, etc.) streams are embedded inside the system stream, and will then create a sometimes pad for each of those elementary streams. At its own choice, it can also create more than one instance of each of those per element instance. The only limitation is that each newly created pad should have a unique name. Sometimes pads are disposed when the stream data is disposed, too (i.e. when going from PAUSED to the READY state). You should not dispose the pad on EOS, because someone might re-activate the pipeline and seek back to before the end-of-stream point. The stream should still stay valid after EOS, at least until the stream data is disposed. In any case, the element is always the owner of such a pad.

The example code below will parse a text file, where the first line is a number (n). The next lines all start with a number (0 to n-1), which is the number of the source pad over which the data should be sent.

3
0: foo
1: bar
0: boo
2: bye

The code to parse this file and create the dynamic “sometimes” pads, looks like this:

typedef struct _GstMyFilter {
[..]
  gboolean firstrun;
  GList *srcpadlist;
} GstMyFilter;

static GstStaticPadTemplate src_factory =
GST_STATIC_PAD_TEMPLATE (
  "src_%u",
  GST_PAD_SRC,
  GST_PAD_SOMETIMES,
  GST_STATIC_CAPS ("ANY")
);

static void
gst_my_filter_class_init (GstMyFilterClass *klass)
{
  GstElementClass *element_class = GST_ELEMENT_CLASS (klass);
[..]
  gst_element_class_add_pad_template (element_class,
    gst_static_pad_template_get (&src_factory));
[..]
}

static void
gst_my_filter_init (GstMyFilter *filter)
{
[..]
  filter->firstrun = TRUE;
  filter->srcpadlist = NULL;
}

/*
 * Get one line of data - without newline.
 */

static GstBuffer *
gst_my_filter_getline (GstMyFilter *filter)
{
  guint8 *data;
  gint n, num;

  /* max. line length is 512 characters - for safety */
  for (n = 0; n < 512; n++) {
    num = gst_bytestream_peek_bytes (filter->bs, &data, n + 1);
    if (num != n + 1)
      return NULL;

    /* newline? */
    if (data[n] == '\n') {
      GstBuffer *buf = gst_buffer_new_allocate (NULL, n + 1, NULL);

      gst_bytestream_peek_bytes (filter->bs, &data, n);
      gst_buffer_fill (buf, 0, data, n);
      gst_buffer_memset (buf, n, '\0', 1);
      gst_bytestream_flush_fast (filter->bs, n + 1);

      return buf;
    }
  }
}

static void
gst_my_filter_loopfunc (GstElement *element)
{
  GstMyFilter *filter = GST_MY_FILTER (element);
  GstBuffer *buf;
  GstPad *pad;
  GstMapInfo map;
  gint num, n;

  /* parse header */
  if (filter->firstrun) {
    gchar *padname;
    guint8 id;

    if (!(buf = gst_my_filter_getline (filter))) {
      gst_element_error (element, STREAM, READ, (NULL),
             ("Stream contains no header"));
      return;
    }
    gst_buffer_extract (buf, 0, &id, 1);
    num = atoi (id);
    gst_buffer_unref (buf);

    /* for each of the streams, create a pad */
    for (n = 0; n < num; n++) {
      padname = g_strdup_printf ("src_%u", n);
      pad = gst_pad_new_from_static_template (src_factory, padname);
      g_free (padname);

      /* here, you would set _event () and _query () functions */

      /* need to activate the pad before adding */
      gst_pad_set_active (pad, TRUE);

      gst_element_add_pad (element, pad);
      filter->srcpadlist = g_list_append (filter->srcpadlist, pad);
    }
  }

  /* and now, simply parse each line and push over */
  if (!(buf = gst_my_filter_getline (filter))) {
    GstEvent *event = gst_event_new (GST_EVENT_EOS);
    GList *padlist;

    for (padlist = srcpadlist;
         padlist != NULL; padlist = g_list_next (padlist)) {
      pad = GST_PAD (padlist->data);
      gst_pad_push_event (pad, gst_event_ref (event));
    }
    gst_event_unref (event);
    /* pause the task here */
    return;
  }

  /* parse stream number and go beyond the ':' in the data */
  gst_buffer_map (buf, &map, GST_MAP_READ);
  num = atoi (map.data[0]);
  if (num >= 0 && num < g_list_length (filter->srcpadlist)) {
    pad = GST_PAD (g_list_nth_data (filter->srcpadlist, num);

    /* magic buffer parsing foo */
    for (n = 0; map.data[n] != ':' &&
                map.data[n] != '\0'; n++) ;
    if (map.data[n] != '\0') {
      GstBuffer *sub;

      /* create region copy that starts right past the space. The reason
       * that we don't just forward the data pointer is because the
       * pointer is no longer the start of an allocated block of memory,
       * but just a pointer to a position somewhere in the middle of it.
       * That cannot be freed upon disposal, so we'd either crash or have
       * a memleak. Creating a region copy is a simple way to solve that. */
      sub = gst_buffer_copy_region (buf, GST_BUFFER_COPY_ALL,
          n + 1, map.size - n - 1);
      gst_pad_push (pad, sub);
    }
  }
  gst_buffer_unmap (buf, &map);
  gst_buffer_unref (buf);
}

Note that we use a lot of checks everywhere to make sure that the content in the file is valid. This has two purposes: first, the file could be erroneous, in which case we prevent a crash. The second and most important reason is that - in extreme cases - the file could be used maliciously to cause undefined behaviour in the plugin, which might lead to security issues. Always assume that the file could be used to do bad things.

Request pads

“Request” pads are similar to sometimes pads, except that request are created on demand of something outside of the element rather than something inside the element. This concept is often used in muxers, where - for each elementary stream that is to be placed in the output system stream - one sink pad will be requested. It can also be used in elements with a variable number of input or outputs pads, such as the tee (multi-output) or input-selector (multi-input) elements.

To implement request pads, you need to provide a padtemplate with a GST_PAD_REQUEST presence and implement therequest_new_pad virtual method in GstElement. To clean up, you will need to implement the release_padvirtual method.

static GstPad * gst_my_filter_request_new_pad   (GstElement     *element,
                         GstPadTemplate *templ,
                                                 const gchar    *name,
                                                 const GstCaps  *caps);

static void gst_my_filter_release_pad (GstElement *element,
                                       GstPad *pad);

static GstStaticPadTemplate sink_factory =
GST_STATIC_PAD_TEMPLATE (
  "sink_%u",
  GST_PAD_SINK,
  GST_PAD_REQUEST,
  GST_STATIC_CAPS ("ANY")
);

static void
gst_my_filter_class_init (GstMyFilterClass *klass)
{
  GstElementClass *element_class = GST_ELEMENT_CLASS (klass);
[..]
  gst_element_class_add_pad_template (klass,
    gst_static_pad_template_get (&sink_factory));
[..]
  element_class->request_new_pad = gst_my_filter_request_new_pad;
  element_class->release_pad = gst_my_filter_release_pad;
}

static GstPad *
gst_my_filter_request_new_pad (GstElement     *element,
                   GstPadTemplate *templ,
                   const gchar    *name,
                               const GstCaps  *caps)
{
  GstPad *pad;
  GstMyFilterInputContext *context;

  context = g_new0 (GstMyFilterInputContext, 1);
  pad = gst_pad_new_from_template (templ, name);
  gst_pad_set_element_private (pad, context);

  /* normally, you would set _chain () and _event () functions here */

  gst_element_add_pad (element, pad);

  return pad;
}

static void
gst_my_filter_release_pad (GstElement *element,
                           GstPad *pad)
{
  GstMyFilterInputContext *context;

  context = gst_pad_get_element_private (pad);
  g_free (context);

  gst_element_remove_pad (element, pad);
}

Different scheduling modes

The scheduling mode of a pad defines how data is retrieved from (source) or given to (sink) pads. GStreamer can operate in two scheduling mode, called push- and pull-mode. GStreamer supports elements with pads in any of the scheduling modes where not all pads need to be operating in the same mode.

So far, we have only discussed _chain ()-operating elements, i.e. elements that have a chain-function set on their sink pad and push buffers on their source pad(s). We call this the push-mode because a peer element will use gst_pad_push () on a srcpad, which will cause our _chain ()-function to be called, which in turn causes our element to push out a buffer on the source pad. The initiative to start the dataflow happens somewhere upstream when it pushes out a buffer and all downstream elements get scheduled when their _chain ()-functions are called in turn.

Before we explain pull-mode scheduling, let's first understand how the different scheduling modes are selected and activated on a pad.

The pad activation stage

During the element state change of READY->PAUSED, the pads of an element will be activated. This happens first on the source pads and then on the sink pads of the element. GStreamer calls the _activate () of a pad. By default this function will activate the pad in push-mode by calling gst_pad_activate_mode () with the GST_PAD_MODE_PUSH scheduling mode. It is possible to override the _activate () of a pad and decide on a different scheduling mode. You can know in what scheduling mode a pad is activated by overriding the_activate_mode ()-function.

GStreamer allows the different pads of an element to operate in different scheduling modes. This allows for many different possible use-cases. What follows is an overview of some typical use-cases.

If all pads of an element are activated in push-mode scheduling, the element as a whole is operating in push-mode. For source elements this means that they will have to start a task that pushes out buffers on the source pad to the downstream elements. Downstream elements will have data pushed to them by upstream elements using the sinkpads _chain ()-function which will push out buffers on the source pads. Prerequisites for this scheduling mode are that a chain-function was set for each sinkpad usinggst_pad_set_chain_function () and that all downstream elements operate in the same mode.
Alternatively, sinkpads can be the driving force behind a pipeline by operating in pull-mode, while the sourcepads of the element still operate in push-mode. In order to be the driving force, those pads start a GstTask when they are activated. This task is a thread, which will call a function specified by the element. When called, this function will have random data access (through gst_pad_pull_range ()) over all sinkpads, and can push data over the sourcepads, which effectively means that this element controls data flow in the pipeline. Prerequisites for this mode are that all downstream elements can act in push mode, and that all upstream elements operate in pull-mode (see below).

Source pads can be activated in PULL mode by a downstream element when they return GST_PAD_MODE_PULL from the GST_QUERY_SCHEDULING query. Prerequisites for this scheduling mode are that a getrange-function was set for the source pad using gst_pad_set_getrange_function ().
Lastly, all pads in an element can be activated in PULL-mode. However, contrary to the above, this does not mean that they start a task on their own. Rather, it means that they are pull slave for the downstream element, and have to provide random data access to it from their _get_range ()-function. Requirements are that the a _get_range ()-function was set on this pad using the functiongst_pad_set_getrange_function (). Also, if the element has any sinkpads, all those pads (and thereby their peers) need to operate in PULL access mode, too.

When a sink element is activated in PULL mode, it should start a task that calls gst_pad_pull_range () on its sinkpad. It can only do this when the upstream SCHEDULING query returns support for the GST_PAD_MODE_PULL scheduling mode.

In the next two sections, we will go closer into pull-mode scheduling (elements/pads driving the pipeline, and elements/pads providing random access), and some specific use cases will be given.

Pads driving the pipeline

Sinkpads operating in pull-mode, with the sourcepads operating in push-mode (or it has no sourcepads when it is a sink), can start a task that will drive the pipeline data flow. Within this task function, you have random access over all of the sinkpads, and push data over the sourcepads. This can come in useful for several different kinds of elements:

Demuxers, parsers and certain kinds of decoders where data comes in unparsed (such as MPEG-audio or video streams), since those will prefer byte-exact (random) access from their input. If possible, however, such elements should be prepared to operate in push-mode mode, too.
Certain kind of audio outputs, which require control over their input data flow, such as the Jack sound server.

First you need to perform a SCHEDULING query to check if the upstream element(s) support pull-mode scheduling. If that is possible, you can activate the sinkpad in pull-mode. Inside the activate_mode function you can then start the task.

#include "filter.h"
#include <string.h>

static gboolean gst_my_filter_activate      (GstPad      * pad,
                                             GstObject   * parent);
static gboolean gst_my_filter_activate_mode (GstPad      * pad,
                                             GstObject   * parent,
                                             GstPadMode    mode,
                         gboolean      active);
static void gst_my_filter_loop      (GstMyFilter * filter);

G_DEFINE_TYPE (GstMyFilter, gst_my_filter, GST_TYPE_ELEMENT);


static void
gst_my_filter_init (GstMyFilter * filter)
{

[..]

  gst_pad_set_activate_function (filter->sinkpad, gst_my_filter_activate);
  gst_pad_set_activatemode_function (filter->sinkpad,
      gst_my_filter_activate_mode);


[..]
}

[..]

static gboolean
gst_my_filter_activate (GstPad * pad, GstObject * parent)
{
  GstQuery *query;
  gboolean pull_mode;

  /* first check what upstream scheduling is supported */
  query = gst_query_new_scheduling ();

  if (!gst_pad_peer_query (pad, query)) {
    gst_query_unref (query);
    goto activate_push;
  }

  /* see if pull-mode is supported */
  pull_mode = gst_query_has_scheduling_mode_with_flags (query,
      GST_PAD_MODE_PULL, GST_SCHEDULING_FLAG_SEEKABLE);
  gst_query_unref (query);

  if (!pull_mode)
    goto activate_push;

  /* now we can activate in pull-mode. GStreamer will also
   * activate the upstream peer in pull-mode */
  return gst_pad_activate_mode (pad, GST_PAD_MODE_PULL, TRUE);

activate_push:
  {
    /* something not right, we fallback to push-mode */
    return gst_pad_activate_mode (pad, GST_PAD_MODE_PUSH, TRUE);
  }
}

static gboolean
gst_my_filter_activate_pull (GstPad    * pad,
                 GstObject * parent,
                 GstPadMode  mode,
                 gboolean    active)
{
  gboolean res;
  GstMyFilter *filter = GST_MY_FILTER (parent);

  switch (mode) {
    case GST_PAD_MODE_PUSH:
      res = TRUE;
      break;
    case GST_PAD_MODE_PULL:
      if (active) {
        filter->offset = 0;
        res = gst_pad_start_task (pad,
            (GstTaskFunction) gst_my_filter_loop, filter, NULL);
      } else {
        res = gst_pad_stop_task (pad);
      }
      break;
    default:
      /* unknown scheduling mode */
      res = FALSE;
      break;
  }
  return res;
}

Once started, your task has full control over input and output. The most simple case of a task function is one that reads input and pushes that over its source pad. It's not all that useful, but provides some more flexibility than the old push-mode case that we've been looking at so far.

#define BLOCKSIZE 2048

    static void
    gst_my_filter_loop (GstMyFilter * filter)
    {
      GstFlowReturn ret;
      guint64 len;
      GstBuffer *buf = NULL;

      if (!gst_pad_query_duration (filter->sinkpad, GST_FORMAT_BYTES, &len)) {
        GST_DEBUG_OBJECT (filter, "failed to query duration, pausing");
        goto stop;
      }

       if (filter->offset >= len) {
        GST_DEBUG_OBJECT (filter, "at end of input, sending EOS, pausing");
        gst_pad_push_event (filter->srcpad, gst_event_new_eos ());
        goto stop;
      }

      /* now, read BLOCKSIZE bytes from byte offset filter->offset */
      ret = gst_pad_pull_range (filter->sinkpad, filter->offset,
          BLOCKSIZE, &buf);

      if (ret != GST_FLOW_OK) {
        GST_DEBUG_OBJECT (filter, "pull_range failed: %s", gst_flow_get_name (ret));
        goto stop;
      }

      /* now push buffer downstream */
      ret = gst_pad_push (filter->srcpad, buf);

      buf = NULL; /* gst_pad_push() took ownership of buffer */

      if (ret != GST_FLOW_OK) {
        GST_DEBUG_OBJECT (filter, "pad_push failed: %s", gst_flow_get_name (ret));
        goto stop;
      }

      /* everything is fine, increase offset and wait for us to be called again */
      filter->offset += BLOCKSIZE;
      return;

    stop:
      GST_DEBUG_OBJECT (filter, "pausing task");
      gst_pad_pause_task (filter->sinkpad);
    }

Providing random access

In the previous section, we have talked about how elements (or pads) that are activated to drive the pipeline using their own task, must use pull-mode scheduling on their sinkpads. This means that all pads linked to those pads need to be activated in pull-mode. Source pads activated in pull-mode must implement a _get_range ()-function set using gst_pad_set_getrange_function (), and that function will be called when the peer pad requests some data with gst_pad_pull_range (). The element is then responsible for seeking to the right offset and providing the requested data. Several elements can implement random access:

Data sources, such as a file source, that can provide data from any offset with reasonable low latency.
Filters that would like to provide a pull-mode scheduling over the whole pipeline.
Parsers who can easily provide this by skipping a small part of their input and are thus essentially "forwarding" getrange requests literally without any own processing involved. Examples include tag readers (e.g. ID3) or single output parsers, such as a WAVE parser.

The following example will show how a _get_range ()-function can be implemented in a source element:

#include "filter.h"
static GstFlowReturn
        gst_my_filter_get_range (GstPad     * pad,
                     GstObject  * parent,
                     guint64      offset,
                     guint        length,
                     GstBuffer ** buf);

G_DEFINE_TYPE (GstMyFilter, gst_my_filter, GST_TYPE_ELEMENT);



static void
gst_my_filter_init (GstMyFilter * filter)
{

[..]

  gst_pad_set_getrange_function (filter->srcpad,
      gst_my_filter_get_range);

[..]
}

static GstFlowReturn
gst_my_filter_get_range (GstPad     * pad,
             GstObject  * parent,
             guint64      offset,
             guint        length,
             GstBuffer ** buf)
{

  GstMyFilter *filter = GST_MY_FILTER (parent);

  [.. here, you would fill *buf ..]

  return GST_FLOW_OK;
}

Caps negotiation

Caps negotiation is the act of finding a media format (GstCaps) between elements that they can handle. This process in GStreamer can in most cases find an optimal solution for the complete pipeline. In this section we explain how this works.

Caps negotiation basics

In GStreamer, negotiation of the media format always follows the following simple rules:

A downstream element suggest a format on its sinkpad and places the suggestion in the result of the CAPS query performed on the sinkpad. See also Implementing a CAPS query function.
An upstream element decides on a format. It sends the selected media format downstream on its source pad with a CAPS event. Downstream elements reconfigure themselves to handle the media type in the CAPS event on the sinkpad.
A downstream element can inform upstream that it would like to suggest a new format by sending a RECONFIGURE event upstream. The RECONFIGURE event simply instructs an upstream element to restart the negotiation phase. Because the element that sent out the RECONFIGURE event is now suggesting another format, the format in the pipeline might change.

In addition to the CAPS and RECONFIGURE event and the CAPS query, there is an ACCEPT_CAPS query to quickly check if a certain caps can be accepted by an element.

All negotiation follows these simple rules. Let's take a look at some typical uses cases and how negotiation happens.

Caps negotiation use cases

In what follows we will look at some use cases for push-mode scheduling. The pull-mode scheduling negotiation phase is discussed in Pull-mode Caps negotiation and is actually similar as we will see.

Since the sink pads only suggest formats and the source pads need to decide, the most complicated work is done in the source pads. We can identify 3 caps negotiation use cases for the source pads:

Fixed negotiation. An element can output one format only. See Fixed negotiation.
Transform negotiation. There is a (fixed) transform between the input and output format of the element, usually based on some element property. The caps that the element will produce depend on the upstream caps and the caps that the element can accept depend on the downstream caps. See Transform negotiation.
Dynamic negotiation. An element can output many formats. See Dynamic negotiation.

Fixed negotiation

In this case, the source pad can only produce a fixed format. Usually this format is encoded inside the media. No downstream element can ask for a different format, the only way that the source pad will renegotiate is when the element decides to change the caps itself.

Elements that could implement fixed caps (on their source pads) are, in general, all elements that are not renegotiable. Examples include:

A typefinder, since the type found is part of the actual data stream and can thus not be re-negotiated. The typefinder will look at the stream of bytes, figure out the type, send a CAPS event with the caps and then push buffers of the type.
Pretty much all demuxers, since the contained elementary data streams are defined in the file headers, and thus not renegotiable.
Some decoders, where the format is embedded in the data stream and not part of the peercaps and where the decoder itself is not reconfigurable, too.
Some sources that produce a fixed format.

gst_pad_use_fixed_caps() is used on the source pad with fixed caps. As long as the pad is not negotiated, the default CAPS query will return the caps presented in the padtemplate. As soon as the pad is negotiated, the CAPS query will return the negotiated caps (and nothing else). These are the relevant code snippets for fixed caps source pads.

[..]
  pad = gst_pad_new_from_static_template (..);
  gst_pad_use_fixed_caps (pad);
[..]

The fixed caps can then be set on the pad by calling gst_pad_set_caps ().

[..]
    caps = gst_caps_new_simple ("audio/x-raw",
        "format", G_TYPE_STRING, GST_AUDIO_NE(F32),
        "rate", G_TYPE_INT, <samplerate>,
        "channels", G_TYPE_INT, <num-channels>, NULL);
    if (!gst_pad_set_caps (pad, caps)) {
      GST_ELEMENT_ERROR (element, CORE, NEGOTIATION, (NULL),
          ("Some debug information here"));
      return GST_FLOW_ERROR;
    }
[..]

These types of elements also don't have a relation between the input format and the output format, the input caps simply don't contain the information needed to produce the output caps.

All other elements that need to be configured for the format should implement full caps negotiation, which will be explained in the next few sections.

Transform negotiation

In this negotiation technique, there is a fixed transform between the element input caps and the output caps. This transformation could be parameterized by element properties but not by the content of the stream (seeFixed negotiation for that use-case).

The caps that the element can accept depend on the (fixed transformation) downstream caps. The caps that the element can produce depend on the (fixed transformation of) the upstream caps.

This type of element can usually set caps on its source pad from the _event() function on the sink pad when it received the CAPS event. This means that the caps transform function transforms a fixed caps into another fixed caps. Examples of elements include:

Videobox. It adds configurable border around a video frame depending on object properties.
Identity elements. All elements that don't change the format of the data, only the content. Video and audio effects are an example. Other examples include elements that inspect the stream.
Some decoders and encoders, where the output format is defined by input format, like mulawdec and mulawenc. These decoders usually have no headers that define the content of the stream. They are usually more like conversion elements.

Below is an example of a negotiation steps of a typical transform element. In the sink pad CAPS event handler, we compute the caps for the source pad and set those.

  [...]

static gboolean
gst_my_filter_setcaps (GstMyFilter *filter,
               GstCaps *caps)
{
  GstStructure *structure;
  int rate, channels;
  gboolean ret;
  GstCaps *outcaps;

  structure = gst_caps_get_structure (caps, 0);
  ret = gst_structure_get_int (structure, "rate", &rate);
  ret = ret && gst_structure_get_int (structure, "channels", &channels);
  if (!ret)
    return FALSE;

  outcaps = gst_caps_new_simple ("audio/x-raw",
      "format", G_TYPE_STRING, GST_AUDIO_NE(S16),
      "rate", G_TYPE_INT, rate,
      "channels", G_TYPE_INT, channels, NULL);
  ret = gst_pad_set_caps (filter->srcpad, outcaps);
  gst_caps_unref (outcaps);

  return ret;
}

static gboolean
gst_my_filter_sink_event (GstPad    *pad,
                  GstObject *parent,
                  GstEvent  *event)
{
  gboolean ret;
  GstMyFilter *filter = GST_MY_FILTER (parent);

  switch (GST_EVENT_TYPE (event)) {
    case GST_EVENT_CAPS:
    {
      GstCaps *caps;

      gst_event_parse_caps (event, &caps);
      ret = gst_my_filter_setcaps (filter, caps);
      break;
    }
    default:
      ret = gst_pad_event_default (pad, parent, event);
      break;
  }
  return ret;
}

  [...]

Dynamic negotiation

A last negotiation method is the most complex and powerful dynamic negotiation.

Like with the transform negotiation in Transform negotiation, dynamic negotiation will perform a transformation on the downstream/upstream caps. Unlike the transform negotiation, this transform will convert fixed caps to unfixed caps. This means that the sink pad input caps can be converted into unfixed (multiple) formats. The source pad will have to choose a format from all the possibilities. It would usually like to choose a format that requires the least amount of effort to produce but it does not have to be. The selection of the format should also depend on the caps that can be accepted downstream (see a QUERY_CAPS function inImplementing a CAPS query function).

A typical flow goes like this:

Caps are received on the sink pad of the element.
If the element prefers to operate in passthrough mode, check if downstream accepts the caps with the ACCEPT_CAPS query. If it does, we can complete negotiation and we can operate in passthrough mode.
Calculate the possible caps for the source pad.
Query the downstream peer pad for the list of possible caps.
Select from the downstream list the first caps that you can transform to and set this as the output caps. You might have to fixate the caps to some reasonable defaults to construct fixed caps.

Examples of this type of elements include:

Converter elements such as videoconvert, audioconvert, audioresample, videoscale, ...
Source elements such as audiotestsrc, videotestsrc, v4l2src, pulsesrc, ...

Let's look at the example of an element that can convert between samplerates, so where input and output samplerate don't have to be the same:

static gboolean
gst_my_filter_setcaps (GstMyFilter *filter,
               GstCaps *caps)
{
  if (gst_pad_set_caps (filter->srcpad, caps)) {
    filter->passthrough = TRUE;
  } else {
    GstCaps *othercaps, *newcaps;
    GstStructure *s = gst_caps_get_structure (caps, 0), *others;

    /* no passthrough, setup internal conversion */
    gst_structure_get_int (s, "channels", &filter->channels);
    othercaps = gst_pad_get_allowed_caps (filter->srcpad);
    others = gst_caps_get_structure (othercaps, 0);
    gst_structure_set (others,
      "channels", G_TYPE_INT, filter->channels, NULL);

    /* now, the samplerate value can optionally have multiple values, so
     * we "fixate" it, which means that one fixed value is chosen */
    newcaps = gst_caps_copy_nth (othercaps, 0);
    gst_caps_unref (othercaps);
    gst_pad_fixate_caps (filter->srcpad, newcaps);
    if (!gst_pad_set_caps (filter->srcpad, newcaps))
      return FALSE;

    /* we are now set up, configure internally */
    filter->passthrough = FALSE;
    gst_structure_get_int (s, "rate", &filter->from_samplerate);
    others = gst_caps_get_structure (newcaps, 0);
    gst_structure_get_int (others, "rate", &filter->to_samplerate);
  }

  return TRUE;
}

static gboolean
gst_my_filter_sink_event (GstPad    *pad,
                  GstObject *parent,
                  GstEvent  *event)
{
  gboolean ret;
  GstMyFilter *filter = GST_MY_FILTER (parent);

  switch (GST_EVENT_TYPE (event)) {
    case GST_EVENT_CAPS:
    {
      GstCaps *caps;

      gst_event_parse_caps (event, &caps);
      ret = gst_my_filter_setcaps (filter, caps);
      break;
    }
    default:
      ret = gst_pad_event_default (pad, parent, event);
      break;
  }
  return ret;
}

static GstFlowReturn
gst_my_filter_chain (GstPad    *pad,
             GstObject *parent,
             GstBuffer *buf)
{
  GstMyFilter *filter = GST_MY_FILTER (parent);
  GstBuffer *out;

  /* push on if in passthrough mode */
  if (filter->passthrough)
    return gst_pad_push (filter->srcpad, buf);

  /* convert, push */
  out = gst_my_filter_convert (filter, buf);
  gst_buffer_unref (buf);

  return gst_pad_push (filter->srcpad, out);
}

Upstream caps (re)negotiation

Upstream negotiation's primary use is to renegotiate (part of) an already-negotiated pipeline to a new format. Some practical examples include to select a different video size because the size of the video window changed, and the video output itself is not capable of rescaling, or because the audio channel configuration changed.

Upstream caps renegotiation is requested by sending a GST_EVENT_RECONFIGURE event upstream. The idea is that it will instruct the upstream element to reconfigure its caps by doing a new query for the allowed caps and then choosing a new caps. The element that sends out the RECONFIGURE event would influence the selection of the new caps by returning the new preferred caps from its GST_QUERY_CAPS query function. The RECONFIGURE event will set the GST_PAD_FLAG_NEED_RECONFIGURE on all pads that it travels over.

It is important to note here that different elements actually have different responsibilities here:

Elements that want to propose a new format upstream need to first check if the new caps are acceptable upstream with an ACCEPT_CAPS query. Then they would send a RECONFIGURE event and be prepared to answer the CAPS query with the new preferred format. It should be noted that when there is no upstream element that can (or wants) to renegotiate, the element needs to deal with the currently configured format.
Elements that operate in transform negotiation according to Transform negotiation pass the RECONFIGURE event upstream. Because these elements simply do a fixed transform based on the upstream caps, they need to send the event upstream so that it can select a new format.
Elements that operate in fixed negotiation (Fixed negotiation) drop the RECONFIGURE event. These elements can't reconfigure and their output caps don't depend on the upstream caps so the event can be dropped.
Elements that can be reconfigured on the source pad (source pads implementing dynamic negotiation inDynamic negotiation) should check its NEED_RECONFIGURE flag with gst_pad_check_reconfigure () and it should start renegotiation when the function returns TRUE.

Implementing a CAPS query function

A _query ()-function with the GST_QUERY_CAPS query type is called when a peer element would like to know which formats this pad supports, and in what order of preference. The return value should be all formats that this elements supports, taking into account limitations of peer elements further downstream or upstream, sorted by order of preference, highest preference first.

static gboolean
gst_my_filter_query (GstPad *pad, GstObject * parent, GstQuery * query)
{
  gboolean ret;
  GstMyFilter *filter = GST_MY_FILTER (parent);

  switch (GST_QUERY_TYPE (query)) {
    case GST_QUERY_CAPS
    {
      GstPad *otherpad;
      GstCaps *temp, *caps, *filt, *tcaps;
      gint i;

      otherpad = (pad == filter->srcpad) ? filter->sinkpad :
                                           filter->srcpad;
      caps = gst_pad_get_allowed_caps (otherpad);

      gst_query_parse_caps (query, &filt);

      /* We support *any* samplerate, indifferent from the samplerate
       * supported by the linked elements on both sides. */
      for (i = 0; i < gst_caps_get_size (caps); i++) {
        GstStructure *structure = gst_caps_get_structure (caps, i);

        gst_structure_remove_field (structure, "rate");
      }

      /* make sure we only return results that intersect our
       * padtemplate */
      tcaps = gst_pad_get_pad_template_caps (pad);
      if (tcaps) {
        temp = gst_caps_intersect (caps, tcaps);
        gst_caps_unref (caps);
        gst_caps_unref (tcaps);
        caps = temp;
      }
      /* filter against the query filter when needed */
      if (filt) {
        temp = gst_caps_intersect (caps, filt);
        gst_caps_unref (caps);
        caps = temp;
      }
      gst_query_set_caps_result (query, caps);
      gst_caps_unref (caps);
      ret = TRUE;
      break;
    }
    default:
      ret = gst_pad_query_default (pad, parent, query);
      break;
  }
  return ret;
}

Memory allocation

Memory allocation and management are very important topics in multimedia. High definition video uses many megabytes to store one single image frame. It is important to reuse memory when possible instead of constantly allocating and freeing it.

Multimedia systems usually use special-purpose chips, such as DSPs or GPUs to perform the heavy lifting (especially for video). These special-purpose chips usually have strict requirements for the memory they operate on and how it is accessed.

This chapter talks about the memory-management features available to GStreamer plugins. We will first talk about the lowlevel GstMemory object that manages access to a piece of memory and then continue with one of it's main users, the GstBuffer, which is used to exchange data between plugins and with the application. We will also discuss the GstMeta. This object can be placed on buffers to provide extra info about it and its memory. We will also discuss the GstBufferPool, which allows to more-efficiently manage buffers of the same size.

To conclude this chapter we will take a look at the GST_QUERY_ALLOCATION query, which is used to negotiate memory management options between elements.

GstMemory

GstMemory is an object that manages a region of memory. This memory object points to a region of memory of “maxsize”. The area in this memory starting at “offset” and size “size” bytes is the accessible memory region. After a GstMemory is created its maxsize can no longer be changed, however, its "offset" and "size" can.

GstAllocator

GstMemory objects are created by a GstAllocator object. Most allocators implement the defaultgst_allocator_alloc() method but some might implement different ones, for example, when additional parameters are needed to allocate the specific memory.

Different allocators exist for system memory, shared memory and memory backed by a DMAbuf file descriptor. To implement support for a new kind of memory type, you must implement a new allocator object.

GstMemory API example

Data access to the memory wrapped by the GstMemory object is always protected with a gst_memory_map()and gst_memory_unmap() pair. An access mode (read/write) must be given when mapping memory. The map function returns a pointer to the valid memory region that can then be accessed according to the requested access mode.

Below is an example on creating a GstMemory object and using the gst_memory_map() to access the memory region.

[...]

  GstMemory *mem;
  GstMapInfo info;
  gint i;

  /* allocate 100 bytes */
  mem = gst_allocator_alloc (NULL, 100, NULL);

  /* get access to the memory in write mode */
  gst_memory_map (mem, &info, GST_MAP_WRITE);

  /* fill with pattern */
  for (i = 0; i < info.size; i++)
    info.data[i] = i;

  /* release memory */
  gst_memory_unmap (mem, &info);

[...]

GstBuffer

A GstBuffer is a lightweight object that is passed from an upstream to a downstream element and contains memory and metadata. It represents the multimedia content that is pushed to or pulled by downstream elements.

A GstBuffer contains one or more GstMemory objects. These objects hold the buffer's data.

Metadata in the buffer consists of:

DTS and PTS timestamps. These represent the decoding and presentation timestamps of the buffer content and are used by synchronizing elements to schedule buffers. These timestamps can beGST_CLOCK_TIME_NONE when unknown/undefined.
The duration of the buffer contents. This duration can be GST_CLOCK_TIME_NONE when unknown/undefined.
Media-specific offset and offset_end values. For video this is the frame number in the stream, for audio, the sample number. Other media might use different definitions.
Arbitrary structures via GstMeta, see below.

Writability

A GstBuffer is writable when the refcount of the object is exactly 1, meaning that only one object is holding a ref to the buffer. You can only modify the buffer when it is writable. This means that you need to call gst_buffer_make_writable() before changing the timestamps, offsets, metadata or adding and removing memory blocks.

API examples

You can create a GstBuffer with gst_buffer_new () and then you can add memory objects to it. You can alternatively use the convenience function gst_buffer_new_allocate () to perform both operations at once. It's also possible to wrap existing memory with gst_buffer_new_wrapped_full () and specify the function to call when the memory should be freed.

You can access the memory of a GstBuffer by getting and mapping the GstMemory objects individually or by using gst_buffer_map (). The latter merges all the memory into one big block and then gives you a pointer to it.

Below is an example of how to create a buffer and access its memory.

[...]
  GstBuffer *buffer;
  GstMemory *mem;
  GstMapInfo info;

  /* make empty buffer */
  buffer = gst_buffer_new ();

  /* make memory holding 100 bytes */
  mem = gst_allocator_alloc (NULL, 100, NULL);

  /* add the buffer */
  gst_buffer_append_memory (buffer, mem);

[...]

  /* get WRITE access to the memory and fill with 0xff */
  gst_buffer_map (buffer, &info, GST_MAP_WRITE);
  memset (info.data, 0xff, info.size);
  gst_buffer_unmap (buffer, &info);

[...]

  /* free the buffer */
  gst_buffer_unref (buffer);

[...]

GstMeta

With the GstMeta system you can add arbitrary structures to buffers. These structures describe extra properties of the buffer such as cropping, stride, region of interest, etc.

The metadata system separates API specification (what the metadata and its API look like) and its implementation (how it works). This makes it possible to have different implementations of the same API, for example, depending on the hardware you are running on.

API example

After allocating a new GstBuffer, you can add metadata to it with the metadata-specific API. This means that you will need to link to the header file where the metadata is defined to use its API.

By convention, a metadata API with name FooBar should provide two methods, agst_buffer_add_foo_bar_meta () and a gst_buffer_get_foo_bar_meta (). Both functions should return a pointer to a FooBarMeta structure that contains the metadata fields. Some of the _add_*_meta () can have extra parameters that will usually be used to configure the metadata structure for you.

Let's have a look at the metadata that is used to specify a cropping region for video frames.

#include <gst/video/gstvideometa.h>

[...]
  GstVideoCropMeta *meta;

  /* buffer points to a video frame, add some cropping metadata */
  meta = gst_buffer_add_video_crop_meta (buffer);

  /* configure the cropping metadata */
  meta->x = 8;
  meta->y = 8;
  meta->width = 120;
  meta->height = 80;
[...]

An element can then use the metadata on the buffer when rendering the frame like this:

#include <gst/video/gstvideometa.h>

[...]
  GstVideoCropMeta *meta;

  /* buffer points to a video frame, get the cropping metadata */
  meta = gst_buffer_get_video_crop_meta (buffer);

  if (meta) {
    /* render frame with cropping */
    _render_frame_cropped (buffer, meta->x, meta->y, meta->width, meta->height);
  } else {
    /* render frame */
    _render_frame (buffer);
  }
[...]

Implementing new GstMeta

In the next sections we show how you can add new metadata to the system and use it on buffers.

Define the metadata API

First we need to define what our API will look like and we have to register this API to the system. This is important because the API definition will be used when elements negotiate what kind of metadata they will exchange. The API definition also contains arbitrary tags that give hints about what the metadata contains. This is important when we see how metadata is preserved as buffers pass through the pipeline.

If you are making a new implementation of an existing API, you can skip this step and move directly to the implementation.

First we start with making the my-example-meta.h header file that will contain the definition of the API and structure for our metadata.

#include <gst/gst.h>

typedef struct _MyExampleMeta MyExampleMeta;

struct _MyExampleMeta {
  GstMeta       meta;

  gint          age;
  gchar        *name;
};

GType my_example_meta_api_get_type (void);
#define MY_EXAMPLE_META_API_TYPE (my_example_meta_api_get_type())

#define gst_buffer_get_my_example_meta(b) \
  ((MyExampleMeta*)gst_buffer_get_meta((b),MY_EXAMPLE_META_API_TYPE))

The metadata API definition consists of the definition of the structure that holds a gint and a string. The first field in the structure must be a GstMeta.

We also define a my_example_meta_api_get_type () function that will register our metadata API definition and a convenience gst_buffer_get_my_example_meta () macro that simply finds and returns the metadata with our new API.

Let's have a look at how the my_example_meta_api_get_type () function is implemented in the my-example-meta.c file:

#include "my-example-meta.h"

GType
my_example_meta_api_get_type (void)
{
  static volatile GType type;
  static const gchar *tags[] = { "foo", "bar", NULL };

  if (g_once_init_enter (&type)) {
    GType _type = gst_meta_api_type_register ("MyExampleMetaAPI", tags);
    g_once_init_leave (&type, _type);
  }
  return type;
}

As you can see, it simply uses the gst_meta_api_type_register () function to register a name and some tags for the API. The result is a new GType pointer that defines the newly registered API.

Implementing a metadata API

Next we can make an implementation for a registered metadata API GType.

The implementation details of a metadata API are kept in a GstMetaInfo structure that you make available to the users of your metadata API implementation with a my_example_meta_get_info () function and a convenience MY_EXAMPLE_META_INFO macro. You also provide a method to add your metadata implementation to aGstBuffer. Your my-example-meta.h header file will need these additions:

[...]

/* implementation */
const GstMetaInfo *my_example_meta_get_info (void);
#define MY_EXAMPLE_META_INFO (my_example_meta_get_info())

MyExampleMeta * gst_buffer_add_my_example_meta (GstBuffer      *buffer,
                                                gint            age,
                                                const gchar    *name);

Let's have a look at how these functions are implemented in the my-example-meta.c file.

[...]

static gboolean
my_example_meta_init (GstMeta * meta, gpointer params, GstBuffer * buffer)
{
  MyExampleMeta *emeta = (MyExampleMeta *) meta;

  emeta->age = 0;
  emeta->name = NULL;

  return TRUE;
}

static gboolean
my_example_meta_transform (GstBuffer * transbuf, GstMeta * meta,
    GstBuffer * buffer, GQuark type, gpointer data)
{
  MyExampleMeta *emeta = (MyExampleMeta *) meta;

  /* we always copy no matter what transform */
  gst_buffer_add_my_example_meta (transbuf, emeta->age, emeta->name);

  return TRUE;
}

static void
my_example_meta_free (GstMeta * meta, GstBuffer * buffer)
{
  MyExampleMeta *emeta = (MyExampleMeta *) meta;

  g_free (emeta->name);
  emeta->name = NULL;
}

const GstMetaInfo *
my_example_meta_get_info (void)
{
  static const GstMetaInfo *meta_info = NULL;

  if (g_once_init_enter (&meta_info)) {
    const GstMetaInfo *mi = gst_meta_register (MY_EXAMPLE_META_API_TYPE,
        "MyExampleMeta",
        sizeof (MyExampleMeta),
        my_example_meta_init,
        my_example_meta_free,
        my_example_meta_transform);
    g_once_init_leave (&meta_info, mi);
  }
  return meta_info;
}

MyExampleMeta *
gst_buffer_add_my_example_meta (GstBuffer   *buffer,
                                gint         age,
                                const gchar *name)
{
  MyExampleMeta *meta;

  g_return_val_if_fail (GST_IS_BUFFER (buffer), NULL);

  meta = (MyExampleMeta *) gst_buffer_add_meta (buffer,
      MY_EXAMPLE_META_INFO, NULL);

  meta->age = age;
  meta->name = g_strdup (name);

  return meta;
}

gst_meta_register () registers the implementation details, like the API that you implement and the size of the metadata structure, alongside methods to initialize and free the memory area. You can also implement a transform function that will be called when a certain transformation (identified by the quark and quark specific data) is performed on a buffer.

Lastly, you implement a gst_buffer_add_*_meta() that adds the metadata implementation to a buffer and sets the values of the metadata.

GstBufferPool

The GstBufferPool object provides a convenient base class for managing lists of reusable buffers. Essential for this object is that all the buffers have the same properties such as size, padding, metadata and alignment.

A GstBufferPool can be configured to manage a minimum and maximum amount of buffers of a specific size. It can also be configured to use a specific GstAllocator for the memory of the buffers. There is also support in the bufferpool to enable bufferpool specific options, such as adding GstMeta to the pool's buffers or enabling specific padding on the buffers' memory.

A GstBufferPool can be either inactivate or active. In the inactive state, you can configure the pool. In the active state, you can't change the configuration anymore but you can acquire and release buffers from/to the pool.

In the following sections we take a look at how you can use a GstBufferPool.

API example

There can be many different GstBufferPool implementations; they are all subclasses of the GstBufferPoolbase class. For this example, we will assume we somehow have access to a buffer pool, either because we created it ourselves or because we were given one as a result of the ALLOCATION query, as we will see below.

The GstBufferPool is initially in the inactive state so that we can configure it. Trying to configure aGstBufferPool that is not in the inactive state will fail. Likewise, trying to activate a bufferpool that is not configured will also fail.

 GstStructure *config;

[...]

  /* get config structure */
  config = gst_buffer_pool_get_config (pool);

  /* set caps, size, minimum and maximum buffers in the pool */
  gst_buffer_pool_config_set_params (config, caps, size, min, max);

  /* configure allocator and parameters */
  gst_buffer_pool_config_set_allocator (config, allocator, &params);

  /* store the updated configuration again */
  gst_buffer_pool_set_config (pool, config);

[...]

The configuration of a GstBufferPool is maintained in a generic GstStructure that can be obtained withgst_buffer_pool_get_config(). Convenience methods exist to get and set the configuration options in this structure. After updating the structure, it is set as the current configuration in the GstBufferPool again with gst_buffer_pool_set_config().

The following options can be configured on a GstBufferPool:

The caps of the buffers to allocate.
The size of the buffers. This is the suggested size of the buffers in the pool. The pool might decide to allocate larger buffers to add padding.
The minimum and maximum amount of buffers in the pool. When minimum is set to \> 0, the bufferpool will pre-allocate this amount of buffers. When maximum is not 0, the bufferpool will allocate up to maximum amount of buffers.
The allocator and parameters to use. Some bufferpools might ignore the allocator and use its internal one.
Other arbitrary bufferpool options identified with a string. a bufferpool lists the supported options with gst_buffer_pool_get_options() and you can ask if an option is supported withgst_buffer_pool_has_option(). The option can be enabled by adding it to the configuration structure with gst_buffer_pool_config_add_option (). These options are used to enable things like letting the pool set metadata on the buffers or to add extra configuration options for padding, for example.

After the configuration is set on the bufferpool, the pool can be activated withgst_buffer_pool_set_active (pool, TRUE). From that point on you can usegst_buffer_pool_acquire_buffer () to retrieve a buffer from the pool, like this:

[...]

  GstFlowReturn ret;
  GstBuffer *buffer;

  ret = gst_buffer_pool_acquire_buffer (pool, &buffer, NULL);
  if (G_UNLIKELY (ret != GST_FLOW_OK))
    goto pool_failed;

  [...]

It is important to check the return value of the acquire function because it is possible that it fails: When your element shuts down, it will deactivate the bufferpool and then all calls to acquire will returnGST_FLOW_FLUSHING.

All buffers that are acquired from the pool will have their pool member set to the original pool. When the last ref is decremented on the buffer, GStreamer will automatically callgst_buffer_pool_release_buffer() to release the buffer back to the pool. You (or any other downstream element) don't need to know if a buffer came from a pool, you can just unref it.

Pre-made base classes

So far, we've been looking at low-level concepts of creating any type of GStreamer element. Now, let's assume that all you want is to create an simple audiosink that works exactly the same as, say, “esdsink”, or a filter that simply normalizes audio volume. Such elements are very general in concept and since they do nothing special, they should be easier to code than to provide your own scheduler activation functions and doing complex caps negotiation. For this purpose, GStreamer provides base classes that simplify some types of elements. Those base classes will be discussed in this chapter.

Writing a sink

Sinks are special elements in GStreamer. This is because sink elements have to take care of preroll, which is the process that takes care that elements going into the GST_STATE_PAUSED state will have buffers ready after the state change. The result of this is that such elements can start processing data immediately after going into the GST_STATE_PLAYING state, without requiring to take some time to initialize outputs or set up decoders; all that is done already before the state-change to GST_STATE_PAUSED successfully completes.

Preroll, however, is a complex process that would require the same code in many elements. Therefore, sink elements can derive from the GstBaseSink base-class, which does preroll and a few other utility functions automatically. The derived class only needs to implement a bunch of virtual functions and will work automatically.

The base class implement much of the synchronization logic that a sink has to perform.

The GstBaseSink base-class specifies some limitations on elements, though:

It requires that the sink only has one sinkpad. Sink elements that need more than one sinkpad, must make a manager element with multiple GstBaseSink elements inside.

Sink elements can derive from GstBaseSink using the usual GObject convenience macro G_DEFINE_TYPE ():

G_DEFINE_TYPE (GstMySink, gst_my_sink, GST_TYPE_BASE_SINK);

[..]

static void
gst_my_sink_class_init (GstMySinkClass * klass)
{
  klass->set_caps = [..];
  klass->render = [..];
[..]
}

The advantages of deriving from GstBaseSink are numerous:

Derived implementations barely need to be aware of preroll, and do not need to know anything about the technical implementation requirements of preroll. The base-class does all the hard work.

Less code to write in the derived class, shared code (and thus shared bugfixes).

There are also specialized base classes for audio and video, let's look at those a bit.

Writing an audio sink

Essentially, audio sink implementations are just a special case of a general sink. An audio sink has the added complexity that it needs to schedule playback of samples. It must match the clock selected in the pipeline against the clock of the audio device and calculate and compensate for drift and jitter.

There are two audio base classes that you can choose to derive from, depending on your needs:GstAudioBasesink and GstAudioSink. The audiobasesink provides full control over how synchronization and scheduling is handled, by using a ringbuffer that the derived class controls and provides. The audiosink base-class is a derived class of the audiobasesink, implementing a standard ringbuffer implementing default synchronization and providing a standard audio-sample clock. Derived classes of this base class merely need to provide a _open (), _close () and a _write () function implementation, and some optional functions. This should suffice for many sound-server output elements and even most interfaces. More demanding audio systems, such as Jack, would want to implement the GstAudioBaseSink base-class.

The GstAudioBaseSink has little to no limitations and should fit virtually every implementation, but is hard to implement. The GstAudioSink, on the other hand, only fits those systems with a simple open () /close () / write () API (which practically means pretty much all of them), but has the advantage that it is a lot easier to implement. The benefits of this second base class are large:

Automatic synchronization, without any code in the derived class.
Also automatically provides a clock, so that other sinks (e.g. in case of audio/video playback) are synchronized.
Features can be added to all audiosinks by making a change in the base class, which makes maintenance easy.
Derived classes require only three small functions, plus some GObject boilerplate code.

In addition to implementing the audio base-class virtual functions, derived classes can (should) also implement the GstBaseSink set_caps () and get_caps () virtual functions for negotiation.

Writing a video sink

Writing a videosink can be done using the GstVideoSink base-class, which derives from GstBaseSinkinternally. Currently, it does nothing yet but add another compile dependency, so derived classes will need to implement all base-sink virtual functions. When they do this correctly, this will have some positive effects on the end user experience with the videosink:

Because of preroll (and the preroll () virtual function), it is possible to display a video frame already when going into the GST_STATE_PAUSED state.
By adding new features to GstVideoSink, it will be possible to add extensions to videosinks that affect all of them, but only need to be coded once, which is a huge maintenance benefit.

Writing a source

In the previous part, particularly Providing random access, we have learned that some types of elements can provide random access. This applies most definitely to source elements reading from a randomly seekable location, such as file sources. However, other source elements may be better described as a live source element, such as a camera source, an audio card source and such; those are not seekable and do not provide byte-exact access. For all such use cases, GStreamer provides two base classes: GstBaseSrc for the basic source functionality, and GstPushSrc, which is a non-byte exact source base-class. The pushsource base class itself derives from basesource as well, and thus all statements about the basesource apply to the pushsource, too.

The basesrc class does several things automatically for derived classes, so they no longer have to worry about it:

Fixes to GstBaseSrc apply to all derived classes automatically.
Automatic pad activation handling, and task-wrapping in case we get assigned to start a task ourselves.

The GstBaseSrc may not be suitable for all cases, though; it has limitations:

There is one and only one sourcepad. Source elements requiring multiple sourcepads must implement a manager bin and use multiple source elements internally or make a manager element that uses a source element and a demuxer inside.

It is possible to use special memory, such as X server memory pointers or mmap ()'ed memory areas, as data pointers in buffers returned from the create() virtual function.

Writing an audio source

An audio source is nothing more but a special case of a pushsource. Audio sources would be anything that reads audio, such as a source reading from a soundserver, a kernel interface (such as ALSA) or a test sound / signal generator. GStreamer provides two base classes, similar to the two audiosinks described in Writing an audio sink; one is ringbuffer-based, and requires the derived class to take care of its own scheduling, synchronization and such. The other is based on this GstAudioBaseSrc and is called GstAudioSrc, and provides a simple open (), close () and read () interface, which is rather simple to implement and will suffice for most soundserver sources and audio interfaces (e.g. ALSA or OSS) out there.

The GstAudioSrc base-class has several benefits for derived classes, on top of the benefits of theGstPushSrc base-class that it is based on:

Does syncronization and provides a clock.
New features can be added to it and will apply to all derived classes automatically.

Writing a transformation element

A third base-class that GStreamer provides is the GstBaseTransform. This is a base class for elements with one sourcepad and one sinkpad which act as a filter of some sort, such as volume changing, audio resampling, audio format conversion, and so on and so on. There is quite a lot of bookkeeping that such elements need to do in order for things such as buffer allocation forwarding, passthrough, in-place processing and such to all work correctly. This base class does all that for you, so that you just need to do the actual processing.

Since the GstBaseTransform is based on the 1-to-1 model for filters, it may not apply well to elements such as decoders, which may have to parse properties from the stream. Also, it will not work for elements requiring more than one sourcepad or sinkpad.

posted @ 2020-05-16 15:46 fellow_jing 阅读(749) 评论(0) 编辑收藏举报

刷新页面返回顶部

fellow_jing

转载：Advanced Concepts of Gstreamer Plugin

Request and Sometimes pads

Sometimes pads

Request pads

Different scheduling modes

The pad activation stage

Pads driving the pipeline

Providing random access

Caps negotiation

Caps negotiation basics

Caps negotiation use cases

Fixed negotiation

Transform negotiation

Dynamic negotiation

Upstream caps (re)negotiation

Implementing a CAPS query function

Memory allocation

GstMemory

GstAllocator

GstMemory API example

GstBuffer

Writability

API examples

GstMeta

API example

Implementing new GstMeta

Define the metadata API

Implementing a metadata API

GstBufferPool

API example

Pre-made base classes

Writing a sink

Writing an audio sink

Writing a video sink

Writing a source

Writing an audio source

Writing a transformation element

公告