ZeroDream/gstreamer
2
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Update

Browse Source
This commit is contained in:
Akkariin Meiko
2022-03-12 03:16:09 +08:00
Unverified
parent 12b76e0c7a
commit 27c4ec74a1
10075 changed files with 5122287 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
subprojects/gst-docs/markdown/plugin-development/introduction/basics.md
+272
View File
@@ -0,0 +1,272 @@
---
title: Foundations
...
# Foundations
This chapter of the guide introduces the basic concepts of GStreamer.
Understanding these concepts will help you grok the issues involved in
extending GStreamer. Many of these concepts are explained in greater
detail in the *GStreamer Application Development Manual*; the basic
concepts presented here serve mainly to refresh your memory.
## Elements and Plugins
Elements are at the core of GStreamer. In the context of plugin
development, an *element* is an object derived from the [`
GstElement`](GstElement) class. Elements
provide some sort of functionality when linked with other elements: For
example, a source element provides data to a stream, and a filter
element acts on the data in a stream. Without elements, GStreamer is
just a bunch of conceptual pipe fittings with nothing to link. A large
number of elements ship with GStreamer, but extra elements can also be
written.
Just writing a new element is not entirely enough, however: You will
need to encapsulate your element in a *plugin* to enable GStreamer to
use it. A plugin is essentially a loadable block of code, usually called
a shared object file or a dynamically linked library. A single plugin
may contain the implementation of several elements, or just a single
one. For simplicity, this guide concentrates primarily on plugins
containing one element.
A *filter* is an important type of element that processes a stream of
data. Producers and consumers of data are called *source* and *sink*
elements, respectively. *Bin* elements contain other elements. One type
of bin is responsible for synchronization of the elements that they
contain so that data flows smoothly. Another type of bin, called
*autoplugger* elements, automatically add other elements to the bin and
links them together so that they act as a filter between two arbitrary
stream types.
The plugin mechanism is used everywhere in GStreamer, even if only the
standard packages are being used. A few very basic functions reside in
the core library, and all others are implemented in plugins. A plugin
registry is used to store the details of the plugins in a binary
registry file. This way, a program using GStreamer does not have to load
all plugins to determine which are needed. Plugins are only loaded when
their provided elements are requested.
See the *GStreamer Library Reference* for the current implementation
details of [`GstElement`](GstElement) and [`GstPlugin`](GstPlugin).
## Pads
*Pads* are used to negotiate links and data flow between elements in
GStreamer. A pad can be viewed as a “place” or “port” on an element
where links may be made with other elements, and through which data can
flow to or from those elements. Pads have specific data handling
capabilities: A pad can restrict the type of data that flows through it.
Links are only allowed between two pads when the allowed data types of
the two pads are compatible.
An analogy may be helpful here. A pad is similar to a plug or jack on a
physical device. Consider, for example, a home theater system consisting
of an amplifier, a DVD player, and a (silent) video projector. Linking
the DVD player to the amplifier is allowed because both devices have
audio jacks, and linking the projector to the DVD player is allowed
because both devices have compatible video jacks. Links between the
projector and the amplifier may not be made because the projector and
amplifier have different types of jacks. Pads in GStreamer serve the
same purpose as the jacks in the home theater system.
For the most part, all data in GStreamer flows one way through a link
between elements. Data flows out of one element through one or more
*source pads*, and elements accept incoming data through one or more
*sink pads*. Source and sink elements have only source and sink pads,
respectively.
See the *GStreamer Library Reference* for the current implementation
details of a [`GstPad`](GstPad).
## GstMiniObject, Buffers and Events
All streams of data in GStreamer are chopped up into chunks that are
passed from a source pad on one element to a sink pad on another
element. *GstMiniObject* is the structure used to hold these chunks of
data.
GstMiniObject contains the following important types:
- An exact type indicating what type of data (event, buffer, ...) this
GstMiniObject is.
- A reference count indicating the number of elements currently
holding a reference to the miniobject. When the reference count
falls to zero, the miniobject will be disposed, and its memory will
be freed in some sense (see below for more details).
For data transport, there are two types of GstMiniObject defined: events
(control) and buffers (content).
Buffers may contain any sort of data that the two linked pads know how
to handle. Normally, a buffer contains a chunk of some sort of audio or
video data that flows from one element to another.
Buffers also contain metadata describing the buffer's contents. Some of
the important types of metadata are:
- Pointers to one or more GstMemory objects. GstMemory objects are
refcounted objects that encapsulate a region of memory.
- A timestamp indicating the preferred display timestamp of the
content in the buffer.
Events contain information on the state of the stream flowing between
the two linked pads. Events will only be sent if the element explicitly
supports them, else the core will (try to) handle the events
automatically. Events are used to indicate, for example, a media type,
the end of a media stream or that the cache should be flushed.
Events may contain several of the following items:
- A subtype indicating the type of the contained event.
- The other contents of the event depend on the specific event type.
Events will be discussed extensively in [Events: Seeking, Navigation and
More](plugin-development/advanced/events.md). Until then, the only event that
will be used is the *EOS* event, which is used to indicate the end-of-stream
(usually end-of-file).
See the *GStreamer Library Reference* for the current implementation
details of a [`GstMiniObject`](GstMiniObject), [`GstBuffer`](GstBuffer)
and [`GstEvent`](GstEvent).
### Buffer Allocation
Buffers are able to store chunks of memory of several different types.
The most generic type of buffer contains memory allocated by malloc().
Such buffers, although convenient, are not always very fast, since data
often needs to be specifically copied into the buffer.
Many specialized elements create buffers that point to special memory.
For example, the filesrc element usually maps a file into the address
space of the application (using mmap()), and creates buffers that point
into that address range. These buffers created by filesrc act exactly
like generic buffers, except that they are read-only. The buffer freeing
code automatically determines the correct method of freeing the
underlying memory. Downstream elements that receive these kinds of
buffers do not need to do anything special to handle or unreference it.
Another way an element might get specialized buffers is to request them
from a downstream peer through a GstBufferPool or GstAllocator. Elements
can ask a GstBufferPool or GstAllocator from the downstream peer
element. If downstream is able to provide these objects, upstream can
use them to allocate buffers. See more in [Memory
allocation](plugin-development/advanced/allocation.md).
Many sink elements have accelerated methods for copying data to
hardware, or have direct access to hardware. It is common for these
elements to be able to create a GstBufferPool or GstAllocator for their
upstream peers. One such example is ximagesink. It creates buffers that
contain XImages. Thus, when an upstream peer copies data into the
buffer, it is copying directly into the XImage, enabling ximagesink to
draw the image directly to the screen instead of having to copy data
into an XImage first.
Filter elements often have the opportunity to either work on a buffer
in-place, or work while copying from a source buffer to a destination
buffer. It is optimal to implement both algorithms, since the GStreamer
framework can choose the fastest algorithm as appropriate. Naturally,
this only makes sense for strict filters -- elements that have exactly
the same format on source and sink pads.
## Media types and Properties
GStreamer uses a type system to ensure that the data passed between
elements is in a recognized format. The type system is also important
for ensuring that the parameters required to fully specify a format
match up correctly when linking pads between elements. Each link that is
made between elements has a specified type and optionally a set of
properties. See more about caps negotiation in [Caps
negotiation](plugin-development/advanced/negotiation.md).
### The Basic Types
GStreamer already supports many basic media types. Following is a table
of a few of the basic types used for buffers in GStreamer. The table
contains the name ("media type") and a description of the type, the
properties associated with the type, and the meaning of each property. A
full list of supported types is included in [List of Defined
Types](plugin-development/advanced/media-types.md#list-of-defined-types).
<table>
<caption>Table of Example Types</caption>
<thead>
<tr class="header">
<th>Media Type</th>
<th>Description</th>
<th>Property</th>
<th>Property Type</th>
<th>Property Values</th>
<th>Property Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>audio/*</td>
<td><em>All audio types</em></td>
<td>rate</td>
<td>integer</td>
<td>greater than 0</td>
<td>The sample rate of the data, in samples (per channel) per second.</td>
</tr>
<tr class="even">
<td></td>
<td></td>
<td>channels</td>
<td>integer</td>
<td>greater than 0</td>
<td>The number of channels of audio data.</td>
</tr>
<tr class="odd">
<td>audio/x-raw</td>
<td>Unstructured and uncompressed raw integer audio data.</td>
<td>format</td>
<td>string</td>
<td>S8 U8 S16LE S16BE U16LE U16BE S24_32LE S24_32BE U24_32LE U24_32BE S32LE S32BE U32LE U32BE S24LE S24BE U24LE U24BE S20LE S20BE U20LE U20BE S18LE S18BE U18LE U18BE F32LE F32BE F64LE F64BE</td>
<td>The format of the sample data.</td>
</tr>
<tr class="even">
<td>audio/mpeg</td>
<td>Audio data compressed using the MPEG audio encoding scheme.</td>
<td>mpegversion</td>
<td>integer</td>
<td>1, 2 or 4</td>
<td>The MPEG-version used for encoding the data. The value 1 refers to MPEG-1, -2 and -2.5 layer 1, 2 or 3. The values 2 and 4 refer to the MPEG-AAC audio encoding schemes.</td>
</tr>
<tr class="odd">
<td></td>
<td></td>
<td>framed</td>
<td>boolean</td>
<td>0 or 1</td>
<td>A true value indicates that each buffer contains exactly one frame. A false value indicates that frames and buffers do not necessarily match up.</td>
</tr>
<tr class="even">
<td></td>
<td></td>
<td>layer</td>
<td>integer</td>
<td>1, 2, or 3</td>
<td>The compression scheme layer used to compress the data <em>(only if mpegversion=1)</em>.</td>
</tr>
<tr class="odd">
<td></td>
<td></td>
<td>bitrate</td>
<td>integer</td>
<td>greater than 0</td>
<td>The bitrate, in bits per second. For VBR (variable bitrate) MPEG data, this is the average bitrate.</td>
</tr>
<tr class="even">
<td>audio/x-vorbis</td>
<td>Vorbis audio data</td>
<td></td>
<td></td>
<td></td>
<td>There are currently no specific properties defined for this type.</td>
</tr>
</tbody>
</table>
subprojects/gst-docs/markdown/plugin-development/introduction/index.md
+20
View File
@@ -0,0 +1,20 @@
---
title: Introduction
...
# Introduction
GStreamer is an extremely powerful and versatile framework for creating
streaming media applications. Many of the virtues of the GStreamer
framework come from its modularity: GStreamer can seamlessly incorporate
new plugin modules. But because modularity and power often come at a
cost of greater complexity (consider, for example,
[CORBA](http://www.omg.org/)), writing new plugins is not always easy.
This guide is intended to help you understand the GStreamer framework
so you can develop new plugins to extend the existing
functionality. The guide addresses most issues by following the
development of an example plugin - an audio filter plugin - written in
C. However, the later parts of the guide also present some issues
involved in writing other types of plugins, and the end of the guide
describes some of the Python bindings for GStreamer.
subprojects/gst-docs/markdown/plugin-development/introduction/preface.md
+238
View File
@@ -0,0 +1,238 @@
---
title: Preface
...
# Preface
## What is GStreamer?
GStreamer is a framework for creating streaming media applications. The
fundamental design comes from the video pipeline at Oregon Graduate
Institute, as well as some ideas from DirectShow.
GStreamer's development framework makes it possible to write any type of
streaming multimedia application. The GStreamer framework is designed to
make it easy to write applications that handle audio or video or both.
It isn't restricted to audio and video, and can process any kind of data
flow. The pipeline design is made to have little overhead above what the
applied filters induce. This makes GStreamer a good framework for
designing even high-end audio applications which put high demands on
latency or performance.
One of the most obvious uses of GStreamer is using it to build a media
player. GStreamer already includes components for building a media
player that can support a very wide variety of formats, including MP3,
Ogg/Vorbis, MPEG-1/2, AVI, Quicktime, mod, and more. GStreamer, however,
is much more than just another media player. Its main advantages are
that the pluggable components can be mixed and matched into arbitrary
pipelines so that it's possible to write a full-fledged video or audio
editing application.
The framework is based on plugins that will provide the various codec
and other functionality. The plugins can be linked and arranged in a
pipeline. This pipeline defines the flow of the data.
The GStreamer core function is to provide a framework for plugins, data
flow, synchronization and media type handling/negotiation. It also
provides an API to write applications using the various plugins.
## Who Should Read This Guide?
This guide explains how to write new modules for GStreamer. The guide is
relevant to several groups of people:
- Anyone who wants to add support for new ways of processing data in
GStreamer. For example, a person in this group might want to create
a new data format converter, a new visualization tool, or a new
decoder or encoder.
- Anyone who wants to add support for new input and output devices.
For example, people in this group might want to add the ability to
write to a new video output system or read data from a digital
camera or special microphone.
- Anyone who wants to extend GStreamer in any way. You need to have an
understanding of how the plugin system works before you can
understand the constraints that the plugin system places on the rest
of the code. Also, you might be surprised after reading this at how
much can be done with plugins.
This guide is not relevant to you if you only want to use the existing
functionality of GStreamer, or if you just want to use an application
that uses GStreamer. If you are only interested in using existing
plugins to write a new application - and there are quite a lot of
plugins already - you might want to check the *GStreamer Application
Development Manual*. If you are just trying to get help with a GStreamer
application, then you should check with the user manual for that
particular application.
## Preliminary Reading
This guide assumes that you are somewhat familiar with the basic
workings of GStreamer. For a gentle introduction to programming concepts
in GStreamer, you may wish to read the *GStreamer Application
Development Manual* first. Also check out the other documentation
available on the [GStreamer web
site](http://gstreamer.freedesktop.org/documentation/).
In order to understand this manual, you will need to have a basic
understanding of the C language. Since GStreamer adheres to the GObject
programming model, this guide also assumes that you understand the
basics of [GObject](http://developer.gnome.org/gobject/stable/pt01.html)
programming. You may also want to have a look at Eric Harlow's book
*Developing Linux Applications with GTK+ and GDK*.
## Structure of This Guide
To help you navigate through this guide, it is divided into several
large parts. Each part addresses a particular broad topic concerning
GStreamer plugin development. The parts of this guide are laid out in
the following order:
- [Building a Plugin][building] - Introduction to the
structure of a plugin, using an example audio filter for
illustration.
This part covers all the basic steps you generally need to perform
to build a plugin, such as registering the element with GStreamer
and setting up the basics so it can receive data from and send data
to neighbour elements. The discussion begins by giving examples of
generating the basic structures and registering an element in
[Constructing the Boilerplate][boilerplate]. Then,
you will learn how to write the code to get a basic filter plugin
working in [Specifying the pads][pads], [The chain function][chainfunc]
and [What are states?][states].
After that, we will show some of the GObject concepts on how to make
an element configurable for applications and how to do
application-element interaction in [Adding
Properties][properties] and [Signals][signals]. Next, you will learn to
build a quick test application to test all that you've just learned
in [Building a Test Application][testapp]. We
will just touch upon basics here. For full-blown application
development, you should look at [the Application Development
Manual](application-development/index.md).
- [Advanced Filter Concepts][advanced] - Information on
advanced features of GStreamer plugin development.
After learning about the basic steps, you should be able to create a
functional audio or video filter plugin with some nice features.
However, GStreamer offers more for plugin writers. This part of the
guide includes chapters on more advanced topics, such as scheduling,
media type definitions in GStreamer, clocks, interfaces and tagging.
Since these features are purpose-specific, you can read them in any
order, most of them don't require knowledge from other sections.
The first chapter, named [Different scheduling
modes][scheduling], will explain some of the basics of
element scheduling. It is not very in-depth, but is mostly some sort
of an introduction on why other things work as they do. Read this
chapter if you're interested in GStreamer internals. Next, we will
apply this knowledge and discuss another type of data transmission
than what you learned in [The chain function][chainfunc]: [Different
scheduling modes][scheduling]. Loop-based elements will give you
more control over input rate. This is useful when writing, for
example, muxers or demuxers.
Next, we will discuss media identification in GStreamer in [Media Types
and Properties][media-types]. You will learn how to
define new media types and get to know a list of standard media
types defined in GStreamer.
In the next chapter, you will learn the concept of request- and
sometimes-pads, which are pads that are created dynamically, either
because the application asked for it (request) or because the media
stream requires it (sometimes). This will be in [Request and
Sometimes pads][request-pads].
The next chapter, [Clocking][clocks], will
explain the concept of clocks in GStreamer. You need this
information when you want to know how elements should achieve
audio/video synchronization.
The next few chapters will discuss advanced ways of doing
application-element interaction. Previously, we learned on the
GObject-ways of doing this in [Adding Properties][properties] and
[Signals][signals]. We will discuss dynamic
parameters, which are a way of defining element behaviour over time
in advance, in [Supporting Dynamic Parameters][dynamic-params].
Next, you will learn about interfaces in [Interfaces][interfaces].
Interfaces are very target- specific ways of application-element
interaction, based on GObject's GInterface. Lastly, you will learn about
how metadata is handled in GStreamer in [Tagging (Metadata and
Streaminfo)][tagging].
The last chapter, [Events: Seeking, Navigation and More][events], will
discuss the concept of events in GStreamer. Events are another way of
doing application-element interaction. They take care of seeking, for
example. They are also yet another way in which elements
interact with each other, such as letting each other know about
media stream discontinuities, forwarding tags inside a pipeline and
so on.
- [Creating special element types][element-types] - Explanation of
writing other plugin types.
Because the first two parts of the guide use an audio filter as an
example, the concepts introduced apply to filter plugins. But many
of the concepts apply equally to other plugin types, including
sources, sinks, and autopluggers. This part of the guide presents
the issues that arise when working on these more specialized plugin
types. The chapter starts with a special focus on elements that can
be written using a base-class ([Pre-made base classes][base-classes]),
and later also goes into writing special types of elements in [Writing a
Demuxer or Parser][one-to-n], [Writing a N-to-1 Element or Muxer][n-to-one]
and [Writing a Manager][manager].
- [Appendices][appendix] - Further information for plugin developers.
The appendices contain some information that stubbornly refuses to
fit cleanly in other sections of the guide. Most of this section is
not yet finished.
The remainder of this introductory part of the guide presents a short
overview of the basic concepts involved in GStreamer plugin development.
Topics covered include [Elements and Plugins][intro-elements],
[Pads][intro-pads], [GstMiniObject, Buffers and Events][intro-miniobjects]
and [Media types and Properties][intro-mediatypes]. If you are already
familiar with this information, you can use this short overview to
refresh your memory, or you can skip to [Building a Plugin][building].
As you can see, there's a lot to learn, so let's get started\!
- Creating compound and complex elements by extending from a GstBin.
This will allow you to create plugins that have other plugins
embedded in them.
- Adding new media types to the registry along with typedetect
functions. This will allow your plugin to operate on a completely
new media type.
[building]: plugin-development/basics/index.md
[boilerplate]: plugin-development/basics/boiler.md
[pads]: plugin-development/basics/pads.md
[chainfunc]: plugin-development/basics/chainfn.md
[states]: plugin-development/basics/states.md
[properties]: plugin-development/basics/args.md
[signals]: plugin-development/basics/signals.md
[testapp]: plugin-development/basics/testapp.md
[advanced]: plugin-development/advanced/index.md
[scheduling]: plugin-development/advanced/scheduling.md
[media-types]: plugin-development/advanced/media-types.md
[request-pads]: plugin-development/advanced/request.md
[clocks]: plugin-development/advanced/clock.md
[dynamic-params]: plugin-development/advanced/dparams.md
[interfaces]: plugin-development/advanced/interfaces.md
[tagging]: plugin-development/advanced/tagging.md
[events]: plugin-development/advanced/events.md
[element-types]: plugin-development/element-types/index.md
[base-classes]: plugin-development/element-types/base-classes.md
[one-to-n]: plugin-development/element-types/one-to-n.md
[n-to-one]: plugin-development/element-types/n-to-one.md
[manager]: plugin-development/element-types/manager.md
[appendix]: plugin-development/appendix/index.md
[intro-elements]: plugin-development/introduction/basics.md#elements-and-plugins
[intro-pads]: plugin-development/introduction/basics.md#pads
[intro-miniobjects]: plugin-development/introduction/basics.md#gstminiobject-buffers-and-events
[intro-mediatypes]: plugin-development/introduction/basics.md#media-types-and-properties