This commit is contained in:
Akkariin Meiko
2022-03-12 03:16:09 +08:00
Unverified
parent 12b76e0c7a
commit 27c4ec74a1
10075 changed files with 5122287 additions and 1 deletions
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,64 @@
# iOS tutorial 5: A Complete media player
## Goal
![screenshot0]
![screenshot1]
This tutorial wants to be the “demo application” that showcases what can
be done with GStreamer on the iOS platform.
It is intended to be built and run, rather than analyzed for its
pedagogical value, since it adds very little GStreamer knowledge over
what has already been shown in [](tutorials/ios/a-basic-media-player.md).
It demonstrates the main functionality that a conventional media player
has, but it is not a complete application yet, therefore it has not been
uploaded to the AppStore.
## Introduction
The previous tutorial already implemented a basic media player. This one
simply adds a few finishing touches. In particular, it adds the
capability to choose the media to play, and disables the screensaver
during media playback.
These are not features directly related to GStreamer, and are therefore
outside the scope of these tutorials. Only a few implementation pointers
are given here.
## Selecting the media to play
A new `UIView` has been added, derived from `UITableViewController`
which shows a list of clips. When one is selected, the
`VideoViewController` from [](tutorials/ios/a-basic-media-player.md) appears
and its URI property is set to the URI of the selected clip.
The list of clips is populated from three sources: Media from the
devices Photo library, Media from the applications Documents folder
(accessible through iTunes file sharing) and a list of hardcoded
Internet addresses, selected to showcase different container and codec
formats, and a couple of bogus ones, to illustrate error reporting.
## Preventing the screen from turning off
While watching a movie, there is typically no user activity. After a
short period of such inactivity, iOS will dim the screen, and then turn
it off completely. To prevent this, the `idleTimerDisabled` property of
the `UIApplication` class is used. The application sets it to YES
(screen locking disabled) when the Play button is pressed, so the screen
is never turned off, and sets it back to NO when the Pause button is
pressed.
## Conclusion
This finishes the series of iOS tutorials. Each one of the preceding
tutorials has evolved on top of the previous one, showing how to
implement a particular set of features, and concluding in this Tutorial
5. The goal of Tutorial 5 is to build a complete media player which can
already be used to showcase the integration of GStreamer and iOS.
It has been a pleasure having you here, and see you soon!
[screenshot0]: images/tutorials/ios-a-complete-media-player-screenshot-0.png
[screenshot1]: images/tutorials/ios-a-complete-media-player-screenshot-1.png
@@ -0,0 +1,659 @@
# iOS tutorial 2: A running pipeline
## Goal
![screenshot]
As seen in the [Basic](tutorials/basic/index.md) and
[Playback](tutorials/playback/index.md) tutorials, GStreamer integrates
nicely with GLibs main loops, so pipeline operation and user interface
can be monitored simultaneously in a very simple way. However, platforms
like iOS or Android do not use GLib and therefore extra care must be
taken to keep track of the pipeline progress without blocking the user
interface (UI).
This tutorial shows:
- How to move the GStreamer-handling code to a separate Dispatch Queue
whereas UI managing still happens from the Main Dispatch Queue
- How to communicate between the Objective-C UI code and the C
GStreamer code
## Introduction
When using a Graphical User Interface (UI), if the application waits for
GStreamer calls to complete the user experience will suffer. The usual
approach, with the [GTK+ toolkit](http://www.gtk.org/) for example, is
to relinquish control to a GLib `GMainLoop` and let it control the
events coming from the UI or GStreamer.
Other graphical toolkits that are not based on GLib, like the [Cocoa
Touch](https://developer.apple.com/library/ios/documentation/General/Conceptual/DevPedia-CocoaCore/Cocoa.html)
framework used on iOS devices, cannot use this option, though. The
solution used in this tutorial uses a GLib `GMainLoop` for its
simplicity, but moves it to a separate thread (a [Dispatch
Queue](http://developer.apple.com/library/ios/#documentation/General/Conceptual/ConcurrencyProgrammingGuide/OperationQueues/OperationQueues.html)
different than the main one) so it does not block the user interface
operation.
Additionally, this tutorial shows a few places where caution has to be
taken when calling from Objective-C to C and vice versa.
The code below builds a pipeline with an `audiotestsrc` and
an `autoaudiosink` (it plays an audible tone). Two buttons in the UI
allow setting the pipeline to PLAYING or PAUSED. A Label in the UI shows
messages sent from the C code (for errors and state changes).
## The User Interface
A toolbar at the bottom of the screen contains a Play and a Pause
button. Over the toolbar there is a Label used to display messages from
GStreamer. This tutorial does not require more elements, but the
following lessons will build their User Interfaces on top of this one,
adding more components.
## The View Controller
The `ViewController` class manages the UI, instantiates
the `GStreamerBackend` and also performs some UI-related tasks on its
behalf:
**ViewController.m**
```
#import "ViewController.h"
#import "GStreamerBackend.h"
#import <UIKit/UIKit.h>
@interface ViewController () {
GStreamerBackend *gst_backend;
}
@end
@implementation ViewController
/*
* Methods from UIViewController
*/
- (void)viewDidLoad
{
[super viewDidLoad];
play_button.enabled = FALSE;
pause_button.enabled = FALSE;
gst_backend = [[GStreamerBackend alloc] init:self];
}
- (void)didReceiveMemoryWarning
{
[super didReceiveMemoryWarning];
// Dispose of any resources that can be recreated.
}
/* Called when the Play button is pressed */
-(IBAction) play:(id)sender
{
[gst_backend play];
}
/* Called when the Pause button is pressed */
-(IBAction) pause:(id)sender
{
[gst_backend pause];
}
/*
* Methods from GstreamerBackendDelegate
*/
-(void) gstreamerInitialized
{
dispatch_async(dispatch_get_main_queue(), ^{
play_button.enabled = TRUE;
pause_button.enabled = TRUE;
message_label.text = @"Ready";
});
}
-(void) gstreamerSetUIMessage:(NSString *)message
{
dispatch_async(dispatch_get_main_queue(), ^{
message_label.text = message;
});
}
@end
```
An instance of the `GStreamerBackend` in stored inside the class:
```
@interface ViewController () {
GStreamerBackend *gst_backend;
}
```
This instance is created in the `viewDidLoad` function through a custom
`init:` method in the `GStreamerBackend`:
```
- (void)viewDidLoad
{
[super viewDidLoad];
play_button.enabled = FALSE;
pause_button.enabled = FALSE;
gst_backend = [[GStreamerBackend alloc] init:self];
}
```
This custom method is required to pass the object that has to be used as
the UI delegate (in this case, ourselves, the `ViewController`).
The Play and Pause buttons are also disabled in the
`viewDidLoad` function, and they are not re-enabled until the
`GStreamerBackend` reports that it is initialized and ready.
```
/* Called when the Play button is pressed */
-(IBAction) play:(id)sender
{
[gst_backend play];
}
/* Called when the Pause button is pressed */
-(IBAction) pause:(id)sender
{
[gst_backend pause];
}
```
These two methods are called when the user presses the Play or Pause
buttons, and simply forward the call to the appropriate method in the
`GStreamerBackend`.
```
-(void) gstreamerInitialized
{
dispatch_async(dispatch_get_main_queue(), ^{
play_button.enabled = TRUE;
pause_button.enabled = TRUE;
message_label.text = @"Ready";
});
}
```
The `gstreamerInitialized` method is defined in the
`GStreamerBackendDelegate` protocol and indicates that the backend is
ready to accept commands. In this case, the Play and Pause buttons are
re-enabled and the Label text is set to “Ready”. This method is called
from a Dispatch Queue other than the Main one; therefore the need for
the
[dispatch_async()](https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/dispatch_async.3.html) call
wrapping all UI code.
```
-(void) gstreamerSetUIMessage:(NSString *)message
{
dispatch_async(dispatch_get_main_queue(), ^{
message_label.text = message;
});
}
```
The `gstreamerSetUIMessage:` method also belongs to the
`GStreamerBackendDelegate` protocol. It is called when the backend wants
to report some message to the user. In this case, the message is copied
onto the Label in the UI, again, from within a
[dispatch_async()](https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/dispatch_async.3.html) call.
## The GStreamer Backend
The `GStreamerBackend` class performs all GStreamer-related tasks and
offers a simplified interface to the application, which does not need to
deal with all the GStreamer details. When it needs to perform any UI
action, it does so through a delegate, which is expected to adhere to
the `GStreamerBackendDelegate` protocol:
**GStreamerBackend.m**
```
#import "GStreamerBackend.h"
#include <gst/gst.h>
GST_DEBUG_CATEGORY_STATIC (debug_category);
#define GST_CAT_DEFAULT debug_category
@interface GStreamerBackend()
-(void)setUIMessage:(gchar*) message;
-(void)app_function;
-(void)check_initialization_complete;
@end
@implementation GStreamerBackend {
id ui_delegate; /* Class that we use to interact with the user interface */
GstElement *pipeline; /* The running pipeline */
GMainContext *context; /* GLib context used to run the main loop */
GMainLoop *main_loop; /* GLib main loop */
gboolean initialized; /* To avoid informing the UI multiple times about the initialization */
}
/*
* Interface methods
*/
-(id) init:(id) uiDelegate
{
if (self = [super init])
{
self->ui_delegate = uiDelegate;
GST_DEBUG_CATEGORY_INIT (debug_category, "tutorial-2", 0, "iOS tutorial 2");
gst_debug_set_threshold_for_name("tutorial-2", GST_LEVEL_DEBUG);
/* Start the bus monitoring task */
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
[self app_function];
});
}
return self;
}
-(void) dealloc
{
if (pipeline) {
GST_DEBUG("Setting the pipeline to NULL");
gst_element_set_state(pipeline, GST_STATE_NULL);
gst_object_unref(pipeline);
pipeline = NULL;
}
}
-(void) play
{
if(gst_element_set_state(pipeline, GST_STATE_PLAYING) == GST_STATE_CHANGE_FAILURE) {
[self setUIMessage:"Failed to set pipeline to playing"];
}
}
-(void) pause
{
if(gst_element_set_state(pipeline, GST_STATE_PAUSED) == GST_STATE_CHANGE_FAILURE) {
[self setUIMessage:"Failed to set pipeline to paused"];
}
}
/*
* Private methods
*/
/* Change the message on the UI through the UI delegate */
-(void)setUIMessage:(gchar*) message
{
NSString *string = [NSString stringWithUTF8String:message];
if(ui_delegate && [ui_delegate respondsToSelector:@selector(gstreamerSetUIMessage:)])
{
[ui_delegate gstreamerSetUIMessage:string];
}
}
/* Retrieve errors from the bus and show them on the UI */
static void error_cb (GstBus *bus, GstMessage *msg, GStreamerBackend *self)
{
GError *err;
gchar *debug_info;
gchar *message_string;
gst_message_parse_error (msg, &err, &debug_info);
message_string = g_strdup_printf ("Error received from element %s: %s", GST_OBJECT_NAME (msg->src), err->message);
g_clear_error (&err);
g_free (debug_info);
[self setUIMessage:message_string];
g_free (message_string);
gst_element_set_state (self->pipeline, GST_STATE_NULL);
}
/* Notify UI about pipeline state changes */
static void state_changed_cb (GstBus *bus, GstMessage *msg, GStreamerBackend *self)
{
GstState old_state, new_state, pending_state;
gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state);
/* Only pay attention to messages coming from the pipeline, not its children */
if (GST_MESSAGE_SRC (msg) == GST_OBJECT (self->pipeline)) {
gchar *message = g_strdup_printf("State changed to %s", gst_element_state_get_name(new_state));
[self setUIMessage:message];
g_free (message);
}
}
/* Check if all conditions are met to report GStreamer as initialized.
* These conditions will change depending on the application */
-(void) check_initialization_complete
{
if (!initialized && main_loop) {
GST_DEBUG ("Initialization complete, notifying application.");
if (ui_delegate && [ui_delegate respondsToSelector:@selector(gstreamerInitialized)])
{
[ui_delegate gstreamerInitialized];
}
initialized = TRUE;
}
}
/* Main method for the bus monitoring code */
-(void) app_function
{
GstBus *bus;
GSource *bus_source;
GError *error = NULL;
GST_DEBUG ("Creating pipeline");
/* Create our own GLib Main Context and make it the default one */
context = g_main_context_new ();
g_main_context_push_thread_default(context);
/* Build pipeline */
pipeline = gst_parse_launch("audiotestsrc ! audioconvert ! audioresample ! autoaudiosink", &error);
if (error) {
gchar *message = g_strdup_printf("Unable to build pipeline: %s", error->message);
g_clear_error (&error);
[self setUIMessage:message];
g_free (message);
return;
}
/* Instruct the bus to emit signals for each received message, and connect to the interesting signals */
bus = gst_element_get_bus (pipeline);
bus_source = gst_bus_create_watch (bus);
g_source_set_callback (bus_source, (GSourceFunc) gst_bus_async_signal_func, NULL, NULL);
g_source_attach (bus_source, context);
g_source_unref (bus_source);
g_signal_connect (G_OBJECT (bus), "message::error", (GCallback)error_cb, (__bridge void *)self);
g_signal_connect (G_OBJECT (bus), "message::state-changed", (GCallback)state_changed_cb, (__bridge void *)self);
gst_object_unref (bus);
/* Create a GLib Main Loop and set it to run */
GST_DEBUG ("Entering main loop...");
main_loop = g_main_loop_new (context, FALSE);
[self check_initialization_complete];
g_main_loop_run (main_loop);
GST_DEBUG ("Exited main loop");
g_main_loop_unref (main_loop);
main_loop = NULL;
/* Free resources */
g_main_context_pop_thread_default(context);
g_main_context_unref (context);
gst_element_set_state (pipeline, GST_STATE_NULL);
gst_object_unref (pipeline);
return;
}
@end
```
#### Interface methods:
```
-(id) init:(id) uiDelegate
{
if (self = [super init])
{
self->ui_delegate = uiDelegate;
GST_DEBUG_CATEGORY_INIT (debug_category, "tutorial-2", 0, "iOS tutorial 2");
gst_debug_set_threshold_for_name("tutorial-2", GST_LEVEL_DEBUG);
/* Start the bus monitoring task */
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
[self app_function];
});
}
return self;
}
```
The `init:` method creates the instance by calling `[super init]`,
stores the delegate object that will handle the UI interaction and
launches the `app_function`, from a separate, concurrent, Dispatch
Queue. The `app_function` monitors the GStreamer bus for messages and
warns the application when interesting things happen.
`init:` also registers a new GStreamer debug category and sets its
threshold, so we can see the debug output from within Xcode and keep
track of our application progress.
```
-(void) dealloc
{
if (pipeline) {
GST_DEBUG("Setting the pipeline to NULL");
gst_element_set_state(pipeline, GST_STATE_NULL);
gst_object_unref(pipeline);
pipeline = NULL;
}
}
```
The `dealloc` method takes care of bringing the pipeline to the NULL
state and releasing it.
```
-(void) play
{
if(gst_element_set_state(pipeline, GST_STATE_PLAYING) == GST_STATE_CHANGE_FAILURE) {
[self setUIMessage:"Failed to set pipeline to playing"];
}
}
-(void) pause
{
if(gst_element_set_state(pipeline, GST_STATE_PAUSED) == GST_STATE_CHANGE_FAILURE) {
[self setUIMessage:"Failed to set pipeline to paused"];
}
}
```
The `play` and `pause` methods simply try to set the pipeline to the
desired state and warn the application if something fails.
#### Private methods:
```
/* Change the message on the UI through the UI delegate */
-(void)setUIMessage:(gchar*) message
{
NSString *string = [NSString stringWithUTF8String:message];
if(ui_delegate && [ui_delegate respondsToSelector:@selector(gstreamerSetUIMessage:)])
{
[ui_delegate gstreamerSetUIMessage:string];
}
}
```
`setUIMessage:` turns the C strings that GStreamer uses (UTF8 `char *`)
into `NSString *` and displays them through the
`gstreamerSetUIMessage` method of the `GStreamerBackendDelegate`. The
implementation of this method is marked as `@optional`, and hence the
check for its existence in the delegate with `respondsToSelector:`
```
/* Retrieve errors from the bus and show them on the UI */
static void error_cb (GstBus *bus, GstMessage *msg, GStreamerBackend *self)
{
GError *err;
gchar *debug_info;
gchar *message_string;
gst_message_parse_error (msg, &err, &debug_info);
message_string = g_strdup_printf ("Error received from element %s: %s", GST_OBJECT_NAME (msg->src), err->message);
g_clear_error (&err);
g_free (debug_info);
[self setUIMessage:message_string];
g_free (message_string);
gst_element_set_state (self->pipeline, GST_STATE_NULL);
}
/* Notify UI about pipeline state changes */
static void state_changed_cb (GstBus *bus, GstMessage *msg, GStreamerBackend *self)
{
GstState old_state, new_state, pending_state;
gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state);
/* Only pay attention to messages coming from the pipeline, not its children */
if (GST_MESSAGE_SRC (msg) == GST_OBJECT (self->pipeline)) {
gchar *message = g_strdup_printf("State changed to %s", gst_element_state_get_name(new_state));
[self setUIMessage:message];
g_free (message);
}
}
```
The `error_cb()` and `state_changed_cb()` are callbacks registered to
the `error` and `state-changed` events in GStreamer, and their goal is
to inform the user about these events. These callbacks have been widely
used in the [Basic tutorials](tutorials/basic/index.md) and their
implementation is very similar, except for two points:
Firstly, the messages are conveyed to the user through the
`setUIMessage:` private method discussed above.
Secondly, they require an instance of a `GStreamerBackend` object in
order to call its instance method `setUIMessage:`, which is passed
through the `userdata` pointer of the callbacks (the `self` pointer in
these implementations). This is discussed below when registering the
callbacks in the `app_function`.
```
/* Check if all conditions are met to report GStreamer as initialized.
* These conditions will change depending on the application */
-(void) check_initialization_complete
{
if (!initialized && main_loop) {
GST_DEBUG ("Initialization complete, notifying application.");
if (ui_delegate && [ui_delegate respondsToSelector:@selector(gstreamerInitialized)])
{
[ui_delegate gstreamerInitialized];
}
initialized = TRUE;
}
}
```
`check_initialization_complete()` verifies that all conditions are met
to consider the backend ready to accept commands and tell the
application if so. In this simple tutorial the only conditions are that
the main loop exists and that we have not already told the application
about this fact. Later (more complex) tutorials include additional
conditions.
Finally, most of the GStreamer work is performed in the app_function.
It exists with almost identical content in the Android tutorial, which
exemplifies how the same code can run on both platforms with little
change.
```
/* Create our own GLib Main Context and make it the default one */
context = g_main_context_new ();
g_main_context_push_thread_default(context);
```
It first creates a GLib context so all `GSource`s are kept in the same
place. This also helps cleaning after GSources created by other
libraries which might not have been properly disposed of. A new context
is created with `g_main_context_new()` and then it is made the default
one for the thread with `g_main_context_push_thread_default()`.
```
/* Build pipeline */
pipeline = gst_parse_launch("audiotestsrc ! audioconvert ! audioresample ! autoaudiosink", &error);
if (error) {
gchar *message = g_strdup_printf("Unable to build pipeline: %s", error->message);
g_clear_error (&error);
[self setUIMessage:message];
g_free (message);
return;
}
```
It then creates a pipeline the easy way, with `gst_parse_launch()`. In
this case, it is simply an `audiotestsrc` (which produces a continuous
tone) and an `autoaudiosink`, with accompanying adapter
elements.
```
/* Instruct the bus to emit signals for each received message, and connect to the interesting signals */
bus = gst_element_get_bus (pipeline);
bus_source = gst_bus_create_watch (bus);
g_source_set_callback (bus_source, (GSourceFunc) gst_bus_async_signal_func, NULL, NULL);
g_source_attach (bus_source, context);
g_source_unref (bus_source);
g_signal_connect (G_OBJECT (bus), "message::error", (GCallback)error_cb, (__bridge void *)self);
g_signal_connect (G_OBJECT (bus), "message::state-changed", (GCallback)state_changed_cb, (__bridge void *)self);
gst_object_unref (bus);
```
These lines create a bus signal watch and connect to some interesting
signals, just like we have been doing in the [Basic
tutorials](tutorials/basic/index.md). The creation of the watch is done
step by step instead of using `gst_bus_add_signal_watch()` to exemplify
how to use a custom GLib context. The interesting bit here is the usage
of a
[__bridge](http://clang.llvm.org/docs/AutomaticReferenceCounting.html#bridged-casts)
cast to convert an Objective-C object into a plain C pointer. In this
case we do not worry much about transferal of ownership of the object,
because it travels through C-land untouched. It re-emerges at the
different callbacks through the userdata pointer and cast again to a
`GStreamerBackend *`.
```
/* Create a GLib Main Loop and set it to run */
GST_DEBUG ("Entering main loop...");
main_loop = g_main_loop_new (context, FALSE);
[self check_initialization_complete];
g_main_loop_run (main_loop);
GST_DEBUG ("Exited main loop");
g_main_loop_unref (main_loop);
main_loop = NULL;
```
Finally, the main loop is created and set to run. Before entering the
main loop, though, `check_initialization_complete()` is called. Upon
exit, the main loop is disposed of.
And this is it! This has been a rather long tutorial, but we covered a
lot of territory. Building on top of this one, the following ones are
shorter and focus only on the new topics.
## Conclusion
This tutorial has shown:
- How to handle GStreamer code from a separate thread using a
[Dispatch
Queue](http://developer.apple.com/library/ios/#documentation/General/Conceptual/ConcurrencyProgrammingGuide/OperationQueues/OperationQueues.html) other
than the Main one.
- How to pass objects between the Objective-C UI code and the C
GStreamer code.
Most of the methods introduced in this tutorial,
like `check_initialization_complete()`and `app_function()`, and the
interface methods `init:`, `play:`, `pause:`,
`gstreamerInitialized:` and `setUIMessage:` will continue to be used in
the following tutorials with minimal modifications, so better get used
to them!
It has been a pleasure having you here, and see you soon!
[screenshot]: images/tutorials/ios-a-running-pipeline-screenshot.png
@@ -0,0 +1,32 @@
# iOS tutorials
## Welcome to the GStreamer iOS tutorials
These tutorials describe iOS-specific topics. General GStreamer
concepts will not be explained in these tutorials, so the
[](tutorials/basic/index.md) should be reviewed first. The reader should
also be familiar with basic iOS programming techniques.
The iOS tutorials have the same structure as the
[](tutorials/android/index.md): Each one builds on top of the previous
one and adds progressively more functionality, until a working media
player application is obtained in
[](tutorials/ios/a-complete-media-player.md).
Make sure to have read the instructions in
[](installing/for-ios-development.md) before jumping into the iOS
tutorials.
All iOS tutorials are split into the following classes:
- The `GStreamerBackend` class performs all GStreamer-related tasks
and offers a simplified interface to the application, which does not
need to deal with all the GStreamer details. When it needs to
perform any UI action, it does so through a delegate, which is
expected to adhere to the `GStreamerBackendDelegate` protocol.
- The `ViewController` class manages the UI, instantiates the
`GStreamerBackend` and also performs some UI-related tasks on its
behalf.
- The `GStreamerBackendDelegate` protocol defines which methods a
class can implement in order to serve as a UI delegate for the
`GStreamerBackend`.
@@ -0,0 +1,135 @@
# iOS tutorial 1: Link against GStreamer
## Goal
![screenshot]
The first iOS tutorial is simple. The objective is to get the GStreamer
version and display it on screen. It exemplifies how to link against the
GStreamer library from Xcode using objective-C.
## Hello GStreamer!
The tutorials code are in the
[`tutorials/xcode iOS` folder](https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-docs/examples/tutorials/xcode%20iOS/).
It was created using the GStreamer Single View
Application template. The view contains only a `UILabel` that will be
used to display the GStreamer's version to the user.
## The User Interface
The UI uses storyboards and contains a single `View` with a centered
`UILabel`. The `ViewController` for the `View` links its
`label` variable to this `UILabel` as an `IBOutlet`.
**ViewController.h**
```
#import <UIKit/UIKit.h>
@interface ViewController : UIViewController {
IBOutlet UILabel *label;
}
@property (retain,nonatomic) UILabel *label;
@end
```
## The GStreamer backend
All GStreamer-handling code is kept in a single Objective-C class called
`GStreamerBackend`. In successive tutorials it will get expanded, but,
for now, it only contains a method to retrieve the GStreamer version.
The `GStreamerBackend` is made in Objective-C so it can take care of the
few C-to-Objective-C conversions that might be necessary (like `char
*` to `NSString *`, for example). This eases the usage of this class by
the UI code, which is typically made in pure Objective-C.
`GStreamerBackend` serves exactly the same purpose as the JNI code in
the [](tutorials/android/index.md).
**GStreamerBackend.m**
```
#import "GStreamerBackend.h"
#include <gst/gst.h>
@implementation GStreamerBackend
-(NSString*) getGStreamerVersion
{
char *version_utf8 = gst_version_string();
NSString *version_string = [NSString stringWithUTF8String:version_utf8];
g_free(version_utf8);
return version_string;
}
@end
```
The `getGStreamerVersion()` method simply calls
`gst_version_string()` to obtain a string describing this version of
GStreamer. This [Modified
UTF8](http://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8) string is then
converted to a `NSString *` by ` NSString:stringWithUTF8String `and
returned. Objective-C will take care of freeing the memory used by the
new `NSString *`, but we need to free the `char *` returned
by `gst_version_string()`.
## The View Controller
The view controller instantiates the GStremerBackend and asks it for the
GStreamer version to display at the label. That's it!
**ViewController.m**
```
#import "ViewController.h"
#import "GStreamerBackend.h"
@interface ViewController () {
GStreamerBackend *gst_backend;
}
@end
@implementation ViewController
@synthesize label;
- (void)viewDidLoad
{
[super viewDidLoad];
// Do any additional setup after loading the view, typically from a nib.
gst_backend = [[GStreamerBackend alloc] init];
label.text = [NSString stringWithFormat:@"Welcome to %@!", [gst_backend getGStreamerVersion]];
}
- (void)didReceiveMemoryWarning
{
[super didReceiveMemoryWarning];
// Dispose of any resources that can be recreated.
}
@end
```
## Conclusion
This ends the first iOS tutorial. It has shown that, due to the
compatibility of C and Objective-C, adding GStreamer support to an iOS
app is as easy as it is on a Desktop application. An extra Objective-C
wrapper has been added (the `GStreamerBackend` class) for clarity, but
calls to the GStreamer framework are valid from any part of the
application code.
The following tutorials detail the few places in which care has to be
taken when developing specifically for the iOS platform.
It has been a pleasure having you here, and see you soon!
[screenshot]: images/tutorials/ios-link-against-gstreamer-screenshot.png
@@ -0,0 +1,578 @@
# iOS tutorial 3: Video
## Goal
![screenshot]
Except for [](tutorials/basic/toolkit-integration.md),
which embedded a video window on a GTK application, all tutorials so far
relied on GStreamer video sinks to create a window to display their
contents. The video sink on iOS is not capable of creating its own
window, so a drawing surface always needs to be provided. This tutorial
shows:
- How to allocate a drawing surface on the Xcode Interface Builder and
pass it to GStreamer
## Introduction
Since iOS does not provide a windowing system, a GStreamer video sink
cannot create pop-up windows as it would do on a Desktop platform.
Fortunately, the `VideoOverlay` interface allows providing video sinks with
an already created window onto which they can draw, as we have seen
in [](tutorials/basic/toolkit-integration.md).
In this tutorial, a `UIView` widget (actually, a subclass of it) is
placed on the main storyboard. In the `viewDidLoad` method of the
`ViewController`, we pass a pointer to this `UIView `to the instance of
the `GStreamerBackend`, so it can tell the video sink where to draw.
## The User Interface
The storyboard from the previous tutorial is expanded: A `UIView `is
added over the toolbar and pinned to all sides so it takes up all
available space (`video_container_view` outlet). Inside it, another
`UIView `is added (`video_view` outlet) which contains the actual video,
centered to its parent, and with a size that adapts to the media size
(through the `video_width_constraint` and `video_height_constraint`
outlets):
**ViewController.h**
```
#import <UIKit/UIKit.h>
#import "GStreamerBackendDelegate.h"
@interface ViewController : UIViewController <GStreamerBackendDelegate> {
IBOutlet UILabel *message_label;
IBOutlet UIBarButtonItem *play_button;
IBOutlet UIBarButtonItem *pause_button;
IBOutlet UIView *video_view;
IBOutlet UIView *video_container_view;
IBOutlet NSLayoutConstraint *video_width_constraint;
IBOutlet NSLayoutConstraint *video_height_constraint;
}
-(IBAction) play:(id)sender;
-(IBAction) pause:(id)sender;
/* From GStreamerBackendDelegate */
-(void) gstreamerInitialized;
-(void) gstreamerSetUIMessage:(NSString *)message;
@end
```
## The View Controller
The `ViewController `class manages the UI, instantiates
the `GStreamerBackend` and also performs some UI-related tasks on its
behalf:
**ViewController.m**
```
#import "ViewController.h"
#import "GStreamerBackend.h"
#import <UIKit/UIKit.h>
@interface ViewController () {
GStreamerBackend *gst_backend;
int media_width;
int media_height;
}
@end
@implementation ViewController
/*
* Methods from UIViewController
*/
- (void)viewDidLoad
{
[super viewDidLoad];
play_button.enabled = FALSE;
pause_button.enabled = FALSE;
/* Make these constant for now, later tutorials will change them */
media_width = 320;
media_height = 240;
gst_backend = [[GStreamerBackend alloc] init:self videoView:video_view];
}
- (void)didReceiveMemoryWarning
{
[super didReceiveMemoryWarning];
// Dispose of any resources that can be recreated.
}
/* Called when the Play button is pressed */
-(IBAction) play:(id)sender
{
[gst_backend play];
}
/* Called when the Pause button is pressed */
-(IBAction) pause:(id)sender
{
[gst_backend pause];
}
- (void)viewDidLayoutSubviews
{
CGFloat view_width = video_container_view.bounds.size.width;
CGFloat view_height = video_container_view.bounds.size.height;
CGFloat correct_height = view_width * media_height / media_width;
CGFloat correct_width = view_height * media_width / media_height;
if (correct_height < view_height) {
video_height_constraint.constant = correct_height;
video_width_constraint.constant = view_width;
} else {
video_width_constraint.constant = correct_width;
video_height_constraint.constant = view_height;
}
}
/*
* Methods from GstreamerBackendDelegate
*/
-(void) gstreamerInitialized
{
dispatch_async(dispatch_get_main_queue(), ^{
play_button.enabled = TRUE;
pause_button.enabled = TRUE;
message_label.text = @"Ready";
});
}
-(void) gstreamerSetUIMessage:(NSString *)message
{
dispatch_async(dispatch_get_main_queue(), ^{
message_label.text = message;
});
}
@end
```
We expand the class to remember the width and height of the media we are
currently playing:
```
@interface ViewController () {
GStreamerBackend *gst_backend;
int media_width;
int media_height;
}
```
In later tutorials this data is retrieved from the GStreamer pipeline,
but in this tutorial, for simplicitys sake, the width and height of the
media is constant and initialized in `viewDidLoad`:
```
- (void)viewDidLoad
{
[super viewDidLoad];
play_button.enabled = FALSE;
pause_button.enabled = FALSE;
/* Make these constant for now, later tutorials will change them */
media_width = 320;
media_height = 240;
gst_backend = [[GStreamerBackend alloc] init:self videoView:video_view];
}
```
As shown below, the `GStreamerBackend` constructor has also been
expanded to accept another parameter: the `UIView *` where the video
sink should draw.
The rest of the `ViewController `code is the same as the previous
tutorial, except for the code that adapts the `video_view` size to the
media size, respecting its aspect ratio:
```
- (void)viewDidLayoutSubviews
{
CGFloat view_width = video_container_view.bounds.size.width;
CGFloat view_height = video_container_view.bounds.size.height;
CGFloat correct_height = view_width * media_height / media_width;
CGFloat correct_width = view_height * media_width / media_height;
if (correct_height < view_height) {
video_height_constraint.constant = correct_height;
video_width_constraint.constant = view_width;
} else {
video_width_constraint.constant = correct_width;
video_height_constraint.constant = view_height;
}
}
```
The `viewDidLayoutSubviews` method is called every time the main view
size has changed (for example, due to a device orientation change) and
the entire layout has been recalculated. At this point, we can access
the `bounds` property of the `video_container_view` to retrieve its new
size and change the `video_view` size accordingly.
The simple algorithm above maximizes either the width or the height of
the `video_view`, while changing the other axis so the aspect ratio of
the media is preserved. The goal is to provide the GStreamer video sink
with a surface of the correct proportions, so it does not need to add
black borders (*letterboxing*), which is a waste of processing power.
The final size is reported to the layout engine by changing the
`constant` field in the width and height `Constraints` of the
`video_view`. These constraints have been created in the storyboard and
are accessible to the `ViewController `through IBOutlets, as is usually
done with other widgets.
## The GStreamer Backend
The `GStreamerBackend` class performs all GStreamer-related tasks and
offers a simplified interface to the application, which does not need to
deal with all the GStreamer details. When it needs to perform any UI
action, it does so through a delegate, which is expected to adhere to
the `GStreamerBackendDelegate` protocol:
**GStreamerBackend.m**
```
#import "GStreamerBackend.h"
#include <gst/gst.h>
#include <gst/video/video.h>
GST_DEBUG_CATEGORY_STATIC (debug_category);
#define GST_CAT_DEFAULT debug_category
@interface GStreamerBackend()
-(void)setUIMessage:(gchar*) message;
-(void)app_function;
-(void)check_initialization_complete;
@end
@implementation GStreamerBackend {
id ui_delegate; /* Class that we use to interact with the user interface */
GstElement *pipeline; /* The running pipeline */
GstElement *video_sink;/* The video sink element which receives VideoOverlay commands */
GMainContext *context; /* GLib context used to run the main loop */
GMainLoop *main_loop; /* GLib main loop */
gboolean initialized; /* To avoid informing the UI multiple times about the initialization */
UIView *ui_video_view; /* UIView that holds the video */
}
/*
* Interface methods
*/
-(id) init:(id) uiDelegate videoView:(UIView *)video_view
{
if (self = [super init])
{
self->ui_delegate = uiDelegate;
self->ui_video_view = video_view;
GST_DEBUG_CATEGORY_INIT (debug_category, "tutorial-3", 0, "iOS tutorial 3");
gst_debug_set_threshold_for_name("tutorial-3", GST_LEVEL_DEBUG);
/* Start the bus monitoring task */
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
[self app_function];
});
}
return self;
}
-(void) dealloc
{
if (pipeline) {
GST_DEBUG("Setting the pipeline to NULL");
gst_element_set_state(pipeline, GST_STATE_NULL);
gst_object_unref(pipeline);
pipeline = NULL;
}
}
-(void) play
{
if(gst_element_set_state(pipeline, GST_STATE_PLAYING) == GST_STATE_CHANGE_FAILURE) {
[self setUIMessage:"Failed to set pipeline to playing"];
}
}
-(void) pause
{
if(gst_element_set_state(pipeline, GST_STATE_PAUSED) == GST_STATE_CHANGE_FAILURE) {
[self setUIMessage:"Failed to set pipeline to paused"];
}
}
/*
* Private methods
*/
/* Change the message on the UI through the UI delegate */
-(void)setUIMessage:(gchar*) message
{
NSString *string = [NSString stringWithUTF8String:message];
if(ui_delegate && [ui_delegate respondsToSelector:@selector(gstreamerSetUIMessage:)])
{
[ui_delegate gstreamerSetUIMessage:string];
}
}
/* Retrieve errors from the bus and show them on the UI */
static void error_cb (GstBus *bus, GstMessage *msg, GStreamerBackend *self)
{
GError *err;
gchar *debug_info;
gchar *message_string;
gst_message_parse_error (msg, &err, &debug_info);
message_string = g_strdup_printf ("Error received from element %s: %s", GST_OBJECT_NAME (msg->src), err->message);
g_clear_error (&err);
g_free (debug_info);
[self setUIMessage:message_string];
g_free (message_string);
gst_element_set_state (self->pipeline, GST_STATE_NULL);
}
/* Notify UI about pipeline state changes */
static void state_changed_cb (GstBus *bus, GstMessage *msg, GStreamerBackend *self)
{
GstState old_state, new_state, pending_state;
gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state);
/* Only pay attention to messages coming from the pipeline, not its children */
if (GST_MESSAGE_SRC (msg) == GST_OBJECT (self->pipeline)) {
gchar *message = g_strdup_printf("State changed to %s", gst_element_state_get_name(new_state));
[self setUIMessage:message];
g_free (message);
}
}
/* Check if all conditions are met to report GStreamer as initialized.
* These conditions will change depending on the application */
-(void) check_initialization_complete
{
if (!initialized && main_loop) {
GST_DEBUG ("Initialization complete, notifying application.");
if (ui_delegate && [ui_delegate respondsToSelector:@selector(gstreamerInitialized)])
{
[ui_delegate gstreamerInitialized];
}
initialized = TRUE;
}
}
/* Main method for the bus monitoring code */
-(void) app_function
{
GstBus *bus;
GSource *bus_source;
GError *error = NULL;
GST_DEBUG ("Creating pipeline");
/* Create our own GLib Main Context and make it the default one */
context = g_main_context_new ();
g_main_context_push_thread_default(context);
/* Build pipeline */
pipeline = gst_parse_launch("videotestsrc ! warptv ! videoconvert ! autovideosink", &error);
if (error) {
gchar *message = g_strdup_printf("Unable to build pipeline: %s", error->message);
g_clear_error (&error);
[self setUIMessage:message];
g_free (message);
return;
}
/* Set the pipeline to READY, so it can already accept a window handle */
gst_element_set_state(pipeline, GST_STATE_READY);
video_sink = gst_bin_get_by_interface(GST_BIN(pipeline), GST_TYPE_VIDEO_OVERLAY);
if (!video_sink) {
GST_ERROR ("Could not retrieve video sink");
return;
}
gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(video_sink), (guintptr) (id) ui_video_view);
/* Instruct the bus to emit signals for each received message, and connect to the interesting signals */
bus = gst_element_get_bus (pipeline);
bus_source = gst_bus_create_watch (bus);
g_source_set_callback (bus_source, (GSourceFunc) gst_bus_async_signal_func, NULL, NULL);
g_source_attach (bus_source, context);
g_source_unref (bus_source);
g_signal_connect (G_OBJECT (bus), "message::error", (GCallback)error_cb, (__bridge void *)self);
g_signal_connect (G_OBJECT (bus), "message::state-changed", (GCallback)state_changed_cb, (__bridge void *)self);
gst_object_unref (bus);
/* Create a GLib Main Loop and set it to run */
GST_DEBUG ("Entering main loop...");
main_loop = g_main_loop_new (context, FALSE);
[self check_initialization_complete];
g_main_loop_run (main_loop);
GST_DEBUG ("Exited main loop");
g_main_loop_unref (main_loop);
main_loop = NULL;
/* Free resources */
g_main_context_pop_thread_default(context);
g_main_context_unref (context);
gst_element_set_state (pipeline, GST_STATE_NULL);
gst_object_unref (pipeline);
return;
}
@end
```
The main differences with the previous tutorial are related to the
handling of the `VideoOverlay` interface:
```
@implementation GStreamerBackend {
id ui_delegate; /* Class that we use to interact with the user interface */
GstElement *pipeline; /* The running pipeline */
GstElement *video_sink;/* The video sink element which receives VideoOverlay commands */
GMainContext *context; /* GLib context used to run the main loop */
GMainLoop *main_loop; /* GLib main loop */
gboolean initialized; /* To avoid informing the UI multiple times about the initialization */
UIView *ui_video_view; /* UIView that holds the video */
}
```
The class is expanded to keep track of the video sink element in the
pipeline and the `UIView *` onto which rendering is to occur.
```
-(id) init:(id) uiDelegate videoView:(UIView *)video_view
{
if (self = [super init])
{
self->ui_delegate = uiDelegate;
self->ui_video_view = video_view;
GST_DEBUG_CATEGORY_INIT (debug_category, "tutorial-3", 0, "iOS tutorial 3");
gst_debug_set_threshold_for_name("tutorial-3", GST_LEVEL_DEBUG);
/* Start the bus monitoring task */
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
[self app_function];
});
}
return self;
}
```
The constructor accepts the `UIView *` as a new parameter, which, at
this point, is simply remembered in `ui_video_view`.
```
/* Build pipeline */
pipeline = gst_parse_launch("videotestsrc ! warptv ! videoconvert ! autovideosink", &error);
```
Then, in the `app_function`, the pipeline is constructed. This time we
build a video pipeline using a simple `videotestsrc` element with a
`warptv` to add some spice. The video sink is `autovideosink`, which
choses the appropriate sink for the platform (currently,
`glimagesink` is the only option for
iOS).
```
/* Set the pipeline to READY, so it can already accept a window handle */
gst_element_set_state(pipeline, GST_STATE_READY);
video_sink = gst_bin_get_by_interface(GST_BIN(pipeline), GST_TYPE_VIDEO_OVERLAY);
if (!video_sink) {
GST_ERROR ("Could not retrieve video sink");
return;
}
gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(video_sink), (guintptr) (id) ui_video_view);
```
Once the pipeline is built, we set it to READY. In this state, dataflow
has not started yet, but the caps of adjacent elements have been
verified to be compatible and their pads have been linked. Also, the
`autovideosink` has already instantiated the actual video sink so we can
ask for it immediately.
The `gst_bin_get_by_interface()` method will examine the whole pipeline
and return a pointer to an element which supports the requested
interface. We are asking for the `VideoOverlay` interface, explained in
[](tutorials/basic/toolkit-integration.md),
which controls how to perform rendering into foreign (non-GStreamer)
windows. The internal video sink instantiated by `autovideosink` is the
only element in this pipeline implementing it, so it will be returned.
Once we have the video sink, we inform it of the `UIView` to use for
rendering, through the `gst_video_overlay_set_window_handle()` method.
## EaglUIView
One last detail remains. In order for `glimagesink` to be able to draw
on the
[`UIView`](http://developer.apple.com/library/ios/#documentation/UIKit/Reference/UIView_Class/UIView/UIView.html),
the
[`Layer`](http://developer.apple.com/library/ios/#documentation/GraphicsImaging/Reference/CALayer_class/Introduction/Introduction.html#//apple_ref/occ/cl/CALayer) associated
with this view must be of the
[`CAEAGLLayer`](http://developer.apple.com/library/ios/#documentation/QuartzCore/Reference/CAEAGLLayer_Class/CAEGLLayer/CAEGLLayer.html#//apple_ref/occ/cl/CAEAGLLayer) class.
To this avail, we create the `EaglUIView` class, derived from
`UIView `and overriding the `layerClass` method:
**EaglUIView.m**
```
#import "EaglUIVIew.h"
#import <QuartzCore/QuartzCore.h>
@implementation EaglUIView
+ (Class) layerClass
{
return [CAEAGLLayer class];
}
@end
```
When creating storyboards, bear in mind that the `UIView `which should
contain the video must have `EaglUIView` as its custom class. This is
easy to setup from the Xcode interface builder. Take a look at the
tutorial storyboard to see how to achieve this.
And this is it, using GStreamer to output video onto an iOS application
is as simple as it seems.
## Conclusion
This tutorial has shown:
- How to display video on iOS using a `UIView `and
the `VideoOverlay` interface.
- How to report the media size to the iOS layout engine through
runtime manipulation of width and height constraints.
The following tutorial plays an actual clip and adds a few more controls
to this tutorial in order to build a simple media player.
It has been a pleasure having you here, and see you soon!
[screenshot]: images/tutorials/ios-video-screenshot.png