Apple juice example for GStreamer operation

Before explaining the pipeline mechanism in GStreamer, let take a real-life example first.

Let imaging the apple juice you drink regularly. But ever you try to know how was it made.

Just take a simple example of a factory chain to make an apple juice bottle. First, apple is the essential material we have from our lovely farmer, thanks to them, but we don't say about farmers here.

And then, the apple needs to peel off, right! Who can accept apple juice from an apple without peeling off? 😒 As we converted an apple to an apple without skin ( ahem, look better everyone looks better without skin, oh I mean clothes 😉)

Next, we need to take the apples go through the presser, this stage we press apples into the juice. Let say apple liquid, because this juice is not as complete yet as our eventual apple juice.

Everyone out there says about how bad is sugar, which sugar is bad for your health, sugar causes obesity, ..etc.. but many people don't want to just drink the pure apple juice then the factory put a bit (ah I think it's a lot 🤣), they know it's less healthy but it's more friendly to human's tasty and money.

Usually, before bottling the juice factory will make a label to help users clarify the detailed ingredients inside the apple juice, how to use the apple juice, and when is the expiration date. Then users will have their best way to use the juice, like just drinking, cooking, or even mix with some wine for example.

Eventually, we have everything ready to package and ship to the customer, so the last stage will be filling it in the bottle and tag them. Now enjoy your apple juice if you already bought any of them from the supper market.

Now turn back to GStreamer, like the factory chain to make apple juice. GStreamer works similarly. They have many different elements, each element has the dedicated responsibility to process the media content. When you put all of them in a chain (Pipeline) with a suitable order, you can do the function you want.

GStreamer with pipeline procedure

Consider our purpose is screen recording. Now we look for the suitable elements (or plugins) need to use to make out our apple juice.

Now we come to GStreamer's documentation, here we can find any elements supported by GStreamer and then look for the one we want. Like we are build up a factory, we need many components, here is where we can look for the component to build up a factory chain.

https://gstreamer.freedesktop.org/documentation/plugins_doc.html?gi-language=c

To make apple juice we need the apple, right? Likewise, to make a screen recording, we need the content of our screen. On the documentation of GStreamer, we look for "captures screen" first, we can find some plugins can use. After trying out all of them I decide to use "dxgiscreencapsrc" as it works the best, you can try out the other plugins as well.

Now I have here for you a sample pipeline to try first. On Windows, open the command line (or terminal on Linux and Mac) and try the below command, if you wish to try the other plugins, you can replace "dxgiscreencapsrc" with the other plugins as well.

gst-launch-1.0 dxgiscreencapsrc cursor=TRUE ! videoconvert ! autovideosink

Let say "dxgiscreencapsrc" already provide us the apple, now we need to peel it off to make it ready to produce, we use video convert, this element will help us convert the video content from the pre element to a suitable format for the post element.

We have the peeled off apple already, now we need to press them into the juice. In this case, we use openh264enc. as the video data after "videoconvert" is just the raw data, if we write it directly to the file, it would be a very big file and the other Media player will be not able to play it, so we need an encoder to encode (or compress) video data into a suitable encoding type, here we use openh264enc to encode it into h264 type (h264 type is the most popular when we want to export an mp4 file).

Like the pure apple juice right after pressing. we have h264 encoded video data. we need to make it more friendly before writing it down to an mp4 file. Then we need to parse it as we put some sugar into the juice. Now we need h264parse.

Actually, at this step, we already can write it down a file and open it by a player like "vlc", but it will be not able to open by other Media player like "Window media player". Because our data now still lack detailed information about it. Somehow vlc can decode it and play because it doesn't care about detail of data, but other media player they cannot read it when they don't know what inside it, like a vigilant person, he will never drink a thing without looking for the ingredients inside. And this step we need "mp4mux" which also can be used to mix the audio inside as well, but we will talk about it in another post.

Finally, package apple juice into a bottle, we need "filesink". This element will help us write the media data into a file.

OK 🙋‍♂️, now we have a perfect pipeline for trial.

Open the command line on Windows or terminal on Linux/Mac and try the above command, do anything with your desktop, and then terminate it to save the file embedcoder.mp4. Open that file by any media player you have and enjoy it.

Pipeline explanation

Option "-et" of gst-launch-1.0:

• -e: let gst-launch know to end the file when you terminate the pipeline by interrupt key (Ctrl + C), without this option, data in your file may be corrupted due to inappropriate ending procedure. Instead of using this option, you can set the duration time for "dxgiscreencapsr" by parameters like x, y, width, height.
• -t: gst-launch will output the tag as the metadata for the file. Recommend to use it but it's ok to run without this option.

dxgiscreencapsrc parameters:

• x, y, width, height: the start position and width height of the windows which to make a recording, calculated from the top-left corner of the screen.
• cursor: set if want to show the cursor in the recording video.

Pad:

• video/x-raw,framerate=30/1: Usually there are the pads between each of the links of the elements and they will be automatically defined when the pipeline running. In this case, we want to control the framerate of the output video, so we will set the pad manually like this. change the framerate as your desire and consider the performance of your computer.

Hope you enjoy all the instruction I wrote here and saw it's useful for your work or whatever 😋. Next post I will guide you to code this pipeline inside a C/C++ code and have more fun with GStreamer in Code. The last post will be the audio integration, I will guide you to go through the complete pipeline to make a screen recording that includes both audios from the PC mic and system audio as well. Stay tuned 😉.

Recap:

1. We know how the apple juice was produced, right. hahaha, at least the conception one.
2. We enhanced our imagination with the operation of the GStreamer pipeline and apple juice factory. I so admire me, LoL. 😂
   
3. We have a detailed and perfect pipeline to record screen activities in an mp4 video file.

Stay tuned and safe everybody, I will come back in the next post soon.