RK3568 基于Gstreamer的多媒体调试记录

1、环境介绍

硬件：飞凌ok3568-c开发板

软件：原厂rk356x sdk

2、概念理清

首先，rk3568等芯片内部是物理存在视频编解码硬件加速模块，包括视频解码器（如RKVDEC），视频编码器（RKVENC）等。有了具体的物理硬件，就一定会有对应的软件驱动。所以在kernel态，会存在这些硬件编解码器的设备驱动。为了屏蔽不同芯片平台和操作系统的差异，诞生了媒体处理软件平台，也就是所谓的MPP（Media Process Platform），MPP会实现对硬件编码器的功能开发。同时MPP向上为用户态应用程序开发者提供了统一的视频媒体处理接口，也就是MPI（Media Process Interface）。各种多媒体中间件如GStreamer、OpenMAX，以GStreamer为例，可以使用MPI接口编写GStreamer硬件编解码插件。也可以直接在应用程序中使用MPI接口实现多媒体处理。

而本文，我们将使用GStreamer来测试多媒体功能。Rockchip开发了一个GStreamer插件，叫作rockchipmpp，这样使用GStreamer工具时（如gst-play，gst-launch）显示指定编解码插件，就可以体验硬件编解码能力。

rochchipmpp插件包含众多元素，如解码相关的：mppvideodec和mppjpegdec。编码相关的：mpph264enc，mppvp8enc，mppjpegenc等。GStreamer在视频解码阶段会优先调⽤rockchipmpp插件。

3、提前准备

本例准备了一个7寸的1024*600分辨率的hdmi屏来配合视频播放测试。

4、GStreamer编译

以下以buildroot系统为例，menuconfig中需要开启相关配置，SDK其实已经默认开启。相关宏均以整理至/buildroot/configs/rockchip/multimedia/gst/，在目标config里直接包含即可。

5、GStreamer基础介绍

Gstreamer的插件主要分为三大类：源插件（Source）、过滤器/类过滤器插件（Filter）、接收插件（Sink）。 源插件只产生数据，但不接收数据，比如filesrc插件，用于读取文件，videotestsrc插件，用于生成指定的图像等。过滤器/类过滤器插件接收数据，对数据进行一些处理，再发送给后级，比如一些解封装器，编解码器等插件。接收插件只接收数据，但不产生数据，比如filesink，用于保存文件，waylandsink，用于渲染画面等。

6、视频播放初体验

直接使用gst-play命令播放mp4视频：

# 由Gstreamer自动查找解码器，由于mppvideodec优先级最高，只要是该插件支持的格式，最终就会选择该插件解码
root@rk3568-buildroot:/# gst-play-1.0 /oem/SampleVideo_1280x720_5mb.mp4
Press 'k' to see a list of keyboard shortcuts.
Now playing /oem/SampleVideo_1280x720_5mb.mp4
Redistribute latency...
Redistribute latency...
Redistribute latency...
Redistribute latency...
Redistribute latency...
0:00:29.5 / 0:00:29.5
Reached end of play list.

7、视频硬编码

7.1、h264

如下命令会生成一个h264编码的mp4文件：

root@rk3568-buildroot:/# gst-launch-1.0 mp4mux name=mux ! filesink location=enc_h264_test.mp4 videotestsrc num-buffers=600 ! video/x-raw,framerate=60/1,width=1920,height=1080,format=NV12 ! mpph264enc ! h264parse ! mux.video_0 -e
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
Pipeline is PREROLLED ...
Prerolled, waiting for async message to finish...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstSystemClock
Got EOS from element "pipeline0".
EOS received - stopping pipeline...
Execution ended after 0:00:17.661260775
Setting pipeline to NULL ...
Freeing pipeline ...

可以自行在windows下打开查看。

7.2、h265

root@rk3568-buildroot:/# gst-launch-1.0 mp4mux name=mux ! filesink location=enc_h265_test.mp4 videotestsrc num-buffers=600 ! video/x-raw,framerate=60/1,width=1920,height=1080,format=NV12 ! mpph265enc ! h265parse ! mux.video_0 -e
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
Pipeline is PREROLLED ...
Prerolled, waiting for async message to finish...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstSystemClock
Got EOS from element "pipeline0".
EOS received - stopping pipeline...
Execution ended after 0:00:17.624120229
Setting pipeline to NULL ...
Freeing pipeline ...

可以自行在windows下打开查看。

8、视频硬解码

8.1、解码视频并播放

# 由Gstreamer自动查找解码器，由于mppvideodec优先级最高，只要是该插件支持的格式，最终就会选择该插件解码
root@rk3568-buildroot:/# gst-play-1.0 /oem/SampleVideo_1280x720_5mb.mp4

# 手动构建pipeline
root@rk3568-buildroot:/# gst-launch-1.0 filesrc location=/oem/SampleVideo_1280x720_5mb.mp4 ! parsebin ! mppvideodec ! waylandsink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...
Prerolled, waiting for async message to finish...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstSystemClock
[13:38:44.139] seeing the first app
Got EOS from element "pipeline0".
Execution ended after 0:00:29.561555096
Setting pipeline to NULL ...
Freeing pipeline ...

解码视频并播放带音频

root@rk3568-buildroot:/# gst-launch-1.0 filesrc location=/oem/SampleVideo_1280x720_5mb.mp4 ! qtdemux name=demux demux.video_0 ! queue ! parsebin ! mppvideodec ! autovideosink demux.audio_0 ! queue ! aacparse ! faad ! alsasink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
Redistribute latency...
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...
Prerolled, waiting for async message to finish...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstAudioSinkClock
Got EOS from element "pipeline0".
Execution ended after 0:00:29.607928266
Setting pipeline to NULL ...
Freeing pipeline ...

插入耳机或者喇叭，就可以体验画音同步。