当前位置：首页 > news >正文

【WebRTC-14】webrtc是如何创建视频的软/硬编码器？

news 来源：原创 2025/6/12 20:55:22

Android-RTC系列软重启，改变以往细读源代码的方式改为带上实际问题分析代码。增加实用性，方便形成肌肉记忆。同时不分种类、不分难易程度，在线征集问题切入点。

问题：webrtc是如何创建视频编码器选用软编码or硬编码的？如何扩展支持OpenH264，OpenH265？

其实这个问题，是上篇文章的一个补充。在上篇文章【WebRTC-13】我们确认创建编码器是如何创建的，找到了编码器的实体类，并大致的勾画出其调用链路，如下所示：

SdpOfferAnswerHandler:: ApplyLocalDescription / ApplyRemoteDescription（sdp信息）
SdpOfferAnswerHandler::UpdateTransceiversAndDataChannels -> UpdateTransceiverChannel（创建RtpTransceiver->Video/VoiceChannel）
SdpOfferAnswerHandler::UpdateSessionState
SdpOfferAnswerHandler::PushdownMediaDescription
BaseChannel::SetLocalContent(const MediaContentDescription* content, ..)
VoiceChannel/VideoChannel::SetLocalContent_w
BaseChannel::UpdateLocalStreams_w(const std::vector<StreamParams>& streams, ..)WebRtcVideoSendChannel::AddSendStream
WebRtcVideoSendChannel::WebRtcVideoSendStream::WebRtcVideoSendStream（Constructor）
WebRtcVideoSendChannel::WebRtcVideoSendStream::SetCodec|::RecreateWebRtcStream|::SetSenderParameters|::ReconfigureEncoder
Call::CreateVideoSendStream
VideoSendStreamImpl() -> VideoStreamEncoder(Interface)

我们可以在VideoStreamEncoder实体类中快速定位成员变量std::unique_ptr<VideoEncoder> encoder_ 代码创建的地方。

VideoStreamEncoderSettings   settings_;
VideoEncoderConfig     encoder_config_;void VideoStreamEncoder::ReconfigureEncoder() {std::unique_ptr<VideoEncoder> encoder_;encoder_ = MaybeCreateFrameDumpingEncoderWrapper(settings_.encoder_factory->CreateVideoEncoder(encoder_config_.video_format)，field_trials_);
}

随着调用链路的回溯，我们定位到WebRtcVideoSendChannel::WebRtcVideoSendStream的构造函数初始化了VideoSendStreamParameters，VideoStreamEncoder需要使用以下的三个输入参数，都是WebRtcVideoSendStream构造函数前在上层传入并创建好了。

webrtc::VideoSendStream::Config parameters_.config
webrtc::VideoEncoderConfig parameters_.encoder_config
VideoCodecSettings parameters_.codec_settings;

也就是在WebRtcVideoSendStream创建的地方，WebRtcVideoSendChannel::AddSendStream。

// cpp\media\engine\webrtc_video_engine.cc
bool WebRtcVideoSendChannel::AddSendStream(const StreamParams& sp) {webrtc::VideoSendStream::Config config(transport());config.encoder_settings.encoder_factory = encoder_factory_;config.encoder_settings.bitrate_allocator_factory = bitrate_allocator_factory_;...WebRtcVideoSendStream* stream = new WebRtcVideoSendStream(call_, sp, std::move(config), default_send_options_,video_config_.enable_cpu_adaptation, bitrate_config_.max_bitrate_bps,send_codec(), send_rtp_extensions_, send_params_);
}

【记住上面出现的transport() 以后会重点讲】显然encoder_factory也是从WebRtcVideoSendChannel外部持有者，webrtc_media_engine传进来的。经过了前面几篇文章的分析，这里直接跳过无聊的跟踪代码，直接给出encoder_factory的根持有者调用链——由PeerConnectionFactory持有的，实际对象位于Java层的DefaultVideoEncoderFactory。

|CreatePeerConnectionFactoryForJava
|---CreateModularPeerConnectionFactory( PeerConnectionFactoryDependencies )
|------ConnectionContext( dep )::EnableMeida = dep->media_factory->CreateMediaEngine
|		auto audio_engine = std::make_unique<WebRtcVoiceEngine>(
|			&env.task_queue_factory(), deps.adm.get(), env.field_trials(),
|			std::move(deps.audio_encoder_factory),
|			std::move(deps.audio_decoder_factory), std::move(deps.audio_mixer),
|			std::move(deps.audio_processing), std::move(deps.audio_frame_processor) );
|		auto video_engine = std::make_unique<WebRtcVideoEngine>(
|			std::move(deps.video_encoder_factory),
|			std::move(deps.video_decoder_factory), env.field_trials());
|---------CreateMediaEngine( deps ) with audio_engine / video_engine

现在我们找到问题答案的入口了，由于C++和Java之间存在一层JNI接口，所以我们先顺着调用接口，看看JNI层的接口对象类。

//文件位置 \sdk\android\src\jni\pc\peer_connection_factory.cc
ScopedJavaLocalRef<jobject> CreatePeerConnectionFactoryForJava(...) {PeerConnectionFactoryDependencies dependencies;...dependencies.adm = std::move(audio_device_module);dependencies.audio_encoder_factory = std::move(audio_encoder_factory);dependencies.audio_decoder_factory = std::move(audio_decoder_factory);dependencies.audio_processing = std::move(audio_processor);dependencies.video_encoder_factory =absl::WrapUnique(CreateVideoEncoderFactory(jni, jencoder_factory));dependencies.video_decoder_factory =absl::WrapUnique(CreateVideoDecoderFactory(jni, jdecoder_factory));EnableMedia(dependencies);rtc::scoped_refptr<PeerConnectionFactoryInterface> factory =CreateModularPeerConnectionFactory(std::move(dependencies));
}//文件位置 \sdk\android\src\jni\pc\video.cc
VideoEncoderFactory* CreateVideoEncoderFactory(JNIEnv* jni,const JavaRef<jobject>& j_encoder_factory) {return IsNull(jni, j_encoder_factory)? nullptr: new VideoEncoderFactoryWrapper(jni, j_encoder_factory);
}

明显这个VideoEncoderFactoryWrapper就是Jni层的接口类，并且看到了它是继承了C++的VideoEncoderFactory 纯虚类。

// 文件位置 sdk\android\src\jni\video_encoder_factory_wrapper.h .cc
// Wrapper for Java VideoEncoderFactory class. Delegates method calls through
// JNI and wraps the encoder inside VideoEncoderWrapper.
class VideoEncoderFactoryWrapper : public VideoEncoderFactory {public:VideoEncoderFactoryWrapper(JNIEnv* jni,const JavaRef<jobject>& encoder_factory);~VideoEncoderFactoryWrapper() override;std::unique_ptr<VideoEncoder> CreateVideoEncoder(const SdpVideoFormat& format) override;// Returns a list of supported codecs in order of preference.std::vector<SdpVideoFormat> GetSupportedFormats() const override;std::vector<SdpVideoFormat> GetImplementations() const override;std::unique_ptr<EncoderSelectorInterface> GetEncoderSelector() const override;private:const ScopedJavaGlobalRef<jobject> encoder_factory_;std::vector<SdpVideoFormat> supported_formats_;std::vector<SdpVideoFormat> implementations_;
};std::unique_ptr<VideoEncoder> VideoEncoderFactoryWrapper::CreateVideoEncoder(const SdpVideoFormat& format) {JNIEnv* jni = AttachCurrentThreadIfNeeded();ScopedJavaLocalRef<jobject> j_codec_info =SdpVideoFormatToVideoCodecInfo(jni, format);ScopedJavaLocalRef<jobject> encoder = Java_VideoEncoderFactory_createEncoder(jni, encoder_factory_, j_codec_info);if (!encoder.obj())return nullptr;return JavaToNativeVideoEncoder(jni, encoder);
}

我们也看到了VideoEncoderFactoryWrapper::CreateVideoEncoder的实现，就是调用Java层的DefaultVideoEncoderFactory.createEncoder。现在到Java层看看实现。

/** Helper class that combines HW and SW encoders. */
public class DefaultVideoEncoderFactory implements VideoEncoderFactory {private final VideoEncoderFactory hardwareVideoEncoderFactory;private final VideoEncoderFactory softwareVideoEncoderFactory = new SoftwareVideoEncoderFactory();/** Create encoder factory using default hardware encoder factory. */public DefaultVideoEncoderFactory(EglBase.Context eglContext, boolean enableIntelVp8Encoder, boolean enableH264HighProfile) {this.hardwareVideoEncoderFactory =new HardwareVideoEncoderFactory(eglContext, enableIntelVp8Encoder, enableH264HighProfile);}@Nullable@Overridepublic VideoEncoder createEncoder(VideoCodecInfo info) {final VideoEncoder softwareEncoder = softwareVideoEncoderFactory.createEncoder(info);final VideoEncoder hardwareEncoder = hardwareVideoEncoderFactory.createEncoder(info);if (hardwareEncoder != null && softwareEncoder != null) {// Both hardware and software supported, wrap it in a software fallbackreturn new VideoEncoderFallback(/* fallback= */ softwareEncoder, /* primary= */ hardwareEncoder);}return hardwareEncoder != null ? hardwareEncoder : softwareEncoder;}}// A combined video encoder that falls back on a secondary encoder if the primary encoder fails.
public class VideoEncoderFallback extends WrappedNativeVideoEncoder {private final VideoEncoder fallback;private final VideoEncoder primary;public VideoEncoderFallback(VideoEncoder fallback, VideoEncoder primary) {this.fallback = fallback;this.primary = primary;}@Overridepublic long createNativeVideoEncoder() {return nativeCreateEncoder(fallback, primary);}@Overridepublic boolean isHardwareEncoder() {return primary.isHardwareEncoder();}
}

Java层的DefaultVideoEncoderFactory是同时创建硬编码器和软编码器，并以Fallback备份的形式，打包成一个WrapperNativeVideoEncoder。上述流程各种类关系如下：

搞清楚各种对象类的上下文之后，我们就得继续深入探究软编码是怎么产生，要如何扩展。也就是上图加粗的SoftwareVideoEncoderFactory -> WrappedNativeVideoEncoder

节省篇幅，以下是 Java.SoftwareVideoEncoderFactory到Cxx.internal_encoder_factory具体实现的依赖链路。

|-org/webrtc/SoftwareVideoEncoderFactory.java
|--sdk/android/src/jni/software_video_encoder_factory.cc
|---api/video_codecs/builtin_video_encoder_factory.cc
|----media/engine/internal_encoder_factory.cc

接下来就是本次问题的答案了，在Cxx.internal_encoder_factory我们可以看到一个动态模板的Factory。这里的VideoEncoderFactoryTemplate是VideoEncoderFactory多模板实现类，

//  media/engine/internal_encoder_factory.cc
namespace {using Factory =VideoEncoderFactoryTemplate<webrtc::LibvpxVp8EncoderTemplateAdapter,
#if defined(WEBRTC_USE_H264)webrtc::OpenH264EncoderTemplateAdapter,
#endif
#if defined(RTC_USE_LIBAOM_AV1_ENCODER)webrtc::LibaomAv1EncoderTemplateAdapter,
#endifwebrtc::LibvpxVp9EncoderTemplateAdapter>;
}std::unique_ptr<VideoEncoder> InternalEncoderFactory::CreateVideoEncoder(const SdpVideoFormat& format) {auto original_format =FuzzyMatchSdpVideoFormat(Factory().GetSupportedFormats(), format);return original_format ? Factory().CreateVideoEncoder(*original_format): nullptr;
}
//  api/video_codecs/video_encoder_factory_template.h
//  api/video_codecs/video_encoder_factory_template_open_h264_adapter.h
//  api/video_codecs/video_encoder_factory_template_libvpx_vp8_adapter.h
//  api/video_codecs/video_encoder_factory_template_libaom_av1_adapter.htemplate <typename V, typename... Vs>std::unique_ptr<VideoEncoder> CreateVideoEncoderInternal(const SdpVideoFormat& format) {if (IsFormatInList(format, V::SupportedFormats())) {return V::CreateEncoder(format);}if constexpr (sizeof...(Vs) > 0) {return CreateVideoEncoderInternal<Vs...>(format);}return nullptr;}

仔细看创建编码器对象的方法实现，逻辑上很好理解，其实就是递归检查模板列表的编码器工厂类(适配器)，命中格式就直接调用模板类的方法实现；不命中则继续往后递归，直至最后一个候选。

比起用STL容器去实现适配器列表，用元模板的实现方式能从编译器层面上去优化实现的代码，提高性能。

而且到这里我们也找到了如何扩展软编码器的入口：仿照OpenH264EncoderTemplateAdapter实现模板静态结构体（VideoEncoderFactory）接口方法；仿照H264Encoder继承VideoEncoder实现功能方法。但其实这里只是找到了扩展软编解码器的入口，要真正实现扩展软编解码，还需要做大量的调查&分析工作，了解webrtc内部大量的调度算法才能得以实现。