RTSP播放器实现回调RGB|YUV给视觉算法，然后二次编码推送到RTMP服务

引言

在本文中，我们将介绍如何基于大牛直播SDK构建一个功能强大的RTSP|RTMP播放器，该播放器利用自定义SDK解码视频、处理RGB帧，并将其推送到RTMP流中进行直播。这个解决方案非常适合需要在实时视频流中集成视觉算法的场景，在处理后将数据推送到RTMP服务器。我们将详细探讨播放器的架构、回调处理以及图像帧的操作过程。

核心组件概述

先看视频演示，左侧是Windows平台轻量级RTSP服务，采集毫秒计数器，然后编码打包，对外提供RTSP拉流的URL，右上角拉取原始的RTSP流，然后回调解码后的RGB|YUV数据，然后点击推送RTMP，实现RGB|YUV数据的二次编码和RTMP推送。可以看到，从原始数据播放回调，到右下角处理后的RTMP流，二次播放，整体延迟在毫秒级，非常低。

1. RTSP/RTMP播放器架构：

   • 该播放器接收RTSP流，将其解码为RGB帧，处理后将这些帧推送到RTMP流进行直播广播。

   • 它利用自定义SDK来处理视频解码、帧处理和流推送。

2. 关键概念：

   • RTSP（实时流协议）：该协议用于控制流媒体服务器，广泛应用于实时视频流的传输。

   • RTMP（实时消息协议）：RTMP用于将视频数据推送到直播服务器，确保低延迟广播。

   • RGB数据处理：播放器将视频帧解码为RGB格式（32位），然后传递给视觉算法进行处理，最后将处理后的数据推送到RTMP服务器。

代码讲解

进入系统后，先播放RTMP、或RTSP流，然后点RTMP推流，那么会模拟把播放器回调的RGB或YUV数据，投递到RTMP推送模块（右上方播放和转推）、右下方播放RTMP服务器二次处理后的RTMP流。

1. 初始化播放器SDK

CSmartPlayerDlg类负责初始化播放器、设置事件回调并准备视频渲染窗口。SDK初始化通过player_api_对象完成。

/* * SmartPlayDlg.cpp* Created by daniusdk.com* WeChat: xinsheng120*/
void CSmartPlayerDlg::OnBnClickedButtonPlay()
{if ( player_handle_ == NULL )return;CString btn_play_str;btn_play_.GetWindowTextW(btn_play_str);if ( btn_play_str == _T("播放") ){if ( !is_recording_ ){if ( !InitCommonSDKParam() ){AfxMessageBox(_T("设置参数错误!"));return;}}player_api_.SetVideoSizeCallBack(player_handle_, GetSafeHwnd(), SP_SDKVideoSizeHandle);bool is_support_d3d_render = false;NT_INT32 in_support_d3d_render = 0;if ( NT_ERC_OK == player_api_.IsSupportD3DRender(player_handle_,wrapper_render_wnd_.RenderWnd(), &in_support_d3d_render)){if ( 1 == in_support_d3d_render ){is_support_d3d_render = true;}}player_api_.SetVideoFrameCallBack(player_handle_, NT_SP_E_VIDEO_FRAME_FORMAT_RGB32,this, SM_SDKVideoFrameHandleV2);if ( is_support_d3d_render ){is_gdi_render_ = false;// 支持d3d绘制的话，就用D3D绘制player_api_.SetRenderWindow(player_handle_, wrapper_render_wnd_.RenderWnd());player_api_.SetRenderScaleMode(player_handle_, btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);}else{is_gdi_render_ = true;// 不支持D3D就让播放器吐出数据来，用GDI绘制wrapper_render_wnd_.SetRenderScaleMode(btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);player_api_.SetVideoFrameCallBack(player_handle_, NT_SP_E_VIDEO_FRAME_FORMAT_RGB32,GetSafeHwnd(), SM_SDKVideoFrameHandle);}if ( BST_CHECKED == btn_check_hardware_decoder_.GetCheck() ){player_api_.SetH264HardwareDecoder(player_handle_, is_support_h264_hardware_decoder_?1:0, 0);player_api_.SetH265HardwareDecoder(player_handle_, is_support_h265_hardware_decoder_?1:0, 0);}else{player_api_.SetH264HardwareDecoder(player_handle_, 0, 0);player_api_.SetH265HardwareDecoder(player_handle_, 0, 0);}player_api_.SetOnlyDecodeVideoKeyFrame(player_handle_, BST_CHECKED == btn_check_only_decode_video_key_frame_.GetCheck() ? 1 : 0);player_api_.SetLowLatencyMode(player_handle_, BST_CHECKED == btn_check_low_latency_.GetCheck() ? 1 : 0);player_api_.SetFlipVertical(player_handle_, BST_CHECKED == btn_check_flip_vertical_.GetCheck() ? 1 :0 );player_api_.SetFlipHorizontal(player_handle_, BST_CHECKED == btn_check_flip_horizontal_.GetCheck() ? 1 : 0);player_api_.SetRotation(player_handle_, rotate_degrees_);player_api_.SetAudioVolume(player_handle_, slider_audio_volume_.GetPos());player_api_.SetUserDataCallBack(player_handle_, GetSafeHwnd(), NT_SP_SDKUserDataHandle);if (NT_ERC_OK != player_api_.StartPlay(player_handle_)){AfxMessageBox(_T("播放器失败!"));return;}btn_play_.SetWindowTextW(_T("停止"));is_playing_ = true;}else{StopPlayback();}
}void CSmartPlayerDlg::StopPlayback()
{if ( player_handle_ == NULL )return;is_gdi_render_ = false;btn_full_screen_.EnableWindow(FALSE);wrapper_render_wnd_.ClearVideoSize();wrapper_render_wnd_.SetPlayerHandle(NULL);width_ = 0; height_ = 0;player_api_.StopPlay(player_handle_);wrapper_render_wnd_.CleanRender();btn_play_.SetWindowTextW(_T("播放"));is_playing_ = false;if (!is_recording_){SetWindowText(base_title_);edit_duration_.SetWindowText(_T(""));edit_playback_pos_.SetWindowText(_T(""));btn_pause_.SetWindowText(_T("暂停"));edit_player_msg_.SetWindowText(_T(""));}
}

上述代码片段初始化了SmartPlayer SDK，这是解码RTSP流、处理视频和音频播放所必需的。

2. 设置事件回调

播放器SDK提供了多个事件回调函数，这些回调函数会在特定事件发生时触发，比如接收到新的视频帧或遇到缓冲事件。这些回调函数在初始化时进行设置。

例如，SetVideoFrameCallBack函数用于定义在接收到新的视频帧时应该执行什么操作：

player_api_.SetVideoFrameCallBack(handle, NT_SP_E_VIDEO_FRAME_FORMAT_RGB32, this, &CSmartPlayerDlg::OnVideoFrameHandle);

OnVideoFrameHandle函数会处理每个RGB帧，然后将其推送到RTMP流。

3. 处理视频帧回调

在OnVideoFrameHandle函数中，我们通过首先检查帧的格式，然后将其数据复制到nt_rgb32_image结构体中来处理RGB帧：

void CSmartPlayerDlg::OnVideoFrameHandle(NT_HANDLE handle, NT_UINT32 status,const NT_SP_VideoFrame* frame)
{if (nullptr == frame)return;std::unique_lock<std::recursive_mutex> lock(push_handle_mutex_);if (!is_pushing_)return;if (GetPushHandle() == nullptr)return;//NT_UINT64 ts = frame->timestamp_;//std::ostringstream ss;//ss << "OnVideoFrameHandle, ts: " << ts << "\r\n";//OutputDebugStringA(ss.str().c_str());NT_PB_Image image;memset(&image, 0, sizeof(image));image.width_ = frame->width_;image.height_ = frame->height_;// timestamp_ 目前不使用//image.timestamp_ = frame->timestamp_;if (NT_SP_E_VIDEO_FRAME_FORMAT_RGB32 == frame->format_){image.format_ = NT_PB_E_IMAGE_FORMAT_RGB32;image.plane_[0] = frame->plane0_;image.stride_[0] = frame->stride0_;image.plane_size_[0] = frame->stride0_ * frame->height_;}else if (NT_SP_E_VIDEO_FRAME_FROMAT_I420 == frame->format_){image.format_ = NT_PB_E_IMAGE_FORMAT_I420;image.plane_[0] = frame->plane0_;image.stride_[0] = frame->stride0_;image.plane_size_[0] = frame->stride0_ * frame->height_;image.plane_[1] = frame->plane1_;image.stride_[1] = frame->stride1_;image.plane_size_[1] = frame->stride1_ * ((frame->height_ + 1) / 2);image.plane_[2] = frame->plane2_;image.stride_[2] = frame->stride2_;image.plane_size_[2] = frame->stride2_ * ((frame->height_ + 1) / 2);}else{return;}int index_ = 0;push_api_.PostLayerImage(push_handle_, 0, index_, &image, 0, NULL);
}

该函数将帧数据转换为图像对象，可以进行后续处理或传递给可视算法，通过PostLayerImage()接口投递到RTMP推送模块。

4. 推送到RTMP

一旦RGB帧处理完成，我们需要将视频数据推送到RTMP服务器。通过使用Smart Publisher SDK的推送功能，我们实现了这一点：

/* SmartPlayerDlg.cpp* Created by daniusdk.com* WeChat: xinsheng120*/
void CSmartPlayerDlg::OnBnClickedButtonPush()
{// TODO: Add your control notification handler code hereCString btn_push_str;btn_push_.GetWindowTextW(btn_push_str);if (btn_push_str == _T("推送RTMP")){StartPush("rtmp://192.168.1.7:1935/hls/stream666");}else{StopPush();}
}bool CSmartPlayerDlg::StartPush(const std::string& url)
{if (is_pushing_)return false;if (url.empty())return false;if (!OpenPushHandle())return false;auto push_handle = GetPushHandle();ASSERT(push_handle != nullptr);if (publisher_handle_count_ < 1){SetCommonOptionToPublisherSDK();}if (NT_ERC_OK != push_api_.SetURL(push_handle, url.c_str(), NULL)){if (0 == publisher_handle_count_){push_api_.Close(push_handle);SetPushHandle(nullptr);}return false;}if (NT_ERC_OK != push_api_.StartPublisher(push_handle, NULL)){if (0 == publisher_handle_count_){push_api_.Close(push_handle);SetPushHandle(nullptr);}return false;}publisher_handle_count_++;btn_push_.SetWindowTextW(_T("停止推送"));is_pushing_ = true;return true;
}void CSmartPlayerDlg::StopPush()
{if (!is_pushing_)return;is_pushing_ = false;std::unique_lock<std::recursive_mutex> lock(push_handle_mutex_);if (nullptr == push_handle_)return;publisher_handle_count_--;push_api_.StopPublisher(push_handle_);if (0 == publisher_handle_count_){push_api_.Close(push_handle_);push_handle_ = nullptr;}btn_push_.SetWindowTextW(_T("推送RTMP"));
}

PushVideoFrame方法将处理后的视频数据实时推送到RTMP服务器。

5. 处理错误与缓冲

SDK还提供了事件回调来处理错误和缓冲事件。例如，当播放器开始缓冲时，将触发以下回调：

extern "C" NT_VOID NT_CALLBACK NT_Push_SDKEventHandle(NT_HANDLE handle, NT_PVOID user_data,NT_UINT32 event_id,NT_INT64  param1,NT_INT64  param2,NT_UINT64 param3,NT_UINT64 param4,NT_PCSTR  param5,NT_PCSTR  param6,NT_PVOID  param7)
{if (user_data == NULL)return;HWND hwnd = (HWND)user_data;if (NT_PB_E_EVENT_ID_CONNECTING == event_id|| NT_PB_E_EVENT_ID_CONNECTION_FAILED == event_id|| NT_PB_E_EVENT_ID_CONNECTED == event_id|| NT_PB_E_EVENT_ID_DISCONNECTED == event_id){if (hwnd != nullptr && ::IsWindow(hwnd)){auto event_info = new ConnectionEventInfo(handle, event_id, param5);::PostMessage(hwnd, WM_USER_PB_SDK_CONNECTION_INFO, (WPARAM)event_info, 0);}}if (NT_PB_E_EVENT_ID_RTSP_URL == event_id){if (hwnd != nullptr && ::IsWindow(hwnd)){if (param5 != nullptr){auto url_event_data = new NT_PushRtspURLEventData(handle, param5);::PostMessage(hwnd, WM_USER_SDK_PUSH_RTSP_URL_EVENT, (WPARAM)url_event_data, 0);}}}
}

该事件确保应用程序能在网络或流中断时做出相应处理。

结论

构建一个RTSP|RTMP播放器并进行RGB帧处理是实时媒体应用中的一项基本技能。通过使用SmartPlayer SDK，我们能够轻松地集成视频解码、帧处理和流推送在一个平台中。这种解决方案允许我们在视频流中进行自定义的视觉处理，并在处理后将其推送到RTMP服务器。大牛直播SDK提供了必要的构建块，包括帧处理、事件驱动的回调和视频渲染，使得它成为开发专业直播应用程序的强大工具。