gstreamer之GstVideoDecoder源码剖析

GStreamer 中的 GstVideoDecoder 基类旨在为实现视频解码器提供一个框架。它定义了一套规则和规范，用于指导基类与其派生子类（具体的视频解码器）之间如何交互与协作。

/*** SECTION:gstvideodecoder* @title: GstVideoDecoder* @short_description: Base class for video decoders** This base class is for video decoders turning encoded data into raw video* frames.** The GstVideoDecoder base class and derived subclasses should cooperate as* follows:** ## Configuration**   * Initially, GstVideoDecoder calls @start when the decoder element*     is activated, which allows the subclass to perform any global setup.**   * GstVideoDecoder calls @set_format to inform the subclass of caps*     describing input video data that it is about to receive, including*     possibly configuration data.*     While unlikely, it might be called more than once, if changing input*     parameters require reconfiguration.**   * Incoming data buffers are processed as needed, described in Data*     Processing below.**   * GstVideoDecoder calls @stop at end of all processing.** ## Data processing**   * The base class gathers input data, and optionally allows subclass*     to parse this into subsequently manageable chunks, typically*     corresponding to and referred to as 'frames'.**   * Each input frame is provided in turn to the subclass' @handle_frame*     callback.*   * When the subclass enables the subframe mode with `gst_video_decoder_set_subframe_mode`,*     the base class will provide to the subclass the same input frame with*     different input buffers to the subclass @handle_frame*     callback. During this call, the subclass needs to take*     ownership of the input_buffer as @GstVideoCodecFrame.input_buffer*     will have been changed before the next subframe buffer is received.*     The subclass will call `gst_video_decoder_have_last_subframe`*     when a new input frame can be created by the base class.*     Every subframe will share the same @GstVideoCodecFrame.output_buffer*     to write the decoding result. The subclass is responsible to protect*     its access.**   * If codec processing results in decoded data, the subclass should call*     @gst_video_decoder_finish_frame to have decoded data pushed*     downstream. In subframe mode*     the subclass should call @gst_video_decoder_finish_subframe until the*     last subframe where it should call @gst_video_decoder_finish_frame.*     The subclass can detect the last subframe using GST_VIDEO_BUFFER_FLAG_MARKER*     on buffers or using its own logic to collect the subframes.*     In case of decoding failure, the subclass must call*     @gst_video_decoder_drop_frame or @gst_video_decoder_drop_subframe,*     to allow the base class to do timestamp and offset tracking, and possibly*     to requeue the frame for a later attempt in the case of reverse playback.** ## Shutdown phase**   * The GstVideoDecoder class calls @stop to inform the subclass that data*     parsing will be stopped.** ## Additional Notes**   * Seeking/Flushing**     * When the pipeline is seeked or otherwise flushed, the subclass is*       informed via a call to its @reset callback, with the hard parameter*       set to true. This indicates the subclass should drop any internal data*       queues and timestamps and prepare for a fresh set of buffers to arrive*       for parsing and decoding.**   * End Of Stream**     * At end-of-stream, the subclass @parse function may be called some final*       times with the at_eos parameter set to true, indicating that the element*       should not expect any more data to be arriving, and it should parse and*       remaining frames and call gst_video_decoder_have_frame() if possible.** The subclass is responsible for providing pad template caps for* source and sink pads. The pads need to be named "sink" and "src". It also* needs to provide information about the output caps, when they are known.* This may be when the base class calls the subclass' @set_format function,* though it might be during decoding, before calling* @gst_video_decoder_finish_frame. This is done via* @gst_video_decoder_set_output_state** The subclass is also responsible for providing (presentation) timestamps* (likely based on corresponding input ones).  If that is not applicable* or possible, the base class provides limited framerate based interpolation.** Similarly, the base class provides some limited (legacy) seeking support* if specifically requested by the subclass, as full-fledged support* should rather be left to upstream demuxer, parser or alike.  This simple* approach caters for seeking and duration reporting using estimated input* bitrates. To enable it, a subclass should call* @gst_video_decoder_set_estimate_rate to enable handling of incoming* byte-streams.** The base class provides some support for reverse playback, in particular* in case incoming data is not packetized or upstream does not provide* fragments on keyframe boundaries.  However, the subclass should then be* prepared for the parsing and frame processing stage to occur separately* (in normal forward processing, the latter immediately follows the former),* The subclass also needs to ensure the parsing stage properly marks* keyframes, unless it knows the upstream elements will do so properly for* incoming data.** The bare minimum that a functional subclass needs to implement is:**   * Provide pad templates*   * Inform the base class of output caps via*      @gst_video_decoder_set_output_state**   * Parse input data, if it is not considered packetized from upstream*      Data will be provided to @parse which should invoke*      @gst_video_decoder_add_to_frame and @gst_video_decoder_have_frame to*      separate the data belonging to each video frame.**   * Accept data in @handle_frame and provide decoded results to*      @gst_video_decoder_finish_frame, or call @gst_video_decoder_drop_frame.*/

GstVideoDecoderClass定义的虚函数：

/*** GstVideoDecoderClass:* @open:           Optional.*                  Called when the element changes to GST_STATE_READY.*                  Allows opening external resources.* @close:          Optional.*                  Called when the element changes to GST_STATE_NULL.*                  Allows closing external resources.* @start:          Optional.*                  Called when the element starts processing.*                  Allows opening external resources.* @stop:           Optional.*                  Called when the element stops processing.*                  Allows closing external resources.* @set_format:     Notifies subclass of incoming data format (caps).* @parse:          Required for non-packetized input.*                  Allows chopping incoming data into manageable units (frames)*                  for subsequent decoding.* @reset:          Optional.*                  Allows subclass (decoder) to perform post-seek semantics reset.*                  Deprecated.* @handle_frame:   Provides input data frame to subclass. In subframe mode, the subclass needs*                  to take ownership of @GstVideoCodecFrame.input_buffer as it will be modified*                  by the base class on the next subframe buffer receiving.* @finish:         Optional.*                  Called to request subclass to dispatch any pending remaining*                  data at EOS. Sub-classes can refuse to decode new data after.* @drain:	    Optional.*                  Called to request subclass to decode any data it can at this*                  point, but that more data may arrive after. (e.g. at segment end).*                  Sub-classes should be prepared to handle new data afterward,*                  or seamless segment processing will break. Since: 1.6* @sink_event:     Optional.*                  Event handler on the sink pad. This function should return*                  TRUE if the event was handled and should be discarded*                  (i.e. not unref'ed).*                  Subclasses should chain up to the parent implementation to*                  invoke the default handler.* @src_event:      Optional.*                  Event handler on the source pad. This function should return*                  TRUE if the event was handled and should be discarded*                  (i.e. not unref'ed).*                  Subclasses should chain up to the parent implementation to*                  invoke the default handler.* @negotiate:      Optional.*                  Negotiate with downstream and configure buffer pools, etc.*                  Subclasses should chain up to the parent implementation to*                  invoke the default handler.* @decide_allocation: Optional.*                     Setup the allocation parameters for allocating output*                     buffers. The passed in query contains the result of the*                     downstream allocation query.*                     Subclasses should chain up to the parent implementation to*                     invoke the default handler.* @propose_allocation: Optional.*                      Propose buffer allocation parameters for upstream elements.*                      Subclasses should chain up to the parent implementation to*                      invoke the default handler.* @flush:              Optional.*                      Flush all remaining data from the decoder without*                      pushing it downstream. Since: 1.2* @sink_query:     Optional.*                  Query handler on the sink pad. This function should*                  return TRUE if the query could be performed. Subclasses*                  should chain up to the parent implementation to invoke the*                  default handler. Since: 1.4* @src_query:      Optional.*                  Query handler on the source pad. This function should*                  return TRUE if the query could be performed. Subclasses*                  should chain up to the parent implementation to invoke the*                  default handler. Since: 1.4* @getcaps:        Optional.*                  Allows for a custom sink getcaps implementation.*                  If not implemented, default returns*                  gst_video_decoder_proxy_getcaps*                  applied to sink template caps.* @transform_meta: Optional. Transform the metadata on the input buffer to the*                  output buffer. By default this method is copies all meta without*                  tags and meta with only the "video" tag. subclasses can*                  implement this method and return %TRUE if the metadata is to be*                  copied. Since: 1.6** Subclasses can override any of the available virtual methods or not, as* needed. At minimum @handle_frame needs to be overridden, and @set_format* and likely as well.  If non-packetized input is supported or expected,* @parse needs to be overridden as well.*/

如注释所说子类至少需要重写handle_frame和set_format，如果non-packetized要重写parse方法。

本文以jpegdec插件为例，重写的关键函数：

  vdec_class->start = gst_jpeg_dec_start;vdec_class->stop = gst_jpeg_dec_stop;vdec_class->flush = gst_jpeg_dec_flush;vdec_class->parse = gst_jpeg_dec_parse;vdec_class->set_format = gst_jpeg_dec_set_format;vdec_class->handle_frame = gst_jpeg_dec_handle_frame;vdec_class->decide_allocation = gst_jpeg_dec_decide_allocation;vdec_class->sink_event = gst_jpeg_dec_sink_event;

: Notifies subclass of incoming data format (caps)

set_format调用在gst_video_decoder_sink_event_default->gst_video_decoder_setcaps

    case GST_EVENT_CAPS:{GstCaps *caps;gst_event_parse_caps (event, &caps);ret = gst_video_decoder_setcaps (decoder, caps);gst_event_unref (event);event = NULL;break;}

static gboolean
gst_video_decoder_setcaps (GstVideoDecoder * decoder, GstCaps * caps)
{GstVideoDecoderClass *decoder_class;GstVideoCodecState *state;gboolean ret = TRUE;decoder_class = GST_VIDEO_DECODER_GET_CLASS (decoder);GST_DEBUG_OBJECT (decoder, "setcaps %" GST_PTR_FORMAT, caps);GST_VIDEO_DECODER_STREAM_LOCK (decoder);if (decoder->priv->input_state) {GST_DEBUG_OBJECT (decoder,"Checking if caps changed old %" GST_PTR_FORMAT " new %" GST_PTR_FORMAT,decoder->priv->input_state->caps, caps);if (gst_caps_is_equal (decoder->priv->input_state->caps, caps))goto caps_not_changed;}state = _new_input_state (caps);if (G_UNLIKELY (state == NULL))goto parse_fail;if (decoder_class->set_format)ret = decoder_class->set_format (decoder, state);if (!ret)goto refused_format;if (decoder->priv->input_state)gst_video_codec_state_unref (decoder->priv->input_state);decoder->priv->input_state = state;GST_VIDEO_DECODER_STREAM_UNLOCK (decoder);return ret;
}

/*** GstVideoCodecState:* @info: The #GstVideoInfo describing the stream* @caps: The #GstCaps used in the caps negotiation of the pad.* @codec_data: a #GstBuffer corresponding to the*     'codec_data' field of a stream, or NULL.* @allocation_caps: The #GstCaps for allocation query and pool*     negotiation. Since: 1.10* @mastering_display_info: Mastering display color volume information*     (HDR metadata) for the stream. Since: 1.20* @content_light_level: Content light level information for the stream.*     Since: 1.20** Structure representing the state of an incoming or outgoing video* stream for encoders and decoders.** Decoders and encoders will receive such a state through their* respective @set_format vmethods.** Decoders and encoders can set the downstream state, by using the* gst_video_decoder_set_output_state() or* gst_video_encoder_set_output_state() methods.*/
/*** GstVideoCodecState.mastering_display_info:** Mastering display color volume information (HDR metadata) for the stream.** Since: 1.20*/
/*** GstVideoCodecState.content_light_level:** Content light level information for the stream.** Since: 1.20*/
struct _GstVideoCodecState
{/*< private >*/gint ref_count;/*< public >*/GstVideoInfo info;GstCaps *caps;GstBuffer *codec_data;GstCaps *allocation_caps;GstVideoMasteringDisplayInfo *mastering_display_info;GstVideoContentLightLevel *content_light_level;/*< private >*/gpointer padding[GST_PADDING_LARGE - 3];
};

上游插件通过GST_EVENT_CAPS事件设置caps，然后保存在了GstVideoCodecState中。

继续看gst_video_decoder_change_state

  switch (transition) {case GST_STATE_CHANGE_NULL_TO_READY:/* open device/library if needed */if (decoder_class->open && !decoder_class->open (decoder))goto open_failed;break;case GST_STATE_CHANGE_READY_TO_PAUSED:GST_VIDEO_DECODER_STREAM_LOCK (decoder);gst_video_decoder_reset (decoder, TRUE, TRUE);GST_VIDEO_DECODER_STREAM_UNLOCK (decoder);/* Initialize device/library if needed */if (decoder_class->start && !decoder_class->start (decoder))goto start_failed;break;default:break;}

会依次调用子类的decoder_class->open和decoder_class->start。jpegdec没有实现open

static gboolean
gst_jpeg_dec_start (GstVideoDecoder * bdec)
{GstJpegDec *dec = (GstJpegDec *) bdec;#ifdef JCS_EXTENSIONSdec->format_convert = FALSE;
#endifdec->saw_header = FALSE;dec->parse_entropy_len = 0;dec->parse_resync = FALSE;gst_video_decoder_set_packetized (bdec, FALSE);return TRUE;
}

上游插件push数据调用chain函数

static GstFlowReturn
gst_video_decoder_chain (GstPad * pad, GstObject * parent, GstBuffer * buf)
{GstVideoDecoder *decoder;GstFlowReturn ret = GST_FLOW_OK;decoder = GST_VIDEO_DECODER (parent);if (G_UNLIKELY (!decoder->priv->input_state && decoder->priv->needs_format))goto not_negotiated;GST_LOG_OBJECT (decoder,"chain PTS %" GST_TIME_FORMAT ", DTS %" GST_TIME_FORMAT " duration %"GST_TIME_FORMAT " size %" G_GSIZE_FORMAT " flags %x",GST_TIME_ARGS (GST_BUFFER_PTS (buf)),GST_TIME_ARGS (GST_BUFFER_DTS (buf)),GST_TIME_ARGS (GST_BUFFER_DURATION (buf)),gst_buffer_get_size (buf), GST_BUFFER_FLAGS (buf));GST_VIDEO_DECODER_STREAM_LOCK (decoder);/* NOTE:* requiring the pad to be negotiated makes it impossible to use* oggdemux or filesrc ! decoder */if (decoder->input_segment.format == GST_FORMAT_UNDEFINED) {GstEvent *event;GstSegment *segment = &decoder->input_segment;GST_WARNING_OBJECT (decoder,"Received buffer without a new-segment. ""Assuming timestamps start from 0.");gst_segment_init (segment, GST_FORMAT_TIME);event = gst_event_new_segment (segment);decoder->priv->current_frame_events =g_list_prepend (decoder->priv->current_frame_events, event);}decoder->priv->had_input_data = TRUE;if (decoder->input_segment.rate > 0.0)ret = gst_video_decoder_chain_forward (decoder, buf, FALSE);elseret = gst_video_decoder_chain_reverse (decoder, buf);GST_VIDEO_DECODER_STREAM_UNLOCK (decoder);return ret;/* ERRORS */
not_negotiated:{GST_ELEMENT_ERROR (decoder, CORE, NEGOTIATION, (NULL),("decoder not initialized"));gst_buffer_unref (buf);return GST_FLOW_NOT_NEGOTIATED;}
}

chain函数中根据decoder->input_segment.rate 调用

  if (decoder->input_segment.rate > 0.0)ret = gst_video_decoder_chain_forward (decoder, buf, FALSE);elseret = gst_video_decoder_chain_reverse (decoder, buf);

重点看下gst_video_decoder_chain_forward

static GstFlowReturn
gst_video_decoder_chain_forward (GstVideoDecoder * decoder,GstBuffer * buf, gboolean at_eos)
{GstVideoDecoderPrivate *priv;GstVideoDecoderClass *klass;GstFlowReturn ret = GST_FLOW_OK;klass = GST_VIDEO_DECODER_GET_CLASS (decoder);priv = decoder->priv;g_return_val_if_fail (priv->packetized || klass->parse, GST_FLOW_ERROR);/* Draining on DISCONT is handled in chain_reverse() for reverse playback,* and this function would only be called to get everything collected GOP* by GOP in the parse_gather list */if (decoder->input_segment.rate > 0.0 && GST_BUFFER_IS_DISCONT (buf)&& (decoder->input_segment.flags & GST_SEEK_FLAG_TRICKMODE_KEY_UNITS))ret = gst_video_decoder_drain_out (decoder, FALSE);if (priv->current_frame == NULL)priv->current_frame = gst_video_decoder_new_frame (decoder);if (!priv->packetized)gst_video_decoder_add_buffer_info (decoder, buf);priv->input_offset += gst_buffer_get_size (buf);if (priv->packetized) {GstVideoCodecFrame *frame;gboolean was_keyframe = FALSE;frame = priv->current_frame;frame->abidata.ABI.num_subframes++;if (gst_video_decoder_get_subframe_mode (decoder)) {/* End the frame if the marker flag is set */if (!GST_BUFFER_FLAG_IS_SET (buf, GST_VIDEO_BUFFER_FLAG_MARKER)&& (decoder->input_segment.rate > 0.0))priv->current_frame = gst_video_codec_frame_ref (frame);elsepriv->current_frame = NULL;} else {priv->current_frame = frame;}if (!GST_BUFFER_FLAG_IS_SET (buf, GST_BUFFER_FLAG_DELTA_UNIT)) {was_keyframe = TRUE;GST_DEBUG_OBJECT (decoder, "Marking current_frame as sync point");GST_VIDEO_CODEC_FRAME_SET_SYNC_POINT (frame);}if (frame->input_buffer) {gst_video_decoder_copy_metas (decoder, frame, frame->input_buffer, buf);gst_buffer_unref (frame->input_buffer);}frame->input_buffer = buf;if (decoder->input_segment.rate < 0.0) {priv->parse_gather = g_list_prepend (priv->parse_gather, frame);priv->current_frame = NULL;} else {ret = gst_video_decoder_decode_frame (decoder, frame);if (!gst_video_decoder_get_subframe_mode (decoder))priv->current_frame = NULL;}/* If in trick mode and it was a keyframe, drain decoder to avoid extra* latency. Only do this for forwards playback as reverse playback handles* draining on keyframes in flush_parse(), and would otherwise call back* from drain_out() to here causing an infinite loop.* Also this function is only called for reverse playback to gather frames* GOP by GOP, and does not do any actual decoding. That would be done by* flush_decode() */if (ret == GST_FLOW_OK && was_keyframe && decoder->input_segment.rate > 0.0&& (decoder->input_segment.flags & GST_SEEK_FLAG_TRICKMODE_KEY_UNITS))ret = gst_video_decoder_drain_out (decoder, FALSE);} else {gst_adapter_push (priv->input_adapter, buf);ret = gst_video_decoder_parse_available (decoder, at_eos, TRUE);}if (ret == GST_VIDEO_DECODER_FLOW_NEED_DATA)return GST_FLOW_OK;return ret;
}

关键流程：

（1）如果priv->packetized == false,会走直接走gst_video_decoder_decode_frame，然后调用子类的decoder_class->handle_frame函数

（2） 如果priv->packetized == true,会走gst_adapter_push数据推送到input_adapter，并且调用

gst_video_decoder_parse_available，继而调用子类的decoder_class->parse函数进行数据解析。

jpegdec的gst_jpeg_dec_handle_frame，完成了数据的解码处理。

static GstFlowReturn
gst_jpeg_dec_handle_frame (GstVideoDecoder * bdec, GstVideoCodecFrame * frame)
{GstFlowReturn ret = GST_FLOW_OK;GstJpegDec *dec = (GstJpegDec *) bdec;GstVideoFrame vframe;gint num_fields;              /* number of fields (1 or 2) */gint output_height;           /* height of output image (one or two fields) */gint height;                  /* height of current frame (whole image or a field) */gint width;guint code;gboolean need_unmap = TRUE;GstVideoCodecState *state = NULL;gboolean release_frame = TRUE;gboolean has_eoi;guint8 *data;gsize nbytes;if (!gst_buffer_map (frame->input_buffer, &dec->current_frame_map,GST_MAP_READ))goto map_failed;data = dec->current_frame_map.data;nbytes = dec->current_frame_map.size;if (nbytes < 2)goto need_more_data;has_eoi = ((data[nbytes - 2] == 0xff) && (data[nbytes - 1] == 0xd9));/* some cameras fail to send an end-of-image marker (EOI),* add it if that is the case. */if (!has_eoi) {GstMapInfo map;GstBuffer *eoibuf = gst_buffer_new_and_alloc (2);/* unmap, will add EOI and remap at the end */gst_buffer_unmap (frame->input_buffer, &dec->current_frame_map);gst_buffer_map (eoibuf, &map, GST_MAP_WRITE);map.data[0] = 0xff;map.data[1] = 0xd9;gst_buffer_unmap (eoibuf, &map);/* append to input buffer, and remap */frame->input_buffer = gst_buffer_append (frame->input_buffer, eoibuf);gst_buffer_map (frame->input_buffer, &dec->current_frame_map, GST_MAP_READ);GST_DEBUG ("fixup EOI marker added");}dec->current_frame = frame;dec->cinfo.src->next_input_byte = dec->current_frame_map.data;dec->cinfo.src->bytes_in_buffer = dec->current_frame_map.size;if (setjmp (dec->jerr.setjmp_buffer)) {code = dec->jerr.pub.msg_code;if (code == JERR_INPUT_EOF) {GST_DEBUG ("jpeg input EOF error, we probably need more data");goto need_more_data;}goto decode_error;}/* read header and check values */ret = gst_jpeg_dec_prepare_decode (dec);if (G_UNLIKELY (ret == GST_FLOW_ERROR))goto done;width = dec->cinfo.output_width;height = dec->cinfo.output_height;/* is it interlaced MJPEG? (we really don't want to scan the jpeg data* to see if there are two SOF markers in the packet to detect this) */if (gst_video_decoder_get_packetized (bdec) &&dec->input_state &&dec->input_state->info.height > height &&dec->input_state->info.height <= (height * 2)&& dec->input_state->info.width == width) {GST_LOG_OBJECT (dec,"looks like an interlaced image: ""input width/height of %dx%d with JPEG frame width/height of %dx%d",dec->input_state->info.width, dec->input_state->info.height, width,height);output_height = dec->input_state->info.height;height = dec->input_state->info.height / 2;num_fields = 2;GST_LOG_OBJECT (dec, "field height=%d", height);} else {output_height = height;num_fields = 1;}gst_jpeg_dec_negotiate (dec, width, output_height,dec->cinfo.jpeg_color_space, num_fields == 2);state = gst_video_decoder_get_output_state (bdec);ret = gst_video_decoder_allocate_output_frame (bdec, frame);if (G_UNLIKELY (ret != GST_FLOW_OK))goto alloc_failed;if (!gst_video_frame_map (&vframe, &state->info, frame->output_buffer,GST_MAP_READWRITE))goto alloc_failed;if (setjmp (dec->jerr.setjmp_buffer)) {code = dec->jerr.pub.msg_code;gst_video_frame_unmap (&vframe);goto decode_error;}GST_LOG_OBJECT (dec, "width %d, height %d, fields %d", width, output_height,num_fields);ret = gst_jpeg_dec_decode (dec, &vframe, width, height, 1, num_fields);if (G_UNLIKELY (ret != GST_FLOW_OK)) {gst_video_frame_unmap (&vframe);goto decode_failed;}if (setjmp (dec->jerr.setjmp_buffer)) {code = dec->jerr.pub.msg_code;gst_video_frame_unmap (&vframe);goto decode_error;}/* decode second field if there is one */if (num_fields == 2) {GstVideoFormat field2_format;/* Checked above before setting num_fields to 2 */g_assert (dec->input_state != NULL);/* skip any chunk or padding bytes before the next SOI marker; both fields* are in one single buffer here, so direct access should be fine here */while (dec->jsrc.pub.bytes_in_buffer > 2 &&GST_READ_UINT16_BE (dec->jsrc.pub.next_input_byte) != 0xffd8) {--dec->jsrc.pub.bytes_in_buffer;++dec->jsrc.pub.next_input_byte;}if (gst_jpeg_dec_prepare_decode (dec) != GST_FLOW_OK) {GST_WARNING_OBJECT (dec, "problem reading jpeg header of 2nd field");/* FIXME: post a warning message here? */gst_video_frame_unmap (&vframe);goto decode_failed;}/* check if format has changed for the second field */
#ifdef JCS_EXTENSIONSif (dec->format_convert) {field2_format = dec->format;} else
#endif{switch (dec->cinfo.jpeg_color_space) {case JCS_RGB:field2_format = GST_VIDEO_FORMAT_RGB;break;case JCS_GRAYSCALE:field2_format = GST_VIDEO_FORMAT_GRAY8;break;default:field2_format = GST_VIDEO_FORMAT_I420;break;}}GST_LOG_OBJECT (dec,"got for second field of interlaced image: ""input width/height of %dx%d with JPEG frame width/height of %dx%d",dec->input_state->info.width, dec->input_state->info.height,dec->cinfo.output_width, dec->cinfo.output_height);if (dec->cinfo.output_width != GST_VIDEO_INFO_WIDTH (&state->info) ||GST_VIDEO_INFO_HEIGHT (&state->info) <= dec->cinfo.output_height ||GST_VIDEO_INFO_HEIGHT (&state->info) > (dec->cinfo.output_height * 2) ||field2_format != GST_VIDEO_INFO_FORMAT (&state->info)) {GST_WARNING_OBJECT (dec, "second field has different format than first");gst_video_frame_unmap (&vframe);goto decode_failed;}ret = gst_jpeg_dec_decode (dec, &vframe, width, height, 2, 2);if (G_UNLIKELY (ret != GST_FLOW_OK)) {gst_video_frame_unmap (&vframe);goto decode_failed;}}gst_video_frame_unmap (&vframe);gst_buffer_unmap (frame->input_buffer, &dec->current_frame_map);ret = gst_video_decoder_finish_frame (bdec, frame);release_frame = FALSE;need_unmap = FALSE;done:exit:if (need_unmap)gst_buffer_unmap (frame->input_buffer, &dec->current_frame_map);if (release_frame)gst_video_decoder_release_frame (bdec, frame);if (state)gst_video_codec_state_unref (state);return ret;/* special cases */
need_more_data:{GST_LOG_OBJECT (dec, "we need more data");ret = GST_FLOW_OK;goto exit;}/* ERRORS */
map_failed:{GST_ELEMENT_ERROR (dec, RESOURCE, READ, (_("Failed to read memory")),("gst_buffer_map() failed for READ access"));ret = GST_FLOW_ERROR;goto exit;}
decode_error:{gchar err_msg[JMSG_LENGTH_MAX];dec->jerr.pub.format_message ((j_common_ptr) (&dec->cinfo), err_msg);GST_VIDEO_DECODER_ERROR (dec, 1, STREAM, DECODE,(_("Failed to decode JPEG image")), ("Decode error #%u: %s", code,err_msg), ret);gst_buffer_unmap (frame->input_buffer, &dec->current_frame_map);gst_video_decoder_drop_frame (bdec, frame);release_frame = FALSE;need_unmap = FALSE;jpeg_abort_decompress (&dec->cinfo);goto done;}
decode_failed:{/* already posted an error message */goto done;}
alloc_failed:{const gchar *reason;reason = gst_flow_get_name (ret);GST_DEBUG_OBJECT (dec, "failed to alloc buffer, reason %s", reason);/* Reset for next time */jpeg_abort_decompress (&dec->cinfo);if (ret != GST_FLOW_EOS && ret != GST_FLOW_FLUSHING &&ret != GST_FLOW_NOT_LINKED) {GST_VIDEO_DECODER_ERROR (dec, 1, STREAM, DECODE,(_("Failed to decode JPEG image")),("Buffer allocation failed, reason: %s", reason), ret);jpeg_abort_decompress (&dec->cinfo);}goto exit;}
}

而priv->packetized流会走流程2也就是parse，parse会不断解析数据，直到解析到完整的一帧数据后，然后会走到have_full_frame流程。

static GstFlowReturn
gst_jpeg_dec_parse (GstVideoDecoder * bdec, GstVideoCodecFrame * frame,GstAdapter * adapter, gboolean at_eos)
{guint size;gint toadd = 0;gboolean resync;gint offset = 0, noffset;GstJpegDec *dec = (GstJpegDec *) bdec;GST_VIDEO_CODEC_FRAME_SET_SYNC_POINT (frame);/* FIXME : The overhead of using scan_uint32 is massive */size = gst_adapter_available (adapter);GST_DEBUG ("Parsing jpeg image data (%u bytes)", size);if (at_eos) {GST_DEBUG ("Flushing all data out");toadd = size;/* If we have leftover data, throw it away */if (!dec->saw_header)goto drop_frame;goto have_full_frame;}if (size < 8)goto need_more_data;if (!dec->saw_header) {gint ret;/* we expect at least 4 bytes, first of which start marker */ret =gst_adapter_masked_scan_uint32 (adapter, 0xffff0000, 0xffd80000, 0,size - 4);GST_DEBUG ("ret:%d", ret);if (ret < 0)goto need_more_data;if (ret) {gst_adapter_flush (adapter, ret);size -= ret;}dec->saw_header = TRUE;}while (1) {guint frame_len;guint32 value;GST_DEBUG ("offset:%d, size:%d", offset, size);noffset =gst_adapter_masked_scan_uint32_peek (adapter, 0x0000ff00, 0x0000ff00,offset, size - offset, &value);/* lost sync if 0xff marker not where expected */if ((resync = (noffset != offset))) {GST_DEBUG ("Lost sync at 0x%08x, resyncing", offset + 2);}/* may have marker, but could have been resyncng */resync = resync || dec->parse_resync;/* Skip over extra 0xff */while ((noffset >= 0) && ((value & 0xff) == 0xff)) {noffset++;noffset =gst_adapter_masked_scan_uint32_peek (adapter, 0x0000ff00, 0x0000ff00,noffset, size - noffset, &value);}/* enough bytes left for marker? (we need 0xNN after the 0xff) */if (noffset < 0) {GST_DEBUG ("at end of input and no EOI marker found, need more data");goto need_more_data;}/* now lock on the marker we found */offset = noffset;value = value & 0xff;if (value == 0xd9) {GST_DEBUG ("0x%08x: EOI marker", offset + 2);/* clear parse state */dec->saw_header = FALSE;dec->parse_resync = FALSE;toadd = offset + 4;goto have_full_frame;}if (value == 0xd8) {GST_DEBUG ("0x%08x: SOI marker before EOI marker", offset + 2);/* clear parse state */dec->saw_header = FALSE;dec->parse_resync = FALSE;toadd = offset;goto have_full_frame;}if (value >= 0xd0 && value <= 0xd7)frame_len = 0;else {/* peek tag and subsequent length */if (offset + 2 + 4 > size)goto need_more_data;elsegst_adapter_masked_scan_uint32_peek (adapter, 0x0, 0x0, offset + 2, 4,&frame_len);frame_len = frame_len & 0xffff;}GST_DEBUG ("0x%08x: tag %02x, frame_len=%u", offset + 2, value, frame_len);/* the frame length includes the 2 bytes for the length; here we want at* least 2 more bytes at the end for an end marker */if (offset + 2 + 2 + frame_len + 2 > size) {goto need_more_data;}if (gst_jpeg_dec_parse_tag_has_entropy_segment (value)) {guint eseglen = dec->parse_entropy_len;GST_DEBUG ("0x%08x: finding entropy segment length (eseglen:%d)",offset + 2, eseglen);if (size < offset + 2 + frame_len + eseglen)goto need_more_data;noffset = offset + 2 + frame_len + dec->parse_entropy_len;while (1) {GST_DEBUG ("noffset:%d, size:%d, size - noffset:%d",noffset, size, size - noffset);noffset = gst_adapter_masked_scan_uint32_peek (adapter, 0x0000ff00,0x0000ff00, noffset, size - noffset, &value);if (noffset < 0) {/* need more data */dec->parse_entropy_len = size - offset - 4 - frame_len - 2;goto need_more_data;}if ((value & 0xff) != 0x00) {eseglen = noffset - offset - frame_len - 2;break;}noffset++;}dec->parse_entropy_len = 0;frame_len += eseglen;GST_DEBUG ("entropy segment length=%u => frame_len=%u", eseglen,frame_len);}if (resync) {/* check if we will still be in sync if we interpret* this as a sync point and skip this frame */noffset = offset + frame_len + 2;noffset = gst_adapter_masked_scan_uint32 (adapter, 0x0000ff00, 0x0000ff00,noffset, 4);if (noffset < 0) {/* ignore and continue resyncing until we hit the end* of our data or find a sync point that looks okay */offset++;continue;}GST_DEBUG ("found sync at 0x%x", offset + 2);}/* Add current data to output buffer */toadd += frame_len + 2;offset += frame_len + 2;}need_more_data:if (toadd)gst_video_decoder_add_to_frame (bdec, toadd);return GST_VIDEO_DECODER_FLOW_NEED_DATA;have_full_frame:if (toadd)gst_video_decoder_add_to_frame (bdec, toadd);GST_VIDEO_CODEC_FRAME_SET_SYNC_POINT (frame);return gst_video_decoder_have_frame (bdec);drop_frame:gst_adapter_flush (adapter, size);return GST_FLOW_OK;
}

其中gst_video_decoder_add_to_frame会将数据push到priv->output_adapter中。还会调用

gst_video_decoder_have_frame，其中关键流程会调用gst_video_decoder_decode_frame继而调用到decoder_class->handle_frame，完成解码。

  /* In reverse playback, just capture and queue frames for later processing */if (decoder->input_segment.rate < 0.0) {priv->parse_gather =g_list_prepend (priv->parse_gather, priv->current_frame);priv->current_frame = NULL;} else {GstVideoCodecFrame *frame = priv->current_frame;frame->abidata.ABI.num_subframes++;/* In subframe mode, we keep a ref for ourselves* as this frame will be kept during the data collection* in parsed mode. The frame reference will be released by* finish_(sub)frame or drop_(sub)frame.*/if (gst_video_decoder_get_subframe_mode (decoder))gst_video_codec_frame_ref (priv->current_frame);elsepriv->current_frame = NULL;/* Decode the frame, which gives away our ref */ret = gst_video_decoder_decode_frame (decoder, frame);}

packetized流，也就是上游送下的数据，并不是可以直接解码的原始数据流，所以必须通过parse从数据中解析出来可解码的原始数据。

/*** gst_video_decoder_set_packetized:* @decoder: a #GstVideoDecoder* @packetized: whether the input data should be considered as packetized.** Allows baseclass to consider input data as packetized or not. If the* input is packetized, then the @parse method will not be called.*/
void
gst_video_decoder_set_packetized (GstVideoDecoder * decoder,gboolean packetized)
{decoder->priv->packetized = packetized;
}

总结：

GstVideoDecoder流程不是很复杂，主要通过set_format设置caps，通过handle_frame我对数据进行解码。