Video Playback and Compositor

Video Playback and Compositor

Authors: [email protected], [email protected]

The Chromium compositor has support for video playback to support offloading work to the GPU and displaying video frames while the main thread is blocked.  There are a few different media engine implementations in Chromium, but they interact with the compositor in similar ways.  Here’s a diagram of the different classes and how they interact:

Things in dotted square boxes are interfaces, things in ovals are concrete classes.  Arrows represent pointers between objects.

Objects by thread (note that a few span multiple threads):

Before video playback:

1.) WebMediaPlayerClientImpl constructed by WebCore

At start of video load:

2.) WebMediaPlayerClientImpl constructs WebMediaPlayer

In SetReadyState, if the video has loaded enough to know we want composited playback:

3.) WebMediaPlayerImpl constructs a WebLayerImpl wrapping a cc::VideoLayer with the provider pointer set to the WebMediaPlayerImpl

4.) WebMediaPlayerImpl provides WebLayer* to WebMediaPlayerClientImpl to register in WebCore’s compositing tree

On the next compositor commit during tree sync (on compositor thread with main thread blocked):

5.) cc::VideoLayer creates a cc::VideoLayerImpl and passes the cc::VideoFrameProvider*

6.) cc::VideoLayerImpl creates a cc::VideoFrameProviderClientImpl with the provider pointer

7.) cc::VideoFrameProviderClientImpl calls SetVideoFrameProviderClient(this) on the cc::VideoFrameProvider*, which is the WebMediaPlayerImpl

On tree activation (immediately after tree sync when not in impl-side painting, at some point later on in impl-side painting)

8.) Active tree cc::VideoLayerImpl sets itself as the cc::VideoFrameProviderClientImpl’s active layer

If we fully initialize without anything shutting down, then the pathway for the video layer to grab a new frame is:

On the compositor thread when preparing to draw a frame:

1.) cc::VideoLayerImpl::WillDraw calls VideoFrameProviderClientImpl::AcquireLockAndCurrentFrame()

2.) VFPCI::ALACF takes its provider_lock_ and calls GetCurrentFrame() on its provider

3.) WebMediaPlayerImpl takes its internal lock_ and returns its current_frame_

4.) cc::VideoLayerImpl uploads data into textures (if needed)

On the compositor thread when drawing a frame:

5.) cc::VideoLayerImpl::AppendQuads adds quads into the draw list referencing the video texture(s)

On the compositor thread after drawing a frame:

6.) cc::VideoLayerImpl::DidDraw calls VideoFrameProviderClientImpl::PutCurrentFrame

7.) VFPCI::PCF calls PutCurrentFrame() on its provider

8.) cc::VideoLayerImpl::DidDraw calls VFPCI::ReleaseLock which releases provider_lock_

The way the media system can notify the compositor of new video frames on the compositor thread is (currently implemented only on android):

On the compositor thread when the media system has a new frame available

1.) Something in the media playback system (content::StreamTextureProxyImpl for android) calls cc::VideoFrameProvider::DidReceiveFrame()

2.) cc::VideoFrameProviderClientImpl checks if it has an active cc::VideoLayerImpl and calls SetNeedsRedraw() on it if so

3.) cc::VideoLayerImpl::SetNeedsRedraw tells the compositor to schedule a new frame

Shutdown

There are a few ways this system can shut down.  All shutdown paths start on the main thread, but they can propagate out to other threads in different ways depending on the type of shutdown.

Video layer removed from compositing tree while playing

This can happen if a <video> element is detached from the DOM tree or display:none is set on an ancestor of the element.

There are two cases to consider here - impl-side painting off and impl-side painting on.  With impl-side painting off, there is only one cc::VideoLayerImpl associated with a given cc::VideoLayer.  The cc::VideoLayerImpl is always created or destroyed during the tree sync part of commit which happens on the compositor thread with the main thread blocked.

With impl-side painting on, there are potentially two cc::VideoLayerImpls associated with a given cc::VideoLayer.  One is in the pending/recycle tree and is always created/destroyed during the commit tree sync on the compositor thread with the main thread blocked.  The other is in the active tree and is created/destroyed during tree activation, which happens on the compositor thread while the main thread is not blocked.  In impl-side painting mode both cc::VideoLayerImpls hold a reference to the same cc::VideoFrameProviderClientImpl.

Preconditions:

1.) WebMediaPlayerClientImpl owns a WebMediaPlayer (which is a WebMediaPlayerImpl)

2.) WebMediaPlayerImpl owns a WebLayerImpl, which has a ref to a cc::VideoLayer

3.) cc::VideoLayer has an associated cc::VideoLayerImpl (two in impl-side painting mode) which reference a cc::VideoFrameProviderClientImpl

4.) WebMediaPlayerImpl’s provider_client_ pointer points to the cc::VideoFrameProviderClientImpl

5.) cc::VideoFrameProviderClientImpl’s provider_ pointer points to the WebMediaPlayerImpl

Sequence:

On the main thread at any point in time

1.) cc::VideoLayer removed from the compositing tree and/or destroyed

In the next compositor commit on the compositor thread with the main thread blocked:

2.) Tree sync starts destruction of the cc::VideoLayerImpl (only cc::VideoLayerImpl when not in impl-side painting, pending/recycle layer in impl-side painting)

3.) ~cc::VideoLayerImpl() calls cc::VideoFrameProviderClientImpl::Stop()

4.) cc::VFPCI::Stop() calls SetVideoFrameProviderClientImpl(NULL) on its cc::VideoFrameProvider*, which is a WebMediaPlayerImpl

5.) cc::VFPCI::StopUsingProvider() takes its provider_lock_ and nulls out its provider_

6.) WebMediaPlayerImpl nulls out its provider_client_ pointer

7.) cc::VFPCI::Stop() nulls out its provider_ pointer (again)

If in impl-side painting mode, on tree activation in the compositor thread:

8.) cc::VideoLayerImpl in active tree destroyed, dropping last reference to cc::VideoFrameProviderClientImpl

Ordering:

Steps 2-7 happen on the compositor thread with the main thread blocked.  This means that the compositor cannot be in a frame for any of these steps and nothing on the main thread can advance (i.e. we can’t start shutting down the WebMediaPlayerImpl).

Postconditions:

1.) WebMediaPlayerImpl’s provider_client_ pointer is NULL (as of step 7)

2.) cc::VideoFrameProviderClientImpl’s provider_ pointer is NULL (as of step 6)

When impl-side painting is not enabled, then

3.) cc::VideoLayerImpl and cc::VideoFrameProviderClientImpl destroyed (as of step 9)

Media engine shut down while video layer is in compositing tree

Preconditions:

1.) WebMediaPlayerClientImpl owns a WebMediaPlayer (which is a WebMediaPlayerImpl)

2.) WebMediaPlayerImpl owns a WebLayerImpl, which has a ref to a cc::VideoLayer

3.) cc::VideoLayer has an associated cc::VideoLayerImpl (two in impl-side painting mode) which reference a cc::VideoFrameProviderClientImpl

4.) WebMediaPlayerImpl’s provider_client_ pointer points to the cc::VideoFrameProviderClientImpl

5.) cc::VideoFrameProviderClientImpl’s provider_ pointer points to the WebMediaPlayerImpl

6.) We’re outside a compositor commit on the main thread

7.) The compositor thread may be in any state, including activation or drawing a frame.

Sequence:

On the main thread at any point in time:

1.) WebMediaPlayerClientImpl destroys its WebMediaPlayer (which is a WebMediaPlayerImpl)

2.) ~WebMediaPlayerImpl() calls SetVideoFrameProviderClient(NULL) on its provider_client_, which is a cc::VideoFrameProvierClientImpl

3.) cc::VideoFrameProviderClientImpl::SetVideoFrameProviderClient() takes its provider_lock_

4.) cc::VideoFrameProviderClientImpl::SetVideoFrameProviderClient() nulls out its provider_ pointer, which pointed to the WebMediaPlayerImpl

5.) cc:VFPCI::SetVideoFrameProviderClient() releases its provider_lock_

6.) ~WebMediaPlayerImpl() completes and the object is destroyed

Ordering:

1.) In step (3) the cc::VFPCI takes its provider_lock_.  If the compositor thread is in the middle of drawing a frame (specifically between WillDraw() and DidDraw()), this call will block until the frame is complete

Postconditions:

1.) WebMediaPlayerImpl’s provider_client_ pointer is NULL (as of step 4)

2.) cc::VideoFrameProviderClientImpl’s provider_ pointer is NULL (as of step 5)

原文地址:https://www.cnblogs.com/huangguanyuan/p/9783528.html

时间: 2024-10-03 01:56:56

Video Playback and Compositor的相关文章

stagefright框架(一)Video Playback的流程

在Android上,預設的多媒體框架(multimedia framework)是OpenCORE. OpenCORE的優點是兼顧了跨平台的移植性,而且已經過多方驗證,所以相對來說較為穩定:但是其缺點是過於龐大複雜,需要耗費相當多的時間去維護.從 Android 2.0開始,Google引進了架構稍為簡潔的Stagefright,並且有逐漸取代OpenCORE的趨勢 (註1). [圖1] Stagefright在Android多媒體架構中的位置. [圖2] Stagefright所涵蓋的模組 (

GStreamer Plugin: Embedded video playback halted; module decodebin20 reported: Your GStreamer installation is missing a plug-in.

标题是在Linux下使用系统yum install 的opencv库来获取视频帧的时候抛出来的错误消息.opencv调用了Gstream的API来处理了视频.错误抛出的代码如下图: http://ubuntuforums.org/archive/index.php/t-1730395.html http://stackoverflow.com/questions/4929721/opencv-python-grab-frames-from-a-video-file 把上面两个链接的回答都看了,还

HDR Video

HDR Video Playback High dynamic range (HDR) video is the next frontier in high-quality video decoding, bringing unmatched scene reproduction qualities. It does so by significantly increasing the dynamic range of the luminance component (from the curr

HTML5 video标签播放视频下载原理

HTML5 video https://github.com/remy/html5demos/blob/master/demos/video.html <video preload="metadata"> <!-- https://bugzilla.mozilla.org/show_bug.cgi?id=676422 --> <source src="assets/dizzy.mp4" type="video/mp4"

Android Video Playback----from Framework to Driver

I will analyze the process of  the local video playback in Android 4.1 on Pandaboard. Hardware acceleration for HD video playback also will be discussed. 1. Video File A video file usually contains two part: File Header and Coded Video Frames. So the

video.js的简单用法

今天项目中需要跨浏览器地播放视频,在网上找了一下,找到了video.js,记录一下video.js的简单用法. ? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 <html> <head> ... <!-- 引入vide

video.js的使用

跨浏览器地播放视频,在网上找了一下,找到了video.js,记录一下video.js的简单用法. ? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 <html> <head> ... <!-- 引入video.js的样式

html5 兼容版本 video

<!-- first try HTML5 playback: if serving as XML, expand `controls` to `controls="controls"` and autoplay likewise --> <!-- warning: playback does not work on iOS3 if you include the poster attribute! fixed in iOS4.0 --> <video wi

一种video视频兼容IE的模式

一:video.html代码: 1 <!DOCTYPE html> 2 <html> 3 <head> 4 <meta charset="utf-8"> 5 <title></title> 6 <link rel="stylesheet" href="css/video-js.css" /> 7 <!--<script type="text/