jdomnitz's picture

Using MediaFoundation and OpenGL

Obviously Microsoft would prefer that media foundation be used with DirectX and has only exposed hardware accelerated interfaces for directx (unless theres a way to convert opengl contexts to a directx device handle). However, the documentation clearly indicates it should work with regular windows forms as well. I developed video playback code and gave it a try but hit a giant problem. If I use the handle of any regular windows form everything works fine....if I use the window handle of an openTK GameWindow I get no video. Any ideas??


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
the Fiddler's picture

How are you accessing the window handle for GameWindow?

It is very unlikely that you'll be able to render with MF on top of an OpenGL window. In general, every pixel must belong to a single technology (i.e. OpenGL or DirectX but not both) - this is called an airspace restriction.

Can MF render to a bitmap? That would be the most straightforward solution.

jdomnitz's picture

It allows rendering to a window and a direct3d surface...but I don't see any presenters for a bitmap. Theres an example in the MediaFoundation.Net samples for creating a custom presenter but apparently it required quite a few hacks and some really ugly code. I guess theoretically it could be ported to render to an opengl surface instead but that might be way above my skillset.

You would think after 3 years there would be some decent samples and documentation but theres barely anything out there.

jdomnitz's picture

On second though this may work:

Maybe I can even use the same approach as directshow (scroll down for the code):

Whats the fastest way to render these bitmaps assuming each bitmap is only displayed once and I need to do 30fps with minimal overhead?

Any interest in an OpenTK.Video library?

the Fiddler's picture

This looks promising. The best approach depends on your target OpenGL version. For modern implementations, Pixelbuffer Objects are likely to provide most efficient. For older implementations, you can ping-pong between two textures (write first, render second and vice versa).

If possible, try to avoid pixel format conversions on the CPU. If MF uses YUV/YUV2 formats (which is likely), consider converting to RGB in a fragment shader.

In general, you should be able to render 720p without issue. 1080p is also possible with a little more fine-tuning.

Edit: OpenTK.Video would be great! Are you planning to release your source code for video playback?

jdomnitz's picture

Keeping the audio and video properly synced is going to be pretty difficult...i guess you keep a running stopwatch for how long it takes to render the frame and use that to update video latency.
A few issues here:
* I need this to work on very low powered hardware (typically embedded devices) which have decent graphics chipsets but rather slow CPUs
* I need this to work with pretty much OpenGL 1.3 or higher-so while PBOs and Fragment shaders are great ways to speed it up...it still needs to perform well without them
* End users are going to expect similar performance to WMP or other video players

A few more questions:
* How widespread are YUV extensions so a fragment shader wouldn't be needed (similar to http://www.mesa3d.org/MESA_ycbcr_texture.spec)?
* How sure are you that 720p is possible? On what kind of hardware?
* How much slower will this be then DXVA?

the Fiddler's picture

How widespread are YUV extensions so a fragment shader wouldn't be needed (similar to http://www.mesa3d.org/MESA_ycbcr_texture.spec)?

My nvidia card does not support this extension, so it is probably not very widespread.

If your hardware supports ARB_imaging, you can convert from YUV to RGB using a color matrix. The conversion will (probably) be done on the CPU, but it should be relatively fast.

Maybe Media Framework can perform the conversion for you?


How sure are you that 720p is possible? On what kind of hardware?

The XBMC media player uses OpenGL exclusively on Linux. I have been able to play 720p on 2GHz Athlon64 and a Nvidia 7600 video card and 1080p with a Nvidia 9500 (which uses a hardware decoder).


How much slower will this be then DXVA?

Sorry, I do not know the answer to that.

jdomnitz's picture

Got it....ok I can start playing around with that route for video and dvd playback. Unfortunately for protected media thats not an option....so is there a way to run an application fullscreen without disabling the DWM? Letting the DWM handle mixing may actually be faster and less CPU intensive since everything remains hardware accelerated.

the Fiddler's picture

Do fullscreen windows disable the DWM? This doesn't happen on my test machines (Ati/Nvidia/Intel).

I'm not sure what your exact goal is: are you trying to create a full-scale media player (in which case where does OpenTK come in?) or are you trying to display video inside an OpenGL application?

In the latter case, this might work:

  1. create a regular (win32), borderless (WS_POPUP), maximized window
  2. set the GameWindow as its parent
  3. render video to that window

If you need to support systems without DWM (e.g. XP), you could create a topmost (not child) window and hide the GameWindow for the duration of the video.

I haven't tested this but it looks plausible.

jdomnitz's picture

Are you sure it doesn't? The DWM will disable itself when in fullscreen and re-enable itself when the window exits fullscreen. According to microsoft this happens anytime a device draws directly to the primary surface. (ref: http://msdn.microsoft.com/en-us/library/aa969538(VS.85).aspx).

When I go fullscreen I am unable to get any windows to display over the fullscreen window. Even set to topmost with the gamewindow set as its parent, nothing displays until I go back to window mode.

I need to display video within a region of the openGL app or fullscreen depending on the users selection. For XP I can just disable rendering under the video window but for vista+ I need the DWM enabled.

A ha....according to here http://www.opengl.org/pipeline/article/vol003_7/ you need to use the PFD_SUPPORT_COMPOSITION flag to keep the DWM enabled. I don't see a WGL equivalent though

second edit:
I managed to keep the DWM enabled by embedding a window before going fullscreen but if I try embedding a window while already fullscreen it fails to display. The other problem is that when the embedded window gains focus the taskbar pops through and screws everything up. Microsoft really screwed things up....that taskbar bug thats been unfixed since vista is proving to be a real PITA. I'm kind of at a loss here.

jdomnitz's picture

Ignore the previous post....I actually saw no effect using the PFD_SUPPORT_COMPOSITION flag-although PFD support in general caused issues. I managed to get everything working while still maintaining a fully hardware accelerated pipeline (1080p video at ~10% CPU inluding the openGL drawing). I embedded a top most window in to my opengl window (which is also set to top most), positioned it and then set that as the target for video rendering. Based on the artifacts when dragging that top window it appears compositing is disabled. However, video playback succeeds above my opengl window and the opengl window continues to render underneath it. CPU usage seems to indicate everything is being done in hardware and somehow working so I guess I can't complain.

For anyone interested in code:

Its in rough shape atm but it shows ffdshow and mediafoundation playback examples with multi instance support.