Iluvatar's picture

Clipboard to Texture?

Hi there!

I have a big problem, because I need to get my rendered WPF-window into a texture somehow.
I have already mastered the part where I render a control (my WPF-window) to a RenderTargetBitmap!
The I use this RenderTargetBitmap and Copy it to the Clipboard.

How is it possible to go on from that point?

Or, if this does not work the easy way, is it possible to create a "Bitmap" or an "Image" out of the Clipboard Data?

Thank you very much :)
Iluvatar


Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Iluvatar's picture

Ok I found a solution to the problem, but I almost instantly got another one :) (as usual)

I need to do the following,
I have to implement this:
Context Preserving Visual Links - PDF

I use a WPF UI to draw all elements of the user-interface.
I already (almost) created the functionality to display different views of one of the hard-drives in the computer, I only need to implement the OpenGL-part of the paper linked above.

So I created a WinFormsHost object and inserted my "CustomGLControl" as described in the GraphicsContext - Class description. It "lies" above the other user-interface - elements (zIndex = 2 for the WinFormsHost)
How is it possible to add the (Custom)GLControl without "visually" interfering with my program-user-interface?
I tried this (and in the designer it is not shown at all, but the whole program is "grey", if I start it):

        <WindowsFormsHost Grid.Column="0" Grid.ColumnSpan="3" Grid.Row="0" Grid.RowSpan="2" Name="overlayHost" Panel.ZIndex="2" IsHitTestVisible="False" Background="Transparent">
            <gl:CustomGLControl x:Name="overlayCanvas" BackColor="Transparent" ForeColor="Transparent"></gl:CustomGLControl>
        </WindowsFormsHost>

Or maybe is it the wrong approch all along?
If you don't want to read the paper, this is, what i try to do:

I have some parts of the screen that should be "highlighted" and connected through links (lines on the screen).
The algorithm is image-based, so I need to get a Screenshot of the monitor/program window (whatever) and do some pomputations based on that screenshot.
One thing is to calculate a "penalty-map" based on color differences for eample!

Using this penalty-map i need to compute the links between the highlighted positions on the screen.

So I though I would use OpenGL for those computations.... Or do you think there's an easier way to do just that??? (using a WPF-Canvas element for example???)

Thank's for the help!!
Iluvatar

Iluvatar's picture

Or maybe asked differently:

  • 1st idea: Is it possible to make the GLControl transparent, and only "colored" if I paint something on it (and the clearing etc...)
  • 2nd idea: I initialize OpenGL WITHOUT ANY panel and control, then I use a WPF - Canvas (which I can set transparent, I already tested it...), then use a Control-less-OpenGL-System to do all calculations, and then draw the results onto that transparent WPF - Canvas (I think that one should work...)

What do you think?
Initializing OpenGL without control, then doing the calculations (in fact render to a texture at the end) and then use the result as (partly) transparent texture for the canvas control??

Thank's for any advise! :)
Ilu

Inertia's picture

I've taken the liberty to correct the link to the PDF in your post above, it didn't show.

There is no need to show the OpenGL window in order to render to it, but there must be a context present (need framebuffer and function pointers etc.). Also PBO and FBO Extensions can be used for offscreen rendering.

Another method to get the picture from WPF to OpenGL may simply be write it on the harddisk or some globally accessible public static Queue<Image> RawInput;

I haven't said anything to this yet, because I don't really see why they use OpenGL (besides speed) you can also implement this on the CPU with .Net 4 parallelism, it will probably be not as fast as the GPU version but it will be easier to tweak/adjust/improve (and most of all debug) the technique until it becomes what you want. When you are done with all features you want to add, then you can still port it to OpenGL or CL.

Iluvatar's picture

Thank you for your reply (and editing the Link - but it worked for me, I tested it...)!

I did some things in the meantime:

  • I was able to create a rendering-context without any GL-Control. Key to success here was this (in a static Constructor of my OpenGLHandler):
                m_viewPortSize = new Size();
                window = new NativeWindow();
                m_glContext = new GraphicsContext(new GraphicsMode(new ColorFormat(32), 24, 8, 4),
                    window.WindowInfo, 3, 2, GraphicsContextFlags.ForwardCompatible);
                m_glContext.MakeCurrent(window.WindowInfo);
                (m_glContext as IGraphicsContextInternal).LoadAll();

    I totally forgot about the last line!!!

  • I already set up the projection (just Ortho) and the Viewport according to the Windowsize
  • I rewrote my TextureHandler from another Project to fit this Project
    • So creating a Framebuffer and attaching a ColorAttachment is ok
    • This includes Texture-Creation from a Bitmap (which I already have, from the Clipboard) and also
    • Creating a Bitmap from a Texture!

I hope the last two points work as expected, because I didn't have the time to test yet.

So the Steps include (in the End :) ):

  1. Doing some Visualization of the HDD-usage (Folder, Files and Extensions with Sizes etc..., and a TreeMap)
  2. Then creating some "Highlighted" Areas on the Window (where the User clicked - and related Areas in the Window)
  3. Grabbing a Screenshot of the Window
  4. Doing the Computations using OpenGL in Shaders (see Paper)
  5. Save the final Result (=Texture) to a Bitmap
  6. Use a WPF Canvas Control to display the Bitmap (created from the OpenGL Texture)

Do you think this approch is ok or would YOU prefer the .NET variant?
Iluvatar

Inertia's picture

If you want to change it in some way, like drawing proper splines or allow the user to load some kind of layout texture which is used to further protect regions of the image, using GL is already an optimization and will make changing/debug things alot harder. (may break some assumptions the optimization is based on)
If it is exactly what you want and you don't need to change a bit, implement it with OpenGL/CL (for the benefit of speed).

Iluvatar's picture

Another question related to this :)

When I grab a Screenshot, I get an "WriteableBitmap" (Syste.Windows.Media.WriteableBitmap). My Texture Creation only allows "old" Bitmaps (System.Drawing.Bitmap)....

I already searched the Web for a Conversion but I wasn't able to find a solution yet.
Maybe you could tell me, how I can "convert" the WriteableBitmap" to an older "Bitmap" or maybe you can come up with a hit, how to rewrite my Constructor??

Here is the "Texture" - Constructor using a Bitmap :) :

public Texture(Bitmap bitmap, String name = "", TextureMinFilter minFilter = TextureMinFilter.LinearMipmapLinear, TextureMagFilter magFilter = TextureMagFilter.Linear)
{
    ///Set the Values
    Name = name;
    m_minFilter = minFilter;
    m_magFilter = magFilter;
 
    ///Generate a Handle for the Texture
    ID = GL.GenTexture();
 
    ///Bind the Texture
    GL.BindTexture(TextureTarget.Texture2D, ID);
 
    try
    {
        ///Set the Bitmap at first
        m_bitmap = bitmap;
 
        ///Set the Texture Informations
        m_data = m_bitmap.LockBits(new Rectangle(0, 0, m_bitmap.Width, m_bitmap.Height), ImageLockMode.ReadOnly, System.Drawing.Imaging.PixelFormat.Format32bppArgb);
        GL.TexImage2D(TextureTarget.Texture2D, 0, PixelInternalFormat.Rgba, m_data.Width, m_data.Height, 0,
            OpenTK.Graphics.OpenGL.PixelFormat.Bgra, PixelType.UnsignedByte, m_data.Scan0);
        m_bitmap.UnlockBits(m_data);
 
        ///Generate the Mipmaps
        GL.GenerateMipmap(GenerateMipmapTarget.Texture2D);
 
        ///Clamp the Texure to prevent Texture flickering at the outer Faces of the Facade
        GL.TexParameter(TextureTarget.Texture2D, TextureParameterName.TextureWrapS, (int)TextureWrapMode.ClampToEdge);
        GL.TexParameter(TextureTarget.Texture2D, TextureParameterName.TextureWrapT, (int)TextureWrapMode.ClampToEdge);
 
        ///Nearet "Interpolation" (== no Interpolation at all)
        GL.TexParameter(TextureTarget.Texture2D, TextureParameterName.TextureMinFilter, (int)m_minFilter);
        GL.TexParameter(TextureTarget.Texture2D, TextureParameterName.TextureMagFilter, (int)m_magFilter);
 
        if (GL.GetError() != ErrorCode.NoError)
        {
            ///send the error-messages
            Logger.SendToLog("Texture creation using a Bitmap caused an error: \"" + GL.GetError().ToString() + "\"", LogType.Errors);
        }
    }
    catch
    {
        ///send the error-messages
        Logger.SendToLog("Texture creation caused an error: \"" + GL.GetError().ToString() + "\"", LogType.Errors);
    }
}

Thank you!

Inertia's picture

From http://msdn.microsoft.com/en-us/library/system.windows.media.imaging.wri...

  1. Call the Lock method to reserve the back buffer for updates.
  2. Obtain a pointer to the back buffer by accessing the BackBuffer property.
  3. Write changes to the back buffer. Other threads may write changes to the back buffer when the WriteableBitmap is locked.
  4. Call the AddDirtyRect method to indicate areas that have changed.
  5. Call the Unlock method to release the back buffer and allow presentation to the screen.

Ignore step 3&4, the IntPtr from BackBuffer can be fed to GL.TexImage2D. (examine Format property for other params)

Iluvatar's picture

Oh thank you!
I am going to try it when I am home again!
I'll keep you posted!

BTW:
Thank you for the support all the time!! :)!!

Iluvatar's picture

Ok, I tried it out now,

I am able to create a Texture now, but I think my Pixel-Format was not right (was it??)!
When I saved the WriteableBitmap and the Bitmap to Disc I found out, that the Texture was in fact... black!
What I did, was exactly what you mentioned:

  1. Locked the WriteableBitmap
  2. Used the BackBuffer - Pointer and several other Properties from the WriteableBitmap to Set the Values of my BitmapData Object (which is used to Create the GLTexture2D)
  3. Unlocke the WriteableBitmap again
  4. Default .... MipMaps... Clamping....

Is it possible, that my Code for Reading the Pixels is not ok? I mean, I it is there, to just read Pixels from the actually bound FrameBuffer right (THEN I understand, why I only get black Images :) )?

Here is the Code for creating a Bitmap and for saving to a File (or is not working this way at all?)

        /// <summary>
        /// Returns a Bitmap with the contents of the current FrameBuffer
        /// </summary>
        /// <returns>The Bitmap containing the Data</returns>
        private Bitmap GetGLBitmapData()
        {
            ///Create the Bitmap
            m_bitmap = new Bitmap(Width, Height);
            m_data = m_bitmap.LockBits(new Rectangle(0, 0, Width, Height), ImageLockMode.WriteOnly, m_data.PixelFormat);
            GL.ReadPixels(0, 0, Width, Height, m_pixelFormat, PixelType.UnsignedByte, m_data.Scan0);
            m_bitmap.UnlockBits(m_data);
 
            ///Return the Bitmap
            return m_bitmap;
        }
        /// <summary>
        /// Save the Content of the FrameBuffer to a File
        /// </summary>
        public void SaveGLImage(String filename = "E:\\textureImage.png")
        {
            GetGLBitmapData().Save(filename, ImageFormat.Png);
        }

BUT: I found a way to create a System.Drawing.Bitmap out of a WriteableBitmap! It's only a hack, but for now it works :)

Here is the Code (implemented as an Extension and using a MemoryStream):

using System.Drawing;
using System.Windows.Media.Imaging;
using System.IO;
 
namespace Helper
{
    public static class ExtensionsHelper
    {
        /// <summary>
        /// Create Bitmap from a WriteableBitmap
        /// </summary>
        /// <param name="wrBitmap">The System.Media.Imaging.WriteableBitmap</param>
        /// <returns>The System.Drawing.Bitmap</returns>
        public static Bitmap GetBitmap(this WriteableBitmap wrBitmap)
        {
            using (MemoryStream mS = new MemoryStream())
            {
                PngBitmapEncoder encoder = new PngBitmapEncoder();
                encoder.Frames.Add(BitmapFrame.Create(wrBitmap));
                encoder.Save(mS);
                Bitmap result = new Bitmap(mS);
                mS.Close();
                return result;
            }
        }
    }
}

When I try to create a Bitmap using my "GetGLBitmapData()" , it doesn't show anything (only a black Image), but I think my Texture is well, because I just call my (well working and tested ) Constructor....
And if I save the converted System.Drawing.Bitmap to the Disc, the Image looks ok :)

I will continue, to tell you, if the Texture is ok :)

Iluvatar's picture

So I tried many different possibilities to access the Texture.
I have no Problem saving the Screenshot to a File (through a Bitmap), but I also need to create a Bitmap out of another Texture...

The other Texture is the "ColorAttachment0" of a FramebufferObject!

Well, for now, I do the following:

Inizialize my FBO and Color-Attachment (WORKS, this is also tested!) and I grab the Screenshot and create another Texture!
I use a simple GLSL Shader to just pass the Color-Values for now onto a "ScreenAlignedQuad"

I make the Conext current
Setup the Viewport
Bind the Framebuffer
Use the right Shader
Bind the ScreenShotTexture (input for the Shader)
Render the ScreenAlignedQuad
Unbind the Objects again.

-> At this Point I want to Create a Bitmap out of the Color-Attachment0 of the FBO...
All I get is a black Texture all the time...

i used the "BitmapData" and a Bitmap.Lock() to achive this but It doesn't work...
The Code works as described above!
I also tried to do something like this instead:

               return new Bitmap(m_data.Width, m_data.Height, m_data.Stride, m_data.PixelFormat, m_data.Scan0);
and I also tried to get the Bitmap-Data like this:

                GL.GetTexImage(TextureTarget.ProxyTexture2DMultisample, 0, m_pixelFormat, PixelType.UnsignedByte, m_data.Scan0);

But nothing works.... (tried all, when the FBO was still bound, or not)

I never get any Errors, not when I create/bind/use/... the Shader/FBO/Texture... all seems ok.
Anybody any suggestions?