OpenTKstart's picture

Relation from 2D to 3D (Screen to Model)

Hey,

i use a GameWindow to draw Quads. Now i want to select a single point of the drawn graphic by the mouse. But i have no idea how to get a relation from 2d-screen to the 3d-model. Maybe this picture will explain it more...

thx


Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
mOfl's picture

The technique you are looking for is called "picking", you will find a lot of sample code on the internet about this topic. There are several approaches to do picking, but the correct way is to implement ray casting such that the ray casted through the pixel of your image plane intersects the object you want to select. There are also workarounds using the depth buffer so that you don't have to implement real ray casting. For a beginning you might take a look at the OpenTK examples, because there is an example called "Picking" which allows you to select triangles of a mesh with the mouse. I don't know how picking is realized in this example, but you'll get some ideas for your application for sure. Also you have to keep in mind that your task is not as trivial as usual object picking because you want to select points/vertices, which is a whole different matter because vertices are usually not present for such computation as they are stored on the video memory. Also, vertices do not have a size so it would not be possible to intersect them with a ray in a suitable way. You might think of representing each vertex of your mesh as an invisible cube and do object picking for this invisible cube instead. Maybe the search term "vertex picking" leads to some sample code, but that's a task beyond the trivial.

OpenTKstart's picture

Hey, thanks for the answer. I read this :
http://opentk.svn.sourceforge.net/viewvc/opentk/trunk/Source/Examples/Op...

but i dont understand everything.

1. Line 103-112 (dont understand what they do)
2. I dont use the method "GL.DrawArrays()". I draw a Array of "graphic-objects" which include GL.Vertex3.
Is it important to use this method?
3.

// Read Pixel under mouse cursor
254 	Byte4 Pixel = new Byte4();
255 	GL.ReadPixels(Mouse.X, this.Height - Mouse.Y, 1, 1, PixelFormat.Rgba, PixelType.UnsignedByte, ref Pixel);
256 	SelectedTriangle = Pixel.ToUInt32();
257 	#endregion Pass 1: Draw Object and pick Triangle 

i understood this, but why they proof this to uint.MaxValue?

263 	if (SelectedTriangle == uint.MaxValue)
264 	GL.ClearColor(.2f, .1f, .3f, 1f); // purple 

4. Can you please explain me the most important thing which i need to do, to select somthing. I thought about your hint to represent each Vertex as a invisible cube. Thats a good idea, i thing i will do this.

Thx

mOfl's picture
OpenTKstart wrote:

1. Line 103-112 (dont understand what they do)

They set up a Vertex Buffer Object. You actually don't use GL.Vertex* anymore, it's outdated and very slow compared to Vertex Buffer Objects where you upload the geometry and vertex attributes only once (and not each frame as with GL.Vertex*). For the picking problem, this has no relevance however, it is a general performance gain.

Quote:

2. I dont use the method "GL.DrawArrays()". I draw a Array of "graphic-objects" which include GL.Vertex3.
Is it important to use this method?

See 1. When using Vertex Buffer Objects, you only have to specify from which uploaded buffers you want to draw geometry. It does not affect picking, just makes everything faster. It is, of course, recommended to use this method. If you're at the beginning of learning OpenGL, I highly recommend you to learn and use the efficient methods from the beginning so you don't get used to the deprecated stuff. This includes Vertex Buffer Objects (VBOs), Vertex Array Objects (VAOs), and Uniform Buffer Objects (UBOs) - look them up on the internet, there's a lot of sample code, even for OpenTK. For Core Profiles above OpenGL 3.2, these constructs are mandatory.

Quote:

3.

// Read Pixel under mouse cursor
254 	Byte4 Pixel = new Byte4();
255 	GL.ReadPixels(Mouse.X, this.Height - Mouse.Y, 1, 1, PixelFormat.Rgba, PixelType.UnsignedByte, ref Pixel);
256 	SelectedTriangle = Pixel.ToUInt32();
257 	#endregion Pass 1: Draw Object and pick Triangle 

i understood this, but why they proof this to uint.MaxValue?

263 	if (SelectedTriangle == uint.MaxValue)
264 	GL.ClearColor(.2f, .1f, .3f, 1f); // purple 

This picking example works in two passes - in the first pass, each triangle is drawn with a unique color which corresponds to a triangle index. Thus, with ReadPixels(), the color of the pixel at the mouse position corresponds to a triangle - that's quite a simple and dirty method of implementing picking, but for this example it's sufficient. The pixel color is converted to an uint value while the background is cleared with 1, 1, 1, 1, which converted to uint is uint.MaxValue. The comparison to uint.MaxValue is therefore a check if a triangle was selected or if the mouse was over the background. If the latter was the case, the background is cleared with a purple color, else with cyan. Then, the triangles are drawn again in white and the selected triangle (according to its triangle ID) is drawn in green.

Quote:

4. Can you please explain me the most important thing which i need to do, to select somthing. I thought about your hint to represent each Vertex as a invisible cube. Thats a good idea, i thing i will do this.

As I said, there are several ways to do picking, but only one way to do it right - with ray casting. I don't think you want to do this. Thus, the hack in this example seems quite suitable for what you want to do. First thing you should do is to impleent a function that adds a cube around every (visible) vertex. Then look at how the colors are assigned to triangles in the OnLoad() function and try to modify it in such a way that the color coded triangle ID is now a vertex ID that is applied to the faces of your cube, so in the first render pass you'd draw uniquely colored cubes instead of triangles. Then, modify the second pass of the example such that the color lookup outputs the cube to be selected.

OpenTKstart's picture

Thanks a lot,

i tried to implement it now and got furthermore questions^^:

Line 79-101:

I think they fill the array with triangle-data which they want to draw there. But i cant find the class in line 81 " VertexT2fN3fV3f".

And is this part important even when i just want to draw quads:

 // Convert from temp mesh to final object, copy position and add triangle Ids for the color attribute.
89	            VBO_Array = new Vertex[temp_VBO.Length];
90	            int TriangleCounter = -1;
91	            for (int i = 0; i < temp_VBO.Length; i++)
92	            {
93	                // Position
94	                VBO_Array[i].Position = temp_VBO[i].Position;
95	
96	                // Index
97	                if (i % 3 == 0)
98	                    TriangleCounter++;
99	                VBO_Array[i].Color = new Byte4(BitConverter.GetBytes(TriangleCounter));
100	            }

Line 107: GL.InterleavedArrays(InterleavedArrayFormat.C4ubV3f, 0, IntPtr.Zero);
what says the InterleavedArrayFormat about my structure?

Greets and Happy New Year!

mOfl's picture
OpenTKstart wrote:

Line 79-101:

I think they fill the array with triangle-data which they want to draw there. But i cant find the class in line 81 " VertexT2fN3fV3f".

With the OpenTK code (you can checkout the SVN repository on the download page) there comes the code for the OpenTK samples - picking included. There are also some auxiliary classes like the SierpinskiTetrahedron that handle mesh creation and storage (using a base class DrawableShape). Another file, Examples/Shapes/VertexStructs.cs, defines some basic structs which define a vertex with some its attributes. The definition for VertexT2fN3fV3f (you can just copy that into your code file so you don't have to look for the VertexStructs.cs file yourself) is:

    public struct VertexT2fN3fV3f
    {
        public Vector2 TexCoord;
        public Vector3 Normal;
        public Vector3 Position;
    }

And yes, you are right, in this section of the code the arrays and Vertex Buffer Object for drawing are set up. A hopefully up-to-date version of the file DrawableShape where the actual buffer creation takes place can be found here to take a look at the implementation: http://www.opensourcejavaphp.net/csharp/opentk/DrawableShape.cs.html

Quote:

And is this part important even when i just want to draw quads:

Yes it is, as this has nothing to do with the type of the primitives you want to draw. "TriangleCounter" is just a name, you'd rename it to "QuadCounter" in your application and change the modulo operation from if (i % 3 == 0) to if (i % 4 == 0) as you want to increment the counter every 4 vertices. You just need to make sure that you set up your index array for drawing right - you need to pass 4 instead of 3 indices per primitive.

Quote:

Line 107: GL.InterleavedArrays(InterleavedArrayFormat.C4ubV3f, 0, IntPtr.Zero);
what says the InterleavedArrayFormat about my structure?

I am very sorry to tell you this again, but this call is deprecated in OpenGL 3.0 Core Profile as well, so better don't get used to it. To get a basic unterstanding of it: instead of specifying the required input in your vertex shader yourself, you could choose out of predefined attribute set arrays for your VBO where the attributes are interleaved, i.e. the attributes of each vertex are packed together and then stored in an array sequentially. C4ubV3f means that the attribute set for one vertex consists of a Color with the size 4 unsigned bytes (RGBA values) and a Vertex position with 3 float values (x, y, z). It would also be possible to add texture coordinates, there were constants for that, too.

OpenTKstart's picture

Problem1:

 protected override void OnRenderFrame(FrameEventArgs e)
        {
          //  base.OnRenderFrame(e);
 
            GL.Color3(Color.White);
	        GL.Enable(EnableCap.ColorArray);
 
            GL.ClearColor(1f, 1f, 1f, 1f);
            GL.Clear(ClearBufferMask.ColorBufferBit | ClearBufferMask.DepthBufferBit);
 
            GL.MatrixMode(MatrixMode.Modelview);
            GL.LoadMatrix(ref modelview);
 
            GL.Scale(scaleX, scaleY, scaleZ); // scale to zoom
 
            GL.Translate(transformation); // translation to move
            GL.Rotate(angleY, Vector3d.UnitY);  //rotation on Y-achis
            GL.Rotate(angleX, Vector3d.UnitX);  //rotation on X-achis
            GL.Rotate(angleZ, Vector3d.UnitZ);  //rotation on Z-achis
 
            //  angleX += 0.5f;  // TEST: auto-rotate
            if (ready)
            {
                #region Pass 1: Draw Object and pick Triangle
                // You may re-enable the shader, but it works perfectly without and will run on intel HW too
                // GL.UseProgram(ProgramObject);
                GL.DrawArrays(VBO_PrimMode, 0, VBO_Array.Length);
                // GL.UseProgram(0);
 
                // Read Pixel under mouse cursor
                Byte4 Pixel = new Byte4();
                GL.ReadPixels(Mouse.X, this.Height - Mouse.Y, 1, 1, PixelFormat.Rgba, PixelType.UnsignedByte, ref Pixel);
                SelectedTriangle = Pixel.ToUInt32();
                #endregion Pass 1: Draw Object and pick Triangle
 
                GL.Color3(Color.White);
                GL.Disable(EnableCap.ColorArray);
 
                #region Pass 2: Draw Shape
                if (SelectedTriangle == uint.MaxValue)
                    GL.ClearColor(.2f, .1f, .3f, 1f); // purple
                else
                    GL.ClearColor(0f, .2f, .3f, 1f); // cyan
                GL.Clear(ClearBufferMask.ColorBufferBit | ClearBufferMask.DepthBufferBit);
 
                GL.Color3(1f, 1f, 1f);
                GL.DrawArrays(VBO_PrimMode, 0, VBO_Array.Length);
 
                GL.PolygonMode(MaterialFace.FrontAndBack, PolygonMode.Line);
                GL.Color3(Color.Red);
                GL.DrawArrays(VBO_PrimMode, 0, VBO_Array.Length);
                GL.PolygonMode(MaterialFace.FrontAndBack, PolygonMode.Fill);
 
                if (SelectedTriangle != uint.MaxValue)
                {
                    GL.Disable(EnableCap.DepthTest);
                    GL.Color3(Color.Green);
                    GL.DrawArrays(VBO_PrimMode, (int)SelectedTriangle * 3, 3);
                    GL.Enable(EnableCap.DepthTest);
                }
                #endregion Pass 2: Draw Shape
            }
         //   drawFramework();  // draws the Graphic-Framework such as X,Y,Z-Achis´s
         //   drawAllObjects(); // draw all object in the graph-list
 
            //Show FPS=======
            this.Title = "OpenTK-GameWindow - FPS: " + RenderFrequency.ToString();
            //===============
            SwapBuffers();
 
            ErrorCode err = GL.GetError();
            if (err != ErrorCode.NoError)
	                Trace.WriteLine("Error at Swapbuffers: " + err);
        }
//...prepare/fill Vertex in  VBO_Array (totally 20160)
 
 #region prepare data for VBO from procedural object
 
            int QuadCounter = -1;
            for (int j = 0; j < VBO_Array.Length; j++)
            {
                // Position
                //  same
 
                // Index
                if (j % 4 == 0)
                    QuadCounter++;
                VBO_Array[j].Color = new Byte4(BitConverter.GetBytes(QuadCounter));
            }
            #endregion prepare data for VBO from procedural object
 
            #region Setup VBO for drawing
            GL.GenBuffers(1, out VBO_Handle);
            GL.BindBuffer(BufferTarget.ArrayBuffer, VBO_Handle);
            GL.BufferData<Vertex>(BufferTarget.ArrayBuffer, (IntPtr)(VBO_Array.Length * Vertex.SizeInBytes), VBO_Array, BufferUsageHint.StaticDraw);
            GL.InterleavedArrays(InterleavedArrayFormat.C4ubV3f, 0, IntPtr.Zero);
 
            ErrorCode err = GL.GetError();
            if (err != ErrorCode.NoError)
                Trace.WriteLine("VBO Setup failed (Error: " + err + "). Attempting to continue.");
            #endregion Setup VBO for drawing
           ready=true;

But i dont see the Object? The coordinates are correct, i used them before...

OpenTKstart's picture

Problem 2 (this is less important)

i get a Exception:
the problem is from this line, but i dont know why. I used this GraphicMode for better GraphicQuality...
static GraphicsMode mode = new GraphicsMode(new OpenTK.Graphics.ColorFormat(32), 24, 0, 2, new OpenTK.Graphics.ColorFormat(32));

StackTrace: http://www.file-upload.net/download-4002160/OpenTK_start.vshost_120104_1...

mOfl's picture

How do you know your second problem comes from your GraphicsMode settings? Does it only occur when you change the GraphicsMode settings? To me it looks like a regular Exception thrown when the wrong part of video memory is addressed in a VBO. As I said, you have to make sure to convert the existing codes to your needs as the example works with triangles but you want to draw quads. Hence, you have to upload an additional vertex index for each quad to your buffer object. First of all, tell me what value the variable VBO_PrimMode has - I expect it to be Triangle which would be wrong in your case. Second, the line where the access violation is thrown should be
GL.DrawArrays(VBO_PrimMode, (int)SelectedTriangle * 3, 3);

GL.DrawArrays(mode, first, count) allows you to draw only a portion of primitives instead of drawing all of them, starting from the index position first and forming primitives of the type mode out of the next count vertices of the active buffer. So in your case, to draw one single quad (I renamed the variable "SelectedTriangle" to "SelectedQuad"), you would write:

GL.DrawArrays(VBO_PrimMode, (int)SelectedQuad * 4, 4);

where VBO_PrimMode is BeginMode.Quads. In the sample code, the DrawableShape class is used which sets PrimitiveMode = BeginMode.Triangles;, so you have to set it to Quads manually where you need it. Is there a reason you want to work with quads in the first place? As you do not seem to have solid knowledge of OpenGL, I'd recommend you to stay with the triangle version of the sample code, so you know everything there is working fine. It should be easy to modify your existing quad drawing code to draw triangles.

This should not fix the Access Violation Exception, however. Please tell me what value the variable SelectedTriangle/SelectedQuad has when the exception occurres.

OpenTKstart's picture

Thats right that the problem only occure when i use the selfmade graphmode and only when i go with the mouse over some triangles.
The Value from SelectedTriangle when the problem occurres is 3217015807. When i use GraphicMode.Default everything works fine, thats the matter why i tried to draw quads now. My Tool is just made to draw quads or better cubes which are made of quads, thats why it´s important to draw quads.

GL.DrawArrays(mode, first, count) allows you to draw only a portion of primitives instead of drawing all of them
Now i understand this, this is the part which draw and overlay the triangle which is selected. In first case i dont need this, because i dont want to select the side of the cubes as more i want to select later the invisible cubes which wrap the Corrd-Points/Vertex.

Thx

mOfl's picture
OpenTKstart wrote:

Thats right that the problem only occure when i use the selfmade graphmode and only when i go with the mouse over some triangles.

The example works with the conversion of color to byte values, so this could be a problem when you define a color mode which is not 32 byte (4 * float).

Quote:

The Value from SelectedTriangle when the problem occurres is 3217015807.

How many quads do you draw, roughly? I think this shouldn't even be possible with the example algorithm to get such a high number. The value 3217015807 is BFBFC3FF in hexadecimal, which is a color, a pale blue, and not a triangle/quad identifier. The identifiers should start with 00000000, the highest id should be the number of your quads - 1. As you use the identifier to access the VBO (the vertices in it), you have to make sure to retrieve the id correctly.

Quote:

When i use GraphicMode.Default everything works fine, thats the matter why i tried to draw quads now.

What do you mean with "everything works fine"? The picking is working?