t7.reyeslua's picture

How to get 3D coordinates from glControl by clicking or selecting with the mouse??

Hello :)

I'm doing some 3d plotter with OpenTK and I need the user to be able to select each individual plotted point and know the coordinates of it. Obviously I know the 3Dcoordinates of all points because that's how i plot them. What I'm not sure how to do is to get the coordinates FROM the glControl with is what the user interacts with. It would be nice if the user could click some point in the glcontrol and know its coordinates. Any ideas/examples of how to do that?

Thanks!! :D


Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
iliak's picture

Look for project/unproject method.

Google is your friend !

t7.reyeslua's picture

Hey....thanks for the quick response! :) ...I've been reading about it and that's exactly what I need! The problem is that I can't seem to find the method anywhere in the OpenTK version I'm using (v1.0). Idon't know if it has not been ported to this version or what. I'm no openGL/TK expert, so if someone could point me where/how to find it, it would be great! :D Thanks again.

EDIT: I'm gonna try with OpenTK.Compatibility where I suppose it is.

iliak's picture

From XNA Viewport :

Projects a 3D vector from object space into screen space. Reference page contains links to related code samples.
Parameters
source
The vector to project.
projection
The projection matrix.
view
The view matrix.
world
The world matrix.
 
public Vector3 Project(Vector3 source, Matrix projection, Matrix view, Matrix world)
{
    Matrix matrix = Matrix.Multiply(Matrix.Multiply(world, view), projection);
    Vector3 vector = Vector3.Transform(source, matrix);
    float a = (((source.X * matrix.M14) + (source.Y * matrix.M24)) + (source.Z * matrix.M34)) + matrix.M44;
    if (!WithinEpsilon(a, 1f))
    {
        vector = (Vector3) (vector / a);
    }
    vector.X = (((vector.X + 1f) * 0.5f) * this.Width) + this.X;
    vector.Y = (((-vector.Y + 1f) * 0.5f) * this.Height) + this.Y;
    vector.Z = (vector.Z * (this.MaxDepth - this.MinDepth)) + this.MinDepth;
    return vector;
}
 
 
 
Converts a screen space point into a corresponding point in world space.
Parameters
source
The vector to project.
projection
The projection matrix.
view
The view matrix.
world
The world matrix.
public Vector3 Unproject(Vector3 source, Matrix projection, Matrix view, Matrix world)
{
    Matrix matrix = Matrix.Invert(Matrix.Multiply(Matrix.Multiply(world, view), projection));
    source.X = (((source.X - this.X) / ((float) this.Width)) * 2f) - 1f;
    source.Y = -((((source.Y - this.Y) / ((float) this.Height)) * 2f) - 1f);
    source.Z = (source.Z - this.MinDepth) / (this.MaxDepth - this.MinDepth);
    Vector3 vector = Vector3.Transform(source, matrix);
    float a = (((source.X * matrix.M14) + (source.Y * matrix.M24)) + (source.Z * matrix.M34)) + matrix.M44;
    if (!WithinEpsilon(a, 1f))
    {
        vector = (Vector3) (vector / a);
    }
    return vector;
}
 
private static bool WithinEpsilon(float a, float b)
{
    float num = a - b;
    return ((-1.401298E-45f <= num) && (num <= float.Epsilon));
}
 
 
 
 

Hope it will help...

LikeKT's picture

Q1) Anyone has codes howing how to efficiently pick triangle of a mesh object picked by a ray cast using OpenTK unproject? I have looked into the picking example of Opentk, not sure how that is related to Unproject of GL.
Q2) is it possible to get local coordinate space from this method to calculate texture coordinates associated with that interception point?

t7.reyeslua's picture

Hey!

I finally got it :) ... I post here what it worked for me so if someone else needs to do something similar it can be found here. Actually I took most of it from http://www.opentk.com/node/480 and just made small changes.

In my code I have a 3D coordinate system where I plot some points. The user must check a checkbox to be allowed to "select" points by clicking them.

private void glControl1_MouseClick(object sender, System.Windows.Forms.MouseEventArgs e)
        {
            if (!loadedTK) return; // Play nice   
 
            int[] viewport = new int[4];
            double[] modelViewMatrix = new double[16];
            double[] projectionMatrix = new double[16];
 
            if (checkBoxSelectPoints.Checked == true)
            {
                int mouseX = e.X;
                int mouseY = e.Y;
 
                //Get Matrix
                OpenTK.Graphics.OpenGL.GL.GetInteger(OpenTK.Graphics.OpenGL.GetPName.Viewport, viewport);
                OpenTK.Graphics.OpenGL.GL.GetDouble(OpenTK.Graphics.OpenGL.GetPName.ModelviewMatrix, modelViewMatrix);
                OpenTK.Graphics.OpenGL.GL.GetDouble(OpenTK.Graphics.OpenGL.GetPName.ProjectionMatrix, projectionMatrix);
 
                //Calculate NearPlane point and FarPlane point. One will get the two end points of a straight line
                //that "almost" intersects the plotted point you "clicked".
                Vector3 win = new Vector3(mouseX, viewport[3] - mouseY, 0);   
                Vector3 worldPositionNear;
                Glu.UnProject(win, modelViewMatrix, projectionMatrix, viewport, out worldPositionNear);
                win.Z = 1.0f;
                Vector3 worldPositionFar;
                Glu.UnProject(win, modelViewMatrix, projectionMatrix, viewport, out worldPositionFar);
 
                //Calculate the lenght of the straigh line (the distance between both points).
                double distanceNF = Math.Sqrt(Math.Pow(worldPositionNear.X - worldPositionFar.X, 2) + 
                                              Math.Pow(worldPositionNear.Y - worldPositionFar.Y, 2)+
                                              Math.Pow(worldPositionNear.Z - worldPositionFar.Z, 2));
                double minDist = distanceNF;
 
 
                 //Calculate which of the plotted points is closest to the line. In other words,
                // look for the point you tried to select. Calculate the distance between the 2 endpoints that passes through
                // each plotted point. The one that is most similar with the straight line will be the selected point.
                int selectedPoint = 0;
                for (int i = 0; i < PointsInfo.Count; i++)
                {
                    double d1 = Math.Sqrt(Math.Pow(worldPositionNear.X - PointsInfo[i].Position.X, 2) +
                                          Math.Pow(worldPositionNear.Y - PointsInfo[i].Position.Y, 2) +
                                          Math.Pow(worldPositionNear.Z - PointsInfo[i].Position.Z, 2));
 
                    double d2 = Math.Sqrt(Math.Pow(PointsInfo[i].Position.X - worldPositionFar.X, 2) +
                                          Math.Pow(PointsInfo[i].Position.Y - worldPositionFar.Y, 2) +
                                          Math.Pow(PointsInfo[i].Position.Z - worldPositionFar.Z, 2));
 
                    if (((d1 + d2) - distanceNF) <= minDist)
                    {
                        minDist = (d1 + d2) - distanceNF;
                        selectedPoint = i;
                    }
                }
 
                //Just select/unselect points if the "click" was really close to a point. Not just by clicking anywhere in the screen
                if (minDist < 0.000065)
                {
                    if (selectedPoints.Contains(selectedPoint))
                        selectedPoints.Remove(selectedPoint);
                    else
                        selectedPoints.Add(selectedPoint);
 
                    glControl1.Invalidate();  //paint again 
                }
            }

In order to get the UnProject method I had to add the OpenTK.Compatibility.dll to my project. The problem was that by adding it there were some "conflicts" with the regular OpenTK.dll. That's why I had to specify in my whole code the "OpenTK.Graphics.OpenGL.GL" in the functions where I had before just "GL". It seems as if the Compatibility dll is just not so compatible after all :P

the Fiddler's picture

OpenTK.Compatibility is there to support projects using the Tao framework or older OpenTK versions. It's not really meant to be used outside of those circumstances (namespaces clashes will arise if you try to include namespaces from both OpenTK and OpenTK.Compatilibity in the same source file).

cengceng's picture

Hi t7.reyeslua

I want to do a project and it seems like your's. I am using Java and JOGL (OpenGL bindings for java)
It is like Opentk. I dont know about openttk I just search about getting world position from mouse click.
I used your algorithm for getting the correct values. And I got wrong values. Here is my code. Can you suggest something

  if(selected==true)
		 {
 
 
 
		 gl.glGetIntegerv(GL.GL_VIEWPORT, viewport, 0);
               gl.glGetDoublev(GL.GL_MODELVIEW_MATRIX, mvmatrix, 0);
             gl.glGetDoublev(GL.GL_PROJECTION_MATRIX, projmatrix, 0);
             /* note viewport[3] is height of window in pixels */
             realy = viewport[3] - (int) y ;
         System.out.println("Coordinates at cursor are (" + x + ", " + realy);
 
 
         FloatBuffer fb=ByteBuffer.allocateDirect(4).order(ByteOrder.nativeOrder()).asFloatBuffer();
 
         gl.glReadPixels(y, realy, 1, 1, GL.GL_DEPTH_COMPONENT, GL.GL_FLOAT, fb);
         fb.rewind();
 
 
 
 
         glu.gluUnProject((double)x,(double)realy,0,mvmatrix,0,projmatrix,0,viewport,0,wcoordnear,0);
         System.out.println("World coords at z=0.0 are ( " //
                 + wcoordnear[0] + ", " + wcoordnear[1] + ", " + wcoordnear[2]
                 + ")");
 
         glu.gluUnProject((double) x, (double) realy, 1.0, //
                 mvmatrix, 0,
                 projmatrix, 0,
                 viewport, 0, 
                 wcoordFar, 0);
             System.out.println("World coords at z=1.0 are (" //
                                + wcoordFar[0] + ", " + wcoordFar[1] + ", " + wcoordFar[2]
                                + ")");
 
             double distance=Math.sqrt( kare(wcoordnear[0]-wcoordFar[0])+kare(wcoordnear[1]-wcoordFar[1])+kare(wcoordnear[2]-wcoordFar[2]) );
 
             System.out.println("ilk distance "+distance);
             double mindist=distance;
             int selctedi=9999,selected=9999; // I use a grid to hold points 
             for (int i = 0; i < 9; i++)
             {
 
            	  for (int j = 0; j < 9; j++)
                  {
 
            		double d1=  Math.sqrt( kare(wcoordnear[0]-t[i][j][0])+kare(wcoordnear[1]-t[i][j][1])+kare(wcoordnear[2]-t[i][j][2]) );
 
            		double d2=  Math.sqrt( kare(wcoordFar[0]-t[i][j][0])+kare(wcoordFar[1]-t[i][j][1])+kare(wcoordFar[2]-t[i][j][2]) );
 
            		if (((d1 + d2) - distance) <= mindist)
                    {
            			mindist = (d1 + d2) - distance;
 
                       selectedi=i;
                     selectedj=j;
                    }
 
 
                  }
             }
 
             System.out.println(selectedi+" "+selectedj+" the point you select ");
 
 
 
 
      selected=false;
		 }
 
 
 
//like your codes pow(number,2)
public double kare(double a)
	{
		return a*a;
	}
Rogue's picture

Disclaimer: I may be 100% wrong about this...

...but I think you're in the wrong place. OpenTK is accessible by .Net languages, like C#, F#, VB.net and IronPython. There may be a few Java devs here, but it's not very likely that you'll get the help you need.

I did some googling for you and found this thread on Gamedev, which seems to be what you're looking for. Hopefully that helps!

xandemon's picture

I had to do this a little while back for my research project, basically I just ported the code from elsewhere. No need to use the compatibility dll, just write your own little UnProject() instead of using glu's one.

I used this is my glCtrl_MouseMove function, with a mouseDrag boolean variable set in glCtrl_MouseDown/glCtrl_MouseUp.

float scale;
Vector3 rotation;
Vector3 depthTestPoint;
private void glCtrlMain_MouseMove(object sender, MouseEventArgs e)
{
    if (mouseDrag)
    {
        switch (e.Button)
        {
            case MouseButtons.Left:
                // get point on mesh
                glCtrlMain.MakeCurrent();
                depthTestPoint = get2Dto3D(e.X, e.Y);
                depthTestPoint /= scale;
                depthTestPoint = Vector3.TransformVector(depthTestPoint,
                    Matrix4.CreateRotationX(MathHelper.DegreesToRadians(-rotation.X)) *
                    Matrix4.CreateRotationY(MathHelper.DegreesToRadians(-rotation.Y)) *
                    Matrix4.CreateRotationZ(MathHelper.DegreesToRadians(-rotation.Z))
                    );
        }
    }
}

This assumes your render code looks something like:

GL.Rotate(rotation.X, Vector3.UnitX);
GL.Rotate(rotation.Y, Vector3.UnitY);
GL.Rotate(rotation.Z, Vector3.UnitZ);
GL.Scale(scale, scale, scale);

...that way, these operations are reversed and you get the 3D position on the surface of the model for any other operations you want to do to it, which I've found rather useful for visualization purposes.

These are the functions that do the actual 'work':

private Vector3 get2Dto3D(int x, int y)
{
    int[] viewport = new int[4];
    Matrix4 modelviewMatrix, projectionMatrix;
    GL.GetFloat(GetPName.ModelviewMatrix, out modelviewMatrix);
    GL.GetFloat(GetPName.ProjectionMatrix, out projectionMatrix);
    GL.GetInteger(GetPName.Viewport, viewport);
 
    // get depth of clicked pixel
    float[] t = new float[1];
    GL.ReadPixels(x, camera.viewport.height - y, 1, 1, OpenTK.Graphics.OpenGL.PixelFormat.DepthComponent, PixelType.Float, t);
 
    return UnProject(new Vector3(x, viewport[3] - y, t[0]), modelviewMatrix, projectionMatrix, viewport);
}
 
private Vector3 UnProject(Vector3 screen, Matrix4 view, Matrix4 projection, int[] view_port)
{
    Vector4 pos = new Vector4();
 
    // Map x and y from window coordinates, map to range -1 to 1 
    pos.X = (screen.X - (float)view_port[0]) / (float)view_port[2] * 2.0f - 1.0f;
    pos.Y = (screen.Y - (float)view_port[1]) / (float)view_port[3] * 2.0f - 1.0f;
    pos.Z = screen.Z * 2.0f - 1.0f;
    pos.W = 1.0f;
 
    Vector4 pos2 = Vector4.Transform(pos, Matrix4.Invert(Matrix4.Mult(view, projection)));
    Vector3 pos_out = new Vector3(pos2.X, pos2.Y, pos2.Z);
 
    return pos_out / pos2.W;
}

Hope that helps!

cengceng's picture

Yes I know I am wrong place :) but I just want to get some suggestions from you.
They are typically same.

gl.glGetIntegerv(GL.GL_VIEWPORT, viewport, 0);
gl.glGetDoublev(GL.GL_MODELVIEW_MATRIX, mvmatrix, 0);
gl.glGetDoublev(GL.GL_PROJECTION_MATRIX, projmatrix, 0);

with this code you get viewport, modelview matrix and projection matris. Yes I can do that with opengl and then
you use

glu.gluUnProject((double)x,(double)realy,0,mvmatrix,0,projmatrix,0,viewport,0,wcoordnear,0);

Ok I use the gluunProject with same input values. First with znear=0 and second zvalue=1
I get the values. I thought they are same methods. I used.But then I use the second part with finding the nearest point
Someone use one algorithm and I though maybe it will work for me :)

Thanks everone