logixoul's picture

Mouse events get missed

I'm writing a drawing app (think MS Paint). So it's important to get precise info about mouse movement.

However, in OpenTK, there's a problem. If I call NativeWindow.ProcessEvents, and the mouse has moved, my mousemove eventhandler gets called only *once*, with the current mouse pos. That happens even if the mouse has gone through several locations before I call ProcessEvents (meaning, the mouse device has sent several events to the computer - mouse devices usually have a high frequency).

That means my drawing app ends up having jaggy drawing. I solved that: I run an "input thread" that polls System.Windows.Forms.Cursor.Position every couple of nanosecs, and if it has changed, writes it to my own "mouse position queue". That works, but it sucks in a lot of ways.

  • On singlecore PCs the CPU gets really hogged
  • The use of threading is complicated to me. E.g. sometimes I get problems like thread starvation (I think), and I don't know how to solve them since I have a weak understanding of threads. I could try to debug those but I'd rather just stop using the thread hack.

I tried to make my input thread wait for mouse events, rather than constantly poll for them. I used MsgWaitForMultipleObjects from WinAPI to do that. But it never got any events - probably because Windows doesn't send events to any thread other than the GUI thread.

I think there should be a flag in OpenTK's MouseDevice, something like "bool MouseDevice.TrackAllPositions", which enables a mode like I described. It should be off by default because it would be detrimental and useless to most apps (like games). The question is, how should it be implemented?

I'm gonna keep experimenting (e.g. with a global mouse hook) but I wanted to give you a heads up.


Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
the Fiddler's picture

OpenTK should report one Mouse.Move event for each WM_MOUSEMOVE event received. Are you using GameWindow? If so, there is no need to call ProcessEvents() manually (it shouldn't hurt but it's called automatically anyway).

If possible, please post a bug report with a short test case that demonstrates the issue.

the Fiddler's picture

Sorry, I can't reproduce. My test case:

// This code is in the Public Domain. It is provided "as is"
// without express or implied warranty of any kind.
 
using System;
using System.Drawing;
using OpenTK;
using OpenTK.Graphics.OpenGL;
using OpenTK.Input;
using System.Collections.Generic;
 
namespace Examples.Tutorial
{
    public class SimpleWindow : GameWindow
    {
        List<Vector2> vs = new List<Vector2>(1024);
        public SimpleWindow() : base(800, 600)
        {
            Mouse.Move += (sender, e) =>
                {
                    vs.Add(new Vector2(2 * e.X / (float)Width - 1, 1 - 2 * e.Y / (float)Height));
                };
        }
 
        protected override void OnResize(EventArgs e)
        {
            GL.Viewport(0, 0, Width, Height);
 
            GL.MatrixMode(MatrixMode.Projection);
            GL.LoadIdentity();
            GL.Ortho(-1.0, 1.0, -1.0, 1.0, 0.0, 4.0);
        }
 
        protected override void OnRenderFrame(FrameEventArgs e)
        {
            GL.Clear(ClearBufferMask.ColorBufferBit);
 
            GL.Begin(BeginMode.LineLoop);
            foreach (var v in vs)
                GL.Vertex2(v);
            GL.End();
 
            SwapBuffers();
        }
 
        public static void Main()
        {
            using (SimpleWindow example = new SimpleWindow())
            {
                example.Run(30.0, 0.0);
            }
        }
    }
}
logixoul's picture

Your testcase exhibits the problem as well. It's just not very visible because your testcase runs at a very high framerate. Now consider an app that may drop to 10fps. This is the case in my app sometimes.

To emulate that in your testcase, add a Thread.Sleep(300) in OnRenderFrame. Of course, this will make the drawn lines "jaggy" because the vertices will get far apart when moving the mouse quickly. If my suggestion ("bool MouseDevice.TrackAllPositions") is implemented, it will fix the "jagginess", because, no matter the framerate, all mouse events will get sent to the window.

Now, you mention WM_MOUSEMOVE, but that's just an implementation detail. I'm not well familiar with the WinAPI, but I know that:

  1. I want OpenTK to support a TrackAllPositions mode.
  2. This *is* possible to implement, because as I said, I've already done it with a hacky constant-polling solution.

The WM_MOUSEMOVE API may not support a TrackAllPositions mode, but if you agree with the necessity of that mode, we'll just figure out a way to implement it without WM_MOUSEMOVE.

the Fiddler's picture

It seems that only one WM_MOUSEMOVE is generated for multiple mouse interrupts when the period between GetMessage calls is larger than the polling rate of the mouse. Linux doesn't suffer from this problem.

GetMouseMovePointsEx might provide a solution. Please file a bug report on the issue.

Stefan Monov's picture

Issue posted at: http://www.opentk.com/node/2157

GetMouseMovePointsEx indeed solved the problem for me. Thanks. How did you find out about it? I looked and looked but couldn't find such an API previously.

Here's the code I used, it may be helpful to you. (for now I'm using it externally to OpenTK, thus breaking the OpenTK abstraction)

using System.Collections.Generic;
using System.Drawing;
using System.Linq;
using System.Runtime.InteropServices;
using OpenTK;
using OpenTK.Input;
using System.ComponentModel;
 
namespace OpenTKFix
{
    static class WinApi
    {
        public const int GMMP_USE_DISPLAY_POINTS = 1;
        public const int GMMP_USE_HIGH_RESOLUTION_POINTS = 2;
 
        [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)] // For GetMouseMovePointsEx
        public struct MOUSEMOVEPOINT
        {
            public int x;               //Specifies the x-coordinate of the mouse
            public int y;               //Specifies the y-coordinate of the mouse
            public int time;            //Specifies the time stamp of the mouse coordinate
            public IntPtr dwExtraInfo;  //Specifies extra information associated with this coordinate. 
        }
 
        [DllImport("user32", ExactSpelling = true, CharSet = CharSet.Auto, SetLastError = true)]
        public static extern int GetMouseMovePointsEx(uint cbSize, [In] ref MOUSEMOVEPOINT pointsIn,
            [Out] MOUSEMOVEPOINT[] pointsBufferOut, int nBufPoints, uint resolution);
   }
 
    class MouseEventQueue
    {
        private Queue<Point> queue = new Queue<Point>();
 
        private GameWindow hostingWindow;
 
        public bool HasEvents
        {
            get { UpdateQueue(); return queue.Count > 0; }
        }
 
        private int lastTimeStamp;
 
        public MouseEventQueue(GameWindow hostingWindow)
        {
            this.hostingWindow = hostingWindow;
        }
 
        public Point RetrieveNextEvent()
        {
            UpdateQueue();
            return queue.Dequeue();
        }
 
        private void UpdateQueue()
        {
            var currentPos = System.Windows.Forms.Cursor.Position;
 
            // If a point is outside the window, we can't pass that point to GetMouseMovePointsEx.
            // This can happen because we use mouse capturing.
            if (!hostingWindow.Bounds.Contains(currentPos))
                return;
 
            // the bitwise-and is from the example, I don't know what it's for.
            // I think it's related to http://support.microsoft.com/kb/269743
            // --Stefan
            var referencePoint = new WinApi.MOUSEMOVEPOINT();
            referencePoint.x = (currentPos.X) & 0x0000FFFF;
            referencePoint.y = (currentPos.Y) & 0x0000FFFF;
 
            const int numPointsRequired = 64; // 64 is the maximum allowed
            var rawPointList = new WinApi.MOUSEMOVEPOINT[numPointsRequired];
            int numPointsReturned = WinApi.GetMouseMovePointsEx((uint)(Marshal.SizeOf(referencePoint)),
                ref referencePoint, rawPointList, numPointsRequired, WinApi.GMMP_USE_DISPLAY_POINTS);
            if (numPointsReturned == -1)
            {
                throw new Win32Exception();
            }
 
            var pointList = new List<Point>();
            for (int i = 0; i < numPointsReturned; i++)
            {
                if (rawPointList[i].time == lastTimeStamp)
                    break;
 
                var p = new Point(rawPointList[i].x, rawPointList[i].y);
 
                if (pointList.Count == 0 || pointList.Last() != p)
                {
                    pointList.Add(hostingWindow.PointToClient(p));
                }
            }
            lastTimeStamp = rawPointList[0].time;
 
            // GetMouseMovePointsEx returns the points from oldest to newest, we need them in the opposite order
            // for the queue.
            pointList.Reverse();
            WorkaroundMicrosoftBug(pointList);
 
            foreach(var p in pointList)
                queue.Enqueue(p);
        }
 
        // workaround for http://support.microsoft.com/kb/269743
        // (adapted from their suggested workaround code)
        private static void WorkaroundMicrosoftBug(List<Point> points)
        {
            for (int i = 0; i < points.Count; i++)
            {
                var p = points[i];
                if (p.X > 32767)
                    p.X -= 65536;
                if (p.Y > 32767)
                    p.Y -= 65536;
                points[i] = p;
            }
        }
    }
}

(this is still me, logixoul, this is my new account)

c2woody's picture
the Fiddler wrote:

It seems that only one WM_MOUSEMOVE is generated for multiple mouse interrupts when the period between GetMessage calls is larger than the polling rate of the mouse.

Is this a plain "GUI thread blocks for too long" problem, since the WM_MOUSEMOVE simply can't be generated more often if the GUI thread has too much load?

Stefan Monov's picture
c2woody wrote:

Is this a plain "GUI thread blocks for too long" problem, since the WM_MOUSEMOVE simply can't be generated more often if the GUI thread has too much load?

No. What we want here is several WM_MOUSEMOVE events after every frame. Imagine an app running at 30fps and a mouse device getting polled by Windows at 125hz and you'll see that if you get just a single WM_MOUSEMOVE after every frame, you're going to miss out on a lot of precision (which is relevant for a MS Paint-like app).

the Fiddler's picture

This is actually a peculiarity in the WM_MOUSEMOVE implementation [1]: when the mouse moves, Windows sets an internal flag but does not actually generate a WM_MOUSEMOVE event. When you call GetMessage, Windows checks this flag and synthesizes a WM_MOUSEMOVE event using the location of the mouse at the time of the GetMessage call. In other words, the rate of WM_MOUSEMOVE messages is bound by the rate of GetMessage calls.

In comparison, X11 adds MouseMove events to the queue even if you don't call NextEvent at all (no message is ever lost here).

@Stefan Monov: you can work around this issue in a cross-platform manner by rendering from a different thread (check out the "Multithreaded Rendering" sample). The code you posted is very useful, I will try to add it to OpenTK.

[1] http://blogs.msdn.com/b/oldnewthing/archive/2003/10/01/55108.aspx

c2woody's picture
Stefan Monov wrote:

No. What we want here is several WM_MOUSEMOVE events after every frame.

Well I wasn't asking what you want, since you'd have to ask Microsoft about issueing more WM_MOUSEMOVE events in this case. It's unlikely that they'll change some fundamental behaviour like this though.