Pixel Perfect Panning Ideas?

Hi I've been making a 3D camera class and I'm almost done, except I haven't figured out how to do pixel perfect panning. What I mean by that is when I click on a point (2D screen space) I want that point (in 3D space) to perfectly line up as I move the mouse. The difficulty is converting that 2D motion to 3D motion. Will I need to use something like unproject to get what I want? Or is there any easier way?

I'm looking for something that will give me changing Up and Right vectors for the camera's view matrix.

Any thoughts?

Thanks in advance