Pointing\Touching objects in a 3D game for the mobile.

If you are making a 3D game for a mobile device you are likely to handle a screen touch input.

In a 2D game, converting the screen touch coordinates into the game position is simple enough.

However, in a 3D game there is more to consider.

When you want the user to click a 3D object in your game, there are several ways to do that.

You can project the object’s position to the screen coordinates and test if the touch position is close to the projected object position.

Or you can convert the touch screen coordinates into a 3D position inside the game’s world coordinates.

I will focus on the second method since it also allows for dragging objects and setting an object’s position.

From Touch to 3D

In order to calculate a 3D position corresponding to our touch screen coordinates, we will need to store the 3D camera\view’s inverted View and Projection matrices separately.

The view and projection matrices are used to project a 3D object into the 2D screen as it would be seen from a camera.

The View matrix of the camera place the 3D object relative to the camera’s origin, and the Projection matrix will project the 3D object into the camera’s “film” or the 2D receiver screen\chip.

What we are going to do is the inverse operation. We will take a coordinate on the camera’s 2D screen\film and find a corresponding 3D position that would have been projected into the camera screen had we did the normal projection from 3D to 2D.

Lets look at some code:

			Graphics2D::Position CalcTouch(double x, double y)
				Graphics2D::Position p2 = InvProj.MulPosNormal(Graphics2D::Position(x/320., -(y/512.), 0));
				Graphics2D::Position p3 = InvProj.MulPosNormal(Graphics2D::Position(x/320., -(y/512.), 1));
				Graphics2D::Position n = (p3-p2).Normalize();
				p2 = InvView.MulPos(p2);
				p3 = InvView.MulPos(p3);
				n = InvView.MulDir(n);
				n = n/n.y;
				p2 = p2-n*p2.Dot(Graphics2D::Position(0, 1, 0));
				return p2;

The function above accepts a 2D coordinate(x and y) and returns a Graphics2D::Position, which is actually a 3D coordinates in my framework.

At first we convert (x, y) into 2 screen space vertices and inverse project them to get 3D vertices(stored into p2 and p3).

The reason we inverse project two vertices is that we want to get a position and a direction(a 3D ray).

After getting the position and direction relative to the camera’s origin, we multiply them by the inverse View matrix to get the position and direction in world space coordinates.

We now have the ray we want but we want to select or drag an object. For that we are intersecting the ray with the plane of our choice(in our case the xz world space plane) and we get our desired 3D point.

After getting the 3D point we can set the object to that position or we can compare the object’s position to that point to see if the player selected that object.