The tessellation shaders are used to subdivide 3D geometry inside the shader pipeline. This allows for dynamic LOD(Level Of Details) but also allows to produce a lot more geometry from a given vertex buffer. This in turn save in bandwidth and processing and even allows for LOD that change within the same model.
The tessellation shaders consist of two Hull shaders, the tessellation stage(which is not programmable) and the Domain shader.
When sending geometry with the tessellation shaders activated, we don’t send normal vertices but rather control points. The control points are processed in the vertex shader as normal vertices and then they are passed to the Hull shaders. The Hull shaders decide what kind of geometry and how many subdivisions the tessellation stage should produce. After the tessellation stage produces the new vertices, the Domain shader process all the new geometry as if they were normal vertices processed by the vertex shader. Which means the Domain shader usually projects the vertices into the screen space for rasterization.
A naive subdivision would be to set a constant subdivision factor per model. Each primitive in our model will subdivide into an identical amount of triangles.
The issue is that some primitives in the original model might be bigger and some smaller, it make no sense to subdivide both the big and small primitive the same. In addition, some primitive might be far from the camera and consist only a few pixels in the screen space. This leads us to try to subdivide the mesh according to how many screen space pixels the triangle or primitive covers.
This might seem simple enough. However, there is an issue. What happens if we subdivide two adjacent triangles, while one triangle will be subdivided into 4 triangles and the adjacent triangle into 9? At first it doesn’t seem like an issue, but if we displace the new vertices with a displacement map(texture), we might have a gap between what were originally two triangles.
We need to make sure that adjacent primitives in the original model, will subdivide exactly the same on their shared edge. This will result that for every new vertex in one primitive, there will be a spatially identical vertex on the neighbouring primitive. Thus when the displacement map translate the vertices, the two new vertices will move to the same place.
In other words, when ever we decide how to subdivide an edge of a primitive, we need to rely only on data that is available to the primitives of both sides of the edge. And we need the tessellation factor of this edge to be identical on both primitives who shared that edge.
With the constant tessellation, its easy, we just provide an identical constant value for all the edges.
When we want it to be tesselated according to screen space area, for each edge we need to calculate the average of the area of the triangles from both sides of the edge.
We use six control points to achieve this. Three for the triangle itself, and another three for the three triangles that share the edges of the current triangle. With those 6 control points we are able to calculate the screen space area of 4 triangles which are used to calculate the tessellation factor of the 3 edges.
Here is a nice video showing screen space tessellation in real-time: