The Stencil buffer's primary use is to apply a mask to the framebuffer. Simply put, you can think of it as a cardboard stencil where you cut out holes, so you may use a can of spraypaint to paint shapes. The paint will only pass the holes you had cut out and be blocked otherwise by the cardboard. OpenGL's Stencil testing allows you to layer several of these masks over each other.
A more OpenGL related example: In any vehicle simulation the interior of the cockpit is usually masked by a stencil buffer, because it does not have to be redrawn every frame. That way alot of fragments of the outside landscape can be skipped, as they would not contribute to the final image anyway.
In order to use the Stencil buffer, the window-system provided framebuffer - or the Stencil attachment of a FBO - must explicitly contain a logical stencil buffer. If there is no stencil buffer, the fragment is always passed to the next pipeline stage.
For the purpose of clarity in this article, the Stencil Buffer is assumed to be 8 Bit large and able to represent the values 0..255
StencilTest functionality is enabled and disabled with EnableCap.StencilTest
GL.Enable( EnableCap.StencilTest ); GL.Disable( EnableCap.StencilTest ); // default
The value used by GL.Clear() commands can be set through:
GL.ClearStencil( int ); // 0 is the default, Range [0..255]
If StencilTest is enabled, writing can be limited by a bitfield through a bitwise AND.
GL.StencilMask( bitmask ); // By default all bits are set to 1.
Note: The command GL.StencilMaskSeparate() behaves exactly like GL.StencilMask() but allows separate comparison functions for front- and back-facing polygons.
The command GL.StencilFunc( func, ref, mask ) is used to specify the conditions under which the StencilTest succeeds or fails. It sets the comparison function, reference value and mask for the Stencil Test.
The word 'pixel' means in this case: The value in the Stencil Buffer at the current pixel.
Note: The command GL.StencilFuncSeparate() behaves exactly like GL.StencilFunc() but allows separate comparison functions for front- and back-facing polygons.
Depending on the result determined by
GL.StencilFunc(), the command
GL.StencilOp( fail, zfail, zpass ) can be used to decide what action should be taken if the fragment passes the test.
The following values are allowed, the default for all operations is StencilOp.Keep
Note: The command GL.StencilOpSeparate() behaves exactly like GL.StencilOp() but allows separate comparison functions for front- and back-facing polygons.
To determine whether StencilTest is enabled or disabled, use
Result = GL.IsEnabled( EnableCap.StencilTest );
The bits available in the Stencil Buffer can be queried by
GL.GetInteger( GetPName.StencilBits, ... );
The value set by GL.ClearStencil() can be queried by
GL.GetInteger( GetPName.StencilClearValue, ... );
The bitfield set by GL.StencilMask() can be queried by
GL.GetInteger( GetPName.StencilWritemask, ... );
The state of the Stencil comparison function can be queried with
GL.GetInteger() and the following parameters:
GetPName.StencilFunc - GL.StencilFunc's parameter 'func'
GetPName.StencilRef - GL.StencilFunc's parameter 'ref'
GetPName.StencilValueMask - GL.StencilFunc's parameter 'mask'
The state of the Stencil operations can be queried with GL.GetInteger and the following parameters:
GetPName.StencilFail - GL.StencilOp's parameter 'fail'
GetPName.StencilPassDepthFail - GL.StencilOp's parameter 'zfail'
GetPName.StencilPassDepthPass - GL.StencilOp's parameter 'zpass'
If the GL.Stencil***Separate() functions have been used, the tokens GetPName.StencilBack*** become available to query the settings for back-facing polygons. With Intellisense you should not have any problems finding them.
Related Extensions for further reading
http://www.opengl.org/registry/specs/EXT/stencil_wrap.txt (promoted to core in GL 2.0)
http://www.opengl.org/registry/specs/ATI/separate_stencil.txt (promoted to core in GL 2.0)
http://www.opengl.org/registry/specs/EXT/stencil_two_side.txt (basically the same functionality as ATI_separate_stencil, not in core though)