04. Stencil Test

The Stencil buffer's primary use is to apply a mask to the framebuffer. Simply put, you can think of it as a cardboard stencil where you cut out holes, so you may use a can of spraypaint to paint shapes. The paint will only pass the holes you had cut out and be blocked otherwise by the cardboard. OpenGL's Stencil testing allows you to layer several of these masks over each other.

A more OpenGL related example: In any vehicle simulation the interior of the cockpit is usually masked by a stencil buffer, because it does not have to be redrawn every frame. That way alot of fragments of the outside landscape can be skipped, as they would not contribute to the final image anyway.

In order to use the Stencil buffer, the window-system provided framebuffer - or the Stencil attachment of a FBO - must explicitly contain a logical stencil buffer. If there is no stencil buffer, the fragment is always passed to the next pipeline stage.

For the purpose of clarity in this article, the Stencil Buffer is assumed to be 8 Bit large and able to represent the values 0..255

StencilTest functionality is enabled and disabled with EnableCap.StencilTest

GL.Enable( EnableCap.StencilTest );
GL.Disable( EnableCap.StencilTest ); // default

The value used by GL.Clear() commands can be set through:

GL.ClearStencil( int ); // 0 is the default, Range [0..255]

If StencilTest is enabled, writing can be limited by a bitfield through a bitwise AND. 

GL.StencilMask( bitmask ); // By default all bits are set to 1.

Note: The command GL.StencilMaskSeparate() behaves exactly like GL.StencilMask() but allows separate comparison functions for front- and back-facing polygons.

GL.StencilFunc()

The command GL.StencilFunc( func, ref, mask ) is used to specify the conditions under which the StencilTest succeeds or fails. It sets the comparison function, reference value and mask for the Stencil Test.

  • ref is an integer value to compare against. By default this value is 0, range [0 .. 255]
  • mask is a bitfield which will be used in a bitwise AND. Only the bits which are set to 1 are considered. By default all bits are set to 1.
  • func of the test can have the following values, the default is StencilFunction.Always.

    • StencilFunction.Always - Test will always succeed.
    • StencilFunction.Never - Test will never succeed.
    • StencilFunction.Less - Test will succeed if ( ref & mask ) < ( pixel & mask )
    • StencilFunction.Lequal - Test will succeed if ( ref & mask ) <= ( pixel & mask )
    • StencilFunction.Equal - Test will succeed if ( ref & mask ) == ( pixel & mask )
    • StencilFunction.Notequal - Test will succeed if ( ref & mask ) != ( pixel & mask )
    • StencilFunction.Gequal - Test will succeed if ( ref & mask ) >= ( pixel & mask )
    • StencilFunction.Greater - Test will succeed if ( ref & mask ) > ( pixel & mask )

    • The word 'pixel' means in this case: The value in the Stencil Buffer at the current pixel.

Note: The command GL.StencilFuncSeparate() behaves exactly like GL.StencilFunc() but allows separate comparison functions for front- and back-facing polygons.

GL.StencilOp()

Depending on the result determined by GL.StencilFunc(), the command GL.StencilOp( fail, zfail, zpass ) can be used to decide what action should be taken if the fragment passes the test.

  • fail - behavior when StencilTest fails, regardless of DepthTest.
  • zfail - behavior when StencilTest succeeds, but DepthTest fails.
  • zpass - behavior when both tests succeed, or if StencilTest succeeds and DepthTest is disabled.

The following values are allowed, the default for all operations is StencilOp.Keep

  • StencilOp.Zero - set Stencil Buffer to 0.
  • StencilOp.Keep - Do not modify the Stencil Buffer.
  • StencilOp.Replace - set Stencil Buffer to ref value as specified by last GL.StencilFunc() call.
  • StencilOp.Incr - increment Stencil Buffer by 1. It is clamped at 255.
  • StencilOp.IncrWrap - increment Stencil Buffer by 1. If the result is greater than 255, it becomes 0.
  • StencilOp.Decr - decrement Stencil Buffer by 1. It is clamped at 0.
  • StencilOp.DecrWrap - decrement Stencil Buffer by 1. If the result is less than 0, it becomes 255.
  • StencilOp.Invert - Bitwise invert. If the Stencil Buffer currently contains the bits 00111001, it is set to 11000110.

Note: The command GL.StencilOpSeparate() behaves exactly like GL.StencilOp() but allows separate comparison functions for front- and back-facing polygons.

State Queries

To determine whether StencilTest is enabled or disabled, use Result = GL.IsEnabled( EnableCap.StencilTest );

The bits available in the Stencil Buffer can be queried by GL.GetInteger( GetPName.StencilBits, ... );

The value set by GL.ClearStencil() can be queried by GL.GetInteger( GetPName.StencilClearValue, ... );

The bitfield set by GL.StencilMask() can be queried by GL.GetInteger( GetPName.StencilWritemask, ... );

The state of the Stencil comparison function can be queried with GL.GetInteger() and the following parameters:

GetPName.StencilFunc - GL.StencilFunc's parameter 'func'
GetPName.StencilRef - GL.StencilFunc's parameter 'ref'
GetPName.StencilValueMask - GL.StencilFunc's parameter 'mask'

The state of the Stencil operations can be queried with GL.GetInteger and the following parameters:

GetPName.StencilFail - GL.StencilOp's parameter 'fail'
GetPName.StencilPassDepthFail - GL.StencilOp's parameter 'zfail'
GetPName.StencilPassDepthPass - GL.StencilOp's parameter 'zpass'

If the GL.Stencil***Separate() functions have been used, the tokens GetPName.StencilBack*** become available to query the settings for back-facing polygons. With Intellisense you should not have any problems finding them.

Related Extensions for further reading

http://www.opengl.org/registry/specs/EXT/stencil_clear_tag.txt
http://www.opengl.org/registry/specs/EXT/stencil_wrap.txt (promoted to core in GL 2.0)
http://www.opengl.org/registry/specs/ATI/separate_stencil.txt (promoted to core in GL 2.0)
http://www.opengl.org/registry/specs/EXT/stencil_two_side.txt (basically the same functionality as ATI_separate_stencil, not in core though)