pokemoen's picture

Geometry shader problem

I am working on a somewhat more advanced geometry shader example:

The idea is to make a simple scene: A box with a checker texture with a sphere inside it. The
camera will be inside the box looking at the sphere.

Rendering will be done in two passes. The first pass will render the box from within the
sphere, looking in along all three axes in the positive and negative directions. These six
different camara orientations will be rendered to individual faces of a cubemap, in one
single pass using the gl_Layer token in the geometry shader. This cubemap will be used the
the second pass.

The second pass will render both the box and the sphere. The box will have the checker
texture, and the sphere will use the cubemap from the first pass.

It's not yet complete, right now I am running into a problem using the geometry shader.

situation 1: (press f1 while running)
a cubemap fbo is being rendered to (textureCubeFBO)
a shader program is being used (shaderProgramCubemap)
a vbo is used (vboCube)

The symptom is that I only see 3 triangles while the vbo has 12. Furthermore if I do not loop 6 times in the geometry shader and if I do not set gl_Layer, I get 4 triangles.

#version 120
#extension GL_EXT_geometry_shader4 : enable
void main(void) {
	int i, layer;
	for (layer = 0; layer < 6; layer++) {
		gl_Layer = layer;
		for (i = 0; i < gl_VerticesIn; i++) {
			gl_Position = gl_PositionIn[i];
			gl_TexCoord[0] = gl_TexCoordIn[i][0];

However when I change the situation to 2: (press f2 while running)
no fbo rendering
another shaderprogram without a geometry shader (shaderProgramBox)
and the same vbo rendering

everything looks as expected.

I have read most of http://www.opengl.org/registry/specs/EXT/geometry_shader4.txt but maybe I am missing something...

Does anyone have experience with this or can anyone try running this on a nvidia card (as I am beginning to suspect this might be an ati problem since the driver (silently, noting the the changelog) added support for this in the latest release (9.8))

Or does anyone have some pointers in which direction to look for solutions?

any help is appreciated ;)

Attached is the full code as it is right now. Its very messy still.

GeometryShader2.cs26.09 KB


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
pokemoen's picture

Ok, one thing I was doing incorrectly here is the place I set program parameters for the geometry shader.

About input and ouput primitives:

The input primitive type is a parameter of the program object, and must be
set before linking by calling ProgramParameteriEXT with ... This setting
will not be in effect until the next time LinkProgram has been called

And about the max vertives out:

To do so, call ProgramParameteriEXT with set
to GEOMETRY_VERTICES_OUT_EXT and set to the maximum number of
vertices the geometry shader will emit in one invocation. This setting
will not be guaranteed to be in effect until the next time LinkProgram has
been called successfully.

So I have changed the code to do these calls before linking. It did not solve the problem of not all primitives being emitted, what it did solve is the low perfromance and the degradation of acceleration in general (like scrolling webpages and other 2d acceleration).

I did however found a workaround so all primitives are emitted, it's weird and I can not explain it, but if I set GEOMETRY_VERTICES_OUT_EXT to 18 in stead of 0x400 (MAX_GEOMETRY_OUTPUT_VERTICES_EXT) all primitives are rendered to all layers of the cubemab fbo.

Another thing I noticed is when setting GEOMETRY_VERTICES_OUT_EXT to 3 I see all primitives on the first layer, which makes perfectly sense because per geometry shader invocation (per triangle) only 3 vertices and one primitive is emitted (to the first layer only). However setting this to 6 does not show primitives on another layer. Only at 15 another layer contains primitives(all of them). And then if I set it to 18, all layers contain all primitives.

I am confused, but at least I can continue now :)

pokemoen's picture

OK the degradation of performance was due to something else: Youtube flashplayer on the second monitor...

pokemoen's picture

I had a problem in the drawCubemapCross() method. I copied this from another project but kinda forgot I never tested it. It turned out to have the same texcoords on 4 faces of the cross :p

So that explained the GEOMETRY_VERTICES_OUT_EXT setting did not do what I expected.

The attached code now renders the all sides of of the cubemap correctly in one pass in the geometry shader, and display the individual faces in to form of a cross.

Next up, drawing a sphere and use the cubemap for reflection :)

GeometryShader2.cs29.49 KB
pokemoen's picture

I have a victory :)

Attached is a not cleaned up, but working as intended version of the original idea described in first post.

It's quite a lot bigger because I was lazy and did not generate the sphere, it is included as a float array. I kept it low poly mode for size, which shows in the reflection. Maybe I'll go crazy and generate the sphere in a geometry shader too!

You can switch between Scene mode and Cubemap viewing mode with F1 and F2.

Anyway what do you guys think of my first real geometry shader use (and cubemap for that matter)?

GeometryShader2.cs74.81 KB
the Fiddler's picture

I'd test this, but I still haven't fixed Linux support...

the Fiddler's picture

Ok, issue fixed. I tested this on a Nvidia card (Quadro NVS135 mobile, equivalent to an 8400, drivers: 190.25, Linux amd64) and it seems that the framebuffer is not complete. Error log:

Vertex info
0(4) : warning C7050: "eyepos" might be used before being initialized
0(4) : warning C7547: extension GL_EXT_geometry_shader4 not supported in profile gp4fp
0(4) : warning C7547: extension GL_EXT_geometry_shader4 not supported in profile gp4vp
Vertex info
0(4) : warning C7547: extension GL_EXT_geometry_shader4 not supported in profile gp4vp
Geometry info
0(62) : warning C7050: "gl_TexCoord[0]" might be used before being initialized
0(62) : warning C7050: "gl_TexCoord[2]" might be used before being initialized
Fragment info
0(4) : warning C7547: extension GL_EXT_geometry_shader4 not supported in profile gp4fp
FBO: Status unknown. (yes, this is really bad.)
max. ColorBuffers: 8 max. AuxBuffers: 8 max. DrawBuffers: 8
Stereo: 0 Samples: 0 DoubleBuffer: 0

The code then fails on the GL.DrawArrays() call with an InvalidFramebufferOperation error (this occurs when you try to use an incomplete framebuffer).

pokemoen's picture

Apparently ati/amd is more forgiving. I did not un-comment the attaching of the depth renderbuffer for the cube fbo. And this caused an incomplete framebuffer.

- Added framebuffer error tokens to switch statement.
- fixed geometry shader warnings.

Now works on a gf 8800 gts 512 with driver (

GeometryShader2.cs74.96 KB
the Fiddler's picture

I have committed an slightly modified version to SVN. The main difference is that I have replaced sphereData by generator functions. The sphere now uses an element buffer and contains more vertices for a smoother look.

I have also added a VertexPositionNormalTexture structure and modified the code to use BlittableValueType.StrideOf() instead of hardcoded stride values.

pokemoen's picture

Thanks, I wasn't aware of BlittableValueType, nice way of doing it!

I was planning to clean up some more and add comments, I'll post a patch when I get around to it.

the Fiddler's picture


BlittableValueType is a new addition. I've been using it in my projects and its quite useful for custom vertex types. I am planning to add another method that retrieves the offsets of fields (useful for methods like GL.NormalPointer). Not too sure about the name, though ("blittable" is the correct term, but "PureValueType" might have been better).