Improvement to text drawing.

Thanks for the code, I've been looking for changes like these. I'm aware of the problems with small font sizes: OpenTK performs kerning on individual glyphs up to 16pt, but doesn't layout text on whole pixels. This can (and will be) included to OpenTK, provided we can find the answers to some important questions:

First, are Gdi functions supported on Mono/X11? I guess they are, but we'll have to find out since which version (if it's Mono 1.2.3 or earlier there's no problem).

Second, MeasureCharacterRanges (Gdi) returns different lengths than MeasureString (Gdi+) - we'll need to take that into account internally in TextureFont.MeasureString(). Actually, these functions don't need to be public - they are just an implementation detail.

Third, up to which font size should this method be used? My personal preference is for sizes up to 16-18pt, but this depends on many factors.

Fourth, does MeasureCharacterRanges allocate memory? If yes, this method should not be used in uncached TextPrinter.Draw calls (ugh).

Fifth (and final), how do we handle text scrolling in this case? We might have layed out the text perfectly on pixel boundaries, but the user can destroy that with a simple call to GL.Translate(0.1, 0, 0). Should we leave this up to the user, or should we somehow try to compensate?

I'll check for Mono support asap, but we'll have to think a little on the rest to find the best approach.

Ok, here's a quick quality comparison of the current implementation, grid-fitting through MeasureCharacterRanges and manual grid-fitting on top of the current implementation:

Apart from the smallest size, MeasureCharacterRanges provides better results (ignore the y-direction, as this is not grid-fitted in any of the above). I'll also provide a screenshot of TextRenderer.DrawText for comparison, which should provide the best results.

I'm still not sure if MeasureCharacterRanges is a good tradeoff, however. It is faster than MeasureString, provides a little better results, but it generates garbage which might have a bigger impact on performance in the long term. It also looks like there is a bug in the native implementation, where trailing spaces are not measured. Last, but not least, the current implementation fails on the Text sample - it renders some text correctly but stops after a number of characters (can you confirm that on your machine?)

[Caching]
TextPrinter.Draw can render both cached and uncached text, so we need to minimize the garbage it generates - which is my main objection on MeasureCharacterRanges.

Now to comment on the rest of the issues.
Yes, the native implementation of MeasureCharacterRanges (and .NET implementation built on it) has a limitation of 32 characters per call. That's why my code was processing 32 characters at a time. Did you hit the limitation via some other means? Is this the limitation you're talking about?

What garbage does MeasureCharacterRanges generate? I thought it would all be cleaned up by properly disposing of the StringFormat and Range objects (which I neglected to do in my implementation, but could easily be done). Dealing with Disposable objects in quite common in GDI. Heck, just to draw a shape of a particular color you have to create a SolidBrush object to contain that color, and that's something that has to be disposed. Is that a problem?

I suppose by "garbage", you're referring to allocated-then-freed memory. Honestly, I've heard about this in academic contexts, but never in real performance optimization discussions that I can recall -- maybe I just haven't been involved in enough optimization discussions. But I'd think twice before discarding this on a theoretical performance hit. .NET does have much better memory management than the average compiler, as I understand it. Now granted, it's interacting with GDI+, and who knows what kind of memory management that has under the covers, but considering the trade-offs as I currently understand them, it might just be worthwhile to get the best text rendering, since I don't see any other way of getting it. I'll tell you one thing. At one time I was creating a new 128x128-pixel bitmap on every MouseMove event in my graphics editing component, and had neglected to dispose of the Bitmap. I thought .NET was good at memory management and knew best how and when to dispose of things when they were no longer referenced. Not so. It was chugging every few seconds as I drew things with the mouse. I assume it was doing automated garbage collection forced by the fact that it had built up a zillion un-diposed bitmap objects that could be disposesd, so then it would periodically clean them up. But then when I realized by folly and introduced a Dispose call, everything zipped right along. So I suspect garbage collection is not so much an issue if you properly dispose of all your disposable objects. What do you think?

My other concern is that, although the smallest font size is a small percentage of the available sizes, I suspect it will be the most popular size and the one people would most want to render cleanly and easily.

On the trailing space issue, did you write your own code or use mine? Mine was intentionally ignoring whitespace and I can see how it might have easily misled you to believing that MeasureCharacterRanges didn't process whitespace properly.

[Garbage]
Actually GetCharacterRanges is currently dealing with up to 32 pieces of garbage per call because it returns an array of (up to) 32 Region objects with each call. But sure, we can leave this alone until more performance data is available.

[Font Rendering]
The potential problem with caching the results of DrawString on a texture is the limitation of texture memory. For an application that is nearing its limit on texture memory or wants to draw (or more importantly, cache) a huge piece of text, it will hit a limitation and be unable to draw/cache as much text this way as it would using another draw method with individual glyphs.

[Samples]
What's TextRenderer? I don't see that in the .NET framework.
I like the output and trade-offs (those that you mentioned) of Graphics.DrawText (of course it's designed to handle all permutations one might want when drawing text), but I'm nervous about other costs: the cost of transferring the image from system memory to video memory might outweigh the cost of MeasureCharacterRanges plus its memory management plus the operations involved in telling GL to draw from a texture (even if the latter sounds like a lot more pieces). And I'm nervous about the memory cost of having to effectively store the whole text output block as a bitmap/texture at least temporarily. Don't you run into the same memory allocation/garbage problems... or are you planning on having a permanent fixed-size bitmap and texture dedicated to rendering text operations?

BTW, I noticed that when I disposed of a TextureFont, the font object passed into the constructor is also disposed. That kind of threw me for a loop. Was that intentional? (I wonder what happens if I pass a system font into the constructor.)

[Memory]
According to your calculations of 1024x1024 at 4:1 compression being 256KB, I assume you're thinking of a 1-byte-per-pixel pixel format. I haven't played around with various pixel formats much, but I'm just curious if that would also allow the application to draw the cached text in any color if the pixel format, for example, is only the alpha channel or something.

[Performance]
So the rasterization would be the bottleneck in this scenario. This would become in issue in scenarios where dynamic text needs to be drawn each frame (like an FPS counter), right? Are you going to perform some tests to compare the performance of this solution versus the original solution (and/or the MeasureCharacterRanges solution) of drawing individual glyphs directly from a texture, which presumably doesn't have this issue (or the System-Video transfer issue)?

I pointed out the potential flaw of system-video RAM transfer overhead, and it seems like the response is "that's not the half of it", which suggests to me that it might not be the end-all perfect solution ;).

I think things were on the right track (as far as performance is concerned) when you had all the glyphs (well, all the ones most people care about anyway) stored in texture memory. It was a truly hardware accelerated solution (right?). And I'm not sure where the best balance between features and performance is (if there isn't a single balance, maybe there needs to be two solutions allowing the client application to choose).

But in the interest of looking for the perfect solution ideal for all clients, let me share some other ideas that have been occurring to me:
1) Instead of caching all the glyphs when the font is created, which I presume is what's happening right now, it might be practical to "lazy-initialize" each glyph as it is used, which I believe is what system text rendering functions do internally. Then you wouldn't be limited to which characters could be drawn (do I understand correctly that the set it limited in the current implementation?). This probably isn't the best idea for a high performance game architecture, but it was an idea that occurred to me, so I figured I'd mention it just in case.
2) I was wondering if there was some way to cache the character sizes/offsets for each glyph as part of the TextureFont object instead of calling measurement functions each time, which might eliminate some concerns about garbage generation. But I don't know if the character size is static or changes based on its context. I feared for a moment that MeasureCharacterRanges would only return an area occupied by actual visible pixels of the glyph, but a test I just performed seems to confirm (with a non-serif font even) that the size of the "l" in "Hello, World!" fully occupies the space between the r and the d. But I also discovered that MeasureCharacterRanges does not in fact return regions that entirely enclose the glyphs as demonstrated by the magnified screenshot below. The overhang in the j is not included in the character range even if it's the only character I'm measuring.

So I'm starting to wonder how far to take this. I don't want things to get all blown out of proportion. Your solution of rendering in system memory and transferring to video memory is probably the best balance for now until/unless at some point we want to start delving into font internals to figure out how font rendering actually works internally just to get the correct character positions manually or something.

"isn't anyone else interested in font rendering?"

Didn't want to disturb the flow of your discussion. It's great that you are working on a sophisticated solution, but I've not much to add since this is way beyond my requirements for printing text on the screen. A greyscale&alpha spritesheet with hardcoded texcoords/width settings for each symbol is all I need (and trivial to implement).

If you want my 2c, use freetype and make that an optional plugin for util.dll. If anyone wants to have super-detailed-pixel-accurate-antialiased-text-printing-with-1000-options-from-an-arbitrary-font: they do have to pay a price for it.
Don't care about execution speed (can always be optimized later if it proves to be a bottleneck) but avoid massive garbage creation. This applies for textures aswell, you might be able to stuff it into some A8L8 format (or even something smaller) and use texture compression, but the price to pay is the continuous create/delete/binding of textures. For a game like Tetris this might be irrelevant, but for a RPG (usually text-heavy) this can become quite complex to handle.

Actually, on Windows TextRenderer.DrawText can be up to an order of magnitude faster than Graphics.DrawString. The first uses GDI (which is hardware accelerated) while the latter uses GDI+ (which is not). Unfortunately, in our case we want to render to a GDI+ bitmap not the screen, and this is quite costly for TextRenderer.DrawText.

Why use TextRenderer.DrawText? The main reason is that its output matches that of the OS, so you could create an OpenGL GUI that matches the surrounding forms. The other reason is that it can render complex scripts when Graphisc.DrawString cannot.

I'll cleanup the code a bit and upload it tomorrow.

Both TextRenderer and Graphics support string alignment and word wrapping. Only the first supports complex scripts, Graphics.DrawString definitely cannot draw any string (this is a limitation of the underlying GDI+ library, which Graphics uses). Check this document which outlines their differences.

The current text rendering method (compositing individual glyphs) moves the burden of implementing alignment and wrapping to us. The proposed method (burn the whole string into a texture) gives us both (as well as better quality) at the expense of texture memory.

Results will always be different between operatins systems, as the underlying rendering libraries are different (FreeType on Linux, GDI/GDI+ on Windows, Quartz (?) on Mac OS X). There is no workaround for that - Windows 98 will render OpenType/TrueType fonts differently than Windows XP and both will render differently than Mac OS X. If what you look for is consistency, you can always burn a texture with the needed glyphs and render text from that (bye bye unicode! :) ) In fact, this should be easy to implement in, or on top of, OpenTK.

Yes, this is the biggest problem with TextRenderer, although this screenshot isn't exactly representative (there's no alpha blending). This problem is visible in the picture I uploaded to the first page, too.

Looking at the source code of Mono, it looks like we can work around the allocation problems of MeasureCharacterRanges by DllImporting gdipMeasureCharacterRanges directly. It also seems that Mono's TextRenderer uses Graphics.DrawString on Linux, so we are limited to GDI+ rendering in all cases.

In this case, MeasureCharacterRanges on top of the current system looks like the most attractive solution. This also means we won't be able to render complex scripts, but this can be addressed in the future if the need arises.

I haven't found the time to polish the code yet for uploading (it's not compatible with the current system), but it looks like that might not be needed after all.

This screenshot isn't representative, because it doesn't perform any alpha-blending at all. The screenshot in the previous page on the other hand does perform alpha blending. How does it do this? It calculates an alpha 'mask' from the rgb values TextRenderer produces (the cached text is stored in an alpha-only texture). The quality *is* worse than correct alpha blending, but it doesn't look quite as bad as this screenshot.

Regarding MeasureCharacterRanges, it's problematic on both runtimes (.Net and Mono), due to memory allocations. I looked into Mono's implementation to see if this can be worked around somehow, and indeed it can (don't you love open-source software? :)) This means MeasureCharacterRanges is a valid approach to the problem.

I wouldn't go so far as to say we've switched places, it's just that it's best to consider all options ;)

Ok, summing up a few thoughts (I think better when writing stuff down :))

There are two parts we need to consider for the font engine:

1. How to lay out and rasterize text (low-level stuff)
2. How to display the result through OpenGL efficiently (high level)

Regarding the first, I can see three options:
a) Use interfaces that come with .Net (e.g. TextRenderer, Graphics)
b) Use OS interfaces through DllImport.
c) Rely on a native library non-system library, like Pango.

Of these:
* TextRenderer can layout complex scripts, but is slow and has problems with antialiasing on Windows.
* Graphics have good quality but cannot layout complex scripts.
* OS interfaces (e.g. GDI on Windows, ATSUI on Mac OS X) are more flexible, fast, can render complex scripts, but require platform-specific code (making ports more difficult).
* Native libraries hide the complexity of OS interfaces, but add a native dependency to OpenTK (which we do not want).

Pick you poison! :) I'd say "Graphics" plus "MeasureCharacterRanges" provide the best compromise between quality/complexity for now.

Regarding (2), namely displaying through OpenGL, we have two options:
a) Store individual glyphs in texture memory, and layout text as an array of textured quads.
b) Store whole strings in texture memory, and draw text as a single textured quad.

The first option makes better use of texture memory, but is more involved when performing layout (we have to position individual glyphs). The second option burns more texture memory, but we can delegate layout to the low-level handler (e.g. layout string "abc" in a 200x100 box, centered).

I honestly cannot say which option is better in the long run (input?) We already have (a) implemented, so we can keep using that for now.

A little offtopic, but not unrelated: has anyone tried to make the font smaller than requested and use bilinear texture filters to scale it up for blurring? (to workaround anti-alias)

This could make 2.b) a more interesting choice, as that would be a convenient way to provide the programmer with decals they can smack on a sign rendered in 3D, like objarni mentioned at 6)
Not sure if this is actually going to be used much, the common approach is building the whole text into the model's texture. objarni's idea is kinda interesting if you plan to localize the app for multiple languages though, this is quite problematic with the common approach if you do not have access to the texture's source image before the layers were collapsed.

My opinion on [1 - Rasterizing Text] is that Graphics.DrawString is the clear winner, and my opinion on [2 - Managing and Displaying the Text] is less clear, but I do lean toward caching whole strings in a texture and drawing them in single blocks. Here's my reasoning:

1) I don't see OpenTK as a text drawing engine but as a thin wrapper around OpenGL. So I don't expect it to be doing too much work for me, just removing some of the tedium of drawing basic text.
2) As such, OpenTK shouldn't have to take so much responsibility for being able to render complex scripts. If the feature support was good enough for the general-purpose Graphics.DrawString function in the framework, it should certainly suffice for OpenTK's text implementation. I for one rarely use non-ASCII characters (in my game development) let alone complex scripts. Even at work (for an international company that releases ERP software in China among other locales) the worst I deal with is Chinese characters and not complex scripts that have to combine characters.
3) I lean slightly toward caching whole strings instead of using MeasureCharacterRanges to position glyphs for one reason: I have one additional concern about MeasureCharacterRanges that I didn't mention. As I may have already mentioned, it can only handle 32 characters at a time on Windows. But my concern then is, if you have to call it 10 times to draw a 320 character string, do you end up with a O(n^2) algorithm as each successive call has to lay out all the text from before in addition to the new 32 characters being calculated? I can think of some possible optimizations (such as removing lines as they are processed) but it may just not be worth the effort of trying to optimize that (there are a couple ways that word wrap complicates how much text you need to provide to make sure the text ends up in the same place for each calculation).

To to summarize my current opinion, I'm liking Graphics.DrawString directly to cached textures with no intermediate glyph management. But I'm not totally turned off of MeasureCharacterRanges if you don't think there's a performance concern.

Unicode's not an issue though. Graphics.DrawString doesn't have any problem drawing Unicode text in general, just complex scripts like Arabic where (as I understand it) characters look different depending on what order they come in. Chinese and Japanese are no problem for any of the solutions being discussed here I think.

Edit: I'm seeing some reports that even Arabic text renders correctly with Graphics.DrawString, so it's hard to come up with examples of what Graphics.DrawString can't render.

[MeasureCharacterRanges and speed]
I think MeasureCharacterRanges has to consider more characters than just the requested range. How could it possibly know where characters 32 through 63 are without knowing how much characters 0 through 31 offset them and how many newlines were in there? I'm assuming MeasureCharacterRanges is stateless and doesn't cache information about previous calls, so it has to lay out all the text each time (at least all the preceding text). Furthermore, if you're performing word wrap, how could MeasureCharacterRanges know that a particular character in a word can fit on the current line without positioning the rest of the characters in the word (regardless of whether they are included in the current range being measured).

The upcoming version, 0.9.1, will only contain bugfixes to the current system. The results of this discussion will appear in 0.9.2+.

0.9.1 has taken a long time to bake due to the large number of internal restructuring - future releases will appear in shorter intervals.