
Load Textures while rendering (Multithreaded)?
Posted Wednesday, 17 March, 2010 - 21:16 by 2k1Toaster inHi,
Currently I am using TexLib to create textures and such. It is working beautifully. If I call all my code synchronously from the same thread, everything works great but takes forever to load. So instead I am loading only what needs to be loaded for an initial displaying, then create 2 threads that run in the background and parse things from resource files while the main thread starts drawing the GUI.
What I am running into is that once the GUI is being drawn, any texture I try to load comes back with an index of '0', and just doesnt work. If I use the same code but call the loading functions directly from the main thread, it all works gloriously.
So is there something prohibiting loading textures to a context while that context is "active" (I dont know the real name for it if one exists)?
I saw one brief thread late last year about multithreading loading, and someone suggested using a second context by creating a new nativewindow and everything in the background. This seems a little convoluted to me, are there really no other ways?
I think the DLL is version 0.9.9.4, but I have made some changes to both the main project files and TexLib, and I am not sure if I changed my local version number or not...


Comments
Re: Load Textures while rendering (Multithreaded)?
You need to create an OpenGL context for each thread that you need to call OpenGL methods from. This is how OpenGL works, there's no way around this limitation. The documentation has more information on multiple contexts.
However, you can still load resources asynchronously using a single context. The trick is to separate the "disk -> system memory" part (slow) from the "system memory -> OpenGL" part (fast) and distribute the slow part to a thread pool. Whenever a background thread finishes loading, you can signal the main thread to generate a new texture object and upload the data to OpenGL.
The reason why this works is because the bottleneck lies in disk access. Reading the texture from disk may take several ms, whereas uploading to OpenGL requires just a few μs, thus it doesn't really hurt to leave the second part to the main thread. Much simpler than creating multiple contexts, too (one for each loader thread).
Note that you'll likely get an even larger performance improvement if you use dds-compressed textures. The reason is that you can send them directly from disk to OpenGL (no need to decode into system memory first, as is the case with jpg or png. Parse the header, find out the format and upload the compressed data directly).
Re: Load Textures while rendering (Multithreaded)?
The problem I have with the single-context approach is that I'm still getting graphical hitches when uploading textures, especially large ones with auto-generating mipmaps, and would like to have a secondary OpenGL context for processing that.
So, before trying this, in the example link above this demonstrates creating a second OpenGL context, but also looks like it's creating a new window as well. Is the width/height thing just for internal use and OpenGL won't create a visible context?
Cheers,
Euan
Icarus Scene Engine. OpenTK-based 3D simulation & games solution:
http://www.pointscape.com.sg/joomla
See Icarus videos on YouTube at:
http://www.youtube.com/user/emacinnes?feature=mhw5
Get the SVN source code at:
http://www.sourceforge.net/projects/icarus
Latest Downloads:
http://www.sourceforge.net/projects/icarus/files
Re: Load Textures while rendering (Multithreaded)?
New windows remain invisible until you call
INativeWindow.Visible = trueor call theIGameWindow.Run()method.That's all there is to it, actually. The new context will automatically be shared if possible (for best results try using the same GraphicsMode and the same context flags and version).
Re: Load Textures while rendering (Multithreaded)?
Right, I've got that far, pretty much as above and what it's doing now is causing an access violation on all OpenGL commands.
Euan
Icarus Scene Engine. OpenTK-based 3D simulation & games solution:
http://www.pointscape.com.sg/joomla
See Icarus videos on YouTube at:
Re: Load Textures while rendering (Multithreaded)?
Try adding
context.MakeCurrent(window.WindowInfo)in the new thread. Also make sure that both contexts are using the same driver (GL.GetString(StringName.Renderer/Vender)), otherwise access violations may occur. (This is uncommon but the new context might be falling back to Microsoft's software renderer for some reason.)Re: Load Textures while rendering (Multithreaded)?
That makes some progress, the makecurrent was required to get the Gl commands working, and the VBO objects are loaded on the second context and now there's no hitching at all during loading, but the textures aren't being shared for some reason, they seem to remain on the background context, haven't figured that one out yet.
Euan
Icarus Scene Engine. OpenTK-based 3D simulation & games solution:
http://www.pointscape.com.sg/joomla
See Icarus videos on YouTube at:
Re: Load Textures while rendering (Multithreaded)?
Alright, added the missing MakeCurrent call to the documentation.
I can reproduce the sharing failure here (Nvidia card) and have added some debugging code to the 1.0 branch to print an error code when sharing fails (debug mode, only). My error code is 170 (resource busy), which might indicate that the main context should not be in use before creating the background context. Investigating.
Edit: indeed, if you make the main context non-current, wglShareLists succeeds. This is not mentioned in the relevant documentation, great. Fixed the code above (and in the documentation) to reflect this discovery.
Re: Load Textures while rendering (Multithreaded)?
Fantastic! that did the trick brilliantly, now it all loads as smooth as butter. Some of the eLearning projects that happen with Icarus have over 80+ textures in them, and that's either a huge load time waiting for enough "free frames" to upload, or having an unresponsive UI to upload them, so it's all silky-smooth now on the asynchronous thread.
That's one beer* I owe you.
*beverage of choice if beer doesn't work.
It's no fun if things are mentioned in documentation. Makes life too easy :-)
Euan
Icarus Scene Engine. OpenTK-based 3D simulation & games solution:
http://www.pointscape.com.sg/joomla
See Icarus videos on YouTube at:
Re: Load Textures while rendering (Multithreaded)?
It may be good to point out you need a GL.Flush() after the "// Perform your processing here". My last few textures were showing up black because OpenGL was not recognizing them until theFLush.