I was playing with my main loop and calling ProcessEvents() 30 times a second. I then profiled my code and found that process events was taking up 40% of my processing time. Switching to the Tao Sdl library, and using the same setup, the processing of events only takes 1.5% of my processing time.
I am just mentioning it in case someone wanted to take a look to see why it is much slower. I'm fine with using the Sdl to create my main window and handle events.