Thanks for the comprehensive reply!
do whatever it takes to "get to the good part" of whatever it is you are most interested in at the moment.
I tend to have the mindset that I need to code everything myself from the ground up so I know exactly what is going on. That's why C is so attractive for these types of applications- you can get down almost to the assembler level (almost) and have precise control over things.
For game programming that may not be necessary and I should take advantage of higher level languages like C# and existing game engines. I think I'll take this to finally learn C#.
I think that having an understanding of what is going on all the way down is super important to writing efficient code, but I also take the view that once I understand it, I can work much more efficiently in systems where I don't have to manage every detail. With C# it is nice because you can see the MSIL code if you really want to, and there are lots of known optimizations on how to write C# code that is efficient or able to minimize garbage collector hits, etc. That's not the same as working with the bare metal, sure, but it's a lot faster and safer (in terms of memory leaks, etc), and it's something that requires just as much knowledge and attention to detail in order to get out the absolute best performance, etc.
I really have trouble using most third party libraries for the same reasons you stick to the bare metal, so I can sympathize. I look at their code, or look at the speed of their code, and just see "bloat! bloat!" With some notable exceptions, like the lidgren network library, which I like very much through and through. And SlimDX seemed to be that way, too. But as a general rule...
As a far as picking a game engine. What are the characteristics that you need to look for besides that it's compatible with the language you plan to program with? For instance, are some game engines more suited towards FPS, PRG, RTS, TBS, etc... I'm still trying to get my arms around exactly what a game engine will provide. I envision that they would provide interfaces to the user inputs, screen outputs, in game unit classes, physics, etc. It that correct?
Well... that answer really varies quite widely, too. Obviously language is one factor, but so too is what the engine provides. Do you want an engine where you can arrange your scenes in 3D or 2D and then add on little scripts, without really coding a lot of the central stuff? There are engines for that. Do you want an engine that has no GUI for the development environment, but which just provides you a code wrapper for things like graphics, sound, and input? There are engines for that as well, and plus there are libraries like DirectX or OpenGL themselves, and things like OpenAL and so on. You could cobble together your own cross-platform engine from just stuff like OpenGL, OpenAL, and similar.
The other big thing is documentation. I don't like working with a lot of the smaller libraries because they are often poorly documented. There's also bugs to consider: if a smaller library is open source, great -- you can fix any bug that crops up if you have to. Another reason I like the lidgren network library. On the other side, if you get some big engine without a source, then you're at the mercy of the central team to fix their own bugs. That might be okay or it might not be.
There are some engines that are literally less powerful and which can't do a lot of the fancier code things, but they trade that for ease of use and a WYSIWYG type of visual editor. I guess it wouldn't be polite to name names in terms of that sort of thing, but it's fairly obvious when you look at a lot of those. They are kind of "game development for hobbyists," though there are a few folks that will inevitably make something really awesome no matter what tools they are given.
Then there's actually the platforms the engines can run on, and the licensing costs to run on the various platforms. The costs can be quite steep: things like 25% of your income on the game, or even $30,000.00 for a license for some of the consoles are pretty common with the major AAA quality engines. And if you're making a AAA game, I imagine those are worth it, too.
Pretty much all engines will give you some sort of input, graphics, and sound capabilities. But they vary widely in the details:
1. Do they have 3D sound? Do you care?
2. How is sound playback handled? On a secondary thread or as part of the primary, streamed or not, etc? What options do you have in terms of when and how sound stuff is loaded into RAM?
3. In a few rarer cases (with smaller or custom engines/libraries mostly), how compatible is the sound functionality with most sound cards or platforms? In a few even rarer cases, does it even have sound capabilities at all?
1. Are 2D or 3D supported? If it's 2D, is it using something older and less functional like DirectDraw, or is it doing it all in software (very functional but fairly slow so low framerates), or is it doing it in a 3D-hardware accelerated fashion (really the fastest way to do it, and quite functional as well, but also the hardest to use by miles). If 2D isn't natively supported at all (which is common), and you want to do 2D, can you get at the bare metal of the DirectX/OpenGL calls in enough of a direct fashion to roll your own?
2. If 3D is your thing, how up to date is the engine on supporting the latest shaders and other functionality, if you care?
3. If you're doing any hardware acceleration at all, how easy is it to tell the hardware caps of the current system GPU and adjust functionality accordingly? Almost anything on OpenGL or DirectX supports querying caps (capabilities), but having graceful failover when certain shaders aren't supported is something I was particularly happy with in Unity, for instance.
1. Keyboard and mouse are pretty much universal and pretty easy to do. But they actually can be messed up to a surprising degree by some libraries, depending if they are event-driven or polling-based, and how much time you want to spend writing a wrapper on top of either. I've done it both ways, and find either works well enough, really.
2. Gamepad support is not universal, and nor are all the functions of gamepads. Things like analog sticks or extended sets of buttons, or specific support for something like force feedback, etc, really vary. I miss not having force feedback since the switch to unity, for instance; I used to use that in SlimDX.
3. If you care about touch devices, support for that varies even more. Just treating someone's finger as a mouse cursor doesn't work all that well in most cases.
And there's other considerations like how often they update their library, how clean it seems in general, if it runs fast or seems bloated, if it has a physics library if you want that, etc, etc. We looked at over 30 different engines when we chose Unity 3D, and for our purposes we are really pleased with it. Though it does have some notable downsides that I've discussed in other threads, and we wound up spending a solid couple of months just "rolling our own" on top of the base unity engine, bypassing all their physics and networking and most of their entity logic, and implementing our own on top of their raw graphics capabilities.
I think there is no engine that is so perfect and amazing that everyone should use it, so the answer to what engine should be used is always really something that comes down to taste. I think that Unity is really great, and definitely my favorite engine these days, but their support is pretty bad and they really make you work to get at the bare metal (and sometimes you can't at all, to my frustration). And to get to the bare metal will require their pro license, which is definitely not free. With the Unreal engine, for instance, you could get started for free and then they just take a cut from profits if you get to that point of selling, and you could even stay in C++.
So it really varies a lot, and is something you'll just have to research to see what strikes you as the best fit for you. There are some really specific and interesting engines from small studios for specific types of projects, too. Like the Love 2D engine, for instance. There's a lot of cool things, and I know quite well how overwhelming it can be to even evaluate all the choices, but hopefully that helps somewhat.