Well, if you can't trust your Players, you have the least rules-knowledge in the group and you can't identify whether behavior is acceptable or inappropriate, then you really shouldn't be DMing in the first place. That goes for any system.

But there is a lot of room for Players who don't or can't optimize in D&D 3.X. I run what I consider a fairly high-op game (average encounters are usually CR+4 to CR+8, mooks are optimized for group tactics, monsters get feats/spells/ability scores shuffled around to make them more effective, etc) but there are always a handful of players who need to add up their Attack Bonus every round, and they do just fine. Part of being a Party means looking out for each other and giving the other PCs room to shine; there's nothing parasitic about having to ask for help every now and again with rules issues. A DM who lets a Player sit there feeling useless is, almost by definition, not doing their job.

Obviously it isn't wrong for you to move to a less optimization-heavy game like nWoD or 4e, but if you really enjoy D&D I recommend finding a group which isn't made entirely of *******s and being "the new guy." As long as your RP is strong and you're willing to ask for help with mechanical issues, it shouldn't be a problem.