So I guess that’s actually several questions, and they each have different answers.

Why does combat feature heavily in D&D? It doesn’t. Or at least, not necessarily. How much or little it features is dependent on your DM.

Ok, so why has it historically been featured heavily? Because of D&D’s lineage. The game evolved mechanically from wargames, where combat was the whole thing, and thematically from works like Conan the Barbarian and Tolkein, where fighting monsters featured prominently.

Why so many types of monsters, then, if works like The Hobbit only had a half dozen or so? Because The Hobbit is a single story, whereas D&D is a framework for creating lots of stories. Maybe one short campaign or a campaign arc has as many monsters as a Tolkein story, but then you go on to the next arc, the next campaign, and you need something new. You can obviously recycle lots; orc bandits are different from orc soldiers are different from orc cultists. But with (tens of?) thousands of games going on continuously, year after year, there’s always a demand for new content to slot in, and monster design is often a handy thing for DMs to outsource. Hence, there are a lot of kinds of monster because there is demand for them.

Are you asking why there are so many kinds of monsters, or why monsters appear so frequently in the campaigns you’ve played in?

Looking good! Those pirates have a pretty sweet setup!

Mhhh…I believe in the right tool for the right job. So there isn’t really an ideal system, for instance, just the best system for what I’m trying to accomplish in a given game.

Sometimes that can be fun, but only if everyone at the table is onboard for a wild tangent. If the other players are bored as shit while the special snowflake starts a unicorn breeding operation, it’s time to use that No. And you, the DM, are included in that too; if your players want to drag you off to write every book in the library and that’s not fun for you, you have the right to say “hey maybe you should play the game I made for you instead.”

Addendum to the “Are you sure you want to do that” bullet: if a player ever does something that seems nonsensical to you, ask them what they expect to achieve by doing that. Understanding their motivation is often what resolves the miscommunication and/or allows you to steer them towards a better way to do what they’re trying to do.