Stripping down a desktop-oriented distro down is probably more effort than it’s worth, so I’d just recommend disabling (when not working on the system) whatever Mint DE that would start-up at boot. Unless disk space is an issue, there’s no reason to do more than this. Otherwise, this endeavor largely becomes a (frustrating) reversion of Mint into the Ubuntu Server.

If you don’t need software from the Ubuntu repositories, Debian would be the best option for a bloat-free, non-Canonical server experience.