“The man” is referring to people in positions of authority such as politicians and business executives. You’ll sometimes hear people say that “the man” is trying to “keep them down”.