I went in the back to tell him the flight was full, and Happiness is being married to someone Dutch shirt, so either he’d have to travel where he was, or get offloaded. As the “I will complain, you should downgrade somebody to accommodate me in the business class” rant went on, I took his baggage details and offloaded his bag in the next half hour. During this time, he was shouting, complaining, being ridiculous, and… woke up the infant who then began to scream uncontrollably. He then said something similar to your question, that he had the right to his peace. By this time, the gate staff I’d called was onboard. The captain shared my opinion and agreed to offload the troublesome passenger.