Once your business has been properly set up with operational programs and effective management techniques, you can reduce the number of hours you spend on the job. I refer to this stage as "maintenance mode" - you maintain the business operations, the business begins to operate more on its systems.
That doesn't mean you can arbitrarily kick back after struggling for years to make the business successful. However, if you have adopted systems that assist you in keeping a tight rein on the company, reducing your time spent at the office is not only possible, it makes a lot of sense.