You can simply remove all users, that is default deny, once you take ownership/permissions and remove inheritance.