The chalky grit of unsealed lime plaster against your palm is proof that perfectly smooth drywall was a mistake. Buildings need enough tooth to catch dust and daylight, otherwise you’re just living inside a render.