You were right. I removed the bezel and the 2 plastic pins on diagonal opposite corners did break. This renders the bezel and base plate useless. The pins are either glued in or have a non-removable taper fit. The base plate, which holds the switch, and the bezel are assembled after the switch is wired and the base plate is screwed to the wall. The screws are only accessible with the bezel removed.
The switch had too much grease in it and this prevented a good contact on occasion.
I was able to salvage the bezel and base plate by drilling 1/16 dia. holes in the location of the plastic pins. I made some shout metal pins and inserted them in the holes. Using these metal pins as guides I glued the bezel to the base plate with a small spot of glue at the two pin locations. Hopefully this will remove more easily next time.