Don't remove entries from an existing scancode keymap
If we can't find the X11 keysym, it's likely that either the keysym is NoSymbol, in which case we won't hit it anyway, or it's been mapped to a character, in which case the existing mapping is correct for the scancode and the character will be reflected in the keycode mapping.