How can I deselect specific points from a selection?

There is an Deselection Mode that, when activated, allows any subsequent selection tool to work in reverse, subtracting points from the current selection.

Additionally, the Invert Selection command toggles the status of all points, selecting what was unselected and vice versa.