While drawing an annotation, you can use the Esc key to cancel drawing. Optionally you can use the toggle button to remove the annotation after canceling.
Left Click to use selected tool