Using the mouse to select multiple entities is an extremely useful tool. To select more items you simply hold shift and keep selecting. Unfortunately if you make a mistake and select unwanted entities, there is no simple way to remove them.
A simple method could be to hold the shift key and 're-select to de-select' the entities. So by holding shift or another simple key, you can then remove items not required in the selection.
You ARE able to keep holding SHIFT or CTRL key and individually select the "unwanted entities" items one by one.