Was Australia Invaded By the english?

i understand that the british wanted to use australia as a dumping ground for the filth of society [ rapist, murderers, peodphiles etc ] but i don't see how the human cargo had anything to do with the invading part even though they did kill thousands of aboriginal ppl. The convicts didn't want to invade they were released like vermin to infest Australia by the unwitting british. So how could you say criminals "invaded" australia when most of them were mentally disabled or just sick in the head how does a population of the lowest form of civilization hatch a plan to invade australia if they were stupid enough to get caught in the first place?