Intent does matter; that's why manslaughter isn't the same as murder.

Racism is bad; almost anything we can do to minimize it will benefit everyone.

But the dichotomy you're advocating here is a gross oversimplification. Where there is no obvious intent, one must look to societal, experiential, and psychological factors that influence a person's behaviour - many of which are beyond a person's control.