Racial relations

Before continuing I would to say that this is not to be racist in any way and is merely an observation. Why is it that only white people can be considered racist? I mean that even if an African American is hateful towards a white man it is not considered racism at all, but if the white guy turns around and says the black man was racist the world turns and says the white guy is racist. All I am saying is that can't anyone be racist not just white people, not that anyone is racist or I have anything against any race of any sort, but we know there is racism and we have to realize that not just the majority can be racist. And if I am called racist for this than that will just prove my point.