I am still unsure about this...

Player A plays Pot of Avarice. He then checks his Graveyard and noticed there are only four(4) monsters and not five(5) which is required for Pot of Avarice.
Player B says Player A's Pot of Avarice was misactivated and thus it's sent to the graveyard.
Player A argues that even though he played, since he only had four(4) monsters, then his Avarice could not have activated thus it would be set face-down.

Just makin' sure, would the Avarice go to the graveyard, or be set face-down.

Anytime a card is activated without legal activation it is returned to the position it was in prior to activation. If the card was set on the field it will be flipped back face down, if the card was in hand it will be returned to the hand.

It does not go to the graveyard, how could it if it couldn't be played in the first place?

This is why, things have to be checked before time, bigger problems can emerge from such events.

The problem is, people mistake when a situation is "similar", but not quite the same.

If your opponent activates Cold Wave in the Main Phase of his turn, and on your turn, you attempt to play Change of Heart, it is a illegal activation since neither player can activate the effect of a Spell or Trap Card, and it would return to hand or Set to field.

Now, lets say your opponent activates Imperial Order in the Main Phase of his turn, and on your turn, you attempt to play Change of Heart. NOW it is a legal activation because Imperial Order only negates the effect of Spell Cards and doesnt prevent them from being activated, so the card would go to the Graveyard.

That's where the confusion comes in from. Players mistake these two instances for the same. You cant activate things during Cold Wave. You can activate things during Imperial Order, and people try to say that they "can't", and take back the Spell Card, when it was legally played.