If a character is adjacent to a colossal character, but the normal size one is on the rim of elevated terrain and a colossal figure down 1 level, are the considered adjacent because he is colossal?

No. Characters on different levels of elevation are not adjacent. Even if they are on opposite sides of "ladder" arrows.

Caveat: Multi-base figures can occupy multiple elevations at the same time. So the colossal could be on level 1, and another elevation making it adjacent.

Jsligh is correct. The colossal can punch the normal sized figure, but the reverse is not true (short of corner cases). Doesn't seem quite right or "logical" in the real world, but that's why I wrote my signature.

To note, while stairs don't make figures adjacent, they can punch each other because the rules say so. Since they aren't adjacent on stairs, they can also shoot each other if they'd like.