The second part of the text on Cosmic Spiderman reads "At the start of the combat phase, Spider-Man gets +1 ATK and +1 DEF this turn for each non-stunned character target opponent controls."

Say combat phase starts and my opponent has 4 non-stunned characters, giving Cosmic Spiderman +4/+4. I have my other charatcers attack first, stunning two of his. Cosmic Spiderman still has +4/+4 and not +2/+2 because the check for non-stunned characters only happened during the start of the combat phase. Is this correct?

Correct, though we can be even a bit more specific.

At the start of the combat phase, Cosmic Spider-Man's power puts a triggered effect on the chain. When this effect resolves from the chain, that is the moment where you determine Spider-Man's bonus for the turn. Afterwards, it doesn't matter if the number of non-stunned characters changes.

That is correct. Spider-man would have +4/+4 for the rest of the turn because the beginning of the combat phase is the only time the check is made.