What will happen in the follow situation:

Player A: Activate Return of the Different Dimension and special summoned 5 monsters.
Player B: Activate Book of Moon on one monster.

Will the monster returned out of play that's face-down on the field. I think the monster stay's one the field because Magical Scientist has something like the same effect.

------------------------------------------------------------------------
Return from the Different Dimension
Normal Trap
Pay half of your Life Points. Special Summon as many of your monsters as possible that have been removed from play on your side of the field. During the End Phase, remove from play all monsters that were Special Summoned by this effect.

- There are no published rulings for "Return from the Different Dimension".

------------------------------------------------------------------------
Magical Scientist
Effect Monster (Spellcaster / DARK / 1 Star / ATK 300 / DEF 300)
Pay 1000 Life Points to Special Summon 1 level 6 or lower Fusion Monster from your Fusion Deck in face-up Attack or Defense Position. That Fusion Monster cannot attack your opponent's Life Points directly, and is returned to your Fusion Deck at the end of the turn.

- You can use "Magical Scientist"'s effect multiple times in the same turn.

- "Magical Scientist"'s effect does not target.

- If the Fusion Monster is flipped face-down with "Book of Moon", or removed from play with "Dimensionhole" or "Interdimensional Matter Transporter" or "Different Dimension Gate", the effect is reset and the Fusion Monster does not go back to the Fusion Deck, and is not prohibited from attacking your opponent's Life Points directly.

- If you Special Summon "Reaper on the Nightmare" with "Magical Scientist", it cannot attack your opponent's Life Points directly (but can attack a monster).

- You cannot activate "Secret Pass to the Treasures" targeting a Fusion Monster Special Summoned with "Magical Scientist" because it cannot attack your opponent's Life Points directly that turn.

- If "Skill Drain" is activated after a Fusion Monster is Special Summoned with "Magical Scientist", the effect of "Magical Scientist" (that makes the Fusion Monster unable to attack directly and have to go back to the Fusion Deck) has been applied already, and is not negated, so the Special Summoned Fusion Monster still cannot attack directly and must go back to the Fusion Deck.

- If "Jowgen the Spiritualist" is on the field, you cannot activate the effect of "Magical Scientist" (so you cannot even pay the cost to try and activate "Last Turn", for example).

Your reasoning is correct; the monster you flipped face-down would remain on the field.