Let's say that my opponent's Arkham Asylum Guard captures one of my figures. Then, I capture my opponent's Arkham Asylum Guard with my own Arkham Asylum Guard. Where would my original, captured figure end up? Would it go back on the board, or would it stay captured in some sort of purgatory?

Mostly this is an opinion without basis. I would assume your original figure would be placed in an adjacent square, similar to as if it had been released.

If you release your captive in your starting area your figure/his captive is defeated ergo your guy is released, no?

The player's guide addresses this