when mole attacks legendary jujitsu master in defense mode which effect works?

Mole's effect activates much sooner.

or more to the point, Mole never goes into the Damage Step, so you never even know what you are attacking. Leg Jitsu Master needs to be flipped face up for his effect to be activated. Flipping a card face up only happens in the damage step.

To be even more correct, Mole DOES go into the Damage Step, Mole just activates at the very, very beginning of it.