Setting Swim volume was not fast enough.
Also added event handler to reset movement mode if my actor thinks its should still be swimming and its movement mode is not set to swimming.

The issue is now unnoticeable and looks like a natural bob in the water.