They are entitled to ask you to leave in exactly the same way that I'd be entitled to ask drivers to stop using the public highway that goes past my house.
It is a SSSI, but that doesn't affect navigation.
It's tidal, you can paddle it. This isn't the disputed part of river access rights law. This is commonly accepted.
There are exceptions to the rule (the Beaulieu river as en example, where a fee must be paid.), but the Test is not one of them.
That said people rarely bother to paddle up there, so the wardens are probably unaware that there is a navigation right. This doesn't excuse their behaviour.