Not to me :) It's like I tried to explain lately with the password encrypted with a password. If it comes to security you can always make things more complicated and building chains. In the end the security is defined by its weakest link. So you need to find out where are weaknesses.
Assuming that if you can login via ssh it doesn't mean anyone else can. Basically there are two possibilities. Either ssh has a security hole or you loose your key with password. What else should happen? If ssh has a security hole it doesn't matter which user tries to log in. And using another user means that someone is on the machine. I think getting access to a machine is a _much_ more difficult than to get root access if you are loggend into a machine.
So my advise would be: Smalltalk and security are the two ends of the productivity scale. While smalltalk can make you really productive, security can keep you from achieving anything. Btw. that is the reason why I decided to go for programming instead of security 15 years ago.
So instead making it complicated for yourself without gaining anything you need to define exactly what is the scope that you want to protect and against which measuements you want to protect yourself.
If you don't trust ssh order a KVM switch with remote capabilities at your provider. With that you can have access to the console of the computer and you can turn off ssh completely. But how do you copy files to the machine then�.??? Ah and btw. while we are at it. I hope you don't use linux. Why designing a tight security strategy if you put it on top of an immature operating system? Use OpenBSD instead!
I hope you can see what I'm trying to tell you. There is no such thing as security. There is only a probability that someone breaks into your machine. Lowering this probability means most of the time making the system a lot more cumbersome. So for every project there is a setting that leverages best for the needs. But before that you have to know what is really necessary so put a probability onto every measurement and not try to be too paranoid.
Norbert