So you do not put passphrases on your ssh keys?

Not always.
��
Because you don't give the private key away why protect it? So imagine you have development process that includes a jenkins that needs to build the source and therefor needs access to the repository. What do you do?

I give it a password-less ssh key, encrypted in some manner. (I do not know what Jenkins offers, but both Travis and GitLab(Runner) support file/data encryption.)
Adding the build server a password on top of the key would make no difference. If you have access to one, you have the access to the other.

Peter