Converting ssh keys from old formats

L ike a lot of people, my main experience with private keys has come from using them for SSH. I’m familiar with the theory, of course - I know generally what asymmetric encryption does,

and I know that it means a compromised server can’t reveal your private key, which is nice although if you only ever use a given private key to SSH into your server and the server is already compromised, is that really so helpful?

What I was less aware of, however, was the various ways in which private keys can be stored, which rather suddenly became a more-than-purely-academic concern to me this past week. I had an old private key lying around which had originally been generated by AWS, and used a rather old format,

and I needed it to be comprehensible by newer software which loftily refused to have anything to do with such outdated ways of expressing itself. No problem, thought I, I’ll just use ssh-keygen to convert the old format to a newer format! Unfortunately this was frustratingly difficult to figure out, so I’m writing it up here for posterity and so that I never have to look it up again.

Preamble: Fantastic Formats and Where to Find Them

If you’re like me, you’re probably aware that private keys are usually delivered as big blobs of Base64-encoded text prefaced by headers like -----BEGIN OPENSSH PRIVATE KEY----, and for some reason never use file extensions.

There are three common formats you’re likely to encounter in the wild:

1. OpenSSH-formatted keys, which start with BEGIN OPENSSH KEY
   Plus the leading and trailing five dashes, but I’m tired of typing those out.
   and are the preferred way of formatting private keys for use with SSH,
2. PCKS#8-formatted keys, which start with BEGIN PRIVATE KEY or BEGIN ENCRYPTED PRIVATE KEY, and
3. PEM or PKCS#1 format, which starts with BEGIN RSA KEY.

The oldest of these is PEM/PKCS#1 - the naming is a bit wishy-washy here. “PEM” when applied specifically to key files for use with SSH, i.e. the way that ssh-keygen uses it, seems to refer to this specific format. But “PEM” more generally is actually just a generic container for binary data that gets used a lot whenever it’s helpful for binary data to be expressible as plaintext. In fact, all of the private key formats I’ve ever seen used with OpenSSH

are some form of PEM file in the end.

Whatever you call it, however, this format has the major limitation that it can only handle RSA keys, which is why it fell out of favor when elliptic-curve cryptography started becoming more popular. The successor seems to have been PKCS#8, which is pretty similar but can hold other types of private keys. I haven’t researched this one in quite as much detail, but I’m guessing that it also is a little nicer in how it handles encrypting private keys, since when you encrypt a PKCS#1 key it gets a couple of extra headers at the top, like this:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,0B0A76ABB134DAFEB5C94C71760442EB

tOSEoYVcYcVEXl6TfBRjFRSihE3660NGRu692gAOqdYayozIvU9xpfeVCSlYO...

whereas when you encrypt a PKCS#8 private key the header just changes from BEGIN PRIVATE KEY to BEGIN ENCRYPTED PRIVATE KEY before starting in on the Base64 bit.

Both PKCS#1 and PKCS#8 use the same method for encoding the actual key data (which, when you get right down to it, is usually just a set of numbers with particular properties and relations between them): ASN.1, or Abstract Syntax Notation One. ASN.1 is… complicated. It seems very flexible, but it also seems like overkill unless you’re trying to describe a very complex document, such as an X.509 certificate, or the legal code of the United States of America. But OpenSSH, it seems, longed for simpler days, when just reading a blasted private key didn’t require pulling in a whole pile of parsing machinery, so it struck out on its own, and thereby was born the OpenSSH Private Key Format. This format does not seem to be described in any RFCs, in fact the only detailed descriptions I can find are a couple of blog posts from people who figured it out for themselves from the OpenSSH source code.

I think this format can support every type of key that OpenSSH does, although I haven’t personally confirmed that.

The ssh-keygen Manpage is a Tapestry of Lies

So, I thought, I can use ssh-keygen to convert between these various and sundry formats, right? It can do that, it has to be able to do that, right?

Well, yes. It can, but good luck figuring out how. For starters, like many older CLI tools, ssh-keygen has an awful lot of flags and options, and it’s hard to distinguish between which are modifiers - “do the same thing, but differently” - and modes of operation - “do a different thing entirely”. The modern way to handle this distinction is with subcommands which take entirely different sets of arguments, but ssh-keygen dates back to a time before that was common.

It also dates back to a time when manpages were the primary way of communicated detailed documentation for CLI tools,

which you’d think would make it possible to figure out how to convert from one private key format to another, but oh-ho-ho! Not so fast, my friend. Here, feast your eyes on this:

-i      This option will read an unencrypted private (or public) key file in the format specified by the -m option and print an
        OpenSSH compatible private (or public) key to stdout.  This option allows importing keys from other software, including
        several commercial SSH implementations.  The default import format is “RFC4716”.

Sounds great, right? Import private keys from other formats, i.e. convert them to our format, right? But it’s lying. The -i mode doesn’t accept private keys at all, that I’ve been able to tell, whatever their format. Giving it one will first prompt you for the passphrase, if any (so it’s lying about needing an unencrypted input, although that’s not a big deal) and then tell you bluntly that the your file is not in a valid format. The specific error message varies slightly with the particular format - attempting to give it a PEM file (with the appropriate option) returns <file> is not a recognized public key format, PKCS8 gets unrecognised raw private key format, and speicfy OpenSSH format just says parse key: invalid format. So really, the only thing this mode is useful for is reading a public key in some PEM-ish format, and spitting out the line that you can used in authorized_keys - the one that starts with ssh-rsa, ssh-ed25519, etc.

Enlightenment Ensues

But wait! The -i option mentions that the formats accepted are specified by the -m option, so let’s take a look there:

-m key_format
        Specify a key format for key generation, the -i (import), -e (export) conversion options, and the -p change passphrase
        operation.  The latter may be used to convert between OpenSSH private key and PEM private key formats.  The supported
        key formats are: “RFC4716” (RFC 4716/SSH2 public or private key), “PKCS8” (PKCS8 public or private key) or “PEM” (PEM
        public key).  By default OpenSSH will write newly-generated private keys in its own format, but when converting public
        keys for export the default format is “RFC4716”.  Setting a format of “PEM” when generating or updating a supported pri‐
        vate key type will cause the key to be stored in the legacy PEM private key format.

Notice anything? I didn’t, the first eleventy-seven times I read through this, because I was looking for a list of known formats, not a hint that you might be able to use yet another option to do something only marginally related to its core functionality. So I missed that “The latter may be used to convert beteween OpenSSH private key and PEM private key

formats”, and instead spent a while chasing my tail about RFC4716. This turns out to be not very helpful because it’s exclusively about a PEM-type encoding for public keys, and doesn’t mention private keys at all! OpenSSH seems to internally consider RFC4716 as the public-key counterpart to its home-grown private-key format, but this isn’t explicitly laid out anywhere. Describing it as “RFC 4716/SSH2 public or private key” is confusing at best, because as we’ve established RFC 4716 doesn’t mention private keys at all. “SSH2 private key” isn’t obviously a thing: RFC 4716 public keys have the header BEGIN SSH2 PUBLIC KEY, but OpenSSH-formate private keys don’t say anything about SSH2. The only way to figure out how it’s being interpreted is to note that whenever ssh-keygen accepts the -m option and happens to condescend to operate on private keys instead of public keys, giving it -m RFC4716 produces and consumes private keys of the OpenSSH flavor.

Anyway. The documentation is so obtuse here that I didn’t even discover this from the manpage at all, in the end. I had to get it from some random Github gist that I unfortunately can’t find any more, but was probably written by someone just as frustrated as I am over the ridiculousness of this whole process. The only way to change the format of a private key is to tell ssh-keygen that you want to change its passphrase? It’s git checkout all over again.

At first I wondered whether maybe this is intentional, for security reasons, so that you don’t accidentally remove the password from a private key while changing its format. But finally I don’t think that makes sense, since if it’s encrypted to start with then ssh-keygen is going to need its passphrase before it can make the conversion anyway, in which case there’s no reason it can’t just keep it hanging around and re-encrypt with the same passphrase after converting.

Most probably this is just a case of a tool evolving organically over time rather than being intentionally designed from the top down, and sure, that’s understandable. Nobody sets out to create a tool that lies to you on its manpage

and re-purposes modes of operation for other, only marginally-related operations, it just happens gradually over time because updates and new features are made in isolation, without consideration of the whole.

Imagining a Brighter Tomorrow

But it doesn’t have to be this way! Nothing (that I can see) prevents the -i option from being updated to accept private keys as well as public keys: it’s clearly perfectly capable of telling when a file isn’t a valid public key in the specified format, so it it seems like it could just parse it as a private key instead, and keep going if successful. Or an entirely new option could be added for converting private keys. -c is already taken for changing comments, but there are a few letters remaining. I don’t see a -j on the manpage, for instance, or an -x.

I realize that an unforgiving reading of my travails in this endeavour might yield the conclusion that I’m an idiot with no reading comprehension, and that the manpage clearly stated the solution to my problem all along, and if I had just RTFM

then I could have avoided all this frustration, but that seems a little unfair to me. When you’re writing the help message or manpage for your tool, you should expect that people will be skimming it, looking to pick out the tidbits that are important to them right now, since for any tool of reasonable complexity 95% of the documentation is going to be irrelevant to any single user in any single situation.

How could the manpage be improved? Well, for starters, it could not lie about the fact that the -i option doesn’t do anything with private keys at all. If it were really trying to be helpful it could even throw in a hint that if you want to work with private keys, you’re barking up the wrong tree. Maybe it could say “Note: This option only accepts public keys. For private key conversions see -p and -m.” or something like that. The -p option should probably also mention somewhere in its description that it happens to also be the preferred method for converting formats, since right now it only talks about changing passphrases.

Anyway, thanks for listening to my TED talk. At least now I’ll never forget how to convert a private key again.