Ceph Bogus Error: “Cannot allocate memory”

More precisely, when attempting to add a new host machine using this command:

ceph orch host add osd2

The response was:

Error EINVAL: Can't communicate with remote host ``, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory

I kept adding memory to the “osd2” VM and same results. Finally, I started talking to the osd2 machine itself. Various queries looked generally OK, so finally, I did this:

cephadm prepare-host --expect-hostname osd2

Ceph is not very good at hostnames. In the *n*x world, hosts can have many names, and the names can be simple or fully qualified domain names (FQDNs). I’d set the primary hostname for this machine fully qualified, meaning “osd2.mousetech.com”.

When I did the prepare-host with “expect-hostname” set to the hostname that ceph knows it by (non-qualified), then the ceph orch host add command kicked off a lot of stuff on osd2 that showed up in the system log. I was running a journalctl -f on osd2 at the time, so I saw it happening.

At the end, it still failed with “Cannot allocate memory”, but I gave it one more shot, since a lot of heavy lifting had been performed and got back an “added host ‘osd2′” message. Success!

So it wasn’t ultimately about insufficient RAM, after all, just about host naming.