Category: Stupid Linux Tricks

cloud-init “gotcha”

I was putting together a project using The Foreman to spin up and manage Amazon EC2 instances and ran into a problem. I could take an AMI and launch it, but I couldn’t ssh into it.

One of the major reasons why this was so was that cloud-init was silently failing and as a result, my ssh key wasn’t being installed.

The AMIs in question were built on top of Ubuntu 14.04LTS. The Foreman creates its own private access key to launch and control EC2 instances and it’s not accessible for general use. You have to supply your own private key if you want to login via ssh.

The recommended way to do that is to inject it via cloud-init. However, cloud-init wasn’t working right.

After a lot of experimentation, I discovered that the issue was in the attempt to also use cloud-init to set the simple hostname and fqdn of the newly-created host. Turns out that including the “host:” line like all the samples out on the Internet do causes the ENTIRE cloud-init file to be ignored. So in order to get my ssh key (AND hostname) injected, I just needed the “fqdn:” line.

Baby Steps with OpenStack

The OpenStack cloud platform is hot these days. Anyone can set up and run their own private cloud without too much difficulty.

Relatively speaking. You do need a huge chunk of RAM and a respectable amount of disk space, even for a minimal cloud. Also a x64-bit hardware VM capable CPU. But considering what you get, it’s not a bad payoff.

An openstack cloud has 3 types of nodes: control, compute, and storage. You need at least one of each, but a single OS instance can host any combination of them, so the simplest cloud would be an all-in-one server.

First Step

There are quite a number of components that make up these nodes – including some that are plug-replaceable, so the easiest way to get started is to use a springboard. One popular route uses a Vagrant VM to launch the DevStack ready-to-run server. This is a good way to get familiar with OpenStack, since everything’s already pretty much set up and running and you can launch it via VirtualBox on your desktop. Assuming you have at least 8GB RAM to spare, since the DevStack VM is going to eat up about half of that.

Second Step

Running a cloud on your desktop is pretty cool, but if you have aspirations on running a real cloud, you need real servers. Since I didn’t have that many spare servers with sufficient capabilities, the next step I did was again launch OpenStack in a VM, but using a KVM under CentOS 5.11. Why not 6 or 7? Primarily because I have legacy Xen VMs on its siblings and I’m not yet ready to migrate them to an OS that can’t host Dom0.

If you do not allocate sufficient RAM or disk space for the OpenStack VM, it may not install properly and almost certainly won’t work properly, and for the most part you’re not going to get much in the way of helpful messages. OpenStack is comprised of a whole raft of component products and there’s not much in the way of centralized detection and reporting of broken components.

Here’s what I used to create the basic OpenStack VM:

#!/bin/sh
VM=icehouse
IMAGES=/var/lib/libvirt/images

virt-install --name $VM \
	--hvm --ram 4096 --cpus 4 \
	--disk path=$IMAGES/$VM.img,size=6 \
	--network bridge:br0 \
	--os-type=linux --os-variant=rhel6 \
	--accelerate --vnc -v \
	--location=http://10.0.1.3/cobbler/ks_mirror/Centos6.5_x86-64/ \
	-x "ks=http://10.0.1.3/cobbler/pub/$VM.ks"

Note that this VM runs the IceHouse release of RDO under Centos 6.5. I tried Juno and CentOS 7, but it kept whining about running out of memory, at least up to about 3GB or so. The network bridge is my VM host’s bridge to the VMs the number of CPUs isn’t important, but I had a few to spare. The kickstart file is nothing special, but it does install and enable ntp and format the disks into a /boot (about 300M) and an LVM partition (everything else), containing a single Logical Volume for the OS.

caution:

A production all-in-one node needs a LOT more disk. You’ll want storage for the client disk images and working permanent storage,

Using PackStack

The PackStack package makes the job of setting up an OpenStack node a lot easier. It fetches the various component packages and uses Puppet to install and configure them. It also creates an “answers” file so you can replay the installation, if needed.

Under CentOS, the easiest way to get things going is to run packstack. My Kickstart had a post-install command to install the Icehouse Yum repository:

rpm -ivh https://repos.fedorapeople.org/repos/openstack/openstack-icehouse/rdo-release-icehouse-4.noarch.rpm

So the sequence once the VM came up post-install went like this:

  1. Install YUM plugin to enforce precedence on repo search/fetch 
    yum -y install yum-plugin-priorities
  2. Upgrade the OS 
    yum -y upgrade
  3. Reboot to get the latest kernel 
    reboot
  4. Install PackStack 
    yum -y install openstack-packstack
  5. Run PackStack 
    packstack --allinone --provision-demo=n

Once all that’s done, with luck, you can open up a web browser on the Openstack console.

Things that can go horribly wrong

The single biggest headache I’ve found with OpenStack is networking. Networking a collection of VMs is a major pain even without clouds. OpenStack raises the ante considerably, since you have 2 options for network stacks (legacy nova network or neutron), and all sorts of real and virtual device/network options. Which, if you’re not already well-read on the subject, you’ll have no clue which ones you should be using or how to set them up.

More on this later.

Beyond that, the most critical functions for OpenStack are the security/identity manager (keystone) and the messaging agent (defaults to rabbitmq, but replaceable). Without the identity manager, nothing can be accessed, without the messaging system, components cannot notify each other about important events. Fortunately, these two are less likely to screw up and appear to be easier to diagnose/repair.

The curse of the mad Puppet

I have been working with various things designed to allow me to control the mousetech.com domain assets in a more centralized way. One of them was to try and use Puppet to provision machines. Puppet is a fairly nice tool, but there are some unexpected pitfalls.

There are several ways to get puppet on a CentOS 5 server. If you’d a glutton for punishment, you can always pull down the source and build from scratch, but I don’t recommend that when just getting started. You can also pull it via YUM from the EPEL repository. Or, you can import the Puppet Labs repo in and pull from there.

I already have EPEL in my stock set of YUM repositories, so that’s what I went for first. In the beginning all was fun and games. Then I got more ambitions and started defining modules. This didn’t work. Worse, the sample documentation used commands that didn’t work. It was getting very frustrating.

It became obvious fairly early that some significant changes have been made to puppet and that what I had wasn’t the Latest and Greatest. That would have been OK, except that attempts to read the online documentation for the older stuff kept leaking back into docs for the newer stuff (a not uncommon problem, best handled I think by archiving the old docs as self-contained PDFs). On top of that, the version of puppet that I was running was sufficiently antique that much of the documentation had fallen off the website (see previous).

To add to the confusion, I wasn’t really sure WHICH version of puppet I was running, since their enterprise product doesn’t keep quite the same set of version numbers as their community version plus I suspected the version (2.6.18) in the EPEL RPM wasn’t indicative, either. I finally came to the conclusion that 2.6.18 – which is the version that Centos10 pulled actually correlates to the community 2.5 version, which is something like 2.3 in Enterprise versioning.

At this point, I went to the source: Puppet Labs and found out about their repo. Unfortunately a network-based RPM install failed for obscure reasons (I’m not sure whether I have lingering LAN issues or it’s them). Fortunately, I was able to wget and install locally. After which I was able to install a Version 3 Puppet, the documentation now matches the commands and module processing works the way they said it did.

One last fly in the ointment, though. It seems that nodes and classes share the same namespace. I was using the same name for my guinea-pug machine node and one of the packages it was trying out and while both node and component parsed, the actual execution was only done against the node – the package was silently ignored. I fixed this by changing the node name to its fully-qualified domain name.

 

[SOLVED] mail loops back to me (MX problem?) for virtual machine

Sometimes they just gang up on you.

I was migrating my sendmail server from a NAT address to a bridge address when it all started.

Xen has this really nasty habit of zapping your hardware MAC address if you don’t get the nat routing configure just right. There’s obviously some way to get it to revert, because occasionally for no obvious reason, the real MAC address will revert, but don’t try searching the web for an answer – all you’ll get is fruitless inquiries and flame responses (you shouldn’t be changing your MAC address, idiot!). Please. There are very good reasons why it’s useful to be able to set a custom MAC address. One place I worked coded their hardware asset IDs in the MAC to assist their DHCP server, for instance.

On the mousetech domain, I’d be happier if it didn’t happen. As it is, the MAC addresses of the primary and secondary NICs got swapped and I didn’t find out until I’d gotten most of the way through fixing things. So the former eth0 became eth1 and vice versa.

Shortly thereafter, outbound mail started bouncing with the infamous “mail loops back to me” message. Since I’d just done major relocation on the mail VM, I wasted a LOT of time messing around with sendmail options to no avail. Normally this message can be cured by putting in a valid MX record in DNS and/or adding all the mailserver alias names to the sendmail local-host-table (cw table). Not this time.

I was fairly sure that the problem had something to do with the fact that the physical host had been set up to forward all port 25 (smtp) requests to the mail VM and that somehow the wrong IP address was getting mixed in, but I could see the actual routing since it was all internal and specific to port 25 to boot.

Turns out I’d been sloppy when I fixed up the iptables forwarding. The correct version (after the NIC mixup) looks like this:

-A PREROUTING -i eth0 -p tcp --dport 25 -j DNAT --to-destination 10.0.1.8:25

Where I went wrong was in being lazy and omitting the NIC ID (eth0) when I repaired the damage that Xen did. As a result, BOTH NICs were being re-routed – the actual internet-facing NIC (which should be routed) and the internal DMZ bridge-facing NIC (which should not). As a result, traffic on port 25 for eth1 was being routed back on itself and sendmail complained.

Gourmet Recipe Manager problems under recent Fedora releases

The Gourmet Recipe Manager program is a very useful way to store and find recipes, but it has been essentially useless since about Fedora 11 (give or take).

The problem is that this app uses an SQLite database to keep its recipes and accesses the database using SQLAlchemy. Version 0.7 is seriously broken relative to Gourmet.

The cure, once you know it, is relatively easy – at least as long as you don’t have any other apps depending on SQLAlchemy (and I didn’t). First, check to see if you do:

rpm -q --whatrequires python-sqlalchemy

If gourmet is your only dependent (or you don’t care/feel brave), remove the 0.7 sqlalchemy package by brute force:

rpm --erase --nodep python-sqlalchemy

Download one of the 0.6 versions of sqlalchemy. If you have a 64-bit system, be sure to get the 64-bit RPM, which includes the 32-bit version.

Install the downloaded sqlalchemy RPM

rpm -ivh python-sqlalchemy-0.6.xxxxxx.rpm

Fire up gourmet. One of the easiest ways to see if it works is just to type a search. If it returns search results, you’re OK!

Making Apache mod_rewrite and the ajp Tomcat connector work together

Apache’s URL rewriting facility can be used to shape URLs piped to Tomcat, but it’s not as simple as it seems.

The basic action is straightforward. First incoming URLs are processed by mod_rewrite, then they are matched up against JkMount definitions to find the appropriate Tomcat connector to use.

The problem is, what Tomcat sees isn’t what you’d expect to have come out from the rewrite process.

What actually happens is that before being submitted to mod_rewrite, the incoming URL is converted to server-relative form. After rewriting, the resulting URL is then reassembled, including the parts that Tomcat doesn’t want or need. Including the Apache DocumentBase path.

To prevent this problem, just qualify the last rewrite rule in the process with the “PT” directive to let Apache know to “Pass Through” the URL without reassembly. So, for example:

RewriteRule ^/$ /mywebapp/index.jsp [L,PT]
JkMount /mywebapp/* ajp13

Howto: Installing TracDiscussion

TracDiscussion allows adding discussion forums to the Trac (http://edgewall.org) system. Unfortunately, there’s not a lot of info on how to to it.

In a perfect world, you’d just issue the command “easy_install TracDiscussion” and bam-you’re-done. In this world, not yet, anyway.

Installing TracDiscussion requires installing some prerequisite packages. If you’re running Trac 11 or later, you don’t need to install the web administraction package, since it’s now part of Trac. Yes, they say that, but rather ambiguously, so this is an unqualified statement. Secondly, you need a spam filter. You’re dealing with email, spam is now about 90% of all email, so you need a filter.

Installing the actual TracDiscussion plugin isn’t all that hard even lacking a functional direct easy_install. Just go to the SVN archives, pick the project directory whose name matches your trac version and use the Subversion export command to download it. Then run easy_install against the downloaded directory.

That’s only half the battle. At that point, restart your HTTP server to reload the Trac configuration. You should see the TracDiscussion plugin listed on the admin plugins page. Select all options. In particular, select the installation option. If you don’t select that one, the extra tables won’t get added to your database.

You’ll probably get an error page backin instructing you to run the trac-admin update command. Do that from a command shell. If using PostgreSQL it will probably tell you to disable database backup on the upgrade. Hopefully you’re doing regular PostgreSQL backups anyway, but you might want to kick off a manual PostgreSQL backup before doing the Trac upgrade just for good luck’s sake.

You may want to edit the [notifications ]section of your trac.ini file in order to configure email.Then restart HTTP. And backup the newly-updated database.

With luck, “discussion” will now appear as one of the items on your Trac menubar and there will be a Discussions management section on the sidebar menu of the admin page.

Trac: Permission denied connecting to PostgreSQL Server

Another one of those obscure things that a web search didn’t turn up. Using the “psql” command-line utility, it was possible to connect to and access the PostgreSQL Trac database. However, the webapp failed, with “cannot connect to server: Permission denied”. No amount of tweaking the firewalls or the pg_hba.conf file would help.

It turns out that the permission wasn’t PostgreSQL, it was selinux. It was forbidding psycopg from opening a database connection while running inside Apache. It wants a “allow httpd_t postgresql_port_t:tcp_socket name_connect;”.

A message shows up in /var/log/audit/audit.log – if you think to look there.

Embedding graphviz-generated scalable graphics in an OpenOffice Document

You can insert graphics into an OpenOffice Writer document – or any of a number of other document editing programs. It’s generally a simple “Insert picture/from file…” menu option.

Getting it not to look like garbage is the hard part. Most images are bitmaps and scaling bitmaps usually results in images that are either “chopped” or have lines that “stairstep”. And, of course, the text can be really ugly.

There is a class of image file that scales, and this is the structured image class of files. It includes SVG, Windows Metafile, and PostScript drawing commands as well as the (Windows-only) Visio embedded object.

Unfortunately, the geniuses determined the supported set of image files that the graphviz utility would generate and the geniuses who defined the images that OpenOffice Writer would import did not have lunch in the same building. About the only commonality is EPS, and in the Fedora standard RPM, EPS is not one of the options presently (mid-2008) included for graphviz.

Failing that, there’s a fairly easy workaround. Have graphviz output a PDF. PDF’s contain PostScript. There’s a utility named “pdftops” that can extract that PostScript and wrap it up in an EPS wrapper. Note that this isn’t the same program as “pdf2ps”!

Example:

dot -oBackingBeans.pdf -v -Tpdf BackingBeans.dot
pdf2ps -eps BackingBeans.pdf BackingBeans.eps

It’s probably pipeable, but this will do.

One other thing remains. The default font selection on my system wasn’t all that beautiful, so I overrode it:

graph[fontpath="/usr/share/fonts/liberation", fontsize=12, fontname="LiberationSans-Regular"];

Here, too, I’ve done the “brute force” thing and supplied the fontpath internally and manually. I should actually set up the environment, but I had other priorities that day and custom fonts are reputedly a pain in graphviz anyway. The “-v” option on graphviz lets you see what font is actually getting pulled, by the way.