Apache, Tomcat and SSL

Its a popular thing to use Apache (or nginx, etc.) as a reverse-proxy server fronting Tomcat. However, documentation on such practices tends to gloss over certain important things. Specifically:

1. Who owns the SSL cert that manages such a configuration. Apache or Tomcat?

2. Is the Apache-to-Tomcat tunnel encrypted? If so, how?

I finally decided to determine by experimentation. Here’s the scoop:

1. Encryption between Apache and Tomcat is not supported by the AJP protocol. If you need back-end encryption, use Apache’s mod_proxy, not mod_ajp.

2. If you make an https connection to a website hosted by Apache or proxied by Apache to Tomcat, the cert that’s applied will be the (x509) cert for that Apache host. Not a Tomcat jks cert.

3. You can configure Apache to proxy incoming SSL traffic to Tomcat even though Tomcat  itself isn’t configured for SSL.  Simply forward from your Apache ProxyPass/ProxyPassReverse to the Tomcat http port (8080 by default).

Note If you forward SSL to Tomcat via its http port, then none of the traffic between Apache and Tomcat will be encrypted. That’s OK if you are doing a forward within the local machine (using loopback) or if you are OK with security on your LAN.

4. If clear-text between Apache and Tomcat is not acceptable, you can do SSL from Apache to Tomcat. In that case, Tomcat needs its own keystore and certs, independent of the Apache certs. Apache will decrypt incoming Internet traffic so that it can do whatever it needs to do with headers and rewrites, then re-encrypt the proxy data using Tomcat’s cert.

To do SSL between Apache and Tomcat, the ProxyPass/ProxyPassReverse directives should address Tomcat’s HTTPS port (8443). Presumably you can even take plain HTTP coming into Apache and SSL it to Tomcat, but I didn’t bother to check.

Note that between Apache and Tomcat, a self-signed cert is probably good enough. In fact, since the cert won’t be officially registered, it’s one less internal secret for people to learn on the Internet. Apache’s handling of the finer aspects of backend certs are tunable, but the defaults are sufficient for most purposes.

Land Mines – Spring Neo4j

One of the primary purposes of this blog is to record what I’ve learned by tedious trial and error and/or spending time down in source code I shouldn’t have had to look at.

This particular topic has more than its share of discoveries.

Spring Neo4j claims that it’s intended to imitate, where possible, existing persistence systems approaches. Unfortunately, it has a long way to go on that.

First, let me mention that after descending through Maven’s equivalent of “DLL Hell”, I have been working with the following version sets:

  • Spring Framework version 4.0.6
  • Spring Neo4j 3.3.0.Release
  • Neo4j version 2.2.0
  • Logging courtesy of slf4j version 1.7.6 and log4j2 version 2.2, which has renamed the log4j config files since log4j v1 and added a few new config options (just for info).

As usual, just finding what was compatible with what was an adventure. I have a nasty, if unprovable suspicion that some of the pain could have been reduced if certain functions and classes had been marked “deprecated” instead of being removed or relocated.

Some of what I’ve learned will break shortly. Spring Neo4j 4 will actually lose a number of functions and annotations that exist in Spring Neo4j 3 (they were probably broken anyway). And some of the fixes I’ve come up with may actually be exploiting bugs rather than clean fixes. It’s the best I could do.

My use case seemed simple. I have 2 entity types: A and B, and a relationship, we’ll call “wants” which has a property named “priority”. I want to set up a network of many-to-many A-wants-B relationships with associated priorities and I want to be able to select a given B and get back an ordered list of A-wants in priority order with the priorities visible.

So using Spring Data Neo4j (SDN, for short), that gives me 2 NodeEntity classes (A and B) and a RelationshipEntity (“Wants”). I’m using GraphRepository extensions to manage persistence for A and B, which gives rise to the first question:

Q: What’s the best way to handle persistence of RelationshipEntities?

There’s a RelationshipGraphRepository, but unlike GraphRepository, it’s a class, not an interface and there is absolutely zero documentation I can find on how to use it or even if I should be explicitly using it. I therefore used ordinary GraphNode for the relationshipEntity. It seems to work. More or less.

And the first observation:

When they say “@Fetch”, they mean “@Fetch”

In JPA, a recommended way of persisting an object is in the form “a = repository.save(a);” This is because JPA may construct new new “a” using the original “a” as a basis, but carrying (visibly or not) extra data and/or meta-data provided by the “save” operation.

This is also the (explicitly) recommended practice for SDN. But there’s a big “gotcha”.

When you persist with JPA, the returned object will contain AT LEAST as much data as the original copy did (it may even be the original copy). When you persist with SDN, the returned object may not. What SDN does is save the data, then do a basic lazy fetch resulting in a new object instance. Which means that unless you annotate your complex properties with “@Fetch”, you’ll end up with a nasty surprise. Unlike JPA, which returns either the original value or an unresolved proxy object, SDN returns null for unresolved lazy fetches. Which is probably a recipe for lost data and is definitely a recipe for confusion (remember, this is things I learned the hard way!)


You can code @Query on both Repository and EntityNode classes

The manual doesn’t mention this. It’s a useful thing to know, although since in EntityNode classes, you often want to base your query relative to the current EntityNode instance, I also spent a lot of time coding things like “{self}” and “{this}” without success. I finally found out that using the clause START node=({self}) does the trick, although since “START” is supposed to be deprecated, there’s likely something that can do the same thing with a simple MATCH. That’s something to tackle another day.

What you get back isn’t what you think it is

Cypher isn’t as intuitively obvious to me as some people seem to think it should be. It’s a form of complex mathematical notation and while the docs on it are fairly illustrative, they are perhaps less complete or explanatory as they might be. So I spent a lot of time trying weird expressions. Here’s what worked as a member query on my “B” node:

List wantsList = new ArrayList(5);

@Query("START b=node({self}) MATCH (b)-[r:WANTS]-(a:A) return r ORDER BY r.priority")
public List getWants() {
    return wantsList;

A lot of the earlier attempts returned false “Wants” objects whose nodeIds were actually the IDs of the “A” objects on the other end of the relationship.

Incidentally, Neo4j normally wants relationships to be unordered and will complain if you attempt to use an ordered collection (such as List) to hold them. However it is smart enough to realize that when you do an “ordered by” query it’s OK to define the collection using an ordering collection object.

But that’s not all!

The “Wants” list that this particular query returned is not directly usable. The nodeIds are valid, but all the other property values were blank. Not merely the lazy-fetch values, but even the primitive property values. So to be actually usable I have to pull the nodeId from the returns pseudo-Wants and use the repository to lookup the actual Wants node.

So I’ve solved what should have been a relatively simple problem – even though it’s an ugly solution – and only lost about 2 days doing so.

Maven: No plugin found for prefix ‘X’ in the current project and in the plugin groups

I just spent 2 days constructing Apache Stratos and it was an apalling experience. Both Stratos and one of its key components: jcloud are massively-complex projects and they brought out the worst in Maven 3.

In theory, Maven is a “write-once, build-anywhere” system. Practice wasn’t quite as kind to me this last week. In addition to the grief I had building Stratos, it turns out that Maven 3 does strange things to the dbunit tests on one of my own major projects for no apparent reason or gain.

Stratos, on the other hand, showed the weaknesses in repository retrieval, as many dependencies simply refused to download from the master repo over and over again and had to be manually yanked from the repository and installed by hand in my local repository. In some cases, the jar didn’t seem to be present on the master repo at all, in others, that particular part of the repository simply timed out when attempting to look at it with the search.maven.org browser and I had to find a secondary source. I thought Maven was supposed to have the intelligence to consult mirror servers!

Having finally downloaded and/or built all the components, the last wall I hit was the error in this post’s title:

No plugin found for prefix ‘X’ in the current project and in the plugin groups

I got this because Maven ran out of memory. PermGen space, no less! So I tried adding the usual Java memory parameters to the Maven command line, and that’s when I got this mysterious message.

Turns out, that the memory options are not passed verbatim to Maven’s JVM, they are passed as though they were command-line arguments, so an “-Xmx” option comes out as something like Maven goal “X”. Obviously not what I needed.

To get the MAVEN_OPTS set in Linux for a 1-shot build command, I used the following command line, instead:

MAVEN_OPTS=” -XX:MaxPermSize=256m” mvn -Dmaven.test.skip=true install

This gave Maven the necessary cues to finish the build. I now have an allegedly useful Stratos system to play with.

DBUnit and CSV reference data

The CSV capabilities of dbUnit are under-documented. Here’s the results of some pain, suffering and debug tracing:

The CSV files are expected to be one-per-table. The tablename is part of the filename, thus: “TABLE1.csv”. Format is the usual, with the first row containing the column names and subsequent rows containing data. It is possible to customize the delimiters and separators, but the defaults work with bog-standard CSV. One possible reason to override is to allow use with pipe-separated format files.

Here’s some code to snapshot an SQL query out to CSV for use as a later test reference (or whatever).

Connection con =
IDatabaseConnection dbUnitCon =
new DatabaseConnection(con, "MYDB");

ITable actualTable =
+ " AND INK_ID in('ABEND001', 'AAA', '07CSI')");

// Take reference snapshot:
IDataSet ds1 = new DefaultDataSet(actualTable);
CsvDataSetWriter.write(ds1, new File("/home/timh/csvdir"));

dbUnit will create csvdir, if needed and output 2 files. The SNAPTABLE.csv file and a file named “table-ordering.txt”.

To load SNAPTABLE for use in validating the results of a test:

// Load expected data from CSV dataset
CsvDataFileLoader ldr = new CsvDataFileLoader();
// NOTE: terminal "/" on URL is MANDATORY!!!
IDataSet expectedDataSet =
ITable expectedTable =

// Assert actual database table match expected table
Assertion.assertEquals(expectedTable, actualTable);

Note that while you can specify case-insensitivity on table names (it’s the default), the case of the SNAPTABLE.csv file and the SNAPTABLE entry in table-ordering.txt must match – at least on case-sensitive OS’s. And it’s good practice regardless. Also note that table-ordering.txt can contain multiple table names, one tablename per line.

Finally, note that the error exception for the comparison assert counts line numbers off by 2. It doesn’t take the column-name row into account, and it starts counting from 0, instead of the more usual case (for databases) of counting starting at 1.

important! The CSV reader is not very flexible when it comes to reading NULL values. Null fields MUST be given a value of “null”, lower case, WITHOUT surrounding quotes. Like so:

“Fred Smith”,”127.0″,null,null,”Jabber”

JSF2 Annotations Ignored

I couldn’t figure out why when I tried to retro-fit an old JSF project for JSF2 annotations the annotations weren’t being processed and the managed beans weren’t being instantiated.

Duh. The JSF2 annotation processor defers to faces-config.xml. My faces-config still said JSF 1.2 in the faces-config element and related xmlns attributes.

On a related note, annotations are only processed by default in the WAR WEB-INF/classes directory. If including JARs in WEB-INF/lib, each jar containing managed beans need a META-INF/faces-config.xml. Doesn’t have to have anything in it, but if it isn’t there, annotations won’t process.

RichFaces “columns” tag doesn’t display columns

The RichFaces manual show this example on using the “columns” tag:

<rich:dataTable value="#{capitalsBean.capitals}" var="cap">
<rich:columns value="#{capitalsBean.labels}" var="col" index="index">

<h:outputText value="#{cap[index]}" />



There is no corresponding sample Java code.

That’s very unfortunate, since in the absence of any better documentation, the natural thing to expect is that “capitalBeans.labels” would be a DataModel object, just as it is for the table itself. And it isn’t.

The model for columns is a simple ordered collection, such as a List.

JPA and overlapping complex keys

If at all possible, you should avoid setting up databases with complex keys (or compound keys, as they are sometimes known). It makes life a whole lot more complex, and in a world where everyone already says you’re taking too long because “All You Have To Do Is…”, complex isn’t a good thing.

Sometimes, however, there’s just no choice.

I have one app that has a table with 2 complex-key relationships. The table’s primary key is (idA, idB) and there’s a foreign key reference to another table item with a key of (idA, idC). The table’s key is an embedded object of type “table1Id” and the foreign reference is a lazy-load “table2” object.

The full details are here.

Tomcat ignores FORM based authentication, uses BASIC authentication

Here’s a week of my life down the drain. I’m setting up a new production site with Apache AJP fronting Tomcat and when the Tomcat webapp needs authentication, it ignores the FORM authentication directive, presenting a BASIC authentication dialog, instead!

I thought it had to do with AJP and SSL tunneling. It didn’t. I finally confirmed that by firing up a text-only browser (this is a remote hosting location with no GUI) and bypassing the firewall, Apache, AJP and everything but Tomcat itself.

The culprit turned out to be insidious. I was overriding an authentication option in order to fine-tune client caching. However, the examples I’d been pilfering from were using BASIC authentication, not FORM authentication. So when I added the config option, it was the Tomcat Valve for BasicAuthenticator.

Turns out that if you do that, the BasicAuthenticator silently takes over, overriding the FORM authentication configuration in web.xml.

The cure: use the FormAuthenticator Valve!

Making Apache mod_rewrite and the ajp Tomcat connector work together

Apache’s URL rewriting facility can be used to shape URLs piped to Tomcat, but it’s not as simple as it seems.

The basic action is straightforward. First incoming URLs are processed by mod_rewrite, then they are matched up against JkMount definitions to find the appropriate Tomcat connector to use.

The problem is, what Tomcat sees isn’t what you’d expect to have come out from the rewrite process.

What actually happens is that before being submitted to mod_rewrite, the incoming URL is converted to server-relative form. After rewriting, the resulting URL is then reassembled, including the parts that Tomcat doesn’t want or need. Including the Apache DocumentBase path.

To prevent this problem, just qualify the last rewrite rule in the process with the “PT” directive to let Apache know to “Pass Through” the URL without reassembly. So, for example:

RewriteRule ^/$ /mywebapp/index.jsp [L,PT]
JkMount /mywebapp/* ajp13

Eclipse: Tomcat FileNotFoundException

After happily using the sysdeo Tomcat plugin for years. I’ve finally been pushed into using the Server facility bundled into Eclipse Ganymede’s JEE drop. For some reason, once started, sysdeo cannot be stopped or restarted – it loses track of the logfile classes.

In theory, you just create a new server of your choice in the Eclipse Servers View. In practice, it’s not quite so simple. Here’s why:

Tomcat actually consists of 2 aspects: a sharable codebase (CATALINA_HOME) and a per-instance part (CATALINA_BASE). This allows multiple copies of Tomcat to run on a single server using one codebase. Most of the time, CATALINA_HOME and CATALINA_BASE are the same value, since more often than not, only one copy of a given version of Tomcat is is use – at least on a developer’s machine.

When you create a Tomcat server using the Eclipse Servers facility, however, it clandestinely creates its own CATALINA_BASE, copying selected files – and only those files into a directory owned by the plugin.

I got burned. I was keeping a file of my own in CATALINA_HOME and using a relative reference to it in server.xml. The file didn’t copy and Tomcat didn’t start clean.

The simplest solution was to edit server.xml and replace the relative path to an absolute path, so that the copied configuration would be able to locate the original (and in this case, the only) copy of my file.

Apparently, however, the copying of the CATALINA_BASE data occurs only when you create a new Server definition. I had to delete the old Server definition from Eclipse and create a new one to get the changes to take.

1 2