Deploying large war files to Tomcat under Hudson: use Cargo

Whilst

mvn tomcat:deploy

works for the majority of war files under Hudson, for a large war (34m) the deploy failed with Out of memory error, though not at the shell prompt. Failure to fix by tweeking memory arguments led me to use Cargo instead. Note that the local tomcat is treated as remote to reduce rework when deploying to a remote server.

<build>
<plugin>
<groupId>org.codehaus.cargo</groupId>
<artifactId>cargo-maven2-plugin</artifactId>
<configuration>
<wait>true</wait>
<container>
<containerId>tomcat6x</containerId>
<type>remote</type>
</container>
<configuration>
<type>runtime</type>
<properties>
<cargo.tomcat.manager.url>http://localhost:8080/manager</cargo.tomcat.manager.url>
<cargo.remote.username>admin</cargo.remote.username>
<cargo.remote.password></cargo.remote.password>
</properties>
</configuration>
<deployer>
<type>remote</type>
<deployables>
<deployable>
<groupId>${project.groupId}</groupId>
<artifactId>${project.artifactId}</artifactId>
<properties>
<context>${project.artifactId}</context>
</properties>
<type>war</type>
</deployable>
</deployables>
</deployer>
</configuration>
<executions>
<execution>
<id>do</id>
<phase>pre-integration-test</phase>
<goals>
<goal>deployer-redeploy</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

Setting up Hudson CI for GWT Development

GWT development requires a 32 bit Sun JDK.

We decided to do this by using a 32 bit VM, which could be moved to another server at a later date.

1. Install VMWare Player (Ensure networking is enabled during installation)
2. Download a 32 bit ubuntu ISO (or 32 bit Debian)
3. Create a new VM from the ISO.
4. Enter VM and establish your ipaddress.
5. Add hudson repo to /etc/apt/sources

   deb http://hudson.gotdns.com/debian binary/
   

6. Install Sun JDK

   apt-get install sun-java6-jdk 
   update-alternatives --config java
   update-alternatives --config javac

7. Install Tomcat

   apt-get install tomcat6
   apt-get install tomcat6-admin
   

   Modify /etc/tomcat/tomcat-users.xml and add
   <user username="admin" password="" roles="manager,admin"/>
   (or add a password and add password to ~/.m2/settings.xml)
   Set Tomcat security off
   

   sed -i "s/TOMCAT6_SECURITY=.\+/TOMCAT6_SECURITY=no/" \
   /etc/default/tomcat6
   

   Increase Tomcat default memory allocation in /etc/defaults/tomcat6
   JAVA_OPTS="-Djava.awt.headless=true -Xmx1024M -Xms1024m"
   
8. Install Hudson

   apt-get install hudson
   

9. Change Hudson port and ajpport by editting /etc/init.d/hudson to add

   HUDSON_ARGS="--httpPort=8081  --ajp13Port=8102" 

10. Install Apache

    apt-get install apache2
    a2enmod proxy
    a2enmod proxy_http

11. Setup /etc/apache2/httpd.conf (or similar)

servername hudson
ProxyPass   /  http://localhost:8081/
ProxyPassReverse  /  http://localhost:8081/
ProxyRequests   Off
<Proxy http://localhost:8081/*>
Order deny,allow
Allow from all
</Proxy>

12. Install Maven

    apt-get install maven2

13. Configure Hudson
    • setup smtp
    • set MAVEN_HOME to /usr/share/maven2
    • set email from address
    • set url to ip address
14. Download GWT toolkit 1.7.1 and extract then install gwt-dev-linux.jar to your local repo

    mvn install:install-file \
    -DgroupId=com.google.gwt -DartifactId=gwt-dev \
    -Dversion=1.7.1 -Dclassifier=linux \
    -Dpackaging=jar -Dfile=gwt-dev-linux.jar

15. Add individual projects to Hudson

Enabling private cross site scripting with a XMLHttpRequest proxy servlet

The problem

You have a list of URLs and want to interrogate them for their Content-Type using javascript in the browser.

You cannot do this because the javascript security model forbids this activity, called cross site scripting, as it could be used to invoke malign code.

The solution

Install a proxy on your server which can make the request on your behalf and relay it back to you. You have not violated the javascript security model and you are free to invoke any url on the net by a call to your own, originating, server.

However the proxy does need to be protected otherwise it could be abused by other sites. To protect the proxy we ensure that it can only be accessed from a page generated by our server, indeed one which has set a named session variable.

The code As implemented within the Melati framework, which wraps the HttpRequest and HttpResponse objects in a Melati object:

private String proxy(Melati melati, ServletTemplateContext context) {
  if (melati.getSession().getAttribute("generatedByMelatiClass") == null)
    throw new AnticipatedException("Only available from within an Admin generated page");
  String method = melati.getRequest().getMethod();
  String url =  melati.getRequest().getQueryString();
  HttpServletResponse response = melati.getResponse();
  HttpMethod httpMethod = null;
  try {

    HttpClient client = new HttpClient();
    if (method.equals("GET"))
      httpMethod = new GetMethod(url);
    else if (method.equals("POST"))
      httpMethod = new PostMethod(url);
    else if (method.equals("PUT"))
      httpMethod = new PutMethod(url);
    else if (method.equals("HEAD"))
      httpMethod = new HeadMethod(url);
    else
      throw new RuntimeException("Unexpected method '" + method + "'");
    try {
      httpMethod.setFollowRedirects(true);
      client.executeMethod(httpMethod);
      for (Header h : httpMethod.getResponseHeaders()) {
        response.setHeader(h.getName(), h.getValue());
      }
      response.setStatus(httpMethod.getStatusCode());
      response.setHeader("Cache-Control", "no-cache");
      byte[] outputBytes = httpMethod.getResponseBody();
      if (outputBytes != null) {
        response.setBufferSize(outputBytes.length);
        response.getWriter().write(new String(outputBytes));
        response.getWriter().flush();
      }
    } catch (Exception e) {
      throw new MelatiIOException(e);
    }
  } finally {
    httpMethod.releaseConnection();
  }
  return null;
}

Software Development with Certainty

My most recent project was to turn an impressive prototype developed by one person into a library, and applications based upon it, that can be simultaneously developed and used by a team.

The development process gains a new dimension once you have published your first version or have your first user. The task is made more tricky due to the absence of the original author.
In other words: situation normal, don't start from here.

The Erewhon applications, Gaboto library and its main dependencies ng4j and Jena are all changing rapidly. New functionality is being added and code and dependencies are being refactored and changed. The challenge is to enable this change without breaking installed systems or at least not breaking them unknowingly. This is ensured by establishing a contract between the code and the design by the use of tests. The tests guarantee that the system actually does do what it claims. Or, more properly, the tests are exactly what the system claims to do.

Rendering ancient languages with CSS3 font-face

Re-reading The Decipherment of Linear B led me to revisit the CSS3 feature @font-face and to address my ignorance of Unicode.

As always it is just about to be ready, really. The browsers which work: Firefox 3.5.2, Android cupcake, Safari (Mac). The browsers which fail: IE6, Firefox 3.0, Konqueror, Safari (iPhone). Opera works, but assumes one font per character sequence.

Unicode has Linear B sorted and will have Egyptian done sometime soon (October 2009).

There are some very nice, free fonts created by George Douros.

See the resulting Browser Unicode rendering test page.

Bye then Ben and Nevis. Coo Cows.

This morning Herbie and the man from the abbertoir will despatch Ben and Nevis. They were brothers and were bought some eighteen years ago. They have kept their fields from turning into woodland and have become local landmarks. Although bullocks their size and six foot horn span mean that they are best viewed from a distance.

Whilst actively seeking out human company, especially in winter when they were fed, they have never been handled and so are impossible to treat for ailments. What has led the vet to say that they must be put down is an insect infestation under their fur.

They were bought by Noel to maintain the bio-diversity of the grassland, which they did spectacularly, the field is thought to have the highest score in the West Country, though this is likely to be as much to do with its bewildering geology as its management.

Noel also chose them because they themselves do not require the attention which he lavished on them anyway, as he just liked Highland cattle. The twice daily walk to feed them undoubtably added years to his life so their ending is even more sad, as another connection to Noel passes.

Hell is other people's code

Refactoring java packages

Maintenance programming on OpenSource projects is much, much more pleasant than using non-writable libraries, as you are able to read with a pen; by which I mean that as you understand the code you can document that understanding. If code was difficult to understand due to it's lacking in structure or unevocative naming you can change it.

The maintenance programmer has a very different relation to the code than does the original author. The original author understands the intention of the code and inhabits the conceptual framework it embodies. The maintenance programmer strives to intuit.

As the understanding of the code grows so does the desire to rewrite it. Once I have tests for the main use cases for a library I am working with I am able to think about giving it a vigorous refactoring.

Refactoring is odd, it is hard and dangerous work yet nothing is meant to change. The functionality of the code should, indeed must, not change. It is tricky to come up with a defense against the injunctions if it ain't broke don't fix it and change for change's sake. I claim that unloved code is dieing code and unchanging software is no longer soft. At the end of the process there should be some increase in beauty and hence comprehensibility and maintainability.

These qualities cannot be measured, however there are things which can be measured: the links between packages (java classes live within a namespace tree where each node is called a package), the number of classes and the proportion of interfaces and abstract classes to concrete classes. These and other metrics are created by the JDepend code metrics tool.

Why package at all?

One of the code smells that refactoring should aim to reduce is the number of import statements within classes. There is a way in which to reduce the number of inter-package linkages: create all classes in the default package and be done with it. However you cannot control, or even be aware of, name clashes in the default namespace, it should NEVER be used. Another possibility is to put all of your code into one single package in a namespace you control. This too gets pretty silly pretty quickly for all but the most trivial project.

The purpose of java package namespaces should be to impose a conceptual structure upon your code which helps you and other programmers understand it. Java packages have no functional effect, given the prevailing practice of restricting visibility modifiers to public or private and eschewing the default package visibility. The compiler does not care how you group your classes. There are no tools that I am aware of which will make any comments about how you do it. The only measurable thing is the number of import statements required in classes within the package. My claim is that for two package schemas with equal number of packages one is better than the other if it results in fewer import statements.

Anti Patterns

Separating all exceptions into a separate package

This will necessarily increase the number of intra-package links, as each exception will have to be imported from the exceptions package.

Exceptions should be declared in the package they are thrown in or in the nearest parent node if they are thrown in two packages.

A new package for a new concept

Today is a new day, I am adding new stuff, I want my stuff to be separate from the previous work are all bad justifications for creating a new package.

Overuse of checked exceptions

Checked exceptions should only be used for events which the application programmer can respond to. Programming bugs should be unchecked exceptions.

Meaningless names and abbreviations

Naming is the most important aspect of writing maintainable code. Good naming removes the need to remember meaning.

Package names should be singular

Analogous to the naming of tables in an SQL database, you might think that you store people records in a table called people but everything falls out much more neatly if you call the table person, not just because you avoid English irregular plurals.

Do not repeat yourself (DRY)

In fully qualified names, as in much else in programming, you should not repeat yourself. So com.mycompany.myproject.mypackage.MyProjectEntity should probably be named com.mycompany.myproject.mypackage.Entity

Use your own default package space

If you are writing a modelling library and its root package is to be net.mylib then use that space for the main guts not net.mylib.model.

My latest refactoring with JDepend metrics before, nearly there and after.

Ant Rant - Version filtering deprecated

Filtering is one of those features which Ant makes available which was overused in the initial excitement when Ant was new.

By using version filtering you are tying your codebase into being built by Ant, or at least a build system which has the same filtering mechanism as Ant.

In Melati, WebMacro and now ten years later in Jena there was a feature to filter the code and insert the version number. The uses are so similar I suspect they were in some popular "Introduction to Ant".

We even had tests to ensure that the version variable had been successfully filtered.

assertNotEquals("@version@", Product.VERSION);

When I came to convert to Maven I wanted to keep this handy feature, but of course the test was failing.

I asked how to do filtering on the Maven lists way back. They were very scathing about the notion of a build process modifying production code. It is not the Maven way.

The question that needs a very convincing answer is what is this for? Does it fit with how people use and develop the software.

As a developer, in your IDE, do you want to see

public static final String NAME = "@Name@";

or more sanely

public static final String NAME = "MyProject";

Filtering may work for the users but not for the developers.

Lets look at the actual use cases, taking Jena as a typical project. A sensible place for these strings is Jena.java. Of the static strings in the following references are found.

PATH - 15 references - interestingly hardcoded, not interpolated.
NAME - 0 references
WEBSITE - 0 references
VERSION - 0 references
MAJOR_VERSION - 0 references
MINOR_VERSION - 0 references
REVISION_VERSION - 0 references
VERSION_STATUS - 0 references
BUILD_DATE - 0 references

We can see that after 6 years this facility has not actually been used. It was put in because the coder knew how to do it and thought it might come in useful. It hasn't. YAGNI.

If you REALLY want to use filtering then filter a properties file using Maven Resource Filtering.

However if you do just want to provide, for example, a version command in your application then explicitly add the version string as a static variable in a prominent place in your code and add a step to your release procedure document to update this variable.

Code should not depend upon the build system: rip it out.

You are using Continuous Integration, aren't you?

This is one of those questions expecting the answer yes, like you have lost your virginity, haven't you?, which has a motivational impact.

Another prod in the right direction was when someone used Hudson as the term for CI server, as you would use Hoover for vaccum cleaner.

I have tried three times to use Continuum, only once with any degree of success. Hudson just works, straight out of the box.

I am proud to say I am, at last and later than most, using Continuous Integration.

The map is not the territory

My new job on the Erewhon Project is largely based upon the work of my predecessor Arno Mittelbach.

In a series of posts leading up to the new OxPoints Arno rediscovers the issues around using first order predicate calculus on continuous dimensions such as time.

This would appear to be a rediscovery of the Yale Shooting Problem: the real world cannot be modelled in a monotonic logic. Locations are not points but areas, names are not simple properties but have implicit usage start and end times. Only in maths is something as simple as its description: the map is not the territory.

amd64 "no such file or directory" error with Android and Skype

Using an amd64 debian (lenny and kubuntu jaunty) install I came across the least informative error message yet:

bash: adb: No such file or directory

and similarly for skype.

This appears to be due to not yet having installed the i36 compatibility libraries:

apt-get install ia32-libs

and it goes away.

Bye then Pepper. Good girl.

On Wednesday morning, at about 8.15, Pepper died.

Whilst she was rather young (eleven maybe) to die, for a mongrel, her timing was pretty good. It was the first day of my new job and as it was all of us were stroking her as she stopped breathing, twitched and was gone.

Pictures tell of thousands of words

Had to give a presentation for an interview with OUCS on "the challenges of creating mobile location-based services to students" so took a walk and photographed the signage.

Did the presentation in OpenOffice Impress, but of course the fonts were different on Windows and the colours were the first ones I chose not the ones I had settled on and saved and which showed on Apple and Linux.
If I don't get the job I will blame grotty software.

Ellie as Shiva

 

Wavemaker 32 bit rpm on amd64 debian

As there is no appropriate .deb file for wavemaker but the rpm is pure java so the following works:

rpm2cpio wavemaker-4.0.2.24308-community.i386.rpm >wm.cpio  
cpio -idv  < wm.cpio
mv opt/wavemaker-4.0.2.24308 /opt/  
/opt/wavemaker-4.0.2.24308/bin/wavemaker.sh 

£900m googlewhack

In Saturday's Guardian Jill Treanor treated us to an 'explanation' of the downfall of Lloyds, which left me perplexed and annoyed, so I turned to Google to try to decipher the jargon.

I discover that a "policy tax charge" is a googlewhack so am none the wiser.

"Yahoo!", the number two, plays a spoiling game

I went to hear Mr Raghu Ramakrishnan of Yahoo! Research give a talk at ComLab, which was a real treat. Lovely people in a lovely building doing interesting things, this is the joy of living in Oxford and I am determined to take more advantage of it.

The talk was rather light on detail, if clearly and energetically given. The problem space was described very well, but the solution was mostly given by pointing to DBLife, so it can be done but the how is left for you to discover.

If, like me, you think of "Yahoo!" as a failing company with an incorrect belief that punctuation can form part of a name then you may not have paid much attention to them.
(Douglas Crockford's json excepted).

However the point was well made that as the number two in the market "Yahoo!" must, necessarily, play a market disrupting game and the way they are doing this is to open their API. The current monetised search model is sown up by big G. The next generation will be 'semantic' search:

- the search engine inferring the reason for the search and possibly tracking episodes in a search which may be engaged in over a number of sessions over a period of days or months.

- the search engine extracting more structure from the pages crawled.

How to do this is unknown so "Yahoo!" have opened their API, lowered the barrier to entry for academics and others to play with the results of spidering. No more having to setup the spiders, parsers or vast datastores previously required to start a websearch project.

"Yahoo!" do not want anything in return, all they want is the current incumbent de-throned, so that they can have another shot, possibly in partnership with the next big idea.

Bibliomania as a search provider

A nasty bug surfaced , which I introduced to poem in 2007, when Bibliomania was reunited with current melati/poem, see Melati Developers for gory details. The site is now even faster.

That done I have created an OpenSearchDescription for Bibliomania, which can be installed from the site or via the Mozilla addon repository.

I would like a few things to happen:

Someone with IE7 or IE8 try clicking on the little search icon at the bottom of the 'your details' login screen on Bibliomania.

Someone with Mozilla go to the Mozilla Addon site, search for 'Bibliomania search', create an account and login, install the addon and then review it positively. Ideally this will be more than one person, but at least one is required, other than me, before it is accepted.

You might also want to check the new entry about search in the Bibliomania FAQ.

First Post

Too busy to get a job!

Since end of the last one I have:

relaunched Bibliomania.

fixed the paneris mail server.

moved paneris.net,paneris.com, melati.org, melati.net to
bytemark, though annoyingly not paneris.org.

Installed OFBiz.

Installed Android.

Joined Twitter

Setup this blog.