Tuesday, 23 November 2010

CSV unification

Unifier on GitHub


A simple problem

Given an excel spreadsheet with three sheets within it, all with approximately the same rows but with differing columns, produce a unified CSV file. If a sheet does not contain a row then insert blank columns. The unique key for each sheet is column 2, called ID, move this column to column 1. No doubt you can see all the issues, as I can in retrospect, but after a week and a bit the job is done. High test coverage enabled me to refactor and add features through to a completely different beast.

Lessons learned

hashcode() for enum constants

The members of an enum inherit their hashCode() method from Object so it varies between JVM invocations. Hence I used
result = prime * result + unificationOption.ordinal();
in CsvTable.hashCode().

Bridging methods

During the compilation of generic code Java quietly generates what are called bridging methods these are visible to Cobertura but not to you, the coder, so Cobertura tries to tell you that you have not exercised these methods by marking the class definition line as not covered. Using the following code to print out all methods
for (Method m : CsvTable.class.getMethods()) {
  System.out.println(m.toGenericString());
}
I discovered the generated bridging methods
public net.pizey.csv.CsvRecord get(java.lang.Object)
public java.lang.Object get(java.lang.Object)

public net.pizey.csv.CsvRecord put(java.lang.String,net.pizey.csv.CsvRecord)
public java.lang.Object put(java.lang.Object,java.lang.Object)

public net.pizey.csv.CsvRecord remove(java.lang.Object)
public java.lang.Object remove(java.lang.Object)
The put(Object key, Object value) cannot be accessed normally, as it is masked by the generic version. So we have to introduce a reflective mechanism of invoking it.
public void testBridgingPut() {
  CsvTable t = new CsvTable("src/test/resources/sheet2.csv", UnificationOptions.LOG);
  Object o = t.get((Object) "1");
  Method method = t.getClass().getMethod("put", new Class[] { Object.class, Object.class });
  method.invoke(t, "jj", o);
}
Finally I had to deal with the generated bridging methods for clone. This is done by not supplying a generic clone method but using the pre-generics signature.

I have come to enjoy Java6 however Generics do feel like an expensive compromise.

Tools and sources

The CSV parser was written by WilliamC and incorporated in a CSV importer in 2000 by MylesC. I started from that framework, but as it was written before Java Collections, let alone Java 6, not a lot remains of the original.
The tools used: GitHub, Hudson CI, Maven, Eclipse and Cobertura.

Sharing and Deployment

The code is at https://github.com/timp21337/unifier.

To include the jar in Maven:
<dependency>
 <groupId>net.pizey.csv>/groupId>
  <artifactId>unifier</artifactId>
  <version>1.0</version>
</dependency>
Deployed to a Maven repository at http://pizey.net/maven2/


Tuesday, 5 October 2010

Adding a framebuffer to a headless unix server


Some java software, including GWT, expects a framebuffer to be present.
This post shows how to install Xvfb (X virtual Frame Buffer) on a Ubuntu/Debian system. This assumes that you do not currently have an X server running (it is a headless server).

apt-get install xvfb xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic

(the fonts are not needed but their absense is warned about)
Place the following in /etc/init.d/xvfb
case "$1" in
start)
echo "Starting virtual X frame buffer: Xvfb"

/usr/bin/Xvfb :0 \
-auth /etc/X99.cfg -screen 0 1024x768x24 \
-fbdir /var/lib/xvfb \
-extension RANDR &

export  DISPLAY=:0

;;
stop)
echo "Stopping virtual X frame buffer: Xvfb"
echo "noop"
;;
*)
echo "Usage: /etc/init.d/xvfb {start|stop}"
exit 1
;;
esac

exit 0


then
chmod u+x /etc/init.d/xvfb 
echo localhost > /etc/X99.cfg
mkdir /var/lib/xvfb
/etc/init.d/xvfb start

should be good to go.

Thursday, 1 July 2010

Using both Http-basic and session based form-login authorisation in Spring Security with wget (--auth-no-challenge)

We recently changed our Spring Security configuration from http-basic to form-login, well actually we used
<http 
auto-config="true" >
which is shorthand for
<http>
<intercept-url pattern="/**" access="ROLE_USER" />
<form-login />
<anonymous />
<http-basic />
<logout />
<remember-me />
</http>

The above, and the rest of the documentation suggests that both http-basic and form-login should work simultaneously. However my wget script broke on the change over. The wget command was
wget  --user=foo@example.org --password=bar --post-data="" \
-d http://localhost/protectedUrl 

This got redirected to the forms login page, as though the authorisation header was being ignored. However alimanfoo was able to use authenticated requests from java to access the protected urls.

Further research revealed that wget only issues authorisation headers in response to a challenge, unless the --auth-no-challenge qualifier is added.
wget  --user=foo@example.org --password=bar --post-data="" \
--auth-no-challenge -d http://localhost/protectedUrl 
You might have thought that supplying username and password meant that you wanted to set them, and this was the behaviour of wget 1.10.2 and prior but no longer.

In fluent manpage we get:
Use of this option is not recommended, and is intended only to
support some few obscure servers, which never send HTTP
authentication challenges, but accept unsolicited auth info, say,
in addition to form-based authentication.

So I am off to explain the use-case to the maintainer list.

Tuesday, 25 May 2010

Android fonts for Debian/Ubuntu

I first came across the mention of the free, Open Source, Android fonts a while ago; here, I think.
They are clean and easy on the eye.

You can install them with
sudo apt-get install ttf-droid 

which will place fonts in /usr/share/fonts/truetype/ttf-droid

However if all you want is to use a droid font in a web page just use
<link href='http://fonts.googleapis.com/css?family=Droid+Sans' 
rel='stylesheet' type='text/css'>

Sunday, 21 February 2010

Bringing WebMacro up for air

WebMacro was the first OpenSource community that I followed, so I love it and still feel a sense of hurt that licensing issues meant that Velocity had to be born.

On top of that I have quite a few WebMacro templates around, and anyway the world needs more than one templating engine.

However, WebMacro went the way of all flesh: The original author moved on, the push to version 1.0 was slow, and I for one stopped following every thread. Versioning was a real mess; (which was more recent 1.0b or 1.0?) Then came the painful, acrid push to 2.0.

I have revisited a few times, cleaning the code of warnings, ensuring tests continue to pass, even regenerating with new versions of javacc.

However what was really needed was a restructuring of the source tree to align with Maven conventions and a thorough cleanout.

The only way to do that used to be to backup the CVS tree, restructure and then ask a SF admin to recreate the repository, which was too big a hurdle.

Recently (well some time in the last ten years) SF have introduced adminrepo, which enables you to lock a repository, restructure it and then replace:

ssh -t timp,webmacro@shell.sourceforge.net create
adminrepo --checkout cvs
cd /cvsroot/webmacro/webmacro
mkdir src/main
mkdir src/main/java
mv src/org src/main/java
cd ~
adminrepo --save cvs
exit


By this means I was able to move current code into the places Maven expects them and move non-core code out to ../contrib.

Then I upgraded from Doug Lea's venerable Concurrent to his java.util.concurrent.

Thanks to a message from Brian Goetz himself I was able to surmount the only real hickup: EDU.oswego.cs.dl.util.concurrent.ClockDaemon becomes java.util.concurrent.ScheduledExecutorService.

Now WebMacro had only one compile-time dependency: the logging system. On inspection it seems that WebMacro's own logging (yes its has a complete logging system built in) should be ripped out in favour of slf4j.

WebMacro 2.2 should be back in the swim.

Thursday, 17 December 2009

Deploying large war files to Tomcat under Hudson: use Cargo

Whilst
mvn tomcat:deploy
works for the majority of war files under Hudson, for a large war (34m) the deploy failed with Out of memory error, though not at the shell prompt. Failure to fix by tweeking memory arguments led me to use Cargo instead. Note that the local tomcat is treated as remote to reduce rework when deploying to a remote server.
<build>
<plugin>
<groupId>org.codehaus.cargo</groupId>
<artifactId>cargo-maven2-plugin</artifactId>
<configuration>
<wait>true</wait>
<container>
<containerId>tomcat6x</containerId>
<type>remote</type>
</container>
<configuration>
<type>runtime</type>
<properties>
<cargo.tomcat.manager.url>http://localhost:8080/manager</cargo.tomcat.manager.url>
<cargo.remote.username>admin</cargo.remote.username>
<cargo.remote.password></cargo.remote.password>
</properties>
</configuration>
<deployer>
<type>remote</type>
<deployables>
<deployable>
<groupId>${project.groupId}</groupId>
<artifactId>${project.artifactId}</artifactId>
<properties>
<context>${project.artifactId}</context>
</properties>
<type>war</type>
</deployable>
</deployables>
</deployer>
</configuration>
<executions>
<execution>
<id>do</id>
<phase>pre-integration-test</phase>
<goals>
<goal>deployer-redeploy</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

Wednesday, 11 November 2009

Setting up Hudson CI for GWT Development

Hudson CI for GWT Development (made with Gliffy)GWT development requires a 32 bit Sun JDK.

We decided to do this by using a 32 bit VM, which could be moved to another server at a later date.
  1. Install VMWare Player (Ensure networking is enabled during installation)
  2. Download a 32 bit ubuntu ISO (or 32 bit Debian)
  3. Create a new VM from the ISO.
  4. Enter VM and establish your ipaddress.
  5. Add hudson repo to /etc/apt/sources
    deb http://hudson.gotdns.com/debian binary/
    
  6. Install Sun JDK
    apt-get install sun-java6-jdk 
    update-alternatives --config java
    update-alternatives --config javac
  7. Install Tomcat
    apt-get install tomcat6
    apt-get install tomcat6-admin
    
    Modify /etc/tomcat/tomcat-users.xml and add
    <user username="admin" password="" roles="manager,admin"/>
    (or add a password and add password to ~/.m2/settings.xml)
    Set Tomcat security off
    sed -i "s/TOMCAT6_SECURITY=.\+/TOMCAT6_SECURITY=no/" \
    /etc/default/tomcat6
    
    Increase Tomcat default memory allocation in /etc/defaults/tomcat6
    JAVA_OPTS="-Djava.awt.headless=true -Xmx1024M -Xms1024m"
  8. Install Hudson
    apt-get install hudson
    
  9. Change Hudson port and ajpport by editting /etc/init.d/hudson to add
    HUDSON_ARGS="--httpPort=8081  --ajp13Port=8102" 
  10. Install Apache
    apt-get install apache2
    a2enmod proxy
    a2enmod proxy_http
  11. Setup /etc/apache2/httpd.conf (or similar)
  12. servername hudson
    ProxyPass   /  http://localhost:8081/
    ProxyPassReverse  /  http://localhost:8081/
    ProxyRequests   Off
    <Proxy http://localhost:8081/*>
    Order deny,allow
    Allow from all
    </Proxy>
  13. Install Maven
    apt-get install maven2
  14. Configure Hudson
    • setup smtp
    • set MAVEN_HOME to /usr/share/maven2
    • set email from address
    • set url to ip address
  15. Download GWT toolkit 1.7.1 and extract then install gwt-dev-linux.jar to your local repo
    mvn install:install-file \
    -DgroupId=com.google.gwt -DartifactId=gwt-dev \
    -Dversion=1.7.1 -Dclassifier=linux \
    -Dpackaging=jar -Dfile=gwt-dev-linux.jar
  16. Add individual projects to Hudson