Debian setup

# used by jenkins
apt-get install daemon
#apt-get install jenkins
apt-get install tomcat6
apt-get install tomcat6-admin
apt-get install curl
apt-get install r-base
apt-get install r-cran-rmysql

apt-get install texlive-xetex
apt-get install texlive-science
apt-get install texlive-latex-extra
apt-get install subversion
apt-get install cvs
apt-get install git
apt-get install wkhtmltopdf
apt-get install okular

apt-get install emacs23-nox

apt-get install jedit

apt-get install git-svn

apt-get install mailutils # smarthost

apt-get install graphviz


R
> install.packages("Rserve")
> install.packages("car") # may not be used
> install.packages("AER") #used by ParasiteClearance
> install.packages("DBI") #used by clinRepRep
> install.packages("RMySQL") #used by clinRepRep
>

Others currently installed on app-dev:
Cairo                   R graphics device using cairo graphics library
                        for creating high-quality bitmap (PNG, JPEG,
                        TIFF), vector (PDF, SVG, PostScript) and
                        display (X11 and Win32) output.
car                     Companion to Applied Regression
Formula                 Extended Model Formulas
lmtest                  Testing Linear Regression Models
sandwich                Robust Covariance Matrix Estimators
strucchange             Testing, Monitoring, and Dating Structural
                        Changes
zoo                     S3 Infrastructure for Regular and Irregular
                        Time Series (Z's ordered observations)


cairoDevice             Cairo-based cross-platform antialiased graphics
                        device driver.
Rserve                  Binary R server



chmod o+w /var/lib/tomcat6/webapps/

cp  /home/timp/.m2/repository/org/springframework/spring-instrument-tomcat/3.0.5.RELEASE/spring-instrument-tomcat-3.0.5.RELEASE.jar /usr/share/tomcat6/lib/

Ensure /etc/tomcat6/tomcat6-users.xml contains

<role rolename="poweruser" />
   <role rolename="poweruserplus" />
   <role rolename="probeuser" />
 
   <user username="admin" password="" roles="manager,admin" />

Get scrollbars in Eclipse:

 sudo su -c 'echo export LIBOVERLAY_SCROLLBAR=0 > /etc/X11/Xsession.d/80overlayscrollbars'

Get innotop working

perl -MCPAN -e shell
CPAN> install Term::ReadKey 

Ensure Rserve and Jenkins start at reboot

cd /etc/init.d
update-rc.d jenkins defaults
update-rc.d Rserve defaults

Set up networking.

Set up GMail as smart host for postfix.

Send me the IP address on boot.

Siemens S16-39 washing machine: code F18 - fixed

Our Siemens S16-39 washing machine started to display F18 and beeping (not the normal 'finished' beep).

We switched it off. Googling suggested that the impeller was broken/unable to move.

Switching back on made a straining noise - switched it off again.

I turned the large pipe (front right bottom) through ninety degrees.

Switched back on, turned to Empty. No bad noise and emptying started.

When empty I placed a towel under outlet and completely undid the large pipe.

There was human hair, a shirt stiffener and piece of slate, removed these and all well!

Representing empty CSV columns in a database

The particular case I want to nail down is a CSV file where field values are not quoted:

key,name,options,
1,one,,

The example can be represented as a three field table with non-nullable fields key and name and a nullable field options (yes I know about the trailing comma - they do occur in the wild).

For these purposes the data can be divided into Strings and others, as an empty string ("") is not a legal value for any other type.

The case is straight forward for XML. Paired tags with no content represent a zero length string. Unpaired tags (empty elements) or missing tags represent null.

For CSV the situation is more confused. When importing CSV files there are three possibilities for a column: it is not present, empty or filled with a value.

If the column is missing then the value in the database should be null.

If the column is present and empty and the column type is not String then the value should be Null; for a String column the value might be either an empty string or null: we have to decide.

Both string treatments can be argued for:

1. A present, empty column, represented by two consecutive commas, is just the limiting case of a normal string and so represents a string of zero length. This ensures that no null value will ever enter that column.
2. String handling should be consistent with the handling of other types where an empty column represents null, hence an empty string is an illegal value in a non-nullable column.

The problem with position 1. is that there is no way to represent null,
the problem with position 2. is that there is no way to represent the empty string.

Our hand is forced, as it happens, by the behaviour of MySQL, which in characteristic fashion chooses the wrong option.

MySQL chooses option 2, so if we wish to gain nulls we must either hand craft the import or post-process the column.

I have chosen to post process all nullable string columns to convert empty strings to null.

I do not believe there is a use case for storing zero length strings in nullable database columns.

Networking, and hence sshd, does not start until after login on Ubuntu

I updated my Ubuntu system and restarted it, as prompted, and left the office.

When I got home I could not ssh into my work machine.

This was because I had not logged into the machine and so networking and hence sshd had not started.

To get networking started before login add the following to /etc/network/interfaces

auto eth0
iface eth0 inet dhcp

Setup Postfix to use Gmail as a smarthost

If you wanted Jenkins to send emails via gmail for instance.

 sudo apt-get install mailutils 

 echo timp.well.ox.ac.uk > /etc/mailname
 emacs /etc/postfix/main.cf

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
html_directory = /usr/share/doc/postfix/html

 emacs /etc/postfix/sasl_passwd # create password file

smtp.gmail.com GMAIL_USER_NAME@gmail.com:GMAIL_USER_PASSWORD

 postmap hash:/etc/postfix/sasl_passwd
 /etc/init.d/postfix reload

Send me the DHCP allocated IP address on boot

Occasionally I reboot, occasionally I get allocated a new IP address.

Create /etc/init.d/sendIpToTimP :

#!/bin/sh                                                                                                                                      
ifconfig eth0 \ 
 |grep "inet addr" \
 |cut -f2 -d":" \
 |cut -f1 -d" " \ 
 | mail timp@paneris.org -s "Work IP address" 

 update-rc.d sendIpToTimP defaults

Configuring MySQL

The defaults that MySQL comes with are probably not what you want.

We got caught by this at work and followed the instructions given in How to calculate a good InnoDB log file size

Edit /etc/mysql/my.cnf to reflect:

query_cache_limit = 4M
query_cache_size  = 32M
table_cache       = 768

innodb_open_files = 512

innodb_buffer_pool_size = 512M

innodb_additional_mem_pool_size = 512M
innodb_log_file_size = 128M
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 8
innodb_concurrency_tickets = 500
innodb_lock_wait_timeout=200
innodb_autoinc_lock_mode= 2
innodb_commit_concurrency=4
innodb_flush_log_at_trx_commit=2
innodb_support_xa=false
innodb_checksums=0
innodb_doublewrite=0
innodb_max_dirty_pages_pct=15

Discover the location of datadir typically datadir = /var/lib/mysql

service mysql stop
mv /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile0_old
mv /var/lib/mysql/ib_logfile1 /var/lib/mysql/ib_logfile1_old
service mysql start