Olivier Mengué – Code & rando

I'm currently playing with XSLT to improve my XMLTV grabbing solution. Much fun playing with a functionnal language available in most web browsers.

I had the need to handle timezone with daylight savings to display date/times in my timezone, Europe/Paris (same as most of western Europe), which has currently the following definition (I'm not interested in history for this project):

• Winter is UTC + 01:00
• Summer is UTC + 02:00
• Transition from winter to summer occurs on the last sunday of March at 01:00 UTC
• Transition from summer to winter occurs on the last sunday of October at 01:00 UTC

So I wrote the following template date-time-Paris which is very specialized as it handles only one specific timezone, but it is very fast. I'm using only two extensions from EXSLT: date:day-in-week() and date:add().

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:tap="test:tap"
  xmlns:date="http://exslt.org/dates-and-times"
  extension-element-prefixes="date"
  >

  <!-- Transformation de l'heure UTC en heure de Paris -->
  <!-- Copyright (c) 2008 Olivier Mengué -->
  <xsl:template name="date-time-Paris">
    <!--
      Daylight savings transitions:
      - last sunday of March at 1:00 UTC
      - last sunday of October at 1:00 UTC
      Time offsets:
      - winter: +01:00
      - summer: +02:00
    -->
    <xsl:param name="date-time-Z"/><!-- UTC -->
    <xsl:variable name="len" select="string-length($date-time-Z)"/>
    <xsl:variable name="seps" select="translate($date-time-Z, '0123456789', '')"/>
    <xsl:if test="$len &lt; 11 or substring($date-time-Z, $len, 1) != 'Z' or ($seps != '--T::Z' and $seps != '--T::.Z' and $seps != '--Z' and $seps != '--TZ' and $seps != '--T:Z')">
      <xsl:message terminate="yes">date-time-Paris: date '<xsl:value-of select="$date-time-Z"/>' invalide.</xsl:message>
    </xsl:if>
    <xsl:variable name="month" select="number(substring($date-time-Z, 6, 2))"/>
    <xsl:variable name="offset">
      <xsl:choose>
        <xsl:when test="$month &lt; 3 or $month &gt; 10">1</xsl:when>
        <xsl:when test="$month &gt; 3 and $month &lt; 10">2</xsl:when>
        <xsl:otherwise>
          <xsl:variable name="transition-day" select="32 - date:day-in-week(concat(substring($date-time-Z, 1, 7), '-31Z'))"/>
          <xsl:variable name="less" select="translate(substring($date-time-Z, 8), '-T:Z', '') &lt; $transition-day*1000000+10000"/>
          <xsl:value-of select="1+number(($less and $month = 10) or (not($less) and $month = 3))"/>
        </xsl:otherwise>
      </xsl:choose>
    </xsl:variable>
    <xsl:value-of select="concat(translate(date:add($date-time-Z, concat('PT', $offset, 'H')), 'Z', ''), '+0', $offset, ':00')"/>
  </xsl:template>

The implementation is really short, however it is quite complex as XSLT 1.0 requires to uses many tricks to workaround languages limitations (no native date type, no string order, no xor operator). Here are some tips to follow the code:

• (a xor b) is equivalent to ((a and not(b) ) and (not(a) and b)).
• number(true()) is 1, number(false()) is 0.
• March and October have the same number of days: 31.
• $transition-day*10000000+10000 for october 2008 (26010000) is an number representation of 2008-10-26T01:00:00Z.
• the second argument to date:add() is a ISO 8601 duration. Ex: PT2H (2 hours).

In fact I present here the fourth version of the code, the first one was much longer (5x) as I was using intermediate templates for computations (last-sunday-of-month, previous-sunday). Of course the key to debug, and then optimise and refactor the implementation was to start with a good test suite. The test suite is of course written in XML and embedded in the stylesheet so it can evolve with the code.

  <tap:suite name="date-time-Paris" xmlns:t="test:date-time-Paris">
    <t:t dt="2008-09-24T21:37:52Z"     r="2008-09-24T23:37:52+02:00"/>
    <t:t dt="2008-09-24T23:37:52Z"     r="2008-09-25T01:37:52+02:00"/>
    <t:t dt="2008-09-24T23:37:52.325Z" r="2008-09-25T01:37:52.325+02:00"/>
    <t:t dt="2008-10-01T00:00:00Z"     r="2008-10-01T02:00:00+02:00"/>
    <t:t dt="2008-10-26T00:59:59Z"     r="2008-10-26T02:59:59+02:00"/>
    <t:t dt="2008-10-26T01:00:00Z"     r="2008-10-26T02:00:00+01:00"/>
    <t:t dt="2008-10-31T12:00:00Z"     r="2008-10-31T13:00:00+01:00"/>
    <t:t dt="2008-11-24T21:37:52Z"     r="2008-11-24T22:37:52+01:00"/>
    <t:t dt="2008-12-31T23:30:00Z"     r="2009-01-01T00:30:00+01:00"/>
    <t:t dt="2009-01-01T00:00:00Z"     r="2009-01-01T01:00:00+01:00"/>
    <t:t dt="2009-03-01T12:00:00Z"     r="2009-03-01T13:00:00+01:00"/>
    <t:t dt="2009-03-29T00:59:59Z"     r="2009-03-29T01:59:59+01:00"/>
    <t:t dt="2009-03-29T01:00:00Z"     r="2009-03-29T03:00:00+02:00"/>
    <t:t dt="2009-03-30T12:00:00Z"     r="2009-03-30T14:00:00+02:00"/>
    <t:t dt="2009-07-01T00:00:00Z"     r="2009-07-01T02:00:00+02:00"/>
  </tap:suite>

A more complete testsuite can be generated from a local dump of the zoneinfo database (directly works on Ubuntu Hardy, however the zdump tool is missing on RHEL/CentOS):

zdump -v Europe/Paris | perl -ne 'm/ (Mar|Oct) (\d{2}) (\d{2}:\d{2}:\d{2}) (20[0-5]\d) UTC = ... (?:Oct|Mar) (\d{2}) (\d{2}:\d{2}:\d{2}) / && do { my $mm = sprintf "%02d", 3+7*($1 eq 'Oct'); print "    <t:t dt=\"$4-$mm-$2T$3\" r=\"$4-$mm-$5T$6\"/>\n"; }'

Now, we need is a way to run the test suite. In XSLT terms, we say: "to apply a template to the test data". This template will just apply a date-time-Paris call to each of the tests.

  <xsl:template match="t:t" xmlns:t="test:date-time-Paris">
    <xsl:call-template name="tap:is">
      <xsl:with-param name="got">
        <xsl:call-template name="date-time-Paris">
          <xsl:with-param name="date-time-Z" select="@dt"/>
        </xsl:call-template>
      </xsl:with-param>
      <xsl:with-param name="expected" select="@r"/>
      <xsl:with-param name="name" select="@dt"/>
    </xsl:call-template>
  </xsl:template>

</xsl:stylesheet>

We now have a complete stylesheet that can be used as a library in other stylesheet. But we have not yet run the tests. Also you probably wonder what are this XML namespace test:tap and the template called tap:is.

test:tap is just a small XSLT test framework that I wrote and the report test results following the Test Anything Protocol that is well known by Perl programmers. tap:is is part of the test:tap API and just check for equality and reports in the TAP format.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:tap="test:tap"
  >

  <!-- Run all the tests in the test suite -->
  <xsl:template match="tap:suite">
    <xsl:message>1..<xsl:value-of select="count(node()[name()!=''])"/></xsl:message>
    <xsl:message># Suite: <xsl:value-of select="@name"/></xsl:message>
    <!-- TODO fix this expression -->
    <xsl:apply-templates select="node()[name()!='']"/>
  </xsl:template>

  <!-- Test Anything Protocol style reporting of tests -->
  <xsl:template name="tap:is">
    <xsl:param name="got"/>
    <xsl:param name="expected"/>
    <xsl:param name="name"/>
    <xsl:choose>
      <xsl:when test="$got = $expected">
        <xsl:message>ok <xsl:value-of select="position()"/> - <xsl:value-of select="concat($name, ' -&gt; ', $got)"/></xsl:message>
      </xsl:when>
      <xsl:otherwise>
        <xsl:message>not ok <xsl:value-of select="position()"/> - <xsl:value-of select="$name"/></xsl:message>
        <xsl:message>#          got: <xsl:value-of select="$got"/></xsl:message>
        <xsl:message>#     expected: <xsl:value-of select="$expected"/></xsl:message>
      </xsl:otherwise>
    </xsl:choose>
  </xsl:template>

</xsl:stylesheet>

We now have two libraries. Let's add a bit more XSLT to glue them together. The main stylesheet applies tap:suite templates to all tap:suite from all imported stylesheets.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:transform version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  >

  <!-- Test framework -->
  <xsl:import href="tap.xslt"/>

  <!-- All stylesheets to test -->
  <xsl:import href="date-time-Paris.xslt"/>

  <!-- Launch the tests embedded in libraires, whatever the input is -->
  <xsl:template match="/">
    <xsl:for-each select="document('')/xsl:transform/xsl:import/@href">
      <xsl:apply-templates select="document(.)//tap:suite" xmlns:tap="test:tap"/>
    </xsl:for-each>
  </xsl:template>

</xsl:transform>

We now have everything to run the test suite:

$ echo '<x/>' | xsltproc tap-run.xslt -
1..15
# Suite: date-time-Paris
ok 1 - 2008-09-24T21:37:52Z -> 2008-09-24T23:37:52+02:00
ok 2 - 2008-09-24T23:37:52Z -> 2008-09-25T01:37:52+02:00
not ok 3 - 2008-09-24T23:37:52.325Z
#          got: 2008-09-25T01:37:52.3249999999998+02:00
#     expected: 2008-09-25T01:37:52.325+02:00
ok 4 - 2008-10-01T00:00:00Z -> 2008-10-01T02:00:00+02:00
ok 5 - 2008-10-26T00:59:59Z -> 2008-10-26T02:59:59+02:00
ok 6 - 2008-10-26T01:00:00Z -> 2008-10-26T02:00:00+01:00
ok 7 - 2008-10-31T12:00:00Z -> 2008-10-31T13:00:00+01:00
ok 8 - 2008-11-24T21:37:52Z -> 2008-11-24T22:37:52+01:00
ok 9 - 2008-12-31T23:30:00Z -> 2009-01-01T00:30:00+01:00
ok 10 - 2009-01-01T00:00:00Z -> 2009-01-01T01:00:00+01:00
ok 11 - 2009-03-01T12:00:00Z -> 2009-03-01T13:00:00+01:00
ok 12 - 2009-03-29T00:59:59Z -> 2009-03-29T01:59:59+01:00
ok 13 - 2009-03-29T01:00:00Z -> 2009-03-29T03:00:00+02:00
ok 14 - 2009-03-30T12:00:00Z -> 2009-03-30T14:00:00+02:00
ok 15 - 2009-07-01T00:00:00Z -> 2009-07-01T02:00:00+02:00

So now I know that I have a failing test due to rounding occuring. For my XMLTV project it is not important as I will not have to handle decimal seconds.

Voici une nouvelle version de mon script de conversion des stations Vélib’ en base POI pour Maemo Mapper sur Nokia N800 (voir mon post précédent).

Parmi les améliorations :

• un script pour automatiser le processus de transformation 
• le renseignement du champ poi_id, ce qui corrige le problème d'édition des POI 
• un index sur la table POI sur les champs latitude, longitude en espérant que cela améliore les temps d'accès. En pratique, je n'ai constaté aucune amélioration ;
• 4 catégories de POI au lieu d'une seule, pour créer votre propre base des stations : Station Vélib’, Station Vélib’ corrigée, Station Vélib’ validée, Station Vélib’ erronée ;
• des modifications d'implémentaiton XSLT : l'utilisation de xsl:foreach et un traitement exhaustif des caractères de l'adresse de la station.

Lisez la suite pour le nouveau code...