Florent Lebreton

Temporarily disable all indexes of a postgresql table

2018-04-10T00:00:00+02:00

When you run a large query (insert/update) on a huge table with several indexes, these indexes can seriously slow the query execution.

With Postgresql it can be very faster to disable the indexes before runing the query and reindex all the table afterwards.

You can do it like this :

Disable all table indexes

UPDATE pg_index
SET indisready=false
WHERE indrelid = (
    SELECT oid
    FROM pg_class
    WHERE relname='<TABLE_NAME>'
);

Run your query

UPDATE <TABLE_NAME> SET ...;

Reenable all table indexes

UPDATE pg_index
SET indisready=true
WHERE indrelid = (
    SELECT oid
    FROM pg_class
    WHERE relname='<TABLE_NAME>'
);

Reindex table

REINDEX <TABLE_NAME>;

I saved some time with this trick so I just wanted to share it!

__fle__

Reset a PostgreSQL sequence and update column values

2014-08-31T00:00:00+02:00

Yesterday, I understood that I had broken a sequence of an auto-increment column in my PostgreSQL database. This quick tip will show you how to reset the sequence, restart it to a specific value, and recreate all values of the column

Example

I have a table which stores estate properties. Each property is associated to list of pictures (gallery). A joint table does this association

db=> \d weather
          Table « public.property_pictures »
   Colonne   |  Type   |                           Modificateurs
-------------+---------+-------------------------------------------------------------------
 id          | integer | non NULL default, nextval('property_gallery_id_seq'::regclass)
 property_id | integer | non NULL
 picture_id  | integer | non NULL

My property_gallery_id_seq sequence is dizzy and generates conflict on the id values.

Solution

Here is a solution.

Set temporary values intentionally far of existing values to avoid conflicts when we recompute the column:

UPDATE property_pictures SET id=10000+nextval('property_gallery_id_seq');

Restart sequence to 1

ALTER SEQUENCE property_gallery_id_seq RESTART WITH 1;

Rewrite all column values

UPDATE property_pictures SET id=nextval('property_gallery_id_seq');

Note: Obviously, be very careful if the column is used a foreign key ... Also, do not forget to search WHY your sequence has been broken anyway ...

This solution is strongly inspired by this stackoverflow post

I hope this trick can save you some time too!

__fle__

Easy Webmapping with django-leaflet & django-geojson

2014-06-29T00:00:00+02:00

Let's start with an important thing: I'm a real newbie in webmapping. I've never used Postgis, Mapnik, OpenLayers or <place any webmapping tool here>. Projection, SRID, ... are just some wild words for me.

For an estate management application, I wanted to place some markers on a map : a list of properties found after a search, the location of a consulted property, ... classic. As a Django developer, this application is written with my favorite framework!

Mathieu told me about django-geojson and django-leaflet, two django apps he baked at Makina Corpus. Clearly, he didn't lie to me, django-geojson and django-leaflet make webmapping really lightweight and friendly.

The magic thing is that you don't need a spatial database or some complex geographic libraries! Mathieu and I have removed the last dependencies to GEOS a few days ago. I'm glad I could make this tiny contribution!

Django-geojson

django-geojson allows to manipulate GeoJSON (a JSON format for encoding geographic data structures) in a Django project. You can (de)serialialize objects and querysets, serve map layers through django views and store geographic data in JSON fields.

Django-leaflet

django-leaflet provides a way to play with Leaflet very easily in a Django project. It embeds Leaflet assets, provides a form widget to edit geometries and a templatetag to display maps. In addition, it's highly configurable through django settings.

An Example

Everybody loves mushrooms? Follow me, I'll show you some spots!

So I want to reference some mushroom spots, search for them and show them on a map. Here is the simple model which uses a PointField provided by django-geojson:

# models.py
from djgeojson.fields import PointField
from django.db import models

class MushroomSpot(models.Model):

    geom = PointField()

For the simplicity, when I create a mushroom spot instance, I want to point its location directly on a map. The admin widget coming with django-leaflet will help me:

# admin.py
from leaflet.admin import LeafletGeoAdmin
from django.contrib import admin

admin.site.register(MushroomSpot, LeafletGeoAdmin)

The admin creation form looks like this and the widget allows me to place my marker easily:

For information - and I think it's crazy you don't even have to know this to make it all work! -, here is the JSON structure stored in database:

{
  "type":"Point",
  "coordinates":[
    -1.4058208465576172,
    47.15301133231325
  ]
}

Edition is easy, now you'll see that display is too. To display a map, I use templatetags and filters provided by the two modules:

leaflet_js and leaflet_css loads my Leaflet assets
geojsonfeature helps me to serialize my MushroomSpot instances in a GeoJSON structure
leaflet_map shows up the map!

My first view aims to show search results, stored in a queryset named qs_results:

# mushroomspot_list.html
{% extends "base.html" %}
{% load leaflet_tags %}
{% load geojson_tags %}

{% block extra_assets %}
  {% leaflet_js %}
  {% leaflet_css %}
{% endblock %}

{% block content %}

    <script type="text/javascript">
      var collection = {{ qs_results|geojsonfeature|safe }};
      function map_init(map, options) {
          L.geoJson(collection).addTo(map);
      }
    </script>

    {% leaflet_map "spots" callback="window.map_init" %}

{% endblock %}

The simple code above gives me something like this:

Django-geojson can serialize a queryset, but it can also serialize a simple model instance. So for the detail view of a mushroom spot, code is almost the same, excepted the variable name of my data:

var collection = {{ mushroom_spot|geojsonfeature|safe }};

Note that this will dump the whole JSON into the DOM. You can also define a layer view, and obtain the data via Ajax.

A little more

A great option of django-geojson allows to automatically serialize instance properties in the standard GeoJSON feature dictionnary properties. Let's use it to add a pop-up on the marker!

I slightly adapt my model to add a description and a photo, which will be parts of my popup content. I also a write a property popupContent whose content will be serialized in the GeoJSON structure:

# models.py
from djgeojson.fields import PointField
from django.db import models

class MushroomSpot(models.Model):

    geom = PointField()
    description = models.TextField()
    picture = models.ImageField()

    @property
    def popupContent(self):
      return '<img src="{}" /><p><{}</p>'.format(
          self.picture.url,
          self.description)

I've just to change the call of geojsonfeature alittle to specify properties I want to serialize and - for this particular use case - to use the Leaflet option onEachFeature:

<script type="text/javascript">
  var collection = {{ mushroom_spot|geojsonfeature:"popupContent"|safe }};

  function onEachFeature(feature, layer) {
    if (feature.properties && feature.properties.popupContent) {
      layer.bindPopup(feature.properties.popupContent);
    }
  }

  function map_init(map, options) {
    L.geoJson(collection, {onEachFeature: onEachFeature}).addTo(map);
  }
</script>

And that's it!

What's next?

My nexts tests will be to play with more complex geometries like lines or polygons. Django-geojson provides more fields than PointField like MultiPointField, PolygonField, ...
For my estate management application, filling may be more simple if the marker was automatically positioned after that the user has wrote the address. I think I could use GeoPy for this.
An other solution would be to integrate Leaflet GeoSearch plugin to the django-leaflet admin widget.

See you!

If you have some observations or questions about this post, please leave a comment below.

Let's keep in touch on twitter or through this blog feed!

Detect value changes between lines with PostgreSQL

2014-05-21T00:00:00+02:00

A few days ago, in a Django project, I had to solve a SQL problem that I had never met yet. Something like : "The last time that this column value has changed between a row and the next one". Crap...How?

By requesting help of regilero, who told me about PostgreSQL window functions.

Window Functions

To solve this, or any SQL query where you have to compare similar rows, PostgreSQL provides a usefull functionnality: Window Functions. PostgreSQL 9.1.13 Documentation introduces this feature by saying:

A window function performs a calculation across a set of table rows that are somehow related to the current row. This is comparable to the type of calculation that can be done with an aggregate function. But unlike regular aggregate functions, use of a window function does not cause rows to become grouped into a single output row — the rows retain their separate identities. Behind the scenes, the window function is able to access more than just the current row of the query result.

Different built-in window functions allows to compute rank of a row in a partition, get previous or next row value, etc. This kind of function must be invoked using window function syntax (i.e. with an OVER clause).

A simple example

Let's take a quite useless but simple example:

Each day, I take note on weather (How warm is it ? Is it rainy ?).
I want to extract some information like:
- On What day did the weather change?
- When did it start to rain for the last time ?
- ...

In a database, a very simple representation looks like this:

db=> \d weather
          Table « public.weather »
    Column      |  Type   | Modifiers
    ------------+---------+---------------
    day         | date    | non NULL
    temperature | integer | non NULL
    rainy       | boolean | non NULL

and with sample data:

db=> SELECT * FROM weather ORDER BY day DESC;
     day       | temperature | rainy
    -----------+-------------+-------
    2014-04-08 |          22 | f
    2014-04-07 |          20 | f
    2014-04-06 |          16 | t
    2014-04-05 |          18 | t
    2014-04-04 |          19 | t
    2014-04-03 |          22 | f
    2014-04-02 |          20 | f
    2014-04-01 |          18 | t

The very intersting part is here : thanks to window functions lag and lead, I can select for each row the column values of the previous and next rows:

SELECT
    day,
    rainy,
    lead(rainy) OVER (ORDER BY day DESC) as prev_rainy,
    lag(rainy) OVER (ORDER BY day DESC) as next_rainy
FROM
    weather
ORDER BY
    day DESC

    day     | rainy | prev_rainy | next_rainy
------------+-------+------------+------------
 2014-04-08 | f     | f          |
 2014-04-07 | f     | t          | f
 2014-04-06 | t     | t          | f
 2014-04-05 | t     | t          | t
 2014-04-04 | t     | f          | t
 2014-04-03 | f     | f          | t
 2014-04-02 | f     | t          | f
 2014-04-01 | t     |            | f

Note: Obviously, the ORDER BY clause is very important here.

By nesting this in an other query, I can detect value changes between rows of my table. For example, the query below gives "each day on which the weather changed" (switch of the rainy boolean):

SELECT
    w1.day, w1.rainy
FROM
    (SELECT
        w2.day,
        w2.rainy,
        lead(w2.rainy) OVER (ORDER BY w2.day DESC) as prev_rainy
     FROM
        weather w2
     ORDER BY
        w2.day DESC) as w1
WHERE
    w1.rainy IS DISTINCT FROM w1.prev_rainy
ORDER BY
    w1.day DESC;

    day    | rainy
-----------+-------
2014-04-07 | f
2014-04-04 | t
2014-04-02 | f
2014-04-01 | t

Based on this first selection, I can easily extract some other information like "the last time the weather began to be nice":

SELECT
    w1.day, w1.rainy
FROM
    (SELECT
        w2.day,
        w2.rainy,
        lead(w2.rainy) OVER (ORDER BY w2.day DESC) as prev_rainy
     FROM
        weather w2
     ORDER BY
        w2.day DESC) as w1
WHERE
    w1.rainy IS DISTINCT FROM w1.prev_rainy
AND
    w1.rainy IS FALSE
ORDER BY
    w1.day DESC
LIMIT 1;

    day    | rainy
-----------+-------
2014-04-07 | f

Go further

Use case above is just an example focused on window functions lag and lead (I have no idea how to solve this kind of problem without them) but PostgreSQL provides other usefull builtin window functions.

In addition, it's possible to call any built-in or user-defined aggregate function as a window function!

Stay tuned

Keep in touch on twitter, through this blog feed or by commenting this article below!

[FR] Ce billet en français sur le blog de Makina Corpus : Détecter un changement de valeurs entre deux lignes avec PostgreSQL !

An efficient GIT workflow for mid/long term projects

2014-02-12T00:00:00+01:00

Our full-web CAMM project at Makina Corpus has been going on for nearly two years and is running in production for over 18 months. I think it's my first project without any headache about our codebase and VCS management. So, I'll present our GIT workflow which has proven to be very effective for now.

Context

Several developers
Several staging/pre-production servers, several (non-synchronous) production servers
Monthly releases (more or less) with delivery on staging, then on production servers
On servers, basecode is directly pulled from the GIT repository with fabric

Rules

To handle this, we have set some simple rules:

One (and only one) maintainer, who manage GIT repository and releases
Never commit directly on master
Never rebase master on any branch
Do not get out of planned workflow

Workflow

Master branch

Our branch master is the common trunk and simply contains all the codebase of the next release. Since we don't work directly on it, it evolves mainly with merges.

Development branches

When a developer starts a new feature or a bugfix, he creates a new branch from master HEAD

$ (master) git checkout -b featureA
$ (featureA) git commit -a -m "featureA part 1"
$ (featureA) git commit -a -m "featureA part 2"

He follows branch master evolution and regularly ensures his code still works, by rebasing his branch featureA on branch master.

$ (featureA) git rebase master

When his developments are done (commits fa1 / fa2 in schema below), he does a last rebase. Thanks to this:

he ensures that the maintainer will be able to merge easily (maintainer should not need to read code deeply and search why there are conflicts)
if tests pass on development branch after rebase, they should pass on master after merge, so we ensure that branch ``master`` is always working well

Possibly, it will be the good time to clean the development branch to let it neat just when it is finished.

The maintainer can now merge this branch in master peacefully, without big conflict troubles. As the maintainer, I like to use no-ff option to force a merge commit, so history can stay really readable (we easily see where the branch has started and where it has been merged).

$ (master) git merge --no-ff featureA

Now that the branch has been merged, the developer should remove his development branch.

$ (master) git branch -d featureA
$ (master) git push origin :featureA

Stable branches

When we prepare a release, we update CHANGELOG (with our workflow, a git log --oneline should be quite clear to do that) and tag the branch master (optional), then we start a stable branch.

$ (master) git tag 1.0
$ (master) git checkout -b stable1.0
$ (stable1.0) git push origin stable1.0

This branch is deployed on different servers.

While development goes on, we possibly have to do some hotfixes (for example: commit hf1 in schema below), that must be sent in production quickly. These hotfixes are done directly on concerned stable branch.

Regularly, the maintainer merges stable branch in master to bring back these commits. This action is particularly important before the next release.

$ (master) git merge --no-ff stable1.0

We found this method really useful because:

each stable branch has its own life and doesn't take care of branch master evolution, so we can hotfix stable branche freely and without stress
we ensure that no hotfix commit has been lost before next release (avoid regressions)

A complete history example

Conclusion

Of course, there are several GIT workflows which can be very efficient, but we found many advantages in working with this method, and no real issue:

Branch master is always clean and working well
Developers don't care about GIT whole workflow
We can fix stable branch without asking ourselves what happened on master since last release
We ensure that each stable release contains new features and possible fixes
Always working with branches and using``-no-ff``option make history really clear !
This workflow is scalable (number of developers or branches doesn't really matter)

If you liked this article, stay tuned ! I sometimes tweet and post GIT articles !

[FR] Ce billet en français sur le blog de Makina Corpus : Un workflow GIT efficace pour les projets à moyen/long terme !

EDIT : Tag and stable branch should not share the same name. Thanks regilero !

GIT tip : A simple .gitconfig file

2014-02-04T00:00:00+01:00

As several friends have asked me this, here is my ~/.gitconfig base file.

Nothing special, just a few aliases and some syntax highlighting :).

[user]
        name = Florent Lebreton
        email = florentlebreton@free.fr
[color]
        ui = auto

[color "branch"]
        current = yellow reverse
        local = yellow
        remote = green

[color "diff"]
        meta = yellow bold
        frag = magenta bold
        old = red bold
        new = green bold
        whitespace = red reverse

[color "status"]
        added = yellow
        changed = green
        untracked = cyan

[core]
        whitespace=fix,-indent-with-non-tab,trailing-space,cr-at-eol

[alias]
        st = status
        ci = commit
        br = branch
        co = checkout
        df = diff
        dc = diff --cached
        lg = log -p
        pr = pull --rebase
        gr = log --all --graph --decorate --oneline

Simple and basic but I hope you'll have found something useful in it. If yes, you may want to know that I sometimes tweet and post stuff about GIT !

Thanks opidentica for having noticed the error!

Combine LDAP and classical authentication in django

2013-12-20T00:00:00+01:00

Still in our CAMM project JOB at Makina Corpus, we needed to provide an authentication system through a LDAP server. The module django-auth-ldap does most of work itself but we decided to combine this LDAP authentication with the classical django authentication for two reasons:

keep possibility to create users only in django without having them in the LDAP directory (mainly for superusers);
be able to fallback on django authentication for all users in case of a LDAP server failure.

So to handle this, we have to:

differentiate LDAP directory users from django database users;
store password of LDAP users in django to be able to switch on classical authentication;
implement two custom authentication backends.

User model

We just implement a custom model user with a boolean from_ldap to diffentiate LDAP users from classical django users.

class MyUser(AbstractUser):
    from_ldap = models.BooleanField(
        _('LDAP user'),
        editable=False,
        default=False)

LDAP authentication backend

This LDAP backend has two goals :

Save user password in django database, thus he'll be able to log in django if LDAP authentication backend is disabled;
Force from_ldap field to True when a user is created by this way.

from django_auth_ldap.backend import LDAPBackend
from django.contrib.auth import get_user_model

class MyLDAPBackend(LDAPBackend):
    """ A custom LDAP authentication backend """

    def authenticate(self, username, password):
        """ Overrides LDAPBackend.authenticate to save user password in django """

        user = LDAPBackend.authenticate(self, username, password)

        # If user has successfully logged, save his password in django database
        if user:
            user.set_password(password)
            user.save()

        return user

    def get_or_create_user(self, username, ldap_user):
        """ Overrides LDAPBackend.get_or_create_user to force from_ldap to True """
        kwargs = {
            'username': username,
            'defaults': {'from_ldap': True}
        }
        user_model = get_user_model()
        return user_model.objects.get_or_create(**kwargs)

Classical authentication backend

We override django.contrib.auth.backends.ModelBackend to ensure LDAP users can't logged in with this one if MyLDAPBackend is available.

from django.contrib.auth import get_backends, get_user_model
from django.contrib.auth.backends import ModelBackend

class MyAuthBackend(ModelBackend):
    """ A custom authentication backend overriding django ModelBackend """

    @staticmethod
    def _is_ldap_backend_activated():
        """ Returns True if MyLDAPBackend is activated """
        return MyLDAPBackend in [b.__class__ for b in get_backends()]

    def authenticate(self, username, password):
        """ Overrides ModelBackend to refuse LDAP users if MyLDAPBackend is activated """

        if self._is_ldap_backend_activated():
            user_model = get_user_model()
            try:
                user_model.objects.get(username=username, from_ldap=False)
            except:
                return None

        user = ModelBackend.authenticate(self, username, password)

        return user

Django settings and fallback solution

Normally, we have our two backends activated :

LDAP users can only connect through MyLDAPBackend;
Django users can connect through MyAuthBackend.

AUTHENTICATION_BACKENDS = (
    'accounts.backends.MyLDAPBackend',
    'accounts.backends.MyAuthBackend',
)

In case of a LDAP directory failure, we just have to disable MyLDAPBackend and everybody can connect with MyAuthBackend.

AUTHENTICATION_BACKENDS = (
    #'accounts.backends.MyLDAPBackend',
    'accounts.backends.MyAuthBackend',
)

Thanks leplatrem for review!

[FR] Ce billet en français sur le blog de Makina Corpus !

GIT tip : Keep your branch clean with fixup and autosquash

2013-12-04T00:00:00+01:00

Who is not tired of committing a "Remove pdb" or a "Fix a typo" few minutes or hours after committing a clean feature ? A few time ago, I discovered two useful options in GIT that work together : git commit --fixup and git rebase --autosquash. With these, you can easily merge little fixes with the original feature and keep your branch clean.

Preferably, you won't use it in a stable or master branch, because rebase rewrites history and can create a big mess, mainly if project counts several developers. It rather can be convenient to clean a development branch before merging it in master.

--fixup & --autosquash

git commit --fixup <commit> automatically marks your commit as a fix of a previous commit
git rebase -i --autosquash automatically organize merging of these fixup commits and associated normal commits

Example

Take a git repos with a branch dev. You intend to commit features A and B:

$ (dev) git add featureA
$ (dev) git commit -m "Feature A is done"
[dev fb2f677] Feature A is done
$ (dev) git add featureB
$ (dev) git commit -m "Feature B is done"
[dev 733e2ff] Feature B is done

Your work is in progress and you find minor mistakes in Feature A : it's time to use --fixup option !

$ (dev) git add featureA                # you've removed a pdb : shameful commit
$ (dev) git commit --fixup fb2f677
[dev c5069d5] fixup! Feature A is done

Here, you see that GIT automatically retrieved featureA commit message prefixed by fixup!.

All work is done, let's see the log:

$ (dev) git log --oneline
c5069d5 fixup! Feature A is done
733e2ff Feature B is done
fb2f677 Feature A is done
ac5db87 Previous commit

Now, you want to clean your branch before merging it : it's time to use --autosquash option !

$ (dev) git rebase -i --autosquash ac5db87
pick fb2f677 Feature A is done
fixup c5069d5 fixup! Feature A is done
fixup c9e138f fixup! Feature A is done
pick 733e2ff Feature B is done

This command has opened your editor with lines above. Just save & quit and ... :

$ (dev) git log --oneline
ff4de2a Feature B is done
5478cee Feature A is done
ac5db87 Previous commit

Your shameful commit has been merged properly with the original feature. It's just a shorcut for something you could do otherwise but I find it very convenient :).

That's all folks !

EDIT : git rebase i <after-this-commit> must be launched with as argument the last commit you want to retain as-is, not the first one you want to change.

Thanks regilero & SebCorbin for reviewing!

Capturez une signature manuscrite avec django-jsignature

2013-11-27T00:00:00+01:00

Aujourd'hui, tous les documents (ou presque) générés par l'activité d'une entreprise sont créés informatiquement. Dans certains cas, l'impression d'un document est nécessaire uniquement pour apposer une signature (de l'envoyeur, du destinataire, ou autre).

Dans le cadre d'un projet de GMAO full web chez Makina, nous avons mis en place, pour les techniciens itinérants, un formulaire de saisie de rapport signé sur tablette tactile.

Ce process, réalisé sur place et immédiatement après l'intervention, permet un gain de temps important en évitant les doubles saisies, les impressions et les échanges mails et courriers.

Bien que la signature "électronique-manuscrite" n'est pas une preuve totalement valable aux yeux de la loi, pouvoir apposer une signature via une tablette tactile peut être vraiment utile et éviter des impressions et des échanges inutiles :).

La fonctionnalité de signature a été réalisée avec jSignature, wrappée dans une app django.

jSignature et django-jsignature

jSignature est un plugin jQuery qui transforme un simple <div> en cadre prêt à recevoir une signature déssinée à la souris ou mieux, sur écran tactile. Quelques options de configuration sont disponibles et l'export de la signature est possible dans différents formats (image base/64, image base/30, json, ...).

Nous avons publié django-jsignature, une petite app django permettant :

d'avoir facilement un champ "signature" dans un formulaire django (avec un field et un widget);
de faire le rendu de l'image en python (avec Pillow);
de stocker la signature (json) et la date de signature dans un modèle (champs fournis par un mixin);

Le billet complet est sur le blog de Makina.

Le code est dispo sur github.

Quand le poney rencontre le lion - Djangocong Belfort 2013

2013-09-29T00:00:00+02:00

Cette année la Djangocong avait perdu son accent du sud pour partir à l'aventure et rencontrer le lion de Belfort. L'aventure est un mot pas si mal choisi quand on vient de Nantes car j'en ai profité pour tester tous les moyens de transport existant (ou presque) :

Tram 2 [Nantes] - 12h30
Tram 1 [Nantes]
TGV [Nantes -> Paris]
Bus 91 [Montparnasse -> Gare de Lyon]
Lyria [Paris -> Belfort-Montbéliard TGV]
Navette [Belfort-Monbeliard TGV -> Montbéliard] (OK là je me suis un peu perdu)
TER [Montbeliard -> Belfort]
Pieds [Gare -> Hotel] - 20h45 o/

Maiiiis ça valait le coup, la liste non ordonnée et non exhaustive de "Pourquoi il faut aller au 'congs ?":

Les gens sont cools
Ça rassure ("Ha tiens il n'y a pas que moi qui ai ce problème !")
Ça motive ("Wouah trop cool ces projets et ces technos qui bougent !")
On découvre plein d'outils pratiques : pytest, les nested formsets, la nouvelle lib django atomic, ...
On boit de la bière et on mange bien
On geek dans l'amphi, en classe, dans le train, ...
...

Ça c'était pour les anecdotes perso : le vrai retour est sur le blog de Makina Corpus.

Merci aux organisateurs et à l'année prochaine o/.

Playing with LiveServerTestCase and Selenium

2012-11-25T00:00:00+01:00

At "Djangocon Tolosa", Julien phalip spoke about a new feature in django 1.4, LiveServerTestCase. A LiveServerTestCase launches a true HTTP server and gives a way to make HTTP requests in test cases. It can be useful to test REST API over a full HTTP protocol stack. Or, associated with Selenium, it can be powerful to make functionnal tests, on a view using ajax for example.

Let's try it with a simple usecase : * a view which displays 10 first results of a dummy search in a <table> * a view which returns dynamically the 5 next results * a button wich launches an ajax request on this second view to fetch 5 more results and append it to the current list

Here is a simple code extract of each part :

# urls.py
...
url(regex=r'^results_page$',
    view='results_page',
    name='results_page'),
url(regex=r'^get_more_lines$',
    view='get_more_lines',
    name='get_more_lines'),
...

# views.py
...
def results_page:
    """ Renders a page with result list in a <table> """

def get_more_lines:
    """ Returns dynamically 5 more lines to be displayed in table """
...

# template.html
...
<table id="results">
  {% for r in results %}
    <tr><td>{{ r }}</td></tr>
  {% endfor %}
</table>
<a id="more-lines" href="#">Get more lines</a>
...

// scripts.js
...
$('#more-lines').click(function(e){
  $.ajax({
    url: '/get_more_lines',
    success: function(response){
      if(response) {
        $('#results').append(response);
      } else {
        $('#more-lines').addClass('disabled');
      }
    }
  });
});
...

Note : code above is incomplete but interesting part is below.

Now we want to test it little bit. For example :

Ensure that table contains 10 lines at begining
Ensure that table contains 15 lines after click on "Get more lines" button
Ensure that button becomes disabled if there is no more result

It's not really a unit test for js function or django view. Looks like more a quick functional test.

Here is the corresponding LiveServerTestCase :

from selenium.webdriver.firefox.webdriver import WebDriver
from selenium.webdriver.support.wait import WebDriverWait
from django.utils.unittest import SkipTest
from django.test import LiveServerTestCase
from django.core.urlresolvers import reverse

class ResultListTestCase(LiveServerTestCase):

    @classmethod
    def setUpClass(cls):
        """ Instantiate selenium driver instance """
        cls.selenium = WebDriver()
        super(ResultListTestCase, cls).setUpClass()

    @classmethod
    def tearDownClass(cls):
        """ Quit selenium driver instance """
        cls.selenium.quit()
        super(BaseSeleniumWebDriverTestCase, cls).tearDownClass()

    def _wait_ajax_complete(self):
        """ Wait until ajax request is completed """
        WebDriverWait(self.selenium, 10).until(
            lambda s: s.execute_script("return jQuery.active == 0")
        )

    def _has_css_class(self, selector, klass):
        """
        Returns True if the element identified by `selector`
        has the CSS class : `klass`.
        """
    return (self.selenium.find_element_by_css_selector(selector)
            .get_attribute('class').find(klass) != -1)


    def test_get_more_lines(self):
        """ Test result list and 'get more lines' button """

        # Display tested page
        url = reverse('results_page')
        self.selenium.get(self.live_server_url + url)

        # Ensure 10 lines are displayed
        rows_length = lambda: len(self.selenium.find_elements_by_css_selector('#results tr'))
        self.assertEqual(rows_length(), 10)

        # Click on 'get-more-lines' button
        self.selenium.find_element_by_id('get-more-lines').click()
        self.wait_ajax_complete()
        self.assertEqual(rows_length(), 15)

        # Click again and check button is disabled
        self.selenium.find_element_by_id('get-more-lines').click()
        self.wait_ajax_complete()
        disabled = self.has_css_class('#increase-history', 'disabled')
        self.assertTrue(disabled)

On my current project, tests are ran by Jenkins on a headless server, so Selenium can't launch a firefox. Awaiting for a specific configuration, I wrapped creation of WebDriver in a try/except like this :

class ResultListTestCase(LiveServerTestCase):

    @classmethod
    def setUpClass(cls):
        try:
            cls.selenium = WebDriver()
            super(ResultListTestCase, cls).setUpClass()
        except Exception as e:
            raise SkipTest('Selenium webdriver is not operational')

This is just a really simple first test but this feature seems pretty cool IMHO :-).

De retour de Drupagora 2011

2011-11-18T00:00:00+01:00

Je reviens de la première édition de Drupagora, à Paris, qui se veut plus oriéntée chefs de projet/décideurs que les habituels DrupalCamps de geeks. Au final, une formule plutôt satisfaisante : Conférences intéressantes, beaucoup de retours d'expérience et un peu de technique quand même (ouf !).

Mobilize, don't miniaturize

Un premier constat de base

Le mobinaute : * est boulimique d'information, très rapide, très volatile * navigue avec une connexion lente et instable * navigue sur un device peu performant

Il faut donc faire : léger, simple, concis, fontionnel

1. La mauvaise idée à la mode : une seule URL + Media Queries

Le responsive design n'est pas la panacée. L'idée est tentante, mais MediaQueries fait seulement de la réorganisation d'affichage, ce qui ne règle pas les problèmes d'ergonomie et de performances : * le volume de données transféré est trop gros : DOM (surtout avec Drupal), images, js, css * la consommation CPU est trop importante : scripts js et animations * le contenu est trop long : articles trop longs, trop de blocs, ...

=> Cette solution est donc à réserver pour un site simple et léger (évenementiel par exemple)

2. Une solution plus aboutie : 2 URL + 2 themes

(MobileTools + Domains + DomainThemes + DomainAccess + DomainViews + Panels, TinySrc)
Une URL de base pour le site sur ordinateur (rien de particulier à ce niveau)
Une URL pour device mobile. Quelques recommandations pour le version mobile :
Une des idées importantes est de profiter de la modernité des navigateurs mobiles (CSS3, HTML5, Storage) !!!

Au niveau du thème
- surcharger les .tpl pour alléger le DOM (DomainThemes)
- penser au responsive design pour les différents devices (OmegaTheme ou AdaptiveTheme + PanelEverywhere + TinySrc)
- éviter au maximum le javascript et la manipulation du DOM (consommation CPU importante) => Penser au CSS3, notamment à Translate3D (qui utilise le GPU et donc soulage le CPU).
- pour les formulaires, l'audio, la vidéo, éviter là encore la javascript / Flash => Penser au HTML5
- pour le js réellement nécessaire, se limiter à DOM Selector API ou jQuery Mobile
- pour les sessions et le cache, éviter l'utilisation de cookies (transfert de data) => Penser au Session storage, Local storage, DB storage
Au niveau du contenu
- offrir la possibilité de publier une version résumée des articles (DomainAccess)
- offrir la possibilité de publier une partie d'une page composite (variantes de Panel, Context, ...)

=> On a une version du site réellement adaptée au mobile avec :

Un contenu allégé, adapté au mode de lecture du mobinaute

Peu de HTML

Pas de JS

Beaucoup de cache navigateur

La gestion des 2 URLS nécessite:

Plan de redirection : il doit être implémenter au plus tôt dans l'architecture (load-balancer -> proxy-cache -> apache -> Drupal si pas possible de faire autrement)

Détection de browser : Device detection => WURFL (plus OpenSource) vs. Browser detection => BrowserCap

Attention à bien mettre en cache la redirection dans les headers HTTP pour ne pas la recalculer à chaque fois !

Exemple d'implémentation complète par Adyax : societegenerale.com

Les slides : http://www.drupagora.com/sites/default/files/slide/Drupal_et_Mobilite.pdf

Réussir son référencement et profiter de Drupal

Quelques trucs Google auxquels on ne pensent pas forcément

"Être en 1ère page de Google n'a plus vraiment de sens" : Penser à se référencer sur "tous" les sites google : maps, youtube, shooping, ... selon le type de site
Il est très largement préférable se positionner sur une multitude d'expressions qualifiées de 4/5/6 mots (longue traine), les expressions génériques sont inutiles
Quand on fait des tests de référencement, attention à la personnalisation des résultats qui peut être trompeuse (cookies, géoloc, sites visités régulièrement, ...)

Évolutions des algorithmes Google en 2010-11-12-...

Google Panda => Les pages à forte valeur ajoutée ont beaucoup plus d'importance que les aggrégateurs (news, comparateurs de prix, ...)
Vers plus de "social ranking" => Facebook like, Google +1
Arrivée du "person ranking" => La popularité de l'auteur de la page (Twitter, Facebook, Blog) va avoir une importance dans le positionnement de la page dans les prochains mois

La règle des 80/20 pour le référencement d'un site Drupal

Avec Drupal, 80% de l'optimisation du référencement d'un site = 5 optimisations:

Un bon title avec les bons mots-clés (en restant naturel : accents, majuscules, ...)

Une bonne URL (sans paramètre, avec les bons mots-clés)

Une bonne meta description pour attirer le clic (accroche du type "Livraison gratuite")

Une bonne structure de site

Une bonne structure sémantique HTML (h*, ul/li)

(bonus) Avoir un title + H1 + URL indépendants les uns des autres !

Les autres 20% :

Fil d'Ariane, taux de rebond (!!!), balises alternatives des images/vidéos, micro-formats, ...

Quelques modules Drupal

URLs : PathAuto, PathAlias, PathRedirect, GlobalRedirect, Token
Title : PageTitle, Taxonomy/Title
Metas : Nodewords
Sitemap : SitemapXML
Linking interne : CKELink

Les slides : http://www.drupagora.com/sites/default/files/slide/presentation-Sebastien-Monnier-Woptimo-drupagora%20-%20V2.pptx

Gestion unifiée des medias chez Radio France

RadioFrance est confronté au problème classique de la gestion des medias (sous D6 pour le moment) sur plusieurs sites. Leurs contraintes sont les suivantes :

Tout type de médias (photo, audio, video, flux externe, objets personnalisés)

Gestion de différents droits, licences, copyrights, expiration selon le media, la source, ...

Différents render de chaque media selon le context d'affichage (dans un wysiwyg ou non, sur device mobile, ...)

~ 1.000.000 de nodes

L'implémentation technique est basée sur le module Scald, initialement développé & utilisé par la radio publique de Chicago :

gère n'importe quel media

a son propre type de stockage unifiant (atoms) pour pallier à l'absence d'Entity sur D6

a sa propre gestion de cache

possède une gestion de "providers" pour pouvoir gérer des types de contenus media personnalisés !

D6 seulement, mais le portage D7 est prévu pour Q1 2012

La solution construite par RadioFrance consiste en une bibliothèque partagée qui liste tous les medias disponibles au contributeur (Scald + Views + Lightbox):

visible et accessible en permanence dans un bandeau droit vertical,

filtres ajax (titre, type de media, tags, licences, ...) + enregistrement de recherches personnalisées,

prévisualisation de n'importe quel media en lightbox

ajout d'un nouveau media lightbox (sans quitter le node en cours de création donc !)

L'intégration d'un media dans un contenu se fait par un simple Drag & Drop :

soit directement dans un WYSIWYG, avec le rendu final automatique. Exemple :

pour une photo : affichage de l'image + légende + copyright,

pour un audio ou une video : affichage du player thémé.

soit dans un fieldset du formulaire (stockage dans un field caché)

Ergonomiquement pour le contributeur ... c'est très sympa !

Pour le passage à D7, RadioFrance hésite entre Scald et Media :

Entity rend redondante la couche d'abstraction créée par Scald avec Atom

Pour le moment, l'interface et les fonctionnalités de Media sont plus basiques que Scald (integration Views inexistante ou très récente, gestion pauvre des licences/expiration, ...), mais il y a bien sûr la possibilité d'ajouter des plugins

Le développement de Media avance vite, et rattrape petit à petit Scald

=> Selon RadioFrance, impossible d'être influent sur l'orientation de Media face à Acquia, donc RadioFrance va choisir d'aider au portage de Scald sur D7 plutôt que de contribuer à Media ...

A tester aussi : le module MediaFront basé sur Media.

My short drush cheat sheet

2011-01-31T00:00:00+01:00

Drush (DRUpal SHell) is a command-line which allows you to administer your Drupal site in console.

It can be very useful when you have broken your site and when you can't access to your administration interface.

Drush command format:

$ drush [options] <command> [argument]

Below, a few basic commands offered by drush :

List all commands and get help:

$ drush
$ drush help

Install Drupal ! (Very useful to deploy a dev instance quickly):

-- Download latest Drupal ...
$ drush dl drupal
-- or select a version ...
$ drush dl drupal --select
-- and install it !
$ drush si --account-name=<USER> --account-pass=<PASS> --db-url=mysql://<DB_USER>:<DB_PASS>@localhost/<DB_NAME>

Clear Drupal caches (used 87 times today):

$ drush cc all

Install one or more module(s):

$ drush dl -y <MODULE> [<MODULE2>, ...]
$ drush en -y <MODULE> [<MODULE2>, ...]

Disable and uninstall module(s):

$ drush dis -y <MODULE> [<MODULE2>, ...]
$ drush pm-uninstall -y <MODULE> [<MODULE2>, ...]

Export database (intelligently, i.e. without cache tables and other stuff):

$ drush sql-dump --result-file=<DUMP_FILE.sql> --gzip

Launch Drupal cron:

$ drush cron

Update your Drupal site:

$ drush up

Of course, it's a very short list of basic drush commands, this post is just my drush newbie notes taken during JulienD presentation at #DCNantes. There is many other commands with drush core, some modules can extend this list and you can implement your own drush commands.

With some configuration, you can administer two sites or more easily.

Generate a weblog with pelican

2011-01-16T00:00:00+01:00

I'm testing pelican, a static blog generator written in python. From simple rst or markdown files (your blog entries), pelican can produce a nice blog with just a cli-tool command. Sources can be managed with your favourite DVCS and final site is easily hosted on any web server.

It seems to be nice so I try this tool, discover at he same time reStructuredText and blog it on the fly!

1 - Start with install pelican from pypi:

$ easy_install pelican

Note : Pelican has some dependancies so it could be better to install it in virtualenv.

2 - Simply create a minimal filesystem:

blog/sources       -- to store blog entry .rst files
blog/www           -- final web site directory
blog/settings.py   -- settings file

3 - Just edit settings.py to add general settings like title, author, URL and other stuff. Mine is very simple for now:

# -*- coding: utf-8 -*-
AUTHOR = u'Florent Lebreton (fle)'
SITENAME = u"/home/fle"
SITEURL = 'http://blog.fle.org'

PDF_GENERATOR = False

LINKS = (('planet django', 'http://planetdjango.org'),)

SOCIAL = (('viadeo', 'http://www.viadeo.com/fr/profile/florent.lebreton'),)

4 - Then, write my first blog entry in sources/01-test.rst:

Hello world
###########

:date: 2011-01-16
:tags: test, pelican
:category: python
:author: Florent Lebreton (fle)

Hello !

5 - Finally, just run the cli-tool like this:

$ cd blog
$ pelican -s settings.py -o www/ sources/

Pelican manages syndication, PDF generation, theming and other cool stuff. Take a look at Pelican page for more information.