Le blog de pingou

To content | To menu | To search

Tag - Database

Entries feed

Thursday, February 25 2021

datanommer/datagrepper investigations

A few team members of the CPE team have investigated how to improve the performances of datanommer/datagrepper.

Continue reading...

Wednesday, August 5 2015

Faitout changes home

Faitout is an application giving you full access to a postgresql database for 30 minutes.

This is really handy to run tests against.

For example, for some of my applications, I run the tests locally against a in-memory sqlite database (very fast) and when I push, the tests are ran on jenkins but this time using faitout (a little slower, but much closer to the production environment). This setup allows me to find early potential error in the code that sqlite does not trigger.

Faitout is running the cloud of the Fedora infrastructure and since this cloud has just been rebuilt, we had to move it. While doing so, faitout got a nice new address:

http://faitout.fedorainfracloud.org/

So if you are using it, don't forget to update your URL ;-)



See also: Previous blog posts about faitout

Thursday, June 26 2014

Faitout, 1000 sessions

A while back, I introduced faitout on this blog.

Since then I have been using it to tests most if not all the project I work on. I basically use the following set-up:

DB_PATH = 'sqlite:///:memory:'
FAITOUT_URL = 'http://209.132.184.152/faitout/'
try:
    import requests
    req = requests.get('%s/new' % FAITOUT_URL)
    if req.status_code == 200:
        DB_PATH = req.text
        print 'Using faitout at: %s' % DB_PATH
except:
    pass

This way, if I have network, the tests are run with faitout and thus against a real postgresql database while if I do not have network, they run against a sqlite in memory database.

This set-up allows me to work offline and still be easily able to run all the unit-tests as I change the code.

What the point of this blog was actually more to announce the fact that despite it's limited spread (only 25 different IP addresses have requested sessions), the tool is used and it has already reached the 1,000 sessions created (and dropped) in less than a year.



If you're not using it, I am inviting you to have a look at it, I find it marvelous in combination with Jenkins and it does help finding bugs in your code.

If you are using it, congrats and keep up the good work!!

Monday, October 28 2013

Faitout - test against a real database

  • Do you do unit-tests?
  • Do you do continuous integration?
  • Do you use sqlite for your tests while deploying against postgresql?
  • Do you hate using sqlite for your tests?


If you answer 'yes' to any of those three questions, the following post is for you.

Otherwise, well, stay, it might still be interesting ;-)

When doing unit-tests you want to have something fast which allows you to quickly see if your last changes affect other part of your code.

sqlite is great for that. You can easily create in memory database, no FileIO, it all goes fast and smooth.

That is until you push your application to production where it is deployed against a real database system such as PostgreSQL. Then suddenly, queries which run fine under sqlite start breaking under PostgreSQL. sqlite and PostgreSQL implements some things differently and this leads to this kind of situation.

The solution for this is of course to run your tests in an environment as close as possible from the production on, ie: run your tests on the same database system as the one you use on production.

But this can also become complex, it means setting up a new database server, create a new database, clean the database after the tests, handle permissions...

With this in mind, project such as postgression appeared.

The idea is simple: easily get access to postgresql databases which are thrown away after a certain time.

The problem is that postgression is not FOSS, thus when a couple of weeks ago there was no way to get a database, there was also no way to set up our own postgression server that could be used by a restricted number of person.

So after discussing it with Aurélien, somewhere between lunch and dessert, faitout appeared.

The idea was simple, have a small web application, create on the fly a user and a database made available to the on who asks and after 30 minutes (via a cron job for the moment) destroy the database and the user.

The API is pretty simple and all is documented on the front page of the application.

So feel free to have a look at it, test it, break it (but let us know how you did that ;-)) at the test instance we have:

http://209.132.184.152/faitout/