Le blog de pingou - Tag - PostgreSQLLe blog de pingou, ses actualités sur Fedora, ses RPMs, ses tests, son Linux... :-)
Pingou's weblog, his fedora's news, his RPMs, his tests, his Linux... :-)2022-02-17T10:46:15+01:00pingouurn:md5:66db5ce1ed1a80cb2f424695b4bb7780Dotcleardatanommer/datagrepper investigationsurn:md5:cad1a1168211ad7359bf3e281c586b982021-02-25T09:31:00+00:002021-02-26T17:25:21+00:00Pierre-YvesGénéralDatabasedatagrepperFedoraFedora-InfraPostgreSQL<p>A few team members of the <a href="https://docs.fedoraproject.org/en-US/cpe/">CPE</a> team have investigated how to improve the performances of datanommer/<a href="https://apps.fedoraproject.org/datagrepper/">datagrepper</a>.</p> <p>A little while ago, we <a href="https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/6NRUH7EP6ERTBUEVTTXYLA25QUSHTKBE/">announced</a> that we would be look at optimization for datanommer/datagrepper.</p>
<p>The main issue being that it has grown over the course of its life (it's been running for 9 years now!) and the 180 millions messages are all stored in a single table making the application slow to perform some of the queries it does. Queries can even be so slow in some situations that the web server ends up raising a 504 gateway time-out error.</p>
<p>So we looked at a few ways to improve performances.</p>
<h4>Default delta</h4>
<p>We found out that most of the queries that time-out are requests that do not include a <code>delta</code> in their arguments. Turns out this is because, if no delta is specified, datagrepper counts all the messages that fit the given criteria, since the beginning of the database and in some cases, that can lead to querying all the 180 millions messages in the database.</p>
<p>Turns out that datagrepper has a configuration option to specify a default delta if none is provided by the user. This alone helps datagrepper quite a bit and allows to not run into time out error while just browsing its UI.</p>
<p><br />
<br /></p>
<p>Once we figured out how to improve the UI, we started looking at how to improve the database side. Our first try was to manually partition the <code>messages</code> table which contains all the messages.</p>
<h4>Manually partitioning the database</h4>
<p>Our first attempt was to partition the message by year, having a partition for each year. However, the way the partitioning works means the field which is used for partitioning is the only field that can be unique across all partitions, as a consequences the field must be part of all foreign keys.</p>
<p>We stopped our first attempt there as we didn't want to adjust all the foreign key constraints.</p>
<p>Our second attempt was to partition the database by <code>id</code> which is the primary key of the <code>messages</code> table, which solves the foreign key question since these constraints rely on that field.
We partitioned the table in chunks of 10 million records. So we created 19 partitions and loaded the data in them.</p>
<p><br />
<br /></p>
<p>With this partitioning and the default delta, we started seeing some good results but we also wanted to test the <a href="https://www.timescale.com/">timescaledb</a> postgresql plugin.</p>
<h4>The timescaledb postgresql plugin</h4>
<p>That plugin is designed to improve performance of databases that store time-related data. In the case of datagrepper its main use case it to store messages and retrieve them based on time information (and potentially other criteria). So timescaledb sounds like a good candidate for our use-case.</p>
<p>Timescaledb gets set-up on <code>timestamp</code> fields. So we set up timescaledb on the <code>timestamp</code> field of the <code>messages</code> table. Once set-up and the data imported we realized that timescaledb also does table partitioning, meaning the issue we had earlier about the foreign key constraints and the year appeared again (but this time on the <code>timestamp</code> field).
This time however, we decided to adjust the tables linked to the <code>messages</code> table to include the <code>timestamp</code> field in the foreign key constraints.</p>
<p>This led to some good gain in performance with one little issue which is that we found some duplicated messages in the database. They have the same <code>msg_id</code> but different <code>timestamp</code>. We considered this to be an artifact of using fedmsg and we expect that moving to fedora-messaging will solve this issue as using rabbitmq will ensure that messages are only processed by one consumer at a time. It could be that this is also cause by the bridge between fedora-messaging and fedmsg, in which case datanommer may have to be adjusted for checking if a <code>msg_id</code> exists in the db before inserting a new message.</p>
<h4>timescaledb without external tables</h4>
<p>We thought that simplifying the database model maybe a way to optimize some of the queries some more. So we've changed the current database schema:
<a href="https://blog.pingoured.fr/public/datanommer_db.jpeg" title="datanommer_db.jpeg"><img src="https://blog.pingoured.fr/public/.datanommer_db_m.jpg" alt="datanommer_db.jpeg" style="display:table; margin:0 auto;" title="datanommer_db.jpeg, Feb 2021" /></a></p>
<p>We moved the user and package information into the messages table, using <a href="https://www.postgresql.org/docs/current/arrays.html">postgresql arrays</a> in the hope that combined this with <a href="https://www.postgresql.org/docs/current/gin-intro.html">Generalized Inverted Index (GIN)</a> we would have some optimization.</p>
<p>However, when testing the queries in postgresql directly, we saw that as soon as the query involved an ordering by timestamp as well as filtering on other criteria (such as package's name or user's name) the performances dropped. We thus never adjusted datanommer and datagrepper to work with this setup (which is why you will not find it in the results section below)</p>
<p><br />
<br /></p>
<p>We have of course tried to measure our different experiments. So let's see how they look.</p>
<h4>Results</h4>
<h5>Environments</h5>
<p>We used four different environments for our tests:</p>
<ul>
<li>prod/openshift</li>
</ul>
<p>This is a openshift deployment of datagrepper which hits the production postgresql database and is configured just like the actual (VM-based) production instance is.</p>
<ul>
<li>prod/aws</li>
</ul>
<p>All our experiments being done in AWS, we needed a production-like instance to avoid comparing the production instance which has all the production traffic and load to single instance on AWS that have no load or traffic.
So this instance is just like the production instance with one difference, it has a default delta specified in its configuration (with a value of 3 days, so if no delta is specified, it returns 3 days worth of messages).</p>
<ul>
<li>partition/aws</li>
</ul>
<p>This is the AWS instance that is running datanommer and datagrepper with the <code>messages</code> table partitioned by <code>id</code>. Datagrepper is also configured with a 3 days default delta.</p>
<ul>
<li>timescaledb/aws</li>
</ul>
<p>This is the AWS instance that is running datanommer and datagrepper with the <code>messages</code> table configured (and thus partitioned) by timescaledb. Datagrepper is also configured with a 3 days default delta.</p>
<h5>Requests</h5>
<p>Our test script launches 10 threads, each of them doing 30 requests (so 300 requests are made in total) and we ran it against six different requests:</p>
<ul>
<li><code>filter_by_topic: /raw?topic=org.fedoraproject.prod.copr.chroot.start</code></li>
<li><code>Plain_raw: /raw</code></li>
<li><code>Filter_by_category: /raw?category=git</code></li>
<li><code>Filter_by_username: /raw?user=pingou</code></li>
<li><code>Filter_by_package: /raw?package=kernel</code></li>
<li><code>Get_by_id: /id?id=2019-cc9e2d43-6b17-4125-a460-9257b0e52d84</code></li>
</ul>
<h5>Graphs</h5>
<p><a href="https://blog.pingoured.fr/public/datanommer_percent_sucess.jpg" title="datanommer_percent_sucess.jpg"><img src="https://blog.pingoured.fr/public/.datanommer_percent_sucess_m.jpg" alt="datanommer_percent_sucess.jpg" style="display:table; margin:0 auto;" title="datanommer_percent_sucess.jpg, Feb 2021" /></a></p>
<p>As you can see timescaledb is the only environment in which all requests returned successfully! It is also important to keep in mind the poor performance of <code>prod/openshift</code> when seeing the other results.
The <code>aws/partition</code> environment performed the least well on the <code>get_by_id</code> request. This can be explained by postgresql having to parse all partitions in parallel to find out which partition contains that specific <code>msg_id</code>. Timescaledb performed better there, potentially thanks to optimizations that timescaledb has that we missed when doing the partitioning manually.</p>
<p><a href="https://blog.pingoured.fr/public/datanommer_req_per_sec.jpg" title="datanommer_req_per_sec.jpg"><img src="https://blog.pingoured.fr/public/.datanommer_req_per_sec_m.jpg" alt="datanommer_req_per_sec.jpg" style="display:table; margin:0 auto;" title="datanommer_req_per_sec.jpg, Feb 2021" /></a></p>
<p>timescaledb pretty much outperform all other environment for all queries here.</p>
<p><a href="https://blog.pingoured.fr/public/datanommer_mean_per_req.jpg" title="datanommer_mean_per_req.jpg"><img src="https://blog.pingoured.fr/public/.datanommer_mean_per_req_m.jpg" alt="datanommer_mean_per_req.jpg" style="display:table; margin:0 auto;" title="datanommer_mean_per_req.jpg, Feb 2021" /></a></p>
<p>We lacked actual results for most of the <code>prod/openshift</code> requests there and here as well timescaledb has the lowest mean request time for each request.</p>
<p><a href="https://blog.pingoured.fr/public/datanommer_max_per_req.jpg" title="datanommer_max_per_req.jpg"><img src="https://blog.pingoured.fr/public/.datanommer_max_per_req_m.jpg" alt="datanommer_max_per_req.jpg" style="display:table; margin:0 auto;" title="datanommer_max_per_req.jpg, Feb 2021" /></a></p>
<p>Here again, timescaledb outperforms all other environments for each request.</p>
<h4>Conclusions</h4>
<p>Seeing these graphs, you can probably already guess what our recommendations are:</p>
<ul>
<li>Set a default delta in datagrepper's configuration file - even though this is going to break the API, but not specifying a delta today results in 504 errors more often than not.</li>
<li>Port the database to use timescaledb. We could probably replicate some of the gain by doing the partitioning manually on the <code>timestamp</code> field, but timescaledb takes care of all of this for us.</li>
</ul>
<p><br />
<br />
<br /></p>
<h5>References:</h5>
<ul>
<li><a href="https://fedora-arc.readthedocs.io/en/latest/datanommer_datagrepper/index.html">Report of the investigations</a></li>
<li><a href="https://pagure.io/fedora-infra/arc/blob/main/f/scripts/migration_timescaledb.sql">SQL script to migrate the database</a> (probably needs some polishing but provides a basis)</li>
<li><a href="https://fedora-arc.readthedocs.io/en/latest/datanommer_datagrepper/pg_timescaledb.html#patch">Datanommer patch to support the new database schema</a></li>
</ul>Setting up pagure on a banana piurn:md5:80ab4a6fe54023902acabe33fef409fe2016-01-05T18:59:00+00:002016-01-18T14:17:03+00:00Pierre-YvesGénéralDocumentationFedoraFedora-planetpagurePostgreSQLPython<p>This is a small blog post about setting up <a href="https://pagure.io/pagure">pagure</a> on a <a href="http://www.bananapi.org/">banana pi</a>.</p> <p><a href="https://blog.pingoured.fr/public/forge.png" title="forge.png"><img src="https://blog.pingoured.fr/public/.forge_m.png" alt="forge.png" style="display:block; margin:0 auto;" title="forge.png, Jan 2016" /></a></p>
<p>My friend <a href="http://kushaldas.in/">Kushal</a> recently convinced me to buy a small <a href="http://www.bananapi.org/">banana pi</a> to play with.</p>
<p>I installed the <a href="https://arm.fedoraproject.org/">Fedora 23 ARM server image</a> on
it, works like a charm and today I thought of trying to set-up
<a href="https://pagure.io/pagure">pagure</a> on it.</p>
<p>I figured this was worth a little blog post.</p>
<p>So here we go</p>
<h5>1/ Install the image on the SD card:</h5>
<p>For me it was something like:</p>
<pre>
sudo fedora-arm-image-installer \
--image=~/Downloads/Fedora-Server-armhfp-23-10-sda.raw.xz \
--target=Bananapi --media=/dev/mmcblk0 \
--selinux=OFF
</pre>
<p>You may have to adjust the path to the image as well as the media on which
you're installing.</p>
<h5>2/ Fix /boot/extlinux/extlinux.conf</h5>
<p>Add console=tty0 to line starting with append</p>
<p>There is a bug in the image preventing the initial-setup from being started
correctly. This fixes it.</p>
<h5>3/ Boot the SD card and fill in the initial-setup as prompted</h5>
<p>and from now on we're on the banana pi.</p>
<h5>4/ Install and enable ntp</h5>
<pre>
sudo dnf install ntp
sudo systemctl enable ntpd
</pre>
<h5>5/ Set-up the network to use networkd</h5>
<p>With these changes we drop the user of NetworkManager or network all together to
rely instead of networkd.</p>
<pre>
sudo systemctl disable NetworkManager network
sudo systemctl enable systemd-networkd systemd-resolved
sudo systemctl start systemd-networkd systemd-resolved
sudo rm -f /etc/resolv.conf
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
</pre>
<p>Then we want to give a static IP to this pi:</p>
<pre>
sudo vim /etc/systemd/network/eth0.network
</pre>
<p>And put in this file the following content:</p>
<pre>
[Match]
Name=eth0
[Network]
Address=192.168.1.20/24
Gateway=192.168.1.255
NTP=pool.ntp.org
</pre>
<p>Of course, adjust the Address and Gateway to your needs</p>
<h5>6/ Install pagure</h5>
<p>Pagure is present in the Fedora repo, but in order to benefit from the latest
changes I made a build of the latest git.
To build the latest version:</p>
<pre>
git clone https://pagure.io/pagure.git
cd pagure
python setup.py sdist
rpmbuild -ts dist/pagure*
koji build --scratch f23-candidate /path/to/your/pagure-...src.rpm
</pre>
<p>Then just get the RPMs from koji and install them:</p>
<pre>
dnf install ./pagure* postgresql-server python-psycopg2 mod_ssl
</pre>
<h5>7/ Configure PostgreSQL</h5>
<p>First instantiate the PostgreSQL server:</p>
<pre>
postgresql-setup --initdb
</pre>
<p>Edit the authentication method to md5 in:</p>
<pre>
vim /var/lib/pgsql/data/pg_hba.conf
</pre>
<p>Create the DB and the user to access it:</p>
<pre>
sudo -u postgres psql
CREATE DATABASE pagure;
CREATE USER pagure;
ALTER USER pagure WITH ENCRYPTED PASSWORD '--';
GRANT ALL PRIVILEGES ON DATABASE pagure to pagure;
GRANT ALL PRIVILEGES ON ALL tables IN SCHEMA public TO pagure;
GRANT ALL PRIVILEGES ON ALL sequences IN SCHEMA public TO pagure;
</pre>
<h5>8/ Adjust the pagure configuration files</h5>
<p>Adjust pagure's configuration file:</p>
<pre>
vim /etc/pagure/pagure.cfg
</pre>
<p>See <a href="https://docs.pagure.org/pagure/configuration.html">the doc for the configuration</a>
for more information about the different options.</p>
<p>Configure sqlalchemy.url in the alembic configuration file:</p>
<pre>
vim /etc/pagure/alembic.ini
</pre>
<p>Un-comment the WSGI file to get pagure running:</p>
<pre>
vim /usr/share/pagure/pagure.wsgi
</pre>
<h5>9/ Create the database and stamp it for alembic</h5>
<p>Once the pagure configuration file is set, we can point pagure_createdb.py to it
and as such create the database where we want it:</p>
<pre>
PAGURE_CONFIG=/etc/pagure/pagure.cfg \
python /usr/share/pagure/pagure_createdb.py
</pre>
<p>Once the database is created, we can tell alembic that we are running the latest
version of the database (so that upgrading the database scheme later using alembic
will work fine).</p>
<pre>
cd /etc/pagure
alembic stamp $(alembic heads |awk '{ print $1 }')
</pre>
<h5>10/ Adjust gitolite</h5>
<p>By default on Fedora, gitolite3 comes with its own gitolite3 user and group.
To make it nicer, instead of offering the repos at gitolite3@host:repos
we want to offers them at git@host:repos. To do this, we rename the
gitolite3 user to git.</p>
<pre>
usermod --move-home --login git --home /srv/git/ gitolite3
groupmod --new-name git gitolite3
</pre>
<h5>11/ Create the folders needed for gitolite and the git repos</h5>
<p>This has to be consistent with the home folder of the git user we adjusted
just above:</p>
<pre>
mkdir -p /srv/git/repositories/{,docs,forks,requests,tickets}
mkdir -p /srv/git/remotes
mkdir -p /srv/git/.gitolite/{conf,keydir,logs}
chown -R git:git /srv/git/.gitolite/
chown -R git:git /srv/git/repositories
chown -R git:git /srv/git/remotes
</pre>
<p>Beware that this should be consistent with what is in pagure.cfg and gitolite requires the repos
to be in a repositories folder.</p>
<h5>12/ Set up the apache server</h5>
<ul>
<li>Create SSL certs with letsencrypt (optional)</li>
</ul>
<pre>
dnf install letsencrypt
letsencrypt --text --email py_pagure@pingoured.fr \
--domains pagure.pingoured.fr \
--agree-tos --renew-by-default --manual certonly
</pre>
<ul>
<li>Create the release folder</li>
</ul>
<pre>
mkdir /var/www/releases
chown git:git /var/www/releases
</pre>
<ul>
<li>Adjust the apache configuration</li>
</ul>
<pre>
cd /etc/httpd/conf.d
vim pagure.conf
</pre>
<p>You will probably need to adjust it for your need. The defaults are basic but
close to what is running on pagure.io. It will be up to you to de-comment the
parts you want.</p>
<h5>13/ Open the ports 80 and 443</h5>
<p>Since this is a Fedora 23 image, it's using firewalld, let's use it to open the
http and https ports.</p>
<pre>
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --reload
</pre>
<h5>14/ Let apache read the git repos so we can clone over http</h5>
<p>We use file ACL to allow apache to read the git repos belonging to the git user</p>
<pre>
setfacl -Rdm user:apache:rx /srv/git
setfacl -Rm user:apache:rx /srv/git
</pre>
<h5>15/ Start and enables the services we use</h5>
<p>Final step:</p>
<pre>
systemctl restart httpd postgresql
systemctl enable httpd postgresql
</pre>
<p><br />
<br /></p>
<p>After this all I had to do was to place the .gitolite.rc file in /srv/git using <a href="https://infrastructure.fedoraproject.org/infra/ansible/roles/pagure/frontend/templates/gitolite.rc">the one from pagure.io</a> and this made gitolite work as well.</p>
<p><br />
<br /></p>
<h5>Conclusion</h5>
<p>This setup is nice and fun to play with, but pagure on banana pi is pretty slow, so I'm not sure I want to keep it there, might require more power than the banana pi can provide.</p>
<p>Something which I want to do though, is adjusting this deployment to work with a local ipsilon, this would make this deployment a little closer to what a good production setting could look like, but that's something for another blog post :-)</p>
<p>Cheers!</p>
<p><br />
<br />
The ironsmith image on the left side is from <a href="https://www.flickr.com/photos/deviceone/2869154930/">deviceone</a> and is licensed CC-2.0 BY-NC-SA</p>Faitout changes homeurn:md5:e72cf44f30565b5eb5d4a65a7685d8fe2015-08-05T11:02:00+01:002015-08-05T10:06:38+01:00Pierre-YvesGénéralDatabasefaitoutFedoraFedora-planetjenkinsPostgreSQLPythonUnit-tests <p><a href="http://faitout.fedorainfracloud.org/">Faitout</a> is an application giving you full access to a postgresql database for 30 minutes.</p>
<p>This is really handy to run tests against.</p>
<p>For example, for some of my applications, I run the tests locally against a in-memory sqlite database (very fast) and when I push, the tests are ran on jenkins but this time using faitout (a little slower, but much closer to the production environment). This setup allows me to find early potential error in the code that sqlite does not trigger.</p>
<p>Faitout is running the cloud of the Fedora infrastructure and since this cloud has just been rebuilt, we had to move it.
While doing so, faitout got a nice new address:</p>
<p><a href="http://faitout.fedorainfracloud.org/">http://faitout.fedorainfracloud.org/</a></p>
<p>So if you are using it, don't forget to update your URL ;-)</p>
<p><br />
<br />
See also: <a href="http://blog.pingoured.fr/index.php?tag/faitout">Previous blog posts about faitout</a></p>Faitout, 1000 sessionsurn:md5:ff0d178095d372a3a1cf7c46d5617f902014-06-26T16:28:00+01:002014-06-26T16:28:00+01:00Pierre-YvesGénéralDatabasefaitoutFedoraFedora-planetjenkinsPostgreSQLPythonUnit-tests <p><a href="http://blog.pingoured.fr/index.php?post/2013/10/28/Faitout-test-against-a-real-database">A while back</a>, I introduced <a href="http://209.132.184.152/faitout/">faitout</a> on this blog.</p>
<p>Since then I have been using it to tests most if not all the project I work on.
I basically use the following set-up:</p>
<pre>
DB_PATH = 'sqlite:///:memory:'
FAITOUT_URL = 'http://209.132.184.152/faitout/'
try:
import requests
req = requests.get('%s/new' % FAITOUT_URL)
if req.status_code == 200:
DB_PATH = req.text
print 'Using faitout at: %s' % DB_PATH
except:
pass
</pre>
<p>This way, if I have network, the tests are run with faitout and thus against a
real postgresql database while if I do not have network, they run against a
sqlite in memory database.</p>
<p>This set-up allows me to work offline and still be easily able to run all the
unit-tests as I change the code.</p>
<p>What the point of this blog was actually more to announce the fact that despite
it's limited spread (only 25 different IP addresses have requested sessions),
the tool is used and it has already reached the 1,000 sessions created (and
dropped) in less than a year.</p>
<p><br />
<br /></p>
<p>If you're not using it, I am inviting you to have a look at it, I find it
marvelous in combination with Jenkins and it does help finding bugs in your
code.</p>
<p>If you are using it, congrats and keep up the good work!!</p>Faitout - test against a real databaseurn:md5:bab888d759c3ca1fd2caca7c395ab5de2013-10-28T15:32:00+00:002013-10-28T15:32:00+00:00Pierre-YvesGénéralDatabasefaitoutFedoraFedora-planetPostgreSQLPythonUnit-tests <ul>
<li>Do you do unit-tests?</li>
</ul>
<ul>
<li>Do you do continuous integration?</li>
</ul>
<ul>
<li>Do you use sqlite for your tests while deploying against postgresql?</li>
</ul>
<ul>
<li>Do you hate using sqlite for your tests?</li>
</ul>
<p><br /></p>
<p>If you answer 'yes' to any of those three questions, the following post is for you.</p>
<p>Otherwise, well, stay, it might still be interesting ;-)</p>
<p>When doing unit-tests you want to have something fast which allows you to quickly see if your last changes affect other part of your code.</p>
<p><a href="http://www.sqlite.org/">sqlite</a> is great for that. You can easily create in memory database, no FileIO, it all goes fast and smooth.</p>
<p>That is until you push your application to production where it is deployed against a real database system such as <a href="https://blog.pingoured.fr/index.php?post/2013/10/28/">PostgreSQL</a>. Then suddenly, queries which run fine under sqlite start breaking under PostgreSQL.
sqlite and PostgreSQL implements some things differently and this leads to this kind of situation.</p>
<p>The solution for this is of course to run your tests in an environment as close as possible from the production on, ie: run your tests on the same database system as the one you use on production.</p>
<p>But this can also become complex, it means setting up a new database server, create a new database, clean the database after the tests, handle permissions...</p>
<p>With this in mind, project such as <a href="http://www.postgression.com/">postgression</a> appeared.</p>
<p>The idea is simple: easily get access to postgresql databases which are thrown away after a certain time.</p>
<p>The problem is that postgression is not FOSS, thus when a couple of weeks ago there was no way to get a database, there was also no way to set up our own postgression server that could be used by a restricted number of person.</p>
<p>So after discussing it with <a href="https://fedoraproject.org/wiki/User:Abompard">Aurélien</a>, somewhere between lunch and dessert, faitout appeared.</p>
<p>The idea was simple, have a small web application, create on the fly a user and a database made available to the on who asks and after 30 minutes (via a cron job for the moment) destroy the database and the user.</p>
<p>The API is pretty simple and all is documented on the front page of the application.</p>
<p>So feel free to have a look at it, test it, break it (but let us know how you did that ;-)) at the test instance we have:</p>
<p><a href="http://209.132.184.152/faitout/">http://209.132.184.152/faitout/</a></p>PostgreSQL vs MongoDBurn:md5:2234e75e59154d171bcd38233a6e8a3f2012-05-20T11:15:00+01:002012-05-20T14:10:51+01:00Pierre-YvesGénéralBenchmarkFedoraFedora-planetmongoDBPostgreSQLPython<p><img src="https://blog.pingoured.fr/public/source.png" alt="source.png" /></p>
<p>A comparative tests of postgresql vs mongodb</p> <p><strong><em>English version</em></strong></p>
<p>As you may know I have spent some time recently working on the <a href="https://fedorahosted.org/hyperkitty/">hyperkitty</a> program. The idea being to offer a new interface to the archives in mailman 3 (which has never been closer to a release).</p>
<p>Hyperkitty aims at implementing a numbers of the ideas developed by Máirín Duffy in <a href="http://blog.linuxgrrl.com/2012/02/29/7750-pixels-of-mailing-list-thread/">her</a> <a href="http://blog.linuxgrrl.com/2012/03/13/mailman-brainstorm/">blog</a> <a href="http://blog.linuxgrrl.com/2012/03/14/mailman-brainstorm-2/">posts</a>. The main one being to try to unify mailing lists and forum (ie: providing a web-interface to mailing lists).</p>
<p>In this quest, I have started to look some time ago to <a href="http://www.mongodb.org/">MongoDB</a>. It is a NoSQL database which is becoming quite popular (probably helped with its integration into <a href="https://openshift.redhat.com/">openshift</a>).
The results were satisfying but then a big question came up:</p>
<ul>
<li>Do we really want to impose the burden of 2 different database systems to our sysadmin for a mailman archives interface ?</li>
</ul>
<p>Of course, if we can avoid it, we would prefer to do it.</p>
<p>But MongoDB was performing really nicely with regards to searching the archives. So testing was needed.</p>
<h3>Hardware</h3>
<p>The machine on which I ran the test has:</p>
<ul>
<li>16G of ram</li>
<li>4 cores</li>
<li>2x1To in RAID1</li>
</ul>
<p>All this operated by RHEL 6.2 (Santiago).</p>
<h3>The databases</h3>
<ul>
<li>PostgreSQL version 8.4.9</li>
<li>MongoDB version 1.8.2</li>
</ul>
<h3>The data</h3>
<p>I used the archives from the <a href="https://lists.fedoraproject.org/mailman/listinfo/devel">devel</a> mailing list, since its creation in 2002.</p>
<ul>
<li>PostgreSQL loaded 166672 emails</li>
<li>MongoDB loaded 166642 emails</li>
</ul>
<p>So there is a difference of 30 emails which I considered to be negligible for the tests.</p>
<h3>The data structure</h3>
<h4>PostgreSQL</h4>
<p>PostgreSQL has one table per list and each table as the same (following) structure:</p>
<pre> id serial NOT NULL,
sender character varying(100) NOT NULL,
email character varying(75) NOT NULL,
subject text NOT NULL,
"content" text NOT NULL,
date timestamp without time zone,
message_id character varying(150) NOT NULL,
stable_url_id character varying(250) NOT NULL,
thread_id character varying(150) NOT NULL,
"references" text,</pre>
<p>Primary key: id</p>
<p>Index: date, message_id, stable_url_id, subject, thread_id</p>
<h4>MongoDB</h4>
<p>MongoDB being a NoSQL database using dictionary has a much more flexible structure.
For the test the same information was stored in a dictionary structure:</p>
<ul>
<li>Content</li>
<li>InReplyTo</li>
<li>From</li>
<li>Subject</li>
<li>ThreadID</li>
<li>Date</li>
<li>References</li>
<li>_id #internal id defined by MongoDB</li>
<li>MessageID</li>
<li>Email</li>
</ul>
<p> UPDATE: The fields which are searched are being indexed. So the following indexes are generated: </p>
<ul>
<li>Content</li>
<li>From</li>
<li>Subject</li>
<li>ThreadID</li>
<li>Date</li>
<li>References</li>
<li>MessageID</li>
<li>Email</li>
</ul>
<h3>The queries</h3>
<p>11 different types of queries were ran:</p>
<ul>
<li>get_thread_length: return how many emails are part of a thread</li>
<li>get_thread_participants: return the list of all the participants in a thread</li>
<li>get_email: return a given email based on its message_id</li>
<li>first_email_in_archive_range: for a time range, return the first email (ordered by date)</li>
<li>get_archives_range: return all the email in a time range</li>
<li>get_archives_length: return information of years and month since the creation of the list</li>
<li>get_list_size: return how many emails are archived for this list</li>
<li>search_subject: performs a search on the subject of the emails</li>
<li>search_content: performs a search on the content of the emails</li>
<li>search_content_subject: performs a search on the subject and the content of the emails</li>
<li>search_sender: performs a search on the sender of the emails (name and email)</li>
</ul>
<p>Each queries was run 30 times, thus allowing caching to take place.</p>
<p>The four search queries were ran using both case insensitive and case sensitive queries</p>
<p>Few addition for PostgreSQL:</p>
<ul>
<li>the search_sender and search_content_subject queries were ran twice in using a union of queries of simply a <strong>or</strong> statement</li>
</ul>
<h3>Results</h3>
<h4>Output</h4>
<p>The first we want to know is</p>
<ul>
<li>Did the queries return the same results ?</li>
</ul>
<p>The following queries returned the same result:</p>
<ul>
<li>get_email</li>
<li>get_archives_range</li>
<li>first_email_in_archives_range</li>
<li>get_thread_length</li>
<li>get_thread_participants</li>
<li>get_archives_length</li>
<li>search_subject</li>
<li>search_subject_cs</li>
<li>search_content</li>
<li>search_content_cs</li>
</ul>
<p>The other ones returned different results:</p>
<ul>
<li>search_content_subject</li>
</ul>
<pre>** Results differs
MG: 28762
PG: 34110
PG-OR: 28762</pre>
<ul>
<li>search_content_subject_cs</li>
</ul>
<pre>** Results differs
MG-CS: 24886
PG-CS: 28578
PG-OR-CS: 24886</pre>
<ul>
<li>search_sender</li>
</ul>
<pre>** Results differs
MG: 1883
PG: 3325
PG-OR: 1883</pre>
<ul>
<li>search_sender_cs</li>
</ul>
<pre>** Results differs
MG-CS: 1882
PG-CS: 1883
PG-OR-CS: 1882</pre>
<ul>
<li>get_list_size</li>
</ul>
<pre>** Results differs
MG: 166642
PG: 166672</pre>
<p>For this last query, the difference in the result was expected nothing new there.</p>
<p>But what about the four queries returning different results, well as mentioned the PG{,-CS} queries are using a union of two queries to retrieve the information, so basically what we have is some duplicates. We could remove the duplicate in python easily using a <code>set</code> but as that will only impact the performance even more.</p>
<p>When we compare the results returned by MongoDB with the results returned by PostgreSQL using a <strong>or</strong> statement, they are similar.
So we can conclude that the different queries are returning the same information.</p>
<p>Let's look at their performances now.</p>
<h4>Performance</h4>
<p>Results are presenting as a boxplot.</p>
<p>Legend:</p>
<ul>
<li><strong>CS </strong>in the title stands for <strong>C</strong>ase <strong>S</strong>ensitive query (as opposed to case insensitive)</li>
<li>MG: results for MongoDB</li>
<li>PG: results for PostgreSQL</li>
<li>PG-OR: results for PostgreSQL using a <strong>or</strong> statement in the query (as opposed to a union)</li>
</ul>
<p><a href="http://pingoured.fr/public/kittybenchmark/overview_simplified.png" target="_new"><img src="http://pingoured.fr/public/kittybenchmark/overview_simplified.png" alt="Overview of the benchmark" width="500" /></a></p>
<p>This is a simplified boxplot (as in the outliers are not presented in the picture) but the full boxplot (with the outliers presented) is also <a href="http://pingoured.fr/public/kittybenchmark/overview.png" target="_new">available</a></p>
<h4>Additionnal results:</h4>
<p>I also looked at the influence of case sensitive vs case insensitive queries within a database system.</p>
<h5>MongoDB</h5>
<p><a href="http://pingoured.fr/public/kittybenchmark/mg_sensitivity.png" target="_new"><img src="http://pingoured.fr/public/kittybenchmark/mg_sensitivity.png" alt="MongoDB sensitivity to case in queries" width="500" /></a></p>
<p>So apparently there is a 'case' effect, but looking at the y scale we can see that this difference is ~0.2 seconds. I believe this is negligible.</p>
<h5>PostgreSQL</h5>
<p><a href="http://pingoured.fr/public/kittybenchmark/pg_sensitivity.png" target="_new"><img src="http://pingoured.fr/public/kittybenchmark/pg_sensitivity.png" alt="PostgreSQL sensitivity to case in queries" width="500" /></a></p>
<p>In this case the 'case' effect is much stronger and can go up to ~4 seconds which is much more significant for us.</p>
<h3>Conclusions</h3>
<p>Looking at the box plot</p>
<ul>
<li>PostgreSQL performs worse than MongoDB in the retrieval queries but the time it takes is negligible (we are way under 1 second)</li>
<li>For PostgreSQL case sensitive queries perform better than MongoDB but worse for case-insensitive queries.</li>
<li>Case sensitivity in the queries has a greater influence in the performances of PostgreSQL than of MongoDB.</li>
<li><strong>Or</strong> statement should be preferred to union of queries (nothing really surprising there).</li>
</ul>
<p><br /><br />So, do we want to perform case-insensitive queries and have 2 database systems to maintain or do we want an option to turn on case-insensitive and have one database system ? Or do we consider 7-8 seconds to be fine while searching in the content of 166000 emails ?</p>
<p><br />On a final note, everything I used to make the tests and the output of the said tests is available on <a href="http://ambre.pingoured.fr/cgit/mm_benchmark.git/">my git</a> and mirrored on <a href="https://github.com/pypingou/kittybenchmark">my github.</a></p>