Le blog de pingou

Every once in a while I want to run a program in parallel but gather its output in a single process so that I do not have concurrent accesses (think for example, several process computing something and storing the output in a file or in a database). I could use locks for this but I figure I could also use a queue.

My problem is that I always forget how I do it and always need to search for it when I want to do it again :-) So for you as much as for me here is an example:

# -*- coding: utf-8 -*-

import itertools
from multiprocessing import Pool, Manager


def do_something(arg):
    """ This function does something important in parallel but where we
    want to centralize the output, thus using the queue
    """
    data, myq = arg
    print data
    myq.put(data)
    myq.task_done()


data = range(100)
m = Manager()
q = m.Queue()
p = Pool(5)
p.map(do_something, itertools.product(data, [q]))


with open('output', 'w') as stream:
    while q.qsize():
        print q.qsize()
        item = q.get()
        print item
        stream.write('%s\n' % item)
    q.join()

There are probably other/better ways to do this but that's a start :-)

Today is the fourth day of flock. As usual the last three days have been really nice. I got to go to a number of interesting conferences and could even present a couple of project that I am or will be working on.

I assisted to the conference from Luke on how pushing updates in Fedora will look like in the coming months. Bodhi 2 is the new version of the application we use to manage our updates in Fedora. Luke and others have been working hard on it but the work they did is really impressive! Bodhi 2 looks better from all angles, UI, Infra, Workflow. Apparently the timeline is to get it deployed before the end of the year but after the release of Fedora 21, so stay tuned it's arriving ;-)

I have been able to assist on the presentation about python 3 in Fedora. I must say that this is looking promising and there are some new shiny things in python 3 that I am already looking forward for (most notably the possibility to have keyword only arguments in functions, this is going to be sweet).

On Thursday, I gave a presentation about the future Fedora Review Server (we couldn't find a better name for it and people seemed to like it :-)), more on that later.

The same day, Adimania presented a little bit his feeling and the state of things with regards to Ansible in the Fedora Infrastructure. I think it was a nice summary of why we are moving and what we like about Ansible.

Thursday afternoon, I went to the talk about NoSQL in Fedora Infrastructure. More than a state of things, it was a plead that we should consider and keep in mind the NoSQL technologies for the Infra and not fear using them where they make sense. Yograterol did a nice job presenting the different NoSQL technologies and clearly we should consider them where it makes sense. Thinking further about it with Raplh we thought that using MongoDB for datagrepper might be interesting, we should benchmark this :)

Finally yesterday I was able to present a little project I have been working on for a little bit progit, I will blog about this in the near future so keep in touch ;-)

Then I attended the talk from Kevin about the present and future of the Fedora infrastructure. This was a good overview of the different irons we have in the fire at the moment and those that near the fire aren't yet too hot. One thing is sure, I am really looking forward having our bugzilla hooked up on fedmsg!

The joint session on Fedora.next chaired by our dear FPL was also quite interesting and provided a very nice overview of what the different working group are currently up to. It was nice to see things moving forward, if some parts are still a little shady, I guess it won't remain this way for long anymore.

Yesterday afternoon, was a session on EPEL.next. There are still a number of concerns and questions about how things could or should be in EPEL. Some things are good and some could be improved, there are some generic idea (such as having a new repo: EPIC which would contain more rapidly evolving software or more recent version of software compared to what is currently in EPEL), but there again the devil is in the details and there will need to be some more thoughts and work before we can see this live.

I guess this is it for the talks, I attended a few more but I can't possibly detail them all here :-)

Next time, more info on what we actually got done during these few days!

If you are a Fedora packager, you are probably aware of the new pkgdb.

One question which has been raised by this new version is: should we change the process to request new branches or integrate new packages in the distribution.

The discussion has occurred on the rel-eng mailing list but I'm gonna try to summarize here what the process is today and what it might become in the coming weeks.

Current new-package procedure:

1. packager opens a review-request on bugzilla
2. reviewer sets the fedora-review flag to ?
3. reviewer does the review
4. reviewer sets the fedora-review flag to +
5. packager creates the scm-request and set fedora-cvs flag to ?
6. cvsadmin checks the review (check reviewer is a packager)
7. cvsadmin processes the scm-request (create git repo, create package in pkgdb)
8. cvsadmin sets fedora-cvs flag to +

New procedure

1. packager opens a review-request on bugzilla
2. reviewer sets the fedora-review flag to ?
3. reviewer does the review
4. reviewer sets the fedora-review flag to +
5. packager goes to pkgdb2 to request new package (specifying: package name, package summary, package branches, bugzilla ticket)
6. requests added to the scm admin queue
7. cvsadmin checks the review (check reviewer is a packager¹)
8. cvsadmin approves the creation of the package in pkgdb
9. package creation is broadcasted on fedmsg
10. fedora-cvs flag set to + on bugzilla
11. git adjusted automatically

Keeping the fedora-cvs flag in bugzilla allows to perform a regular (daily?) check that there are no fedora-review flag set as + that have been approved in pkgdb and whose fedmsg message hasn't been processed.

Looking at the number, it looks like there are more steps on the new procedure but eventually, most of them can be automated.

New branch process

For new branches, the process would be very similar:

1. packager goes to pkgdb2 to request new branch
2. requests added to the scm admin queue
3. cvsadmin checks the request (requester is a packager...)
4. cvsadmin approves the creation of the branch in pkgdb
5. branch creation is broadcasted on fedmsg
6. git adjusted automatically

A while back, I introduced faitout on this blog.

Since then I have been using it to tests most if not all the project I work on. I basically use the following set-up:

DB_PATH = 'sqlite:///:memory:'
FAITOUT_URL = 'http://209.132.184.152/faitout/'
try:
    import requests
    req = requests.get('%s/new' % FAITOUT_URL)
    if req.status_code == 200:
        DB_PATH = req.text
        print 'Using faitout at: %s' % DB_PATH
except:
    pass

This way, if I have network, the tests are run with faitout and thus against a real postgresql database while if I do not have network, they run against a sqlite in memory database.

This set-up allows me to work offline and still be easily able to run all the unit-tests as I change the code.

What the point of this blog was actually more to announce the fact that despite it's limited spread (only 25 different IP addresses have requested sessions), the tool is used and it has already reached the 1,000 sessions created (and dropped) in less than a year.

If you're not using it, I am inviting you to have a look at it, I find it marvelous in combination with Jenkins and it does help finding bugs in your code.

If you are using it, congrats and keep up the good work!!

It has been a little while since I last spoke about dgroc, the daily git rebuild on copr program.

The problem is that it kinda already lost its name...

Thanks to the great work of Miro Hrončok dgroc now supports mercurial as well :)

Miro took that opportunity to add a generic structure making it easy for other source configuration management software to be added!

Miro also fixed dgroc to take into account the release number used and automatically bump it upon rebuild.

So if you wanted to use dgroc but could not because you project was using mercurial, well now you have no excuses anymore!

Last week I have had the opportunity to attend my first Libre Graphic Meeting conference, this year located in Leipzig (Germany). Not being much of a graphic person, I must say that I was sometime lost a bit in some of the talks (being during a presentation or at the coffee corner), but on the other side I have learned a lot! I discovered a whole new side of the Open-Source Software community working on low-level tools and algorythms for image and video manipulation. Meeting these people with such a deep understanding of computer science and photo, for example, was a really extremely enriching experience.

I have learned about image manipulation and the cool effects provided in gimp by the g'mic project. I have met some of the dev of the darktable and inkscape projects, these guys are doing a remendous work, kudos!

On the Friday, we had a presentation about the gooseberry project by the Blender foundation. If you have not pledged to help them, go do it now!. Their project is amazing and need more help!

Another amazing talk was Sebastian Koenig presenting us his work on reviving a medial manikin with Blender. Basically the story is that a museum in Leipzig (the GRASSI Museum für Angewandte Kunst Leipzig) had this old maniki (22.5cm tall) which used to be animated, you could move her arms, legs, fingers or toes but it got old and stucked. So the museum in collaboration with the university did some sort of scan of the manikin and they asked Sebastian Köning to see if he could reconstruct the manikin using Blender. The resulting movie is now on display in the museum next to the actual manikin. I found this research amazing both from a technical and a historical point of view! And icing on the cake, since they had the mesh to reconstruct the animation on Blender they have also been able to make a 3D printed replicate of the manikin giving it back its ability to move.

On the last day, I was able to attend a workshop offered by Tobias Ellinghaus darktable developer and Patrick David about image manipulation on darktable and gimp. I had to leave before the end but the workshop was really, really interesting. Patrick did a live image editing demo in gimp, performing it on photo taken just a few minutes before. I need to practice this a little bit otherwise I'm going to forget all the good tips that were given.

I could also discovered some cool project related to DSLR such as the Magic Lantern which provides a new OS for Canon DSLR, awesome right? There is also the entangle project allowing to remotely control your DSLR, quite handy for macro or astro photos.

I was also given the possibility to contribute to the party. gnokii and I gave a presentation about nuancier as a FOSS contributing and voting application for wallpapers. I think it was well-received and there is apparently already interest to update it to support font. In addition, I took the opportunity, during a lightning talks session, to present HyperKitty and its demo instance which also seem to have brought quite some interest.

All in all, this was my first LGM meeting, I learned a lot about the whole libre graphic ecosystem, met a lot a new people and was given the opportunity to introduce a couple of projects dear to me. I really enjoyed it and would advise it to anyone interested, even remotely, into libre graphic.

Thank you for anyone that helped me attend or enjoying it (Gnokii, Ryan, Garrett, Chris and all the others ;-))!

As you may know, Fedora is under-going a rather large change with Fedora.Next proposition/evolution. One of the point that Fedora.Next addresses is the loss of users observed in Fedora for few years.

The statistics page on the wiki as well as my own representation of the same numbers are both out-dated so we don't really have a clear view on this.

However, since October 2012, all the messages sent onto the fedmsg bus are being stored in datanommer. And that's information we can use to see how we're doing with regards to our contributors.

I asked and got a dump of the datanommer database (the data is anyway publicly accessible in datagrepper) and ran my traditional script on it to gather some numbers.

I generated on a daily, weekly and monthly basis the graph of the number of (distinct) active contributors we have.

Here are the results :-)

Three interesting periods:

• the period without any messages in March 2013 is the period where the bus was down and we did not realize it (since we have a nagios check for the bus)
• on the daily graph you can really see the dip created by Christmas and its holidays
• on the monthly graph, the pick in August 2013 coincides with Flock and the launch of badges!

Looking at the graph generated by this week in fedora, in December we launched COPR and it seems that the number of posts on the planet has also quite increased in December and January. Does this alone explain the bump we observe here?

February is always a busy month and this year will be no exception.

Later today I am leaving for Brussel to attend FOSDEM over the week-end, with as added bonus a presentation on Sunday.

The presentation will be about fedmsg and its ecosystem and one of the particularity of this talk is that it will be done together with Nicolas Dandrimont (his blog) who is a Debian developer.

Think about how awesome it is to have in the distribution track a talk about a technology and its possibilities given by two person from two different distributions.

I must say I am looking forward this presentation :-)

Recently, while working on pkgdb2 I had a RFE for "dynamic" ownership.

The idea was to automatically change the owner based on who is actually working on the package.

With the change from "owner" to "point of contact" of a package, I thought that this might be an interesting idea. Of course in order to assess the feasibility and to investigate if it is really a good idea, we need some stats.

So I wrote a script that retrieves all the packages present in rawhide in Fedora. For each package it takes the last 100 actions (git commits, koji build and bodhi updates) and order the contributors from the most the least active. The script then checks the most active user versus the owner/point of contact of the package.

There is the output after running for 6h35:

14546 packages retrieved
14546 packages checked
85 packages w/ no package information
2877 packages w/ ausil as active point of contact
7132 packages won't change their point of contact
4451 packages will change of point of contact

I had to put appart ausil as he is the one doing the Mass-Rebuild and as such would become the point of contact of too many packages that have no other activity than Mass-Rebuild.

I still have the matrix of data available to extract more information about the distribution of the packages among the packager but I thought I would share this first.

What do you think?

Recently I have been thinking about a way to do mass-rebuild but only of packages that have not been built in a while (since the last release?).

At the moment, we only do mass-rebuild when there is a specific need to, for example a new version of GCC.

This is a very specific process which is ran over multiple days and just rebuilds all the packages. As a results, some packages that are of very low maintenance may just seat around, un-touched until the next mass-rebuild.

I was wondering if we could not simply take all the packages on rawhide and run, say once a month (or once a week, every day?), check when their last successfull build was and if older than X (to be defined), do a simple scratch build of the package. We could query koji or fedmsg via datagrepper to get the date of the last successful build of the package.

So technically it is duable, in theory it makes sense but the question is, in practice does it?

The first check to assess this is simply looking at the distribution last successfull dates of the packages.

So I wrote a small script querying the packagedb to get the list of all the packages and then queries datagrepper to retrieve the date of the last successful build. The number of days between this date and today is then computed and the output provides the number of packages that have been rebuild on each day.

Graphically it looks like this: On the X axis is presented the number of packages built on that day, on the Y axis is how many days ago that day was.

Converted to a log scale, we get: On the X axis is a log of the number of packages built on that day, on the Y axis is how many days ago that day was.

To provide some more statistics:

• 14397 packages were checked
• 49 packages were built yesterday (day 0, when the data was gathered),
• 1 package has not been successfully built since 271 days ago
• 66 packages have not been sucessfully re-built for 200 days or more
• 11418 packages have not been sucessfully re-built for 100 days or more
• The two peaks that can be seen are from 132 and 133 days ago (last mass-rebuild?)

Is this something worth persuing? Should we automatically re-build packages after a while and report in case the build fails?

What do you think?