Le blog de pingou

Recently I have had to prepare a couple of demo about some work I have been doing. As my internet connection isn't the fastest, I chose to do a screencast that I could then upload somewhere and share. This prevented issues with my internet as well as gave me the possibility to show the full thing by editing the recording.

However, I ran into a few problems.

I first tried quite a few screencast apps:

• The screencast tool that is built-in in gnome (simply pressing ctrl+alt+shift+r). However, this is recording only very short screencast by default, changing this default meant editing a configuration value in dconf (and thus having some idea beforehand of how long the recording will be).
• recapp, that one simply didn't start me for
• peek, that one seemed to work but intercepted all my mouse clicks, so I could only navigate with the keyboard, could not highlight anything with my cursor and when I looked at the recording, it was all black
• SimpleScreenRecorder, could not seem to be stopped once started
• OBS studio, recorded a black screen

After all this, I gave up and re-log into my session using X11 instead of Wayland. Suddenly all of the screencast app worked fine... :)

So once I was able to record what I wanted to show, I still had over 10 minutes of video to show at a demo review, so I wanted to edit it, cut the parts where there are no progress, increase the speed of the parts where things are happening but do not need to be shown real-time (for example, when a system boots, or when it is being installed, if the speed there is x2, it is fine).

I've looked around at different tools and found:

• kdenlive
• VidCutter
• Video Trimmer
• ShotCut

I ended up settling for kdenlive for two reasons:

• someone I know uses it and recommended it to me (thus I knew it is able to do what I was looking for)
• I've found this tutorial on youtube explaining me exactly how to do what I wanted to do:

The kdenlive UI changed a little bit since this video was recorded (like the "Change speed" button is now available via a "right click" on the video track) but this tutorial is enough to give you some basis on video editing with kdenlive.

I use virt-manager to create and manage the virtual machines running on my laptops. However, by default, virt-manager wants to store the disk image for any new virtual machine it creates on /root/.local/share/libvirt/images/... (on Fedora 34, I remember it was on /var/lib/ on some earlier releases).

The issue is that my / is on a different partition than my /home and this one is much larger (400G) than / (60G). So if I place the disk images on either of the two locations above, I will end up quickly filling up my / which causes all kind of problems :-)

Since last week, I've created a bunch of virtual machines so I needed to figure out this storage location question.

I ended up finding out that I can simply create an empty qcow2 image that I can then use with virt-manager.

To create the qcow2 image simply run:

qemu-img create -f cow2 <name>.qcow2 <size>

For example:

qemu-img create -f qcow2 os_tree_gnome.qcow2 20G

This creates a 20G disk image named os_tree_gnome.cqow2, which can then be used by virt-manager.

To do this, when creating a new virtual machine with virt-manager, select import existing disk image, then browse to the qcow2 image you just created. Select the name of the OS, specify the number of CPUs and the memory the VM will have and on the last screen click on Customize configuration before install. This will allow to see the settings of the virtual machine before it is installed. That allows you to click on Add Hardware of type Storage. There select Device type and make it a CDROM device which allows you via the Manage button to pick the iso of your choice (boot.iso, dvd.iso...). Once the CDROM device is added, go to the Boot Options in the settings of the virtual machine and ensure that the CDROM is checked.

At this point you can click on Begin Installation and the VM will boot from the qcow2 file, which is empty so it will fall back on booting from the CDROM device which contains your ISO.

Recently we announced on devel-announce the upcoming changes to integrate anitya with dist-git.

Following this announcement, Till H. asked if there was a way to have an overview of the monitoring status of all the packages they maintain. I have replied that there is no such thing today but that I could cook up a little script to help with this.

So here it is: get_monitoring_status.py

Since the feature is only enabled in staging, you will need to specify the staging dist-git server for the script to be meaningful.

Here is a small example of output:

$ python3 get_monitoring_status.py pingou --url https://src.stg.fedoraproject.org/
rpms/ampy                                                              : monitoring-with-scratch
    https://src.stg.fedoraproject.org/rpms/ampy
rpms/audiofile                                                         : no-monitoring
    https://src.stg.fedoraproject.org/rpms/audiofile
rpms/boom                                                              : no-monitoring
    https://src.stg.fedoraproject.org/rpms/boom
rpms/bugwarrior                                                        : monitoring-with-scratch
    https://src.stg.fedoraproject.org/rpms/bugwarrior
rpms/datagrepper                                                       : no-monitoring
    https://src.stg.fedoraproject.org/rpms/datagrepper
rpms/datanommer                                                        : no-monitoring
    https://src.stg.fedoraproject.org/rpms/datanommer
...

This week, a good part of the Fedora Infrastructure team as well as some members from the CentOS Infrastructure team met up in Frederisksburg (Virginia, USA) for a few days of hacking together.

simple-koji-ci is a small fedmsg-based service that just got deployed in the Fedora Infrastructure.

It aims at doing something really simple: for each pull-request opened in pagure on dist-git, kick off a scratch-build in koji and report the outcome of this build to the pull-request.

This way, when someone opens a pull-request against a package that you are maintaining you can quickly see if that change would build (at least at the time the pull-request was opened).

This service is currently really simple and straight forward, dumb in many ways and still missing some desired features such as: - kick off a new scratch build in the PR is rebased/updated - allow package maintainer to retrigger the build manually but it is a start and we will work on improving it :)

Happy packaging!

PS: live example

I recently started looking at our dist-git usage but my data was a little limited.

Instead of querying datagrepper I managed to access directly the data in the database to get some stats:

Dist-git commits

Here is the output:

Over 1582 days (from 2012-10-08 to 2017-02-28)
   There was an average of 376.300884956 commits per day
   The median is of 327.0 commits per day
   The minimum number of commits was 1
   The maximum number of commits was 34716

For the average and median we removed all the days where there were more than 3,000 builds since they mostly concern mass-rebuilds (18 days were above 3000, and thus removed).

This is how it looks in a graph:

Commits in dist-git per day

Bodhi updates

Using the same data source, I went on to look at the number of bodhi updates flagged go to testing and the number of bodhi updates flagged to go to stable per day.

Here is the output:

Over 1541 days (from 2012-10-08 to 2017-02-28)
   there was an average of 76.9000648929 requests to testing per day
   The median is of 75 requests to testing per day
   the minimum number of requests to testing was 4
   the maximum number of requests to testing was 291
Over 1561 days (from 2012-10-08 to 2017-02-28)
   there was an average of 57.4477898783 requests to stable per day
   The median is of 54 requests to stable per day
   the minimum number of requests to stable was 1
   the maximum number of requests to stable was 217

(No data were removed there since there are no equivalent to mass-rebuild for these).

Graphically:

Updates requests for testing:

Updates requests for stable:

Flock is always a peculiar time of the year for me. For one it is one of the few time I get to meet with my colleagues but more than that, it's also one of the few time I get to spend a few days with fellows from this Fedora community that is so dear to me.

I have to say that this year was no exception. Flock 2016 has been really nice. I can, of course, only speak for myself, but from what I have seen we got a lot of work done and we are now ready to move forward on quite a few subjects.

One of the most important aspect of flock is the fact that an important part of the community gathers in one place, but we need to be careful as the conference only represent about 10% of all the Fedora contributors. So it is our duty as attendee to report to the broader community about the subjects that were discussed and the talks we have had.

It is of course practically impossible to mention everything here, for one because I took very little note during the conference, but I would like to point out the topics that appeared the most important to me during that conference.

Fedora at large and its community

During the opening keynote, mattdm gave an overview of how Fedora is appreciated outside of our community. It seems that Fedora 24 has been doing great, same for Fedora 23 before that. The IT world seems to appreciate the Fedora.next program we have started and what it is leading to.

Matt also gave a few numbers on the side of our community and our contributor base. These were numbers that had already been presented in his talk at DevConf 2016 (talk that I watched on youtube). So there were really new to me, but I still like the fact that there is about 66% of our community that is not working for our primary sponsor, Red Hat. This is healthy for our community, this diversity ensures that we are not just an echo chamber and that it is not just us liking what we do.

The Fedora Infrastructure

This is a part of the project that I am directly involved in and that I think made some really good progress during these few days. We had a few session. It started with a presentation from Kevin and I about the state of the Fedora Infrastructure. We went a little bit through the changes that happened in the last year and ones planned in the coming year, both from an infrastructure and an application point of view.

This has lead a few questions and discussions, all in a nice atmosphere. I had one comment on the presentation that we have not included as much numbers in it as we had last year making it a little harder for people not accustomed with our work to follow. Something to work on for next year.

We also had a workshop session. Over two hours we went through the changes we want to make to the infrastructure (opening our private cloud to our contributors, start investigate where and how to use docker, reflect the level of support provided to services by using different domain names for examples) and for each of these we came to some agreement and made a plan on how to move forward with it.

I will not go to much in the details of what we discussed and what agreement we reached in this blog post as it has already been summarized on the fedora infrastructure list.

Fedora Docker Layered Images

So this an project that has been worked on for a few months now by Adam Miller in coordination with the rel-eng team. The idea is to allow Fedora to start distributing more than just RPMs and in this case, Docker images.

This service is about to land. There are still a few aspect to be worked on, including how to distribute the images to the mirrors and how to ensure users are being redirected to a mirror that is up to date. Dennis, Randy and I had a very interesting discussion around the work that remains to be done for this. It will imply making changes to MirrorManager and likely all of its three components (MirrorManager, MirrorList and the backend services). It might also imply work on the docker side.

Being able to have these discussion while seating on comfortable harmchairs facing each other was really nice. We managed to have a list of applications that needs to be adjusted and a good idea of how the different pieces will work together.

Fedora atomic on a workstation

Patrick gave a very interesting presentation on how he builds and uses Fedora atomic to run it on its laptop. This was really most interesting but it gave a little bit of mixed feelings. On the one side, it looks really promising and exciting to work with, on the other side it seems not really user-friendly and a little hard/time-consuming. I do wonder if, some aspects could not be simplified for me (for example retrieving the list of RPMs currently installed on my machine to insert in the kickstart file instead of more or less starting from scratch).

Maybe I will try to make a little time available this year to try to play with this, it is really tempting.

Automation

Ralph has lead a very nice workshop on automation whose idea was to brainstorm around what we do and that we could automate and what we all have built script to do for us and which thus may need to be generalized.

The discussion has been lively and quite a few ideas were exchanged.

Two of them sticked with me a little more

• Generate a cron job gathering information from pkgdb, koji, bodhi, fedocal to

give access at a single location about releases. What are they koji tags? What are their bodhi name? What are their current status (released? beta-freeze? beta released? alpha?...)? There are a few applications relying on this information and while a good part of it is present in pkgdb, not all is and it does not necessarily make sense to add it there.

• Create some sort of service that triggers builds upon git push and even the

creation of bodhi update if we want. There are quite a few use-case that people would like to see supported and some people do not want this at all, so this should be entirely opt-in. Currently is idea is to ask packager to place a ChangeLog file in their git repo, next to the spec and the sources files and place in this file the information needed to create the bodhi update. If in a push, this ChangeLog file is updated, automatically trigger the build in koji and if it finished successfully, create the update in bodhi. That means that: without this ChangeLog file, nothing changes from the current situation, if the ChangeLog is not touched, nothing changes from the current situation, if the ChangeLog file is touched but the build fails in koji, the only change from the current situation is that this service will have saved the user from triggering the build manually.

These were the two ideas that stick with me the most. There has been more discussed and there are more possibilities (like making the service something that is ran locally by the user, as opposed to something ran in the infrastructure).

Pagure

Pagure has been a really nice surprise to me. Many people talked to me about it most often in good terms and sometime with some interesting ideas. I am not sure all the ideas provided will be implemented but there is food for thoughts and enough to keep me busy a little while!

Modularity

I had been following the modularity working group from a little far and I have been quite happy to discuss directly with the people working on this about the work they are trying to achieve.

Ralph and I have had a few lengthy discussion around the life-cycle of packages in this new model and, among others, the impact this would have on tools such as pkgdb. It seems clear to me that while the data model might not necessarily change that much, we will need to adjust pkgdb for this new distribution model. All the details are still not entirely clear, some features will need to be added, the UI will need to be adjusted, overall probably not enough to worth a rewrite of pkgdb but still enough that I will need to spend some cycles on it.

The work done by the modularity group is quite fascinating, I have been involved or the spectator of some the discussion they had and there is really quite a lot of work still to be done and this is sounding really interesting.

If you have not had a chance to see what they are doing, I encourage you to check out their wiki page and check Langdon's presentation at flock as soon as it is available on youtube.

Hubs

Fedora-hubs is a cool project aiming at simplifying the steps new contributors need to take to reach the old contributors. IRC, mailing lists, tickets are all places where activities happen but that might be obscure to new contributors. fedora-hubs tries to fill this gap by aggregating all the activity around a group of people and provide it to new contributors so they know where to look to get aboard.

We ran a workshop with a nice demo at flock. We received some good positive feedbacks and people seem to like how things are looking. Personally flock has also been the occasion to pass the torch on this project. I have been leading it since Ralph changed team but I am really not the expert in the technologies needed for hubs' frontend. So I passed on the torch to Sayan who is much more experienced than I am and who I'm sure will do a great job leading hubs. I will still be around, I am very much interested in helping with backend bits and pieces. FMN still needs some work and a few other applications that I maintain might require adjustments to integrate with hubs the way we want it. So, do expect me around :)

Zanata

Finally, flock has also been the occasion for me to meet up with the folks from zanata (the platform used by fedora's translators). We exchanged a few emails before the conference as we asked them to expand on their web-hooks so we could gather some more stats and include them on fedora-hubs. It was really nice to be able to discuss with them regarding their plans and ours and how we may be able to help each other.

Final words

Well, this has been quite a lengthy blog post, if you made it so far : congratulations!

As a final note, I would like to thank all the organizers of the conference, having tried to place a bid for this year, I have a small idea of the amount of work involved but they managed wonderfully and it was an excellent flock!

New FMN architecture and tests

Introduction

FMN is the FedMsg Notification service. It allows any contributors (or actually, anyone with a FAS account) to tune what notification they want to receive and how.

For example it allows saying things like:

• Send me a notification on IRC for every package I maintain that has successfully built on koji

• Send me a notification by email for every request made in pkgdb to a package I maintain

• Send me a notification by IRC when a new version of a package I maintain is found

How it works

The principile is that anyone can log in on the web UI of FMN there, they can create filters on a specific backend (email or IRC mainly) and add rules to that filter. These rules must either be validated or invalited for the notification to be sent.

Then the FMN backend listens to all the messages sent on Fedora's fedmsg and for each message received, goes through all the rules in all the filters to figure out who wants to be notified about this action and how.

The challenge

Today, computing who wants to be notified and how takes about 6 seconds to 12 seconds per message and is really CPU intensive. This means that when we have an operation sending a few thousands messages on the bus (for example, mass-branching or a packager maintaining a lot of packages orphaning them), the queue of messages goes up and it can take hours to days for a notification to be delivered which could be problematic in some cases.

The architecture

This is the current architecture of FMN:

|                        +--------\
|                   read |  prefs | write
|                  +---->|  DB    |<--------+
|                  |     \--------+         |
|        +-----+---+---+            +---+---+---+---+   +----+
|        |     |fmn.lib|            |   |fmn.lib|   |   |user|
v        |     +-------+            |   +-------+   |   +--+-+
fedmsg+->|consumer     |            |central webapp |<-----+
+        +-----+  +---+|            +---------------+
|        |email|  |irc||
|        +-+---+--+-+-++
|          |        |
|          |        |
v          v        v

As you can see it is not clear where the CPU intensive part is and that's because it is in fact integrated in the fedmsg consumer. This design, while making things easier brings the downside of making it pratically impossible to scale it easily when we have an event producing lots of messages. We multi-threaded the application as much as we could, but we were quickly reaching the limit of the GIL.

To try improving on this situation, we reworked the architecture of the backend as follow:

                                                     +-------------+
                                              Read   |             |   Write
                                              +------+  prefs DB   +<------+
                                              |      |             |       |
   +                                          |      +-------------+       |
   |                                          |                            |   +------------------+   +--------+
   |                                          |                            |   |    |fmn.lib|     |   |        |
   |                                          v                            |   |    +-------+     |<--+  User  |
   |                                    +----------+                       +---+                  |   |        |
   |                                    |   fmn.lib|                           |  Central WebApp  |   +--------+
   |                                    |          |                           +------------------+
   |                             +----->|  Worker  +--------+
   |                             |      |          |        |
fedmsg                           |      +----------+        |
   |                             |                          |
   |                             |      +----------+        |
   |   +------------------+      |      |   fmn.lib|        |       +--------------------+
   |   | fedmsg consumer  |      |      |          |        |       | Backend            |
   +-->|                  +------------>|  Worker  +--------------->|                    |
   |   |                  |      |      |          |        |       +-----+   +---+  +---+
   |   +------------------+      |      +----------+        |       |email|   |IRC|  |SSE|
   |                             |                          |       +--+--+---+-+-+--+-+-+
   |                             |      +----------+        |          |        |      |
   |                             |      |   fmn.lib|        |          |        |      |
   |                             |      |          |        |          |        |      |
   |                             +----->|  Worker  +--------+          |        |      |
   |                         RabbitMQ   |          |    RabbitMQ       |        |      |
   |                                    +----------+                   |        |      |
   |                                                                   v        v      v
   |
   |
   |
   v

The idea is that the fedmsg consumer listens to Fedora's fedmsg, put the messages in a queue. These messages are then picked from the queue by multiple workers who will do the CPU intensive task and put their results in another queue. The results are then picked from this second queue by a backend process that will do the actually notification (sending the email, the IRC message).

We also included an SSE component to the backend, which is something we want to do for fedora-hubs but this still needs to be written.

Testing the new architecture

The new architecture looks fine on paper, but one would wonder how it performs in real-life and with real data.

In order to test it, we wrote two scripts (one for the current architecture and one for the new) sending messages via fedmsg or putting in messages in the queue that the workers listens to, therefore mimiking there the behavior of the fedmsg consumer. Then we ran different tests.

The machine

The machine on which the tests were run is:

• CPU: Intel i5 760 @ 2.8GHz (quad-core)
• RAM: 16G DDR2 (1333 Mhz)
• Disk: ScanDisk SDSSDA12 (120G)
• OS: RHEL 7.2, up to date
• Dataset: 15,000 (15K) messages

The results

The current architecture

The current architecture only allows to run one test, send 15K fedmsg messages and let the fedmsg consumer process them and monitor how long it takes to digest them.

Test #0 - fedmsg based
  Lasted for 9:05:23.313368
  Maxed at:  14995
  Avg processing: 0.458672376874 msg/s

The new architecture

The new architecture being able to scale we performed a different tests with it, using 2 workers, then 4 workers, then 6 workers and finally 8 workers. This gives us an idea if the scaling is linear or not and how much improvement we get by adding more workers.

Test #1 - 2 workers - 1 backend
  Lasted for 4:32:48.870010
  Maxed at:  13470
  Avg processing: 0.824487297215 msg/s
Test #2 - 4 workers - 1 backend
  Lasted for 3:18:10.030542
  Maxed at:  13447
  Avg processing: 1.1342276217 msg/s
Test #3 - 6 workers - 1 backend
  Lasted for 3:06:02.881912
  Maxed at:  13392
  Avg processing: 1.20500359971 msg/s
Test #4 - 8 workers - 1 backend
  Lasted for 3:14:11.669631
  Maxed at:  13351
  Avg processing: 1.15160928467 msg/s

Conclusions

Looking at the results of the tests, the new architecture is clearly handling its load better and faster. However, the progress aren't as linear as we like. My feeling is that retrieve information from the cache (here redis) is at one point getting slower, eventually also because of the central lock we tell redis to use.

As time permits, I will try to investigate this further to see if we can still gain some speed.

Accessing pagure's performances via mod_wsgi-express

Every once in a while we make changes to dist-git in the Fedora infrastructure. This means, we need to test our changes to make sure they do not break (ideally, at all).

These days, we are working on adding namespacing to our git repos so that we can support delivering something else than rpms (the first use-case being, docker). So with the current set-up we have, we added namespacing to pkgdb which remains our main endpoint to manage who has access to which git repo (pkgdb being in a way a glorified interface to manage our gitolite). The next step there is to teach gitolite about this namespacing.

The idea is to move from:

 /srv/git/repositories/<pkg1>.git
 /srv/git/repositories/<pkg2>.git
 /srv/git/repositories/<pkg3>.git
 /srv/git/repositories/<pkg4>.git

To something like:

 /srv/git/repositories/rpms/<pkg1>.git
 /srv/git/repositories/rpms/<pkg2>.git
 /srv/git/repositories/rpms/<pkg3>.git
 /srv/git/repositories/rpms/<pkg4>.git
 /srv/git/repositories/docker/<pkg2>.git
 /srv/git/repositories/docker/<pkg5>.git

But, in order to keep things working with the current clone out there, we'll symlink the rpms namespace to one level higher in the hierarchy which should basically keep things running as they are currently.

So the question at hand is, now that we have adjusted our staging pkgdb and dist-git, how do we test that fedpkg still works.

This is a recipe from bochecha to make it easy to test fedpkg in staging while not breaking it for regular use.

It goes in three steps:

1. Edit the file /etc/rpkg/fedpkg.conf and add to it:

[fedpkgstg]
lookaside = http://pkgs.stg.fedoraproject.org/repo/pkgs
lookasidehash = md5
lookaside_cgi = https://pkgs.stg.fedoraproject.org/repo/pkgs/upload.cgi
gitbaseurl = ssh://%(user)s@pkgs.stg.fedoraproject.org/%(module)s
anongiturl = git://pkgs.stg.fedoraproject.org/%(module)s
tracbaseurl = https://%(user)s:%(password)s@fedorahosted.org/rel-eng/login/xmlrpc
branchre = f\d$|f\d\d$|el\d$|olpc\d$|master$
kojiconfig = /etc/koji.conf
build_client = koji

2. Create a fedpkgstg (the name of the cli must be the same as the title of the section entered in the config file above)

sudo ln -s /usr/bin/fedpkg /usr/bin/fedpkgstg

3. call fedpkgstg to test staging and fedpkg to do your regular operation against the production instances

Thanks bochecha!