Le blog de pingou

To content | To menu | To search

Tag - pkgdb

Entries feed - Comments feed

Friday, December 11 2015

Testing distgit in staging with fedpkgstg

Every once in a while we make changes to dist-git in the Fedora infrastructure. This means, we need to test our changes to make sure they do not break (ideally, at all).

These days, we are working on adding namespacing to our git repos so that we can support delivering something else than rpms (the first use-case being, docker). So with the current set-up we have, we added namespacing to pkgdb which remains our main endpoint to manage who has access to which git repo (pkgdb being in a way a glorified interface to manage our gitolite). The next step there is to teach gitolite about this namespacing.

The idea is to move from:

 /srv/git/repositories/<pkg1>.git
 /srv/git/repositories/<pkg2>.git
 /srv/git/repositories/<pkg3>.git
 /srv/git/repositories/<pkg4>.git

To something like:

 /srv/git/repositories/rpms/<pkg1>.git
 /srv/git/repositories/rpms/<pkg2>.git
 /srv/git/repositories/rpms/<pkg3>.git
 /srv/git/repositories/rpms/<pkg4>.git
 /srv/git/repositories/docker/<pkg2>.git
 /srv/git/repositories/docker/<pkg5>.git

But, in order to keep things working with the current clone out there, we'll symlink the rpms namespace to one level higher in the hierarchy which should basically keep things running as they are currently.

So the question at hand is, now that we have adjusted our staging pkgdb and dist-git, how do we test that fedpkg still works.

This is a recipe from bochecha to make it easy to test fedpkg in staging while not breaking it for regular use.

It goes in three steps:

1. Edit the file /etc/rpkg/fedpkg.conf and add to it:

[fedpkgstg]
lookaside = http://pkgs.stg.fedoraproject.org/repo/pkgs
lookasidehash = md5
lookaside_cgi = https://pkgs.stg.fedoraproject.org/repo/pkgs/upload.cgi
gitbaseurl = ssh://%(user)s@pkgs.stg.fedoraproject.org/%(module)s
anongiturl = git://pkgs.stg.fedoraproject.org/%(module)s
tracbaseurl = https://%(user)s:%(password)s@fedorahosted.org/rel-eng/login/xmlrpc
branchre = f\d$|f\d\d$|el\d$|olpc\d$|master$
kojiconfig = /etc/koji.conf
build_client = koji

2. Create a fedpkgstg (the name of the cli must be the same as the title of the section entered in the config file above)

sudo ln -s /usr/bin/fedpkg /usr/bin/fedpkgstg

3. call fedpkgstg to test staging and fedpkg to do your regular operation against the production instances



Thanks bochecha!

Friday, June 26 2015

Packagers AFK in pkgdb

I just wanted to point out a small feature added to pkgdb recently.

Basically, it integrates with the vacation calendar of fedocal to show on the packager's info page if the person is on vacations or not.

If you are dealing with someone who is slow to answer on bugs, irc or emails, it may give you an insight as to why that is.

pkgdb_vacations2.png



Note: I am in no way saying that Paul is slow to answer bugs, irc or email, and have merely used him to illustrate my thoughts following up on his post about the Red Hat summit and I shall not be held responsible for any variations in Paul's response time :-)

Wednesday, June 17 2015

Contribute to pkgdb2

How to get started with contributing to pkgdb2.

Continue reading...

Thursday, May 7 2015

Check packages in anitya and pkgdb2 for monitoring

A little while ago I presented a script allowing to search for the packages of a specified user and see which are missing from either anitya or are not being monitored in pkgdb2.

This script however, only check someone's packages and someone time we want to check a number of packages at once, eventually, all the packages matching a given template.

This new script does just that:

 $ python pkgs_not_in_anitya_2.py 'drupal-*'
   drupal-service_links                 Monitor=False   Anitya=False
   drupal-calendar                      Monitor=False   Anitya=False
   drupal-cck                           Monitor=False   Anitya=False
   drupal-date                          Monitor=False   Anitya=False
   drupal-workspace                     Monitor=False   Anitya=False
   drupal-views                         Monitor=False   Anitya=False

If you are interested, feel free to use the script

Friday, April 3 2015

OpenSearch integration in pkgdb

One of the earliest feature request of pkgdb2 (that was present in pkgdb1) is the browser search integration.

This integration is based on the OpenSearch specifications and basically allows to use pkgdb as one of the search engine of your web browser just like you can use google, duckduckgo or wikipedia.

I recently found out this feature is not so well known, so I thought I would present it and explain how to set it up (screenshot are on Firefox).

1/ Go to https://admin.fedoraproject.org/pkgdb and click on the list of search engines at the top right.

2/ Select the entry Add "Fedora PkgDB2: Packages"

That's it you are done for the most important step :)

pkgdb_search_3.1.png

Now something which I do and find most useful is:

3/ Go to Manage Search Engines...

There, with the search engine pkgdb packages associate the keyword pkgdb

pkgdb_search_5.png

Now, you can use your url bar as usual but when you enter pkgdb <something> it will search this <something> in pkgdb directly. So for example, if you want to search for guake in pkgdb, you would type in your url bar pkgdb guake.

pkgdb_search_6.png

The bonus point is that since there is only one package with this name, you will be immediately redirected to its page.

This way, when you want to quickly find information about a package in pkgdb, you can get it from your browser in one simple step (eventually two if several package match the keyword you entered).

Final bonus point? To access pkgdb directly, enter in the url bar: "pkgdb " (with a space at the end), without a keyword, Firefox will bring you directly to the front page of the application.

Tuesday, March 24 2015

New package & new branch process

A little while ago, I blogged about the new package and new branch request processes.

These changes have been pushed to production yesterday.

What does this change for you, packager?

New package

If you already a packager, you know the current process to get packages into Fedora, you know that once your package has been approved on bugzilla, you have to file a SCM request.

With the new process, this step is no longer necessary. You can directly go to pkgdb and file the request there.

From there admins will review the package review on bugzilla and create the package in pkgdb (or refuse with an explanation).

New branch

If your package is already in Fedora, you can now directly request a new branch in pkgdb. Here there are multiple options

  • You have approveacls on the package (thus you are a package admin) and the request is regarding a new Fedora branch: The branch will be created automatically
  • You have approveacls on the package (thus you are a package admin) and the request is regarding a new EPEL branch: The request will be submitted to the pkgdb admins who will process it in their next run
  • You do not have approveacls on the package, then your request will be marked as: `Pending`, this means that the admins of the package have one week to react. They can either approve your request and by setting it to Awaiting Review, or they can decline the request (for which they must specify a reason). After this one week (or sooner if the package admin set the request to Awaiting Review) the pkgdb admin will process the request like they do with the other.

Note: Even with this new workflow, requests are still manually reviewed, so the requests will not necessarily be processed faster (but if it is easier for the admins, they may run it more often!).

What does this change for you, admins?

Hopefully, the process will be much simpler for you. In short

  • no need to log onto any system, you can do everything from your own machine and it should work out of the box
  • much more automated testing (including checking if a package is present in RHEL and on which arch for EPEL requests)
  • one tool to process the requests: pkgdb-admin distributed as part of packagedb-cli (aka: pkgdb-cli)



I hope this process makes sense to you and will make your life easier.

You are welcome to already use these processes, just let us know if you run into some problems, but for the time being both the old and the new processes are supported :-)

Thursday, January 22 2015

New branch request process

A little while ago I blogged about a new process to request a new branch on an existing package.

The code to support this change is now under review but I thought I should document the workflow a little bit, so here is how I tried to design things:

pkgdb_new_branch_flow_2.png

Ideally, when the branch is approved and created in pkgdb by the admin, pkgdb will send a message on fedmsg, message that will be seen by a fedmsg-consummer that will automatically update the git repos within, say 2 minutes. That last part if almost ready and hopefully will be running soon.

Wednesday, October 15 2014

Fedora-Infra: Did you know? The package information are now updated weekly in pkgdb2!

The package database pkgdb2 is the place where is managed the permission on the git repositories.

In simple words, it is the place managing the "who is allowed to do what on which package".

For each package, when they are created, the summary, the description and the upstream URL from the spec file are added to the database, which allow us to display the information on the page concerning the package. However, until two weeks ago, this information was never updated. That means that if you had an old package whose description had changed over time, pkgdb would present the one from the time the package was created in the database.

Nowadays, we have a script running on a weekly basis and updating the database. Currently, this script relies on the information provided by yum's metadata on the rawhide repo. This means that packages that are only present in EPEL or that are retired on rawhide but present in F21, will not have their information updated. This is likely something we will fix in the future though.

In the mean-time, you can now enjoy a pkgdb with summary and description information for almost all packages!

As an example, checkout the fedocal page, you can now see a link to the upstream website, a short summary and a little longer description of the project.

Also, to give you a little hint on the amount of updates we did:

The first time we ran the script:

 16638 packages checked
 15723 packages updated

Last week's run:

 16690 packages checked
 50 packages updated

Wednesday, July 23 2014

New package, new branch, new workflow?

If you are a Fedora packager, you are probably aware of the new pkgdb.

One question which has been raised by this new version is: should we change the process to request new branches or integrate new packages in the distribution.

The discussion has occurred on the rel-eng mailing list but I'm gonna try to summarize here what the process is today and what it might become in the coming weeks.

Current new-package procedure:
  1. packager opens a review-request on bugzilla
  2. reviewer sets the fedora-review flag to ?
  3. reviewer does the review
  4. reviewer sets the fedora-review flag to +
  5. packager creates the scm-request and set fedora-cvs flag to ?
  6. cvsadmin checks the review (check reviewer is a packager)
  7. cvsadmin processes the scm-request (create git repo, create package in pkgdb)
  8. cvsadmin sets fedora-cvs flag to +
New procedure
  1. packager opens a review-request on bugzilla
  2. reviewer sets the fedora-review flag to ?
  3. reviewer does the review
  4. reviewer sets the fedora-review flag to +
  5. packager goes to pkgdb2 to request new package (specifying: package name, package summary, package branches, bugzilla ticket)
  6. requests added to the scm admin queue
  7. cvsadmin checks the review (check reviewer is a packager¹)
  8. cvsadmin approves the creation of the package in pkgdb
  9. package creation is broadcasted on fedmsg
  10. fedora-cvs flag set to + on bugzilla
  11. git adjusted automatically

Keeping the fedora-cvs flag in bugzilla allows to perform a regular (daily?) check that there are no fedora-review flag set as + that have been approved in pkgdb and whose fedmsg message hasn't been processed.

Looking at the number, it looks like there are more steps on the new procedure but eventually, most of them can be automated.

New branch process

For new branches, the process would be very similar:

  1. packager goes to pkgdb2 to request new branch
  2. requests added to the scm admin queue
  3. cvsadmin checks the request (requester is a packager...)
  4. cvsadmin approves the creation of the branch in pkgdb
  5. branch creation is broadcasted on fedmsg
  6. git adjusted automatically

Monday, February 3 2014

Dynamic point of contact assignment (2)

Recently I spoke about dynamic point of contact assignment in pkgdb2, I had generated some global stats about general changes but nothing specific, so here is a little more information about the impact this would have for packagers:

 652 packagers lose at least one package
 In average they lose 6.82668711656 packages
 392 packagers gain at least one package
 In average they gain 11.7168367347 packages
 
 Top 10 packagers losing packages:
    iarnell loses 305 packages : 385 -> 80
    spot loses 185 packages : 348 -> 163
    jplesnik loses 173 packages : 214 -> 41
    than loses 167 packages : 185 -> 18
    steve loses 129 packages : 137 -> 8
    eseyman loses 97 packages : 190 -> 93
    psabata loses 76 packages : 150 -> 74
    jwrdegoede loses 70 packages : 173 -> 103
    corsepiu loses 53 packages : 112 -> 59
    mathstuf loses 51 packages : 63 -> 12
 
 Top 10 packagers gaining packages:
    ppisar gains 1353 packages : 295 -> 1648
    rdieter gains 241 packages : 135 -> 376
    kalev gains 127 packages : 43 -> 170
    limb gains 123 packages : 234 -> 357
    pghmcfc gains 114 packages : 176 -> 290
    pbrobinson gains 106 packages : 58 -> 164
    remi gains 101 packages : 212 -> 313
    mizdebsk gains 97 packages : 54 -> 151
    spot gains 93 packages : 348 -> 441
    petersen gains 75 packages : 141 -> 216

Graphically this is how it looks:

packages_lost_per_packagers.png

packages_won_per_packagers2.png

What about relative lost/gained, so instead of the number of packages let's look at the % of packages lost/gained relative to the number of packages that this user is the current POC:

 Top 10 packagers losing packages (in %):
    <... 142 packagers...> lose 100.0% packages
    steve loses 94% packages
    than loses 90.2% packages
    silfreed loses 90% packages
    packaging-team loses 89% packages
    yyang, sxw lose 87.5% packages
    amdunn, rocha, kylev lose 85% packages
    pvrabec, huwang, jdornak, hvad lose 83% packages
    timn loses 82% packages
    mathstuf loses 81% packages
 
 Top 10 packagers gaining packages (in %):
    paragn gains 1325.0% packages
    atkac gains 1100.0% packages
    crobinso gains 1000.0% packages
    hegdevasant gains 700.0% packages
    aalvarez gains 550.0% packages
    ppisar gains 459% packages
    airlied gains 433% packages
    pmachata gains 417% packages
    jkastner, jpeeler gain 400% packages
    adamwill gains 350% packages

Below is a graphical distribution of the packagers by the percentage of packages they lose:

packages_lost_per_packagers_pc.png

So from this we can conclude:

  • We have two packagers standing out with regards to the number of packages they would gain with a dynamic POC assignment
  • We have one packager standing out as it appears he/she has a lot of packages but is not working on them so much anymore
  • Some packager are both losing and gaining packages
  • We need to filter out the groups maintaining packages, but with the move to pkgdb2 it will be easier to filter them out
  • Quite a number of person are losing all their packages with dynamic POC assignment. There are also two peaks around 50% and 33%.
  • Amusingly, ppisar who gains the most packages is not the one gaining the most packages relatively to the number of packages he already owns


As before I put the script I wrote to gather these stats on my cgit

Thursday, January 23 2014

Dynamic point of contact assignment

Recently, while working on pkgdb2 I had a RFE for "dynamic" ownership.

The idea was to automatically change the owner based on who is actually working on the package.

With the change from "owner" to "point of contact" of a package, I thought that this might be an interesting idea. Of course in order to assess the feasibility and to investigate if it is really a good idea, we need some stats.

So I wrote a script that retrieves all the packages present in rawhide in Fedora. For each package it takes the last 100 actions (git commits, koji build and bodhi updates) and order the contributors from the most the least active. The script then checks the most active user versus the owner/point of contact of the package.

There is the output after running for 6h35:

14546 packages retrieved
14546 packages checked
85 packages w/ no package information
2877 packages w/ ausil as active point of contact
7132 packages won't change their point of contact
4451 packages will change of point of contact

I had to put appart ausil as he is the one doing the Mass-Rebuild and as such would become the point of contact of too many packages that have no other activity than Mass-Rebuild.

I still have the matrix of data available to extract more information about the distribution of the packages among the packager but I thought I would share this first.



What do you think?

Friday, August 5 2011

PackageDB-cli 1.1.0

rpm.pngsource.png

Version 1.1.0 du client text pour pkgdb.

Release 1.1.0 of the command-line interface for pkgdb.

Continue reading...

Wednesday, June 29 2011

PackageDB-cli 1.0.0

rpm.pngsource.png

Version 1.0.0 du client text pour pkgdb.

Release 1.0.0 of the command-line interface for pkgdb.

Continue reading...

Wednesday, June 8 2011

PackageDB-cli

source.png

Un client text pour pkgdb.

A command-line interface for pkgdb.

Continue reading...

Wednesday, May 11 2011

Get pkgdb info (2)

source.png

Amélioration du script pour pkgdb.

Improvement of the script to query pkgdb.

Continue reading...

Tuesday, May 10 2011

Get pkgdb info

source.png

Un petit script pour interroger pkgdb

A small script to query pkgdb.

Continue reading...