Testing CPU features with Qemu

The Ceph erasure code plugin must run on Intel CPU that have no SSE4.2 support. A Qemu is run without SSE4.2 support:

qemu-system-x86_64 -machine accel=kvm:tcg -m 2048 \
  -drive file=server.img -boot c \
  -display sdl \
  -net nic -net user,hostfwd=tcp::2222-:22 \
  -fsdev local,security_model=passthrough,id=fsdev0,path=~/ceph \
  -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=hostshare

The qemu CPU has no SSE4.2 although the native CPU has it:

$ grep sse4.2 /proc/cpuinfo | wc -l
$ ssh -p 2222 loic@ grep sse4.2 /proc/cpuinfo | wc -l

The local development directory is a Plan 9 folder shared over Virtio mounted inside the VM:

sudo mount -t 9p -o trans=virtio,version=9p2000.L hostshare /home/loic/ceph

and the functional test is run to assert that encoding and decoding an object:

$ cd /home/loic/ceph/src
$ ./unittest_erasure_code_jerasure
[----------] Global test environment tear-down
[==========] 16 tests from 8 test cases ran. (30 ms total)
[  PASSED  ] 16 tests.

Vue subjective de la naissance de l'Erasure Code dans Ceph

L’erasure code, c’est aussi le RAID5, qui permet de perdre un disque dur sans perdre ses données. Du point de vue de l’utilisateur, le concept est simple et utile, mais pour la personne qui est chargée de concevoir le logiciel qui fait le travail, c’est un casse-tête. On trouve des boîtiers RAID5 à trois disques dans n’importe quelle boutique : quand l’un d’eux cesse de fonctionner, on le remplace et les fichiers sont toujours là. On pourrait imaginer ça avec six disques dont deux cessent de fonctionner simultanément. Mais non : au lieu d’avoir recours à une opération XOR, assimilable en cinq minutes, il faut des corps de Galois, un bon bagage mathématique et beaucoup de calculs. Pour corser la difficulté, dans un système de stockage distribué tel que Ceph, les disques sont souvent déconnectés temporairement pour cause d’indisponibilité réseau.
Continue reading “Vue subjective de la naissance de l'Erasure Code dans Ceph”

random read disk stress test

When a GNU/Linux machine exhibits a high iowait (i.e. more than 20% of the processor time is locked waiting for IO to complete), it does not always mean a lot of bytes are read or written. It is demonstrated that reading randomly on the disk will create conditions in which less than 5Mb/s are read and the disk will be busy most of the time, which translates into an iowait greater than 20%. A workaround is to give more RAM to the (virtual) machine so that pages read are cached in memory and read only once, therefore reducing the probability of random reads.
Continue reading “random read disk stress test”

SourceForge drifting

Over the past few months the SourceForge development facility, which hosts a large number of Free Software projects, has changed its policies. Features for exporting a project from SourceForge have been removed. The implementation used to be exclusively Free Software but is now based on non-free software. Finally, VA Linux[1] has become rather underhand in their attempts to grasp exclusive control of contributors’ work.

SourceForge did a lot of good for the Free Software community, but it’s now time to break free.

Locking users in a non-free software world

SourceForge brought to Free Software a unified and standard development methodology based on modern tools. Before SourceForge, such tools (bug tracking, cvs, web, support, forums, polls, news, etc.) were available individually, but few developers used many of them together, because they had to set up the combined facilities on their own. SourceForge made the combination conveniently available for both new and experienced developers.

Because of the convenience of SourceForge, many Free Software developers have come to take this collection of features for granted, and would be reluctant to go back to the old way of doing things. Unfortunately, this means that when SourceForge itself takes a turn for the worse, it tends to pull Free Software developers down with it.

The second important thing SourceForge did was to provide this environment based exclusively on Free Software. By doing this, SourceForge not only provided a powerful methodology for the Free Software community, it also demonstrated what Free Software could do, and promoted the use of Free Software. And since the special software for SourceForge was itself free, anyone could set up a similar site. The SourceForge software became permanently available to developers everywhere. Developers in (say) India who can’t afford the bandwidth to use the SourceForge site could have the benefit of the same features on their own server.

In August 2001, VA Linux reversed those policies and introduced non-free software on the SourceForge server. In announcing this (the original document was removed or moved shortly after the publication of this article), Larry Augustin (VA Linux CEO) claims that SourceForge.net users will “see virtually no changes”. That may be true if they narrow their vision and consider only what job the site does and how to operate it. But when we consider the implications, things are very different now. Instead of a showcase for Free Software, SourceForge is now a demo site for non-free software. There is a danger that the many thousands of people registered on SourceForge will become increasingly hooked on the SourceForge site and on features implemented by proprietary software.

As a Free Software developer, you are still free to use the SourceForge server, but you won’t have the freedom to copy, modify, study and distribute the software it runs; you won’t be free to set up a similar site yourself, or adapt it to your own needs. The last published release of the SourceForge software is one year old. The move to non-free software was the culmination of a series of steps designed to lock users in. There never was a way to fully extract projects from SourceForge, but efforts were made in this direction–then this year they were removed. At present the only things you can get are the CVS tree and tracker data /export/sf_tracker_export.php. Few people are aware of the later because it is undocumented. The export page explains how to use scripts that don’t exist anymore; implementation of facilities to ease project extraction was stopped. The developer community is exclusively made of VA Linux employees and a few people who are asked not to disclose the current code.

The mailing lists archives, a major service of SourceForge recently became unmaintained. Will it be replaced by a non-free software based solution ?

Contributors’ work appropriation

Here is what happened to me shortly before the announcement that SourceForge would use and develop non-free software. Because I’m listed as a contributor (in the sources and documentation) to the SourceForge software, I received a request from VA Linux to assign copyright to them. I was not surprised or unhappy with this; many Free Software projects ask contributors to assign copyright of their changes to the main author. Assigning copyright to a single holder is a strategy for defending the GNU GPL more effectively, and I would have been happy to cooperate in that regard.

But when I read the details of their copyright assignment, I saw major problems. I was asked to assign copyright of my work that “is, or may in the future be, utilized in the SourceForge collaborative software development platform”. The assignment was not limited to my contribution to the SourceForge code, it potentially covered all my past and future work if it was of some interest to SourceForge.

I was also expecting a promise that my work would be released under the GNU GPL, but the assignment said nothing about Free Software. VA Linux would be allowed to release the software I wrote under a non-free software license and not let the community have it at all. But I wasn’t sure at the time if this was a real concern, because VA Linux only produced and used Free Software. Two weeks later they decided to introduce non-free software on SourceForge and that cast a different light on the question.

VA Linux told me that they only sent the assignment to two people, in the hope to refine it. We started a long discussion that lasted two months. I assumed this discussion was to make the copyright assignment more palatable to the Free Software community, so I worked hard to give constructive feedback. Finally I was sent the version of the copyright assignment produced by the legal department. I quote it here in its entirety:

                 SourceForge Copyright Assignment

         Thank you for your interest in contributing software code to SourceForge.

            In order for us to include the code in our product, we will need you
            to provide us with the rights to the code.

            By signing this agreement, you, the undersigned, hereby assign to VA
            Linux all right, title and interest in and to the software code
            described below, and all copyright, patent, proprietary information,
            trade secret, and other intellectual property rights therein. You also
            agree to take all actions and sign all documents (such as copyright
            assignments or registrations) reasonably requested by VA Linux to
            evidence and record the above assignments.

This was even more of a power grab than the first draft. “You give us total control; we promise nothing”. At this point, I knew that the attempts to clarify the copyright assignment were a waste of time; VA Linux clearly wasn’t collecting copyright assignments in order to enforce the GNU GPL.

Escape entrapment

It’s time for people who value freedom to escape from SourceForge. It has become a tar pit from which escape will become increasingly difficult. Development hosting platforms based completely on Free Software flourish all over the world. You can create your own, join an existing one or help write the underlying software. Some months ago I helped to launch Savannah for the GNU project because I felt the need of a collaboratively run platform. With friends and co-developpers we are now re-writing and packaging distributed development hosting software. The idea is to be able to install and operate a SourceForge-like site within hours. Savannah will run this software at the end of this year. At first it may have less functionality than SourceForge, but it has a bright future because it is rooted in a cooperative effort of people sharing Free Software.

SourceForge is free as in free beer because it was designed this way. It was a very expensive and ephemeral gift to the Free Software community. We could resent VA Linux for such a poisoned gift. On the contrary I think we should thank them. They brought us methodology, and taught us that a development hosting facility must be built in a distributed and collaborative way, not by a single company controlling everything from top to bottom. Of course that means everyone needs to spend a little time developing and maintaining these hosting facilities. We’ve finished our beer, it’s time to win our freedom.

[1] VA Linux is the owner of the SourceForge domain name, provides and owns the hardware, pays for the bandwidth, hire people maintaining SourceForge. VA Linux is also the owner of most OSDN sites, the largest concentration of Free Software related resources in the hands of a single company.