recovering from lost partition table: a use case

This happened today and may be of use to other Ubuntu or Debian users with a similar configuration. The disk of a laptop lost its partition table because it was mistaken for a USB key and modified. There was no way to recover the old GPT partition table from the backup because it was not corrupted. The new partition table was both in the GPT partition table and its backup.

The laptop was installed with Ubuntu 16.04 about two years ago and it was assumed to:

  • Have an EFI partition
  • Have a separate boot partition

The partition table is rebuilt to start with a 512MB partition for EFI (type 1) followed by another parition for the rest of the disk. When the second partition table is created using fdisk, it will issue a warning about an ext2 signature: it must be preserved.

Using dumpe2fs on the second partition the actual size of the partition is displayed, for instance:

$ dumpe2fs /dev/nvme0n1p2
...
Block count:              249856
...
Block size:               1024

The partition table is adjusted accordingly, taking into account that filesystem blocks are 1024 bytes and fdisk sectors are 512 bytes (i.e. 249856 * 2 == 499712):

$ fdisk -l /dev/nvme0n1
Device           Start        End   Sectors   Size Type
/dev/nvme0n1p1    2048    1050623   1048576   512M EFI System
/dev/nvme0n1p2 1050624    1550335    499712   244M Linux filesystem

Note: for some reason fdisk adds 1 to the specified number of sectors in some cases. Care must be taken to verify the actual number of sectors allocated to the partition and adjust them accordingly.

The signature of the third partition gives a hint regarding its content. If it is crypto_LUKS, it can be open with cryptsetup luksOpen /dev/nvme0n1p3 root and the device will show under /dev/mapper.

Why I deleted my GitHub account

Today I permanently deleted the GitHub account that was in my name. I used it to contribute to Free Software projects that only use this proprietary platform. I started using GitHub a few years ago and constantly tried (and failed) to convince projects to self-host using Free Software instead. As more and more Free Software projects I care about moved to GitHub, I ended up using it daily and it made me sad.

The recent acquisition of GitHub by Microsoft turned it into a service that belongs to a company that is actively working against the Free Software movement by:

  • Lobbying for Software Patents and being one of the largest Software Patents holder in the world
  • Lobbying for DRMs and implementing them in their software products
  • Convincing people they must not fight in court when their rights under copyleft licenses are violated

By using Microsoft services daily, I would also implicitly support them, even if indirectly. I feel better knowing I no longer depend on Microsoft in any way.

HOWTO nginx & letsencrypt on Debian GNU/Linux stretch/9

The goal is to configure a nginx server with automatic Let’s Encrypt renewal, assuming a new dedicated virtual machine running a pristine Debian GNU/Linux stretch/9.

Install docker-compose:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates dirmngr
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo deb https://apt.dockerproject.org/repo debian-stretch main | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
sudo apt-get install -y docker-engine
sudo bash -c 'curl -L https://github.com/docker/compose/releases/download/1.13.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose'
sudo chmod +x /usr/local/bin/docker-compose

Assuming the FQDN of the machine is download.securedrop.club and the person responsible can be reached at admin@securedrop.cub, create the docker-compose.yml with:

cat > docker-compose.yml <<EOF
version: '2'
services:
  web:
    image: nginx:1.13.3
    volumes:
      - ./html:/usr/share/nginx/html:ro
    ports:
      - "8080:80"
    environment:
      - VIRTUAL_HOST=download.securedrop.club
      - LETSENCRYPT_HOST=download.securedrop.club
      - LETSENCRYPT_EMAIL=admin@securedrop.club
  proxy:
    image: jwilder/nginx-proxy
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./certs:/etc/nginx/certs:ro
      - /etc/nginx/vhost.d
      - /usr/share/nginx/html
    ports:
      - "80:80"
      - "443:443"
    restart: always
    depends_on:
      - web
  letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./certs:/etc/nginx/certs:rw
    volumes_from:
      - proxy
EOF

and run docker-compose up in the same directory as the docker-compose.yml file.

logging udev events at boot time

Adapted from Peter Rajnoha post:

  • create a special systemd unit to monitor udev during boot:
    cat > /etc/systemd/system/systemd-udev-monitor.service <<EOF
    [Unit]
    Description=udev Monitoring
    DefaultDependencies=no
    Wants=systemd-udevd.service
    After=systemd-udevd-control.socket systemd-udevd-kernel.socket
    Before=sysinit.target systemd-udev-trigger.service
    
    [Service]
    Type=simple
    ExecStart=/usr/bin/sh -c "/usr/sbin/udevadm monitor --udev --env > /udev_monitor.log"
    
    [Install]
    WantedBy=sysinit.target
    EOF
    
  • run systemctl daemon-reload
  • run systemctl enable systemd-udev-monitor.service
  • reboot
  • append “systemd.log_level=debug systemd.log_target=kmsg udev.log-priority=debug log_buf_len=8M” to kernel command line
  • collect the logs in /udev_monitor.log

Continue reading “logging udev events at boot time”

Semi-reliable GitHub scripting

The githubpy python library provides a thin layer on top of the GitHub V3 API, which is convenient because the official GitHub documentation can be used. The undocumented behavior of GitHub is outside of the scope of this library and needs to be addressed by the caller.

For instance creating a repository is asynchronous and checking for its existence may fail. Something similar to the following function should be used to wait until it exists:

    def project_exists(self, name):
        retry = 10
        while retry > 0:
            try:
                for repo in self.github.g.user('repos').get():
                    if repo['name'] == name:
                        return True
                return False
            except github.ApiError:
                time.sleep(5)
            retry -= 1
        raise Exception('error getting the list of repos')

    def add_project(self):
        r = self.github.g.user('repos').post(
            name=GITHUB['repo'],
            auto_init=True)
        assert r['full_name'] == GITHUB['username'] + '/' + GITHUB['repo']
        while not self.project_exists(GITHUB['repo']):
            pass

Another example is merging a pull request. It sometimes fails (503, cannot be merged error) although it succeeds in the background. To cope with that, the state of the pull request should be checked immediately after the merge failed. It can either be merged or closed (although the GitHub web interface shows it as merged). The following function can be used to cope with that behavior:

    def merge(self, pr, message):
        retry = 10
        while retry > 0:
            try:
                current = self.github.repos().pulls(pr).get()
                if current['state'] in ('merged', 'closed'):
                    return
                logging.info('state = ' + current['state'])
                self.github.repos().pulls(pr).merge().put(
                    commit_message=message)
            except github.ApiError as e:
                logging.error(str(e.response))
                logging.exception('merging ' + str(pr) + ' ' + message)
                time.sleep(5)
            retry -= 1
        assert retry > 0

These two examples have been implemented as part of the ceph-workbench integration tests. The behavior described above can be reproduced by running the test in a loop during a few hours.

write-only ssh based rsync server

A write-only rsync server can be used by anyone to upload content with no risk of deleting existing files. Assuming access to the rsync server is handled via ssh, the following line can be added to the ~/.ssh/authorized_keys file

command="rrsync /usr/share/nginx/html" ssh-rsa AAAAB3NzaC1y...

The rrsync script is found in the rsync package documentation and installed with:

gzip -d < /usr/share/doc/rsync/scripts/rrsync.gz > /usr/bin/rrsync
chmod +x /usr/bin/rrsync

DNS spoofing with RPZ and bind9

When two web services reside on the same LAN, it may be convenient to spoof DNS entries to use the LAN IP instead of the public IP. It can be done using RPZ and bind9.
For instance workbench.dachary.org can be mapped to 10.0.2.21 with

$ cat /etc/bind/rpz.db
$TTL 60
@            IN    SOA  localhost. root.localhost.  (
                          2   ; serial
                          3H  ; refresh
                          1H  ; retry
                          1W  ; expiry
                          1H) ; minimum
                  IN    NS    localhost.

workbench.dachary.org        A    10.0.2.21

The zone is declared in

$ cat /etc/bind/named.conf.local
zone "rpz" {
      type master;
      file "/etc/bind/rpz.db";
      allow-query {none;};
};

and the response-policy is set in the options file with

$ cat /etc/bind/named.conf.options
...
	response-policy { zone "rpz"; };
};

When bind9 is restarted with /etc/init.d/bind9 restart, the mapping can be verified with

$ dig @127.0.0.1 workbench.dachary.org
workbench.dachary.org.	5	IN	A	10.0.2.21

If the bind9 server runs on a docker host, it can be used by docker containers with

docker run  ... --dns=172.17.42.1 ...

Using a cloud image with kvm

It would be convenient to have a virt-builder oneliner such as

$ virt-builder --arch i386 --ssh-inject ~/.ssh/id_rsa.pub fedora-21

to get an image suitable to run and login with

$ qemu-kvm -m 1024 -net user,hostfwd=tcp::2222-:22 \
  -drive file=fedora-21.qcow2 &
$ ssh -p 2222 localhost grep PRETTY /etc/os-release
PRETTY_NAME="Fedora 21 (Twenty One)"

Docker users have a simpler form because there is no need to ssh to enter the container:

$ docker run fedora:21 grep PRETTY /etc/os-release
PRETTY_NAME="Fedora 21 (Twenty One)"

It is not currently possible to use virt-builder as described above because

  • the set of images available by default is limited (no i386 architecture for instance)
  • the –inject-ssh option is only available in the development version

The libguestfs.org toolbox can however be used to implement a script modifying images prepared for the cloud (see ubuntu cloud images for instance):

  • wget the image
    wget -O my.img http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-i386-disk1.img
    
  • create a config-drive for cloud-init to feed it the ssh public key.
    mkdir -p config-drive
    cat > config-drive/user-data <<EOF
    #cloud-config
    ssh_authorized_keys:
     - $(cat ~/.ssh/id_rsa.pub)
    chpasswd: { expire: False }
    EOF
    cat > config-drive/meta-data <<EOF
    instance-id: iid-123459
    local-hostname: testhost
    EOF
    ( cd config-drive ; LIBGUESTFS_BACKEND=direct virt-make-fs \
      --type=msdos --label=cidata .  ../config-drive.img )
    
  • launch the image with the config drive attached and it will be auto detected
    qemu-kvm -m 1024 -net user,hostfwd=tcp::2222-:22 \
      -drive file=my.img -drive config-drive.img
    

Continue reading “Using a cloud image with kvm”

Upgrade nodejs on Ubuntu 14.04

To run gh a version of nodejs more recent than the one packaged by default on Ubuntu 14.04 is required:

$ apt-cache policy nodejs
nodejs:
  Installed: 0.10.25~dfsg2-2ubuntu1
  Candidate: 0.10.25~dfsg2-2ubuntu1
  Version table:
 *** 0.10.25~dfsg2-2ubuntu1 0
        500 http://fr.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
        100 /var/lib/dpkg/status
$ gh watch
fatal: Please update your NodeJS version: http://nodejs.org/download

The recommended way to upgrade is currently broken and the following can be used instead:

sudo add-apt-repository 'deb https://deb.nodesource.com/node trusty main'
sudo apt-get update
sudo apt-get install nodejs

If either apt-get update or apt-get install fail with a message like SSL: certificate subject name:

...
Err https://deb.nodesource.com trusty/main amd64 Packages
  SSL: certificate subject name (login.meteornetworks.com) does not match target host name 'deb.nodesource.com'
Ign http://ceph.com trusty/main Translation-en
Err https://deb.nodesource.com trusty/main i386 Packages
  SSL: certificate subject name (login.meteornetworks.com) does not match target host name 'deb.nodesource.com'
Ign https://deb.nodesource.com trusty/main Translation-en_US
Ign https://deb.nodesource.com trusty/main Translation-en
Ign http://get.docker.io docker/main Translation-en_US
Ign http://get.docker.io docker/main Translation-en
W: Failed to fetch https://deb.nodesource.com/node/dists/trusty/main/binary-amd64/Packages  SSL: certificate subject name (login.meteornetworks.com) does not match target host name 'deb.nodesource.com'
W: Failed to fetch https://deb.nodesource.com/node/dists/trusty/main/binary-i386/Packages  SSL: certificate subject name (login.meteornetworks.com) does not match target host name 'deb.nodesource.com'
E: Some index files failed to download. They have been ignored, or old ones used instead.

The following will fix it:

echo 'Acquire::https::deb.nodesource.com::Verify-Peer "false";' > /etc/apt/apt.conf.d/99verify

Alternatively a version of gh that does not require a recent version of nodejs can be installed with

sudo npm install -g gh@1.9.4

An example of controlled technical debt

When I started working to help with Ceph backports, I was not familiar with the workflow (who does what, when and why) or the conventions (referencing commits from redmine issues, the redmine backport field, …). I felt the need for scripts to help me cross reference information (from git, github and redmine) and consolidate them into an inventory which I could use as a central point to measure progress and find what needed to be done. But I was not able to formulate this in so many words and at the beginning it was little more than a vague feeling that I would quickly be lost if I did not write down my findings. I chose to write a script, with no tests and no structure, to do things like matching a pull request with a redmine issue when the only clue was a Fixes: #XXX embedded in the comment one of the commits.

After a few weeks the script grew into a 500 lines monstrosity, extremely useful and quite impossible to maintain in the long run. My excuse was that I had no clue what I needed to begin with and that I could not have understood the backport workflow without this script. After the first backport release was declared ready, I stopped adding functionalities and re-started from scratch what became the ceph-workbench backport sub command.

This refactor was done without modifying the behavior of the original script (there were only a few occurrences where it was impossible to preserve). The architecture of the script was completely new: the original script was a near linear sequence of operations with only global variables. The quick summary is that the script pulls information from a few sources (one class for redmine, one for gitlab, one for git), cross reference them with ad-hoc methods and display them into rdoc pages to be displayed in the wiki.

Writing unit tests helped proceed incrementally, pulling one code snippet after the other and checking they were not broken by the refactor. Instead of unit testing the top level command, integration tests were written and run via tox, using real gitlab and redmine instances as fixtures running in docker containers. It will help when adding new use cases such as scrapping the ceph-qa mailing list to match teuthology job failures with the corresponding redmine issue or interpreting the Backport: field in commit messages.