Onboarding devops with infrastructure as code

Enough is a software released as a pypi package and a docker image to maintain an infrastructure composed of multiple machines, physical or virtual. In other words, it is an infrastructure as code based on Ansible and OpenStack. In addition it is built exclusively on Free Software, primarily because it helps with backdoors and viruses. These two properties combined create a unique opportunity to onboard volunteers devops willing to help nonprofit organizations defending whistleblowers.

Trusting volunteers to manage the infrastructure of a nonprofit is a non trivial chicken and egg problem. Trust is built over time, after working with a volunteer over a period of months and getting to know them. But the work a devop could do, such as maintaining a system to detect intrusions, requires access to the infrastructure of the nonprofit… which requires trust.

This is the part where it helps to have the infrastructure separated from its code. It becomes possible for the volunteer to work on the code without having access to the infrastructure. A recent example in the context of Enough illustrates this: Wazuh started sending noisy alerts about CVE-2019-20367 on a production platform and had to be ignored. The manual approach would have been to login the wazuh server and add the following to /var/ossec/etc/rules/local_rules.xml.

<!-- Local rules -->
<group name="vulnerability-detector">
  <rule id="100010" level="0">
    <field name="vulnerability.cve">CVE-2019-20367</field>
    <description>Vulnerable non-upgradeable packages</description>

The IaC approach is to first add an Ansible task doing the same in Enough via a merge request (which require no privilege) and then apply the change to the target infrastructure (with the appropriate privileges).

The benefit of a fully Free Software stack comes with testing. The above change must be verified to work properly before it is merged, in order to avoid problems when applied in production. To do so the entire infrastructure is created and configured every time an integration tests is required, and destroyed when it completes. tox -e wazuh will :

This would not be possible to do independently if there was even one dependency on a software or service that is not based on Free Software because automated tests would require permission from a third party to run. The test itself is straightforward once the stack is deployed:

def test_wazuh_vulnerability_ignored(host):
    with host.sudo():
        systemctl stop wazuh-manager
        rm /var/ossec/logs/alerts/alerts.log
        rm /var/ossec/queue/vulnerabilities/cve.db
        apt-get install -y libbsd0=0.9.1-2
        systemctl start wazuh-manager

        @retry.retry(AssertionError, tries=8)
        def get_alert():
            host.file("/var/ossec/logs/alerts/alerts.log").contains('non-upgradeable packages')

This effectively moves most of the devop workload outside of the infrastructure that requires privileges to a public place. In a less sensitive environment, trust can be built over time by working with volunteers while effectively solve real problems that the nonprofit are facing.

Thanks to pimthepoi for their contribution!