Development machines made easy
From lazyness to awesomeness

Let’s dig into how I manage creating and managing development machines. The Why and the How.

Table of Contents

For work, I sometimes need to create and use development machines to hack on a specific case. There is multiple reason I would need to create those machines instead of working directly on my laptop or desktop. Let’s look at those, and how I try to automate the hell out of it (because I am really lazy).

1 Use case

Let’s look into some use-case that are useful to me

  • Create and/or test packages for a specific distribution — most likely RPM-based (Fedora, RHEL, …) and Debian-based (Ubuntu, Debian, …).
  • From scratch machine,
    • to make sure some documentation are complete for people to start hacking on a project, and using a tool.
    • to make demo, recording or something 👼
  • Cluster machines for Kubernetes or Openshift.

Some requirements and nice-to-have:

  • Automate provisioning of these machines.
  • Use virtual machine for most case (libvirt, qemu+kvm).
  • Auto updates of the “provisioning”

Targeted system are, for now :

  • Fedora, RHEL
  • Debian, Ubuntu
  • NixOS

2 Base images

Let’s use packer with qemu for those cases — and let’s create a repository where we’re gonna write the development machine recipes : vdemeester/machines.

2.1 NixOS recipes

The initial source of packer recipes comes from nix-community/nixbox, but I’m tailoring them to my needs

{
  "builders": [
    {
      "boot_wait": "40s",
      "boot_command": [
        "echo http://{{ .HTTPIP }}:{{ .HTTPPort}} > .packer_http<enter>",
        "mkdir -m 0700 .ssh<enter>",
        "curl $(cat .packer_http)/install_rsa.pub > .ssh/authorized_keys<enter>",
        "systemctl start sshd<enter>"
      ],
      "http_directory": "scripts",
      "iso_checksum_type": "sha256",
      "shutdown_command": "shutdown -h now",
      "ssh_private_key_file": "./scripts/install_rsa",
      "ssh_port": 22,
      "ssh_username": "root",
      "type": "qemu",
      "iso_url": "https://d3g5gsiof5omrk.cloudfront.net/nixos/18.09/nixos-18.09.1799.b9fa31cea0e/nixos-minimal-18.09.1799.b9fa31cea0e-x86_64-linux.iso",
      "iso_checksum": "cc7c399c5fe4672383fe54cb1d648854a0d6732765fe1a61bb38b3fe3b7c6d2f",
      "disk_interface": "virtio-scsi",
      "qemuargs": [
        [
          "-m",
          "1024"
        ]
      ]
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "script": "./scripts/install.sh"
    }
  ]
}

Let’s look at the provisioning script. We don’t want to create a full specific configuration for these images as we will use ansible for the final provisioning.

  • scripts/install.sh

    #!/bin/sh -e
    
    packer_http=$(cat .packer_http)
    
    # Partition disk
    cat <<FDISK | fdisk /dev/sda
    n
    
    
    
    
    a
    w
    
    FDISK
    
    # Create filesystem
    mkfs.ext4 -j -L nixos /dev/sda1
    
    # Mount filesystem
    mount LABEL=nixos /mnt
    
    # Setup system
    nixos-generate-config --root /mnt
    
    curl -sf "$packer_http/machine.nix" > /mnt/etc/nixos/machine.nix
    curl -sf "$packer_http/builders/$PACKER_BUILDER_TYPE.nix" > /mnt/etc/nixos/hardware-builder.nix
    curl -sf "$packer_http/configuration.nix" > /mnt/etc/nixos/configuration.nix
    curl -sf "$packer_http/custom-configuration.nix" > /mnt/etc/nixos/custom-configuration.nix
    
    ### Install ###
    nixos-install
    
    ### Cleanup ###
    curl "$packer_http/postinstall.sh" | nixos-install
    
  • scripts/postinstall.sh

    #!/bin/sh
    
    # Make sure we are totally up to date
    nix-channel --add https://nixos.org/channels/nixos-18.09 nixos
    nix-channel --update
    nixos-rebuild switch --upgrade
    
    # Cleanup any previous generations and delete old packages that can be
    # pruned.
    
    for x in $(seq 0 2) ; do
      nix-env --delete-generations old
      nix-collect-garbage -d
    done
    
    
    # Remove install ssh key
    rm -rf /root/.ssh /root/.packer_http
    
    # Zero out the disk (for better compression)
    dd if=/dev/zero of=/EMPTY bs=1M
    rm -rf /EMPTY
    
  • scripts/machine.nix

    # This file is overwritten by the vagrant-nixos plugin
    { config, pkgs, ... }:
    {
      networking.hostName = "nixos-machine";
    }
    
  • scripts/configuration.nix

    { config, pkgs, ... }:
    
    {
      imports =
        [ # Include the results of the hardware scan.
          ./hardware-configuration.nix
          ./hardware-builder.nix
          ./machine.nix
          ./custom-configuration.nix
        ];
    
      # Use the GRUB 2 boot loader.
      boot.loader.grub.enable = true;
      boot.loader.grub.version = 2;
      boot.loader.grub.device = "/dev/sda";
    
      # remove the fsck that runs at startup. It will always fail to run, stopping
      # your boot until you press *.
      boot.initrd.checkJournalingFS = false;
    
      # Services to enable:
    
      # Enable the OpenSSH daemon.
      services.openssh.enable = true;
    
      # Enable DBus
      services.dbus.enable    = true;
    
      # Replace nptd by timesyncd
      services.timesyncd.enable = true;
    
      # Packages for Vagrant
      environment.systemPackages = with pkgs; [
        iputils
      ];
    
      # Creates a "vincent" users with password-less sudo access
      users = {
        extraGroups = [ { name = "vincent"; } ];
        extraUsers  = [
          # Try to avoid ask password
          { name = "root"; password = "vincent"; }
          {
            description     = "Vincent User";
            name            = "vincent";
            group           = "vincent";
            extraGroups     = [ "users" "wheel" ];
            password        = "vincent";
            home            = "/home/vincent";
            createHome      = true;
            useDefaultShell = true;
            openssh.authorizedKeys.keys = [
              "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO1sx5h44xnK/k0ODnQ3aQR8+nr7HC7u94fS3OhwQ6AvjqDGLnI6EP4sr4Yh2eXf8lHX+lkg8iZ6Z+y9dVnnzwveZfqbfOyh6t8Hg+M1nl26rwdYv+guU8khvh+Kzl9Vdb5dexf/hWQ/LcWvsuPO+tBmqajNTLYbGinqrMm3Bw2jJS/+DitgoT8hiuSTU1smY1CGzggHEdsx4+oDMuDMvRYwOBBHrUF00lZLx3zB3nGl1VFYD2St3vzlmzoZNrW7Rx8TRg02BTVAwd4qPHOMz8Kg+JmDhVig9yeqHo4FCwXxQ8+jk54Cd2el6TjfaA5HD2+e4FYLP6bMSLIabLTfLP vincent@wakasu"
            ];
          }
        ];
      };
    
      security.sudo.configFile =
        ''
          Defaults:root,%wheel env_keep+=LOCALE_ARCHIVE
          Defaults:root,%wheel env_keep+=NIX_PATH
          Defaults:root,%wheel env_keep+=TERMINFO_DIRS
          Defaults env_keep+=SSH_AUTH_SOCK
          Defaults lecture = never
          root   ALL=(ALL) SETENV: ALL
          %wheel ALL=(ALL) NOPASSWD: ALL, SETENV: ALL
        '';
    
    }
    
  • scripts/custom-configuration.nix

    { config, pkgs, ... }:
    
    {
    # Place here any custom configuration specific to your organisation (locale, ...)
    # if you want it to be part of the packer base image to be used with vagrant.
    }
    
  • scripts/builders/qemu.nix

    { modulesPath, ... }:
    {
      imports = [
        "${toString modulesPath}/profiles/qemu-guest.nix"
      ];
    }
    

And to build this image, a simple packer build nixos.json is required.

2.2 TODO Fedora recipes

2.3 TODO Debian recipes

2.4 TODO Ubuntu recipes

3 TODO Provisionning

Now that we have base images, we can start to play around, most likely with ansible to easily and quickly provision setups based on those. Those setups can includes :

  • development environment
    • Nix-based (aka Nixos or Nixpkgs on other machines)
    • Native-based (aka no Nix)
  • multiple node testing environment

3.1 TODO Development machines

3.2 TODO Kubernetes cluster

4 References

Emacs 24.3.50.3 (Org mode 8.0.3)

Vincent Demeester. Last Updated 2019-03-23 Sat 15:15 (exported 2019-05-15 Wed 18:20).