6 minutes
Proxmox Homelab Software
Intro
I’ve been on an exciting path of building and running a homelab over the past few months. This post shares my vision, how i’ve built it to date, and what might be next.
Requirements / Decisions / Constraints
- Use version controlled infrastructure as code to manage the lifecycle of linux guests running on the homelab. Preview infrastructure changes prior to execution.
- Use version controlled config management tools to setup linux guests on creation and if the config is modified.
- Keep the homelab as self contained as possible (there will still be lots of external requirements).
- Use automatically renewed certificates for tls/https everywhere.
- Use public DNS for private (RFC1918) A records
Key Components
- Self-hosted Gitlab for version control and collaboration.
- Self-hosted terraform state in postgresql
- Self-hosted Atlantis for Terraform workflows via Gitlab merge requests.
ansible-pull
on a systemd timer running directly on guests- local docker registry which caches docker images (pull-through cache)
note: there are lots of ways to set this up. this is how i’ve done it.
Prerequisites
- One or more Proxmox Virtual Environment physical servers
- Root access to the PVE web console on port
8006
- A homelab domain name. ex:
my-special-homelab.xyz
- Cloudflare account, authoritative dns for your homelab domain.
- Password manager for all of the various credentials needed
One-time Setup Steps
Gitlab
- Setup initial admin user
- Create homelab project
- Create repo for terraform
- Create repo for ansible
- Create a gitlab user for atlantis
Atlantis
- postgres creds
- proxmox creds
- cloudflare creds
- gitlab creds
- Generate an ssh keypair which is used to do minimal bootstrapping of new lxc guests following creation.
- gitlab: debian bookworm LXC. caddy binary, gitlab “fat” docker image, systemd units.
- My current setup uses the now-archived, gitlab role from Jeff Geerling
- atlantis: debian bookworm LXC. caddy binary, atlantis binary or docker image, systemd units. (i’m not using docker)
- postgresql: Create from the PVE host console shell using tteck pve helper scripts, databases > postgresql.
How to add a new LXC guest
- Reserve the next virtual MAC address in my MAC address table I maintain in a Notion doc.
- Reserve the next private IP address in my Homelab CIDR range, which I maintain in a Notion doc.
- Create a DHCP reservation in Unifi for the MAC address and IP Address pair above.
- Select the docker image name and tag I intend to run in the LXC behind Caddy.
- Select the subdomain I intend to use for the DNS entry. ex:
my-app
->my-app.my-homelab.cloud
- Create the playbook which invokes the docker, caddy, and systemd bits. Mostly referencing existing roles, and ensure the changes are merged to
main
- Create the terraform bits with the new
proxmox_lxc
resource- which defines: the proxmox host to use, the hostname, the lxc os template (debian 12 - the same version stored on the same device on all hosts), run the lxc unprivileged, set memory and cpu cores, allow nested virtualization (for docker to work), the root filesystem size, ssh authorized keys, the mac address, the provisioner script, and ignore changes to the ostemplate (when I upgrade the version in the future i don’t want the guest to be destroyed)
- provisioner script: install ansible and git. install ansible roles. Setup the ansible-pull systemd timer to run every 15 minutes.
ansible-pull
can be configured to run every time or only when there is a new commit in git. Install the dependent ansible roles I need. Finally run the systemd unit once immediately.
the terraform manifest:
terraform {
backend "pg" {
conn_str = "postgres://pg.myhomelab.net:5432/terraform_backend"
}
required_version = "1.5.7"
required_providers {
proxmox = {
source = "Telmate/proxmox"
version = "2.9.14"
}
cloudflare = {
source = "cloudflare/cloudflare"
version = "4.20.0"
}
random = {
source = "hashicorp/random"
}
}
}
provider "proxmox" {
pm_api_url = "https://pve.myhomelab.net:8006/api2/json"
}
# export CLOUDFLARE_API_TOKEN="token"
provider "cloudflare" {}
data "cloudflare_zone" "myhomelab_net" {
name = "myhomelab.net"
}
locals {
# the intention is for this template to be available on all pve hosts.
# it needs to be added one time on each host and updated periodically here and downloaded.
debian_12_bookwork_lxc_template = "local:vztmpl/debian-12-standard_12.2-1_amd64.tar.zst"
public_keys = {
gitlab_myhomelab_net = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMQa6UoZoNZouT9y7udMlsMRh2nZaZZ0aoy72sDHjkyQ"
github_com = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZY+CBnJyRZDM+IQHRevG43mtk1Jat2j0IdqEPn8bU7"
atlantis_root = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICW6T8sU7fxNLQ+9DxtMOxlpzfZMb8tIpJ+w/W+TXhPj [email protected]"
}
guests = {
it_tools = {
mac = "0A:BC:DE:00:00:15"
dhcp_reservation = "10.20.71.113"
domain = "mailcatcher.my-homelab.net"
}
}
}
resource "random_password" "mailcatcher" {
length = 30
special = true
override_special = "_%@"
}
resource "proxmox_lxc" "mailcatcher" {
target_node = "pve6"
start = true
onboot = true
hostname = local.guests.mailcatcher.domain
ostemplate = local.debian_12_bookwork_lxc_template
password = random_password.mailcatcher.result
unprivileged = true
memory = "512"
cores = 1
features {
nesting = true
}
rootfs {
storage = "local-lvm"
size = "5G"
}
ssh_public_keys = <<-EOT
${local.public_keys.gitlab_myhomelab_net}
${local.public_keys.github_com}
${local.public_keys.atlantis_root}
EOT
network {
name = "eth0"
bridge = "vmbr0"
ip = "dhcp"
hwaddr = local.guests.mailcatcher.mac
}
provisioner "file" {
source = "setup-ansible-pull-cron.sh"
destination = "/tmp/script.sh"
}
provisioner "remote-exec" {
inline = [
"chmod +x /tmp/script.sh",
"/tmp/script.sh ${local.guests.mailcatcher.playbook}"
]
}
connection {
type = "ssh"
user = "root"
private_key = file("/root/.ssh/id_ed25519")
host = local.guests.mailcatcher.dhcp_reservation
}
lifecycle {
ignore_changes = [ostemplate]
}
}
resource "cloudflare_record" "mailcatcher" {
zone_id = data.cloudflare_zone.myhomelab_net.zone_id
name = element(split(".", local.guests.mailcatcher.domain), 0)
value = local.guests.mailcatcher.dhcp_reservation
type = "A"
ttl = 1
proxied = false
comment = local.dns_comment
}
the guest provisioner script (setup-ansible-pull-cron.sh):
#!/bin/bash
ANSIBLE_PLAYBOOK=$1
apt-get update
apt-get install ansible -y
apt-get install git -y
cat > /etc/systemd/system/ansible-pull.timer <<EOF
[Unit]
Description=Run ansible-pull every 15 minutes
[Timer]
OnBootSec=5min
OnUnitActiveSec=15min
Unit=ansible-pull.service
[Install]
WantedBy=timers.target
EOF
cat > /etc/systemd/system/ansible-pull.service <<EOF
[Unit]
Description=ansible-pull
[Service]
ExecStart=/usr/bin/ansible-pull -U https://gitlab.my-homelab.net/my-homelab.net/ansible.git -C main -i localhost, $ANSIBLE_PLAYBOOK
EOF
ansible-galaxy install geerlingguy.docker,7.0.2
ansible-galaxy install git+https://github.com/tphummel/ansible-role-caddy-tls-dns.git,main
systemctl daemon-reload
systemctl enable ansible-pull.timer
systemctl start ansible-pull.timer
# run ansible-pull once, adhoc, right now
systemctl start ansible-pull.service
# block until ansible-pull has done a first run
while systemctl is-active --quiet ansible-pull.service;
do
echo "Waiting for adhoc run of ansible-pull to finish...";
sleep 2;
done
Ansible pull playbook
---
- hosts: localhost
become: yes
gather_facts: yes
vars:
container:
name: it-tools
description: 'self-hosted it-tools.tech'
image_name: 'corentinth/it-tools'
image_tag: '2023.12.21-5ed3693'
exposed_port: '8080'
internal_port: '8080'
domain: 'it-tools.my-homelab.net'
dns_api_token: 'cloudflare token to edit dns records for acme tls challenge'
tls_email: '[email protected]'
roles:
- geerlingguy.docker
- role: ansible-role-caddy-tls-dns
vars:
caddy_domain: '{{ domain }}'
caddy_tls_email: '{{ tls_email }}'
caddy_dns_api_token: '{{ dns_api_token }}'
caddy_target_port: "{{ container.exposed_port }}"
tasks:
- name: Create systemd service file for it-tools.tech
copy:
content: |
[Unit]
Description={{ container.description }}
After=docker.service
Requires=docker.service
[Service]
ExecStart=/usr/bin/docker run --name {{ container.name }} -p {{ container.exposed_port }}:{{ container.internal_port }} {{ container.image_name}}:{{ container.image_tag }}
ExecStop=/usr/bin/docker stop {{ container.name }}
ExecStopPost=/usr/bin/docker rm -f {{ container.name }}
[Install]
WantedBy=multi-user.target
dest: /etc/systemd/system/{{ container.name }}.service
mode: 0644
notify:
- Reload systemd
- Start service
handlers:
- name: Reload systemd
systemd:
daemon_reload: yes
- name: Start service
systemd:
name: "{{ container.name }}"
enabled: yes
state: restarted
Applications
Once this is all set up, what can you do?
Self host your:
- private access to your application
- rss feed reader, link saver, and link archiver.
- games
- location data
- uptime monitoring
- backup your critical data to offsite storage. tools like syncthing, rclone, airbyte
- ifttt-style automation. tools like activepieces, node-red
- misc handy tools: it-tech.tools, hrconvert2
- mailhog
- olivetin
- nocodb
- wiki
- pastebin / secret sharing
- a single html page with all your links in one spot
- backup images from your phone
- podcast and audiobook library
- connect cloudflare workers to software running in your homelab
- connect to your apps privately over cloudflare tunnel (or tailscale, etc)
Scan awesome-selfhosted and let your imagination run. Focus on utility. Focus on paid services you use every day.