B61 Native Kubernetes Example

This example compiles a SeedEMU mini-Internet topology to native Kubernetes manifests and deploys it with seedemu.k8sTools. Kube-OVN/OVS is used as the secondary network fabric for SeedEMU simulated networks. k8sTools.py keeps this example directory small by copying setup/running resources into temporary directories while commands run.

Files

Compile

Run this once before deploying any of the four YAML examples:

cd examples/internet/b61_k8s_compile
python3 ./mini_internet_k8s.py

The default output directory is ./output. To write elsewhere:

python3 ./mini_internet_k8s.py --output-dir /tmp/seedemu-b61-output

To compile macvlan manifests instead of the default Kube-OVN/OVS manifests:

python3 ./mini_internet_k8s.py --cni-type macvlan

The default compiler mode writes k8s.kube-ovn.yaml, images.yaml, and one Docker build context per compiled SeedEMU node. --cni-type macvlan writes k8s.yaml with macvlan NetworkAttachmentDefinitions. k8sTools.py up also accepts images.txt metadata when present. The default namespace is seedemu.

Common Deploy Commands

Every setup mode follows the same command shape:

python3 ./k8sTools.py build \
  --input <one-config-yaml> \
  --config-k3s configK3s.yaml \
  --kubeconfig kubeconfig.yaml \
  --inventory inventory.yaml

python3 ./k8sTools.py up -f ./output -k kubeconfig.yaml -d configK3s.yaml
python3 ./k8sTools.py clean -f ./output -k kubeconfig.yaml
python3 ./k8sTools.py destroy -d configK3s.yaml

build creates or prepares infrastructure and writes configK3s.yaml, kubeconfig.yaml, and inventory.yaml. up builds/pushes workload images and applies the SeedEMU namespace. clean removes only the workload namespace. destroy removes the K3s/OVN setup and any KVM infrastructure recorded in configK3s.yaml.

Add --keep-temp to any k8sTools.py command when you need to inspect its temporary setup/running directory after a failure.

YAML Choices

configKvmOvnSimply.yaml

Use this when you want the shortest local KVM configuration.

python3 ./k8sTools.py build \
  --input configKvmOvnSimply.yaml \
  --config-k3s configK3s.yaml \
  --kubeconfig kubeconfig.yaml \
  --inventory inventory.yaml

This file is a minimal template for kind: kvmOvn. It only specifies the cluster name, master/worker resources, worker count, and SSH user/key. The k8sTools layer fills the normal KVM defaults such as VM names, IP/MAC planning, registry port, Kube-OVN fabric, output paths, and K3s defaults. Treat it as the starting point for a new local KVM experiment.

configKvmOvn.yaml

Use this for a more explicit local KVM deployment.

python3 ./k8sTools.py build \
  --input configKvmOvn.yaml \
  --config-k3s configK3s.yaml \
  --kubeconfig kubeconfig.yaml \
  --inventory inventory.yaml

Compared with configKvmOvnSimply.yaml, this file pins VM names, IP ranges, MAC ranges, registry port, and fabric.type: ovn. It also disables the legacy single base-image path with kvm.legacyBaseImagePath: "". This is the safer choice when the same libvirt host may already contain other SeedEMU/K3s VMs and you want predictable naming.

configK3sOvn.yaml

Use this when Kubernetes should be built on existing physical machines.

python3 ./k8sTools.py build \
  --input configK3sOvn.yaml \
  --config-k3s configK3s.yaml \
  --kubeconfig kubeconfig.yaml \
  --inventory inventory.yaml

Each nodes[] entry describes a real machine: name, role, management ip, optional connection, and ssh.user/key. A node with connection: local runs setup commands locally; other nodes are reached by SSH. Edit the IPs, users, and keys before running this on another lab.

This path can install or reinstall K3s on the listed machines. Confirm SSH key-based login and sudo -n true on every node first.

configMultiHostKvmOvn.yaml

Use this when multiple physical machines should act as KVM hypervisors for one K3s cluster.

python3 ./k8sTools.py build \
  --input configMultiHostKvmOvn.yaml \
  --config-k3s configK3s.yaml \
  --kubeconfig kubeconfig.yaml \
  --inventory inventory.yaml

The hypervisors[] section describes each physical KVM host. Each hypervisor gets its own routed libvirt subnet through routedSubnet, and vmCount controls how many K3s VMs are created there. vmSsh is the SSH account/key injected into those guest VMs. master.placement selects which hypervisor owns the K3s master. multiHostKvm.routingTunnel can create VXLAN next hops between hypervisors when the underlay does not provide a direct L2 route.

This mode modifies every listed hypervisor: it may create libvirt networks, routes, optional VXLAN interfaces, VM disks, cloud-init data, and K3s nodes.

Prerequisites

All modes require Python 3.10+, Docker/buildx, kubectl, ansible-playbook, ssh, scp, and curl.

The KVM modes also require libvirt/KVM tools such as virsh, virt-install, qemu-img, and a working libvirt network. The physical-node and multi-host flows require SSH key authentication and non-interactive sudo on all remote machines.

Generate a dedicated SSH key if needed:

ssh-keygen -t ed25519 -f ~/.ssh/seedemu_k8s -C "seedemu-k8s"
chmod 600 ~/.ssh/seedemu_k8s
ssh-keygen -y -f ~/.ssh/seedemu_k8s > ~/.ssh/seedemu_k8s.pub

Then update the selected YAML:

ssh:
  user: ubuntu
  key: ~/.ssh/seedemu_k8s

For physical machines, install the public key and verify sudo:

ssh-copy-id -i ~/.ssh/seedemu_k8s.pub seed@192.0.2.11
ssh -i ~/.ssh/seedemu_k8s seed@192.0.2.11 'sudo -n true'

Runtime Notes

• k8sTools.py up infers the logical compiler image prefix from output/images.yaml or output/images.txt; use --image-registry-prefix only for custom outputs.
• During up, generated Dockerfile FROM images such as handsonsecurity/seedemu-base:2.0 and handsonsecurity/seedemu-router:2.0 are prepared on the local host first and then loaded into the registry/master Docker daemon before remote buildx runs.
• k8sTools.py clean keeps the cluster infrastructure and only removes the deployed SeedEMU workload.
• k8sTools.py destroy uses metadata embedded in generated configK3s.yaml; do not delete that file before cleanup.
• Real build, up, down, and destroy commands modify local or remote infrastructure. The compile step alone is non-destructive.

Quick Kubernetes Debug Commands

Run these commands from this directory. Use the local stable kubeconfig explicitly so a stale KUBECONFIG from another experiment is not used by mistake:

export B61_NS=seedemu
export B61_KUBECONFIG=./examples/internet/b61_k8s_compile/kubeconfig.yaml

Check namespace, nodes, and pod status:

kubectl --kubeconfig "${B61_KUBECONFIG}" get ns
kubectl --kubeconfig "${B61_KUBECONFIG}" get namespace "${B61_NS}"
kubectl --kubeconfig "${B61_KUBECONFIG}" get nodes -o wide
kubectl --kubeconfig "${B61_KUBECONFIG}" -n "${B61_NS}" get pods -o wide
kubectl --kubeconfig "${B61_KUBECONFIG}" -n "${B61_NS}" get pods -l seedemu.io/role=brd -o wide
kubectl --kubeconfig "${B61_KUBECONFIG}" -n "${B61_NS}" get deploy -o wide
kubectl --kubeconfig "${B61_KUBECONFIG}" -n "${B61_NS}" get events --sort-by='.lastTimestamp'

Inspect or enter one pod/deployment. Replace the object name with a real name from kubectl get pods or kubectl get deploy:

kubectl --kubeconfig "${B61_KUBECONFIG}" -n "${B61_NS}" describe pod <pod-name>
kubectl --kubeconfig "${B61_KUBECONFIG}" -n "${B61_NS}" logs <pod-name> --tail=80
kubectl --kubeconfig "${B61_KUBECONFIG}" -n "${B61_NS}" exec -it <pod-name> -- bash
kubectl --kubeconfig "${B61_KUBECONFIG}" -n "${B61_NS}" exec -it deploy/<deploy-name> -- sh
kubectl --kubeconfig "${B61_KUBECONFIG}" -n "${B61_NS}" exec deploy/<deploy-name> -- birdc show protocols

as1862brd-r13-1.13.7.70-7564dbb49c-9wctj

Large post-BIRD runs can intermittently return TLS handshake timeout, Client.Timeout exceeded, or http2: client connection lost while VM CPUs are saturated. First retry the same command with an explicit request timeout:

kubectl --kubeconfig "${B61_KUBECONFIG}" --request-timeout=120s -n "${B61_NS}" get pods -l seedemu.io/role=brd -o wide