QEMU/KVM上で Kubernetes The Hard Way （事前準備からTLS証明書の作成まで）

QEMU/KVM上で https://github.com/kelseyhightower/kubernetes-the-hard-wayを試してみたので、ログを残しておく。元々の文章では、GCPを前提としているが、今回は手元のQEMU/KVM上で構築した。

バージョン一覧

kubernetes-the-hard-way bf2850974e19c118d04fdc0809ce2ae8a0026a27
Kubernetes 1.12.0
containerd Container Runtime 1.2.0-rc.0
gVisor 50c283b9f56bb7200938d9e207355f05f79f0d17
CNI Container Networking 0.6.0
etcd v3.3.9
CoreDNS v1.2.2

1. 事前準備

VMイメージを取得して、起動するまで。

# VMイメージの準備
$ http_dir=http://ftp.jaist.ac.jp/pub/Linux/Fedora/releases/30/Cloud/x86_64/images
$ wget $http_dir/Fedora-Cloud-Base-30-1.2.x86_64.qcow2
$ virt-customize -a /var/lib/libvirt/images/Fedora-Cloud-Base-30-1.2.x86_64.qcow2 \
  --run-command 'yum remove cloud-init* -y' \
  --root-password password:root
$ cd /var/lib/libvirt/images/
$ for i in 0 1 2 ; do \
  sudo cp ./Fedora-Cloud-Base-30-1.2.x86_64.qcow2 controller-$i.qcow2; \
  sudo cp ./Fedora-Cloud-Base-30-1.2.x86_64.qcow2 worker-$i.qcow2; \
  done

# VM起動
$ for name in controller-0 controller-1 controller-2 worker-0 worker-1 worker-2
  do
    sudo virt-install --name=$name --virt-type kvm --graphics none \
                      --disk /var/lib/libvirt/images/$name.qcow2 --vcpus=1 \
                      --ram=512 --network network=default,model=virtio \
                      --import --noautoconsole
  done
$ sudo virsh list --all
Id   Name           State
--------------------------date: 2020-07-28T00:00:00-00:00
draft: false
---
4    controller-0   実行中
5    controller-2   実行中
6    controller-1   実行中
7    worker-0       実行中
8    worker-1       実行中
9    worker-2       実行中

# VM初期設定（VMごとに）
$ hostnamectl set-hostname controller-0
$ nmcli c m "System eth0" ipv4.addresses "192.168.122.10/24" \
                          ipv4.method manual \
                          connection.autoconnect yes \
                          ipv4.gateway 192.168.122.1 \
                          ipv4.dns 8.8.8.8
$ systemctl restart NetworkManager

# host側のNW設定
$ sudo sysctl -p /etc/sysctl.conf
net.ipv4.ip_forward = 1
$ cat /etc/hosts | grep 192.168.122
192.168.122.10 controller-0
192.168.122.11 controller-1
192.168.122.12 controller-2
192.168.122.13 worker-0
192.168.122.14 worker-1
192.168.122.15 worker-2

2. クライアントツールのインストール

# Public Key Infrastructure (PKI)のため
$ wget -q --show-progress --https-only --timestamping \
   https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 \
   https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
$ chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
$ mv cfssl_linux-amd64 /usr/local/bin/cfssl
$ mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
$ cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.6

# kubectlのインストール
$ wget https://storage.googleapis.com/kubernetes-release/release/v1.12.0/bin/linux/amd64/kubectl
$ chmod +x kubectl
$ mv kubectl /usr/local/bin
$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0",
                             GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0",
                             GitTreeState:"clean",
                             BuildDate:"2018-09-27T17:05:32Z",
                             GoVersion:"go1.10.4", Compiler:"gc",
                             Platform:"linux/amd64"}

3. コンピュートリソースの準備

今回はGCPを使わないので省略。

4. 認証局のセットアップとTLS証明書の生成

CloudFlareのPKIツールキットを使って、PKI基盤を作る。まず、CA（Certificate Authority）のための設定ファイル、証明書、秘密鍵を作る。

$ cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

$ cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "Oregon"
    }
  ]
}
EOF

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[INFO] generating a new CA key and certificate from CSR
[INFO] generate received request
[INFO] received CSR
[INFO] generating key: rsa-2048
[INFO] encoded CSR
[INFO] signed certificate with serial number XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

$ ls ca*pem
ca-key.pem  ca.pem

続いて、各コンポーネントのクライアント・サーバ証明書を作る。また、adminユーザのためのクライアント証明書を作る。

$ cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:masters",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

$ cfssl gencert \
   -ca=ca.pem \
   -ca-key=ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   admin-csr.json | cfssljson -bare admin
[INFO] generate received request
[INFO] received CSR
[INFO] generating key: rsa-2048
[INFO] encoded CSR
[INFO] signed certificate with serial number XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

$ ls admin*pem
admin-key.pem  admin.pem

Node AuthorizerがKubeletを認証する。証明書と秘密鍵をノードごとに作成する。

$ for instance in worker-0 worker-1 worker-2; do
cat > ${instance}-csr.json <<EOF
{
  "CN": "system:node:${instance}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:nodes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

ip=`cat /etc/hosts | grep $instance | awk '{print $1;}'`
echo $instance $ip
cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${instance},${ip} \
  -profile=kubernetes \
  ${instance}-csr.json | cfssljson -bare ${instance}
done

$ ls worker*pem
worker-0-key.pem  worker-0.pem  worker-1-key.pem  worker-1.pem  worker-2-key.pem  worker-2.pem

kube-controller-managerのためのクライアント証明書と秘密鍵を作る。

$ cat > kube-controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

$ cfssl gencert \
   -ca=ca.pem \
   -ca-key=ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
[INFO] generate received request
[INFO] received CSR
[INFO] generating key: rsa-2048
[INFO] encoded CSR
[INFO] signed certificate with serial number XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

$ ls kube-controller-manager*pem
kube-controller-manager-key.pem kube-controller-manager.pem

kube-proxyのための証明書も作る。

$ cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:node-proxier",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

$ cfssl gencert \
   -ca=ca.pem \
   -ca-key=ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   kube-proxy-csr.json | cfssljson -bare kube-proxy
[INFO] generate received request
[INFO] received CSR
[INFO] generating key: rsa-2048
[INFO] encoded CSR
[INFO] signed certificate with serial number XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

$ ls kube-proxy*pem
kube-proxy-key.pem  kube-proxy.pem

kube-scheduler用のクライアント証明書と秘密鍵を作る。

$ cat > kube-scheduler-csr.json <<EOF
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

$ cfssl gencert \
   -ca=ca.pem \
   -ca-key=ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   kube-scheduler-csr.json | cfssljson -bare kube-scheduler
[INFO] generate received request
[INFO] received CSR
[INFO] generating key: rsa-2048
[INFO] encoded CSR
[INFO] signed certificate with serial number XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
$ ls kube-scheduler*pem
kube-scheduler-key.pem  kube-scheduler.pem

kubernetes APIサーバのクライアント証明書と秘密鍵を作成する。 LBをたてるのはめんどうなので、controller-0をAPI public addressとした。

$ KUBERNETES_PUBLIC_ADDRESS=192.168.122.10

$ cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

$ cfssl gencert \
   -ca=ca.pem \
   -ca-key=ca-key.pem \
   -config=ca-config.json \
   -hostname=10.32.0.1,10.240.0.10,10.240.0.11,10.240.0.12,${KUBERNETES_PUBLIC_ADDRESS},127.0.0.1,kubernetes.default \
   -profile=kubernetes \
   kubernetes-csr.json | cfssljson -bare kubernetes
[INFO] generate received request
[INFO] received CSR
[INFO] generating key: rsa-2048
[INFO] encoded CSR
[INFO] signed certificate with serial number XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
$ ls kubernetes*pem
kubernetes-key.pem  kubernetes.pem

service-accountの証明書と秘密鍵を作る。

$ cat > service-account-csr.json <<EOF
{
  "CN": "service-accounts",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

$ cfssl gencert \
   -ca=ca.pem \
   -ca-key=ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   service-account-csr.json | cfssljson -bare service-account
[INFO] generate received request
[INFO] received CSR
[INFO] generating key: rsa-2048
[INFO] encoded CSR
[INFO] signed certificate with serial number XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
$ ls service-account*pem
service-account-key.pem  service-account.pem

サーバ・クライアント証明書を配布する。

$ for instance in worker-0 worker-1 worker-2; do
    scp ca.pem ${instance}-key.pem ${instance}.pem root@${instance}:~/
  done

$ for instance in controller-0 controller-1 controller-2; do
    scp ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \
      service-account-key.pem service-account.pem root@${instance}:~/
  done

続きは余裕があるときにやっていく。

バージョン一覧#

1. 事前準備#

2. クライアントツールのインストール#

3. コンピュートリソースの準備#

4. 認証局のセットアップとTLS証明書の生成#

バージョン一覧

1. 事前準備

2. クライアントツールのインストール

3. コンピュートリソースの準備

4. 認証局のセットアップとTLS証明書の生成