etcd + TLS 集群部署

不知在哪篇技术文档中看到, kubernetes master和etcd分开部署模式,因为集群的状态都保存在etcd中,这样当kubernetes master挂掉后,通过API Server交互的Scale等功能无法使用外,其他已经部署的Pod仍然能继续工作。

基于这种考虑,通过yum以及修改etcd.conf方式部署了一个三节点的etcd集群,但对于企业使用而言,虽然在局域网内访问,多数情况下还是需要配置安全证书,就好像很多政府部门因为三级等保的要求必须在weblogic中配置ssl一样,自己尝试在之前的环境中通过修改conf文件下配置,启动时遭遇各种问题失败,但同样的证书后修改为命令行方式配置后以及手工安装etcd后部署成功。记录如下:

安装cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

chmod +x cfssl_linux-amd64 cfssljson_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

证书相关处理

证书名称配置文件用途
etcd-root-ca.pemetcd-root-ca-csr.jsonetcd 根 CA 证书
etcd.pemetcd-gencert.json、etcd-csr.jsonetcd 集群证书

1

 

 

 

 

Etcd 证书生成所需配置文件如下:

etcd-root-ca-csr.json

{   "key": {     

        "algo": "rsa",     

        "size": 4096

       },   

    "names": [     

       {       

       "O": "etcd",       

       "OU": "etcd Security",       

       "L": "Beijing",       

       "ST": "Beijing",       

       "C": "CN"

       }   

     ],   

   "CN": "etcd-root-ca"

 }

etcd-gencert.json

{  

   "signing": {    

      "default": {        

          "usages": [          

              "signing",

              "key encipherment",

              "server auth",

              "client auth"

             ],        

           "expiry": "87600h"

    }

  }

}

etcd-csr.json

{  

   "key": {    

      "algo": "rsa",    

      "size": 4096

    },  

   "names": [

     {      

        "O": "etcd",      

        "OU": "etcd Security",      

        "L": "Beijing",      

        "ST": "Beijing",      

        "C": "CN"

     }

   ],  

  "CN": "etcd",  

  "hosts": [    

     "127.0.0.1",

     "localhost",

     "192.168.0.153",

     "192.168.0.154",

     "192.168.0.164",

     "master",

     "node1",

     "node2"

  ]

}

最后生成 Etcd 证书

cfssl gencert --initca=true etcd-root-ca-csr.json | cfssljson --bare etcd-root-ca

cfssl gencert --ca etcd-root-ca.pem --ca-key etcd-root-ca-key.pem --config etcd-gencert.json etcd-csr.json | cfssljson --bare etcd

生成的证书列表如下

 1207331-20171026153118851-1877842771.png

三、部署 HA ETCD

安装前准备

关闭 selinux, setenforce 0

关闭防火墙, systemctl stop firewalld; iptables -F

ntpdate 时间同步

ntpdate time1.aliyun.com

安装 Etcd

以下操作都是在master节点上操作

ETCD 直接采用 rpm 安装,RPM 可以从 Fedora 官方仓库  获取 spec 文件自己 build,或者直接从 rpmFind 网站  搜索

下载 rpm包

wget ftp://195.220.108.108/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/e/etcd-3.2.7-1.fc28.x86_64.rpm  

#分发并安装

I="192.168.0.153 192.168.0.154 192.168.0.164"

for IP in $I; do

   etcd-3.2.7-1.fc28.x86_64.rpm root@$IP:~    

   ssh root@$IP rpm -ivh etcd-3.2.7-1.fc28.x86_64.rpm

done

分发证书

I="192.168.0.153 192.168.0.154 192.168.0.164"

for IP in $I; do

    ssh root@$IP mkdir /etc/etcd/ssl/    

    scp *.pem root@$IP:/etc/etcd/ssl/    

    ssh root@$IP chown -R etcd:etcd /etc/etcd/ssl/    

    ssh root@$IP chmod -R 755 /etc/etcd/

done

修改配置

rpm 安装好以后直接修改  /etc/etcd/etcd.conf  配置文件即可,其中单个节点配置如下(其他节点只是名字和 IP 不同)

# [member]

ETCD_NAME

etcd0

ETCD_DATA_DIR

"/var/lib/etcd/etcd0.etcd"

ETCD_WAL_DIR

"/var/lib/etcd/wal"

ETCD_SNAPSHOT_COUNT

"100"

ETCD_HEARTBEAT_INTERVAL

"100"

ETCD_ELECTION_TIMEOUT

"1000"

ETCD_LISTEN_PEER_URLS

"https://192.168.0.153:2380"

ETCD_LISTEN_CLIENT_URLS

"https://192.168.0.153:2379,http://127.0.0.1:2379"

ETCD_MAX_SNAPSHOTS

"5"

ETCD_MAX_WALS

"5"

#ETCD_CORS=""

# [cluster]

ETCD_INITIAL_ADVERTISE_PEER_URLS

"https://192.168.0.153:2380"

# if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."

ETCD_INITIAL_CLUSTER

"etcd0=https://192.168.0.153:2380,etcd1=https://192.168.0.154:2380,etcd2=https://192.168.0.164:2380"

ETCD_INITIAL_CLUSTER_STATE

"new"

ETCD_INITIAL_CLUSTER_TOKEN

"etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS

"https://192.168.0.153:2379"

 

#ETCD_DISCOVERY="" 

#ETCD_DISCOVERY_SRV="" 

#ETCD_DISCOVERY_FALLBACK="proxy" 

#ETCD_DISCOVERY_PROXY="" 

#ETCD_STRICT_RECONFIG_CHECK="false" 

#ETCD_AUTO_COMPACTION_RETENTION="0"

# [proxy] 

#ETCD_PROXY="off"/ 

#ETCD_PROXY_FAILURE_WAIT="5000" 

#ETCD_PROXY_REFRESH_INTERVAL="30000" 

#ETCD_PROXY_DIAL_TIMEOUT="1000" 

#ETCD_PROXY_WRITE_TIMEOUT="5000" 

#ETCD_PROXY_READ_TIMEOUT="0"

# [security]  

ETCD_CERT_FILE

"/etc/etcd/ssl/etcd.pem"  

ETCD_KEY_FILE

"/etc/etcd/ssl/etcd-key.pem"  

ETCD_CLIENT_CERT_AUTH

"true"  

ETCD_TRUSTED_CA_FILE

"/etc/etcd/ssl/etcd-root-ca.pem"  

ETCD_AUTO_TLS

"true"  

ETCD_PEER_CERT_FILE

"/etc/etcd/ssl/etcd.pem"  

ETCD_PEER_KEY_FILE

"/etc/etcd/ssl/etcd-key.pem"  

ETCD_PEER_CLIENT_CERT_AUTH

"true"  

ETCD_PEER_TRUSTED_CA_FILE

"/etc/etcd/ssl/etcd-root-ca.pem"  

ETCD_PEER_AUTO_TLS

"true"

# [logging] 

#ETCD_DEBUG="false" 

# examples for -log-package-levels etcdserver=WARNING,security=DEBUG 

#ETCD_LOG_PACKAGE_LEVELS=" "

node 节点要修改的地方:

ETCD_NAME

ETCD_LISTEN_PEER_URLS

ETCD_LISTEN_CLIENT_URLS

ETCD_INITIAL_ADVERTISE_PEER_URLS

ETCD_ADVERTISE_CLIENT_URLS

修改完成后,还需要修改/usr/lib/systemd/system/etcd.service文件内容如下:

1.  [Unit ]

2. Description =Etcd  Server

3. After

network . target

4. After

network

online . target

5. Wants

network

online . target

6.  [Service ]

7. Type

notify

8. WorkingDirectory =/ var / lib / etcd /

9. EnvironmentFile =-/ etc / etcd / etcd . conf

10. User

etcd

11.  # set GOMAXPROCS to number of processors

12. ExecStart =/ bin / bash

c "GOMAXPROCS=$(nproc) /usr/bin/etcd \

13.  --name="${ETCD_NAME}" \

14.  --data-dir="${ETCD_DATA_DIR}" \

15.  --listen-peer-urls="${ETCD_LISTEN_PEER_URLS}" \

16.  --advertise-client-urls="${ETCD_ADVERTISE_CLIENT_URLS}" \

17.  --initial-cluster-token="${ETCD_INITIAL_CLUSTER_TOKEN}" \

18.  --initial-cluster="${ETCD_INITIAL_CLUSTER}" \

19.  --initial-cluster-state="${ETCD_INITIAL_CLUSTER_STATE}" \

20.  --listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}""

21. Restart

on

failure

22. LimitNOFILE

65536

23.  [Install ]

24. WantedBy

multi

user . target

 

4、启动验证

配置修改后在每个节点进行启动即可,注意,Etcd 各个节点间必须保证时钟同步,否则会造成启动失败等错误

systemctl daemon-reload

systemctl start etcd

systemctl enable etcd

启动成功后验证节点状态

export ETCDCTL_API=3

etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.0.153:2379,https://192.168.0.154:2379,https://192.168.0.164:2379 endpoint health

1207331-20171026161541898-1047037684.png

本文出自https://www.cnblogs.com/Tempted/p/7737361.html

参考http://www.361way.com/etcd-cluster/5468.html

 

转载于:https://blog.51cto.com/lookingdream/2108136

代码交流 2021