Appreciate life and record it

Kustomize

 2927  6    
Kustomize

Kustomize

kustomize介绍

模板化，方便管理多个kubernets的配置文件，并且通过patch更新，类似make和sed

了解这个需要过一遍官方文档和视频

官网链接https://kustomize.io/

kustomize较helm比较大的优势是已经植入到了kubectl apply里面，只需要加-k 参数就可以完成kustomize的效果

安装

https://kubectl.docs.kubernetes.io/installation/kustomize/binaries/

1

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

使用

准备hello world

假设部署一个hello word

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

➜  llc-blog-docker cat app.py 
from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'i am in blog http://blog.liuliancao.com'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3333)

➜  llc-blog-docker cat requirements.txt 
flask
➜  llc-blog-docker cat Dockerfile 
# Dockerfile
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "app.py"]
➜  llc-blog-docker docker build -t llc-blog-image .            
[+] Building 22.5s (10/10) FINISHED                                                                                                            docker:default
 => [internal] load build definition from Dockerfile                                                                                                     0.3s
 => => transferring dockerfile: 202B                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/python:3.11-slim                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                        0.3s
 => => transferring context: 2B                                                                                                                          0.0s
 => [internal] load build context                                                                                                                        0.8s
 => => transferring context: 488B                                                                                                                        0.1s
 => [1/5] FROM docker.io/library/python:3.11-slim                                                                                                        2.6s
 => [2/5] WORKDIR /app                                                                                                                                   1.3s
 => [3/5] COPY requirements.txt .                                                                                                                        1.5s
 => [4/5] RUN pip install --no-cache-dir -r requirements.txt                                                                                             9.5s
 => [5/5] COPY . .                                                                                                                                       1.8s 
 => exporting to image                                                                                                                                   2.8s 
 => => exporting layers                                                                                                                                  2.1s 
 => => writing image sha256:de81b2bfaedadb86fe1b1e12bfa4feeeb38f296f99ab197aada8ad2d7a067cf5                                                             0.1s 
 => => naming to docker.io/library/llc-blog-image

备注下 如果你docker拉不下来就是百度或者google下2024年11月可用docker镜 像加速站点，然后选一个放到你的 /etc/docker/daemon.json ，类似这样，然后 systemctl restart docker ,再 docker info|grep -i mirror -C3 看看 registry，比如这个是我找的

1
2
3
4
5
6
7

➜  llc-blog-docker cat /etc/docker/daemon.json
{
   #...
   "registry-mirrors": [
        "https://docker.registry.cyou"
   ]
}

这个时候标记下版本

1
2
3
4
5
6
7

➜  llc-blog-docker docker tag llc-blog-image liuliancao/llc-blog-image:v1
➜  llc-blog-docker docker images|grep llc-blog         
llc-blog-image                latest      de81b2bfaeda   2 hours ago     145MB
liuliancao/llc-blog-image     v1.0        de81b2bfaeda   2 hours ago     145MB
➜  llc-blog-docker docker images|grep llc-blog         
llc-blog-image                latest      de81b2bfaeda   2 hours ago     145MB
liuliancao/llc-blog-image     v1.0        de81b2bfaeda   2 hours ago     145MB

创建下deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

➜  llc-blog kubectl apply -f deployment.yaml     
deployment.apps/llc-frontend created
➜  llc-blog cat deployment.yaml 
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: llc-frontend
  namespace: kustomize-blog-pre
  labels:
    app: blog
spec:
  selector:
    matchLabels:
      app: blog
  replicas: 1
  template:
    metadata:
      labels:
        app: blog
    spec:
      containers:
      - name: flask-hello
        image: liuliancao/llc-blog-image:v1.0
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: BLOG_WEBSITE
          value: https://blog.liuliancao.com
        ports:
        - containerPort: 3333

测试kustomize简单使用

经过上面操作，我们已经有个service.yaml和deployments.yaml了 参考 https://kubectl.docs.kubernetes.io/guides/introduction/kustomize/

https://www.qikqiak.com/post/kustomize-101/

https://kubernetes.io/zh-cn/docs/tasks/manage-kubernetes-objects/kustomization/

初步使用

需要编写下kustomization.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

➜  kustomize tree llc-blog
llc-blog
├── deployment.yaml
└── service.yaml

1 directory, 2 files

➜  kustomize cat llc-blog/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - service.yaml
  - deployment.yaml

然后就可以kubectl apply -k的方式来更新kustomization

可以看到最基础的kustomize是把所有的文件拼接起来，这个我们可以通过 kustomize build 看出来

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

➜  kustomize kustomize build llc-blog       
apiVersion: v1
kind: Service
metadata:
  name: llc-hello-svc
  namespace: kustomize-blog-pre
spec:
  ports:
  - port: 3333
    targetPort: 3333
  selector:
    app: blog
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: blog
  name: llc-frontend
  namespace: kustomize-blog-pre
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blog
  template:
    metadata:
      labels:
        app: blog
    spec:
      containers:
      - env:
        - name: BLOG_WEBSITE
          value: https://blog.liuliancao.com
        image: liuliancao/llc-blog-image:v1
        name: flask-hello
        ports:
        - containerPort: 3333
        resources:
          requests:
            cpu: 100m
            memory: 100Mi

在实际使用中，我们可能有自定义的需求 一般来说，我们会放基础的部分，也会放一些环境的部分，这个用过terraform 应 该了解，类似ansible的层层变量也比较类似

这里稍微调整了下，就可以支持多个环境了，只是里面的东西都是写死的

目录结构和使用

首先看看kustomization.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

➜  kustomize cat llc-blog/base/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

➜  kustomize cat llc-blog/overlays/prod/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

这样就复用了老的yaml 所以base里面的相当于是模板

引入变量

在原来的deployments继续修改一个能标记到的资源，然后添加一个环境变量

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

➜  kustomize cat llc-blog/overlays/prod/custom-env.yaml 
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: llc-frontend
  namespace: kustomize-blog-prod
spec:
  template:
    spec:
      containers:
      - name: flask-hello
        env:
        - name: BLOG_AUTHOR
          value: liuliancao

这个时候的kustomization.yaml内容

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

➜  kustomize cat llc-blog/overlays/prod/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

patches:
- path: custom-env.yaml
➜  kustomize

我们通过patches引入一个patch,告诉在base的基础上面刚刚的diff,也就是增加 一个环境变量，所以类似拼接，我们也可以拼接类似image等，在base不写就好 了，在patch里面引入

这个时候继续build发现

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

➜  kustomize kustomize build llc-blog/overlays/prod 
apiVersion: v1
kind: Service
metadata:
  name: llc-hello-svc
  namespace: kustomize-blog-prod
spec:
  ports:
  - port: 3333
    targetPort: 3333
  selector:
    app: blog
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: blog
  name: llc-frontend
  namespace: kustomize-blog-prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blog
  template:
    metadata:
      labels:
        app: blog
    spec:
      containers:
      - env:
        - name: BLOG_AUTHOR
          value: liuliancao
        - name: BLOG_WEBSITE
          value: https://blog.liuliancao.com
        image: liuliancao/llc-blog-image:v1
        name: flask-hello
        ports:
        - containerPort: 3333
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
特性列表

其实官网里面已经说了，Kustomize可以有这些特性，这些特性有的是小工具， 有的是和模板相关的，比如namespace,里面的所有资源都是这个namespace，这 样就不用定义一个namespace变量了

字段 类型 解释 namespace string 为所有资源添加名字空间 namePrefix string 此字段的值将被添加到所有资源名称前面 nameSuffix string 此字段的值将被添加到所有资源名称后面 commonLabels map[string]string 要添加到所有资源和选择算符的标签 commonAnnotations map[string]string 要添加到所有资源的注解 resources []string 列表中的每个条目都必须能够解析为现有的资源配置文件 configMapGenerator []ConfigMapArgs 列表中的每个条目都会生成一个 ConfigMap secretGenerator []SecretArgs 列表中的每个条目都会生成一个 Secret generatorOptions GeneratorOptions 更改所有 ConfigMap 和 Secret 生成器的行为 bases []string 列表中每个条目都应能解析为一个包含 kustomization.yaml 文件的目录 patchesStrategicMerge []string 列表中每个条目都能解析为某 Kubernetes 对象的策略性合并补丁 patchesJson6902 []Patch 列表中每个条目都能解析为一个 Kubernetes 对象和一个 JSON 补丁 vars []Var 每个条目用来从某资源的字段来析取文字 images []Image 每个条目都用来更改镜像的名称、标记与/或摘要，不必生成补丁 configurations []string 列表中每个条目都应能解析为一个包含 Kustomize 转换器配置 的文件 crds []string 列表中每个条目都应能够解析为 Kubernetes 类别的 OpenAPI 定义文件

官方的examples

https://github.com/kubernetes-sigs/kustomize/blob/master/examples/zh/README.md

示例：早餐配置

https://github.com/kubernetes-sigs/kustomize/blob/master/examples/zh/breakfast.md

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

➜  kustomize DEMO_HOME=$(mktemp -d)
# 创建模板目录
➜  kustomize mkdir -p $DEMO_HOME/breakfast/base
# 模板里面增加kustomize的定义文件，要求包含当前目录下的coffee.yaml和pancakes.yaml
➜  kustomize cat <<EOF >$DEMO_HOME/breakfast/base/kustomization.yaml
resources:
- coffee.yaml
- pancakes.yaml
EOF

# 定义coffee怎么实现的
➜  kustomize cat <<EOF >$DEMO_HOME/breakfast/base/coffee.yaml
kind: Coffee
metadata:
  name: morningCup
temperature: lukewarm
data:
  greeting: "Good Morning!"
EOF

# 定义pancake实现
➜  kustomize cat <<EOF >$DEMO_HOME/breakfast/base/pancakes.yaml
kind: Pancakes
metadata:
  name: comfort
stacksize: 3
topping: none
EOF

# 为喜欢热咖啡的 Alice 定制她的早餐 这里相当于是早餐的一个实现,比如什么样的咖啡，什么样的煎饼

# 新增alice这个用户或者个体
➜  kustomize mkdir -p $DEMO_HOME/breakfast/overlays/alice
# 增加alice这个用户的自定义需求，通过patches,并且在resources引入了base文件夹，引入了区分的标签
➜  kustomize cat <<EOF >$DEMO_HOME/breakfast/overlays/alice/kustomization.yaml
commonLabels:
  who: alice
resources:
- ../../base
patches:
- path: temperature.yaml
EOF
# 定义需要自定义的具体内容，比如温度，咖啡的名称等等
➜  kustomize cat <<EOF >$DEMO_HOME/breakfast/overlays/alice/temperature.yaml
kind: Coffee
metadata:
  name: morningCup
temperature: hot!
EOF

# 继续增加bob的，5 块薄煎饼和草莓
➜  kustomize mkdir -p $DEMO_HOME/breakfast/overlays/bob

cat <<EOF >$DEMO_HOME/breakfast/overlays/bob/kustomization.yaml
commonLabels:
  who: bob
resources:
- ../../base
patches:
- path: topping.yaml
EOF

cat <<EOF >$DEMO_HOME/breakfast/overlays/bob/topping.yaml
kind: Pancakes
metadata:
  name: comfort
stacksize: 5
topping: strawberries
EOF

现在尝试为alice和bob生成早餐

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

➜  kustomize kustomize build $DEMO_HOME/breakfast/overlays/alice
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
data:
  greeting: Good Morning!
kind: Coffee
metadata:
  labels:
    who: alice
  name: morningCup
temperature: hot!
---
kind: Pancakes
metadata:
  labels:
    who: alice
  name: comfort
stacksize: 3
topping: none

这里提示commonLabels过期了，那我们执行下kustomize edit fix看看

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# 切到alice的目录
➜  kustomize cd $DEMO_HOME/breakfast/overlays/alice
# 执行下fix
➜  alice kustomize edit fix --vars commonLabels

Fixed fields:
  patchesJson6902 -> patches
  patchesStrategicMerge -> patches
  commonLabels -> labels
➜  alice cd $DEMO_HOME/breakfast/overlays/bob 
# 把bob也修复下
➜  bob kustomize edit fix --vars commonLabels
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.

Fixed fields:
  patchesJson6902 -> patches
  patchesStrategicMerge -> patches
  commonLabels -> labels
  vars -> replacements
➜  bob less kustomization.yaml

再次build，可以发现kustomize模板化生成了一个k8s的资源清单，只需要传给 kubectl apply即可

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

➜  alice kustomize build $DEMO_HOME/breakfast/overlays/alice
data:
  greeting: Good Morning!
kind: Coffee
metadata:
  labels:
    who: alice
  name: morningCup
temperature: hot!
---
kind: Pancakes
metadata:
  labels:
    who: alice
  name: comfort
stacksize: 3
topping: none

build下bob的

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

➜  bob kustomize build $DEMO_HOME/breakfast/overlays/bob 
data:
  greeting: Good Morning!
kind: Coffee
metadata:
  labels:
    who: bob
  name: morningCup
temperature: lukewarm
---
kind: Pancakes
metadata:
  labels:
    who: bob
  name: comfort
stacksize: 5
topping: strawberries

参考文档