K8s部署ret2shell

1. 背景#

前段时间看到信安协会的同学去打了西电的MoeCTF，感觉他们的哪个平台（Ret2shell）在颜值和功能上都挺不错的，于是想把现在使用的 GZCTF 升级成 Ret2shell

2. 部署环境#

目前 GZCTF平台部署在一个三节点的K8S集群上，由于学校局域网对外端口封禁非常严重，于是使用MetalLB提供的LoadBlancer对上层的反向代理服务器进行暴露。

系统：Debian GNU/Linux 12 (bookworm)
集群：K8s v1.25.0 + Helm v3.11.2
存储：使用宿主机目录
镜像仓库：使用外部的Harbor作为镜像仓库

3. 部署#

3.1 创建命名空间#

1
apiVersion: v1
2
kind: Namespace
3
metadata:
4
  name: ret2shell-platform
5
  labels:
6
    app.kubernetes.io/name: ret2shell
7
    app.kubernetes.io/part-of: ret2shell

3.2 给节点打标签#

1
# 负责运行平台的节点
2
kubectl label node master1 ret.sh.cn/workload=platform
3

4
# 负责运行题目容器的节点
5
kubectl label node master2 ret.sh.cn/workload=challenge
6
kubectl label node master3 ret.sh.cn/workload=challenge

3.3 创建PV#

Important

因为集群的空间吃紧，所以我在master1节点新加了一块硬盘并挂载到了 /opt/ret2shell-data ，这里手动创建PV和PVC并在 values.yaml 里面手动引用

创建 nats 的 PV

1
apiVersion: v1
2
kind: PersistentVolume
3
metadata:
4
  name: ret2shell-nats-data
5
  labels:
6
    app.kubernetes.io/name: ret2shell
7
    app.kubernetes.io/component: nats
8
spec:
9
  capacity:
10
    storage: 1Gi
11
  accessModes:
12
    - ReadWriteOnce
13
  persistentVolumeReclaimPolicy: Retain
14
  storageClassName: ret2shell-local
15
  hostPath:
16
    path: /opt/ret2shell-data/nats
17
    type: DirectoryOrCreate

这个PV用于nats的JetStream消息持久化，分配1G

创建 postgres 的 PV

1
apiVersion: v1
2
kind: PersistentVolume
3
metadata:
4
  name: ret2shell-postgresql-data
5
  labels:
6
    app.kubernetes.io/name: ret2shell
7
    app.kubernetes.io/component: postgresql
8
spec:
9
  capacity:
10
    storage: 15Gi
11
  accessModes:
12
    - ReadWriteOnce
13
  persistentVolumeReclaimPolicy: Retain
14
  storageClassName: ret2shell-local
15
  hostPath:
16
    path: /opt/ret2shell-data/postgresql
17
    type: DirectoryOrCreate

这个PV用于平台数据库的数据存储，考虑到需要长期使用，分配15G

创建 valkey 的 PV

1
apiVersion: v1
2
kind: PersistentVolume
3
metadata:
4
  name: ret2shell-valkey-data
5
  labels:
6
    app.kubernetes.io/name: ret2shell
7
    app.kubernetes.io/component: valkey
8
spec:
9
  capacity:
10
    storage: 1Gi
11
  accessModes:
12
    - ReadWriteOnce
13
  persistentVolumeReclaimPolicy: Retain
14
  storageClassName: ret2shell-local
15
  hostPath:
16
    path: /opt/ret2shell-data/valkey
17
    type: DirectoryOrCreate

这个是键值对数据库的数据持久化PV，分配1G

创建 victoria 的 PV

1
apiVersion: v1
2
kind: PersistentVolume
3
metadata:
4
  name: ret2shell-victoria-logs-data
5
  labels:
6
    app.kubernetes.io/name: ret2shell
7
    app.kubernetes.io/component: victoria-logs
8
spec:
9
  capacity:
10
    storage: 3Gi
11
  accessModes:
12
    - ReadWriteOnce
13
  persistentVolumeReclaimPolicy: Retain
14
  storageClassName: ret2shell-local
15
  hostPath:
16
    path: /opt/ret2shell-data/victoria-logs
17
    type: DirectoryOrCreate

这个PV用于平台日志存储，分配3G

创建 platform 的 PV

1
apiVersion: v1
2
kind: PersistentVolume
3
metadata:
4
  name: ret2shell-platform-data
5
  labels:
6
    app.kubernetes.io/name: ret2shell
7
    app.kubernetes.io/component: platform
8
spec:
9
  capacity:
10
    storage: 70Gi
11
  accessModes:
12
    - ReadWriteOnce
13
  persistentVolumeReclaimPolicy: Retain
14
  storageClassName: ret2shell-local
15
  hostPath:
16
    path: /opt/ret2shell-data/platform
17
    type: DirectoryOrCreate

这个PV存储bucket 仓库、附件、媒体文件、题目环境捕获文件、平台本地日志，分配70G

3.4 创建对应的PVC#

创建 nats 的 PVC

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: ret2shell-nats-data
5
  namespace: ret2shell-platform
6
  labels:
7
    app.kubernetes.io/name: ret2shell
8
    app.kubernetes.io/component: nats
9
spec:
10
  accessModes:
11
    - ReadWriteOnce
12
  storageClassName: ret2shell-local
13
  volumeName: ret2shell-nats-data
14
  resources:
15
    requests:
16
      storage: 1Gi

创建 postgres 的 PVC

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: ret2shell-postgresql-data
5
  namespace: ret2shell-platform
6
  labels:
7
    app.kubernetes.io/name: ret2shell
8
    app.kubernetes.io/component: postgresql
9
spec:
10
  accessModes:
11
    - ReadWriteOnce
12
  storageClassName: ret2shell-local
13
  volumeName: ret2shell-postgresql-data
14
  resources:
15
    requests:
16
      storage: 10Gi

创建 valkey 的 PVC

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: ret2shell-valkey-data
5
  namespace: ret2shell-platform
6
  labels:
7
    app.kubernetes.io/name: ret2shell
8
    app.kubernetes.io/component: valkey
9
spec:
10
  accessModes:
11
    - ReadWriteOnce
12
  storageClassName: ret2shell-local
13
  volumeName: ret2shell-valkey-data
14
  resources:
15
    requests:
16
      storage: 1Gi

创建 victoria 的 PVC

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: ret2shell-victoria-logs-data
5
  namespace: ret2shell-platform
6
  labels:
7
    app.kubernetes.io/name: ret2shell
8
    app.kubernetes.io/component: victoria-logs
9
spec:
10
  accessModes:
11
    - ReadWriteOnce
12
  storageClassName: ret2shell-local
13
  volumeName: ret2shell-victoria-logs-data
14
  resources:
15
    requests:
16
      storage: 3Gi

创建 platform 的 PVC

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: ret2shell-platform-data
5
  namespace: ret2shell-platform
6
  labels:
7
    app.kubernetes.io/name: ret2shell
8
    app.kubernetes.io/component: platform
9
spec:
10
  accessModes:
11
    - ReadWriteOnce
12
  storageClassName: ret2shell-local
13
  volumeName: ret2shell-platform-data
14
  resources:
15
    requests:
16
      storage: 25Gi

3.5 使用 Helm 进行安装#

编写 values.yaml

1
global:
2
  labels: {}
3
platform:
4
  image:
5
    repository: ghcr.io/ret2shell/ret2shell
6
    tag: 3.12.2
7
  exposure:
8
    type: nodePort
9
  service:
10
    nodePort: 30307
11
  config:
12
    auth:
13
      signingKey: "strong-password"
14
    server:
15
      externalDomain: ctf.example.com
16
      externalHttps: true
17
      corsOrigins: "https://ctf.example.com"
18
    cluster:
19
      enabled: true
20
      nodeSelector: "challenge"
21
      enableCapture: true
22
  nodeSelector:
23
    ret.sh.cn/workload: platform
24
  persistence:
25
    enabled: true
26
    accessMode: ReadWriteOnce
27
    existingClaim: "ret2shell-platform-data"
28

29
postgresql:
30
  mode: internal
31
  image:
32
    repository: docker.jiaxin.site/library/postgres
33
    tag: 18-alpine
34
  auth:
35
    password: "strong-password"
36
  nodeSelector:
37
    ret.sh.cn/workload: platform
38
  persistence:
39
    enabled: true
40
    accessMode: ReadWriteOnce
41
    existingClaim: "ret2shell-postgresql-data"
42

43
valkey:
44
  mode: internal
45
  image:
46
    repository: docker.jiaxin.site/valkey/valkey
47
    tag: 8-alpine
48
  auth:
49
    enabled: true
50
    password: "strong-password"
51
  nodeSelector:
52
    ret.sh.cn/workload: platform
53
  persistence:
54
    enabled: true
55
    accessMode: ReadWriteOnce
56
    existingClaim: "ret2shell-valkey-data"
57

58
nats:
59
  mode: internal
60
  image:
61
    repository: docker.jiaxin.site/library/nats
62
    tag: 2-alpine
63
  auth:
64
    enabled: true
65
    token: "strong-password"
66
  nodeSelector:
67
    ret.sh.cn/workload: platform
68
  persistence:
69
    enabled: true
70
    accessMode: ReadWriteOnce
71
    existingClaim: "ret2shell-nats-data"
72

73
registry:
74
  mode: disabled
75

76
victoriaLogs:
77
  mode: internal
78
  image:
79
    repository: docker.jiaxin.site/victoriametrics/victoria-logs
80
    tag: v1.43.1
81
  nodeSelector:
82
    ret.sh.cn/workload: platform
83
  persistence:
84
    enabled: true
85
    accessMode: ReadWriteOnce
86
    existingClaim: "ret2shell-victoria-logs-data"

Important

使用 existingClaim 字段引用刚刚创建的PVC

我们使用外部的Harbor作为比赛使用的镜像仓库，所以不使用ret2shell部署的受控registry，将 registry.mode 设置为 disabled

开始部署

1
# 安装jq
2
sudo apt install -y jq
3

4
# 创建配置文件夹
5
mkdir -p ~/ret2shell-deploy
6
cd ~/ret2shell-deploy
7

8
# 获取最新版本
9
R2S_VERSION="$(curl -fsSL https://api.github.com/repos/ret2shell/ret2shell/releases/latest | jq -r '.tag_name')"
10

11
# 下载最新版本的Chart和签名
12
curl -fLO "https://github.com/ret2shell/ret2shell/releases/download/${R2S_VERSION}/helm-chart-${R2S_VERSION}.tgz"
13
curl -fLO "https://github.com/ret2shell/ret2shell/releases/download/${R2S_VERSION}/helm-chart-${R2S_VERSION}.tgz.sha256"
14

15
# 校验一下sha256
16
EXPECTED_SHA256="$(awk '{print $1}' "helm-chart-${R2S_VERSION}.tgz.sha256")"
17
ACTUAL_SHA256="$(sha256sum "helm-chart-${R2S_VERSION}.tgz" | awk '{print $1}')"
18
test "$EXPECTED_SHA256" = "$ACTUAL_SHA256"
19

20
# 安装
21
helm upgrade --install ret2shell helm-chart-3.12.2.tgz -n ret2shell-platform -f values.yaml

4. 暴露服务#

4.1 使用MetalLB#

创建IP地址池

1
apiVersion: metallb.io/v1beta1
2
kind: IPAddressPool
3
metadata:
4
  name: ret2shell-ips
5
  namespace: metallb-system
6
spec:
7
  addresses:
8
    - 192.168.1.119/32
9
  autoAssign: false

创建负载均衡的 SVC

1
apiVersion: v1
2
kind: Service
3
metadata:
4
  name: ret2shell-loadblancer
5
  namespace: ret2shell-platform
6
  labels:
7
    app.kubernetes.io/name: ret2shell
8
    app.kubernetes.io/component: platform
9
  annotations:
10
    metallb.universe.tf/address-pool: ret2shell-ips
11
spec:
12
  type: LoadBalancer
13
  ports:
14
    - name: http
15
      port: 80
16
      targetPort: http
17
      protocol: TCP
18
  selector:
19
    app.kubernetes.io/name: ret2shell
20
    app.kubernetes.io/component: platform

4.2 配置反向代理#

1
server {
2
    include common.d/listen.conf;
3
    include ssl/main.conf;
4
    server_name ctf.example.com;
5

6
    location / {
7
      proxy_pass http://192.168.1.119;
8
        include common.d/proxy_pass.conf;
9
    }
10
}

5. 添加镜像仓库的凭据#

1
kubectl create secret docker-registry harbor-secret \
2
  --docker-server=<你的仓库地址> \
3
  --docker-username=<用户名> \
4
  --docker-password=<密码或访问令牌> \
5
  --namespace=ret2shell-challenge

Important

ret2shell拉取镜像时使用的凭据是 docker-registry 形式的，需要提前在集群内配置，在创建题目时设置，启动题目环境时平台会自动使用该凭据拉取Harbor内的镜像

6. 结尾#

没了喵，谢谢欣赏

音乐

1. 背景#

2. 部署环境#

3. 部署#

3.1 创建命名空间#

3.2 给节点打标签#

3.3 创建PV#

3.4 创建对应的PVC#

3.5 使用 Helm 进行安装#

4. 暴露服务#

4.1 使用MetalLB#

4.2 配置反向代理#

5. 添加镜像仓库的凭据#

6. 结尾#

文章分享

评论区

目录