DevOps实战：用Kubernetes和Argo打造自动化CI/CD流程（2）

DevOps实战：用Kubernetes和Argo打造自动化CI/CD流程（2）

背景

Tips

翻遍国内外的文档，关于 Argo 作为 CI/CD 当前所有开源的文档，博客，argo官方文档。得出的结论是： argo官方给出的例子都相对独立，要么国内是一些搬运 argo 的官方的简单例子，根本没有一套可以用于实际工作中完整的流程。整套 Kubernetes和Argo打造自动化CI/CD流程 都是根据我的个人实践，一步一步摸索出来的。

架构图

正片开始之前，请一定先熟悉上面的架构图，跟着我的步骤，一步一步执行成功，相信后续根据自己特定的需求定制CI/CD。

需求

用户更新代码，提交commit 到 master branch，DevOps Argo自动进行服务的测试，构建，更新服务。

正片开始

文件目录

GitHub - jackwillsmith/go-gin

.
|-- Dockerfile
|-- Dockerfile-bk
|-- Makefile
|-- ab_test.md
|-- docker-compose.yaml
|-- go.mod
|-- go.sum
|-- install_argo.sh
|-- main.go                                   # 程序入口
|-- main_test.go                              # 单元测试文件
|-- manifest
|   |-- argo-events-clusterrolebinding.yaml   # argo-events sa default
|   |-- argo-workflow-clusterrole.yaml        # argo clusterrole
|   |-- argo-workflow-clusterrolebinding.yaml # argo clusterrolebinding
|   |-- github-eventsources.yaml              # github eventsource
|   |-- github-sensor.yaml                    # github webhook
|   |-- go-gin-deployment-workflow.yaml       # go-gin workflow
|   |-- mani.yaml                             # go-gin deployments,service
|-- readme.md

go-gin manifest都创建在 argo-events namespace下

ArgoCD

1. 登录argocd UI

root@master:/home/eilinge/argo-cd# kubectl -n argocd get svc
NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
argocd-server                             NodePort    10.43.238.233   <none>        80:30878/TCP,443:32063/ TCP   11d # ClusterIP -> NodePort# 获取argocd admin 密码
root@master:/home/eilinge/argo-cd# kubectl -n argocd get secret argocd-initial-admin-secret --output=jsonpath={.data.password} |base64 -d

2. 创建go-gin的deployment,service

创建成功

等待同步。点击进入详情

root@master:/home/eilinge# kubectl -n argo-events get all|grep go-gin
pod/go-gin-577b868bd6-79cf7                      1/1     Running     0               24hservice/go-gin                      ClusterIP   10.43.245.228   <none>        8080/TCP                     46hdeployment.apps/go-gin                      1/1     1            1           46hreplicaset.apps/go-gin-577b868bd6                      1         1         1       24h

Argo Workflow

部署Argo Workflow

DevOps实战：用Kubernetes和Argo打造自动化CI/CD流程（1）-CSDN博客

root@master:/home/eilinge/argo-cd# kubectl -n argo get all
NAME                                      READY   STATUS    RESTARTS        AGE
pod/argo-server-67bfcbc559-bxqwd          1/1     Running   3 (2d4h ago)    9d
pod/workflow-controller-b84cc4f5b-fg5ss   1/1     Running   9 (3h43m ago)   30hNAME                  TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/argo-server   NodePort   10.43.242.65   <none>        2746:30865/TCP   9dNAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argo-server           1/1     1            1           9d
deployment.apps/workflow-controller   1/1     1            1           9dNAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/argo-server-58f9864f85          0         0         0       9d
replicaset.apps/argo-server-67bfcbc559          1         1         1       9d
replicaset.apps/argo-server-b99696f87           0         0         0       9d
replicaset.apps/workflow-controller-b84cc4f5b   1         1         1       9d

登录argo workflow UI

第一次登录时，需要进行token认证。 Access Token - Argo Workflows - The workflow engine for Kubernetes

go-gin-workflow.yaml

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:name: buildkit
spec:arguments:parameters:- name: repovalue: https://github.com/jackwillsmith/go-gin.git- name: branchvalue: master- name: pathvalue: .- name: imagevalue: eilinge/go-gin:v1.2- name: servernamevalue: go-gin- name: namespacevalue: argo-events- name: portvalue: 8080entrypoint: main# We use a volume claim template so that we can have a shared workspace.volumeClaimTemplates:- metadata:name: workspec:accessModes: [ "ReadWriteOnce" ]resources:requests:storage: 64Mitemplates:- name: maindag:tasks:  # 部署的流程- name: clone     # 1. clone 从远程仓库下载到本地template: clonearguments:parameters:- name: repovalue: "{{workflow.parameters.repo}}"- name: branchvalue: "{{workflow.parameters.branch}}"- name: gotest     # 2. gotest 执行go test,进行单元测试template: gotestarguments:parameters:- name: pathvalue: "{{workflow.parameters.path}}"depends: "clone"- name: build       # 3. 在pod中构建go build -o 可执行文件template: buildarguments:parameters:- name: pathvalue: "{{workflow.parameters.path}}"depends: "gotest"- name: image       # 4. 在pod中构建 imagetemplate: imagearguments:parameters:- name: pathvalue: "{{workflow.parameters.path}}"- name: imagevalue: "{{workflow.parameters.image}}"depends: "build"- name: workload    # 5. 更新go-gin deployment服务template: go-gin-serverarguments:parameters:- name: servernamevalue: "{{workflow.parameters.servername}}"- name: namespacevalue: "{{workflow.parameters.namespace}}"- name: imagevalue: "{{workflow.parameters.image}}"depends: "image"- name: cloneinputs:parameters:- name: repo- name: branchcontainer:volumeMounts:- mountPath: /workname: workimage: docker.m.daocloud.io/alpine/git:v2.26.2workingDir: /work         # 不同task 之间通过/work 目录进行传递文件# Do a shallow clone, which is the fastest way to clone, by using the# --depth, --branch, and --single-branch optionsargs:- clone- --depth               # 根据具体项目进行调整- "1"- --branch- "{{inputs.parameters.branch}}"- --single-branch- "{{inputs.parameters.repo}}"- .- name: gotestinputs:parameters:- name: pathcontainer:image: golang:1.22.5volumeMounts:- mountPath: /workname: workworkingDir: /work/{{inputs.parameters.path}}env:                       # golang容器中执行 go test -v ./...# Because this is not a Gomodule, we must turn modules off.- name: GO111MODULEvalue: "on"- name: CGO_ENABLEDvalue: "0"- name: GOPROXYvalue: "https://goproxy.cn,direct"command:- goargs:- test- -v- ./...- name: buildinputs:parameters:- name: pathcontainer:image: golang:1.22.5volumeMounts:- mountPath: /workname: workworkingDir: /work/{{inputs.parameters.path}}env:                      # golang容器中执行 go build -o# Because this is not a Gomodule, we must turn modules off.- name: GO111MODULEvalue: "on"- name: CGO_ENABLEDvalue: "0"- name: GOPROXYvalue: "https://goproxy.cn,direct"command:- goargs:- build- -v- -o- /work/out/app       # golang main.go可执行文件- name: imageinputs:parameters:- name: path- name: image# Mount the configuration so we can push the image.# This should create the /.docker/config.json file.volumes:- name: buildkitd-sockethostPath:path: /run/buildkit/buildkitd.sock # 需要将k3s节点的builkitd.sock 挂载到容器中type: Socketcontainer:readinessProbe:exec:command: [ sh, -c, "buildctl debug workers" ]image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/moby/buildkit:latestvolumeMounts:- name: workmountPath: /work- name: buildkitd-socketmountPath: /run/buildkit/buildkitd.sock   # 构建image的buildkitd.sockworkingDir: /work/{{inputs.parameters.path}}env:- name: BUILDKITD_FLAGSvalue: --oci-worker-no-process-sandboxcommand:- buildctl-daemonless.sh  # 可进入容器，查看详情 相当于执行 docker buildargs:- build- --frontend- dockerfile.v0- --local- context=.- --local- dockerfile=.- --output- type=image,name=docker.io/{{inputs.parameters.image}},push=false- name: go-gin-serverdaemon: trueinputs:parameters:- name: servername- name: namespace- name: imageresource:action: patch            # 这里通过patch 修改argocd创建的deployment资源，而不是createmanifest: |apiVersion: apps/v1kind: Deploymentmetadata:name: {{inputs.parameters.servername}}namespace: {{inputs.parameters.namespace}}spec:template:metadata:creationTimestamp: "{{workflow.creationTimestamp}}"  # 只修改创建时间即可，可以使最新image生效spec:containers:- image: "{{inputs.parameters.image}}"name: "{{inputs.parameters.servername}}" # 执行pod中具体container

创建workflow

argo-events

部署 github-eventsource

kubectl -n argo-events apply -f github-eventsources.yaml

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:name: github
spec:service:                            # 创建service :12000ports:- name: exampleport: 12000targetPort: 12000github:example:repositories:                   # 关联github 仓库- owner: jackwillsmithnames:- go-ginwebhook:                              # 监听 :12000/push 路由# endpoint to listen to events onendpoint: /push# port to run internal HTTP server onport: "12000"# HTTP request method to allow. In this case, only POST requests are acceptedmethod: POSTevents:                               # 监听 events:push- "push"# type of the connection between event-source and Github.# You should set it to false to avoid man-in-the-middle and other attacks.insecure: true# Determines if notifications are sent when the webhook is triggeredactive: true# The media type used to serialize the payloadscontentType: json

root@master:/home/eilinge/argo-cd# kubectl -n argo-events get all |grep github-eventsource
pod/github-eventsource-d6zmx-665c64c5c8-59svh    1/1     Running     0                30hservice/github-eventsource-svc      NodePort    10.43.229.201   <none>        12000:31906/TCP              5d1hdeployment.apps/github-eventsource-d6zmx    1/1     1            1           5d1hreplicaset.apps/github-eventsource-d6zmx-665c64c5c8    1         1         1       5d1h
ch

创建Sensor

部署github-sensor

kubectl -n argo-events apply -f github-sensor.yaml

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:name: github
spec:template:serviceAccountName: operate-workflow-sadependencies:- name: test-depeventSourceName: githubeventName: examplefilters:data:# Type of Github event that triggered the delivery: [pull, push, issues, label, test,...]# https://docs.github.com/en/developers/webhooks-and-events/webhook-events-and-payloads- path: headers.X-Github-Event   # 定义监听 webhook event pushtype: stringvalue:- push- path: body.ref                 # 定义github go-gin master branchtype: stringvalue:- master- "refs/heads/master"triggers:- template:name: github-workflow-triggerargoWorkflow:operation: resubmit  # resubmit argo workflowsource:resource:apiVersion: argoproj.io/v1alpha1kind: Workflowmetadata:name: buildkit # workflow name exists in argo workflowretryStrategy:steps: 3

root@master:/home/eilinge/argo-cd# kubectl -n argo-events get all |grep github-sensor
pod/github-sensor-jwwvn-654f5d584-p9cvz          1/1     Running     0                25h
deployment.apps/github-sensor-jwwvn         1/1     1            1           28h
replicaset.apps/github-sensor-jwwvn-654f5d584          1         1         1       25h

github go-gin项目创建webhook

由于作者是在自己电脑的虚拟机中，部署的k3s节点，github无法直接进行访问，需要内网穿透才能在公网进行访问。可以通过Frp服务实现。

开发个人Ollama-Chat–9 Frp穿透_ollama api frps-CSDN博客

测试

经常上述的部署流程，已经将架构图中所需的资源都创建成功了，现在进行测试。

ISSUE

1. Argo Rollouts 实现蓝绿发布未写明？

蓝绿发布属于网关层，后续会更新通过专业网关服务Higress进行发布

2. Argo Workflow资源创建后，会有用户权限不足，无法操作kubernetes 资源。

解决方法放置在go-gin项目的manifest文件夹下的clusterrole.yaml, clusterrolebinding.ayml