linuxea: 使用tekton构建镜像(2) 现在我门将这些组合起来形成一个流水线，使用的 CRD 对象是 Pipeline 。假如是一个新的环境，我门仍然需要配置此前的凭据，比如git认证，镜像仓库认证，必要的配置和版本号的传递等首先配置一个使用的凭据在清单中1.配置凭据和SAapiVersion: v1 kind: Secret metadata: name: ucloud-auth annotations: tekton.dev/docker-0: http://uhub.service.ucloud.cn type: kubernetes.io/basic-auth stringData: username: username password: password --- apiVersion: v1 kind: ServiceAccount metadata: name: build-sa secrets: - name: ucloud-auth --- apiVersion: tekton.dev/v1alpha1 kind: PipelineResource metadata: name: git-res namespace: default spec: params: - name: url value: https://gitee.com/marksugar/argocd-example - name: revision value: master type: git --- apiVersion: tekton.dev/v1alpha1 kind: PipelineResource metadata: name: ucloud-image-go spec: type: image params: - name: url value: uhub.service.ucloud.cn/linuxea/golang #构建完的镜像名称 PipelineResource的ucloud-image-go中并没有写入版本号，这是因为在task会进行生成2.配置task1task1作为一个假定的测试apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: test spec: resources: inputs: - name: repo type: git steps: - name: run-test image: registry.cn-zhangjiakou.aliyuncs.com/marksugar-k8s/golang:1.18.10-alpine3.17 workingDir: /workspace/repo script: | #!/usr/bin/env sh cd tekton/go && go test3.生成版本号我门需要提供一个版本号，使用task生成一个由环境变量的传入-年月日时间的版本号apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: generate-build-id spec: description: >- Given a base version, this task generates a unique build id by appending the base-version to the current timestamp. params: - name: base-version description: Base product version type: string default: "1.0" results: - name: timestamp description: Current timestamp - name: build-id description: ID of the current build steps: - name: get-timestamp image: uhub.service.ucloud.cn/marksugar-k8s/bash:5.0.18 script: | #!/usr/bin/env bash sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories apk add -U tzdata cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime ts=`date "+%Y%m%d-%H%M%S"` echo "Current Timestamp: ${ts}" echo ${ts} | tr -d "\n" | tee $(results.timestamp.path) - name: get-buildid image: uhub.service.ucloud.cn/marksugar-k8s/bash:5.0.18 script: | #!/usr/bin/env bash ts=`cat $(results.timestamp.path)` buildId=$(inputs.params.base-version)-${ts} echo ${buildId} | tr -d "\n" | tee $(results.build-id.path)3.配置镜像构建我门需要修改构建时候的命令，以便于添加传入的版本号# task-build-push.yaml apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: build-and-push-test spec: resources: inputs: # 定义输入资源 - name: repo #输入资源，就是gitee的那个仓库 type: git outputs: # 定义输出资源 - name: linuxea # 输出镜像名字 type: image params: # 定义参数 - name: DockerfileURL #指明 dockerfile 在仓库中的哪个位置 type: string default: $(resources.inputs.repo.path)/tekton/go/Dockerfile # repo资源的路径 description: The path to the dockerfile to build - name: pathToContext #指明构建上下文的路径 type: string default: $(resources.inputs.repo.path) # repo资源的路径 description: the build context used by docker daemon - name: imageTag type: string default: "v0.2.0" description: the docker image tag steps: - name: build-and-push image: docker:stable script: | #!/usr/bin/env sh docker login uhub.service.ucloud.cn cd /workspace/repo/tekton/go docker build -t $(resources.outputs.linuxea.url):$(params.imageTag) . docker push $(resources.outputs.linuxea.url):$(params.imageTag) # 这边的参数都是在 input 和 output 中定义的 env: - name: DOCKER_HOST value: tcp://docker-dind.tekton-pipelines:23754.配置pipeline管道中，我门需要注意每个task运行的顺序是由runAfter来决定的，在build-and-push-test阶段，传入了环境变量imageTagapiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: test-pipeline spec: resources: # 为 Tasks 提供输入和输出资源声明 - name: git-res type: git - name: ucloud-image-go type: image params: - name: image-tag type: string tasks: # 添加task到流水线中 - name: get-build-id taskRef: name: generate-build-id # 引入generate-build-id task params: - name: base-version value: $(params.image-tag) # 运行应用测试 - name: test taskRef: name: test resources: inputs: - name: repo # Task 输入名称 resource: git-res # Pipeline 资源名称 # 构建并推送 Docker 镜像 - name: build-and-push-test taskRef: name: build-and-push-test runAfter: - test # 测试任务执行之后 - get-build-id resources: inputs: - name: repo # 指定输入的git仓库资源 resource: git-res outputs: # 指定输出的镜像资源 - name: linuxea resource: ucloud-image-go params: - name: imageTag value: "$(tasks.get-build-id.results.build-id)"5.pipelinerun接着在spec中引入build-sa的sa，和各个资源和管道apiVersion: tekton.dev/v1beta1 kind: PipelineRun metadata: name: test-pipelinerun spec: serviceAccountName: build-sa # 关联带有认证信息的 ServiceAccount pipelineRef: name: test-pipeline resources: - name: git-res # 指定输入的git仓库资源 resourceRef: name: git-res - name: ucloud-image-go # 指定输出的镜像资源 resourceRef: name: ucloud-image-go params: - name: image-tag # 传入版本号 value: "v0.3.0"注意，这里的image-tag的变量是传递给管道中名称为 get-build-id的task, 而这个task指向的是generate-build-id这个实际的task定义，而generate-build-id这个task在接收到image-tag的参数后返回了一个 buildId的环境变量，这个变量被获取后赋值给imageTag，而在构建镜像之前这个变量已经获取完成并且被传入到build-and-push-test的构建步骤中。6.构建查看创建完成后，通过命令查看[root@master1 pipeline]# tkn pipelinerun describe test-pipelinerun Name: test-pipelinerun Namespace: default Pipeline Ref: test-pipeline Service Account: build-sa Labels: tekton.dev/pipeline=test-pipeline Status STARTED DURATION STATUS 6 seconds ago --- Running ⏱ Timeouts Pipeline: 1h0m0s查看当前的taskrun[root@master1 pipeline]# tkn taskrun list NAME STARTED DURATION STATUS test-pipelinerun-build-and-push-test 15 minutes ago 11s Succeeded test-pipelinerun-get-build-id 15 minutes ago 9s Succeeded test-pipelinerun-test 15 minutes ago 9s Succeeded build-and-push 7 hours ago 12m13s Succeeded testrun 3 days ago 2m45s Succeeded根据name查看日志[root@master1 pipeline]# tkn taskrun logs test-pipelinerun-test [git-source-repo-gknf2] {"level":"info","ts":1686559991.0322871,"caller":"git/git.go:178","msg":"Successfully cloned https://gitee.com/marksugar/argocd-example @ d270fd8931fb059485622edb2ef1aa1209b7d42c (grafted, HEAD, origin/master) in path /workspace/repo"} [git-source-repo-gknf2] {"level":"info","ts":1686559991.0433397,"caller":"git/git.go:217","msg":"Successfully initialized and updated submodules in path /workspace/repo"} [run-test] PASS [run-test] ok test 0.002s [root@master1 pipeline]# tkn taskrun logs test-pipelinerun-get-build-id [get-timestamp] fetch https://mirrors.ustc.edu.cn/alpine/v3.18/main/x86_64/APKINDEX.tar.gz [get-timestamp] fetch https://mirrors.ustc.edu.cn/alpine/v3.18/community/x86_64/APKINDEX.tar.gz [get-timestamp] (1/1) Installing tzdata (2023c-r1) [get-timestamp] OK: 11 MiB in 19 packages [get-timestamp] Current Timestamp: 20230612-165311 [get-timestamp] 20230612-165311 [get-buildid] v0.3.0-20230612-165311 [root@master1 pipeline]# tkn taskrun logs test-pipelinerun-build-and-push-test [create-dir-linuxea-q6w6n] 2023/06/12 08:53:16 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/home/nonroot": unable to create destination directory: mkdir /home/nonroot: permission denied [git-source-repo-w5dcj] {"level":"info","ts":1686559998.4194562,"caller":"git/git.go:178","msg":"Successfully cloned https://gitee.com/marksugar/argocd-example @ d270fd8931fb059485622edb2ef1aa1209b7d42c (grafted, HEAD, origin/master) in path /workspace/repo"} [git-source-repo-w5dcj] {"level":"info","ts":1686559998.4337826,"caller":"git/git.go:217","msg":"Successfully initialized and updated submodules in path /workspace/repo"} [build-and-push] Authenticating with existing credentials... [build-and-push] WARNING! Your password will be stored unencrypted in /root/.docker/config.json. [build-and-push] Configure a credential helper to remove this warning. See [build-and-push] https://docs.docker.com/engine/reference/commandline/login/#credentials-store [build-and-push] [build-and-push] Login Succeeded [build-and-push] Sending build context to Docker daemon 5.12kB [build-and-push] Step 1/5 : FROM registry.cn-zhangjiakou.aliyuncs.com/marksugar-k8s/golang:1.18.10-alpine3.17 [build-and-push] ---> a77f45e5f987 [build-and-push] Step 2/5 : RUN mkdir /test -p [build-and-push] ---> Using cache [build-and-push] ---> 48d724b29eff [build-and-push] Step 3/5 : WORKDIR /test [build-and-push] ---> Using cache [build-and-push] ---> 9b6eda13d6c1 [build-and-push] Step 4/5 : COPY . . [build-and-push] ---> Using cache [build-and-push] ---> a5c71d579512 [build-and-push] Step 5/5 : CMD ["go test"] [build-and-push] ---> Using cache [build-and-push] ---> 2f377c99476e [build-and-push] Successfully built 2f377c99476e [build-and-push] Successfully tagged uhub.service.ucloud.cn/linuxea/golang:v0.3.0-20230612-165311 [build-and-push] The push refers to repository [uhub.service.ucloud.cn/linuxea/golang] [build-and-push] 6a7300559c98: Preparing [build-and-push] 17600296503b: Preparing [build-and-push] 4ad1c2ef216c: Preparing [build-and-push] c23db623ee98: Preparing [build-and-push] c1bfd5512d71: Preparing [build-and-push] 8e012198eea1: Preparing [build-and-push] 8e012198eea1: Waiting [build-and-push] 17600296503b: Layer already exists [build-and-push] 6a7300559c98: Layer already exists [build-and-push] 4ad1c2ef216c: Layer already exists [build-and-push] c1bfd5512d71: Layer already exists [build-and-push] c23db623ee98: Layer already exists [build-and-push] 8e012198eea1: Layer already exists [build-and-push] v0.3.0-20230612-165311: digest: sha256:b649a5f42e753e36562b574e26018301e8e839f129f92e3f87fd29f4b30734fc size: 1572 [image-digest-exporter-htxn8] {"severity":"INFO","timestamp":"2023-06-12T08:53:22.548887491Z","caller":"logging/config.go:116","message":"Successfully created the logger."} [image-digest-exporter-htxn8] {"severity":"INFO","timestamp":"2023-06-12T08:53:22.548992849Z","caller":"logging/config.go:117","message":"Logging level set to: info"} [image-digest-exporter-htxn8] {"severity":"INFO","timestamp":"2023-06-12T08:53:22.549269268Z","caller":"imagedigestexporter/main.go:59","message":"No index.json found for: linuxea","commit":"68f2a66"}此时pod构建结束[root@master1 build]# kubectl get pods -w NAME READY STATUS RESTARTS AGE build-and-push-pod 0/4 Completed 0 7h26m test-pipelinerun-build-and-push-test-pod 0/4 Completed 0 8m18s test-pipelinerun-get-build-id-pod 0/2 Completed 0 8m33s test-pipelinerun-test-pod 0/2 Completed 0 8m33s testrun-pod 0/2 Completed 0 3d7h test-pipelinerun-get-build-id-pod 0/2 Terminating 0 8m37s test-pipelinerun-test-pod 0/2 Terminating 0 8m37s test-pipelinerun-build-and-push-test-pod 0/4 Terminating 0 8m22s test-pipelinerun-get-build-id-pod 0/2 Terminating 0 8m37s test-pipelinerun-test-pod 0/2 Terminating 0 8m37s test-pipelinerun-build-and-push-test-pod 0/4 Terminating 0 8m22s test-pipelinerun-get-build-id-pod 0/2 Pending 0 0s test-pipelinerun-test-pod 0/2 Pending 0 0s test-pipelinerun-get-build-id-pod 0/2 Pending 0 0s test-pipelinerun-test-pod 0/2 Pending 0 0s test-pipelinerun-get-build-id-pod 0/2 Init:0/3 0 0s test-pipelinerun-test-pod 0/2 Init:0/4 0 0s test-pipelinerun-test-pod 0/2 Init:1/4 0 1s test-pipelinerun-get-build-id-pod 0/2 Init:1/3 0 1s test-pipelinerun-test-pod 0/2 Init:2/4 0 2s test-pipelinerun-get-build-id-pod 0/2 Init:2/3 0 2s test-pipelinerun-test-pod 0/2 Init:3/4 0 3s test-pipelinerun-get-build-id-pod 0/2 PodInitializing 0 3s test-pipelinerun-get-build-id-pod 2/2 Running 0 4s test-pipelinerun-test-pod 0/2 PodInitializing 0 4s test-pipelinerun-get-build-id-pod 2/2 Running 0 4s test-pipelinerun-test-pod 2/2 Running 0 5s test-pipelinerun-test-pod 2/2 Running 0 5s test-pipelinerun-test-pod 1/2 NotReady 0 8s test-pipelinerun-get-build-id-pod 1/2 NotReady 0 8s test-pipelinerun-test-pod 0/2 Completed 0 9s test-pipelinerun-get-build-id-pod 0/2 Completed 0 9s test-pipelinerun-build-and-push-test-pod 0/4 Pending 0 0s test-pipelinerun-build-and-push-test-pod 0/4 Pending 0 0s test-pipelinerun-build-and-push-test-pod 0/4 Init:0/3 0 0s test-pipelinerun-build-and-push-test-pod 0/4 Init:1/3 0 0s test-pipelinerun-build-and-push-test-pod 0/4 Init:2/3 0 2s test-pipelinerun-build-and-push-test-pod 0/4 PodInitializing 0 3s test-pipelinerun-build-and-push-test-pod 4/4 Running 0 4s test-pipelinerun-build-and-push-test-pod 4/4 Running 0 4s test-pipelinerun-build-and-push-test-pod 3/4 NotReady 0 5s test-pipelinerun-build-and-push-test-pod 2/4 NotReady 0 6s test-pipelinerun-build-and-push-test-pod 1/4 NotReady 0 9s test-pipelinerun-build-and-push-test-pod 0/4 Completed 0 10s任务成功[root@master1 pipeline]# tkn pipelinerun describe test-pipelinerun Name: test-pipelinerun Namespace: default Pipeline Ref: test-pipeline Service Account: build-sa Labels: tekton.dev/pipeline=test-pipeline STARTED DURATION STATUS 1 minute ago 20s Succeeded Timeouts Pipeline: 1h0m0s Params NAME VALUE ∙ image-tag v0.3.0镜像也推送完成在tekton中看到详细的细节，其中test的task1做了go test，build-and-push进行了镜像打包和推送。

linuxea:jenkins基于钉钉的构建通知(11) 在之前的几篇中，我分别介绍了基础环境的配置，skywaling+nacos的配置，nexus3的配置，围绕sonarqube的配置和构建镜像的配置。这一篇中，主要配置消息通知阅读此篇，你将了解如下列表中简单的实现方式：jenkins和gitlab触发(已实现)jenkins凭据使用(已实现)juit配置(已实现)sonarqube简单扫描(已实现)sonarqube覆盖率(已实现)打包基于java的skywalking agent(上一章已实现)sonarqube与gitlab关联 (上一章已实现)配置docker中构建docker (上一章已实现)mvn打包(上一章已实现)sonarqube简单分支扫描(上一章已实现)基于gitlab来管理kustomize的k8s配置清单 (上一章已实现)kubectl部署 (上一章已实现)kubeclt deployment的状态跟踪 (上一章已实现)钉钉消息的构建状态推送(本章实现)前面我们断断续续的将最简单的持续集成做好，在cd阶段，使用了kustomize和argocd，并且搭配了kustomize和argocd做了gitops的部分事宜，现在们在添加一个基于钉钉的构建通知我们创建一个钉钉机器人，关键字是DEVOPS我们创建一个函数，其中采用markdown语法，如下：分别需要向DingTalk传递几个行参，分别是：mdTitle 标签，这里的标签也就是我们创建的关键字: DEVOPSmdText 详细文本atUser 需要@谁atAll @所有人SedContent 通知标题函数体如下：def DingTalk(mdTitle, mdText, atAll, atUser = '' ,SedContent){ webhook = "https://oapi.dingtalk.com/robot/send?access_token=55d35d6f09f05388c1a8f7d73955cd9b7eaf4a0dd38" sh """ curl --location --request POST ${webhook} \ --header 'Content-Type: application/json' \ --data '{ "msgtype": "markdown", "markdown": { "title": "${mdTitle}", "text": "${SedContent}\n ${mdText}" }, "at": { "atMobiles": [ "${atUser}" ], "isAtAll": "${atAll}" } }' """ }而在流水线阶段添加post，如下 post { success{ script{ // ItmesName="${JOB_NAME.split('/')[-1]}" env.SedContent="构建通知" mdText = "### ✅ \n ### 发起人: ${BUILD_TRIGGER_BY} \n ### 项目: ${JOB_NAME} \n ### 标签: $IPATH \n ### 时间: ${TIMENOW_CN} \n ### 提交SHA: ${GIT_COMMIT_TAGSHA} \n ### Commit Info: ${GIT_COMMIT_DESCRIBE} \n ### By:  \n" DingTalk("DEVOPS", mdText, true, SedContent) } } failure{ script{ env.SedContent="构建通知" mdText = "### ❌ \n 发起人: ${BUILD_TRIGGER_BY} \n ### 项目: ${JOB_NAME} \n ### 标签: $IPATH \n ### 时间: ${TIMENOW_CN} \n ### 提交SHA: ${GIT_COMMIT_TAGSHA} \n ### Commit Info: ${GIT_COMMIT_DESCRIBE} \n ### By:  \n" DingTalk("DEVOPS", mdText, true, SedContent) } } }当然，现在你看到了上面的函数传递中有很多变量，这些需要我们去获取我们在任意一个阶段中的script中，并用env.声明到全局环境变量，添加如下GIT_COMMIT_DESCRIBE： 提交信息GIT_COMMIT_TAGSHA：提交的SHA值TIMENOW_CN：可阅读的时间格式 env.GIT_COMMIT_DESCRIBE = "${sh(script:'git log --oneline --no-merges|head -1', returnStdout: true)}" env.GIT_COMMIT_TAGSHA=sh(script: """cut -b -40 .git/refs/remotes/origin/master""",returnStdout: true).trim() env.TIMENOW_CN=sh(script: """date +%Y年%m月%d日%H时%M分%S秒""",returnStdout: true).trim()进行构建，一旦构建完成，将会发送一段消息到钉钉如下而最终的管道流水线试图如下：完整的流水线管道代码如下try { if ( "${onerun}" == "gitlabs"){ println("Trigger Branch: ${info_ref}") RefName="${info_ref.split("/")[-1]}" //自定义显示名称 currentBuild.displayName = "#${info_event_name}-${RefName}-${info_checkout_sha}" //自定义描述 currentBuild.description = "Trigger by user ${info_user_username} 自动触发 \n branch: ${RefName} \n commit message: ${info_commits_0_message}" BUILD_TRIGGER_BY="${info_user_username}" BASEURL="${info_project_git_http_url}" } }catch(e){ BUILD_TRIGGER_BY="${currentBuild.getBuildCauses()[0].userId}" currentBuild.description = "Trigger by user ${BUILD_TRIGGER_BY} 非自动触发 \n branch: ${branch} \ngit: ${BASEURL}" } pipeline{ //指定运行此流水线的节点 agent any environment { def tag_time = new Date().format("yyyyMMddHHmm") def IPATH="harbor.marksugar.com/java/${JOB_NAME}:${tag_time}" def kustomize_Git="git@172.16.100.47:devops/k8s-yaml.git" def JOB_NAMES=sh (script: """echo ${kustomize_Git.split("/")[-1]} | cut -d . -f 1""",returnStdout: true).trim() def Projects_Area="dev" def apps_name="java-demo" def projectGroup="java-demo" def PACK_PATH="/usr/local/package" } //管道运行选项 options { skipDefaultCheckout true skipStagesAfterUnstable() buildDiscarder(logRotator(numToKeepStr: '2')) } //流水线的阶段 stages{ //阶段1 获取代码 stage("CheckOut"){ steps { script { println("下载代码 --> 分支： ${env.branch}") checkout( [$class: 'GitSCM', branches: [[name: "${branch}"]], extensions: [], userRemoteConfigs: [[ credentialsId: 'gitlab-mark', url: "${BASEURL}"]]]) } } } stage("unit Test"){ steps{ script{ env.GIT_COMMIT_DESCRIBE = "${sh(script:'git log --oneline --no-merges|head -1', returnStdout: true)}" env.TIMENOW_CN=sh(returnStdout: true, script: 'date +%Y年%m月%d日%H时%M分%S秒') env.GIT_COMMIT_TAGSHA=sh (script: """cut -b -40 .git/refs/remotes/origin/master""",returnStdout: true).trim() sh """ cd linuxea && mvn test -s /var/jenkins_home/.m2/settings.xml2 """ } } post { success { script { junit 'linuxea/target/surefire-reports/*.xml' } } } } stage("coed sonar"){ environment { def JOB_NAMES=sh (script: """echo ${BASEURL.split("/")[-1]} | cut -d . -f 1""",returnStdout: true).trim() def Projects_GitId=sh (script: """curl --silent --heade "PRIVATE-TOKEN: zrv1vpfZTtEFCJGrJczB" "http://gitlab.marksugar.com/api/v4/projects?simple=true"| /usr/local/package/jq-1.6/jq -rc '.[]|select(.path_with_namespace == "java/java-demo")'| /usr/local/package/jq-1.6/jq .id""",returnStdout: true).trim() def SONAR_git_TOKEN="K8DtxxxifxU1gQeDgvDK" def GitLab_Address="http://172.16.100.47" } steps{ script { withCredentials([string(credentialsId: 'sonarqube-token', variable: 'SONAR_TOKEN')]) { sh """ cd linuxea && \ /usr/local/package/sonar-scanner-4.6.2.2472-linux/bin/sonar-scanner \ -Dsonar.host.url=${GitLab_Address}:9000 \ -Dsonar.projectKey=${JOB_NAME} \ -Dsonar.projectName=${JOB_NAME} \ -Dsonar.projectVersion=${BUILD_NUMBER} \ -Dsonar.login=${SONAR_TOKEN} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=${env.BASEURL} \ -Dsonar.links.ci=${BUILD_URL} \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.core.codeCoveragePlugin=jacoco \ -Dsonar.jacoco.reportPaths=target/jacoco.exec \ -Dsonar.branch.name=${branch} \ -Dsonar.gitlab.commit_sha=${GIT_COMMIT_TAGSHA} \ -Dsonar.gitlab.ref_name=${branch} \ -Dsonar.gitlab.project_id=${Projects_GitId} \ -Dsonar.dynamicAnalysis=reuseReports \ -Dsonar.gitlab.failure_notification_mode=commit-status \ -Dsonar.gitlab.url=${GitLab_Address} \ -Dsonar.gitlab.user_token=${SONAR_git_TOKEN} \ -Dsonar.gitlab.api_version=v4 """ } } } } stage("mvn build"){ steps { script { sh """ cd linuxea mvn clean install -Dautoconfig.skip=true -Dmaven.test.skip=false -Dmaven.test.failure.ignore=true -s /var/jenkins_home/.m2/settings.xml2 """ } } } stage("docker build"){ steps{ script{ sh """ cd linuxea docker ps -a cp -r /usr/local/package/skywalking-agent ./ docker build -f ./Dockerfile -t $IPATH . docker push $IPATH docker rmi -f $IPATH """ } } } stage('Deploy') { steps { sh ''' [ ! -d ${JOB_NAMES} ] || rm -rf ${JOB_NAMES} } git clone ${kustomize_Git} && cd ${JOB_NAMES} && git checkout ${apps_name} echo "push latest images: $IPATH" echo "`date +%F-%T` imageTag: $IPATH buildId: ${BUILD_NUMBER} " >> ./buildhistory-$Projects_Area-${apps_name}.log cd overlays/$Projects_Area ${PACK_PATH}/kustomize edit set image $IPATH cd ../.. git add . git config --global push.default matching git config user.name zhengchao.tang git config user.email usertzc@163.com git commit -m "image tag $IPATH-> ${imageUrlPath}" git push -u origin ${apps_name} ${PACK_PATH}/argocd app sync ${apps_name} --retry-backoff-duration=10s -l marksugar/app=${apps_name} ''' // ${PACK_PATH}/argocd app sync ${apps_name} --retry-backoff-duration=10s -l marksugar/app=${apps_name} } // ${PACK_PATH}/kustomize build overlays/$Projects_Area/ | ${PACK_PATH}/kubectl --kubeconfig /var/jenkins_home/.kube/config-1.23.1-dev apply -f - } stage('status watch') { steps { sh ''' ${PACK_PATH}/kubectl --kubeconfig /var/jenkins_home/.kube/config-1.23.1-dev -n ${projectGroup} rollout status deployment ${apps_name} --watch --timeout=10m ''' } } } post { success{ script{ // ItmesName="${JOB_NAME.split('/')[-1]}" env.SedContent="构建通知" mdText = "### ✅ \n ### 发起人: ${BUILD_TRIGGER_BY} \n ### 项目: ${JOB_NAME} \n ### 标签: $IPATH \n ### 时间: ${TIMENOW_CN} \n ### 提交SHA: ${GIT_COMMIT_TAGSHA} \n ### Commit Info: ${GIT_COMMIT_DESCRIBE} \n ### By:  \n" DingTalk("DEVOPS", mdText, true, SedContent) } } failure{ script{ env.SedContent="构建通知" mdText = "### ❌ \n 发起人: ${BUILD_TRIGGER_BY} \n ### 项目: ${JOB_NAME} \n ### 标签: $IPATH \n ### 时间: ${TIMENOW_CN} \n ### 提交SHA: ${GIT_COMMIT_TAGSHA} \n ### Commit Info: ${GIT_COMMIT_DESCRIBE} \n ### By:  \n" DingTalk("DEVOPS", mdText, true, SedContent) } } } } def DingTalk(mdTitle, mdText, atAll, atUser = '' ,SedContent){ webhook = "https://oapi.dingtalk.com/robot/send?access_token=55d35d6f09f05388c1a8f7d73955cd9b7eaf4a0dd3803abdd1452e83d5b607ab" sh """ curl --location --request POST ${webhook} \ --header 'Content-Type: application/json' \ --data '{ "msgtype": "markdown", "markdown": { "title": "${mdTitle}", "text": "${SedContent}\n ${mdText}" }, "at": { "atMobiles": [ "${atUser}" ], "isAtAll": "${atAll}" } }' """ }现在，一个最简单的gitops的demo项目搭建完成参考gitops

linuxea:基于kustomize的argocd发布实现(10) 在此前我们配置了Kustomize清单，并且通过kubectl将清单应用到k8s中，之后又做另一个状态跟踪，但这还不够。我们希望通过一个cd工具来配置管理，并且提供一个可视化界面。我们选择argocd我不会在这篇章节中去介绍uI界面到底怎么操作，因为那些显而易见。我只会介绍argocd的二进制程序客户端的操作使用，但是也仅限于完成一个app的创建，集群的添加，项目的添加。仅此而已。argocd是一个成熟的部署工具，如果有时间，我将会在后面的时间里更新其他的必要功能。阅读此篇，你将了解argocd客户端最简单的操作，和一些此前的流水线实现方式列表如下：jenkins和gitlab触发(已实现)jenkins凭据使用(已实现)juit配置(已实现)sonarqube简单扫描(已实现)sonarqube覆盖率(已实现)打包基于java的skywalking agent(已实现)sonarqube与gitlab关联 (已实现)配置docker中构建docker (已实现)mvn打包(已实现)sonarqube简单分支扫描(已实现)基于gitlab来管理kustomize的k8s配置清单(已实现)kubectl部署(已实现)kubeclt deployment的状态跟踪(已实现)kustomize和argocd(本章实现)钉钉消息的构建状态推送1.1 安装2.4.2我们在gitlab上获取此配置文件，并修改镜像此前我拉取了2.4.0和2.4.2的镜像，如下2.4.0 image: registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:dex-v2.30.2 image: registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:haproxy-2.0.25-alpine image: registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:v2.4.0 image: registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:redis-7.0.0-alpine2.4.2 image: registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:dex-v2.30.2 image: registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:haproxy-2.0.25-alpine image: registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:v2.4.2 image: registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:redis-7.0.0-alpine分别替换所有镜像地址,如果是install.yaml就替换，如果是ha-install.yaml也替换sed -i 's@redis:7.0.0-alpine@registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:redis-7.0.0-alpine@g' sed -i 's@ghcr.io/dexidp/dex:v2.30.2@registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:dex-v2.30.2@g' sed -i 's@quay.io/argoproj/argocd:v2.4.0@registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:v2.4.0@g' sed -i 's@haproxy:2.0.25-alpine@registry.cn-hangzhou.aliyuncs.com/marksugar/argocd:haproxy-2.0.25-alpine@g'创建名称空间并applykubectl create namespace argocd kubectl apply -n argocd -f argocd.yaml更新删除不掉的时候的解决办法kubectl patch crd/appprojects.argoproj.io -p '{"metadata":{"finalizers":[]}}' --type=merge等待，到argocd组件准备完成[root@linuxea-11 ~/argocd]# kubectl -n argocd get pod NAME READY STATUS RESTARTS AGE argocd-application-controller-0 1/1 Running 0 7m33s argocd-applicationset-controller-7bbcd5c9bd-rqn84 1/1 Running 0 7m33s argocd-dex-server-75c668865-s9x5d 1/1 Running 0 7m33s argocd-notifications-controller-bc5954bd7-gg4ks 1/1 Running 0 7m33s argocd-redis-ha-haproxy-8658c76475-hdzkv 1/1 Running 0 7m33s argocd-redis-ha-haproxy-8658c76475-jrrtl 1/1 Running 0 7m33s argocd-redis-ha-haproxy-8658c76475-rk868 1/1 Running 0 7m33s argocd-redis-ha-server-0 2/2 Running 0 7m33s argocd-redis-ha-server-1 2/2 Running 0 5m3s argocd-redis-ha-server-2 2/2 Running 0 4m3s argocd-repo-server-567dd6c487-6k89z 1/1 Running 0 7m33s argocd-repo-server-567dd6c487-rt4vq 1/1 Running 0 7m33s argocd-server-677d79497b-k72h2 1/1 Running 0 7m33s argocd-server-677d79497b-pb5gt 1/1 Running 0 7m33s配置域名访问apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: argocd-server-ingress namespace: argocd annotations: cert-manager.io/cluster-issuer: letsencrypt-prod kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: "true" nginx.ingress.kubernetes.io/ssl-passthrough: "true" nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" spec: rules: - host: argocd.linuxea.com http: paths: - path: / pathType: Prefix backend: service: name: argocd-server port: name: https创建[root@linuxea-11 ~/argocd]# kubectl apply -f argocd-ingress.yaml ingress.networking.k8s.io/argocd-server-ingress created [root@linuxea-11 ~/argocd]# kubectl -n argocd get ingress NAME CLASS HOSTS ADDRESS PORTS AGE argocd-server-ingress nginx argocd.linuxea.com 80 11s配置nodeport我们直接使用nodeport来配置apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: server app.kubernetes.io/name: argocd-server app.kubernetes.io/part-of: argocd name: argocd-server namespace: argocd spec: ports: - name: http port: 80 nodePort: 31080 protocol: TCP targetPort: 8080 - name: https port: 443 nodePort: 31443 protocol: TCP targetPort: 8080 selector: app.kubernetes.io/name: argocd-server type: NodePort用户名admin, 获取密码[root@linuxea-11 ~/argocd]# kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d && echo QOOMW76CV8bEczKO1.2 客户端登录安装完成后，我们通过一个二进制的客户端来操作整个流程，于是我们需要下载一个Linux客户端注意： 和此前的其他包一样，如果是docker运行的jenkins，要将二进制包放到容器内，因此我提供了两种方式wget https://github.com/argoproj/argo-cd/releases/download/v2.4.2/argocd-linux-amd64如果你用私有域名的话，你本地hosts解析需要配置[root@linuxea-48 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.16.100.11 argocd.linuxea.com下载二进制文件后进行登录即可, 我使用的是nodeportargocd login 172.16.100.11:31080 --grpc-web[root@linuxea-48 ~/.kube]# argocd login 172.16.100.11:31080 --grpc-web WARNING: server certificate had error: x509: cannot validate certificate for 172.16.100.11 because it doesn't contain any IP SANs. Proceed insecurely (y/n)? y Username: admin Password: 'admin:login' logged in successfully Context '172.16.100.11:31080' updated登录会在一段时间后失效，于是我门需要些一个脚本过一段时间登录一次argocd login 172.16.100.11:31080 --grpc-web # 登录 argocd login 172.16.15.137:31080 --grpc-web最好写在脚本里面登录即可容器外脚本# cat /login.sh KCONFIG=/root/.kube/config-1.23.1-dev argocd login 172.16.100.11:31080 --username admin --password $(kubectl --kubeconfig=$KCONFIG -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ) --insecure --grpc-web容器内下载argocd二进制文件存放到已经映射的目录内，并添加执行权限[root@linuxea-48 /data/jenkins-latest/jenkins_home]# cp /usr/local/sbin/argocd /data/jenkins-latest/package/ [root@linuxea-48 /data/jenkins-latest/jenkins_home]# ll /data/jenkins-latest/package/ total 251084 drwxr-xr-x 6 root root 99 Sep 5 2021 apache-maven-3.8.2 -rw-r--r-- 1 root root 131352410 Jul 9 17:24 argocd drwxr-xr-x 6 root root 105 Sep 6 2021 gradle-6.9.1 drwxr-xr-x 2 root root 16 Oct 18 2021 jq-1.6 -rwxr-xr-x 1 root root 40230912 Jul 9 15:08 kubectl -rwxr-xr-x 1 root root 11976704 Jul 9 15:08 kustomize drwxr-xr-x 6 1001 1001 108 Aug 31 2021 node-v14.17.6-linux-x64 drwxrwxr-x 10 1001 1002 221 Jun 18 11:37 skywalking-agent -rw-r--r-- 1 root root 30443381 Jun 29 23:46 skywalking-java-8.11.0.tar.gz drwxr-xr-x 6 root root 51 May 7 2021 sonar-scanner-4.6.2.2472-linux -rw-r--r-- 1 root root 43099390 Sep 11 2021 sonar-scanner-cli-4.6.2.2472-linux.zip [root@linuxea-48 /data/jenkins-latest/jenkins_home]# chmod +x /data/jenkins-latest/package/argocd 还需要k8s的config配置文件，如果你阅读了上一篇基于jenkins的kustomize配置发布(9)，那这里当然是轻车熟路了我的二进制文件存放在/usr/local/package - /data/jenkins-latest/package:/usr/local/package由于我门在容器里面，我门复制config文件到一个位置而后指定即可[root@linuxea-48 ~]# cp -r ~/.kube /data/jenkins-latest/jenkins_home/ [root@linuxea-48 ~]# ls /data/jenkins-latest/jenkins_home/.kube/ cache config config-1.20.2-test config-1.22.1-prod config-1.22.1-test config-1.23.1-dev config2 marksugar-dev-1 marksugar-prod-1容器内登录KUBE_PATH=/usr/local/package KCONFIG=/var/jenkins_home/.kube/config-1.23.1-dev ${KUBE_PATH}/argocd login 172.16.100.11:31080 --username admin --password $(${KUBE_PATH}/kubectl --kubeconfig=$KCONFIG -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ) --insecure --grpc-web如下bash-5.1# KUBE_PATH=/usr/local/package bash-5.1# KCONFIG=/var/jenkins_home/.kube/config-1.23.1-dev bash-5.1# ${KUBE_PATH}/argocd login 172.16.100.11:31080 --username admin --password $(${KUBE_PATH}/kubectl --kubeconfig=$KCONFIG -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ) --insecure --grpc-web 'admin:login' logged in successfully Context '172.16.100.11:31080' updated在上面我们说过，一旦登录了只会，登录的凭据是会失效的，因此我们需要在计划任务里面，5个小时登录一次。而后使用计划任务进行登录即可0 5 * * * /bin/bash /login.sh查看版本信息[root@linuxea-48 ~]# argocd version --grpc-web argocd: v2.4.2+c6d0c8b BuildDate: 2022-06-21T21:03:41Z GitCommit: c6d0c8baaa291cd68465acd7ad6bef58b2b6f942 GitTreeState: clean GoVersion: go1.18.3 Compiler: gc Platform: linux/amd64 argocd-server: v2.4.2+c6d0c8b BuildDate: 2022-06-21T20:42:05Z GitCommit: c6d0c8baaa291cd68465acd7ad6bef58b2b6f942 GitTreeState: clean GoVersion: go1.18.3 Compiler: gc Platform: linux/amd64 Kustomize Version: v4.4.1 2021-11-11T23:36:27Z Helm Version: v3.8.1+g5cb9af4 Kubectl Version: v0.23.1 Jsonnet Version: v0.18.01.2.1. 集群凭据管理通常可能存在多个集群，因此，我们使用配置参数指定即可如果只有一个，无需指定，默认config[root@linuxea-48 ~]# ll ~/.kube/ total 56 drwxr-x--- 4 root root 35 Jun 22 00:09 cache -rw-r--r-- 1 root root 6254 Jun 21 23:58 config-1.20.2-test -rw-r--r-- 1 root root 6277 Jun 22 00:07 config-1.22.1-prod -rw-r--r-- 1 root root 6277 Jun 22 00:06 config-1.22.1-test -rw-r--r-- 1 root root 6193 Jun 22 00:09 config-1.23.1-dev -rw-r--r-- 1 root root 6246 Mar 4 23:55 config2 -rw-r--r-- 1 root root 6277 Aug 22 2021 marksugar-dev-1 -rw-r--r-- 1 root root 6277 Aug 22 2021 marksugar-prod-1 如果有多个，需要指定配置文件[root@linuxea-48 ~/.kube]# kubectl --kubeconfig /root/.kube/config-1.23.1-dev -n argocd get pod NAME READY STATUS RESTARTS AGE argocd-application-controller-0 1/1 Running 1 (12m ago) 23h argocd-applicationset-controller-7bbcd5c9bd-rqn84 1/1 Running 1 (12m ago) 23h argocd-dex-server-75c668865-s9x5d 1/1 Running 1 (12m ago) 23h argocd-notifications-controller-bc5954bd7-gg4ks 1/1 Running 1 (12m ago) 23h argocd-redis-ha-haproxy-8658c76475-hdzkv 1/1 Running 1 (12m ago) 23h argocd-redis-ha-haproxy-8658c76475-jrrtl 1/1 Running 1 (12m ago) 23h argocd-redis-ha-haproxy-8658c76475-rk868 1/1 Running 1 (12m ago) 23h argocd-redis-ha-server-0 2/2 Running 2 (12m ago) 23h argocd-redis-ha-server-1 2/2 Running 2 (12m ago) 23h argocd-redis-ha-server-2 2/2 Running 2 (12m ago) 23h argocd-repo-server-567dd6c487-6k89z 1/1 Running 1 (12m ago) 23h argocd-repo-server-567dd6c487-rt4vq 1/1 Running 1 (12m ago) 23h argocd-server-677d79497b-k72h2 1/1 Running 1 (12m ago) 23h argocd-server-677d79497b-pb5gt 1/1 Running 1 (12m ago) 23h\1.2.2 将集群加入argocd仍然需要重申下环境变量的配置export KUBECONFIG=$HOME/.kube/config-1.23.1-dev而后在查看当前的集群[root@linuxea-48 ~/.kube]# kubectl config get-contexts -o name context-cluster1将此集群加入到argocd[root@linuxea-48 ~/.kube]# argocd cluster add context-cluster1 WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `context-cluster1` with full cluster level privileges. Do you want to continue [y/N]? y INFO[0008] ServiceAccount "argocd-manager" created in namespace "kube-system" INFO[0008] ClusterRole "argocd-manager-role" created INFO[0008] ClusterRoleBinding "argocd-manager-role-binding" created Cluster 'https://172.16.100.11:6443' added这里添加完成后，在settings->Clusters 中也将会看到容器内首先将config文件复制到映射的目录内，比如/var/jenkins_home/# 配置kubeconfig位置 bash-5.1# export KUBECONFIG=/var/jenkins_home/.kube/config-1.23.1-dev # 复制二进制文件到sbin，仅仅是方便操作 bash-5.1# cp /usr/local/package/argocd /usr/sbin/ bash-5.1# cp /usr/local/package/kubectl /usr/sbin/ # 测试 bash-5.1# kubectl get pod NAME READY STATUS RESTARTS AGE nfs-client-provisioner-59bd97ddb-qcrpj 1/1 Running 18 (7h51m ago) 26d # 查看当前contexts名称 bash-5.1# kubectl config get-contexts -o name context-cluster1 # 添加到argocd bash-5.1# argocd cluster add context-cluster WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `kubernetes-admin@kubernetes` with full cluster level privileges. Do you want to continue [y/N]? WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `kubernetes-admin@kubernetes` with full cluster level privileges. Do you want to continue [y/N]? y INFO[0003] ServiceAccount "argocd-manager" created in namespace "kube-system" INFO[0003] ClusterRole "argocd-manager-role" created INFO[0003] ClusterRoleBinding "argocd-manager-role-binding" created Cluster 'https://172.16.100.11:6443' added添加完成1.3 定义repo存储库定于存储库有两种方式分别是ssh和http，都可以使用，参考官方文档1.3.1 密钥如果已经有现成的密钥，则不需要创建，如果没有，可以使用ssh-keygen -t ed25519 生成密钥， 并且添加到gitlab中# ssh-keygen -t ed25519 -f /home/jenkins_home/.ssh/ # ls /home/jenkins_home/.ssh/ -ll 总用量 8 -rw------- 1 root root 399 7月 8 16:44 id_rsa -rw-r--r-- 1 root root 93 7月 8 16:44 id_rsa.pubargocd添加git，指定~/.ssh/id_rsa,并使用--insecure-ignore-host-key选项[root@linuxea-48 ~/.kube]# argocd repo add git@172.16.100.47:pipeline-ops/marksugar-ui.git --ssh-private-key-path ~/.ssh/id_rsa --insecure-ignore-host-key Repository 'git@172.16.100.47:pipeline-ops/marksugar-ui.git' added这里添加完成在settings->repositories界面将会看到一个存储库容器内和上面一样，如果已经有现成的密钥，则不需要创建，如果没有，可以使用ssh-keygen -t ed25519 生成密钥， 并且将id_rsa.pub添加到gitlab中下面是docker-compose的密钥 volumes: .... - /home/jenkins_home/.ssh/:/root/.ssh我们在上面已经添加了marksugar-ui, 如果有多个项目，多次添加即可我们开始添加 java-demogit@172.16.100.47:devops/k8s-yaml.git是kustmoize配置清单的地址argocd repo add git@172.16.100.47:devops/k8s-yaml.git --ssh-private-key-path ~/.ssh/id_rsa --insecure-ignore-host-keybash-5.1# argocd repo add git@172.16.100.47:devops/k8s-yaml.git --ssh-private-key-path ~/.ssh/id_rsa --insecure-ignore-host-key Repository 'git@172.16.100.47:devops/k8s-yaml.git' added1.3.2 http我门仍然可以考虑使用http来使用，官方的示例如下argocd repo add https://github.com/argoproj/argocd-example-apps --username <username> --password <password>我的环境如下配置：argocd repo add http://172.16.15.136:180/devops/k8s-yaml --username root --password gitlab.com # 添加repo root@ca060212e6f6:/var/jenkins_home# argocd repo add http://172.16.15.136:180/devops/k8s-yaml.git --username root --password gitlab.com Repository 'http://172.16.15.136:180/devops/k8s-yaml.git' added1.4 定义项目AppProject CRD 是代表应用程序逻辑分组的 Kubernetes 资源对象。它由以下关键信息定义：sourceRepos引用项目中的应用程序可以从中提取清单的存储库。destinations引用项目中的应用程序可以部署到的集群和命名空间（不要使用该name字段，仅server匹配该字段）。roles定义了他们对项目内资源的访问权限的实体列表。一个示例规范如下：在创建之前，我们先在集群内创建一个名称空间:marksugarkubectl create ns marksugar声明式配置如下，指定name，指定marksugar部署的名称空间，其他默认 destinations: - namespace: marksugar server: 'https://172.16.100.11:6443'更多时候我们限制项目内使用的范围，比如我们只配置使用的如：deployment,service,configmap,这些配置取决于控制器apiVersion: v1 kind: ConfigMap ... --- apiVersion: v1 kind: Service ...and DeploymentapiVersion: apps/v1 kind: Deployment如果此时有ingress，那么配置就如下 - group: 'networking.k8s.io' kind: 'Ingress'以此推论。最终我的配置如下： namespaceResourceWhitelist: - group: 'apps' kind: 'Deployment' - group: '' kind: 'Service' - group: '' kind: 'ConfigMap'一个完整的配置如下：apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: my-linuxea # 名称 # name: marksugar namespace: argocd # finalizers: # - resources-finalizer.argocd.argoproj.io spec: description: Example Project(测试) # 更详细的内容 sourceRepos: - '*' destinations: - namespace: marksugar # 名称空间 server: 'https://172.16.100.11:6443' # k8s api地址 # clusterResourceWhitelist: # - group: '' # kind: Namespace # namespaceResourceBlacklist: # - group: '' # kind: ResourceQuota # - group: '' # kind: LimitRange # - group: '' # kind: NetworkPolicy namespaceResourceWhitelist: - group: 'apps' kind: 'Deployment' # 名称空间的内允许让argocd当前app使用的的kind - group: '' kind: 'Service' # 名称空间的内允许让argocd当前app使用的的kind - group: '' kind: 'ConfigMap' # 名称空间的内允许让argocd当前app使用的的kind # kind: Deployment # - group: 'apps' # kind: StatefulSet # roles: # - name: read-only # description: Read-only privileges to my-project # policies: # - p, proj:my-project:read-only, applications, get, my-project/*, allow # groups: # - test-env # - name: ci-role # description: Sync privileges for guestbook-dev # policies: # - p, proj:my-project:ci-role, applications, sync, my-project/guestbook-dev, allow # jwtTokens: # - iat: 1535390316上面的这个有太多注释，精简一下，并进行成我门实际的参数，最终如下：apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: my-linuxea-java-demo namespace: argocd spec: description: Example Project(测试) sourceRepos: - '*' destinations: - namespace: java-demo server: 'https://172.16.100.11:6443' namespaceResourceWhitelist: - group: 'apps' kind: 'Deployment' - group: '' kind: 'Service' - group: '' kind: 'ConfigMap'执行PS E:\ops\k8s-1.23.1-latest\gitops\argocd> kubectl.exe apply -f .\project-new.yaml appproject.argoproj.io/my-linuxea-java-demo created执行完成后，将会创建一个projects，在settings->projects查看1.5 定义应用Application CRD 是 Kubernetes 资源对象，表示环境中已部署的应用程序实例。它由两个关键信息定义：source对 Git 中所需状态的引用（存储库、修订版、路径、环境）destination对目标集群和命名空间的引用。对于集群，可以使用 server 或 name 之一，但不能同时使用两者（这将导致错误）。当服务器丢失时，它会根据名称进行计算并用于任何操作。一个最小的应用程序规范如下：apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: marksugar-ui namespace: argocd labels: marksugar/marksugar-ui: prod # 标签 spec: project: my-linuxea # 定义的项目名 source: repoURL: git@172.16.100.47:pipeline-ops/marksugar-ui.git # git地址 targetRevision: master # git分支 path: overlays/marksugar-ui/prod/ # git路径对应到目录下的配置 destination: server: https://172.16.100.11:6443 # k8s api namespace: marksugar # 名称空间有关其他字段，请参阅application.yaml。只要您完成了入门的第一步，您就可以应用它kubectl apply -n argocd -f application.yaml，Argo CD 将开始部署留言簿应用程序。或者使用下面客户端命令进行配置，比如我此前配置去的marksugar-ui就是命令行配置的，如下：argocd app create marksugar-ui --repo git@172.16.100.47:pipeline-ops/marksugar-ui.git --revision master --path overlays/marksugar-ui/prod/ --dest-server https://172.16.100.11:6443 --dest-namespace marksugar --project=my-linuxea --label=marksugar/marksugar-ui=prod我门仍然进行修改成我门希望的配置样子，yaml如下我这里使用的是httpapiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: java-demo namespace: argocd labels: marksugar/app: java-demo spec: project: my-linuxea-java-demo source: repoURL: git@172.16.100.47:devops/k8s-yaml.git targetRevision: java-demo path: overlays/dev/ destination: server: https://172.16.100.11:6443 namespace: java-demo此时创建了一个appPS E:\ops\k8s-1.23.1-latest\gitops\argocd\java-demo> kubectl.exe apply -f .\app.yaml application.argoproj.io/java-demo created如下只有同步正常，healthy才会变绿如果有多个名称空间，不想混合显示，我们在页面中在做左侧，选择cluster的名称空间后，才能看到名称空间下的app,也就是应用如果你配置的是http的git地址就会是下面这个样子配置apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: java-demo namespace: argocd labels: marksugar/app: java-demo spec: project: my-linuxea-java-demo source: repoURL: http://172.16.15.136:180/devops/k8s-yaml.git targetRevision: java-demo path: overlays/dev/ destination: server: https://172.16.15.137:6443 namespace: java-demo视图1.6 手动同步我门可以点击web页面的上面的sync来进行同步，也可以用命令行手动同步使其生效我门通过argocd app list查看当前的已经有的项目示例：密钥root@9c0cad5ebce8:/# argocd app list NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET java-demo https://172.16.15.137:6443 java-demo my-linuxea-java-demo Unknown Healthy <none> ComparisonError git@172.16.15.136:23857/devops/k8s-yaml.git overlays/dev/ java-demohttproot@ca060212e6f6:/var/jenkins_home# argocd app list NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET java-demo https://172.16.15.137:6443 java-demo my-linuxea-java-demo OutOfSync Missing <none> <none> http://172.16.15.136:180/devops/k8s-yaml.git overlays/dev/ java-demo而我们现在的是这样的bash-5.1# argocd app list NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET java-demo https://172.16.100.11:6443 java-demo my-linuxea-java-demo OutOfSync Missing <none> <none> git@172.16.100.47:devops/k8s-yaml.git overlays/dev/ java-demo marksugar-ui https://172.16.100.11:6443 marksugar my-linuxea Synced Healthy <none> <none> git@172.16.100.47:pipeline-ops/marksugar-ui.git overlays/marksugar-ui/prod/ master而后进行同步即可argocd app sync java-demo --retry-backoff-duration=10s -l marksugar/app=java-demo如下bash-5.1# argocd app sync java-demo --retry-backoff-duration=10s -l marksugar/app=java-demo TIMESTAMP GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE 2022-07-09T19:20:26+08:00 ConfigMap java-demo envinpod-74t9b8htb6 Synced 2022-07-09T19:20:26+08:00 Service java-demo java-demo OutOfSync Missing 2022-07-09T19:20:26+08:00 apps Deployment java-demo java-demo Synced Healthy 2022-07-09T19:20:27+08:00 Service java-demo java-demo OutOfSync Healthy 2022-07-09T19:20:27+08:00 ConfigMap java-demo envinpod-74t9b8htb6 Synced configmap/envinpod-74t9b8htb6 unchanged 2022-07-09T19:20:27+08:00 Service java-demo java-demo OutOfSync Healthy service/java-demo created 2022-07-09T19:20:27+08:00 apps Deployment java-demo java-demo Synced Healthy deployment.apps/java-demo configured Name: java-demo Project: my-linuxea-java-demo Server: https://172.16.100.11:6443 Namespace: java-demo URL: https://172.16.100.11:31080/applications/java-demo Repo: git@172.16.100.47:devops/k8s-yaml.git Target: java-demo Path: overlays/dev/ SyncWindow: Sync Allowed Sync Policy: <none> Sync Status: Synced to java-demo (fd1286f) Health Status: Healthy Operation: Sync Sync Revision: fd1286f64d1edac2def43d4a37bcc13a9f0286d0 Phase: Succeeded Start: 2022-07-09 19:20:26 +0800 CST Finished: 2022-07-09 19:20:27 +0800 CST Duration: 1s Message: successfully synced (all tasks run) GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE ConfigMap java-demo envinpod-74t9b8htb6 Synced configmap/envinpod-74t9b8htb6 unchanged Service java-demo java-demo Synced Healthy service/java-demo created apps Deployment java-demo java-demo Synced Healthy deployment.apps/java-demo configured TIMESTAMP GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE 2022-07-09T19:20:28+08:00 apps Deployment java-demo java-demo Synced Healthy 2022-07-09T19:20:28+08:00 ConfigMap java-demo envinpod-74t9b8htb6 Synced 2022-07-09T19:20:28+08:00 Service java-demo java-demo Synced Healthy 2022-07-09T19:20:28+08:00 apps Deployment java-demo java-demo Synced Healthy deployment.apps/java-demo configured 2022-07-09T19:20:28+08:00 ConfigMap java-demo envinpod-74t9b8htb6 Synced configmap/envinpod-74t9b8htb6 unchanged 2022-07-09T19:20:28+08:00 Service java-demo java-demo Synced Healthy service/java-demo unchanged Name: java-demo Project: my-linuxea-java-demo Server: https://172.16.100.11:6443 Namespace: java-demo URL: https://172.16.100.11:31080/applications/java-demo Repo: git@172.16.100.47:devops/k8s-yaml.git Target: java-demo Path: overlays/dev/ SyncWindow: Sync Allowed Sync Policy: <none> Sync Status: Synced to java-demo (fd1286f) Health Status: Healthy Operation: Sync Sync Revision: fd1286f64d1edac2def43d4a37bcc13a9f0286d0 Phase: Succeeded Start: 2022-07-09 19:20:27 +0800 CST Finished: 2022-07-09 19:20:28 +0800 CST Duration: 1s Message: successfully synced (all tasks run) GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE ConfigMap java-demo envinpod-74t9b8htb6 Synced configmap/envinpod-74t9b8htb6 unchanged Service java-demo java-demo Synced Healthy service/java-demo unchanged apps Deployment java-demo java-demo Synced Healthy deployment.apps/java-demo configured同步完成后状态就会发生改变命令行查看bash-5.1# argocd app list NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET java-demo https://172.16.100.11:6443 java-demo my-linuxea-java-demo Synced Healthy <none> <none> git@172.16.100.47:devops/k8s-yaml.git overlays/dev/ java-demo marksugar-ui https://172.16.100.11:6443 marksugar my-linuxea Synced Healthy <none> <none> git@172.16.100.47:pipeline-ops/marksugar-ui.git overlays/marksugar-ui/prod/ master打开页面查看如果是http的这里会显示http此时正在拉取镜像状态是 Progressing，我们等待拉取完成，而后选中后会点击进入详情页面项目内的仪表盘功能如下图一旦镜像完成拉取，并且runing起来，则显示健康仪表盘功能如下图回到k8s查看[root@linuxea-01 .ssh]# kubectl get all -n java-demo NAME READY STATUS RESTARTS AGE pod/java-demo-6474cb8fc8-6zwlt 1/1 Running 0 7m45s pod/java-demo-6474cb8fc8-92sw7 1/1 Running 0 7m45s pod/java-demo-6474cb8fc8-k8985 1/1 Running 0 7m45s pod/java-demo-6474cb8fc8-ndzpl 1/1 Running 0 7m45s pod/java-demo-6474cb8fc8-rxg2k 1/1 Running 0 7m45s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/java-demo NodePort 10.111.26.148 <none> 8080:31180/TCP 24h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/java-demo 5/5 5 5 7m45s NAME DESIRED CURRENT READY AGE replicaset.apps/java-demo-6474cb8fc8 5 5 5 7m45s1.7 加入流水线阅读过上一篇基于jenkins的kustomize配置发布(9)你大概就知道，整个简单的流程是怎么走的，我们复制过来修改一下，如下当前阶段流水线阶段，步骤大致如下：1.判断本地是否有git的目录，如果有就删除2.拉取git，并切换到分支3.追加当前的镜像版本到一个buildhistory的文件中4.cd到目录中修改镜像5.修改完成后上传修改你被人6.argocd同步与之不同的就是将kustomize和kubectl改成了argocd代码快如下： stage('Deploy') { steps { sh ''' [ ! -d ${JOB_NAMES} ] || rm -rf ${JOB_NAMES} } git clone ${kustomize_Git} && cd ${JOB_NAMES} && git checkout ${apps_name} echo "push latest images: $IPATH" echo "`date +%F-%T` imageTag: $IPATH buildId: ${BUILD_NUMBER} " >> ./buildhistory-$Projects_Area-${apps_name}.log cd overlays/$Projects_Area ${PACK_PATH}/kustomize edit set image $IPATH cd ../.. git add . git config --global push.default matching git config user.name zhengchao.tang git config user.email usertzc@163.com git commit -m "image tag $IPATH-> ${imageUrlPath}" git push -u origin ${apps_name} ${PACK_PATH}/argocd app sync ${apps_name} --retry-backoff-duration=10s -l marksugar/app=${apps_name} ''' } } 仅此而已在上一篇中忘了截图与此同时，gitlab上已经有了一个版本的历史记录argocd最简单的示例到此告一段落参考gitops

linuxea:基于jenkins的kustomize配置发布(9) 在之前的几篇中，我分别介绍了基础环境的配置，skywaling+nacos的配置，nexus3的配置，围绕sonarqube的配置和构建镜像的配置。这一篇中，基于构建的镜像进行清单编排。我们需要一种工具来管理配置清单。阅读此篇，你将了解如下列表中简单的实现方式：jenkins和gitlab触发(已实现)jenkins凭据使用(已实现)juit配置(已实现)sonarqube简单扫描(已实现)sonarqube覆盖率(已实现)打包基于java的skywalking agent(上一章已实现)sonarqube与gitlab关联 (上一章已实现)配置docker中构建docker (上一章已实现)mvn打包(上一章已实现)sonarqube简单分支扫描(上一章已实现)基于gitlab来管理kustomize的k8s配置清单(本章实现)kubectl部署(本章实现)kubeclt deployment的状态跟踪(本章实现)钉钉消息的构建状态推送没错，我移情别恋了，在Helm和kustomize中，我选择后者。最大的原因是因为kustomize简单，易于维护。无论从那个角度，我都找不到不用kustomize的理由。这倒不是因为kustomize是多么优秀，仅仅是因为kustomize的方式让一切变得都简单。Helm和kustomizehelm几乎可以完成所有的操作，但是helm的问题是学习有难度，对于小白不友好，配置一旦过多调试将会更复杂。也是因为这种限制，那么使用helm的范围就被缩小了，不管在什么条件下，它都不在是优选。kustomize更直白，无论是开发，还是运维新手，都可以快速上手进行修改添加等基础配置。kustomizekustomize用法在官网的github上已经有所说明了，并且这里温馨的提供了中文示例。讨论如何学习kustomize不在本章的重点遵循kustmoize的版本，在https://github.com/kubernetes-sigs/kustomize/releases找到一个版本，通过http://toolwa.com/github/加速下载Kubectl 版本自定义版本< v1.14不适用v1.14-v1.20v2.0.3v1.21v4.0.5v1.22v4.2.0[root@k8s-01 linuxea]# kustomize version {Version:kustomize/v4.5.5 GitCommit:daa3e5e2c2d3a4b8c94021a7384bfb06734bcd26 BuildDate:2022-05-20T20:25:40Z GoOs:linux GoArch:amd64}创建必要的目录结构阅读示例中的示例：devops和开发配合管理配置数据有助于理解kustomize配置方法场景：在生产环境中有一个基于 Java 由多个内部团队对于业务拆分了不通的组并且有不同的项目的应用程序。这些服务在不同的环境中运行：development、 testing、 staging 和 production，有些配置需要频繁修改的。如果只是维护一个大的配置文件是非常麻烦且困难的 ，而这些配置文件也是需要专业运维人员或者devops工程师来进行操作的，这里面包含了一些片面且偏向运维的工作是开发人员不必知道的。例如：生产环境的敏感数据关键的登录凭据等这些在kustomize中被分成了不通的类因此，kustomize提供了混合管理办法基于相同的 base 创建 n 个 overlays 来创建 n 个集群环境的方法我们将使用 n==2，例如，只使用 development 和 production ，这里也可以使用相同的方法来增加更多的环境。运行 kustomize build 基于 overlay 的 target 来创建集群环境。为了让这一切开始运行，准备如下创建kustomize目录结构创建并配置kustomize配置文件最好创建gitlab项目，将配置存放在gitlab开始此前我写了一篇kustomize变量传入有过一些介绍，我们在简单补充一下。kustomize在1.14版本中已经是Kubectl内置的命令，并且支持kubernetes的原生可复用声明式配置的插件。它引入了一种无需模板的方式来自定义应用程序配置，从而简化了现成应用程序的使用。Kustomize 遍历 Kubernetes 清单以添加、删除或更新配置选项。它既可以作为独立的二进制文件使用，也可以作为kubectl来使用更多的背景可参考它的白皮书，这些在github的Declarative application management in Kubernetes存放。因为总的来说，这篇不是让你如何去理解背后的故事，而是一个最简单的示例常见操作在项目中为所有 Kubernetes 对象设置贯穿性字段是一种常见操作。 贯穿性字段的一些使用场景如下：为所有资源设置相同的名字空间为所有对象添加相同的前缀或后缀为对象添加相同的标签集合为对象添加相同的注解集合为对象添加相同的资源限制以及以及副本数这些通过在overlays目录下不同的配置来区分不通的环境所用的清单信息安装遵循github版本对应规则Kubectl versionKustomize version< v1.14n/av1.14-v1.20v2.0.3v1.21v4.0.5v1.22v4.2.0我的集群是1.23.1，因此我下载4.5.4PS E:\ops\k8s-1.23.1-latest\gitops> kustomize version {Version:kustomize/v4.5.4 GitCommit:cf3a452ddd6f83945d39d582243b8592ec627ae3 BuildDate:2022-03-28T23:12:45Z GoOs:windows GoArch:amd64}java-demo我这里已经配置了一个已经配置好的环境，我将会在这里简单介绍使用方法和配置，我不会详细说明deployment控制器的配置清单，也不会说明和kustomize基本使用无关的配置信息，我只会尽可能的在这个简单的示例中说明整个kustomize的在本示例中的用法。简述：kustomize需要base和Overlays目录，base可以是多个，overlays也可以是多个，overlays下的文件最终会覆盖到base的配置之上，只要配置是合理的，base的配置应该将有共性的配置最终通过overlays来进行配置，以此应对多个环境的配置。java-demo是一个无状态的java应用，使用的是Deployment控制器进行配置，并且创建一个service，于此同时传入skywalking的环境变量信息。1. 目录结构目录结构如下：# tree ./ ./ ├── base │   ├── deployment.yaml │   ├── kustomization.yaml │   └── service.yaml ├── overlays │   ├── dev │   │   ├── env.file │   │   ├── kustomization.yaml │   │   └── resources.yaml │   └── prod │   ├── kustomization.yaml │   ├── replicas.yaml │   └── resources.yaml └── README.md 4 directories, 11 files其中两目录如下：./ ├── base ├── overlays └── README.mdbase: 目录作为基础配置目录，真实的配置文件在这个文件下overlays: 目录作为场景目录，描述与 base 应用配置的差异部分来实现资源复用而在overlays目录下，又有两个目录，分别是dev和prod，分别对应俩个环境的配置，这里可以任意起名来区分，因为在这两个目录下面存放的是各自不通的配置./ ├── base ├── overlays │   ├── dev │   └── prod └── README.md1.1 imagePullSecrets除此之外，我们需要一个拉取镜像的信息使用cat ~/.docker/config.json |base64获取到base64字符串编码，而后复制到.dockerconfigjson: >-下即可apiVersion: v1 data: .dockerconfigjson: >- ewoJImkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTkuMDMuMTIgKGxpbnV4KSIKCX0KfQ== kind: Secret metadata: name: 156pull namespace: java-demo type: kubernetes.io/dockerconfigjson2. base目录base目录下分别有三个文件，分别如下├── base │   ├── deployment.yaml │   ├── kustomization.yaml │   └── service.yaml在deployment.yaml中定义必要的属性不定义场景的指标，如标签，名称空间，副本数量和资源限制定义名称，镜像地址，环境变量名这些不定义的属性通过即将配置的overlays中的配置进行贯穿覆盖到这个基础配置之上必须定义的属性表明了贯穿的属性和基础的配置是一份这里的环境变量用的是configmap的方式，值是通过后面传递过来的。如下deployment.yamlapiVersion: apps/v1 kind: Deployment metadata: name: java-demo spec: selector: matchLabels: template: metadata: labels: spec: containers: - image: harbor.marksugar.com/java/linuxea-2022 imagePullPolicy: IfNotPresent name: java-demo ports: - containerPort: 8080 env: - name: SW_AGENT_NAME valueFrom: configMapKeyRef: name: envinpod key: SW_AGENT_NAME - name: SW_AGENT_TRACE_IGNORE_PATH valueFrom: configMapKeyRef: name: envinpod key: SW_AGENT_TRACE_IGNORE_PATH - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES valueFrom: configMapKeyRef: name: envinpod key: SW_AGENT_COLLECTOR_BACKEND_SERVICES imagePullSecrets: - name: 156pull restartPolicy: Alwaysservice.yamlapiVersion: v1 kind: Service metadata: name: java-demo spec: type: NodePort ports: - port: 8080 targetPort: 8080 nodePort: 31180kustomization.yamlkustomization.yaml引入这两个配置文件resources: - deployment.yaml - service.yaml执行 kustomize build /base ，得到的结果如下，这就是当前的原始清单apiVersion: v1 kind: Service metadata: name: java-demo spec: ports: - nodePort: 31180 port: 8080 targetPort: 8080 type: NodePort --- apiVersion: apps/v1 kind: Deployment metadata: name: java-demo spec: selector: matchLabels: null template: metadata: labels: null spec: containers: - env: - name: SW_AGENT_NAME valueFrom: configMapKeyRef: key: SW_AGENT_NAME name: envinpod - name: SW_AGENT_TRACE_IGNORE_PATH valueFrom: configMapKeyRef: key: SW_AGENT_TRACE_IGNORE_PATH name: envinpod - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES valueFrom: configMapKeyRef: key: SW_AGENT_COLLECTOR_BACKEND_SERVICES name: envinpod image: harbor.marksugar.com/java/linuxea-2022:202207091551 imagePullPolicy: IfNotPresent name: java-demo ports: - containerPort: 8080 imagePullSecrets: - name: 156pull restartPolicy: Always3. overlays目录首先，在overlays目录下是有dev和prod目录的，我们先看在dev目录下的kustomization.yamlkustomization.yaml中的内容，包含一组资源和相关的自定义信息，如下更多用法参考官方文档或者github社区kustomization.yamlapiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization patchesStrategicMerge: - resources.yaml # 当如当前的文件 namespace: java-demo # 名称空间 images: - name: harbor.marksugar.com/java/linuxea-2022 # 镜像url必须保持和base中一致 newTag: '202207072119' # 镜像tag bases: - ../../base # 引入bases基础文件 # configmap变量 configMapGenerator: - name: envinpod # 环境变量名称 env: env.file # 环境变量位置 # 副本数 replicas: - name: java-demo # 名称必须保持一致 count: 5 # namePrefix: dev- # pod前缀 # nameSuffix: "-001" # pod后缀 commonLabels: app: java-demo # 标签 # logging: isOk # commonAnnotations: # oncallPager: 897-001删掉那些注释后如下apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization patchesStrategicMerge: - resources.yaml namespace: java-demo images: - name: harbor.marksugar.com/java/linuxea-2022 newTag: '202207071059' bases: - ../../base configMapGenerator: - name: envinpod env: env.file replicas: - name: java-demo count: 5 commonLabels: app: java-demoresources.yaml resources.yaml 中的name必须保持一致apiVersion: apps/v1 kind: Deployment metadata: name: java-demo spec: template: spec: containers: - name: java-demo resources: limits: cpu: "1" memory: 2048Mi requests: cpu: "1" memory: 2048Mienv.fileenv.file定义的变量是对应在base中的，这些是skwayling中的必要信息，参考kubernetes中skywalking9.0部署使用,env的用法参考kustomize变量引入SW_AGENT_NAME=test::java-demo SW_AGENT_TRACE_IGNORE_PATH=GET:/health,GET:/aggreg/health,/eureka/**,xxl-job/** SW_AGENT_COLLECTOR_BACKEND_SERVICES=skywalking-oap.skywalking:11800查看 kustomize build overlays/dev/后的配置清单。如下所示：apiVersion: v1 data: SW_AGENT_COLLECTOR_BACKEND_SERVICES: skywalking-oap.skywalking:11800 SW_AGENT_NAME: test::java-demo SW_AGENT_TRACE_IGNORE_PATH: GET:/health,GET:/aggreg/health,/eureka/**,xxl-job/** kind: ConfigMap metadata: labels: app: java-demo name: envinpod-74t9b8htb6 namespace: java-demo --- apiVersion: v1 kind: Service metadata: labels: app: java-demo name: java-demo namespace: java-demo spec: ports: - nodePort: 31180 port: 8080 targetPort: 8080 selector: app: java-demo type: NodePort --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: java-demo name: java-demo namespace: java-demo spec: replicas: 5 selector: matchLabels: app: java-demo template: metadata: labels: app: java-demo spec: containers: - env: - name: SW_AGENT_NAME valueFrom: configMapKeyRef: key: SW_AGENT_NAME name: envinpod-74t9b8htb6 - name: SW_AGENT_TRACE_IGNORE_PATH valueFrom: configMapKeyRef: key: SW_AGENT_TRACE_IGNORE_PATH name: envinpod-74t9b8htb6 - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES valueFrom: configMapKeyRef: key: SW_AGENT_COLLECTOR_BACKEND_SERVICES name: envinpod-74t9b8htb6 image: harbor.marksugar.com/java/linuxea-2022:202207071059 imagePullPolicy: IfNotPresent name: java-demo ports: - containerPort: 8080 resources: limits: cpu: "1" memory: 2048Mi requests: cpu: "1" memory: 2048Mi imagePullSecrets: - name: 156pull restartPolicy: Alwaysbase作为基础配置，Overlays作为覆盖来区分。base是包含 kustomization.yaml 文件的一个目录，其中包含一组资源及其相关的定制。 base可以是本地目录或者来自远程仓库的目录，只要其中存在 kustomization.yaml 文件即可。 Overlays 也是一个目录，其中包含将其他 kustomization 目录当做 bases 来引用的 kustomization.yaml 文件。 base不了解Overlays的存在，且可被多个Overlays所使用。 Overlays则可以有多个base，且可针对所有base中的资源执行操作，还可以在其上执行定制。通过sed替换Overlays下的文件内容或者kustomize edit set，如：在Overlays下执行kustomize edit set image harbor.marksugar.com/java/linuxea-2022:202207091551:202207071059:1.14.b替换镜像文件。一切符合预期后，使用kustomize.exe build .\overlays\dev\ | kubectl apply -f -使其生效。4. 部署到k8s命令部署两种方式kustomizekustomize build overlays/dev/ | kubectl apply -f -kubectlkubectl apply -k overlays/dev/使用kubectl apply -k生效，如下PS E:\ops\k8s-1.23.1-latest\gitops> kubectl.exe apply -k .\overlays\dev\ configmap/envinpod-74t9b8htb6 unchanged service/java-demo created deployment.apps/java-demo created如果使用的域名是私有的，需要在本地hosts填写本地解析172.16.100.54 harbor.marksugar.com并且需要修改/etc/docker/daemon.json{ "data-root": "/var/lib/docker", "exec-opts": ["native.cgroupdriver=systemd"], "insecure-registries": ["harbor.marksugar.com"], "max-concurrent-downloads": 10, "live-restore": true, "log-driver": "json-file", "log-level": "warn", "log-opts": { "max-size": "50m", "max-file": "1" }, "storage-driver": "overlay2" }查看部署情况PS E:\ops\k8s-1.23.1-latest\gitops\kustomize-k8s-yaml> kubectl.exe -n java-demo get all NAME READY STATUS RESTARTS AGE pod/java-demo-6474cb8fc8-6xs8t 1/1 Running 0 41s pod/java-demo-6474cb8fc8-9z9sd 1/1 Running 0 41s pod/java-demo-6474cb8fc8-jfqv6 1/1 Running 0 41s pod/java-demo-6474cb8fc8-p5ztd 1/1 Running 0 41s pod/java-demo-6474cb8fc8-sqt7b 1/1 Running 0 41s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/java-demo NodePort 10.111.26.148 <none> 8080:31180/TCP 41s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/java-demo 5/5 5 5 41s NAME DESIRED CURRENT READY AGE replicaset.apps/java-demo-6474cb8fc8 5 5 5 42s与此同时,skywalking也加入成功创建git项目在gitlab创建了一个组，在组织里面创建了一个项目，名称以项目命名，在项目内每个应用对应一个分支如： devops组内内新建一个k8s-yaml的项目，项目内创建一个java-demo分支，java-demo分支中存放java-demo的配置文件现在创建key，将密钥加入到项目中ssh-keygen -t ed25519将文件推送到git上$ git clone git@172.16.100.47:devops/k8s-yaml.git Cloning into 'k8s-yaml'... remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (3/3), done. $ cd k8s-yaml/ $ git checkout -b java-demo Switched to a new branch 'java-demo $ ls -ll total 1024 -rw-r--r-- 1 Administrator 197121 12 Jul 7 21:09 README.MD drwxr-xr-x 1 Administrator 197121 0 Jun 28 20:15 base/ -rw-r--r-- 1 Administrator 197121 774 Jul 6 18:05 imagepullsecrt.yaml drwxr-xr-x 1 Administrator 197121 0 Jun 28 20:15 overlays/ $ git add . $ git commit -m "first commit" [java-demo a9701f7] first commit 11 files changed, 185 insertions(+) create mode 100644 base/deployment.yaml create mode 100644 base/kustomization.yaml create mode 100644 base/service.yaml create mode 100644 imagepullsecrt.yaml create mode 100644 overlays/dev/env.file create mode 100644 overlays/dev/kustomization.yaml create mode 100644 overlays/dev/resources.yaml create mode 100644 overlays/prod/kustomization.yaml create mode 100644 overlays/prod/replicas.yaml create mode 100644 overlays/prod/resources.yaml $ git push -u origin java-demo Enumerating objects: 19, done. Counting objects: 100% (19/19), done. Delta compression using up to 8 threads Compressing objects: 100% (15/15), done. Writing objects: 100% (17/17), 2.90 KiB | 329.00 KiB/s, done. Total 17 (delta 2), reused 0 (delta 0), pack-reused 0 remote: remote: To create a merge request for java-demo, visit: remote: http://172.16.100.47/devops/k8s-yaml/-/merge_requests/new?merge_request%5Bsource_branch%5D=java-demo remote: To 172.16.100.47:devops/k8s-yaml.git bb67227..a9701f7 java-demo -> java-demo Branch 'java-demo' set up to track remote branch 'java-demo' from 'origin'.添加到流水线首先，kustomize是配置文件是存放在gitlab上，因此，这个git需要我们拉取下来，而后修改镜像名称，应用kustomize的配置后，在push到gitlab上在这里的是kustomize是仅仅来管理yaml清单文件，在后面将使用argocd来做我们在流水线里面配置一个环境变量，指向kustomize配置文件的git地址，并切除git拉取后的目录地址尽可能的在gitlab和jenkins上的项目名称保持一直，才能做好流水线取值或者切出值的时候方便def kustomize_Git="git@172.16.100.47:devops/k8s-yaml.git" def JOB_NAMES=sh (script: """echo ${kustomize_Git.split("/")[-1]} | cut -d . -f 1""",returnStdout: true).trim() 但是kustomize是不能直接去访问集群的，因此还必须用kubectl，那就以为这需要config文件我们使用命令指定配置文件位置kubectl --kubeconfig /var/jenkins_home/.kube/config-1.23.1-dev另外，如果你的jenkins的docker镜像没有kustomize，或者kubectl，需要挂载进去，因此我的就变成了 environment { def tag_time = new Date().format("yyyyMMddHHmm") def IPATH="harbor.marksugar.com/java/${JOB_NAME}:${tag_time}" def kustomize_Git="git@172.16.100.47:devops/k8s-yaml.git" def JOB_NAMES=sh (script: """echo ${kustomize_Git.split("/")[-1]} | cut -d . -f 1""",returnStdout: true).trim() def Projects_Area="dev" def apps_name="java-demo" def projectGroup="java-demo" def PACK_PATH="/usr/local/package" }并且在容器内生成一个密钥，而后加到gitlab中，以供git拉取和上传bash-5.1# ssh-keygen -t rsa而后在复制到/var/jenkins_home下，并且挂载到容器内- /data/jenkins-latest/jenkins_home/.ssh:/root/.ssh第一次拉取需要输入yes，我们规避它echo ' Host * StrictHostKeyChecking no UserKnownHostsFile=/dev/null' >>/root/.ssh/config如果你使用的是宿主机运行的Jenkins，这一步可省略因为资源不足的问题，我们手动修改副本数为1流水线阶段，步骤大致如下：1.判断本地是否有git的目录，如果有就删除2.拉取git，并切换到分支3.追加当前的镜像版本到一个buildhistory的文件中4.cd到目录中修改镜像5.修改完成后上传修改你被人6.kustomize和kubectl应用配置清单代码快如下： stage('Deploy') { steps { sh ''' [ ! -d ${JOB_NAMES} ] || rm -rf ${JOB_NAMES} } git clone ${kustomize_Git} && cd ${JOB_NAMES} && git checkout ${apps_name} echo "push latest images: $IPATH" echo "`date +%F-%T` imageTag: $IPATH buildId: ${BUILD_NUMBER} " >> ./buildhistory-$Projects_Area-${apps_name}.log cd overlays/$Projects_Area ${PACK_PATH}/kustomize edit set image $IPATH cd ../.. git add . git config --global push.default matching git config user.name zhengchao.tang git config user.email usertzc@163.com git commit -m "image tag $IPATH-> ${imageUrlPath}" git push -u origin ${apps_name} ${PACK_PATH}/kustomize build overlays/$Projects_Area/ | ${PACK_PATH}/kubectl --kubeconfig /var/jenkins_home/.kube/config-1.23.1-dev apply -f - ''' } } 观测状态配置清单被生效后，不一定符合预期，此时有很多种情况出现，特别是在使用原生的这些命令和脚本更新的时候我们需要追踪更新后的状态，以便于我们随时做出正确的动作。我此前写过一篇关于kubernetes检测pod部署状态简单实现，如果感兴趣可以查看仍然使用此前的方式，如下 stage('status watch') { steps { sh ''' ${PACK_PATH}/kubectl --kubeconfig /var/jenkins_home/.kube/config-1.23.1-dev -n ${projectGroup} rollout status deployment ${apps_name} --watch --timeout=10m ''' } }构建一次到服务器上查看[root@linuxea-11 ~]# kubectl -n java-demo get pod NAME READY STATUS RESTARTS AGE java-demo-66b98564f6-xsc6z 1/1 Running 0 9m24s其他参考kubernetes中skywalking9.0部署使用,kustomize变量引入

linuxea:jenkins流水线集成sonar分支扫描/关联gitlab/docker和mvn打包配置二(8) 在前面的jenkins流水线集成juit/sonarqube/覆盖率扫描配置一中介绍了juilt，覆盖率以及soanrqube的一些配置实现。接着上一篇中，我们继续。阅读此篇，你将了解如下列表中简单的实现方式：jenkins和gitlab触发(上一章已实现)jenkins凭据使用(上一章已实现)juit配置(上一章已实现)sonarqube简单扫描(上一章已实现)sonarqube覆盖率(上一章已实现)打包基于java的skywalking agent(上一章已实现)sonarqube与gitlab关联 (本章实现)配置docker中构建docker (本章实现)mvn打包 (本章实现)sonarqube简单分支扫描(本章实现)基于gitlab来管理kustomize的k8s配置清单kubectl部署kubeclt deployment的状态跟踪钉钉消息的构建状态推送4.6 分支扫描我们可能更希望扫描某一个分支，于是我们需要sonarqube-community-branch-plugin插件我们在https://github.com/mc1arke/sonarqube-community-branch-plugin/releases中，留意支持的版本Note: This version supports Sonarqube 8.9 and above. Sonarqube 8.8 and below or 9.0 and above are not supported in this release使用下表查找每个 SonarQube 版本的正确插件版本SonarQube 版本插件版本9.1+1.12.09.01.9.08.91.8.28.7 - 8.81.7.08.5 - 8.61.6.08.2 - 8.41.5.08.11.4.07.8 - 8.01.3.27.4 - 7.71.0.2于是，我们在nexus3上下载1.8.1版本https://github.com/mc1arke/sonarqube-community-branch-plugin/releases/download/1.8.0/sonarqube-community-branch-plugin-1.8.0.jar 或者 https://github.91chifun.workers.dev//https://github.com/mc1arke/sonarqube-community-branch-plugin/releases/download/1.8.0/sonarqube-community-branch-plugin-1.8.0.jar根据安装提示https://github.com/mc1arke/sonarqube-community-branch-plugin#manual-install而后直接将 jar包下载在/data/sonarqube/extensions/plugins/下即可wget http://172.16.100.48/jenkins/sonar-plugins/sonarqube-community-branch-plugin-1.8.0.jar -o /data/sonarqube/extensions/plugins/sonarqube-community-branch-plugin-1.8.0.jar实际上/data/sonarqube/extensions/目录被挂载到nexus的容器内的/opt/sonarqube/extensions下而容器内的位置是不变的，因此挂载映射关系如下： volumes: - /etc/localtime:/etc/localtime - /data/sonarqube/conf:/opt/sonarqube/conf - /data/sonarqube/extensions:/opt/sonarqube/extensions - /data/sonarqube/logs:/opt/sonarqube/logs - /data/sonarqube/data:/opt/sonarqube/data[root@linuxea-47 /data/sonarqube/extensions]# ll plugins/ total 17552 -rwx------ 1 1000 1000 10280677 Oct 10 2021 sonar-gitlab-plugin-4.1.0-SNAPSHOT.jar -rwx------ 1 1000 1000 61903 Sep 11 2021 sonar-l10n-zh-plugin-8.9.jar -rwx------ 1 1000 1000 7623167 Oct 10 2021 sonarqube-community-branch-plugin-1.8.0.jar而后，我们在本地是/data/sonarqube/conf下的创建一个配置文件sonar.properties，内容如下sonar.web.javaAdditionalOpts=-javaagent:./extensions/plugins/sonarqube-community-branch-plugin-1.8.0.jar=web sonar.ce.javaAdditionalOpts=-javaagent:./extensions/plugins/sonarqube-community-branch-plugin-1.8.0.jar=ce这个配置文件被映射到容器内的/opt/sonarqube/conf进入容器查看[root@linuxea-47 /data/sonarqube]# ls extensions/plugins/ -ll total 17552 -rwx------ 1 1000 1000 61903 Sep 11 2021 sonar-l10n-zh-plugin-8.9.jar -rwx------ 1 1000 1000 7623167 Oct 10 2021 sonarqube-community-branch-plugin-1.8.0.jar分支扫描参数增加 –Dsonar.branch.name=-Dsonar.branch.name=master那现在的projetctkey就不需要加分支名字了 -Dsonar.projectKey=${JOB_NAME}_${branch} \ -Dsonar.projectName=${JOB_NAME}_${branch} \直接在一个项目中就可以看到多个分支的扫描结果了 stage("coed sonar"){ steps{ script { withCredentials([string(credentialsId: 'sonarqube-token', variable: 'SONAR_TOKEN')]) { sh """ cd linuxea && \ /usr/local/package/sonar-scanner-4.6.2.2472-linux/bin/sonar-scanner \ -Dsonar.host.url=http://172.16.100.47:9000 \ -Dsonar.projectKey=${JOB_NAME} \ -Dsonar.projectName=${JOB_NAME} \ -Dsonar.projectVersion=${BUILD_NUMBER} \ -Dsonar.login=${SONAR_TOKEN} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=${env.BASEURL} \ -Dsonar.links.ci=${BUILD_URL} \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.core.codeCoveragePlugin=jacoco \ -Dsonar.jacoco.reportPaths=target/jacoco.exec \ -Dsonar.branch.name=${branch} """ } } } }此时我们分别构建master和web后，在sonarqube的UI中就会有两个分支的扫描结果注意事项如果你使用的是不同的版本，而不同的版本配置是不一样的。见github的每个分支，比如：1.5.04.7 关联gitlab在https://github.com/gabrie-allaigre/sonar-gitlab-plugin下载插件，参阅用法中版本对应，我们下载4.1.0https://github.com/gabrie-allaigre/sonar-gitlab-plugin/releases/download/4.1.0/sonar-gitlab-plugin-4.1.0-SNAPSHOT.jar而后仍然存放到sonarqube的plugin目录下[root@linuxea-47 ~]# ls /data/sonarqube/extensions/plugins/ -ll total 17552 -rwx------ 1 1000 1000 10280677 Oct 10 2021 sonar-gitlab-plugin-4.1.0-SNAPSHOT.jar -rwx------ 1 1000 1000 61903 Sep 11 2021 sonar-l10n-zh-plugin-8.9.jar -rwx------ 1 1000 1000 7623167 Oct 10 2021 sonarqube-community-branch-plugin-1.8.0.jar这在启动的时候，实际上可以看到日志加载根据文档，要完成扫描必须提供如下必要参数-Dsonar.gitlab.commit_sha=1632c729e8f78f913cbf0925baa2a8c893e4473b \ 版本sha -Dsonar.gitlab.ref_name=master \ 分支 -Dsonar.gitlab.project_id=16 \ 项目id -Dsonar.dynamicAnalysis=reuseReports \ 扫描方式 -Dsonar.gitlab.failure_notification_mode=commit-status \ 更改提交状态 -Dsonar.gitlab.url=http://192.168.1.200 \ gitlab地址 -Dsonar.gitlab.user_token=k8xLe6dYTzdtoewSysmy \ gitlab token -Dsonar.gitlab.api_version=v41.配置一个全局token至少需要如下权限令牌如下K8DtxxxifxU1gQeDgvDK其他信息根据现有的项目输入即可-Dsonar.gitlab.commit_sha=4a5bb3db1c845cddc86290d137ef694b3b076d0e \ 版本sha -Dsonar.gitlab.ref_name=master \ 分支 -Dsonar.gitlab.project_id=19 \ 项目id -Dsonar.dynamicAnalysis=reuseReports \ 扫描方式 -Dsonar.gitlab.failure_notification_mode=commit-status \ 更改提交状态 -Dsonar.gitlab.url=http://172.16.100.47 \ gitlab地址 -Dsonar.gitlab.user_token=K8DtxxxifxU1gQeDgvDK \ gitlab token -Dsonar.gitlab.api_version=v42.将上述命令添加到sonarqube的流水线中/var/jenkins_home/package/sonar-scanner/bin/sonar-scanner \ -Dsonar.host.url=http://172.16.15.136:9000 \ -Dsonar.projectKey=java-demo \ -Dsonar.projectName=java-demo \ -Dsonar.projectVersion=120 \ -Dsonar.login=636558affea60cc5f264247de36e7c27c817530b \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=http://172.16.15.136:180/devops/java-demo.git \ -Dsonar.links.ci=http://172.16.15.136:8088/job/java-demo/120/ \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.branch.name=main \ -Dsonar.gitlab.commit_sha=9353e89a7b42e0d93ddf95520408ecfde9a5144a \ -Dsonar.gitlab.ref_name=main \ -Dsonar.gitlab.project_id=2 \ -Dsonar.dynamicAnalysis=reuseReports \ -Dsonar.gitlab.failure_notification_mode=commit-status \ -Dsonar.gitlab.url=http://172.16.15.136:180 \ -Dsonar.gitlab.user_token=9mszu2KXx7nHXiwJveBs \ -Dsonar.gitlab.api_version=v4运行测试正常是什么样的呢，换一个环境配置下/usr/local/package/sonar-scanner-4.6.2.2472-linux/bin/sonar-scanner \ -Dsonar.host.url=http://172.16.100.47:9000 \ -Dsonar.projectKey=java-demo \ -Dsonar.projectName=java-demo \ -Dsonar.projectVersion=20 \ -Dsonar.login=bc826f124d691127c351388274667d7deb1cc9b2 \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=www.baidu.com \ -Dsonar.links.ci=20 \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.core.codeCoveragePlugin=jacoco \ -Dsonar.jacoco.reportPaths=target/jacoco.exec \ -Dsonar.branch.name=master \ -Dsonar.gitlab.commit_sha=4a5bb3db1c845cddc86290d137ef694b3b076d0e \ -Dsonar.gitlab.ref_name=master \ -Dsonar.gitlab.project_id=19 \ -Dsonar.dynamicAnalysis=reuseReports \ -Dsonar.gitlab.failure_notification_mode=commit-status \ -Dsonar.gitlab.url=http://172.16.100.47 \ -Dsonar.gitlab.user_token=K8DtxxxifxU1gQeDgvDK \ -Dsonar.gitlab.api_version=v4 执行之后INFO: SCM Publisher SCM provider for this project is: git INFO: SCM Publisher 2 source files to be analyzed INFO: SCM Publisher 2/2 source files have been analyzed (done) | time=704ms INFO: CPD Executor 2 files had no CPD blocks INFO: CPD Executor Calculating CPD for 0 files INFO: CPD Executor CPD calculation finished (done) | time=0ms INFO: Analysis report generated in 42ms, dir size=74 KB INFO: Analysis report compressed in 14ms, zip size=13 KB INFO: Analysis report uploaded in 468ms INFO: ANALYSIS SUCCESSFUL, you can browse http://172.16.100.47:9000/dashboard?id=java-demo&branch=master INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report INFO: More about the report processing at http://172.16.100.47:9000/api/ce/task?id=AYHOP018DZyaRsN1subY INFO: Executing post-job 'GitLab Commit Issue Publisher' INFO: Waiting quality gate to complete... INFO: Quality gate status: OK INFO: Duplicated Lines : 0 INFO: Lines of Code : 18 INFO: Report status=success, desc=SonarQube reported QualityGate is ok, with 2 ok, no issues INFO: Analysis total time: 7.130 s INFO: ------------------------------------------------------------------------ INFO: EXECUTION SUCCESS INFO: ------------------------------------------------------------------------ INFO: Total time: 7.949s INFO: Final Memory: 17M/60M INFO: ------------------------------------------------------------------------流水线已通过3.获取参数现在的问题是，手动输入gitlab的这些值不可能在jenkins中输入，我们需要自动获取这些。分支的环境变量通过传递来，用变量获取即可commit_sha通过读取当前代码中的文件实现gitlab token放到密钥管理当中于是，我们通过jq来获取格式化gitlab api返回值获取缺省的项目id需要下载一个jq程序在jenkins节点上。于是我们在https://stedolan.github.io/jq/download/页面下载一个 binaries二进制的即可https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64获取项目id curl --silent --header "PRIVATE-TOKEN: K8DtxxxifxU1gQeDgvDK" "http://gitlab.marksugar.com/api/v4/projects?simple=true"| jq -rc '.[]|select(.name == "java-demo")'|jq .id示例1：如果项目名称在所有组内是唯一的，就可以使用jq -rc '.[]|select(.name == "java-demo")',如下.name == "java-demo"： 项目名curl --silent --header "PRIVATE-TOKEN: K8DtxxxifxU1gQeDgvDK" "http://gitlab.marksugar.com/api/v4/projects?simple=true"| jq -rc '.[]|select(.name == "java-demo")' | jq .id示例2：如果项目名称在所有组内不是唯一，且有多个的，用jq -rc '.[]|select(.path_with_namespace == "java/java-demo")',如下.path_with_namespace == java/java-demo ： 组名/项目名curl --silent --header "PRIVATE-TOKEN: K8DtxxxifxU1gQeDgvDK" "http://gitlab.marksugar.com/api/v4/projects?simple=true"| jq -rc '.[]|select(.path_with_namespace == "java/java-demo")'|jq .id获取当前的sha版本号获取办版本号只需要在当前项目目录内读取文件或者命令即可，it log --pretty=oneline|head -1| cut -b 1-40，如下[root@linuxea-48 /data/jenkins-latest/jenkins_home/workspace/linuxea-2022]# git log --pretty=oneline|head -1| cut -b 1-40 4a5bb3db1c845cddc86290d137ef694b3b076d0e除此之外使用cut -b -40 .git/refs/remotes/origin/master 能获得一样的效果[root@linuxea-48 /data/jenkins-latest/jenkins_home/workspace/linuxea-2022]# cut -b -40 .git/refs/remotes/origin/master 4a5bb3db1c845cddc86290d137ef694b3b076d0e项目名称项目名称，我们可以使用Jenkins的项目名字。但是，这个名字有时候未必和git的项目名称一样，于是，我们直接截取项目的地址名称JOB_NAMES=sh (script: """echo ${BASEURL.split("/")[-1]} | cut -d . -f 1""",returnStdout: true).trim() 那么现在已经具备上面的几个关键参数，现在分别命名GIT_COMMIT_TAGSHA和Projects_GitId,JOB_NAMESenvironment { def GIT_COMMIT_TAGSHA=sh (script: """cut -b -40 .git/refs/remotes/origin/master""",returnStdout: true).trim() def JOB_NAMES=sh (script: """echo ${BASEURL.split("/")[-1]} | cut -d . -f 1""",returnStdout: true).trim() def Projects_GitId=sh (script: """curl --silent --header "PRIVATE-TOKEN: zrv1vpfZTtEFCJGrJczB" "http://gitlab.marksugar.com/api/v4/projects?simple=true"| ${buildMap["jq"]} -rc '.[]|select(.path_with_namespace == "java/java-demo")'| ${buildMap["jq"]} .id""",returnStdout: true).trim() }那么现在的环境变量就是 environment { def tag_time = new Date().format("yyyyMMddHHmm") def IPATH="harbor.marksugar.com/java/${JOB_NAME}:${tag_time}" def GIT_COMMIT_TAGSHA=sh (script: """cut -b -40 .git/refs/remotes/origin/master""",returnStdout: true).trim() def JOB_NAMES=sh (script: """echo ${BASEURL.split("/")[-1]} | cut -d . -f 1""",returnStdout: true).trim() def Projects_GitId=sh (script: """curl --silent --header "PRIVATE-TOKEN: zrv1vpfZTtEFCJGrJczB" "http://gitlab.marksugar.com/api/v4/projects?simple=true"| ${buildMap["jq"]} -rc '.[]|select(.path_with_namespace == "java/java-demo")'| ${buildMap["jq"]} .id""",returnStdout: true).trim() def SONAR_git_TOKEN="K8DtxxxifxU1gQeDgvDK" def GitLab_Address="http://172.16.100.47" } 而新增的调用的命令如下 -Dsonar.gitlab.commit_sha=${GIT_COMMIT_TAGSHA} \ -Dsonar.gitlab.ref_name=${branch} \ -Dsonar.gitlab.project_id=${Projects_GitId} \ -Dsonar.dynamicAnalysis=reuseReports \ -Dsonar.gitlab.failure_notification_mode=commit-status \ -Dsonar.gitlab.url=${GitLab_Address} \ -Dsonar.gitlab.user_token=${SONAR_git_TOKEN} \ -Dsonar.gitlab.api_version=v4 构建一次能够看到已经获取到的值，构建成功的完整的阶段代码如下： stage("coed sonar"){ environment { def GIT_COMMIT_TAGSHA=sh (script: """cut -b -40 .git/refs/remotes/origin/master""",returnStdout: true).trim() def JOB_NAMES=sh (script: """echo ${BASEURL.split("/")[-1]} | cut -d . -f 1""",returnStdout: true).trim() def Projects_GitId=sh (script: """curl --silent --heade "PRIVATE-TOKEN: zrv1vpfZTtEFCJGrJczB" "http://gitlab.marksugar.com/api/v4/projects?simple=true"| /usr/local/package/jq-1.6/jq -rc '.[]|select(.path_with_namespace == "java/java-demo")'| /usr/local/package/jq-1.6/jq .id""",returnStdout: true).trim() def SONAR_git_TOKEN="K8DtxxxifxU1gQeDgvDK" def GitLab_Address="http://172.16.100.47" } steps{ script { withCredentials([string(credentialsId: 'sonarqube-token', variable: 'SONAR_TOKEN')]) { sh """ cd linuxea && \ /usr/local/package/sonar-scanner-4.6.2.2472-linux/bin/sonar-scanner \ -Dsonar.host.url=${GitLab_Address}:9000 \ -Dsonar.projectKey=${JOB_NAME} \ -Dsonar.projectName=${JOB_NAME} \ -Dsonar.projectVersion=${BUILD_NUMBER} \ -Dsonar.login=${SONAR_TOKEN} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=${env.BASEURL} \ -Dsonar.links.ci=${BUILD_URL} \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.core.codeCoveragePlugin=jacoco \ -Dsonar.jacoco.reportPaths=target/jacoco.exec \ -Dsonar.branch.name=${branch} \ -Dsonar.gitlab.commit_sha=${GIT_COMMIT_TAGSHA} \ -Dsonar.gitlab.ref_name=${branch} \ -Dsonar.gitlab.project_id=${Projects_GitId} \ -Dsonar.dynamicAnalysis=reuseReports \ -Dsonar.gitlab.failure_notification_mode=commit-status \ -Dsonar.gitlab.url=${GitLab_Address} \ -Dsonar.gitlab.user_token=${SONAR_git_TOKEN} \ -Dsonar.gitlab.api_version=v4 """ } } } }4.8 mvn 打包我们是哟个一条命令直接进行打包-Dmaven.test.skip=true，不执行测试用例，也不编译测试用例类-Dmaven.test.failure.ignore=true ，忽略单元测试失败-s ~/.m2/settings.xml，指定mvn构建的配置文件位置mvn clean install -Dautoconfig.skip=true -Dmaven.test.skip=false -Dmaven.test.failure.ignore=true -s /var/jenkins_home/.m2/settings.xml阶段如下 stage("mvn build"){ steps { script { sh """ cd linuxea mvn clean install -Dautoconfig.skip=true -Dmaven.test.skip=false -Dmaven.test.failure.ignore=true -s /var/jenkins_home/.m2/settings.xml """ } } }4.9 推送镜像我们先需要将docker配置好，首先容器内需要安装docker，而后挂载socket如果你的系统是和容器系统的库文件一样，你可以将本地的docker二进制文件挂载到容器内，但是我使用的是alpine，因此我在容器内安装了docker，此时只需要挂载目录和sock即可也可以将docker挂载到容器内即可 - /usr/bin/docker:/usr/bin/docker - /etc/docker:/etc/docker - /var/run/docker.sock:/var/run/docker.sock并在容器内登录docker容器内登录，或者在流水线阶段中登录也可以[root@linuxea-48 /data/jenkins-latest/jenkins_home]# docker exec -it jenkins bash bash-5.1# cat ~/.docker/config.json { "auths": { "harbor.marksugar.com": { "auth": "YWRtaW46SGFyYm9yMTIzNDU=" } } }将配置复制到主机并挂载到容器内，或者在主机登录挂载到容器都可以- /data/jenkins-latest/.docker:/root/.docker能够在容器内查看docker命令bash-5.1# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 536cb1dbeb3f registry.cn-hangzhou.aliyuncs.com/marksugar/jenkins:2.332-3-alpine-ansible-maven3-nodev16.15-latest "/sbin/tini -- /usr/…" About an hour ago Up About an hour jenkins而后配置docker推送阶段开始之前要配置环境变量，用于获取镜像的时间tag_time随机时间 agent any environment { def tag_time = new Date().format("yyyyMMddHHmm") def IPATH="harbor.marksugar.com/java/${JOB_NAME}:${tag_time}" }docker阶段请注意：此时在COPY skywalking-agent的时候，需要将包拷贝到当前目录才能COPY到容器内 stage("docker build"){ steps{ script{ sh """ cd linuxea docker ps -a cp -r /usr/local/package/skywalking-agent ./ docker build -f ./Dockerfile -t $IPATH . docker push $IPATH docker rmi -f $IPATH """ } } }与此同时需要修改Dockerfile中的COPY 目录而后创建harbor仓库开始构建一旦构建完成，镜像将会推送到harbor仓库此时的pipeline流水线i清单如下try { if ( "${onerun}" == "gitlabs"){ println("Trigger Branch: ${info_ref}") RefName="${info_ref.split("/")[-1]}" //自定义显示名称 currentBuild.displayName = "#${info_event_name}-${RefName}-${info_checkout_sha}" //自定义描述 currentBuild.description = "Trigger by user ${info_user_username} 自动触发 \n branch: ${RefName} \n commit message: ${info_commits_0_message}" BUILD_TRIGGER_BY="${info_user_username}" BASEURL="${info_project_git_http_url}" } }catch(e){ BUILD_TRIGGER_BY="${currentBuild.getBuildCauses()[0].userId}" currentBuild.description = "Trigger by user ${BUILD_TRIGGER_BY} 非自动触发 \n branch: ${branch} \ngit: ${BASEURL}" } pipeline{ //指定运行此流水线的节点 agent any environment { def tag_time = new Date().format("yyyyMMddHHmm") def IPATH="harbor.marksugar.com/java/${JOB_NAME}:${tag_time}" } //管道运行选项 options { skipDefaultCheckout true skipStagesAfterUnstable() buildDiscarder(logRotator(numToKeepStr: '2')) } //流水线的阶段 stages{ //阶段1 获取代码 stage("CheckOut"){ steps { script { println("下载代码 --> 分支： ${env.branch}") checkout( [$class: 'GitSCM', branches: [[name: "${branch}"]], extensions: [], userRemoteConfigs: [[ credentialsId: 'gitlab-mark', url: "${BASEURL}"]]]) } } } stage("unit Test"){ steps{ script{ sh """ cd linuxea && mvn test -s /var/jenkins_home/.m2/settings.xml2 """ } } post { success { script { junit 'linuxea/target/surefire-reports/*.xml' } } } } stage("coed sonar"){ environment { def GIT_COMMIT_TAGSHA=sh (script: """cut -b -40 .git/refs/remotes/origin/master""",returnStdout: true).trim() def JOB_NAMES=sh (script: """echo ${BASEURL.split("/")[-1]} | cut -d . -f 1""",returnStdout: true).trim() def Projects_GitId=sh (script: """curl --silent --heade "PRIVATE-TOKEN: zrv1vpfZTtEFCJGrJczB" "http://gitlab.marksugar.com/api/v4/projects?simple=true"| /usr/local/package/jq-1.6/jq -rc '.[]|select(.path_with_namespace == "java/java-demo")'| /usr/local/package/jq-1.6/jq .id""",returnStdout: true).trim() def SONAR_git_TOKEN="K8DtxxxifxU1gQeDgvDK" def GitLab_Address="http://172.16.100.47" } steps{ script { withCredentials([string(credentialsId: 'sonarqube-token', variable: 'SONAR_TOKEN')]) { sh """ cd linuxea && \ /usr/local/package/sonar-scanner-4.6.2.2472-linux/bin/sonar-scanner \ -Dsonar.host.url=${GitLab_Address}:9000 \ -Dsonar.projectKey=${JOB_NAME} \ -Dsonar.projectName=${JOB_NAME} \ -Dsonar.projectVersion=${BUILD_NUMBER} \ -Dsonar.login=${SONAR_TOKEN} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=${env.BASEURL} \ -Dsonar.links.ci=${BUILD_URL} \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.core.codeCoveragePlugin=jacoco \ -Dsonar.jacoco.reportPaths=target/jacoco.exec \ -Dsonar.branch.name=${branch} \ -Dsonar.gitlab.commit_sha=${GIT_COMMIT_TAGSHA} \ -Dsonar.gitlab.ref_name=${branch} \ -Dsonar.gitlab.project_id=${Projects_GitId} \ -Dsonar.dynamicAnalysis=reuseReports \ -Dsonar.gitlab.failure_notification_mode=commit-status \ -Dsonar.gitlab.url=${GitLab_Address} \ -Dsonar.gitlab.user_token=${SONAR_git_TOKEN} \ -Dsonar.gitlab.api_version=v4 """ } } } } stage("mvn build"){ steps { script { sh """ cd linuxea mvn clean install -Dautoconfig.skip=true -Dmaven.test.skip=false -Dmaven.test.failure.ignore=true -s /var/jenkins_home/.m2/settings.xml2 """ } } } stage("docker build"){ steps{ script{ sh """ cd linuxea docker ps -a cp -r /usr/local/package/skywalking-agent ./ docker build -f ./Dockerfile -t $IPATH . docker push $IPATH docker rmi -f $IPATH """ } } } } }

linuxea:jenkins流水线集成juit/sonarqube/覆盖率扫描配置一(7) 到了当前阶段流水线阶段，我们需要配置mvn去打包，sonarqube，以及容器的镜像构建，这些东西都会被安排到流水线上，直到docker镜像构建完成，此阶段，我将一个docker镜像构建完成视作为一个制品的完成。这些流程被放置在一起，因此称作持续集成。当然，这在这里只是一个非常简单的模型。阅读此篇，你将了解如下列表中简单的实现方式：jenkins和gitlab触发(上一章实现)jenkins凭据使用(上一章实现)juit配置(本章实现)sonarqube简单扫描(本章实现)sonarqube简单分支扫描(本章实现)sonarqube覆盖率(本章实现)sonarqube与gitlab关联配置docker中构建dockermvn打包打包基于java的skywalking agent(上一章实现)基于gitlab来管理kustomize的k8s配置清单kubectl部署kubeclt deployment的状态跟踪钉钉消息的构建状态推送开始下载一个java程序包另外，为了演示这个效果，我们需要一个合适的jar包来测试，而不是在前面提供的那个hello-world的包。于是在https://start.spring.io/页面默认选择，选择java 8，而后点击CENERATE下载demo包，解压这个包将代码推送到gitlab而后选择ADD最后点击GENERATE，下载一个java-demo的包错误修改如果构建失败，mvn使用aliyun的直接构建即可<mirrors> .... <mirror> <id>nexus-aliyun</id> <mirrorOf>*</mirrorOf> <name>Nexus aliyun</name> <url>http://maven.aliyun.com/nexus/content/groups/public</url> </mirror> </mirrors>而后，我们创建一个新分支来存放cd /tmp git clone git@172.16.100.47:java/java-demo.git cd java-demo git checkout -b web mkdir linuxea而后将java-demo.zip解压复制进去unzip java-demo.zip cp java-demo/* /tmp/java-demo/linuxea/上传到git仓库git add . git commit -m "first commit" git push -u origin web另外还需要将Dockerfile放入4.1 配置凭据你可以使用密钥或者账号密码来进行配置，如果是账号密码需要配置一个Credentials后来进行引用。因此，需要安装Credentials插件install plugins "Credentials"在Manage Jenkins -> Manage Credentials -> 选择全局，而后添加凭据而后复制创建完成的id，将会在拉取代码时候用到，如下 //阶段1 获取代码 stage("CheckOut"){ steps { script { println("下载代码 --> 分支： ${env.branch}") checkout( [$class: 'GitSCM', branches: [[name: "${branch}"]], extensions: [], userRemoteConfigs: [[ credentialsId: 'gitlab-mark', url: "${BASEURL}"]]]) } } }总的如下try { if ( "${onerun}" == "gitlabs"){ println("Trigger Branch: ${info_ref}") RefName="${info_ref.split("/")[-1]}" //自定义显示名称 currentBuild.displayName = "#${info_event_name}-${RefName}-${info_checkout_sha}" //自定义描述 currentBuild.description = "Trigger by user ${info_user_username} 自动触发 \n branch: ${RefName} \n commit message: ${info_commits_0_message}" BUILD_TRIGGER_BY="${info_user_username}" BASEURL="${info_project_git_http_url}" } }catch(e){ BUILD_TRIGGER_BY="${currentBuild.getBuildCauses()[0].userId}" currentBuild.description = "Trigger by user ${BUILD_TRIGGER_BY} 非自动触发 \n branch: ${branch} \ngit: ${BASEURL}" } pipeline{ //指定运行此流水线的节点 agent any //管道运行选项 options { skipDefaultCheckout true skipStagesAfterUnstable() buildDiscarder(logRotator(numToKeepStr: '2')) } //流水线的阶段 stages{ //阶段1 获取代码 stage("CheckOut"){ steps { script { println("下载代码 --> 分支： ${env.branch}") checkout( [$class: 'GitSCM', branches: [[name: "${branch}"]], extensions: [], userRemoteConfigs: [[ credentialsId: 'gitlab-root', url: "${BASEURL}"]]]) } } } } }但是这里面涉及到两个变量，分别是：$branch: 分支这里我们修改为： web$BASEURL： git地址这个就是这个项目地地址这两个参数化构建过程中需要在选项参数中进行添加4.2 配置mvn源地址如果没有配置mvn的,参考如下.m2下的settings.xml文件不好用了，或者反正就不好用了，我们直接配置绝对路径找到mirrors字段，字段内全部删除，复制下面的粘贴进去即可<mirrors> <mirror> <id>alimaven</id> <name>aliyun maven</name> <url>http://maven.aliyun.com/nexus/content/groups/public/</url> <mirrorOf>central</mirrorOf> </mirror> </mirrors>像这样，而后存放在一个位置<?xml version="1.0" encoding="UTF-8"?> <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"> <pluginGroups> </pluginGroups> <proxies> </proxies> <servers> <server> <id>maven-releases</id> <username>admin</username> <password>admin</password> </server> <server> <id>maven-snapshots</id> <username>admin</username> <password>admin</password> </server> </servers> <mirrors> <!-- <mirror> <id>nexus</id> <mirrorOf>local</mirrorOf> <name>nexus</name> <url>http://172.16.15.136:8081/repository/maven-public/</url> </mirror>--> <mirror> <id>alimaven</id> <name>aliyun maven</name> <url>http://172.16.15.136:8081/repository/maven2-group/</url> <mirrorOf>central</mirrorOf> </mirror> </mirrors> <profiles> </profiles> </settings>4.3 juit测试pom文件增加依赖 <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> 我们的目录结构中，首先是一个Linuxea的目录，于是我们每次都需要cd进去 stage("unit Test"){ steps{ script{ sh """ cd linuxea && mvn test -s /var/jenkins_home/.m2/settings.xml """ } } post { success { script { junit 'linuxea/target/surefire-reports/*.xml' } } } }正常情况下会在surefire-reports下创建一个xml文件，jenkins读取此文件[root@linuxea-48 /data/jenkins-latest/jenkins_home/workspace/linuxea-2022/linuxea/target]# ll total 0 drwxr-xr-x 3 root root 47 Jul 3 18:31 classes drwxr-xr-x 3 root root 25 Jul 3 01:30 generated-sources drwxr-xr-x 3 root root 30 Jul 3 01:45 generated-test-sources drwxr-xr-x 3 root root 35 Jul 3 01:40 maven-status drwxr-xr-x 2 root root 257 Jul 3 18:32 surefire-reports drwxr-xr-x 3 root root 17 Jul 3 18:31 test-classes如果代码里没有单元测试，这里是没有报告产生的，并且有可能因为版本问题会报错现在我们将web合并到master，开始构建。最终如下4.4 sonarqube简单使用，安装插件即可，SonarQube Scanner，或者下载SonarQube Scanner安装挂载到容器内1.首先我们配置token信息和Jenkins互联复制token01c2f60766bd896fe82a378bb105e1a73d9161c32.回到jenkins添加凭据插件配置如果此时用的插件就可以这么配置，在Manage Jenkins->Configure System流水线阶段如下 stage('SonarQube analysis') { // sonarqube installation name steps{ withSonarQubeEnv( installationName: 'sonarqube-servers') { sh """ cd demo && mvn clean verify sonar:sonar -s /var/jenkins_home/.m2/settings.xml """ } } }不使用插件(现在就是)不用插件就需要下载sonar-scanner-cli包使用，如下wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.6.2.2472-linux.zip unzi sonar-scanner-cli-4.6.2.2472-linux.zip -d /usr/local/ ln -s /usr/local/sonar-scanner-4.6.2.2472-linux/bin/sonar-scanner /usr/local/sbin/ sed -i 's/use_embedded_jre=true/use_embedded_jre=false/g' /usr/local/sonar-scanner-4.6.2.2472-linux/bin/sonar-scanner这个包最终是被挂载到容器的/usr/local/package/下如果不用插件，用命令组合变量，因此需要生成一下命令的格式在流水线语法中-选中"withCredentials; Build credentials to variables"，而后绑定到"sonarqube-token"变量中将生成的流水线脚本复制，并进行修改 stage("coed sonar"){ steps{ script { withCredentials([string(credentialsId: 'sonarqube-token', variable: 'SONAR_TOKEN')]) { sh """ cd linuxea && \ /usr/local/package/sonar-scanner-4.6.2.2472-linux/bin/sonar-scanner \ -Dsonar.host.url=http://172.16.100.47:9000 \ -Dsonar.projectKey=${JOB_NAME}_${branch} \ -Dsonar.projectName=${JOB_NAME}_${branch} \ -Dsonar.projectVersion=${BUILD_NUMBER} \ -Dsonar.login=${SONAR_TOKEN} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=${env.BASEURL} \ -Dsonar.links.ci=${BUILD_URL} \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports """ } } } }此阶段运行后如下现在在sonarqube生成的就是master分支的4.5 代码覆盖率在8.9.2的版本里面jacoco默认是有的，因此我们不需要额外插件但是仍然需要配置pom.xml中的dependency，如下 <dependency> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.8.3</version> <scope>test</scope> </dependency>还有一个plugins <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> </plugin> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.8.2</version> <configuration> <destFile>target/jacoco.exec</destFile> <detaFile>target/jacoco.exec</detaFile> </configuration> <executions> <execution> <id>jacoco-initialize</id> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>jacoco-site</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution> </executions> </plugin>修改完成的pom.xml如下<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.7.1</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>com.linuxea</groupId> <artifactId>java-demo</artifactId> <version>0.0.1-SNAPSHOT</version> <name>java-demo</name> <description>Demo project for Spring Boot</description> <properties> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.8.3</version> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> </plugin> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.8.2</version> <configuration> <destFile>target/jacoco.exec</destFile> <detaFile>target/jacoco.exec</detaFile> </configuration> <executions> <execution> <id>jacoco-initialize</id> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>jacoco-site</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project>而后在构建的命令中添加如下-Dsonar.core.codeCoveragePlugin=jacoco \ -Dsonar.jacoco.reportPaths=target/jacoco.exec添加到流水线 stage("coed sonar"){ steps{ script { withCredentials([string(credentialsId: 'sonarqube-token', variable: 'SONAR_TOKEN')]) { sh """ cd linuxea && \ /usr/local/package/sonar-scanner-4.6.2.2472-linux/bin/sonar-scanner \ -Dsonar.host.url=http://172.16.100.47:9000 \ -Dsonar.projectKey=${JOB_NAME}_${branch} \ -Dsonar.projectName=${JOB_NAME}_${branch} \ -Dsonar.projectVersion=${BUILD_NUMBER} \ -Dsonar.login=${SONAR_TOKEN} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=${env.BASEURL} \ -Dsonar.links.ci=${BUILD_URL} \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.core.codeCoveragePlugin=jacoco \ -Dsonar.jacoco.reportPaths=target/jacoco.exec """ } } } }构建后sonarqube的覆盖率将会出现值如果没有配置测试用例，这里是0

linuxea:gitlab和jenkins自动和手动触发构建(6) 在前面几章里面，我们配置了基本的组件，围绕java构建配置了nexus3，配置了skywalking，能够打包和构建镜像，但是我们需要将这些串联起来，组成一个流水线，并且需要将skywalking的agent打包在镜像内，并配置必要的参数。与此同时，我们使用一个简单的实现方式用作在jenkins上，那就是pipeline和部分groovy语法的函数，至少来完成一下场景。场景1： A方希望提交代码或者打TAG来触发jenkins构建，在构建之前使用sonarqube进行代码扫描，并且配置简单的阈值。而后去上述的流水线整合。按道理，sonarqube的配置是有一些讲究的，处于整体考虑sonarqube只用作流水线管道的一部分，本次不去考虑sonarqube的代码扫描策略，也不会将扫描结果关联到gitlab，只是仅仅将文件反馈到Jenkins。这些在后面如果有时间在进行配置在本次中我只仅仅使用pipeline，并不是共享库。阅读此篇，你将了解如下列表中简单的实现方式：jenkins和gitlab触发(本章实现)jenkins凭据使用(本章实现)juit配置sonarqube简单扫描配置docker中构建docker打包基于java的skywalking agent(本章实现)基于gitlab来管理kustomize的k8s配置清单kubectl部署kubeclt deployment的状态跟踪钉钉消息的构建状态推送拓扑如下图：1.添加skywalking agent此前在基于nexus3代码构建和harbor镜像打包(3)一骗你中，我们已经有了一个java-hello-world的包，提供了一个8086的端口，并且我们将Dockerfile之类的都已准备妥当，此时延续此前的流程继续走。如果没有，在此页面克隆。1.现在我们下载一个skywaling的agent(8.11.0)端来到Dockerfile中，要实现，需要下载包到jenkins服务器上，或者打在镜像内。https://www.apache.org/dyn/closer.cgi/skywalking/java-agent/8.11.0/apache-skywalking-java-agent-8.11.0.tgz2.gitlab上创建一个java组，创建一个java-demo的项目，将代码和代码中的Dockerfile推送到gitlab仓库中3.在Dockerfile中添加COPY agent，并在启动的时候添加到启动命令中，如下docker-compose中映射关系中，/data/jenkins-latest/package:/usr/local/package。于是我们将skywalking包存放在/data/jenkins-latest/package下，而后在Dockerfile中/usr/local/package的路径即可COPY /usr/local/package/skywalking-agent /skywalking-agent而后启动的时候引入到启动命令中 -javaagent:/skywalking-agent/skywalking-agent.jarCMD java ${JAVA_OPTS} -javaagent:/skywalking-agent/skywalking-agent.jar -jar *.jarDockerfile如下我们需要修改下目录 结构，提前创建/skywalking-agent/logs并且授权并且，skywalking-agent目录需要提前在流水线中复制到当前目录中来FROM registry.cn-hangzhou.aliyuncs.com/marksugar/jdk:8u202 MAINTAINER by mark ENV JAVA_OPTS="\ -server \ -Xms2048m \ -Xmx2048m \ -Xmn512m \ -Xss256k \ -XX:+UseConcMarkSweepGC \ -XX:+UseCMSInitiatingOccupancyOnly \ -XX:CMSInitiatingOccupancyFraction=70 \ -XX:+HeapDumpOnOutOfMemoryError \ -XX:HeapDumpPath=/data/logs" \ MY_USER=linuxea \ MY_USER_ID=316 RUN addgroup -g ${MY_USER_ID} -S ${MY_USER} \ && adduser -u ${MY_USER_ID} -S -H -s /sbin/nologin -g 'java' -G ${MY_USER} ${MY_USER} \ && mkdir /data/logs /skywalking-agent -p COPY target/*.jar /data/ COPY skywalking-agent /skywalking-agent/ RUN chown -R 316.316 /skywalking-agent WORKDIR /data USER linuxea CMD java ${JAVA_OPTS} -javaagent:/skywalking-agent/skywalking-agent.jar -jar *.jar 注意事项我们需要通过trace-ignore-plugin来过滤跟踪系统需要忽略的url，因此我们需要根据trace-ignore-plugin进行配置。这很有必要匹配规则遵循 Ant Path 匹配风格，如 /path/、/path/*、/path/?。将apm-trace-ignore-plugin-x.jar复制到agent/plugins，重启agent即可生效插件。于是，我们将这个插件复制到plugins下tar xf apache-skywalking-java-agent-8.11.0.tar.gz cd skywalking-agent/ cp optional-plugins/apm-trace-ignore-plugin-8.11.0.jar plugins/忽略参数(忽略参数在k8syaml中进行配置)有两种方法可以配置忽略模式。通过系统环境设置具有更高的优先级。1.系统环境变量设置，需要在系统变量中添加skywalking.trace.ignore_path，值为需要忽略的路径，多条路径之间用,分隔2.将/agent/optional-plugins/apm-trace-ignore-plugin/apm-trace-ignore-plugin.config 复制到/agent/config/ 目录下，并添加过滤跟踪的规则trace.ignore_path=/your/path/1/,/your/path/2/4.将gitlab的java-demo项目拉到本地后，将java-helo-word项目文件移动到私有gitlab，并且将Dockerfile放入[root@Node-172_16_100_48 /data]# git clone git@172.16.100.47:java/java-demo.git Cloning into 'java-demo'... remote: Enumerating objects: 3, done. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3 Receiving objects: 100% (3/3), done. [root@Node-172_16_100_48 /data]# mv java-helo-word/* java-demo/ [root@Node-172_16_100_48 /data]# tree java-demo/linuxea/ java-demo/ ├── bbin.png ├── cn-site-service.iml ├── Dockerfile .......... 23 directories, 26 files放入完成后的Dockerfile的内容如下FROM registry.cn-hangzhou.aliyuncs.com/marksugar/jdk:8u202 MAINTAINER by mark ENV JAVA_OPTS="\ -server \ -Xms2048m \ -Xmx2048m \ -Xmn512m \ -Xss256k \ -XX:+UseConcMarkSweepGC \ -XX:+UseCMSInitiatingOccupancyOnly \ -XX:CMSInitiatingOccupancyFraction=70 \ -XX:+HeapDumpOnOutOfMemoryError \ -XX:HeapDumpPath=/data/logs" \ MY_USER=linuxea \ MY_USER_ID=316 RUN addgroup -g ${MY_USER_ID} -S ${MY_USER} \ && adduser -u ${MY_USER_ID} -S -H -s /sbin/nologin -g 'java' -G ${MY_USER} ${MY_USER} \ && mkdir /data/logs /skywalking-agent -p COPY target/*.jar /data/ COPY skywalking-agent /skywalking-agent/ RUN chown -R 316.316 /skywalking-agent WORKDIR /data USER linuxea CMD java ${JAVA_OPTS} -javaagent:/skywalking-agent/skywalking-agent.jar -jar *.jar Dockerfile添加skywalking到此完成。[root@linuxea-48 /data/java-demo]# git add . && git commit -m "first commit" && git push -u origin main [main 2f6d866] first commit 25 files changed, 545 insertions(+) create mode 100644 Dockerfile create mode 100644 bbin.png create mode 100644 cn-site-service.iml create mode 100644 fenghuang.png create mode 100644 index.html create mode 100644 pom.xml create mode 100644 src/main/java/com/dt/info/InfoSiteServiceApplication.java create mode 100644 src/main/java/com/dt/info/controller/HelloController.java create mode 100644 src/main/resources/account.properties create mode 100644 src/main/resources/application.yml create mode 100644 src/main/resources/log4j.properties ........... remote: remote: To create a merge request for main, visit: remote: http://172.16.100.47/java/java-demo/-/merge_requests/new?merge_request%5Bsource_branch%5D=main remote: To git@172.16.100.47:java/java-demo.git * [new branch] main -> main Branch main set up to track remote branch main from origin.代码上传到gitlab后开始配置jenkinsB.new jar你也可以生成一个空的java包来测试准备一个jar包，可以是一个java已有的程序或者下载一个空的，如下在https://start.spring.io/页面默认选择，选择java 8，而后点击CENERATE下载demo包，解压这个包将代码推送到gitlab将项目拉到本地后在上传demo包Administrator@DESKTOP-RD8S1SJ MINGW64 /h/k8s-1.20.2/gitops $ git clone git@172.16.100.47:pipeline-ops/2022-test.git Cloning into '2022-test'... remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (3/3), done. Administrator@DESKTOP-RD8S1SJ MINGW64 /h/k8s-1.20.2/gitops $ unzip demo.zip -d 2022-test $ git add . && git commit -m "first commit" && git push2.关联jenkins和gitlab为了能够实现gitlab自动触发，我们需要配置一个webhook，并且jenkins安装插件来完成。首先我们需要插件，并且在gitlab配置一个webhook，一旦gitlab发生事件后就会触发到jenkins，jenkins启动流水线作业。我将会在流水线作业来判断作业被触发是通过gitlab还是其他的。2.1 jenkins插件安装Generic Webhook Trigger插件，而后点击新建imtes->输入一个名称—>选择pipeline例如，创建了一个Linuxea-2022的项目，勾选了Generic Webhook Trigger，并且在下方的token，输入了一个marksugar测试pipelinepipeline{ //指定运行此流水线的节点 agent any //管道运行选项 options { skipStagesAfterUnstable() } //流水线的阶段 stages{ //阶段1 获取代码 stage("CheckOut"){ steps{ script{ println("获取代码") } } } stage("Build"){ steps{ script{ println("运行构建") } } } } post { always{ script{ println("流水线结束后，经常做的事情") } } success{ script{ println("流水线成功后，要做的事情") } } failure{ script{ println("流水线失败后，要做的事情") } } aborted{ script{ println("流水线取消后，要做的事情") } } } }手动触发测试curl --location --request GET 'http://172.16.100.48:58080/generic-webhook-trigger/invoke/?token=marksugar'运行一次[root@linuxea-47 ~]# curl --location --request GET 'http://172.16.100.48:58080/generic-webhook-trigger/invoke/?token=marksugar' {"jobs":{"linuxea-2022":{"regexpFilterExpression":"","triggered":true,"resolvedVariables":{},"regexpFilterText":"","id":4,"url":"queue/item/4/"}},"message":"Triggered jobs."}You have new mail in /var/spool/mail/root2.2 配置gitlab webhook1.在右上角的preferences中的最下方Localization选择简体中文保存2.管理元->设置->网络->下拉菜单中的“出战请求”勾选 允许来自 web hooks 和服务对本地网络的请求回到gitlab首先进入项目后选择settings->webhooks->urlurl输入http://172.16.100.48:58080/generic-webhook-trigger/invoke/?token=marksugar?marksugar, 问号后面为设置的token而后选中push events和tag push events和最下面的ssl verfication : Enable SSL verification兵点击Add webhook测试在最下方 -> test下拉菜单中选择一个被我们选中过的事件，而后点击。模拟一次push在edit中 -> 最下方 View details 查看Request body，Request body就是发送的内容，这些内容可以被获取到并且解析回到jenkins，查看已经开始构建3.1 自动与手动关联在上面已经配置了自动触发jenkins构建，但是这还不够，我们想在jenkins上体现出来，那一次是自动构建，那一次是手动点击，于是我们添加try和catch因此，我们配置try，并且在现有的阶段添加两个必要的环境变量来应对手动触发3.2 添加手动参数branch: 分支BASEURL：git地址3.2 配置识别try语法我们需要获取请求过来的数据，因此我们获取所有的json请求，配置如下自动触发而后获取的变量方式解析后，我将必要的值进行拼接后如下println("Trigger User: ${info_user_username}") println("Trigger Branch: ${info_ref}" ) println("Trigger event: ${info_event_name}") println("Trigger application: ${info_project_name}") println("Trigger version number: ${info_checkout_sha}") println("Trigger commit message: ${info_commits_0_message}") println("Trigger commit time: ${info_commits_0_timestamp}")而我们只需要部分，因此就变成了如下这般try { println("Trigger Branch: ${info_ref}") RefName="${info_ref.split("/")[-1]}" //自定义显示名称 currentBuild.displayName = "#${info_event_name}-${RefName}-${info_checkout_sha}" //自定义描述 currentBuild.description = "Trigger by user ${info_user_username} 非gitlab自动触发 \n branch: ${RefName} \n commit message: ${info_commits_0_message}" BUILD_TRIGGER_BY="${info_user_username}" BASEURL="${info_project_git_http_url}" }catch(e){ BUILD_TRIGGER_BY="${currentBuild.getBuildCauses()[0].userId}" currentBuild.description = "Trigger by user ${BUILD_TRIGGER_BY} 手动触发 \n branch: ${branch} \n git url: ${BASEURL}" } pipeline{ //指定运行此流水线的节点 agent any //管道运行选项 options { skipDefaultCheckout true skipStagesAfterUnstable() buildDiscarder(logRotator(numToKeepStr: '2')) } //流水线的阶段 stages{ //阶段1 获取代码 stage("env"){ steps{ script{ println("${BASEURL}") } } } } }一旦被触发后，如下图3.3 判断gitlab但这还没玩，尽管此时已经能够识别自动触发了，但是我们无法识别到底是从哪里来的自动触发。但是，我们只需要知道那些是gitlab来的请求和不是gitlab来的请求即可。简单来说，就是需要一个参数来判断触发这次构建的来源。于是，我们配置请求参数来识别判断在Request parameters-> 输入onerunonerun： 用来判断的参数而后在gitlab的url中添加上传递参数http://172.16.100.48:58080/generic-webhook-trigger/invoke/?onerun=gitlabs&token=marksugar这里的onerun=gitlabs，如下我们在try中进行判断即可onerun=gitlabstry { if ( "${onerun}" == "gitlabs"){ println("从带有gitlabs请求来的构建") } }catch(e){ println("从没有带有gitlabs请求来的构建") }在本次中，配置如下try { if ( "${onerun}" == "gitlabs"){ println("Trigger Branch: ${info_ref}") RefName="${info_ref.split("/")[-1]}" //自定义显示名称 currentBuild.displayName = "#${info_event_name}-${RefName}-${info_checkout_sha}" //自定义描述 currentBuild.description = "Trigger by user ${info_user_username} 自动触发 \n branch: ${RefName} \n commit message: ${info_commits_0_message}" BUILD_TRIGGER_BY="${info_user_username}" BASEURL="${info_project_git_http_url}" } }catch(e){ BUILD_TRIGGER_BY="${currentBuild.getBuildCauses()[0].userId}" currentBuild.description = "Trigger by user ${BUILD_TRIGGER_BY} 非自动触发 \n branch: ${branch} \ngit: ${BASEURL}" } pipeline{ //指定运行此流水线的节点 agent any //管道运行选项 options { skipDefaultCheckout true skipStagesAfterUnstable() buildDiscarder(logRotator(numToKeepStr: '2')) } //流水线的阶段 stages{ //阶段1 获取代码 stage("env"){ steps{ script{ println("${BASEURL}") } } } } }手动构建一次通过命令构建[root@linuxea-48 ~]# curl --location --request GET 'http://172.16.100.48:58080/generic-webhook-trigger/invoke/?onerun=gitlabs&token=marksugar' && echo {"jobs":{"linuxea-2022":{"regexpFilterExpression":"","triggered":true,"resolvedVariables":{"info":"","onerun":"gitlabs","onerun_0":"gitlabs"},"regexpFilterText":"","id":14,"url":"queue/item/14/"}},"message":"Triggered jobs."}如下通过这样的配置，我们至少能从jenkins上看到构建的触发类型而已。