Gitea(Git 호스팅 서비스), Harbor(도커 레지스트리), 그리고 ArgoCD(쿠버네티스 애플리케이션 배포 도구)를 사용하여 자동화된 애플리케이션 배포 방법을 소개합니다.
1. Gitea
Gitea Helm Charts 다운로드 및 설치(Minimal)
[root@m-k8s ~]# helm repo add gitea-charts https://dl.gitea.com/charts/
[root@m-k8s ~]# helm repo list
NAME URL
prometheus-community https://prometheus-community.github.io/helm-charts
gitea-charts https://dl.gitea.com/charts/
[root@m-k8s ~]# helm pull gitea-charts/gitea
[root@m-k8s ~]# tar xvfz gitea-10.1.4.tgz
[root@m-k8s ~]# cd gitea
[root@m-k8s gitea]# vi values.yaml
service:
http:
type: LoadBalancer // ClusterIP -> LoadBalancer
redis-cluster:
enabled: false // true -> false
postgresql:
enabled: false// true -> false
postgresql-ha:
enabled: false// true -> false
persistence:
enabled: false// true -> false
gitea:
config: // 아래 내용 추가
database:
DB_TYPE: sqlite3
session:
PROVIDER: memory
cache:
ADAPTER: memory
queue:
TYPE: level
[root@m-k8s gitea]# kubectl create ns gitea
[root@m-k8s gitea]# helm install gitea -n gitea -f values.yaml .
[root@m-k8s gitea]# kubectl get all -n gitea
NAME READY STATUS RESTARTS AGE
pod/gitea-d59784fb-s2l8m 1/1 Running 0 34s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/gitea-http LoadBalancer 10.99.62.47 192.168.1.201 3000:32001/TCP 34s
service/gitea-ssh ClusterIP None <none> 22/TCP 34s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/gitea 1/1 1 1 34s
NAME DESIRED CURRENT READY AGE
replicaset.apps/gitea-d59784fb 1 1 1 34s
브라우저에서 접속 후 로그인
[root@m-k8s gitea]# cat values.yaml | grep -A gitea_admin
username: gitea_admin
password: r8sA8CPHD9!bt6d
LoadBalancer IP 확인 후 접속
[root@m-k8s 021]# k get svc -n gitea NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE gitea-http LoadBalancer 10.101.73.252 192.168.1.154 3000:32270/TCP 2m49s gitea-ssh ClusterIP None <none> 22/TCP 2m49s http://192.168.1.154:3000
Repository 생성하기
우측 + 버튼 클릭
저장소 이름을 지정하고 저장소 만들기
생성한 Repository에 쿠버네티스 manifest 업로드 하기
해당 manifest는 Harbor로 구축한 Private Registry를 이용해 이미지를 가져올 예정입니다.
[root@m-k8s gitea]# cd ~
# git 인증서 검증 비활성화
[root@m-k8s ~]# git config --global http.sslVerify false
# git 유저 정보 설정
[root@m-k8s ~]# git config --global user.name "gitea-admin"
[root@m-k8s ~]# git config --global user.email "gitea@local.domain"
[root@m-k8s ~]# git clone http://192.168.1.152:3000/gitea_admin/my_application.git
Cloning into 'my_application'...
warning: You appear to have cloned an empty repository
[root@m-k8s ~]# cd my_application
[root@m-k8s my_application]# vi mynginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: mynginx-deployment
labels:
app: mynginx
spec:
replicas: 3
selector:
matchLabels:
app: mynginx
template:
metadata:
labels:
app: mynginx
spec:
containers:
- name: mynginx
image: 192.168.1.202/library/myimage:latest ## 다음번 lb ip 예측 (gitea ip +1)
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: mynginx-service
spec:
selector:
app: mynginx
ports:
- protocol: TCP
port: 80
targetPort: 80
type: LoadBalancer
[root@m-k8s my_application]# git add .
[root@m-k8s my_application]# git status
[root@m-k8s my_application]# git commit -m "myngnix manifest upload"
[master (root-commit) e7be4fc] myngnix manifest upload
1 file changed, 34 insertions(+)
[root@m-k8s my_application]# git push origin main
Username for 'http://192.168.1.201:3000': gitea_admin
Password for 'http://gitea_admin@192.168.1.201:3000':
Counting objects: 3, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 454 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: . Processing 1 references
remote: Processed 1 references in total
To http://192.168.1.201:3000/gitea_admin/my_application.git
* [new branch] master -> master
웹에서 업로드 확인
2. Harbor
Harbor Helm Charts 다운로드 및 설치
helm install harbor -n harbor -f values.yaml .
[root@m-k8s ~]# helm repo add harbor https://helm.goharbor.io
[root@m-k8s ~]# helm repo list
NAME URL
prometheus-community https://prometheus-community.github.io/helm-charts
gitea-charts https://dl.gitea.com/charts/
harbor https://helm.goharbor.io
[root@m-k8s ~]# helm pull harbor/harbor
[root@m-k8s ~]# tar xvfz harbor-1.14.2.tgz
[root@m-k8s ~]# cd harbor
[root@m-k8s harbor]# vi values.yaml
expose:
type: loadBalancer // Ingress -> loadBalancer
tls:
enabled: false // true -> False
loadBalancer:
name: harbor
IP: "192.168.1.155" // IP 지정
persistence:
enabled: false // true -> false
externalURL: http://192.168.1.155 // URL 지정
=======================================================
[root@m-k8s harbor]# kubectl create ns harbor
[root@m-k8s harbor]#
[root@m-k8s harbor]# kubectl get all -n harbor
NAME READY STATUS RESTARTS AGE
pod/harbor-core-657496664d-xqf5r 1/1 Running 0 84s
pod/harbor-database-0 1/1 Running 0 84s
pod/harbor-jobservice-78666596dd-ntjcf 1/1 Running 3 (51s ago) 84s
pod/harbor-nginx-5b76d7588d-k4mv9 1/1 Running 0 84s
pod/harbor-portal-68bcb5dd4c-rbdl2 1/1 Running 0 84s
pod/harbor-redis-0 1/1 Running 0 84s
pod/harbor-registry-695b7468f8-gdmmz 2/2 Running 0 84s
pod/harbor-trivy-0 1/1 Running 0 84s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/harbor LoadBalancer 10.97.176.227 192.168.1.155 80:30022/TCP 9m49s 84s
service/harbor-core ClusterIP 10.97.130.242 <none> 80/TCP 84s
service/harbor-database ClusterIP 10.111.247.242 <none> 5432/TCP 84s
service/harbor-jobservice ClusterIP 10.96.71.100 <none> 80/TCP 84s
service/harbor-portal ClusterIP 10.96.111.107 <none> 80/TCP 84s
service/harbor-redis ClusterIP 10.106.39.229 <none> 6379/TCP 84s
service/harbor-registry ClusterIP 10.111.118.33 <none> 5000/TCP,8080/TCP 84s
service/harbor-trivy ClusterIP 10.97.214.78 <none> 8080/TCP 84s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/harbor-core 1/1 1 1 84s
deployment.apps/harbor-jobservice 1/1 1 1 84s
deployment.apps/harbor-nginx 1/1 1 1 84s
deployment.apps/harbor-portal 1/1 1 1 84s
deployment.apps/harbor-registry 1/1 1 1 84s
NAME DESIRED CURRENT READY AGE
replicaset.apps/harbor-core-657496664d 1 1 1 84s
replicaset.apps/harbor-jobservice-78666596dd 1 1 1 84s
replicaset.apps/harbor-nginx-5b76d7588d 1 1 1 84s
replicaset.apps/harbor-portal-68bcb5dd4c 1 1 1 84s
replicaset.apps/harbor-registry-695b7468f8 1 1 1 84s
NAME READY AGE
statefulset.apps/harbor-database 1/1 84s
statefulset.apps/harbor-redis 1/1 84s
statefulset.apps/harbor-trivy 1/1 84s
브라우저에서 접속 후 로그인
[root@m-k8s harbor]# cat values.yaml | grep harborAdminPassword: harborAdminPassword: "Harbor12345"
LoadBalancer IP 확인 후 접속
http://192.168.1.155
기본으로 생성되어 있는 project 인 library를 사용하겠습니다.
Harbor Registry에 이미지 push 하기
# 이미지 빌드용 docker-ce 설치
[root@m-k8s ~]# yum -y install docker-ce
# Harbor 주소를 insecure Registry 로 등록
[root@m-k8s ~]# echo '{ "insecure-registries" : ["192.168.1.202"] }' > /etc/docker/daemon.json
[root@m-k8s ~]# systemctl start docker
# myimage 생성
[root@m-k8s ~]# docker pull nginx
[root@m-k8s harbor]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 61395b4c586d 6 hours ago 187MB
[root@m-k8s ~]# docker tag 61395b4c586d 192.168.1.155/library/myimage:latest
# Harbor Registry 로그인
[root@m-k8s ~]# docker login 192.168.1.202
Username: admin
Password:
# 이미지 push
[root@m-k8s ~]# docker push 192.168.1.155/library/myimage:latest
The push refers to repository [192.168.1.202/library/myimage]
aae231785348: Pushed
e48f2ce44b27: Pushed
3bfd54ea739a: Pushed
bf4045499bea: Pushed
1b34d645672f: Pushed
c74e4ebd2844: Pushed
a2d7501dfb35: Pushed
latest: digest: sha256:9504f3f64a3f16f0eaf9adca3542ff8b2a6880e6abfb13e478cca23f6380080a size: 1778
브라우저에서 push 된 이미지 확인
쿠버네티스에서 Insecure Registry를 사용 가능 하도록 등록
모든 노드에서 containerd 설정 추가 및 재시작이 필요합니다.
# vi /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry.configs] // 아래 두줄 추가
[plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.1.155".tls]
insecure_skip_verify = true
[plugins."io.containerd.grpc.v1.cri".registry.mirrors] // 아래 두줄 추가
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.1.155"]
endpoint = ["http://192.168.1.155"]
# systemctl restart containerd
다시 마스터 노드로 돌아와 이미지 활용이 가능한지 테스트합니다.
[root@m-k8s ~]# kubectl run test --image=192.168.1.155/library/myimage:latest
[root@m-k8s ~]# kubectl get pod
[root@m-k8s ~]# kubectl describe pod test
Image: 192.168.1.202/library/myimage:latest
Normal Pulling 102s kubelet Pulling image "192.168.1.202/library/myimage:latest"
Normal Pulled 101s kubelet Successfully pulled image "192.168.1.202/library/myimage:latest" in 155ms (155ms including waiting)
3. ArgoCD
ArgoCD Helm Charts 다운로드 및 설치
[root@m-k8s ~]# helm repo add argo https://argoproj.github.io/argo-helm
[root@m-k8s ~]# helm repo list
NAME URL
prometheus-community https://prometheus-community.github.io/helm-charts
gitea-charts https://dl.gitea.com/charts/
harbor https://helm.goharbor.io
argo https://argoproj.github.io/argo-helm
[root@m-k8s ~]# helm pull argo/argo-cd
[root@m-k8s ~]# tar xvfz argo-cd-5.46.2.tgz
[root@m-k8s ~]# cd argo-cd
[root@m-k8s argo-cd]# vi values.yaml
service:
type: LoadBalancer // ClusterIP -> LoadBalancer
[root@m-k8s argo-cd]# kubectl create ns argocd
[root@m-k8s argo-cd]# helm install argocd -n argocd -f values.yaml .
[root@m-k8s argo-cd]# kubectl get all -n argocd
NAME READY STATUS RESTARTS AGE
pod/argocd-application-controller-0 1/1 Running 0 61s
pod/argocd-applicationset-controller-747d4d99bf-4wrlj 1/1 Running 0 61s
pod/argocd-dex-server-7fc6cddbf7-76j5v 1/1 Running 0 61s
pod/argocd-notifications-controller-79997876f4-r8hns 1/1 Running 0 61s
pod/argocd-redis-fd4478856-zp4vj 1/1 Running 0 61s
pod/argocd-repo-server-d74bfdf4-xzqcq 1/1 Running 0 60s
pod/argocd-server-75bc9fc56-pr9ls 1/1 Running 0 61s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/argocd-applicationset-controller ClusterIP 10.106.49.225 <none> 7000/TCP 62s
service/argocd-dex-server ClusterIP 10.111.104.149 <none> 5556/TCP,5557/TCP 62s
service/argocd-redis ClusterIP 10.103.169.122 <none> 6379/TCP 62s
service/argocd-repo-server ClusterIP 10.103.64.77 <none> 8081/TCP 62s
service/argocd-server LoadBalancer 10.110.84.177 192.168.1.203 80:30691/TCP,443:31156/TCP 62s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/argocd-applicationset-controller 1/1 1 1 61s
deployment.apps/argocd-dex-server 1/1 1 1 61s
deployment.apps/argocd-notifications-controller 1/1 1 1 61s
deployment.apps/argocd-redis 1/1 1 1 61s
deployment.apps/argocd-repo-server 1/1 1 1 61s
deployment.apps/argocd-server 1/1 1 1 61s
NAME DESIRED CURRENT READY AGE
replicaset.apps/argocd-applicationset-controller-747d4d99bf 1 1 1 61s
replicaset.apps/argocd-dex-server-7fc6cddbf7 1 1 1 61s
replicaset.apps/argocd-notifications-controller-79997876f4 1 1 1 61s
replicaset.apps/argocd-redis-fd4478856 1 1 1 61s
replicaset.apps/argocd-repo-server-d74bfdf4 1 1 1 61s
replicaset.apps/argocd-server-75bc9fc56 1 1 1 61s
NAME READY AGE
statefulset.apps/argocd-application-controller 1/1 61s
브라우저에서 접속 후 로그인
argrocd는 최초 접속 패스워드를 시크릿으로 무작위 생성합니다.
따라서 이를 base64 디코딩하여 확인합니다.
[root@m-k8s argo-cd]# kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
wUC74FsFelTdySaJ
LoadBalancer IP 확인 후 접속
http://192.168.1.203
ArgoCD를 활용하여 쿠버네티스 클러스터에 배포하기
Repository 등록하기
Settings → Repositories 클릭
좌측 상단 + CONNECT REPO
Git Repository 설정
앞서 만든 gitea repository를 등록하는 작업입니다.
Connection Method: VIA HTTPS
Type: git
Project: default
Repository URL: http://192.168.1.154:3000/gitea_admin/my_application
Skip server verification: 체크
이후 상단 CONNECT 버튼 클릭
성공적으로 연결된 모습입니다.
Application 배포하기
메인 화면 좌측 상단 + NEW APP 버튼 클릭
GENERAL 항목 설정
Application Name: mynginx-test
Project Name: default
SYNC POLICY: Manual 수동으로 git 리포지토리를 SYNC
Automatic, 주기적으로 git 리포지토리를 체크하여 SYNC
Automatic SYNC로 Git의 브랜치 변화를 자동으로 감지하고 지속적인 배포가 가능합니다.
SOURCE 항목 설정
Repository URL : http://192.168.1.154:3000/gitea_admin/my_application
Revision: HEAD
Path: .
DESTINATION 항목 설정
어느 쿠버네티스 클러스터에 배포할지 설정하는 부분입니다.
Cluster URL: https://kubernetes.default.svc
Namespace: default
상단 CREATE 버튼 클릭
Application 생성이 완료된 모습입니다.
SYNC 정책을 Manual로 했기 때문에 배포가 시작되지 않았습니다.
SYNC 버튼을 클릭해 배포를 시작합시다.
배포가 정상적으로 완료된 모습입니다.
Application을 클릭하여 자세한 배포 현황을 확인 가능합니다.
실제로 배포되었는지 쿠버네티스에서도 확인합니다.
[root@m-k8s argo-cd]# kubectl get svc,po,deployment NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d6h service/mynginx-service LoadBalancer 10.110.141.42 192.168.1.204 80:30836/TCP 2m27s NAME READY STATUS RESTARTS AGE pod/mynginx-deployment-b4548cd9c-4mn5r 1/1 Running 0 2m26s pod/mynginx-deployment-b4548cd9c-4r6rt 1/1 Running 0 2m26s pod/mynginx-deployment-b4548cd9c-8cd6k 1/1 Running 0 2m26s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mynginx-deployment 3/3 3 3 2m27s
Application의 변경 적용
쿠버네티스 manifest 변경 후 git 에 반영
mariadb 이미지를 추가 하도록 변경하였습니다.
[root@m-k8s my_application]# cd ~/my_application/
[root@m-k8s my_application]# vi mynginx.yaml
# 아래 내용을 하단에 추가
---
apiVersion: v1
kind: Pod
metadata:
labels:
run: mariadb
name: mariadb
spec:
containers:
- image: mariadb:latest
name: mariadb
ports:
- containerPort: 3306
env:
- name: MARIADB_ROOT_PASSWORD
value: "1234"
---
apiVersion: v1
kind: Service
metadata:
name: mariadb-service
spec:
selector:
app: mariadb-service
ports:
- protocol: TCP
port: 3306
targetPort: 3306
type: ClusterIP
[root@m-k8s my_application]# git status
# On branch master
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: mynginx.yaml
#
no changes added to commit (use "git add" and/or "git commit -a")
[root@m-k8s my_application]# git add .
[root@m-k8s my_application]# git commit -m "add mariadb"
[master eaa21d9] add mariadb
1 file changed, 28 insertions(+), 1 deletion(-)
[root@m-k8s my_application]# git push origin master
Username for 'http://192.168.1.201:3000': gitea_admin
Password for 'http://gitea_admin@192.168.1.201:3000':
Username for 'http://192.168.1.201:3000': gitea_admin
Password for 'http://gitea_admin@192.168.1.201:3000':
Counting objects: 10, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 778 bytes | 0 bytes/s, done.
Total 6 (delta 2), reused 0 (delta 0)
remote: . Processing 1 references
remote: Processed 1 references in total
To http://192.168.1.201:3000/gitea_admin/my_application.git
e6fa25d..8ca9737 master -> master
ArgoCD 배포 확인
Refresh 버튼을 통해 새로 고침을 하니 Git에 변경 사항이 생겨 Missing, OutOfSync 상태가 되었습니다.
다시 SYNC 버튼으로 동기화를 진행하고 pod가 배포된 모습을 확인할 수 있습니다.
앞서 설명된 것과 같이 SYNC 정책을 Auto로 변경하면 자동으로 Git의 변경 사항을 감지해 반영합니다.
쿠버네티스에서도 mariadb pod와 서비스가 추가 되었음을 확인 가능합니다.
[root@m-k8s my_application]# kubectl get po,svc NAME READY STATUS RESTARTS AGE pod/mariadb 1/1 Running 0 89s pod/mynginx-deployment-b4548cd9c-cjj68 1/1 Running 0 7m7s pod/mynginx-deployment-b4548cd9c-k78hq 1/1 Running 0 7m7s pod/mynginx-deployment-b4548cd9c-pzkd6 1/1 Running 0 7m7s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5d service/mariadb-service ClusterIP 10.111.59.13 <none> 3306/TCP 89s service/mynginx-service LoadBalancer 10.109.178.255 192.168.1.204 80:32658/TCP 7m7s
argocd password reset
-----------------------------
# 여기 들어가서 admin.password 와 admin.passwordMtime 의 key, value를 모두 지우고 save
kubectl edit secret argocd-secret -n argocd
# argocd-server 파드 재시작
kubectl delete pod argocd-server-6d98c8dd8-5wvl7 -n argocd
# 새로운 비밀번호 얻기
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d