| 목차 | ||
|---|---|---|
|
...
1. 쿠버네티스 모니터링
이번 장에서는 모니터링을 위해 Dashboard, Prometheus+Grafana 를 직접 설치하여 실습해 봅니다.
1.1 Kubernetes Dashboard
대시보드는 웹기반으로 쿠버네티스를 모니터링하고 관리 할 수 있습니다
...
다운로드 받은 파일에서 서비스 타입을 NodePort나 LoadBalancer로 수정합니다.
12/00-dashboard-install.txt
| 코드 블럭 |
|---|
curl -LO https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml |
...
(3) 로그인 시 필요한 토큰을 생성하여 접속합니다.
...
12/01-dashboard-user.yaml
...
| 코드 블럭 |
|---|
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
|
| 코드 블럭 |
|---|
kubectl create -f 017.01-dashboard-user.yaml serviceaccount/admin-user created kclusterrolebinding.rbac.authorization.k8s.io/admin-user created 토큰을 복사하여 로그인할 때 사용합니다. kubectl -n kubernetes-dashboard create token admin-user eyJhbGciOiJSUzI1NiIsImtpZCI6ImNlZ015NjBDby15R1c4NVlIMi1vTGVVQXhPdGVzeEk5cEV0NGJiWl92RFEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjk0Njc3NDgxLCJpYXQiOjE2OTQ2NzM4ODEsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiYTRjOGRmMzQtNGY4Yy00OGE5LWJmMjEtYTgyZDliODhhYzAwIn19LCJuYmYiOjE2OTQ2NzM4ODEsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.hF6NmSdHK-_nysVWZkQdLZZPtKDzPKq1VUfVKn3TjP1d44QfFoBYxVvcbC9mjcphPT0ggA-NKFcZalUN5a8X04lmk77Vel7PFyrXy4Q7WNx2OxiQN88U7F291pyptif0_sk44pZA58MMLYKEq9WKFGnKufyfZJeDsLcFq2YwkTtk4DYrJ47KeibKRnXf3QJ6Wcw4e61x0vBy_VU_vpCPkD4lC9bQKVRRy-Y-thevbAfE5FRthvADsNodBdIEZReXMGnDkvog_LL0okSGz6wKfWE8CpvEGhZQpydlagJGCcs6A15_wr05RyZBNtaoxOdFTDieeYEqKcio0qUJNTGOhA kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes-dashboard kubernetes-dashboard NodePort 10.109.126.84 <none> 443:31963/TCP 8m35s nodeip:nodeport로 접근하여 위에서 확이한 token은 이용하여 접속합니다 |
...
(4) 브라우저를 통해 대시보드에 접속합니다.
...
1.2 Prometheus와 Grafana
프로메테우스와 그라파나는 모니터링에 특화된 솔루션입니다.
실제 운영 환경 모니터링은 그라파나를 가장 많이 사용합니다.
프로메테우스로 매트릭을 수집하여 저장하고 그라파나에서 필요한 데이터를 불러와 대시보드 형식으로 표현 합니다.
쿠버네티스와 노드의 매트릭 수집을 위해 kube-state-metric과 nodeexporter를 사용합니다.
프로메테우스 매트릭은 클러스터 규모가 커질수록 수집 대상 데이터도 많아지고 매트릭정보도 많아지므로 트래픽과 스토리지를 잘 설계해야 합니다.
...
(1) Prometheus&Grafana 설치
12/02-monitoring-install.txt
| 코드 블럭 |
|---|
git clone https://github.com/prometheus-operator/kube-prometheus.git cd kube-prometheus/ kubectl create -f manifests/setup customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheusagents.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/scrapeconfigs.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created namespace/monitoring created 프로메테우스 servicemonitor 라는 crd가 설치되기 까지 기다리는 명령 "No resources found" 가 뜨면 성공 until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done kubectl create -f manifests/ alertmanager.monitoring.coreos.com/main created networkpolicy.networking.k8s.io/alertmanager-main created poddisruptionbudget.policy/alertmanager-main created .. 생략 .. 설치에 시간이 소요될 수 있습니다. 전부 배포된 모습입니다. kubectl get all -n monitoring NAME READY STATUS RESTARTS AGE pod/alertmanager-main-0 2/2 Running 0 45s pod/alertmanager-main-1 2/2 Running 0 45s pod/alertmanager-main-2 2/2 Running 0 45s pod/blackbox-exporter-6cfc4bffb6-srf94 3/3 Running 0 51s pod/grafana-748964b847-rwrk5 1/1 Running 0 49s pod/kube-state-metrics-6b4d48dcb4-jb7qt 3/3 Running 0 49s pod/node-exporter-5v9lk 2/2 Running 0 48s pod/node-exporter-9b7md 2/2 Running 0 48s pod/node-exporter-bmdpz 2/2 Running 0 48s pod/prometheus-adapter-79c588b474-72k7p 1/1 Running 0 47s pod/prometheus-adapter-79c588b474-nq4pb 1/1 Running 0 47s pod/prometheus-k8s-0 2/2 Running 0 44s pod/prometheus-k8s-1 2/2 Running 0 44s pod/prometheus-operator-68f6c79f9d-x4jjm 2/2 Running 0 47s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/alertmanager-main ClusterIP 10.105.88.232 <none> 9093/TCP,8080/TCP 51s service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 45s service/blackbox-exporter ClusterIP 10.96.11.27 <none> 9115/TCP,19115/TCP 51s service/grafana ClusterIP 10.101.206.55 <none> 3000/TCP 50s service/kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 49s service/node-exporter ClusterIP None <none> 9100/TCP 49s service/prometheus-adapter ClusterIP 10.104.99.31 <none> 443/TCP 48s service/prometheus-k8s ClusterIP 10.97.36.1 <none> 9090/TCP,8080/TCP 48s service/prometheus-operated ClusterIP None <none> 9090/TCP 44s service/prometheus-operator ClusterIP None <none> 8443/TCP 47s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-exporter 3 3 3 3 3 kubernetes.io/os=linux 49s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/blackbox-exporter 1/1 1 1 51s deployment.apps/grafana 1/1 1 1 50s deployment.apps/kube-state-metrics 1/1 1 1 49s deployment.apps/prometheus-adapter 2/2 2 2 48s deployment.apps/prometheus-operator 1/1 1 1 47s NAME DESIRED CURRENT READY AGE replicaset.apps/blackbox-exporter-6cfc4bffb6 1 1 1 51s replicaset.apps/grafana-748964b847 1 1 1 50s replicaset.apps/kube-state-metrics-6b4d48dcb4 1 1 1 49s replicaset.apps/prometheus-adapter-79c588b474 2 2 2 48s replicaset.apps/prometheus-operator-68f6c79f9d 1 1 1 47s NAME READY AGE statefulset.apps/alertmanager-main 3/3 45s statefulset.apps/prometheus-k8s 2/2 44s |
...
참고) https://grafana.com/grafana/dashboards/
kube-prometheus-stack 설치
| 코드 블럭 |
|---|
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update |
| 코드 블럭 |
|---|
helm pull prometheus-community/kube-prometheus-stack --untar
# cd kube-prometheus-stack/ |
| 코드 블럭 |
|---|
helm install prometheus-stack . -f values.yaml -n monitoring --create-namespace |
| 코드 블럭 |
|---|
kubectl -n monitoring edit svc prometheus-stack-grafana
ClusterIP -> LoadBalancer |
