{"id":729,"date":"2021-05-08T15:37:52","date_gmt":"2021-05-08T07:37:52","guid":{"rendered":"https:\/\/www.yinyubo.cn\/?p=729"},"modified":"2022-05-17T09:28:53","modified_gmt":"2022-05-17T01:28:53","slug":"k3s","status":"publish","type":"post","link":"https:\/\/www.yinyubo.com\/?p=729","title":{"rendered":"K3S\u73af\u5883\u4e0b\u63a5\u5165Prometheus,grafana,\u7b49\u76d1\u63a7\u5957\u4ef6"},"content":{"rendered":"\n
\n
\n

\u80cc\u666f<\/h1>\n

\u56e0\u4e3a\u5e38\u89c4\u7684\u76d1\u63a7\u90fd\u662f\u7528K8S\u505a\u7684\uff0c\u800cK3S\u4e0a\u7684\u76d1\u63a7\u65b9\u6848\u5c11\u4e4b\u53c8\u5c11\uff0c\u5982\u679c\u76f4\u63a5\u7528rancher\u4e0a\u7684prometheus\u76d1\u63a7\uff0c\u4f1a\u6d88\u8017\u81f3\u5c112G\u7684\u5185\u5b58\uff0c\u4e8e\u662f\u6211\u4eec\u5c31\u81ea\u5df1\u505a\u4e86K3S\u548cpg\u6570\u636e\u5e93\u7684\u76d1\u63a7,\u5e76\u4e14\u901a\u8fc7\u6211\u4eec\u81ea\u5df1\u505a\u7684\u76d1\u63a7\uff0c\u53ef\u4ee5\u51cf\u5c11\u4e00\u4e9b\u4e0d\u5fc5\u8981\u7684\u6027\u80fd\u5f00\u9500\u3002\u4e3b\u8981\u76d1\u63a7\u5bb9\u5668\u8d44\u6e90\u6d88\u8017\uff0c\u5bbf\u4e3b\u673a\u8d44\u6e90\u6d88\u8017\uff0cpg\u6570\u636e\u5e93\u8d44\u6e90\u6d88\u8017
\"\"
\u8fd9\u4e2a\u662f\u8d44\u6e90\u4f7f\u7528\u60c5\u51b5\uff0c\u5927\u7ea6\u4f1a\u7528\u6389600M\u5185\u5b58\u548c100MCPU<\/p>\n

\n
\n
\n
user@user:\/$ kubectl top pod -n kube-ops<\/code>
\nNAME                                            CPU(cores)   MEMORY(bytes)<\/code>
\ngrafana-0                                       1m           47Mi<\/code>
\nkube-state-metrics-594fb7bc84-5mg6g             3m           10Mi<\/code>
\nnode-exportor-prometheus-node-exporter-sznls    31m          8Mi<\/code>
\nnode-exportor-prometheus-node-exporter-tx7rm    13m          7Mi<\/code>
\nprometheus-0                                    46m          565Mi<\/code>
\nprometheus-postgres-exporter-6c858f47d4-dj9hj   13m          6Mi<\/code><\/pre>\n
\n
prometheus\u76d1\u63a7\u7ec4\u4ef6\u53c2\u8003helm chart<\/p>\n
https:\/\/github.com\/prometheus-community\/helm-charts\/tree\/main\/charts<\/a>
\u6211\u9b54\u6539\u7684github\u8fde\u63a5
https:\/\/github.com\/lizhenwei\/k3s-prometheus<\/a><\/p>\n<\/blockquote>\n
\n
Prometheus\u7684\u5b89\u88c5<\/h1>\n
1.\u521b\u5efa\u4e00\u4e2a\u547d\u540d\u7a7a\u95f4kube-ops\uff0c\u628a\u76d1\u63a7\u7528\u7684\u4e1c\u897f\u90fd\u653e\u5728\u8fd9\u4e2a\u547d\u540d\u91cc\u9762\u3002
2.\u4f7f\u7528\u6211\u9b54\u6539\u7684chart\u8fdb\u884c\u5b89\u88c5\uff0c\u9ed8\u8ba4\u7248\u672c\u662fv2.26.0\uff0c\u9ed8\u8ba4\u542f\u7528\u4e0a\u56fe\u63d0\u5230\u7684<\/p>\n
cadvisor,kube_state_metrics,node_exportor,postgres_exporter<\/p>\n
\u00a0<\/p>\n
kubectl create ns kube-ops
\nhelm install -n kube-ops prometheus lizhenwei-prometheus

<\/pre>\n
\u5982\u679c\u8981\u5173\u95ed\u5176\u4e2d\u7684\u67d0\u4e2a\u76d1\u63a7\u9879\uff0c\u5b89\u88c5\u65f6\u53c2\u8003\u6dfb\u52a0\u53d8\u91cf –set node_exportor.enable=false\uff0c
prometheus\u7684\u6570\u636e\u4f1a\u7528\u9ed8\u8ba4\u7684storageClass\u751f\u6210PVC\uff0c\u5982\u679c\u6ca1\u6709\u9ed8\u8ba4\u7684\uff0c\u53c2\u8003\u624b\u52a8\u8bbe\u7f6enfs\u5b58\u50a8 –set persistence.storageClass=nfs-client
\u5b89\u88c5\u6210\u529f\u540e\uff0c\u56e0\u4e3a\u6211\u662fnodeport\u66b4\u9732\u51fa\u6765\u7684\uff0c\u6240\u4ee5\u53ef\u4ee5\u901a\u8fc7\u6d4f\u89c8\u5668\u53bb\u68c0\u67e5\u4e00\u4e0b\u914d\u7f6e,\u4f8b\u5982\u8fd9\u91cc\u6211\u4eec\u66b4\u9732\u51fa\u6765\u662f32331\u7aef\u53e3\u53ef\u4ee5\u8bbf\u95ee,\u4f8b\u5982\uff1a<\/p>\n
\n
\n
\n
$ kubectl get svc -n kube-ops<\/code>
\nNAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE<\/code>
\nprometheus                     NodePort    10.43.178.128   <none>        9090:32331\/TCP      2d4h<\/code>

<\/pre>\n
\u8bbf\u95eehttp:\/\/master\u673a\u5668IP:32331\/targets\uff0c\u53ef\u4ee5\u770b\u5230prometheus\u81ea\u5df1\u5df2\u7ecf\u662f\u6210\u529f\u8d77\u6765\u4e86\uff0c\u6211\u4eec\u4e5f\u53ef\u4ee5\u53bbhttp:\/\/master\u673a\u5668IP:32331\/config\u9875\u9762\u770b\u770b\u914d\u7f6e\u9879\u662f\u5426\u90fd\u6b63\u786e\"\"<\/span>
\u8981\u542f\u7528\u6216\u505c\u7528\u67d0\u4e9b\u76d1\u63a7\u529f\u80fd\uff0c\u53ef\u4ee5\u4fee\u6539lizhenwei-prometheus\/valuse.yaml\u91cc\u7684\u914d\u7f6e\uff0c\u4fee\u6539\u6210true\u6216false<\/p>\n
\n
kube-state-metrics\u7684\u5b89\u88c5<\/h1>\n
kube-state-metrics\u662f\u7528\u6765\u6536\u96c6k8s\u96c6\u7fa4\u7684CPU\uff0c\u5185\u5b58\u7b49\u4fe1\u606f\u7684\u6536\u96c6\u5668\uff0c\u8fd9\u91cc\u6211\u7528\u7684\u662fv1.9.7\u7248\u672c\u3002
\u4f7f\u7528\u6211github\u4e0a\u7684helm\u8fdb\u884c\u5b89\u88c5\uff1a<\/p>\n
helm install -n kube-ops prometheus lizhenwei-kube-state-metrics
\n<\/pre>\n
\u8981\u60f3\u9a8c\u8bc1\u90e8\u7f72kube-state-metrics\u4e4b\u540e\uff0c\u662f\u5426\u80fd\u6210\u529f\u4f7f\u7528\uff0c\u6211\u4eec\u53ef\u4ee5\u770bprometheus\u7684target\u9875\u9762\u662f\u4e0d\u662f\u663e\u793akube-state-metrics(X\/X up) x\u4e3a\u673a\u5668\u8282\u70b9\u6570\u91cf\u3002
\u7136\u540e\u6211\u4eec\u53bbgraph\u9875\u9762\u770b\u770b\uff0chttp:\/\/master\u673a\u5668IP:32331\/graph\u3002\u53ef\u4ee5\u53c2\u8003\u5b98\u7f51\u7ed9\u7684promql\u6587\u6863\uff0c\u8f93\u5165\u51e0\u4e2a\u8bed\u53e5\u8bd5\u8bd5<\/span><\/p>\n
\n
\n
PromQL\u53c2\u8003\u94fe\u63a5<\/p>\n
https:\/\/github.com\/kubernetes\/kube-state-metrics\/tree\/master\/docs<\/a><\/p>\n<\/blockquote>\n
\u4f8b\u5982\u8f93\u5165kube_configmap_info\uff0c\u70b9\u51fbexecute\u4f1a\u8fd4\u56de\u4e00\u4e9b\u7ed3\u679c\uff0c\u5982\u4e0b\u56fe\"\"<\/p>\n
\n
K3S\u81ea\u5e26\u7684CADVISOR<\/h1>\n
K3S\u81ea\u5e26kubelet \uff0c\u800ccadvisor\u5df2\u7ecf\u96c6\u6210\u5728kubelet\u91cc\u4e86\uff0cprometheus\u53ef\u4ee5\u901a\u8fc7\u914d\u7f6e\u627e\u5230\u672c\u5730\u5b89\u88c5\u7684cadvisor\uff0c\u4f8b\u5982\u4e0b\u5217\u4ee3\u7801\uff1a<\/p>\n
- job_name: 'cadvisor'<\/code>\n  scheme: https<\/code>\n  tls_config:<\/code>\n    ca_file: \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/ca.crt<\/code>\n  bearer_token_file: \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/token<\/code>\n  kubernetes_sd_configs:<\/code>\n  - role: node<\/code>\n  relabel_configs:<\/code>\n  - action: labelmap<\/code>\n    regex: __meta_kubernetes_node_label_(.+)<\/code>\n  - target_label: __address__<\/code>\n    replacement: kubernetes.default.svc:443<\/code>\n  - source_labels: [__meta_kubernetes_node_name]<\/code>\n    regex: (.+)<\/code>\n    target_label: __metrics_path__<\/code>\n    replacement: \/api\/v1\/nodes\/${1}\/proxy\/metrics\/cadvisor<\/code>

<\/pre>\n
\u63a5\u5165prometheus\u7684\u65f6\u5019\uff0c\u53ef\u4ee5\u6839\u636e\u4ee5\u4e0b\u94fe\u63a5\uff0c\u67e5\u770bPromQL\u8bed\u53e5<\/p>\n
\n
\n
PromQL\u53c2\u8003\u94fe\u63a5<\/p>\n
https:\/\/github.com\/google\/cadvisor\/blob\/master\/docs\/storage\/prometheus.md<\/a><\/p>\n<\/blockquote>\n
\n
node-exportor\u7684\u5b89\u88c5<\/h1>\n
\u8fd9\u4e2a\u4f7f\u7528\u5b98\u65b9\u7f51\u7ad9\u7ed9\u7684\u4e00\u4e2a\u516c\u5171\u5e93\u5c31\u53ef\u4ee5\u4e86<\/p>\n
\n
\n
\n
helm repo add prometheus-community https:\/\/prometheus-community.github.io\/helm-charts<\/code>\nhelm repo update<\/code>\nhelm install node-exportor prometheus-community\/prometheus-node-exporter<\/code><\/pre>\n
\n
grafana\u53c2\u8003\u94fe\u63a5https:\/\/grafana.com\/grafana\/dashboards\/1860<\/a><\/p>\n<\/blockquote>\n
\n
postgres-exportor<\/h1>\n
\u4e0b\u8f7d\u4ee3\u7801https:\/\/github.com\/prometheus-community\/helm-charts\/tree\/main\/charts\/prometheus-postgres-exporter<\/a>\u00a0\u5230\u673a\u5668\u4e0a
\u4fee\u6539valuse.yaml\u91cc\u7684queries\u91cc\u7684\u53c2\u6570\uff0c\u5728\u6700\u540e\u9762\u52a0\u4e0a<\/p>\n
\n
\n
\n
pg_stat_activity:<\/code>\n      query: |<\/code>\n        SELECT\n          datname,\n          SUM(EXTRACT(EPOCH FROM (CURRENT_TIMESTAMP - state_change))::bigint)::float AS process_idle_seconds_sum,\n          COUNT(*) AS process_idle_seconds_count\n        FROM pg_stat_activity\n        WHERE state = 'idle'\n        GROUP BY datname\n      metrics:\n        - datname:\n            usage: \"LABEL\"\n            description: \"datname\"\n        - process_idle_seconds:\n            usage: \"COUNTER\"\n            description: \"Idle time of server processes\"\n        - process_idle_seconds_sum:\n            usage: \"GUAGE\"\n            description: \"sum of Idle time of server processes\"<\/code><\/pre>\n
\u5728valuse.yaml\u6587\u4ef6\u7684config.datasource\u90e8\u5206\u6dfb\u52a0\u6211\u4eec\u7684postgres\u8d85\u7ba1\u7528\u6237\u4fe1\u606f\u3002<\/p>\n
helm\u5b89\u88c5\u8bed\u53e5<\/p>\n
\n
\n
\n
helm install -n kube-ops prometheus-postgres-exporter prometheus-postgres-exporter\n<\/pre>\n
\n
Grafana\u5bfc\u5165dashboard<\/h1>\n
\u53ef\u4ee5\u53c2\u8003\u8fd9\u4f4d\u5927\u795e\u7684github\u94fe\u63a5https:\/\/github.com\/starsliao\/Prometheus<\/a><\/p>\n
\n
\u9875\u9762\u622a\u56fe<\/h1>\n
\"\"\"\" \"\"<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
\u80cc\u666f \u56e0\u4e3a\u5e38\u89c4\u7684\u76d1\u63a7\u90fd\u662f\u7528K8S\u505a\u7684\uff0c\u800cK3S\u4e0a\u7684\u76d1\u63a7\u65b9\u6848\u5c11\u4e4b\u53c8\u5c11\uff0c\u5982\u679c\u76f4\u63a5\u7528rancher\u4e0a\u7684prometh […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-729","post","type-post","status-publish","format-standard","hentry","category-k8s"],"_links":{"self":[{"href":"https:\/\/www.yinyubo.com\/index.php?rest_route=\/wp\/v2\/posts\/729","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.yinyubo.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.yinyubo.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.yinyubo.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.yinyubo.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=729"}],"version-history":[{"count":22,"href":"https:\/\/www.yinyubo.com\/index.php?rest_route=\/wp\/v2\/posts\/729\/revisions"}],"predecessor-version":[{"id":848,"href":"https:\/\/www.yinyubo.com\/index.php?rest_route=\/wp\/v2\/posts\/729\/revisions\/848"}],"wp:attachment":[{"href":"https:\/\/www.yinyubo.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=729"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.yinyubo.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=729"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.yinyubo.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=729"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}