开源监控系统Prometheus的前世今生

http://www.uml.org.cn/itil/202012101.asp

https://prometheus.io/docs/introduction/overview/

info数据结构

可以存储任意信息

https://prometheus.github.io/client_python/instrumenting/info/

from prometheus_client import Info

i = Info('my_build_version', 'Description of info')

i.info({'version': '1.2.3', 'buildhost': 'foo@bar'})

其它四种常用数据类型

https://prometheus.io/docs/concepts/metric_types/

查询接口

https://prometheus.io/docs/prometheus/latest/querying/api/

Querying exemplars

$ curl -g 'http://localhost:9090/api/v1/query_exemplars?query=test_exemplar_metric_total&start=2020-09-14T15:22:25.479Z&end=2020-09-14T15:23:25.479Z'
{
    "status": "success",
    "data": [
        {
            "seriesLabels": {
                "__name__": "test_exemplar_metric_total",
                "instance": "localhost:8090",
                "job": "prometheus",
                "service": "bar"
            },
            "exemplars": [
                {
                    "labels": {
                        "trace_id": "EpTxMJ40fUus7aGY"
                    },
                    "value": "6",
                    "timestamp": 1600096945.479
                }
            ]
        },
        {
            "seriesLabels": {
                "__name__": "test_exemplar_metric_total",
                "instance": "localhost:8090",
                "job": "prometheus",
                "service": "foo"
            },
            "exemplars": [
                {
                    "labels": {
                        "trace_id": "Olp9XHlq763ccsfa"
                    },
                    "value": "19",
                    "timestamp": 1600096955.479
                },
                {
                    "labels": {
                        "trace_id": "hCtjygkIHwAN9vs4"
                    },
                    "value": "20",
                    "timestamp": 1600096965.489
                }
            ]
        }
    ]
}

https://zhuanlan.zhihu.com/p/671479426

在现代云原生应用的监控体系中，Prometheus无疑是一颗璀璨的明星，而Python则是一门多才多艺的编程语言。将它们结合，通过Python读取Prometheus接口数据，成为了实时监控和数据分析的一项重要任务。

本篇文章将分享Prometheus API的基本概念到PromQL查询语言的应用，再到如何通过Python与Prometheus API进行无缝交互。通过丰富的示例代码和详细的讲解，将解锁使用Python进行实时监控的奇妙世界，为读者打开更广阔的数据分析视野。

Prometheus API是Prometheus监控系统提供的接口，通过该接口，用户可以方便地查询和获取监控数据。Prometheus API的设计灵感来自于RESTful风格，采用HTTP协议，为用户提供了丰富的端点用于不同的监控操作。

常用的Prometheus API端点包括：

/api/v1/query： 用于执行单个即时查询，返回指定查询的结果。

/api/v1/query_range： 允许用户执行范围查询，获取一段时间内的时间序列数据。

/api/v1/label： 提供有关标签的信息，包括标签名称、标签值等。

/api/v1/targets： 返回所有已知的目标信息，包括目标的标签和状态。

`prometheus-api-client`

https://www.cnblogs.com/rainbow-tan/p/16813768.html

from prometheus_api_client import PrometheusConnect

prom = PrometheusConnect(url="http://172.17.140.17:9090", headers=None, disable_ssl=True)
ok = prom.check_prometheus_connection()  # 检查连接状态
print(f"连接Prometheus:{prom.url}, 状态:{'连接成功' if ok else '连接失败'}")
"""
url - (str) url for the prometheus host
headers – (dict) A dictionary of http headers to be used to communicate with the host. Example: {“Authorization”: “bearer my_oauth_token_to_the_host”}
disable_ssl – (bool) If set to True, will disable ssl certificate verification for the http requests made to the prometheus host

url Prometheus的连接url
如果Prometheus需要认证, 则需要在headers中添加认证 {“Authorization”: “bearer my_oauth_token_to_the_host”}
disable_ssl 是否禁止ssl认证 http就禁止 https 则需要验证
"""

airflow-prometheus-pushgateway

https://github.com/DanielMorales9/airflow-prometheus-pushgateway

https://github.com/DanielMorales9/airflow-prometheus-pushgateway/blob/master/airflow/dags/hello_world.py

import time
from datetime import datetime
from datetime import timedelta
from random import randint, random
from timeit import default_timer as timer

from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator
from prometheus_client import CollectorRegistry, push_to_gateway, Summary, Counter

GATEWAY = 'pushgateway'


def print_hello():
    registry = CollectorRegistry()
    c = Counter('count_exceptions', 'counts number of successes and failures',
                labelnames=['type'], registry=registry)
    s = Summary('time_delta', 'execution time of print_hello function', registry=registry)

    for i in range(randint(1, 10)):
        start = timer()
        time.sleep(random()*10)

        try:

            if randint(0, 1) == 1:
                raise Exception

            c.labels(type='success').inc()

        except:
            c.labels(type='failure').inc()

        end = timer()
        s.observe(timedelta(seconds=end - start).seconds)

    push_to_gateway('%s:9091' % GATEWAY, job='print_hello', registry=registry)

    return 'Hello world!'


dag = DAG('hello_world',
          description='Simple tutorial DAG',
          schedule_interval='*/1 * * * *',
          start_date=datetime(2017, 3, 20),
          catchup=False)

dummy_operator = DummyOperator(task_id='dummy_task', retries=3, dag=dag)

hello_operator = PythonOperator(task_id='hello_task', python_callable=print_hello, dag=dag)

dummy_operator >> hello_operator

自定义node exporter

https://www.cnblogs.com/liukx/p/15549688.html

#!/usr/bin/python3
# coding:utf-8
from prometheus_client import Gauge, start_http_server, Counter
from prometheus_client.core import CollectorRegistry
import prometheus_client
import uvicorn
from fastapi import FastAPI
from fastapi.responses import PlainTextResponse

# 定义api对象
app = FastAPI()

# 设置接口访问路径/metrics
# @app语法糖需要放在最终入口函数
@app.get('/metrics', response_class=PlainTextResponse)
def get_data():
    '''
    该函数为最终函数入口（采集数据函数）,该例子模拟采集到数据标签label1、label2和label3，数据data1
    '''
    # 定义client_python里提供的prometheus Gauge数据类型
    REGISTRY = CollectorRegistry(auto_describe=False)
    example_G = Gauge("this_is_a_metric_name", "this is a metric describe", ["label1", "label2", "label3"], registry=REGISTRY)
    
    label1 = '111'
    label2 = '222'
    label3 = '333'
    data1 = '444'
    # 调用Gauge数据类型，把采集的数据放入到设定的example_G
    example_G.labels(label1,label2,label3).set(data1)
    # return需放在函数下最外层，否则返回的数据有缺失
    return prometheus_client.generate_latest(REGISTRY)
 
# 用uvicorn调用接口，启用端口为9330
if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=9330, log_level="info")

Prometheus Agent

https://zhuanlan.zhihu.com/p/438726797

Prometheus Proxy

https://blog.csdn.net/gitblog_00908/article/details/143883256

Prometheus Proxy 是一个用于解决 Prometheus 监控系统在防火墙环境下无法直接访问被监控端点的问题的开源项目。Prometheus 是一个优秀的系统监控和报警工具包，它使用拉取模型来收集指标数据。然而，当 Prometheus 服务器和其指标端点之间存在防火墙时，拉取模型会遇到问题。Prometheus Proxy 通过在防火墙内外分别运行代理和代理端点的方式，使得 Prometheus 能够继续使用拉取模型来收集数据。