diff --git a/prometheus_grafana_loki/readme.md b/prometheus_grafana_loki/readme.md index 1010cbb..bb74296 100644 --- a/prometheus_grafana_loki/readme.md +++ b/prometheus_grafana_loki/readme.md @@ -24,7 +24,6 @@ Lot of the prometheus stuff here is based off the magnificent # Chapters * **[Core prometheus+grafana](#Overview)** - nice dashboards with metrics of docker host and containers -* **[PromQL](#PromQL)** - links to various learning resources * **[Pushgateway](#Pushgateway)** - push data to prometheus from anywhere * **[Alertmanager](#Alertmanager)** - setting alerts and getting notifications * **[Loki](#Loki)** - prometheus for logs @@ -354,6 +353,10 @@ To **add** pushgateway functionality to the current stack: * Adding pushgateway's **scrape point** to `prometheus.yml`
+ Of note is **honor_labels** set to true, + which makes sure that **conflicting labels** like `job`, set during push + are kept over labels set in `prometheus.yml` for the scrape job. + [Docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).
prometheus.yml @@ -373,7 +376,7 @@ To **add** pushgateway functionality to the current stack: ### The basics -![veeam-dash](https://i.imgur.com/TOuv9bM.png) +![push-web](https://i.imgur.com/9Jk0HKu.png) To **test pushing** some metric, execute in linux:
* `echo "some_metric 3.14" | curl --data-binary @- https://push.example.com/metrics/job/blabla/instance/whatever` @@ -381,8 +384,9 @@ To **test pushing** some metric, execute in linux:
* In Grafana > Explore > query for `some_metric` and see its value there. In that command you see the metric itself: `some_metric` and it's value: `3.14`
-But you also see **labels** being set, one label named `job`, which is required, -but after that it's whatever you want.
+But there are also **labels** being set as part of the url. One label named `job`, +which is required, but after that it's whatever you want. +They just need to be in **pairs** - label name and label value. The metrics sit on the pushgateway **forever**, unless deleted or container shuts down. **Prometheus will not remove** the metrics **after scraping**,