Adding Targets to Prometheus
Introduction
In this tutorial, you’re going to learn how to set up and add targets to Prometheus via Kubernetes. Don’t worry, it’s totally not complicated, and there are just a few super easy steps to cover. However, for learning’s sake, we’ll also be going over configuration maps, target formatting, and how to add the targets to your config map.
If you’re looking for some deeper and more advanced options check out this link: prometheus.io/docs/prometheus/latest/configuration/configuration
Prerequisites
Before you begin, you’re going to need a few things already done:
- Kubernetes environment installed
- Prometheus installed on a Kubernetes environment.
- Kubernetes Command Line Tool: Kubectl. This tool allows you to run commands on kubernetes clusters.
Learn more deets here: https://kubernetes.io/docs/tasks/tools/
Prometheus Configuration Map
Adding targets to your configuration map means that you basically have to edit the map. You’ll have to channel the great explorers of the world while you do this. It doesn’t help, but it feels cool!
Your config map’s name should be the same one you used when you first installed Prometheus. Find it, then run the following command to see a comprehensive list of other config maps within Prometheus: kubectl get configmaps
Once you find the name of the map you want to edit, you can enter the command: “kubectl edit cm” to start adding your targets. kubectl edit cm [config map name]
Targets
For our examples, we’ll use 3 different jobs:
In the first job, we’ll be monitoring a node on the standard node-exporter port, 9100, and on a separate port that is hosting custom metrics, 9200.
The second job will monitor the 5.6.7.8 node on the 5678 port.
In the third and final job, we will be monitoring the 9.10.11.12 node on the HTTPS port, 443, using the blackbox-exporter service. To do this, you’ll need to have the blackbox exporter service installed in Kubernetes, and will need to change the [blackbox ip]:[port] part under the relabel_configs: section with the IP and port of your blackbox instance.
Remember to check the names of your jobs, since each have their own labels that you’ll be able to use in Prometheus Query Language (PromQL) queries:
For our examples, we'll use the instance as a unique label that will indicate the source of the metrics. Then, we'll use the cluster label to group the jobs together.
scrape_configs:
- job_name: node-1-metrics
static_configs:
- labels:
instance: node-1
cluster: cluster-a
targets:
- 1.2.3.4:9100
- 1.2.3.4:9200
- job_name: node-2-metrics
- labels:
instance: node-2
cluster: cluster-a
targets:
- 5.6.7.8:5678
- job_name: node-3-https-metrics
metrics_path: /probe
params:
module: http_2xx
static_configs:
- targets:
- http://9.10.11.12:443
- labels:
instance: node-3
cluster: cluster-a
relabel_configs:
- source_labels: [__address__]
separator: ;
regex: http://9.10.11.12:443
target_label: name
replacement: 'node-3-https'
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: [blackbox ip]:[port] # The blackbox exporter's real hostname:port.
Once you have beautifully formatted your jobs like so, you’ll need to add them to the end of the scrape_configs section in the Prometheus configuration map.
As long as your targets are formatted correctly, saving the file will output "configmap/[config map name] edited". If you didn’t format them properly, your config will be saved to the /tmp directory, instead of the map, and you’ll have to go back and format again.
Conclusion
That’s all, folks!
Now you know how to set up targets in Prometheus. Not too difficult, right?
If you have any questions about Prometheus or other DevOps software contact us today!