Create and export 🔥 Prometheus metrics

Prometheus is a titan in greek mythology that brought fire (hence the logo). Prometheus is also a modern monitoring system that has uses time series to display the data. It provides a query language for better exploiting your metrics compiled with an alerting service and all of that with great compatibility and integration with other system. Making it fairly easy to deploy and use.

Implement Prometheus Metrics

You can create different types of metrics depending on your needs. Here we are going to look at a couple of example in Kotlin.

Metrics types

There are 4 types of metrics:

Counter: simple that can only go up (number of transactions, errors)
Gauge: can go up and down (request duration, queue size)
Histogram: measure the frequency of events in pre-defined buckets (response duration)
Summary: measure with more accuracy on the application side the frequency of events in dynamic buckets.

Enable it with Springboot in Kotlin

Set up

First would be the springboot Actuator library. That will provide all the prod ready feature you may want including the metrics one. Then besides all the other common springboot dependency, here is a summary of what you may need to add in your gradle.build.kts.

implementation("org.springframework.boot:spring-boot-starter-actuator")
// To create your own prometheus metrics
implementation("io.micrometer:micrometer-registry-prometheus:1.3.5")

Configuration

Those will enable you to create and display the metrics. Micrometer is an application metrics that supports multiple monitoring services. Then don’t forget to update your application.yml with metrics enabled. This way you’ll have all basic springboot prometheus metrics already created and advertised.

management:
  # Set the management port, if not set will take the default one.
  server:
    port: 9100
  # Enable prometheus and metrics endpoints
  endpoint:
    metrics:
      enabled: true
    prometheus:
      enabled: true
  # Expose all metrics endpoints at "/"
  endpoints:
    web:
      base-path: "/"
      exposure:
        include: '*'
  # Enable prometheus metrics generation
  metrics:
    export:
      prometheus:
        enabled: true

With the above config you should be able to check the metrics at localhost:9100/prometheus. We set the base path to be / instead of actuator. Also you may need to set jmx to true (which is false by default in spring 2.x) for some metrics (like for kafka)

spring:
  jmx:
    enabled: true

JMX is the Java Management Extension library that supply tools for run time managing and monitoring.

Test it

Since you have enabled the metric, you can now test it. Depending if you set the management port to 9100 or let it default to be same as your app, you can get the endpoint and retrieve the data.

Here is how you would test it:

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ExtendWith(SpringExtension::class)
internal class EnppointTests {

  var testRestTemplate = TestRestTemplate()

  @Test
  fun metricsTest() {
    val result: ResponseEntity<Void> = testRestTemplate.exchange(
        URI("http://localhost:9100/metrics"),
        HttpMethod.GET,
        HttpEntity(""),
        Void::class.java)

    Assertions.assertEquals(HttpStatus.OK, result.statusCode)
  }
}

If you use the default port, you can use @LocalServerPort to get it so that you don’t need to hard code it in your test.

Add New Metrics

Counter with Micrometer Core

For the counter you can use the MeterRegistry then set a name, you can add some other parameters but we keep it simple. The tag .tag("key", "value") is used when exploiting the metrics in the PromQL so you can differentiate them. You can add as many tags as you want:

import io.micrometer.core.instrument.MeterRegistry
import io.micrometer.core.instrument.Counter

@Component
class MetricLogger(registry: MeterRegistry) {
    private var requestReceivedCounter: Counter = Counter
        .builder("request_received")
        .tag("app","my-app")
        .register(registry)

    fun requestReceived() = requestReceivedCounter.increment()
}

With that you can call your logger and increment your counter:

@Autowired
private lateinit var metricLogger: MetricLogger
    
fun myMethodThatReceiveRequest() {
   metricLogger.requestReceived()
   // ... and do some more stuff
}

You can use it within your code this way, a bit like you would with log lines.

Gauge with Prometheus Client

For the Gauge, you will need the CollectorRegistry a name, some help message and a label. As before you can add as many tags as you wish.

import io.prometheus.client.CollectorRegistry
import io.prometheus.client.Gauge

@Component
class MetricLogger(collectorRegistry: CollectorRegistry) {
    private var queueSizeGauge: Gauge = Gauge
        .build("queue_size", "Gauge for queue size")
        .labelNames("type")
        .register(collectorRegistry)

    fun updateQueuSize(value: Long) = latencyGauge.labels("my-app").set(value.toDouble())
}

I could also use .inc() and .dec() to increment the value of my gauge instead of using .set(). As for the usage it’s the same as the Counter one. You can also use Micrometer for gauge and other metrics. Both can work and be used, check the concepts Micrometer doc page for more info.

Exploit your metrics with PromQL

PromQL

Prometheus Query Language PromQL lets you apply some filters, operators, functions and more to your metrics. Particularly useful if you scrape metrics for multiple application on multiple servers.

Usage

Depending on your metrics you can apply different kind of query. Also those query can be used in the default prometheus GUI or if you have it connected with Grafana, you can use them there to create your custom graphs.

Grafana is an “observability platform” that lets you use the data from prometheus to create dashboards. You can use the same Prometheus queries and display the data as you wish.

For more information on how to use your metrics, check the other sources, it will pretty much depend on your use case: the data you have or what you want to monitor.

Other sources

Here are some interesting sources you may want to take a look when creating, exploiting your metrics: