undefined## Circuit Breaker: Hystrix Dashboard {#circuit-breaker-hystrix-dashboard}
One of the main benefits of Hystrix is the set of metrics it gathers about each HystrixCommand. The Hystrix Dashboard displays the health of each circuit breaker in an efficient manner.
Figure 3. Hystrix Dashboard
To run the Hystrix Dashboard annotate your Spring Boot main class with @EnableHystrixDashboard
. You then visit /hystrix
and point the dashboard to an individual instances /hystrix.stream
endpoint in a Hystrix client application.
Turbine
Looking at an individual instances Hystrix data is not very useful in terms of the overall health of the system. Turbine is an application that aggregates all of the relevant /hystrix.stream
endpoints into a combined /turbine.stream
for use in the Hystrix Dashboard. Individual instances are located via Eureka. Running Turbine is as simple as annotating your main class with the @EnableTurbine
annotation (e.g. using spring-cloud-starter-turbine to set up the classpath). All of the documented configuration properties from the Turbine 1 wiki) apply. The only difference is that the turbine.instanceUrlSuffix
does not need the port prepended as this is handled automatically unless turbine.instanceInsertPort=false
.
The configuration key turbine.appConfig
is a list of eureka serviceIds that turbine will use to lookup instances. The turbine stream is then used in the Hystrix dashboard using a url that looks like: [http://my.turbine.sever:8080/turbine.stream?cluster=<CLUSTERNAME>](http://my.turbine.sever:8080/turbine.stream?cluster=<CLUSTERNAME>");
(the cluster parameter can be omitted if the name is "default"). The cluster
parameter must match an entry in turbine.aggregator.clusterConfig
. Values returned from eureka are uppercase, thus we expect this example to work if there is an app registered with Eureka called "customers":
turbine: aggregator: clusterConfig: CUSTOMERS appConfig: customers
The clusterName
can be customized by a SPEL expression in turbine.clusterNameExpression
with root an instance of InstanceInfo
. The default value is appName
, which means that the Eureka serviceId ends up as the cluster key (i.e. the InstanceInfo
for customers has an appName
of "CUSTOMERS"). A different example would be turbine.clusterNameExpression=aSGName
, which would get the cluster name from the AWS ASG name. Another example:
turbine: aggregator: clusterConfig: SYSTEM,USER appConfig: customers,stores,ui,admin clusterNameExpression: metadata['cluster']
In this case, the cluster name from 4 services is pulled from their metadata map, and is expected to have values that include "SYSTEM" and "USER".
To use the "default" cluster for all apps you need a string literal expression (with single quotes):
turbine: appConfig: customers,stores clusterNameExpression: 'default'
Spring Cloud provides a spring-cloud-starter-turbine
that has all the dependencies you need to get a Turbine server running. Just create a Spring Boot application and annotate it with @EnableTurbine
.
Turbine AMQP
In some environments (e.g. in a PaaS setting), the classic Turbine model of pulling metrics from all the distributed Hystrix commands doesn’t work. In that case you might want to have your Hystrix commands push metrics to Turbine, and Spring Cloud enables that with AMQP messaging. All you need to do on the client is add a dependency to spring-cloud-netflix-hystrix-amqp
and make sure there is a Rabbit broker available (see Spring Boot documentation for details on how to configure the client credentials, but it should work out of the box for a local broker or in Cloud Foundry).
On the server side Just create a Spring Boot application and annotate it with @EnableTurbineAmqp
and by default it will come up on port 8989 (point your Hystrix dashboard to that port, any path). You can customize the port using either server.port
or turbine.amqp.port
. If you have spring-boot-starter-web
and spring-boot-starter-actuator
on the classpath as well, then you can open up the Actuator endpoints on a separate port (with Tomcat by default) by providing a management.port
which is different.
You can then point the Hystrix Dashboard to the Turbine AMQP Server instead of individual Hystrix streams. If Turbine AMQP is running on port 8989 on myhost, then put [http://myhost:8989](http://myhost:8989)
in the stream input field in the Hystrix Dashboard. Circuits will be prefixed by their respective serviceId, followed by a dot, then the circuit name.
Spring Cloud provides a spring-cloud-starter-turbine-amqp
that has all the dependencies you need to get a Turbine AMQP server running. You need Java 8 to run the app because it is Netty-based.