Circuit Breakers

I first encountered the idea of software circuit breakers when reading the first edition of the book Release It. In this lab we’re going to gain familiarity with Hystrix: Netflix’s open source implementation of the circuit breaker pattern. A number of Netflix’s open source projects have been adapted for Spring and are now offered as a Spring Cloud project named Spring Cloud Netflix.

The main problem we’re dealing with here is what happens if some microservice dependency begins to degrade or fail? There’s a danger that the failure will cascade to clients, who will also begin to degrade and fail. Circuit breakers can make a system of microservices less fragile, in other words, more resilient to such failures.

Launch both fortune-service and greeting, and visit greeting’s endpoint. We should see a greeting and a fortune.
Stop the fortune service
Refresh the greeting application’s view in your browser

What happens? We don’t handle the situation when the service is not available. The call to the fortune service fails, which throws an exception that we don’t handle.

At this point we could simply wrap a try-catch clause in greeting, around the call to the fortune service. That would be an improvement, but it doesn’t solve the issue of cascading failures.

1. Adding Hystrix to the Greeting Application

Add the hystrix dependency to the greeting application's build file:

Add to build.gradle

@@ -30,6 +30,8 @@ dependencies {
   compile 'org.springframework.boot:spring-boot-starter-actuator'
   compile 'org.springframework.boot:spring-boot-starter-freemarker'

+  compile 'org.springframework.cloud:spring-cloud-starter-hystrix'
+
   runtime 'org.springframework.boot:spring-boot-devtools'

   compileOnly 'org.projectlombok:lombok'

In GreetingApplication add the @EnableCircuitBreaker annotation, like so:

Enable circuit breakers

@@ -2,10 +2,12 @@ package io.pivotal.training;

 import org.springframework.boot.SpringApplication;
 import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
 import org.springframework.context.annotation.Bean;
 import org.springframework.web.client.RestTemplate;

 @SpringBootApplication
+@EnableCircuitBreaker
 public class GreetingApplication {

   public static void main(String[] args) {

Finally, in FortuneServiceClient, target the call to the fortune service, and "wrap" it with a circuit breaker.

Add a breaker around the call to the fortune service

@@ -1,5 +1,6 @@
 package io.pivotal.training.greeting;

+import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.beans.factory.annotation.Value;
 import org.springframework.stereotype.Component;
@@ -19,10 +20,17 @@ public class FortuneServiceClient {
     this.restTemplate = restTemplate;
   }

+  @HystrixCommand(fallbackMethod = "defaultFortune") (1)
   public String getFortune() {
     Map<String,String> result = restTemplate.getForObject(baseUrl, Map.class);
     String fortune = result.get("fortune");
     log.info("received fortune '{}'", fortune);
     return fortune;
   }
+
+  public String defaultFortune() { (2)
+    log.info("Default fortune used.");
+    return "Your future is uncertain";
+  }
+
 }

1	Tag the method with the annotation
2	Provide a fallback implementation, invoked when getFortune() fails

The fallback method’s signature must match the signature (return type, method arguments) of the method it stands in for.

The above is only one mechanism to apply a circuit breaker, but it’s an elegant one. Similar to other Spring annotations (@Transactional and @Cacheable come to mind), this annotation applies a type of cross-cutting concern to the target method. To learn more about the @HystrixCommand, see the hystrix-contrib project javanica.

Let’s repeat our manual test:

Start up both apps once more
Make sure the greeting application is fetching and displaying fortunes
Stop the fortune service
Refresh the greeting page

Does the greeting application fall back to the default fortune?

⇒ Restart the fortune-service

Does the greeting application revert to using the fortune-service right away?

It should, because a single failure will not trip the circuit. By default Hystrix is configured to trip the circuit when it reaches a 50% failure rate against a minimum volume of 20 requests in a 10-second period. But if we do manage to trip the circuit, it will then back off and short-circuit calls for a short period. The Hystrix Wiki's How it works and Configuration sections provide the details.

Each Hystrix command can be configured directly via the annotation. Here’s one example (from the documentation) worth calling out:

Configuring the Hystrix Command with a custom timeout

@HystrixCommand(commandProperties = {
    @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "500")
})
public User getUserById(String id) {
    return userResource.getUserById(id);
}

A timeout is one type of failure that will count toward tripping a circuit. Above, the timeout is customized to 500ms.

2. One more Test

We just verified that when a call to the fortune service fails, we fall back to the default fortune. It wouldn’t be a bad idea to add an automated test into our suite that exercises this behavior regularly. We basically want to make sure that our circuit breaker is in place and functioning.

Here is a simple implementation of such a test. It’s a variation on the FortuneServiceClientTests. This test is designed not to stand up a stub, and specifically we instrument a mock restTemplate to throw an exception when invoked. The goal here is to verify that the circuit breaker is in place and will fall back to the default fortune:

Testing our circuit breaker

package io.pivotal.training.greeting;

import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.web.client.RestTemplate;

import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Matchers.any;
import static org.mockito.Matchers.anyString;
import static org.mockito.Mockito.when;

@RunWith(SpringRunner.class)
@SpringBootTest
public class FortuneServiceClientFallbackTests {
  @MockBean RestTemplate restTemplate;
  @Autowired private FortuneServiceClient fortuneServiceClient;

  @Before
  public void setup() {
    when(restTemplate.getForObject(anyString(), any()))
        .thenThrow(new RuntimeException("something went wrong"));
  }

  @Test
  public void shouldFallbackToDefaultFortune() {
    String defaultFortune = fortuneServiceClient.defaultFortune();

    String fortune = fortuneServiceClient.getFortune();

    assertThat(fortune).isEqualTo(defaultFortune);
  }
}

3. The Dashboard

One of the stated design goals of the Hystrix project is to Enable near real-time monitoring, alerting, and operational control. Hystrix computes real-time statistics with information about each Hystrix command. With Spring Cloud, this information is exposed as a stream via the actuator endpoint /hystrix.stream.

⇒ In a browser, visit the /hystrix.stream endpoint for the greeting application

It should look something like this:

In that output, you should see the name of the hystrix command[s] in question, the state of the circuit (open or closed), and many additional statistics including request counts, successes, failures, and latencies.

The Hystrix project provides a dashboard that can consume and present the information from that stream in a way that is very easy to interpret and consume. The following graphic, taken directly from the project’s wiki page, details how the information is presented and what it represents:

3.1. Stand up a Hystrix Dashboard

Visit the spring initializr at http://start.spring.io/ and build yourself a hystrix dashboard project, as shown below:

Click Generate Project, unzip the downloaded file
Import the project into your IDE
Configure the dashboard to run on a unique port, say, 8082:
/src/main/resources/application.properties
```
@@ -0,0 +1 @@
+server.port=8082
```

Turn on, or enable the Hystrix Dashboard, by adding a specific annotation to the project’s Application class:

/src/main/java/io/pivotal/training/springcloud/hystrixdashboard/HystrixDashboardApplication.java

 import org.springframework.boot.SpringApplication;
 import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.cloud.netflix.hystrix.dashboard.EnableHystrixDashboard;

 @SpringBootApplication
+@EnableHystrixDashboard
 public class HystrixDashboardApplication {

Start up: fortune-service, greeting, and this dashboard.
Visit the hystrix dashboard endpoint at http://localhost:8082/hystrix
Paste the greeting application’s /hystrix.stream endpoint into the form field, as shown below:
Finally, click Monitor Stream. You should now be able to view and monitor our getFortune circuit breaker (notice how the hystrix command’s name is inferred from the method the annotation decorates).

We have much more visibility into the state of our circuit, the amount of traffic going through it, number of requests, failures, etc..

Take the time once more to experiment with the system. Take down the fortune-service. Consider using a tool such as apache ab to submit multiple requests in a short time interval to the greeting application to trip the circuit, and observe the failure count go up, the circuit tripping.

Then, do the reverse: restart the fortune-service, observe that the greeting application will continue to short-circuit calls to the fortune service for some time before it tries again and re-establishes communication with it, at which time the circuit’s state should revert to closed.

4. Congratulations

Circuit breakers help make a microservices-based system less fragile, and allows that system to continue to operate in the face of failures (or latency) in dependent downstream services.

At the moment, our dashboard is consuming a single Hystrix stream. In a subsequent lab, we’ll look at another Netflix project named Turbine, and explore how multiple streams can be aggregated and fed to the dashboard to allow us to monitor multiple instances of a service, as well as multiple distinct services.