1. What You Will Learn

  • How to use Ribbon as a client side load balancer

  • How to use a Ribbon enabled RestTemplate

2. Start the foundation services

  1. Start the config-server in a terminal window. You may have terminal windows still open from previous labs. They may be reused for this lab.

    $ cd config-server
    $ mvn spring-boot:run
  2. Start the service-registry

    $ cd service-registry
    $ mvn spring-boot:run
  3. Start two instances of the fortune-service, as follows:

    1. Start your first instance as usual:

      $ cd fortune-service
      $ mvn spring-boot:run

      This will give us a running fortune-service on http://localhost:8787/

    2. Start a second instance by picking a different port for the second instance to listen on:

      Mac, Linux:

      SERVER_PORT=8788 mvn spring-boot:run

      Windows:

      $ set SERVER_PORT=8788
      $ mvn spring-boot:run
    3. Now check the eureka dashboard at http://localhost:8761/ and verify that you have two local instances of the fortune service running

3. Set up greeting-ribbon

No additions to the pom.xml

In this case, we don’t need to explicitly include Ribbon support in the pom.xml. Ribbon support is pulled in through transitive dependencies (dependencies of the dependencies we have already defined).

  1. Review the class greeting-ribbon/src/main/java/io/pivotal/greeting/GreetingController.java. Notice the loadBalancerClient. It is a client-side load balancer (Ribbon). Review the fetchFortuneServiceUrl() method. Ribbon is integrated with Eureka so that it can discover services as well. Notice how the loadBalancerClient chooses a service instance by name.

    package io.pivotal.greeting;
    
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.cloud.client.ServiceInstance;
    import org.springframework.cloud.client.loadbalancer.LoadBalancerClient;
    import org.springframework.stereotype.Controller;
    import org.springframework.ui.Model;
    import org.springframework.web.bind.annotation.RequestMapping;
    import org.springframework.web.client.RestTemplate;
    
    @Controller
    public class GreetingController {
    
      private final Logger logger = LoggerFactory.getLogger(GreetingController.class);
    
      private final LoadBalancerClient loadBalancerClient;
    
      public GreetingController(LoadBalancerClient loadBalancerClient) {
        this.loadBalancerClient = loadBalancerClient;
      }
    
      @RequestMapping("/")
      String getGreeting(Model model) {
    
        logger.debug("Adding greeting");
        model.addAttribute("msg", "Greetings!!!");
    
        RestTemplate restTemplate = new RestTemplate();
        String fortune = restTemplate.getForObject(fetchFortuneServiceUrl(), String.class);
    
        logger.debug("Adding fortune: {}", fortune);
        model.addAttribute("fortune", fortune);
    
        return "greeting"; // resolves to the greeting.ftl template
      }
    
      private String fetchFortuneServiceUrl() {
        ServiceInstance instance = loadBalancerClient.choose("fortune-service");
    
        logger.debug("uri: {}", instance.getUri().toString());
        logger.debug("serviceId: {}", instance.getServiceId());
    
        return instance.getUri().toString();
      }
    
    }
  2. Open a new terminal window. Start the greeting-ribbon app.

    $ cd greeting-ribbon
    $ mvn spring-boot:run
  3. After the a few moments, check the service-registry dashboard http://localhost:8761. Confirm the greeting-ribbon app is registered.

  4. Browse to the greeting-ribbon application at http://localhost:8080/. Confirm you are seeing fortunes. As you refresh the greeting-ribbon app’s main page, observe the log output for greeting-ribbon in the terminal. Inspect the logged uri and serviceId values. You should see these go back and forth between the two fortune-service instances, in round-robin fashion. Ribbon is doing client-side load balancing!

  5. Stop the greeting-ribbon application.

4. Refactor to use a load-balanced RestTemplate

LoadBalancerClient provides a simple way to ask Ribbon to choose() an application instance from among multiple running instances. Spring Cloud Netflix provides an alternative, more elegant mechanism to make load-balanced REST API calls, which transparently invokes the Ribbon load balancer, via a cross-cutting annotation named @LoadBalanced.

Let’s see how this works by refactoring our existing implementation:

  1. In our application class, GreetingRibbonApplication, explicitly inject a RestTemplate bean, and decorate it with the @LoadBalanced annotation, as shown in this diff:

    package io.pivotal;
    
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
    +import org.springframework.cloud.client.loadbalancer.LoadBalanced;
    +import org.springframework.context.annotation.Bean;
    +import org.springframework.web.client.RestTemplate;
    
    @SpringBootApplication
    @EnableDiscoveryClient
    public class GreetingRibbonApplication {
    
      public static void main(String[] args) {
        SpringApplication.run(GreetingRibbonRestApplication.class, args);
      }
    
    +  @LoadBalanced
    +  @Bean
    +  RestTemplate restTemplate() {
    +    return new RestTemplate();
    +  }
    
    }

    Simply adding the annotation will ensure that calls made using this RestTemplate instance will be load-balanced by Ribbon.

  2. To use this RestTemplate instance, let’s inject it into our GreetingController using Spring constructor injection, as shown by this diff:

    -  private final LoadBalancerClient loadBalancerClient;
    +  private final RestTemplate restTemplate;
    
    -  public GreetingController(LoadBalancerClient loadBalancerClient) {
    +  public GreetingController(RestTemplate restTemplate) {
    -    this.loadBalancerClient = loadBalancerClient;
    +    this.restTemplate = restTemplate;
      }
  3. Next, inside our getGreeting() method, we can remove the line that explicitly instantiates a RestTemplate and use the Autowired instance in its place. Furthermore, instead of calling fetchFortuneServiceUrl(), we use a clever String parameter that looks like a url, but that is in fact a template or placeholder for the actual url that Ribbon will derive from it:

    @RequestMapping("/")
    String getGreeting(Model model) {
    
      logger.debug("Adding greeting");
      model.addAttribute("msg", "Greetings!!!");
    
    -  RestTemplate restTemplate = new RestTemplate();
    -  String fortune = restTemplate.getForObject(fetchFortuneServiceUrl(), String.class);
    +  String fortune = restTemplate.getForObject("http://fortune-service", String.class);
    
      logger.debug("Adding fortune: {}", fortune);
      model.addAttribute("fortune", fortune);
    
      return "greeting"; // resolves to the greeting.ftl template
    }

    Basically http://fortune-service references the name of the application we wish to invoke. Spring Cloud Netflix automatically parses the string, looks up the application name against a locally cached copy of the Eureka registry, and asks Ribbon to choose an instance, whose url is then inserted in the place of the application name.

  4. We can now delete the unused method fetchFortuneServiceUrl()

A reference implementation of greeting-ribbon refactored as described above is available under the project named greeting-ribbon-rest.

Let’s take our refactored implementation for a spin:

$ cd greeting-ribbon
$ mvn clean spring-boot:run
  1. After the a few moments, check the service-registry dashboard at http://localhost:8761. Confirm the greeting-ribbon app is registered once more.

  2. Browse to http://localhost:8080/ to the greeting-ribbon application. Confirm you are seeing fortunes. Refresh as desired. Review the terminal output for the greeting-ribbon app.

  3. More interestingly, review the logs for the two fortune-service applications. With each refresh of greeting-ribbon, one of these two will serve the response, in alternating fashion.

5. Retrying Failed Requests

Conduct the following experiment:

  • Make sure you have two instances of fortune-service running

  • Visit the greeting application, and again, verify that requests round-robin between the two instances

  • Stop or kill one of the two fortune-service instances

  • Continue making requests to the greeting application

One of two requests simply fails.

If we wait long enough for Eureka to remove the insance from the list (since it won’t be receiving heartbeats any longer), and for the greeting application to fetch an updated registry, this matter will resolve itself.

Given that a second instance is available to handle these requests, it’s preferable if Ribbon were to just retry the request against the surviving instance.

Let’s rectify this problem.

Netflix Ribbon supports retrying failed requests. The Ribbon GitHub Wiki cites configuration properties such as MaxAutoRetries and MaxAutoRetriesNextServer that govern this behavior. The Spring Cloud team supports this capability via the Spring Retry project. Ryan Baxter, a member of the Spring Cloud team, discusses this integration work in this blog (so does the Spring Cloud Netflix reference documentation, for that matter).

➜ Add the Spring Retry dependency to pom file for greeting-ribbon:

greeting-ribbon/pom.xml
@@ -41,6 +41,10 @@
       <groupId>io.pivotal.spring.cloud</groupId>
       <artifactId>spring-cloud-services-starter-service-registry</artifactId>
     </dependency>
+    <dependency>
+      <groupId>org.springframework.retry</groupId>
+      <artifactId>spring-retry</artifactId>
+    </dependency>
     <dependency>
       <groupId>org.springframework.boot</groupId>
       <artifactId>spring-boot-starter-freemarker</artifactId>

➜ Configure MaxAutoRetriesNextServer for the ribbon client:

greeting-ribbon/src/main/resources/application.yml
---
fortune-service:
  ribbon:
    MaxAutoRetriesNextServer: 1

This setting can be interpreted as if a request to one of the servers from the list fails, try another server.

➜ Also, configure the logging level for the package org.springframework.retry to DEBUG:

greeting-ribbon/src/main/resources/application.yml
logging:
  level:
    org.springframework.retry: DEBUG

Finally, repeat the test:

  • Ensure two instances of fortune-service are running and registered with eureka.

  • Start greeting-ribbon, and ensure that requests round-robin between the two instances.

  • Take down one of the fortune-service instances.

  • Access the greeting application a few more times: you should see no failures.

Also observe that the console output for greeting-ribbon contains a log message from the Spring Retry library stating that the retry count was 1 (i.e. that it actually retried a request):

...
o.s.retry.support.RetryTemplate          : Retry: count=1
io.pivotal.greeting.GreetingController   : Adding fortune: You can always find happiness at work on Friday
io.pivotal.greeting.GreetingController   : Adding greeting
...

6. Customize the Load Balancing Rule

The Ribbon API was designed to be flexible, with multiple interfaces whose implementations define its behavior:

  • IRule: defines the rules that Ribbon uses for load balancing. The default is RoundRobinRule

  • IPing: the mechanism Ribbon uses to ping services (to check if they are alive)

  • ILoadBalancer: a more general interface that includes how Ribbon obtains the list of servers to load-balance across

Out of the box, Ribbon provides multiple alternative implementations of IRule. Our task here is to configure Ribbon to use the WeightedResponseTimeRule instead of the default RoundRobinRule.

We can simulate two instances of fortune-service with different response times by using an artificial delay in the response for one of our two fortune-service instances. The fortune service has been configured with the property (or environment variable) DELAY_MS to allow us to do just that.

  1. Start two instances of fortune-service once more, but this time, as follows:

    mvn spring-boot:run

    ..and:

    Mac, Linux:

    SERVER_PORT=8788 DELAY_MS=1000 mvn spring-boot:run

    Windows:

    $ set SERVER_PORT=8788
    $ set DELAY_MS=1000
    $ mvn spring-boot:run

    At this point you can verify once more that you have two instances of fortune-service registered with Eureka, and that one of them takes longer to respond than the other.

  2. Refer to the following Spring Cloud Ribbon documentation which discusses how different aspects of Ribbon can be configured using properties.

  3. In greeting-ribbon src/main/resources folder, create the application.yml file and edit it as follows:

    fortune-service:
      ribbon:
        NFLoadBalancerRuleClassName: com.netflix.loadbalancer.WeightedResponseTimeRule

    Note above how a spring boot application can define and configure multiple independent Ribbon clients. Each client has a name, in this case corresponding to the service being called.

  4. Start the greeting-ribbon application once more, and refresh the page to see Ribbon again load-balance requests across the two fortune-service instances.

    1. You should begin to see log messages indicating that a background thread is running that computes updated response times for each instance.

    2. Within a short time, greeting-ribbon will adapt to invoking the faster fortune-service instance in proportion to its response time; i.e. it will be invoked more often than the slower instance.

Congratulations! You’ve now configured Ribbon with a custom load balancing rule. You can now stop the config-server, service-registry, fortune-service and greeting-ribbon applications.

7. Deploy the greeting-ribbon to PCF

7.1. PCF and the Eureka Registration Method

In the previous lab, we used the route registration method. This had the effect of giving client applications the same url for all application instances. This setting defeats Ribbon, in that no matter which application instance we select, we end up with the same application url, which by definition will be routed through the PCF GoRouter.

In order to bypass the GoRouter and leverage Ribbon, the registration method has to be set to direct.

When using the direct registration method, all application instances register their internal IP address and port number with Eureka. This can be a problem in PCF however, for a number of reasons:

  • Cloud Foundry may not be configured for direct container-to-container networking,

  • Even with container-to-container networking enabled, platform security policy may not allow applications to call one another directly

A new feature of PCF version 1.10 known as "container-to-container" networking, provides a mechanism for an operator for applications to call each other directly by IP address, effectively bypassing the GoRouter.

A recent blog post by Chris Sterling gives a great overview of how this feature works together with Spring Cloud Services.

Assuming you’re working with an instance of Cloud Foundry that has direct networking enabled, let’s begin by modifying our application configuration:

  1. In your app-config git repository, locate application.yml

  2. Modify the value of the property spring.cloud.services.registrationMethod from route to direct

  3. Commit and push your changes back to your github repository

  4. Finally, restart your running fortune-service instances, to cause them to re-register with eureka using their IP addresses:

    cf restart fortune-service

7.2. Deploy 'greeting-ribbon'

  1. Package and push the greeting-ribbon application.

    $ mvn clean package
    $ cf push greeting-ribbon -p target/greeting-ribbon-0.0.1-SNAPSHOT.jar -m 768M --random-route --no-start
  2. Bind services for the greeting-ribbon application.

    cf bind-service greeting-ribbon config-server

    ..and:

    cf bind-service greeting-ribbon service-registry

    You can safely ignore the TIP: Use 'cf restage' to ensure your env variable changes take effect message from the CLI. We don’t need to restage at this time.

  3. Start the greeting-ribbon app.

    cf start greeting-ribbon
  4. After the a few moments, check the service-registry. Confirm the greeting-ribbon app is registered.

Instead of deploying greeting-ribbon as described above, did you consider using a CloudFoundry manifest? Below is an example (please don’t copy verbatim; for one, you’ll likely want to change the value for host).

---
applications:
- name: greeting-ribbon
  path: target/greeting-ribbon-0.0.1-SNAPSHOT.jar
  host: eitan-gr
  memory: 768M
  services:
  - config-service
  - discovery-service

Manifest files simplify the task of deploying an application, and a single cf push command can replace the multiple individual cf commands we had to invoke above.

View Instance Registrations in PCF

Observe the PCF discovery service dashboard.

Eureka Dashboard

Did you notice how each microservice is registering using its direct IP address?

This gives Ribbon a distinct url for each application instance, allowing it to control load balancing.

One advantage of direct networking between microservices is that we are no longer forced to create a publicly accessible route to backing services. By removing these routes, we ensure that these services are always accessed via other microservices. In other words, they can be made inaccessible to direct calls from outside Cloud Foundry.

For more details, please read the following.

Refresh the greeting-ribbon root endpoint.

This should fail.

In a nutshell, you’ll need to invoke the following command to allow direct communication from the greeting-ribbon application to the fortune-service application:

cf add-network-policy greeting-ribbon --destination-app fortune-service --protocol tcp --port 8080

Now retry refreshing the greeting-ribbon root endpoint. If fortune-service is still scaled at three instances, you should see Ribbon load balance requests across these instances.

How do you tell if calls to fortune-service are load-balanced?

Here’s one way: in the log output below for fortune-service, look at the text inside the first set of square brackets:

fortune-service log output
➜  cf logs fortune-service
Connected, tailing logs for app fortune-service in org eitan-org / space dev as admin...

2017-04-04T20:55:14.37-0500 [APP/PROC/WEB/2] io.pivotal.fortune.FortuneController     : fetching fortune.
2017-04-04T20:55:17.31-0500 [APP/PROC/WEB/0] io.pivotal.fortune.FortuneController     : fetching fortune.
2017-04-04T20:55:17.58-0500 [APP/PROC/WEB/1] io.pivotal.fortune.FortuneController     : fetching fortune.
2017-04-04T20:55:17.69-0500 [APP/PROC/WEB/2] io.pivotal.fortune.FortuneController     : fetching fortune.
2017-04-04T20:55:17.84-0500 [APP/PROC/WEB/0] io.pivotal.fortune.FortuneController     : fetching fortune.
2017-04-04T20:55:18.09-0500 [APP/PROC/WEB/1] io.pivotal.fortune.FortuneController     : fetching fortune.
2017-04-04T20:55:19.95-0500 [APP/PROC/WEB/2] io.pivotal.fortune.FortuneController     : fetching fortune.
2017-04-04T20:55:20.15-0500 [APP/PROC/WEB/0] io.pivotal.fortune.FortuneController     : fetching fortune.
2017-04-04T20:55:20.29-0500 [APP/PROC/WEB/1] io.pivotal.fortune.FortuneController     : fetching fortune.

Did you notice how the instance index is reliably cycling from 2 to 0 to 1?

In the log output above, did you also notice how there are no log messages with the RTR log type? This is not a coincidence. It is because the calls from greeting-ribbon to fortune-service are direct: they no longer travel via the GoRouter.