How to automatically Retry DynamoDB Write on ProvisionedThroughputExceededException

Carvia Tech | July 27, 2018 | 3 min read | 321 views | Amazon DynamoDB


Amazon DynamoDB requires us to provision read and write capacity in advance, otherwise if exceeded will start throwing ProvisionedThroughputExceededException. So lets say you have configured write capacity of 1 record per second, and all of a sudden lots of write requests queue up, you will start receiving this exception in your code. Please be noted that Amazon DynamoDB maintains a sliding window buffer of 5 minutes, i.e. if you do not use any write capacity in last 5 minutes, you can use it in one go, so effectively a spike of 300 writes can be handled gracefully if you have un-utilized quota of last 5 minutes

ProvisionedThroughputExceededException

Your request rate is too high. The AWS SDKs for DynamoDB automatically retry requests that receive this exception. Your request is eventually successful, unless your retry queue is too large to finish. Reduce the frequency of requests and use exponential backoff. For more information, go to Error Retries and Exponential Backoff in the Amazon DynamoDB Developer Guide.

Gracefully handling the failures using Spring Retry with exponential back-off

Spring Retry module provides us a convenient mechanism to handle retry execution with exponential back offs.

You need to make the below changes in your project’s build.gradle file:

build.gradle
compile group: 'org.springframework.retry', name: 'spring-retry', version: '1.1.2.RELEASE'

Now add @EnableRetry annotation to your main Application class.

/src/main/java/com/foo/Application.java
@SpringBootApplication
@EnableRetry    (1)
public class Application {

    public static void main(String[] args) {
        new SpringApplicationBuilder(Application.class).run(args);
    }

}
1 This line is important

Finally we need to apply @Retryable annotation to the method that we want to retry automatically in case of certain exceptions (ProvisionedThroughputExceededException in this case).

/src/main/java/com/foo/ViewsService.java
private static final Logger logger = LoggerFactory.getLogger(ViewService.class);

@Retryable(maxAttempts = 5, value = {ProvisionedThroughputExceededException.class},
        backoff = @Backoff(value = 5000, multiplier = 2))           (1)
public void incrementViews(String app, String id, long delta) {
    customDao.incrementViews(app, id, delta);
}

@Recover
public void recover(ProvisionedThroughputExceededException exception) {
    logger.error("All retries failed - recover from exception ", exception);
}
1 method incrementViews will be retried 5 times (at 5, 10, 20, 40, 80 seconds interval) when ProvisionedThroughputExceededException is thrown by Dao layer.

That’s all we need to do. Exponential backoff will make sure that subsequent retries succeed.

Exponential backoff

An exponential back-off is an algorithm that uses a pre-defined process to gradually, multiplicatively decrease the rate of a given operation to find a more acceptable rate.

Other options worth considering

There are other options worth considering when you are integrating AWS DynamoDB in your application, which will increase fault tolerance of your application. Here they are:

Using SQS for queuing all requests

Amazon SQS is a simple queue service that can be added to your application to make it more resilient to failures. What you can do is put the write requests into SQS and a separate job can poll records from SQS at a limited rate and insert them into Dynamo Db without the fear of having ProvisionedThroughputExceededException. But this approach may not work well where your client needs a synchronous response to the requests.

Using Redis for temporarily handling the spike

In a similar fashion, Redis can be used to handle the temporary spike of incoming resuests and then slowly insert the data into dynamoDB.


Top articles in this category:
  1. What are Conditional Writes in AWS DynamoDB
  2. How will you ensure that no two threads update the same db record in parallel in amazon DynamoDB
  3. How to implement Atomic Counters in DynamoDB for high throughput
  4. Amazon DynamoDB Java Interview Questions
  5. What are Best Practices for Using Amazon DynamoDB?
  6. scan all records of a Amazon DynamoDB table using a Java Code
  7. What is Eventual Consistency in DynamoDB?



Find more on this topic:
NoSQL image
NoSQL

No SQL design and development - Amazon DynamoDB, MongoDB, MySQL document DB.

Last updated 1 month ago


Recommended books for interview preparation:

This website uses cookies to ensure you get the best experience on our website. more info