WebClient for synchronous HTTP Requests

Outgoing REST calls through HTTP. This is something rather basic, and pretty much every application requires it. In this article, we will cover how to consume a third party REST interface using the Spring WebClient. This particular WebClient will serve the requests towards CoinGecko to retrieve prices for cryptocurrencies. In earlier articles, we’ve covered the handling of our own accounts, but the main difference is, that we can know the balances of them ourselves. In the case of crypto, we do need that information from a better source.

• Why Webclient
• What is the WebCLient
• Implementing WebClient
• Flux To List
• Testing

Why WebClient

Starting out with some bad news - at least for me. I always liked using Spring’s RestTemplate. It provided an easy way to do precisely what we’re out to do - making REST calls. Unfortunately, since Spring 5.0, the RestTemplate is in maintenance mode.

So, since we want to go with the flow and not implement something that’s to be deprecated in future Spring versions, we will follow their guidance and implement this functionality the new shiny way - using the WebClient.

What is the WebClient

The WebClient is Spring’s way of making outgoing web requests through HTTP, as already mentioned. Important to note however, is that it was part of Spring’s Web Reactive module. This means that the WebClient can be used as a non-blocking, reactive solution.

Now, I’m not a huge fan of using the Reactive Code principles everywhere. Certainly, there are usecases where it makes a lot of sense and brings huge advantages. The main reason I dislike it is that it is extremely hard to debug. Technically, nothing happens in the entire code until it is really required. Afterwards, everything is handling in a Flux or a Mono, and conventional tooling simply does not cover an elegant way of looking what is inside those Fluxes or Monoes.

Additionally, there are some parts that are yet to be figured out. For instance, JPA can technically handle the reactive stack and has its own reactive implementation. However, in that implementation, the OneToMany or** ManyToMany** relationships are not yet covered.

Fortunately though, the WebClient can be used for synchronous functionality also. This is what we’ll do in this case, by simply calling the .block() method on the Flux/Mono. When the other kinks are worked out - who knows, I just may migrate over to the Reactive stack fully.

Implementing WebClient

WebClient

First, we’ll create a basic WebClient with some minimal configuration.

@Configuration
class WebClientConfiguration {

    @Bean
    fun coinGeckoWebClient(@Value("\${service.coingecko.base-url}") baseUrl: String): WebClient {
        return WebClient.builder()
            .baseUrl(baseUrl)
            .clientConnector(ReactorClientHttpConnector(httpClient()))
            .defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
            .defaultHeader(HttpHeaders.ACCEPT, "${MediaType.APPLICATION_JSON}")
            .defaultHeader(HttpHeaders.ACCEPT_CHARSET, StandardCharsets.UTF_8.toString())
            .build()
    }

    @Bean
    fun httpClient(): HttpClient {
        return HttpClient.create()
            .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 5000)
            .responseTimeout(Duration.ofMillis(5000))
            .doOnConnected { conn ->
                conn.addHandlerLast(ReadTimeoutHandler(5000, TimeUnit.MILLISECONDS))
                    .addHandlerLast(WriteTimeoutHandler(5000, TimeUnit.MILLISECONDS))
            }
    }
}

As mentioned earlier, we will utilize the WebClient to retrieve information from CoinGecko, so we’ve set that as a base URL.

Additionally, we set some basic headers regarding the Content-Type, Accept. These are potentially not required, but many services offer their responses using different representations, and this way, we’ll be sure that we’re served with the items that we expect.

Gateway

Now that we’ve created the WebClient for CoinGecko, we will create a gateway that will use the WebClient to make the request and then perform some additional operations on it.

The use of the gateway is to keep the separation of concerns clean, and this is also the latest stage where the DTOs are still DTOs. They’ll be mapped in here, so that any place that makes use of the gateway will have our nice domain objects, which we own.

Talking about the DTO, that looks as follows (created from the API definition):

data class CryptoAssetDto(
    val id: String,
    val symbol: String,
    val name: String,
    @JsonProperty("current_price")
    val price: Double,
    @JsonProperty("market_cap_rank")
    val marketCapRank: Int
)

Its mapping is created using an extension function, just as all our other mappings:

fun CryptoAssetDto.toDomain() = CryptoAsset(
    id = this.id,
    name = this.name,
    symbol = this.symbol.uppercase(),
    marketCapRank = this.marketCapRank,
    price = this.price
)

@Component
class CoinGeckoGateway @Inject constructor(private val coinGeckoWebClient: WebClient) {

    fun loadGreatestCoins(page: Int): Flux<CryptoAsset> {

        return coinGeckoWebClient.get().uri { uriBuilder ->
            uriBuilder
                .path("/coins/markets")
                .queryParam("vs_currency", "usd")
                .queryParam("order", "market_cap_desc")
                .queryParam("per_page", "250")
                .queryParam("sparkline", "false")
                .queryParam("page", page)
                .build()
        }
            .retrieve()
            .bodyToFlux(CryptoAssetDto::class.java)
            .map { it.toDomain() }
    }
}

Some things to note:

• The .path() sets the actual endpoint beyond the baseUrl
• .queryParam() adds query parameters and can be used many times
• .retrieve() defines that the building of the request is done, and everything that follows will perform operations on the response
• In the bodyToFlux() line, the response is deserialized into a Flux of CryptoAssetDto
• .map uses the Flux API to convert the Flux items into our domain objects

Testing it

Now, if we hook up the Gateway to some part of the code (more on that below) and we test it out, we can already run into the issue of how Flux cannot be easily debugged. For instance, if we set a breakpoint towards the end of the gateway, we can see the following:

As you can see, there is nothing close to helpful in the debugger…

There can sometimes be ways to mitigate these issues. In this case here for instance, we additionally put a breakpoint in the mapping. If we’d never reach the breakpoint, we’d know the response was either empty, or some other issue has occurred.

Fortunately, everything was set up correctly:

Even more closely to the Flux, we could also put the breakpoint inside the .map() lambda. Feel free to try this.

Flux To List

Okay, so now we’ve got a potential Flux of CryptoAssets. We do need to do something with them as well though.

Since our Database is not reactive as of now, we need to convert the Flux to a List. Our usecase here is to load the prices daily for the top 2000 cryptocurrencies, and store the prices in the DB.

Essentially, any new asset will be stored precisely as it was returned, and recurring currencies will have a new snapshot created, which is added with the timestamp to the existing entry.

@Service
class CryptoAssetService @Inject constructor(
    private val cryptoAssetRepository: CryptoAssetRepository,
    private val coinGeckoGateway: CoinGeckoGateway
) {

    fun loadDailyPrices() {
        for (page in 1..8) {
            val coinList: MutableList<CryptoAsset>? = coinGeckoGateway.loadGreatestCoins(page).collectList().block()
            coinList?.map {
                cryptoAssetRepository.findById(it.id).toNullable()?.also { coin -> coin.price = it.price } ?: it
            }
                ?.map { it.addNewSnapshot() }
                ?.let { cryptoAssetRepository.saveAll(it) }
        }
    }

    fun getAllAssets(): List<CryptoAsset> {
        return cryptoAssetRepository.findAllByOrderByMarketCapRank()
    }
}

You can ignore the getAllAssets() method for now. This is really only for debugging practices.

There is not much to state here, except for the one very important method. As is custom for the WebClient, it is reactive - meaning that everything happens asynchronously. Essentially, nothing happens until it is really required, or at least it’s not possible to state when anything happens.

We can make the request synchronous by calling the .block() method. This method can only be called on a Mono, which is why we first call collectList(), which transforms the Flux<CryptoAsset> we’ve had earlier into a Mono<List<CryptoAsset>>. Calling block() on the Mono will return, synchronously the List<CryptoAsset>.

Testing

Okay, this may be cheating a little, as I will only show snippets of the manual tests. Unit Tests and Automated integration Tests for WebClient deserve their own, separate article.

However, we now want to check whether everything works as expected. For this, we have the getAllAssets() functions from earlier.

So, we will run a couple of requests after starting the application. Upon the first retrieval of our stored assets, it should be an empty list. After the first loading, there should be lots of entries, all with one snapshot. Then, finally, we’ll make a second request, and we should see some multiple snapshots, with some having different values.

These operations are triggered through our stubs. Here are the corresponding Controllers:

@RestController
@CrossOrigin
@Profile("!prd")
class CryptoAssetStubController @Inject constructor(
    private val cryptoAssetService: CryptoAssetService,
    private val cryptoAssetRepository: CryptoAssetRepository
) {

    @PostMapping("/stubs/assets/load")
    fun loadAssets(): ResponseEntity<Any> {
        val assets = cryptoAssetService.loadDailyPrices()
        return ResponseEntity.ok(assets)
    }
}

The StubController is only there for helping in testing. It will not be available in Production, where the daily retrieval will actually happen in regular, timed intervals.

The loading of crypto assets from our database will actually also be needed, as this may be required by the frontend. Here is that endpoint:

@RestController
@CrossOrigin
class CryptoController @Inject constructor(private val cryptoPositionService: CryptoService) {
...

    @GetMapping(CRYPTOS)
    fun getCryptoPositions(): ResponseEntity<List<CryptoPositionDto>> {
        return ResponseEntity.ok(cryptoPositionService.getCryptoPositions().map { it.toDto() })
    }
    ...
}

Round 0

So, first retrieval of stored values…

As expected, we have no values, as we’re starting with an empty table.

Round 1

After calling /stubs/assets/load once, we’ve supposedly loaded the entries in the table. Let’s check it out by retrieving /cryptos/assets again.

Round 2

Now we’ve already got some values. Let’s run another round of calling /stubs/assets/load followed by /cryptos/assets, to see whether we now have multiple snapshots for each crypto.

There you have it. We now have a way of creating snapshots of crypto values by consuming APIs from a third party!

Plus, we saw that Bitcoin actually went up a little during those couple of minutes. Way to go!!