How To Test Servant Applications

Even with a nicely structured API that passes Haskell’s strict type checker, it’s a good idea to write some tests for your application.

In this recipe we’ll work through some common testing strategies and provide examples of utlizing these testing strategies in order to test Servant applications.

Testing strategies

There are many testing strategies you may wish to employ when testing your Servant application, but included below are three common testing patterns:

• We’ll use servant-client to derive client functions and then send valid requests to our API, running in another thread. This is great for testing that our business logic is correctly implemented with only valid HTTP requests.
• We’ll also use hspec-wai to make arbitrary HTTP requests, in order to test how our application may respond to invalid or otherwise unexpected requests.
• Finally, we can also use servant-quickcheck for whole-API tests, in order to assert that our entire application conforms to best practices.

Useful Libraries

The following libraries will often come in handy when we decide to test our Servant applications:

• hspec
• hspec-wai
• QuickCheck
• servant-quickcheck

Imports and Our Testing Module

This recipe starts with the following ingredients:

{-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds,
  DeriveGeneric, TypeOperators #-}
import           Prelude ()
import           Prelude.Compat

import qualified Control.Concurrent               as C
import           Control.Concurrent.MVar
import           Control.Exception                (bracket)
import           Control.Lens              hiding (Context)
import           Data.Aeson
import           Data.Aeson.Lens
import qualified Data.HashMap.Strict              as HM
import           Data.Text                        (Text, unpack)
import           GHC.Generics
import           Network.HTTP.Client       hiding (Proxy)
import           Network.HTTP.Types
import           Network.Wai
import qualified Network.Wai.Handler.Warp         as Warp

import           Servant
import           Servant.Client
import           Servant.Server
import           Servant.QuickCheck
import           Servant.QuickCheck.Internal (serverDoesntSatisfy)

import           Test.Hspec
import           Test.Hspec.Wai
import           Test.Hspec.Wai.Matcher

We’re going to produce different Specs that represent different aspects of our application, and we’ll ask hspec to run all of our different Specs. This is a common organizational method for testing modules:

spec :: Spec
spec = do
  businessLogicSpec
  thirdPartyResourcesSpec
  servantQuickcheckSpec

Often, codebases will use hspec’s autodiscover pragma to find all testing modules and Specs inside, but we’re going to explicitly make a main function to run our tests because we have only one spec defined above:

main :: IO ()
main = hspec spec

Testing Your Business Logic

Let’s say we have an API that looks something like this:

data User = User {
  name :: Text
  , user_id :: Integer
  } deriving (Eq, Show, Generic)

instance FromJSON User
instance ToJSON User

type UserApi =
  -- One endpoint: create a user
  "user" :> Capture "userId" Integer :> Post '[JSON] User

A real server would likely use a database to store, retrieve, and validate users, but we’re going to do something really simple merely to have something to test. With that said, here’s a sample handler, server, and Application for the endpoint described above:

userApp :: Application
userApp = serve (Proxy :: Proxy UserApi) userServer

userServer :: Server UserApi
userServer = createUser

createUser :: Integer -> Handler User
createUser userId = do
  if userId > 5000
    then pure $ User { name = "some user", user_id = userId }
    else throwError $ err400 { errBody = "userId is too small" }

Strategy 1: Spin Up a Server, Create a Client, Make Some Requests

One of the benefits of Servant’s type-level DSL for describing APIs is that once you have provided a type-level description of your API, you can create clients, documentation, or other tools for it somewhat magically.

In this case, we’d like to test our server, so we can use servant-client to create a client, after which we’ll run our server, and then make requests of it and see how it responds.

Let’s write some tests:

withUserApp :: IO () -> IO ()
withUserApp action =
  -- we can spin up a server in another thread and kill that thread when done
  -- in an exception-safe way
  bracket (liftIO $ C.forkIO $ Warp.run 8888 userApp)
    C.killThread
    (const action)


businessLogicSpec :: Spec
businessLogicSpec =
  -- `around` will start our Server before the tests and turn it off after
  around_ withUserApp $ do
    -- create a test client function
    let createUser = client (Proxy :: Proxy UserApi)
    -- create a servant-client ClientEnv
    baseUrl <- runIO $ parseBaseUrl "http://localhost:8888"
    manager <- runIO $ newManager defaultManagerSettings
    let clientEnv = mkClientEnv manager baseUrl

    -- testing scenarios start here
    describe "POST /user" $ do
      it "should create a user with a high enough ID" $ do
        result <- runClientM (createUser 50001) clientEnv
        result `shouldBe` (Right $ User { name = "some user", user_id = 50001})
      it "will it fail with a too-small ID?" $ do
        result <- runClientM (createUser 4999) clientEnv
        result `shouldBe` (Right $ User { name = "some user", user_id = 50001})

Running These Tests

Let’s run our tests and see what happens:

$ cabal new-test all
POST /user
  should create a user with a high enough ID
  should fail with a too-small ID FAILED [1]

  Failures:

  Testing.lhs:129:7:
  1) POST /user should fail with a too-small ID
       expected: Right (User {name = "some user", user_id = 50001})
        but got: Left (FailureResponse (Response {responseStatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, responseHeaders = fromList [("Transfer-Encoding","chunked"),("Date","Fri, 12 Oct 2018 04:36:22 GMT"),("Server","Warp/3.2.25")], responseHttpVersion = HTTP/1.1, responseBody = "userId is too small"}))

  To rerun use: --match "/POST /user/should fail with a too-small ID/"

Hmm. One passed and one failed! It looks like I was expecting a success response in the second test, but I actually got a failure. We should fix that, but first I’d like to introduce hspec-wai, which will give us different mechanisms for making requests of our application and validating the responses we get. We’re also going to spin up a fake Elasticsearch server, so that our server can think it’s talking to a real database.

Mocking 3rd Party Resources

Often our web applications will need to make their own web requests to other 3rd-party applications. These requests provide a lot of opportunity for failure and so we’d like to test that the right messages and failure values (in addition to success values) are returned from our application.

Define the 3rd-Party Resource

With Servant’s type-level API definitions, assuming you’ve already defined the API you want to mock, it’s relatively trivial to create a simple server for the purposes of running tests. For instance, consider an API server that needs to get data out of Elasticsearch. Let’s first define the Elasticsearch server and client using Servant API descriptions:

type SearchAPI =
  -- We're using Aeson's Generic JSON `Value` to make things easier on
  -- ourselves. We're also representing only one Elasticsearch endpoint:
  -- get item by id
  "myIndex" :> "myDocType" :> Capture "docId" Integer :> Get '[JSON] Value

-- Here's our Servant Client function
getDocument = client (Proxy :: Proxy SearchAPI)

-- We can use these helpers when we want to make requests
-- using our client function
clientEnv :: Text -> Text -> IO ClientEnv
clientEnv esHost esPort = do
  baseUrl <- parseBaseUrl $ unpack $ esHost <> ":" <> esPort
  manager <- newManager defaultManagerSettings
  pure $ mkClientEnv manager baseUrl

runSearchClient :: Text -> Text -> ClientM a -> IO (Either ServantError a)
runSearchClient esHost esPort = (clientEnv esHost esPort >>=) . runClientM

Servant Server Example Using this 3rd-Party Resource

So we’ve got an Elasticsearch server and a client to talk to it. Let’s now build a simple app server that uses this client to retrieve documents. This is somewhat contrived, but hopefully it illustrates the typical three-tier application architecture.

One note: we’re also going to take advantage of lens-aeson here, which may look a bit foreign. The gist of it is that we’re going to traverse a JSON Value from Elasticsearch and try to extract some kind of document to return.

Imagine, then, that this is our real server implementation:

type DocApi =
  "docs" :> Capture "docId" Integer :> Get '[JSON] Value

docsApp :: Text -> Text -> Application
docsApp esHost esPort = serve (Proxy :: Proxy DocApi) $ docServer esHost esPort

docServer :: Text -> Text -> Server DocApi
docServer esHost esPort = getDocById esHost esPort

-- Our Handler tries to get a doc from Elasticsearch and then tries to parse
-- it. Unfortunately, there's a lot of opportunity for failure in these
-- actions
getDocById :: Text -> Text -> Integer -> Handler Value
getDocById esHost esPort docId = do
  -- Our Servant Client function returns Either ServantError Value here:
  docRes <- liftIO $ runSearchClient esHost esPort (getDocument docId)
  case docRes of
    Left err -> throwError $ err404 { errBody = "Failed looking up content" }
    Right value -> do
      -- we'll either fail to parse our document or we'll return it
      case value ^? _Object . ix "_source" of
        Nothing -> throwError $ err400 { errBody = "Failed parsing content" }
        Just obj -> pure obj

Testing Our Backend

So the above represents our application and is close to a server we may actually deploy. How then shall we test this application?

Ideally, we’d like it to make requests of a real Elasticsearch server, but we certainly don’t want our tests to trigger requests to a live, production database. In addition, we don’t want to depend on our real Elasticsearch server having specific, consistent results for us to test against, because that would make our tests flaky (and flaky tests are sometimes described as worse than not having tests at all).

One solution to this is to create a trivial Elasticsearch server as part of our testing code. We can do this relatively easily because we already have an API definition for it above. With a real server, we can then let our own application make requests of it and we’ll simulate different scenarios in order to make sure our application responds the way we expect it to.

Let’s start with some helpers which will allow us to run a testing version of our Elasticsearch server in another thread:

-- | We'll run the Elasticsearch server so we can test behaviors
withElasticsearch :: IO () -> IO ()
withElasticsearch action =
  bracket (liftIO $ C.forkIO $ Warp.run 9999 esTestApp)
    C.killThread
    (const action)

esTestApp :: Application
esTestApp = serve (Proxy :: Proxy SearchAPI) esTestServer

esTestServer :: Server SearchAPI
esTestServer = getESDocument

-- This is the *mock* handler we're going to use. We create it
-- here specifically to trigger different behavior in our tests.
getESDocument :: Integer -> Handler Value
getESDocument docId
  -- arbitrary things we can use in our tests to simulate failure:
  -- we want to trigger different code paths.
  | docId > 1000 = throwError err500
  | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")]
  | otherwise = pure $ Object $ HM.fromList [("_source", Object $ HM.fromList [("a", String "b")])]

Now, we should be ready to write some tests.

In this case, we’re going to use hspec-wai, which will give us a simple way to run our application, make requests, and make assertions against the responses we receive.

Hopefully, this will simplify our testing code:

thirdPartyResourcesSpec :: Spec
thirdPartyResourcesSpec = around_ withElasticsearch $ do
  -- we call `with` from `hspec-wai` and pass *real* `Application`
  with (pure $ docsApp "localhost" "9999") $ do
    describe "GET /docs" $ do
      it "should be able to get a document" $
        -- `get` is a function from hspec-wai`.
        get "/docs/1" `shouldRespondWith` 200
      it "should be able to handle connection failures" $
        get "/docs/1001" `shouldRespondWith` 404
      it "should be able to handle parsing failures" $
        get "/docs/501" `shouldRespondWith` 400
      it "should be able to handle odd HTTP requests" $
        -- we can also make all kinds of arbitrary custom requests to see how
        -- our server responds using the `request` function:
        -- request :: Method -> ByteString -> [Header]
        --   -> LB.ByteString -> WaiSession SResponse
        request methodPost "/docs/501" [] "{" `shouldRespondWith` 405
      it "we can also do more with the Response using hspec-wai's matchers" $
        -- see also `MatchHeader` and JSON-matching tools as well...
        get "/docs/1" `shouldRespondWith` 200 { matchBody = MatchBody bodyMatcher }

bodyMatcher :: [Network.HTTP.Types.Header] -> Body -> Maybe String
bodyMatcher _ body = case (decode body :: Maybe Value) of
  -- success in this case means we return `Nothing`
  Just val | val == (Object $ HM.fromList [("a", String "b")]) -> Nothing
  _ -> Just "This is how we represent failure: this message will be printed"

Out of the box, hspec-wai provides a lot of useful tools for us to run tests against our application. What happens when we run these tests?

$ cabal new-test all
...

GET /docs
  should be able to get a document
  should be able to handle connection failures
  should be able to handle parsing failures
  should be able to handle odd HTTP requests
  we can also do more with the Response using hspec-wai's matchers

Fortunately, they all passed! Let’s move to another strategy: whole-API testing.

Servant Quickcheck

servant-quickcheck is a project that allows users to write tests for whole Servant APIs using quickcheck-style property-checking mechanisms.

servant-quickcheck is great for asserting API-wide rules, such as “no endpoint throws a 500” or “all 301 status codes also come with a Location header”. The project even comes with a number of predicates that reference the RFCs they originate from.

In other words, it’s one way to assert that your APIs conform to specs and best practices.

Quickcheckable API

Let’s make an API and a server to demonstrate how to use servant-quickcheck:

type API = ReqBody '[JSON] String :> Post '[JSON] String
      :<|> Get '[JSON] Int
      :<|> BasicAuth "some-realm" () :> Get '[JSON] ()

api :: Proxy API
api = Proxy

server :: IO (Server API)
server = do
    mvar <- newMVar ""
    return $ (\x -> liftIO $ swapMVar mvar x)
        :<|> (liftIO $ readMVar mvar >>= return . length)
        :<|> (const $ return ())

Using servant-quickcheck

Let’s build some tests for our API using servant-quickcheck.

Similar to the above examples, we’re going to create Specs, but in this case, we’ll rely on a number of predicates available from servant-quickcheck to see if our API server conforms to best practices:

-- Let's set some QuickCheck values
args :: Args
args = defaultArgs { maxSuccess = 500 }

-- Here's a Servant Context object we'll use
ctx :: Context '[BasicAuthCheck ()]
ctx = BasicAuthCheck (const . return $ NoSuchUser) :. EmptyContext


servantQuickcheckSpec :: Spec
servantQuickcheckSpec = describe "" $ do
  it "API demonstrates best practices" $
    -- `withServerServer` and `withServantServerAndContext` come from `servant-quickcheck`
    withServantServerAndContext api ctx server $ \burl ->
      -- `serverSatisfies` and the predicates also come from `servant-quickcheck`
      serverSatisfies api burl args (unauthorizedContainsWWWAuthenticate
                                 <%> not500
                                 <%> onlyJsonObjects  -- this one isn't true!
                                 <%> mempty)

  it "API doesn't have these things implemented yet" $
    withServantServerAndContext api ctx server $ \burl -> do
      serverDoesntSatisfy api burl args (getsHaveCacheControlHeader
                                    <%> notAllowedContainsAllowHeader
                                    <%> mempty)

Let’s see what happens when we run these tests:

  API demonstrates best practices FAILED [2]
+++ OK, passed 500 tests.
  API doesn't have these things implemented yet

  src/Servant/QuickCheck/Internal/QuickCheck.hs:143:11:
  2) Main[339:25] API demonstrates best practices
       Failed:
       Just Predicate failed
            Predicate: onlyJsonObjects

            Response:
                 Status code: 200
                 Headers:  "Transfer-Encoding": "chunked"
                           "Date": "Fri, 12 Oct 2018 04:36:22 GMT"
                           "Server": "Warp/3.2.25"
                           "Content-Type": "application/json;charset=utf-8"
                 Body: ""

  To rerun use: --match "/Main[339:25]/API demonstrates best practices/"

Randomized with seed 1046277487

Finished in 0.4306 seconds

Hmm. It looks like we thought our API only returned JSON objects, which is a best practice, but in fact, we did have an endpoint that returned an empty body, which you can see in the printed response above: Body: "". We should consider revising our API to only return top-level JSON Objects in the future!

Other Cool Things

servant-quickcheck also has a cool mechanism where you can compare two API servers to demonstrate that they respond identically to requests. This may be useful if you are planning to rewrite one API in another language or with another web framework. You have to specify whether you’re looking for jsonEquality vs regular ByteString equality, though.

Conclusion

There are lots of techniques for testing and we only covered a few here.

Useful libraries such as hspec-wai have ways of running Wai Applications and sending requests to them, while Servant’s type-level DSL for defining APIs allows us to more easily mock out servers and to derive clients, which will only craft valid requests.

Lastly, if you want a broad overview of where your application fits in with regard to best practices, consider using servant-quickcheck.

This program is available as a cabal project here.