Every application with a reasonable complexity obviously has to be tested before releasing to the customer – Yesod applications are no different in this case. Of course, type safety gives us a strong guarantees – guarantees which are far beyond reach of Django, Rails or Node developers. Still, that’s not enough to purge all the bugs (or at least most of them). That’s why we have to write tests.

There are hundreds of levels of testing that are specified by dozens of documents, standards and specifications – we won’t be discussing most of them here, as there are only a few people that care about technicalities, and most of us care only about “what to do to make the app work and guarantee that it will keep working”. Formally, these are tests of the functionality (there are also other groups, like performance or UX), and are the most common group of tests. For our application, I’d divide it into three or four kinds of tests that require different setups and different testing methods (we won’t be implementing all of them here – that’s just a conceptual discussion):

• Unit tests
• Server (backend) tests
• Interface (frontend) tests
• System tests

They become needed with the increasing complexity, so you won’t necessarily need all of them in your first project, but with time, they become increasingly useful (so do performance tests and UX ones, but they are a different story). Unit tests are one of the most popular tools, and pretty much everybody claim that they use it – it’s the basic tool for assessing application sanity in unityped languages (like Python or JavaScript). Of course, Haskell also has several frameworks for writing them, of which ones of the most popular are Tasty and Hspec. Tasty is a framework, but the actual tests and assertions are provided by other libraries, like QuickCheck, SmallCheck or HUnit. The third one is quite a typical xUnit library for Haskell, but the first two are a bit different – instead of testing specific input/output combinations, they are testing properties of your code. They do this by injecting pseudo-random values and analyzing features of the result (instead of the result value). For example, we if the define that prepend is a function which – after applying it to a List – increases its length by 1 and is the first element of the new structure, we could define it as:

prepend :: a -> [a] -> [a]
prepend elem list = elem : list

x = testGroup "prepend features" 
  [
    QC.testProperty "after prepend list is bigger" $ 
      \list elem -> (length $ prepend elem (list :: [Int])) == length list + 1,
    QC.testProperty "after prepend becomes first element" $ 
      \list elem -> (head $ prepend elem (list :: [Int])) == elem
  ]

of course, there are lots of different properties that can be assessed on most structures. However, there is a catch in these tests – you cannot ignore intermittent failures. Since they use random input (pseudo-random, but the seed is usually really random), every test failure may be an actual bug. Plus, it’s possible that some bugs will go unnoticed for a few runs. That’s a bit different philosophy from the usual approach, where data is always the same and bug is either detected or not on each run (assuming no environment “intermittent” failures). It’s not better or worse, it’s simply different. Arguably better for lower-level tests, such as unit tests, and that’s why it’s used there. They are not suitable for edge-case testing, but are very good at exploring the domain.

There is nothing special in unit tests for applications that are using Yesod – they are just plain Haskell UTs, ignoring the whole Yesod thing.

Next group of tests are server tests – ones that use backend. They should be ran against a fairly-complete app, with backend database set up (preferably on the same host), but without connection to external services or full deployment (proxies, load balancers etc.). It should mostly test reactions on API calls – in most cases you will not want to test the HTML page but rather some JSON or XML responses (testing HTML is much harder). Yesod provides such tests (called integeration tests there, but this name is often used in many contexts) in a helper library, yesod-test. Examples of such tests together with Hspec framework are included in the default scaffolding in test/Handler directory. As you can see, these are quite like HTTP requests, except that they don’t really go through any port, and the communication happens inside a single binary. I really recommend writing this kind of tests – they give a (nearly) end-to-end view on the processing, and are still quite efficient (single binary with occassional DB access). One more thing about the database: beware. While you’ll probably want to use it in these tests, you have to be sure which applications communicate directly with the database. I don’t mean “the same instance of database as your test database”, as I’m sure that you’ll have a special database for tests, preferably set up from scratch on each test run. What I mean is that it’s quite common that more than one application communicate with the same database – for example, for market analysis. That’s an important point, because if you use a shared database, database is also one of your interfaces and you should treat it with the same care as you treat your other interfaces.

There are two types of interface (UI) tests – first one is testing for view – as in Jest, a library for testing React views – and the second type is testing for functionality – as in Selenium. I’m focusing on tools for web projects here, because Yesod is a web framework, but same types are important in pretty much any area, including mobile and desktop applications. These types of tests are probably responsible for the most hatred from the development teams to testing of all tools. That is because both these types are brittle, and small, seemingly not connected changes can break them. These changes include moving the button a few pixels left or right, changing the tone of the background, removing one or two nested divs. Of course, properly written tests will yield more reasonable results (if you’re using Selenium, check out the Page Object pattern), but still, they’re much less change-tolerant than the other types. Additionally, they are not yet fully mature – despite the fact that Selenium is there for a few years already, the driver for Firefox is still not ready (I know that it was broken by Firefox 48 and that geckodriver is not responsibility of Selenium team – still, lack of driver for one of the top browsers signals immaturity of tooling as a whole), so you may encouter quite a few glitches. Nevertheless, I really recommend to implement tests for some basic functionalities of the app. In the beginning it might seem that manual clicking through the app is faster, but amount of manual clicking never ceases to increase, and our patience does – and the quality of testing suffers. Of course, I’m not asking you to implement every single detail in UI test – but at least check basic features – that checkboxes work, that submit buttons cause submits and that data is available in the UI after its submission. Oh, and there are Selenium bindings for Haskell. For web tests your setup should be similar to the one created for server tests, while for view tests it may be simpler – for example as simple as four UT setup.

The last type of tests is arguably the most complex one and most IT projects don’t have them. The are run in the actual production environment (with the exception for no serving of real clients yet) and/or its clone. Their purpose is to guarantee that deployment was done properly, communication with external services is fine and generally the application is ready to start serving real clients. This is no longer time for checking the functionality – this should have been done earlier – only a few basic scenarios are executed, mostly to guarantee that interfaces between system components (services, external world and things like OS) are working fine. Static typing help here as well – perhaps Yesod is not the best example, but Servant is kinda famous for generating type-safe WebAPIs. Still, we have to check that services were built in compatible versions, ports are not blocked etc. Altogether – this step is more of a job for a DevOps guy and simplifies rather operations than development, but hey – in your startup you’ll have to write everything on your own and deal with the administration as well, so you’ll better get to know it!

By the way, that’s precisely what we’re going to deal with in the next post – setting up an automated deployment routine to provide us a fully automated continous deployment pipeline. The whole task probably won’t fit in a single post, but hey – let’s see.

Stay tuned!