Where are the bugs?
If you're are a developer, you've already written bugs. Not just one or two bugs, you have written many of them. A bug is just an error, and hey, we're only human! But if you don't find a better approach, this will still stay the case, and things won't improve. After some time, say a few months or a few years, you'll start to realise your code is hard to maintain and improve.
To avoid this you need to add tests to make sure that your code produces the expected result, this process spans from the first day and until the last day of a project. The most specific kind of test that we can perform is a unit test. These offer the possibility to test a very narrow part of your code. This is the first big step in finding a bug.
Make sure you're writing these tests beforehand (not rushing through the process) and taking the time to think about all acceptance criteria and complex cases of your business logic.
At PALO IT this is our standard process, otherwise known as test-driven development (TDD)—we don’t write a single line of business code if there’s not already a test to cover it.
Revisiting our framework documentation
If you write unit tests and respect the TDD process you should produce good, quality code.
But does this suffice? Let’s imagine that you want to create a REST API that returns a JSON list of 50 bookings. To make it happen, you'll need to support the HTTP GET method and return a JSON. If you're a Java developer, you'll probably choose Spring because it offers you the web support that you need, and Jackson to serialise your list of objects into JSON. So, you write unit tests for the expected list, create a service to build that list, expose this service in a controller, and run your tests: all green 🚦
But a problem can appear when you want to try this API for the first time. For example, if you are familiar with the Java encapsulation principle, you have probably (and by reflex) set all your fields as private. If that's the case, a call to your service will fail. In fact, by default, Jackson will not serialise all your private fields, so your API will not be able to return anything. To solve the issue you need to configure Jackson or change your class.
In modern Java programming, several frameworks are often piled up: Spring, Aspect J, JPA, Jackson, Lombok, and so on. These frameworks can sometimes be incompatible with each other.
This example echoes something very important—you should include all the frameworks and libraries that you use in your test strategy. In other words, your tests should go through all layers of the application to validate that the product works, not only the business code. You should include all frameworks and libraries you use in your test strategy.
With this in mind, you'll be able to test the parsing/marshalling, the security, the network, and the integration of all our smaller parts of code where the framework acts like a glue between them. This is the second level of test, and you're probably familiar with it too, the integration test.
Those tests are not well maintained by some that consider them too slow to run and maintain compared to unit tests, but they're the only way to test what you are exposing to other teams or partners.
It’s also important to see that it’s not an easy job to know if we have a good integration test, because it’s hard to measure the coverage of your code from a client perspective. But, we always aim for top quality, so we need to reach this level of test and still follow the TDD process.
The “it runs on my laptop” effect
Now that you're writing unit and integration tests, and still following the TDD process, you should produce a quality project. But, does it meet our client's expectations? If we're creating an API, we need to deploy it to consider the job done, and this API is probably not the only one that you'll build and should run somewhere. It can be on a container that you deploy on a Kubernetes, exposed through a Kong gateway, secured by Keycloak, with data from an Oracle database. You might also be integrated into an existing CRM and a lot of other external dependencies relative to the business of your client. You need to do that within different contexts, because you probably have more than just one environment. These are complex but real-world obstacles.
A serverless architecture, like the one that we are using today when we deploy on AWS Serverless, also has a lot of complexity because you still have a gateway and external dependencies. We want to be able to control risk by detecting bugs on the developer's computer, not after deployment or in continuous integration.
To address this goal, we are using a specific framework, designed for this specific purpose and maintained by us. You can find this open source project on here. This framework allows us to emulate an AWS API gateway, the lambda runtime environment, and provide enough integration to measure the code and branch coverage of integration tests.
In effect, we merge the concept of unit and integration tests — because we write micro services, the code is so simple that the granularity of unit tests and integration tests is identical.
Time for a real-world example!
Imagine a method that allows us to find a user by using his or her phone number. The code is available here.
Here we check the JWT token and, regarding the role, decide to throw an error to the expected user. If the user is not present, we return a 404 with a message to explain that the user is not found.
Here are two examples of tests that we have for this endpoint.
As you can see, we are testing code from a real API consumer, we call the endpoint with RestAssured and check the response that we receive. A success in the first case and an error with a description of what happened in the second case, and we're also able to measure the coverage of that. Here's a part of the Jacoco code coverage report:
The creation of the user is not part of this test. In fact, we are able to run DynamoDB locally and inject the data required by the tests. If we need to run other dependencies, we can mock them if it’s sufficient.
Closing thoughts
This approach is fast, complete, and offers a distinct advantage—our integration tests cover what was previously done by unit test, so we don't need to maintain them anymore, and can stay focused on the best level of test. This strategy helps us to find 95% of issues locally.
Concretely:
- We write our API tests using a unit testing framework (Junit)
- In a test, we use an HTTP client library like RestAssured to actually invoke our API gateway and lambda based API, so we traverse all applicative layers, including framework and database
- We usually run a real database locally on the developer’s workstation, and simulators for complex external systems like 3rd party APIs
- Our framework allows us to run exactly the same tests locally (on a workstation) and remotely (after deployment on AWS). The remote testing helps capturing the remaining 5% of problems linked to configuration, scalability, security policies, throttling and other niceties, and is an integral part of our DevOps strategy
- Finally, we validate our TDD practice by continuously measuring the code and branch coverage of our business code. If we apply TDD properly, it should be 100% all the time. In our CI setting, we even fail the build if we don’t hit 100% (of course, we also parse the code base to check that lazy developers don’t use magical annotations like @lombok.Generated to disable code coverage 😁)
The principles we exposed here work for a serverless architecture, but they can be applied as well for a more traditional, Spring Boot based approach, and of course also for other technologies.
