Building a Microservices-based REST API with RestExpress, Java EE, and MongoDB: Part 3

Develop a well-architected and well-documented REST API, built on a tightly integrated collection of Java EE-based microservices.

Note: All code available on GitHub. For the version of the code that matches the details in this blog post, check out the master branch, v1.0.0 tag (after running git clone …, run a git checkout tags/v1.0.0 command).

Previous Posts

In Part One of this series, we introduced the microservices-based Virtual-Vehicles REST API example. The vehicle-themed Virtual-Vehicles microservices offers a comprehensive set of functionality, through a REST API, to application developers. In Part Two, we installed a copy of the Virtual-Vehicles project from GitHub. In Part Two, we also gained a basic understanding of how RestExpress works. Finally, we discovered how to get the Virtual-Vehicles microservices up and running.

Part Three

In part three of this series, we will take the Virtual-Vehicles for a test drive (get it? maybe it was funnier the first time…). There are several tools we can use to test the Virtual-Vehicles API. One of my favorite tools is Postman.  We will explore how to use Postman, along with the Virtual-Vehicles API documentation, to test the Virtual-Vehicles microservice’s endpoints, which compose the Virtual-Vehicles API.

Testing the API

There are three categories of tools available to test RESTful APIs, which are GUI-based applications, command line tools, and testing frameworks. Postman, Advanced REST Client, REST Console, and SmartBear’s SoapUI and SoapUI NG Pro, are examples of GUI-based applications, designed specifically to test RESTful APIs. cURL and GNU Wget are two examples of command line tools, which among other capabilities, can test APIs. Lastly, JUnit is an example of a testing framework that can be used to test a RESTful API. Surprisingly, JUnit is not only designed to manage unit tests. Each category of testing tools has their pros and cons, depending on your testing needs. We will explore all of these categories in this post as we test the Virtual-Vehicles REST API.

JUnit

JUnit is probably the best known of all Java unit testing frameworks. JUnit’s website describes JUnit as ‘a simple, open source framework to write and run repeatable tests. It is an instance of the xUnit architecture for unit testing frameworks.’ Most Java developers turn to JUnit for unit testing. However, JUnit is capable of other forms of testing, including integration testing. In his post, ‘Unit Testing with JUnit – Tutorial’, Lars Vogel states ‘an integration test has the target to test the behavior of a component or the integration between a set of components. The term functional test is sometimes used as a synonym for integration test. This kind of tests allow you to translate your user stories into a test suite, i.e., the test would resemble an expected user interaction with the application.’

Testing the Virtual-Vehicles RESTful API’s operations with JUnit would be considered integration (functional) testing. At a minimum, to complete requests, we call one microservice, which in turn authenticates the JWT by calling another microservice. If authenticated, the first microservice makes a request to its MongoDB database. As Vogel stated, whereas a unit test targets a small unit of code, such as a method, the request/response operation is integration between a set of components. testing an API call requires several dependencies.

The simplest example of testing the Virtual-Vehicles API with JUnit, would be to test an HTTP GET request to return a single instance of a vehicle. The code below demonstrates how this might be done. Notice the request depends on helper methods (not included, for brevity). To request the vehicle, assuming we already have a registered client, we need a valid JWT. We also need a valid vehicle ObjectId. To obtain these two pieces of data, we call helper methods, which in turn makes the necessary request to retrieve a JWT and vehicle ObjectId.

	/**
	* Test of HTTP GET to read a single vehicle.
	*/
	@Test
	public void testVehicleRead() {
	String responseBody = "";
	String output;
	Boolean result = true;
	Boolean expResult = true;
	try {
	URL url = new URL(getBaseUrlAndPort() + "/vehicles/" + getVehicleObjectId());
	HttpURLConnection conn = (HttpURLConnection) url.openConnection();
	conn.setRequestMethod("GET");
	conn.setRequestProperty("Authorization", "Bearer " + getJwt());
	conn.setRequestProperty("Accept", "application/json");
	if (conn.getResponseCode() != 200) {
	// if not 200 response code then fail test
	result = false;
	}
	BufferedReader br = new BufferedReader(new InputStreamReader(
	(conn.getInputStream())));
	while ((output = br.readLine()) != null) {
	responseBody = output;
	}
	if (responseBody.length() < 1) {
	// if response body is empty then fail test
	result = false;
	}
	conn.disconnect();
	} catch (IOException e) {
	// if MalformedURLException, ConnectException, etc. then fail test
	result = false;
	}
	assertEquals(expResult, result);
	}

view raw testVehicleRead.java hosted with ❤ by GitHub

Below are the results of the above test, run in NetBeans IDE, using the built-in support for JUnit.

JUnit can also be run from the command line using the Maven goal, surefire:test:

mvn -q -Dtest=com.example.vehicle.objectid.VehicleControllerIT surefire:test

view raw vehicle_surefire.bash hosted with ❤ by GitHub

cURL

One of the best-known command line tools for calling for all types of operations centered around calling a URL is cURL. According to their website, ‘curl is a command line tool and library for transferring data with URL syntax, supporting…HTTP, HTTPS…curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5, NTLM, Negotiate, and Kerberos), file transfer resume, proxy tunneling and more.’ I prefer the website’s  briefer description, cURL ‘groks those URLs’.

Using cURL, we could make an HTTP PUT request to the Vehicle microservice’s /vehicles/{oid}.{format} endpoint. With cURL, we have the ability to add the JWT-based Authorization header and the raw request body, containing the modified vehicle object. Below is an example of that cURL command, which can be run from a terminal prompt.

	curl --url 'http://virtual-vehicles.com:8581/vehicles/557310cfec7291b25cd3a1c2' \
	-X PUT \
	-H 'Pragma: no-cache' \
	-H 'Cache-Control: no-cache' \
	-H 'Accept: application/json; charset=UTF-8' \
	-H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJ2aXJ0dWFsLXZlaGljbGVzLmNvbSIsImFwaUtleSI6IlJncjg0YzF6VkdtMFd1N25kWjd5UGNURSIsImV4cCI6MTQzMzY2ODEwNywiYWl0IjoxNDMzNjMyMTA3fQ.xglaKWufcj4TZtMXW3DLa9uy5JB_FJHxxtk_iF1WT6U' \
	--data-binary $'{ "year": 2015, "make": "Chevrolet", "model": "Corvette Stingray", "color": "White", "type": "Coupe", "mileage": 902, "createdAt": "2015-05-09T22:36:04.808Z" }' \
	--compressed

view raw curl_command.bash hosted with ❤ by GitHub

The response body contains the expected modified vehicle object in JSON-format, along with a 201 Created response status.

The cURL commands may be incorporated into many types of automated testing processes. These might be as simple as a bash script. The script could a series of automated tests, including the following: register an API client, use the API key to create a JWT, use the JWT to create a new vehicle, use the new vehicle’s ObjectId to modify that same vehicle, delete that vehicle, confirm the vehicle is removed using the count operation and returns a test results report to the user.

cURL Commands from Chrome
Quick tip, instead of hand-coding complex cURL commands, containing form data, URL parameters, and Headers, use Chrome. First, open the Chrome Developer Tools (f12). Next, using the Postman – REST Client for Chrome, available in the Chrome App Store, execute your HTTP request. Finally, in the ‘Network’ tab of the Developers tools, find and right-click on the request and select ‘Copy as cURL’. You have a complete cURL command equivalent of your Postman request, which you can paste directly into the command line or insert into a script. Below is an example of using the Postman – REST Client for Chrome to generate a cURL command.

	curl --url 'http://virtual-vehicles.com:8581/vehicles/554e8bd4c830093007d8b949' \
	-X PUT \
	-H 'Pragma: no-cache' \
	-H 'Origin: chrome-extension://fdmmgilgnpjigdojojpjoooidkmcomcm' \
	-H 'Accept-Encoding: gzip, deflate, sdch' \
	-H 'Accept-Language: en-US,en;q=0.8' \
	-H 'CSP: active' \
	-H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJ2aXJ0dWFsLXZlaGljbGVzLmNvbSIsImFwaUtleSI6IlBUMklPSWRaRzZoU0VEZGR1c2h6U04xRyIsImV4cCI6MTQzMzU2MDg5NiwiYWl0IjoxNDMzNTI0ODk2fQ.4q6EMuxE0vS43zILjE6e1tYrb5ulCe69-1QTFLYGbFU' \
	-H 'Content-Type: text/plain;charset=UTF-8' \
	-H 'Accept: */*' \
	-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36' \
	-H 'Cache-Control: no-cache' \
	-H 'Connection: keep-alive' \
	-H 'X-FirePHP-Version: 0.0.6' \
	--data-binary $'{ "year": 2015, "make": "Chevrolet", "model": "Corvette Stingray", "color": "White", "type": "Coupe", "mileage": 902, "createdAt": "2015-05-09T22:36:04.808Z" }' \
	--compressed

view raw curl_command.bash hosted with ❤ by GitHub

The generated command is a bit verbose. Compare this command to the cURL command, earlier.

Wget

Similar to cURL, GNU Wget provides the ability to call the Virtual-Vehicles API’s endpoints. According to their website, ‘GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive command line tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.’ Again, like cURL, we can run Wget commands from the command line or incorporate them into scripted testing processes. The Wget website contains excellent documentation.

Using Wget, we could make the same HTTP PUT request to the Vehicle microservice /vehicles/{oid}.{format} endpoint. Like cURL, we have the ability to add the JWT-based Authorization header and the raw request body, containing the modified vehicle object.

	wget -O - 'http://virtual-vehicles.com:8581/vehicles/557310cfec7291b25cd3a1c2' \
	--method=PUT \
	--header='Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJ2aXJ0dWFsLXZlaGljbGVzLmNvbSIsImFwaUtleSI6IlJncjg0YzF6VkdtMFd1N25kWjd5UGNURSIsImV4cCI6MTQzMzY2ODEwNywiYWl0IjoxNDMzNjMyMTA3fQ.xglaKWufcj4TZtMXW3DLa9uy5JB_FJHxxtk_iF1WT6U' \
	--header='Content-Type: text/plain;charset=UTF-8' \
	--header='Accept: application/json' \
	--body-data=$'{ "year": 2015, "make": "Chevrolet", "model": "Corvette Stingray", "color": "White", "type": "Coupe", "mileage": 902, "createdAt": "2015-05-09T22:36:04.808Z" }'

view raw wget_command.bash hosted with ❤ by GitHub

The response body contains the expected modified vehicle object in JSON-format, along with a 201 Created response status.

cURL Bash Testing

We can combine cURL and Wget with several of the tools bash provides, to develop fairly complex integration tests. The bash-based script below just scratches the surface as a complete set of integration tests. However, the tests demonstrate an efficient multi-stage test approach to handling the complex nature of RESTful service request requirements. The tests build upon each other.

After setting up some variables and doing a quick health check on one service, the tests register a new API client by calling the Authentication service. Next, they use the new client’s API key to obtain a JWT. The tests then use the JWT to authenticate themselves and create a new vehicle. Finally, they use the new vehicle’s id and the JWT to verify the existence for the new vehicle.

Although some may consider using bash to test somewhat primitive, the following script demonstrates the effectiveness of bash’s  curl, grep, sed, awk, along with regular expressions, to test our RESTful services. Note how we grep certain values from the response, such as the new client’s API key, and then use that value as a parameter for the following test request, such as to obtain a JWT.

	#!/bin/sh
	########################################################################
	#
	# title: Virtual-Vehicles Project Integration Tests
	# author: Gary A. Stafford (https://programmaticponderings.com)
	# url: https://github.com/garystafford/virtual-vehicles-docker
	# description: Performs integration tests on the Virtual-Vehicles
	# microservices
	# to run: sh tests_color.sh -v
	#
	########################################################################
	echo --- Integration Tests ---
	### VARIABLES ###
	hostname="localhost"
	application="Test API Client $(date +%s)" # randomized
	secret="$(date +%s | sha256sum | base64 | head -c 15)" # randomized
	echo hostname: ${hostname}
	echo application: ${application}
	echo secret: ${secret}
	### TESTS ###
	echo "TEST: GET request should return 'true' in the response body"
	url="http://${hostname}:8581/vehicles/utils/ping.json"
	echo ${url}
	curl -X GET -H 'Accept: application/json; charset=UTF-8' \
	--url "${url}" \
	| grep true > /dev/null
	[ "$?" -ne 0 ] && echo "RESULT: fail" && exit 1
	echo "RESULT: pass"
	echo "TEST: POST request should return a new client in the response body with an 'id'"
	url="http://${hostname}:8587/clients"
	echo ${url}
	curl -X POST -H "Cache-Control: no-cache" -d "{
	\"application\": \"${application}\",
	\"secret\": \"${secret}\"
	}" --url "${url}" \
	| grep '"id":"[a-zA-Z0-9]\{24\}"' > /dev/null
	[ "$?" -ne 0 ] && echo "RESULT: fail" && exit 1
	echo "RESULT: pass"
	echo "SETUP: Get the new client's apiKey for next test"
	url="http://${hostname}:8587/clients"
	echo ${url}
	apiKey=$(curl -X POST -H "Cache-Control: no-cache" -d "{
	\"application\": \"${application}\",
	\"secret\": \"${secret}\"
	}" --url "${url}" \
	| grep -o '"apiKey":"[a-zA-Z0-9]\{24\}"' \
	| grep -o '[a-zA-Z0-9]\{24\}' \
	| sed -e 's/^"//' -e 's/"$//')
	echo apiKey: ${apiKey}
	echo
	echo "TEST: GET request should return a new jwt in the response body"
	url="http://${hostname}:8587/jwts?apiKey=${apiKey}&secret=${secret}"
	echo ${url}
	curl -X GET -H "Cache-Control: no-cache" \
	--url "${url}" \
	| grep '[a-zA-Z0-9_-]\{1,\}\.[a-zA-Z0-9_-]\{1,\}\.[a-zA-Z0-9_-]\{1,\}' > /dev/null
	[ "$?" -ne 0 ] && echo "RESULT: fail" && exit 1
	echo "RESULT: pass"
	echo "SETUP: Get a new jwt using the new client for the next test"
	url="http://${hostname}:8587/jwts?apiKey=${apiKey}&secret=${secret}"
	echo ${url}
	jwt=$(curl -X GET -H "Cache-Control: no-cache" \
	--url "${url}" \
	| grep '[a-zA-Z0-9_-]\{1,\}\.[a-zA-Z0-9_-]\{1,\}\.[a-zA-Z0-9_-]\{1,\}' \
	| sed -e 's/^"//' -e 's/"$//')
	echo jwt: ${jwt}
	echo "TEST: POST request should return a new vehicle in the response body with an 'id'"
	url="http://${hostname}:8581/vehicles"
	echo ${url}
	curl -X POST -H "Cache-Control: no-cache" \
	-H "Authorization: Bearer ${jwt}" \
	-d '{
	"year": 2015,
	"make": "Test",
	"model": "Foo",
	"color": "White",
	"type": "Sedan",
	"mileage": 250
	}' --url "${url}" \
	| grep '"id":"[a-zA-Z0-9]\{24\}"' > /dev/null
	[ "$?" -ne 0 ] && echo "RESULT: fail" && exit 1
	echo "RESULT: pass"
	echo "SETUP: Get id from new vehicle for the next test"
	url="http://${hostname}:8581/vehicles?filter=make::Test|model::Foo&limit=1"
	echo ${url}
	id=$(curl -X GET -H "Cache-Control: no-cache" \
	-H "Authorization: Bearer ${jwt}" \
	--url "${url}" \
	| grep '"id":"[a-zA-Z0-9]\{24\}"' \
	| grep -o '[a-zA-Z0-9]\{24\}' \
	| tail -1 \
	| sed -e 's/^"//' -e 's/"$//')
	echo vehicle id: ${id}
	echo "TEST: GET request should return a vehicle in the response body with the requested 'id'"
	url="http://${hostname}:8581/vehicles/${id}"
	echo ${url}
	curl -X GET -H "Cache-Control: no-cache" \
	-H "Authorization: Bearer ${jwt}" \
	--url "${url}" \
	| grep '"id":"[a-zA-Z0-9]\{24\}"' > /dev/null
	[ "$?" -ne 0 ] && echo "RESULT: fail" && exit 1
	echo "RESULT: pass"

view raw vehicle_tests.sh hosted with ❤ by GitHub

Since these tests are just a bash script, they can from the command line, or easily called from a continuous integration tool, Such as Jenkins CI or Hudson.

Postman

Postman, like several similar tools, is an application designed specifically for test API endpoints. The Postman website describes Postman as tool that allows you to ‘build, test, and document your APIs faster.’  There are two versions of Postman in the Chrome Web Store. They are Postman – REST Client, the in-browser extension, which we mentioned above, and Postman, the standalone application. There is also Postman Interceptor, which helps you send requests that use browser cookies through the Postman application.

Postman and similar applications, have add-ons and extensions to extend their features. In particular, Postman, which is free, offers the Jetpacks paid extension. Jetpacks add the ability to ‘write and run tests inside Postman, extract data from responses, chain requests together and test requests with thousands of variations’. Jetpacks allow you to move beyond basic one-off API request-based testing, to automated regression and performance testing.

Using Postman
Let’s use the same HTTP PUT example we used with cURL and Wget, and see how we would perform the same task with Postman. In the first screen grab below, you can see all elements of the HTTP request, including the RESTful API’s URL, URI including the vehicle’s ObjectId (/vehicles/{ObjectId}.{format}), HTTP method (PUT), Authorization Header with JWT (Bearer), and the raw request body. The raw request body contains a JSON representation of the vehicle we want to update. Note how Postman saves the request in history so we can easily replay it later.

In the next screen-grab, we see the response to the HTTP PUT request. Note the response body, response status, timing, and response headers.

Looking at the response body in Postman, you easily see the how RestExpress demonstrates the RESTful principle we discussed in Part Two of the series, HATEOAS (Hypermedia as the Engine of Application State). Note the link to this vehicle’s ‘self’ href) and the entire vehicles collection (‘up’ href).

Postman Collections
A great feature of Postman with Jetpacks is Collections. Collections are sets of requests, which can be saved, recalled, and shared. The Collection Runner runs requests in a collection, in the order in which you set them. Ordered collections are ideal for the Virtual-Vehicles API. The screen grab below shows a collection of requests, arranged in the order we would execute them to test the Virtual-Vehicles API, as it applies to specifically to vehicle CRUD operations:

1. Execute HTTP POST request to register the new API client, passing the application name and a shared secret in the request
   Receive the new client’s API key in response
2. Execute HTTP GET to request, passing the new client’s API key and the shared secret in the request
   Receive the new JWT in response
3. Execute HTTP POST request to create a new vehicle, passing the JWT in the header for authentication (used for all following requests)
   Receive the new vehicle object in response
4. Execute HTTP PUT request to modify the new vehicle, using the vehicle’s ObjectId
   Receive the modified vehicle object in response
5. Execute HTTP GET to request the modified vehicle, to confirm it exists in the expected state
   Receive the vehicle object in response
6. Execute HTTP DELETE request to delete the new vehicle, using the vehicle’s ObjectId
7. Execute HTTP GET to request the new vehicle and to confirm it has been removed
   Receive a 404 Not Found status response, as expected

Using saved collections for testing the Virtual-Vehicles API is a real-time saving. However, the collections cannot easily be re-run without hand-editing or some advanced scripting. In the simple example above, we hard-coded a JWT and vehicle ObjectId in the requests. Unfortunately, the JWT has an expiration of only 10 hours by default. More immediately, the ObjectId is unique. The earlier collection test run created, then deleted, the vehicle with that ObjectId.

Negative Testing
You may also perform negative testing with Postman. For example, do you receive the expected response when you don’t include the Authorization Header with JWT in a request (401 Unauthorized status)? When you include a JWT, which has expired (401 Unauthorized status)? When you request a vehicle, whose ObjectId is incorrect or is not found in the database (400 Bad Request status)? Do you receive the expected response when you call an actual service, but an endpoint that doesn’t exist (405 Method Not Allowed)?

Postman Test Automation

In addition to manually viewing the HTTP response, to verify the results of a request, Postman allows you to write and run automated tests for each request. According to their website, a ‘Postman test is essentially JavaScript code which sets values for the special tests object. You can set a descriptive key for an element in the object and then say if it’s true or false’. This allows you to write a set of response validation tests for each request.

Below is a quick example of testing the same HTTP POST request, used to create the new API client, above. In this example, we:

1. Test that the Content-Type response header is present
2. Test that the HTTP POST successfully returned a 201 status code
3. Test that the new client’s API key was returned in the response body
4. Test that the response time was less than 200ms

Reviewing Postman’s ‘Tests’ tab, above, observe the four tests have run successfully. Using the Postman’s testing feature, you can create even more advanced tests, eliminating the need to manually validate responses.

This post demonstrates a small subset of the features Postman and other similar applications provide for testing RESTful API. The tools and processes you use to test your RESTful API will depend on the stage of development and testing you are in, as well as the existing technology stacks you build, and on which you host your services.