Pros and cons of running API mocks inside CI/CD

Our customers choose to run their mocks inside and also outside their CI/CD builds and pipelines. There are advantages and disadvantages to both approaches.

Typically, if the system under test is running (microservice under test) inside CI/CD then our customers would also run their API mocks inside CI/CD.

On the first diagram below, we show how running API mocks inside CI/CD would look like, on the second diagram we show how running outside CI/CD would look like. 

Pros of running inside CI/CD

• Major: More resilient builds because they do not rely on any external components or infrastructure
• Major: More resilient builds due to relying on known API mock state by using ephemeral per-test-run API mock instances with the test-primed state instead of long-running API mock instances with undefined state
• Major: Easier to set up and maintain builds by running API mocking (service virtualization) tool from maven/Gradle before the tests and tear down after tests

Cons of running inside CI/CD

• Critical: To run API mocks inside CI/CD the system under test needs to be capable of running inside CI/CD as well
• Major: People need to be trained on how to use mocks inside CI/CD
• Major: Need an API mocking (service virtualization) tool with a low footprint that is designed to be run per test or per build (i.e Wiremock, Mountebank or Traffic Parrot)
• Major: Performance testing might need additional hardware resources to support 1000+ TPS (req/s)
• Minor: Debugging builds with no access to a running API mock tool instance means the observability of the API mocking tool needs to be high - the logs need to contain enough information to   
• Minor: additional hardware resources needed on the build agent/slave/runner