Monday, 13 June 2022

How to get a FedEx testing tracking number?

The short answer (detailed answer below)

To get one of the testing FedEx tracking numbers, you can use one of the tracking numbers from the table below (copied from Mock Tracking Numbers for FedEx Express and FedEx Ground) or the number 123456789012.

Note that the FedEx Sandbox environment might be unstable during your testing and you might need to use FedEx API mocks instead (either built in-house using Wiremock or ready-made by a vendor).

Scan EventTracking NumberTrack By Reference Info.Shipment Account NumberShip DateDestination Postal Code
Shipment information sent to FedEx449044304137821Customer Reference: 115380173Ground Shipment ID: DMWsGWdnNN51008800015-08-202033126
Tendered149331877648230Ground Shipment ID: 14933187764823051008800015-08-202028752
Picked up020207021381215Ground Shipment ID: 5308952851008800015-08-202030549
Arrived at FedEx location403934084723025Ground Shipment ID: 403934084723025Department: 3182672251008800015-08-202099206
At local FedEx facility920241085725456Customer Reference: 0014243047/34684523 Ground Shipment ID: 92024108572545651008800015-08-202019720
At destination sort facility568838414941Shipper Reference: P218101_004154359Purchase Order: P218101_00415435951008800015-08-202085388
Departed FedEx location039813852990618Customer Reference: 4631962Ground Shipment ID: THE HOUSEDepartment: McGeePurchase Order: 338515851008800015-08-202024740
On FedEx vehicle for delivery231300687629630Customer Reference: W62283340102Ground Shipment ID: 231300687629630Purchase Order: 622833451008800015-08-202033126
International shipment release797806677146N/A
Customer not available or business closed (Delivery Exception 007)377101283611590Ground Shipment ID: 37710128361159051008800015-08-202095815
Local delivery restriction (Delivery Exception 083)852426136339213Customer Reference: 11840271301351008800015-08-202011375
Incorrect address (Delivery Exception 03)797615467620Shipper Reference: OTHER-TK10451008800015-08-202070810
Unable to deliver (Shipment Exception 099)957794015041323N/A
Returned to sender/shipper076288115212522Invoice: 8190951008800015-08-202050323
Clearance delay (International)581190049992N/A
Delivered122816215025810Customer Reference: PO#17472451008800015-08-202024273
Hold at location843119172384577N/A
Shipment canceled070358180009382Customer Reference: 15241402Ground Shipment ID: 07035818000938251008800015-08-202094545
Duplicate Tracking Number713062653486Unique Identifier 1: 2457821000~713062653486~FXUnique Identifier 2: 2457835000~713062653486~FX

The longer answer

The FedEx Sandbox has numerous stability and test data problems. For example, if you use tracking code 123456789012, you will get a different response status depending on the state of the FedEx Sandbox environment.

If you are performing a one-off manual test of your Magento integration with FedEx API, then using the FedEx Sandbox should be sufficient, and you can go ahead and use one of the codes provided above.

If, however, you are running automated regression or integration tests on a schedule or per commit, then the instability of the environment might cripple your testing. For example, you might see flaky tests and flaky builds and have to spend time investigating them even though there is nothing wrong with your code or tests; it's just an issue with the FedEx Sandbox API.

To resolve the issue with FedEx Sandbox environment stability, you have three options, which are essentially about using test-doubles instead of the real FedEx Sandbox API:

  • Create and use mocks in your code instead of using the FedEx Sandbox. For example, Mockito in Java or uniuttest.mock in Python. This means you will have to write the FedEx Mockito mocks yourself. And it comes with several tradeoffs. This is a popular option for developers that have time on their hands and can build solutions internally, and don't work on a tight deadline.
  • Create over-the-wire mocks with tools like Wiremock or Mountebank instead of using the FedEx Sandbox. There is a list of tools like that on Wikipedia. The issue here again is that you have to build those over-the-wire mocks yourself, which can be problematic if it's more than one API and you have a lot of work on your plate anyway.
  • Use ready-made FedEx API mocks like the one from Traffic Parrot instead of using the FedEx Sandbox. The advantage is that experts create them and can use them within minutes. Just copy their mock API URL and use it instead of the FedEx one. Good idea if you are on a tight deadline. You will have to pay for using them, which can sometimes be a downside of this option.

enter image description here

Wednesday, 8 June 2022

Avoid production data in test environments!

As ThoughtWorks have observed in 2021, using production test data in test environments has significant risks and drawbacks, for example:

  • test environments have lower security constraints than production environments, so they expose client data to threats
  • incorrect email test alerts sent to real users 
  • obfuscation often only addresses parts of the data sets
  • copying data can infringe on data protection laws
One way our customers avoid having to copy production data into the test environment is by using synthetic data created in virtual services, or API mocks in our commercial tool Traffic Parrot and a combination of open-source tools like Wiremock or Mountebank.

This often requires an upfront investment to define user personas that can be reused across test cases but removes the risks listed above. It also allows for increased test coverage and enables automated testing. 

As one of our customers put it:

"We have used Traffic Parrot to simulate third-party HTTP services and test our product in isolation. Traffic Parrot consultants have developed custom-made simulators for us. During the first testing cycle, it allowed us to find 28 issues with our product’s release candidate and fix them early in the software lifecycle. This means completing the testing schedule more than 2 months earlier than anticipated just for the first release alone.
We have also used Traffic Parrot to create third party system simulators that communicate over IBM MQ queues. The previous process took at least 30 minutes of manual effort per test case to verify when the interface was available—and sometimes up to 3 days when testing needed to be scheduled with external teams. Our Traffic Parrot simulators respond within milliseconds and are always up. They enabled our team to create automated tests that rely on those service responses and meet our department automated testing KPIs. This allowed us to reduce the regression testing cycle from 30 days to a week."
QA Director working for a software company responsible for government projects

If you would like to see what would be the Return On Investment (ROI) for your team for switching from using production data in test environments to service virtualization and API mocking, please reach out to us.

Self-service service virtualization tool for platform engineering product teams

As Thoughtworks have observed, platform engineering product teams are a sensible default to delivering results to customers in competitive fast-paced environments.

"The emphasis here is on API-driven self-service and supporting tools, with delivery teams still responsible for supporting what they deploy onto the platform. Organizations that consider establishing such a platform team should be very cautious not to accidentally create a separate DevOps team, nor should they simply relabel their existing hosting and operations structure as a platform."  - ThoughtWorks Technology Radar 2017 edition

Commercial tool Traffic Parrot and opensource tools like Wiremock or Mountebank are compatible with self-service teams where there is no centralised admin team managing the service virtualization and API mocking infrastructure. The cross-functional product teams manage Traffic Parrot instances and mocks themselves for their product needs.

As one of our customers put it back in 2017:

'We are building a new platform based on microservices and needed a testing infrastructure that allowed all of our testers to work autonomously, using local machines to record their communication streams between their services and inside our pipelines in order to virtualize a service. Traffic Parrot, while not as fully featured as offerings from HP and IBM, offered the key capabilities that our teams needed such as HTTPS and JMS IBM MQ and a substantially smaller footprint that allowed it to run locally. We have been very pleased by Traffic Parrot's email support and responsiveness to our requests and would recommend them to other teams doing microservice-based architectures."Chief Architect at a Fortune 500 company

And another team that migrated off a centrally managed solution:

"We have migrated our department off CA Lisa to reduce our operational costs in performance and system testing environments. We used WireMock for HTTP stubbing and mocking and Traffic Parrot for IBM MQ. Neither WireMock nor Traffic Parrot provides all the features available in CA Lisa. Still, after initial proof of concept assessment, we have decided those tools' feature sets will be sufficient for our needs and significantly cut costs. It took our team six months to migrate HTTP to WireMock and another three months working closely with the Traffic Parrot team to improve their offering to meet our load testing and mainframe integration requirements. With this investment, we cut our tooling costs in half." - Executive working for a global retail bank.

If you currently use WireMock or Mountebank but need gRPC, JMS, ActiveMQ, RabbitMQ, IBM® WebSphere MQ 7.5+, AMQP, or other protocol support, let’s schedule a 20-minute call to explore if Traffic Parrot is a better fit for your needs.

Friday, 3 June 2022

Traffic Parrot 5.33.11 released, what's new?

We have just released version 5.33.11. Here is a list of the changes that came with the release:


  • Added access logging for UI interactions, written to logs/http-ui-access/http-ui-access-yyyy_mm_dd.log


  • Upgraded bundled JRE from 8u282 to 8u332
  • Upgraded gRPC from 1.43.1 to 1.46.0
  • Upgraded protoc from 3.19.1 to 3.20.1
  • Upgraded WireMock from 2.31.0 to 2.33.2
    • WireMock removed support for parameter normalizing in response templates
    • To restore parameter normalizing behaviour, please set: trafficparrot.virtualservice.handlebars.normalizeHelperParameters=true
  • HTTP VS access logs are now written to logs/http-vs-access/http-vs-access-yyyy_mm_dd.log
  • Upgraded Gradle version in examples project from 4.10.1 to 7.4.2
  • Logs file listing in UI is now ordered by last modified date


  • Library upgrades to fix OWASP issues

Monday, 9 May 2022

Traffic Parrot 5.33.5 released, what's new?

We have just released version 5.33.5. Here is a list of the changes that came with the release:


  • Added new to allow sending custom headers to the on premise license server: trafficparrot.license.usage.custom.headers=ExampleA:ValueA,ExampleB:ValueB


  • Pressing the Cancel button on the Add mapping form now clears the form fields


  • Library upgrades to fix OWASP issues

Tuesday, 3 May 2022

Testing FedEx API Integration: How to work around FedEx Sandbox Issues?

Many companies use FedEx to ship goods to their customers and they will often connect to FedEx APIs to fulfil those needs. One challenge developers and testers face early on and then throughout the product lifecycle is testing that the application integrates well with the FedEx APIs.

Diagram 1: Application connects to FedEx APIs

Diagram 1: Application connects to FedEx APIs

We hear many developers and testers complain about the FedEx Sandbox API environment. Their challenges include that the FedEx Sandbox API environment can have issues without an estimated time to be fixed, return intermittent errors, issues with test data, intermittent downtime and others. There are also issues with performance testing. See “Appendix A - FedEx Sandbox Typical Integration Issues” below.

Diagram 2: Development and Testing with FedEx Sandbox APIs

Diagram 2: Development and Testing with FedEx Sandbox APIs

To work around that problem, we see companies use one of the following options:

  • They continue testing with FedEx Sandbox APIs.

    • They live with the FedEx Sandbox API problems mentioned, ultimately compromising on their software quality, incurring additional development and testing costs and compromising on time to market.

  • They test with production FedEx APIs.

    • This has its drawbacks, as you are actually executing real business transactions inside FedEx systems like ordering shipments etc. Although it's easy to get started with, it is typically not sustainable long term if you plan to test your moderate to high complexity applications, particularly if you have automated regression test suites.

  • They build FedEx API mocks or stubs in-house.

    • Mocks allow them for more flexibility around testing. The challenge is the team has to have the capability to develop, maintain and support them.

  • They rely on third-party vendor solutions such as Traffic Parrot FedEx Sandbox environment that provides FedEx API mocks but also more.

    • Instead of re-inventing the wheel, they purchase ready-made and proven solutions to the problem.

Diagram 3: Using Traffic Parrot FedEx Sandbox and ready-made APIs
Diagram 3: Using Traffic Parrot FedEx Sandbox and ready-made APIs




Continue testing with FedEx Sandbox APIs.

Supported directly by FedEx.

Has a negative impact on development  testing costs and schedules. 

Minimal hypothetical situation testing capabilities.

Slows projects as complexity grows.

Can get costly as complexity grows.

Schedule slips due to intermittent FedEx Sandbox issues.

Risk of not following the testing pyramid.

Test with production FedEx API

Supported directly by FedEx.

Better uptime and availability than the FedEx Sandbox environment.

Impacts production business transactions, which need to be rolled back.

Need to pay for production API usage.

Minimal hypothetical situation testing capabilities.

Slows projects as complexity grows.

Can get costly as complexity grows.

Risk of committing to real business transactions, impacting real customers or inventories.

Risk of not following the testing pyramid.

Build FedEX API mocks or stubs in-house.

Hypothetical scenarios can increase your test coverage.

Need in-house development and maintenance efforts.

Need in-house infrastructure to host the mocks.

Re-inventing the wheel, off-the-shelf solutions are available.

Developing in-house puts schedules at risk, as your team has to develop and maintain mocks.

Developers and testers that work under pressure often do not have time to work on additional tools, they would rather use a ready-made solution. 

Rely on third party vendor solutions such as Traffic Parrot FedEx Sandbox environment

Quick to get started with, ready within 10-40 minutes.

Provide not just mocks but a full Sandbox environment, for more details see “Diagram 4: In-house development of mocks vs. using Traffic Parrot Sandbox environments”

Need a purchase from a third party vendor.


For a detailed comparison between mocks and simulators/sandbox environments, please review Testing Microservices: Examining the Tradeoffs of Twelve Techniques - Part 2.

Diagram 4: In-house development of mocks vs. using Traffic Parrot Sandbox environments
Diagram 4: In-house development of mocks vs. using Traffic Parrot Sandbox environments

Next steps

Contact us to schedule a free 30-minute call so we can discuss the best option for your situation.

Appendix A - FedEx API Sandbox Environment Typical Integration Issues

Service Unavailable

HTTP/1.1 503 Service Unavailable

X-API-Mode: Sandbox

Content-Encoding: gzip

Content-Type: application/json;charset=UTF-8

Content-Length: 236

Server: Layer7-API-Gateway

Date: Mon, 18 Apr 2022 13:59:07 GMT

Connection: close

Vary: Accept-Encoding

Server-Timing: cdn-cache; desc=MISS

Server-Timing: edge; dur=105

Server-Timing: origin; dur=6067


    "transactionId": "18b5df2b-2786-41f3-83e2-22f1307f2672",

    "errors": [


            "code": "SERVICE.UNAVAILABLE.ERROR",

            "message": "The service is currently unavailable and we are working to resolve the issue. We apologize for any inconvenience. Please check back at a later time."




Unknown tracking number

HTTP/1.1 200 OK

X-API-Mode: Sandbox

Content-Encoding: gzip

Content-Type: application/json

Content-Length: 301

Server: Layer7-API-Gateway

Date: Tue, 19 Apr 2022 12:29:04 GMT

Connection: keep-alive

Vary: Accept-Encoding

Set-Cookie: JSESSIONID=7A3EB75A8A39D3F4C5231A03786A1EC8; Path=/track/v1; Secure; HttpOnly;

Set-Cookie: __VCAP_ID__=e063ffd3-0bf4-41a1-6598-4e4f; Path=/track/v1; HttpOnly; Secure;

Server-Timing: cdn-cache; desc=MISS

Server-Timing: edge; dur=93

Server-Timing: origin; dur=197


    "transactionId": "a6a9dbff-7d26-477d-bdbd-688645a9585f",

    "customerTransactionId": "22467615-2115-4c0b-b169-fade80e11aec",

    "output": {

        "completeTrackResults": [


                "trackingNumber": "920241085725456",

                "trackResults": [


                        "trackingNumberInfo": {

                            "trackingNumber": "920241085725456",

                            "trackingNumberUniqueId": "",

                            "carrierCode": ""


                        "error": {

                            "code": "TRACKING.TRACKINGNUMBER.NOTFOUND",

                            "message": "Tracking number cannot be found. Please correct the tracking number and try again."








System unavailable

HTTP/1.1 200 OK

X-API-Mode: Sandbox

Content-Encoding: gzip

Content-Type: application/json

Content-Length: 262

Server: Layer7-API-Gateway

Date: Tue, 19 Apr 2022 12:31:05 GMT

Connection: keep-alive

Vary: Accept-Encoding

Set-Cookie: JSESSIONID=1CA447C0AC9255093CB82E169C292684; Path=/track/v1; Secure; HttpOnly;

Set-Cookie: __VCAP_ID__=f54d4735-4d59-46e2-7967-6877; Path=/track/v1; HttpOnly; Secure;

Server-Timing: cdn-cache; desc=MISS

Server-Timing: edge; dur=92

Server-Timing: origin; dur=5235


    "transactionId": "1378803b-d312-4014-af90-a7ad734dd4a8",

    "customerTransactionId": "c5dd0962-7c96-436d-8a6b-35dea6c32f08",

    "output": {

        "completeTrackResults": [


                "trackingNumber": "568838414941",

                "trackResults": [


                        "trackingNumberInfo": {

                            "trackingNumber": "568838414941",

                            "trackingNumberUniqueId": "",

                            "carrierCode": ""


                        "error": {

                            "code": "SYSTEM.UNAVAILABLE.EXCEPTION",

                            "message": "default.error"








Ambiguous message Invalid Account Number

Fedex API Rates and Transit Times API + LTL Freight: Account Number Invalid 


Lacking clarity which credentials to use 

Rating is temporarily unavailable - Error in FEDEX

“Error returned in processing transaction

Severity: FAILURE

Source: crs

Code: 7

Message: Rating is temporarily unavailable, please try again later.

LocalizedMessage: Rating is temporarily unavailable, please try again later.”

“try using production credentials. I did has the same issue with test server ans was able to resolve it through production credentials”

General Failure when servers are down

fedex General Failure

“when the FedEx servers are down, you get "General failure" as the error message”

Fedex Sandbox has limited number of test credentials

fedex shipping web service, test account authentication failed

“appears Fedex has not gotten their act together regarding their test credentials. I was told by someone in Web services tech support that there is a limited array of test credentials and there are some that are simply not valid. Turns out, once you get test credentials with your user account, they will always be the same when requested from that account.”

FedEx Shipping Codes are not clear

FedEx Shipping Codes

“The problem is, the values your 3PL expects may not be the true "carrier" values so there is no way to provide you with such a list. These are the values that FedEx expects for Web Services which is the current list. The XML API has been retired and did not support all services.”

The courtesy rates service intermittent availability

Unable to obtain courtesy rates in Fed Ex Web Service

“Unable to obtain courtesy rates”

“There is nothing wrong in the code this is mainly due to website issue. I tried the same thing after few hours and it was working perfectly.”

Lack of clarity on available test data

How to get Fedex testing tracking number?

“** Every carrier (UPS, USPS, etc...) has lots of tracking numbers for lots of use-cases, but not FedEx... I find that very strange.

* Also... I am guessing FedEx expects you to use real tracking numbers even in their test environment. It's how I have been testing for multiple use-cases.

I hope this helps anyone, I have spent a while digging around.”

Monday, 4 April 2022

The difference in service virtualization for monolithic and microservice architectures?

We see our customers embark on a migration to microservices from monolithic architectures. 

We get asked often: How does Service Virtualization (SV) and API mocking fit into both worlds and what changes when you migrate from a monolithic architecture to a microservices architecture?

In the diagram below, we demonstrate how our customers typically deploy two instances of an SV solution in a monolithic environment. They would have a sandbox service virtualization environment used by developers and testers to create their virtual services and then a test service used by the development and test environments. See Figure 1.

Figure 1: Service Virtualization in a monolithic architecture

When moving from monolithic architectures to microservice architectures, we see our customers:

  • Create production-like environments prepared on-demand using scripts instead of manually configured environments

  • Use containerisation technologies such as Docker and Kubernetes (OpenShift, Rancher and similar) instead of running on bare hardware of virtual machines.

  • They run production-like environments even locally on developers’ and tester’s laptops or desktops and in CI/CD builds and pipelines instead of development and testing in shared DEV, SIT and QA environments.

A distributed architecture with many production-like development and test environments on local machines, in CD/CI or Kubernetes (or OpenShift or Rachner), requires a service virutlaization tool that can be deployed in many of those environments as well. Those service virtualization tools allow for the new decentralised development and testing approach to be used throughout the lifecycle. This is demonstrated in Figure 2.

Figure 2: Service Virtualization while migrating to a microservices architecture

OpenSource tools like Wiremock or Mountebank allow for that usage pattern, as well as selected commercial offerings such as Traffic Parrot.

Once the migration to microservices has been completed, the centralised SV infrastructure can be deprecated and decommissioned, as demonstrated in Figure 3.

Figure 3: Service Virtualization in microservice architectures

Some of our customers ask us why would they not just use the existing SV approach in the new world of microservices?

Although using the centrally managed solutions in microservice architectures is possible, it comes with maintenance costs caused by:

  • Cannot create production-like environments on-demand by programmable scripts resulting in increased maintenance costs

  • Many laptops, CI/CD pipelines and K8S environments connecting to the same instance of SV create a bottleneck resulting in increased maintenance costs and a single point of failure.

  • Investing in building SV artefacts in monolithic tooling that is costly to migrate to a fit for purpose microservice and container compatible SV tooling incurs long term migration costs.

If you would like to learn more about microservice architectures and what changes when moving away from a monolithic approach, see the introduction to microservices by Martin Fowler and the article by Paul Hammant on release schedules.

Monday, 14 March 2022

Traffic Parrot 5.33.2 released, what's new?

We have just released version 5.33.2. Here is a list of the changes that came with the release:


  • Native IBM® MQ mapping multiple response messages now support zero response messages. This allows for consuming request messages without sending any response message.
  • Added new to allow starting JMS queue replay on startup:
    • trafficparrot.jms.start.queue.replay.on.startup=false
    • trafficparrot.jms.start.queue.replay.on.startup.consumeRequestsFromConnectionId=1
    • trafficparrot.jms.start.queue.replay.on.startup.replayResponsesToConnectionId=1
  • Extended virtual service support for using multiple JMS virtual services linked to separate mappings:
    • The scenarios/VirtualServiceName/ now allows for configuring:
      • jms.queue.replay.on.startup=true
    • The scenarios/VirtualServiceName/jms-connections.json file is used to specify the queue manager connections per virtual service
    • The scenarios/VirtualServiceName/jms-mappings directory contains the mappings files per virtual service
    • The UI allows starting JMS replay for multiple virtual services at a time
  • RabbitMQ JMS queue declare arguments can now be specified in the mapping JSON. Please note that the UI does not yet support this feature.
  • RabbitMQ JMS queue names may now be specified in the format exchange/routingKey instead of a direct queue name


  • Upgraded bundled JRE from 8u292 to 8u322
  • RabbitMQ JMS with AMQP compatibility mode enabled does not support topics and will now fail with an error message

Thursday, 17 February 2022

Traffic Parrot 5.32.0 released, what's new?

We have just released version 5.32.0. Here is a list of the changes that came with the release:


  • Added support for using a proxy server with path based routing between Traffic Parrot and the license server

Sunday, 30 January 2022

Traffic Parrot 5.31.2 released, what's new?

We have just released version 5.31.2. Here is a list of the changes that came with the release:


  • Added ability to import and export IBM MQ mappings in ZIP file format
  • JMS response messages now populate the response JMSCorrelationID using the JMSCorrelationID from the request message if it is present or else the JMSMessageID from the request
  • JMS response messages are now sent to the JMSReplyTo destination if present or else the mapping destination is used


  • Updated VS HTTPS certificates with new 3 year validity period


  • Library upgrades to fix OWASP issues

Wednesday, 26 January 2022

Is trunk-based development the same as a monorepo?

"Is trunk-based development the same as a monorepo?" - an executive working for a large financial institution 

Monorepo means you have one source code repository for many projects/deliverables/modules/... which means all sources are version controlled in the same place.

Trunk based development means everybody works on the same branch (trunk/master/...) which allows for high levels of continuous integration, because everybody sees the changes you commit as soon as you commit them, and you see other developers changes soon as well.

The alternative is feature branching, where you integrate when you merge your branch to the trunk/master.

Do you need help with understanding the details of trunk-based development and monorepos and the impact on your organization? Feel free to contact us.

Monday, 24 January 2022

Accelerate delivery, improve quality and reduce time-to-market

Traffic Parrot can help you with:

  • Unblocking testers and developers waiting for APIs (faster time to market)
  • Easier and faster to reproduce production bugs (faster time to market)
  • Speed up setting up test data (faster time to market, higher quality)
  • Eliminate the need for scheduling time on environments (faster time to market, higher quality)
  • Reduce third-party transaction costs (lower costs)
  • Make your performance tests more reliable (higher quality)
  • Allow to develop automated test sooner and run the builds run faster (faster time to market, higher quality)
Please, contact us for more details. Feel free to review our information on "Getting started with stubbing, mocking and service virtualization".

Friday, 7 January 2022

Traffic Parrot 5.30.1 released, what's new?

We have just released version 5.30.1. Here is a list of the changes that came with the release:


  • Added support for connecting to an internal or external ActiveMQ broker via JMS AMQP 1.0
  • Added support for username/password when connecting to external ActiveMQ broker


  • Include ActiveMQ sample JMS AMQP 1.0 connection


  • Library upgrades to fix OWASP issues
  • Fixed an issue with the {{ regex }} helper when using an {{ xPath }} result as a parameter

Tuesday, 4 January 2022

Three case studies on API-first Development and Consumer-Driven Contracts

At Traffic Parrot we have seen several companies embark on the API-first journey, a few of which also approach their API strategy consumer-first and implemented Consumer-Driven contracts.

In this short article, we summarise two categories of engagements we have seen faced by the teams:

  • Implementing API-first approach for teams that have not done API-first before and starting their journey with microservices (up to 5-20 microservices in production)

  • Implementing Consumer-Driven Contract Testing on teams that are already proficient in API-first development approach and face issues with large scale contract testing of microservices (more than 50-100 microservices in production)

Useful terms to know

  • API-first development: the move to microservices typically also drives an API-first development approach, where teams define upfront business contracts between each other using APIs. A sample API specification language: OpenAPI or Protobuf

  • Consumer-Driven Contracts: When you design an API, its syntax and semantics will be designed by the API producer team by working closely with the teams that will consume the API as opposed to the API producer team making assumptions based on the data model how the API should look like. The API consumer teams drive the shape of the APIs.

  • Consumer-Driven Contract Testing: A type of contract testing where we are ensuring in automated tests that the contracts designed in a consumer-driven fashion are met. This allows confidence that a new release of a microservice will not result in breaking API changes in production. A sample tool: Pact-JVM

High-level summary

It has been our clients’ experience that:

  1. The API-first approach is an effective way to parallelise work between teams working in fast-paced environments and microservice-based architectures.

  2. Allowing teams to work closely with each other and design APIs in a consumer-driven fashion helps drive quality and reduce time to market no matter the team size and deployment scale.

  3. Although consumer-driven testing is a well-grounded way of testing contracts it can be counterproductive to introduce it to teams that are not experienced in the API-first approach or run less than 5-20 microservices in production.

Case study: a large media company using an API-first approach and Consumer-driven testing 

A global media company had an existing stack of 100+ microservices that was primarily tested with automated BDD E2E tests. The releases were done every 2-4 weeks.

These tests were costly to maintain because it took significant developer time to write and debug the suite of tests. The suite of tests took 2 hours to run, and up to a week to investigate issues before every release.

Developers were often frustrated as the tests were flaky due to the complexity of the system under test, leading to many non-deterministic failure points. These tests would prevent them from releasing new features on-demand as the tests took a few hours to run. 

With this experience, the company decided to avoid E2E testing for the new product they were working on.

The main way they decided to grow confidence in the contracts between new microservices and the behaviour of the product as a whole was to design contracts in a consumer-driven way. The company chose consumer-driven contract testing with Pact-JVM to test those contracts. Most of the teams were entirely new to consumer-driven contracts. 

Equipped with a solid API-first design approach background across all teams, and several years of experience with automated BDD-style Acceptance and E2E testing, as well as TDD on unit-level they felt confident in learning the consumer-driven contract testing approach. Unfortunately, that proved not to be true after a few months of working on the problem. It was the teams’ experience that about a fifth of the developers did pick up the new workflow rapidly but the majority of the team had issues understanding the new workflow even after a few months of working on it and about a fifth of the team even after 12 months did not onboard themselves fully to the new process. 

There was a change to infrastructure happening at the same time, moving away from bare-metal to Kuberenetes and Docker deployments. 

The J-Curve Effect of this transformation was noticeable but manageable. Teams adopted just a few new tools and practices as part of this transformation (PactJVM, Docker, Kubernetes) and it was enough to keep developers and testers busy changing their daily habits. In retrospect, the investment in consumer-driven contract testing was justified but the ROI was not significant, close to breaking even. 

Case study: a global e-commerce giant using API-first approach

The company decided to move away from a monolithic architecture to more autonomous teams and microservices. As part of that transition, they decided to recommend good practices rather than force the use of specific technologies and solutions onto teams, trusting the teams to make the right judgements on which tools and techniques will make the most positive impact on their daily activities.

The developers wrote integration tests and used an API mocking tool Traffic Parrot to mock dependent components. They also wrote Cucumber/Gherkin BDD acceptance API tests to capture the business requirements (they called these "contract tests"), which use a Docker image of the microservice and a Docker image with the Traffic Parrot API mocks. The BDD tests verify both the microservice API and interactions with dependent components by verifying the interactions on the API mocks. That way, the BDD tests verify both microservice API request and response and all communication with dependent components by assertions and verifications.

The company decided to create the API mocks in two ways.

First, if the API that a developer wants to consume already exists, they create the API mocks by recording requests and responses. A developer starts by creating a new test on their computer. They then run the test and create API mocks by recording them. They commit the tests and mocks to the microservice project in Git. In a QA pipeline (a pipeline that is run per commit to checking the quality of the product), they start a Docker container that runs the API mocking tool and mounts the mock definitions from the microservice project.

Second, if the API the microservice will consume does not exist yet, a developer will create the API mocks from OpenAPI specifications for HTTP REST APIs or create the API mocks from protocol buffer specification files for gRPC APIs.

They also develop and run automated E2E smoke tests. This is one of the techniques to test contracts between microservices and it makes sure groups of microservices work well together. The presence of the E2E test suite is justified as it is tested not only the producer side of contracts, which is tested in the BDD tests, but also the consumer side, and so provides more confidence. The architects monitor the number of E2E tests. They keep the complexity of the E2E test suite at a level that does not cripple the release process or daily development activities.

The move to consumer-driven contract testing was not justified in the eyes of the architects and individual teams on an enterprise-scale as the J-Curve Effect impact would be too significant and the ROI not justifiable in a reasonable timeframe. Out of the hundreds of developers that work for the company, only a handful decided to use consumer-driven contract testing, as they were already very familiar with the API-first approach and had several years of experience in BDD.

Case study: an InsurTech startup using API-first approach

The company had a handful of teams working on a dozen of microservices. The microservices were replacing part of a deprecated monolith.  

The producer teams designed the APIs for the teams to consume. To manage contracts between teams and allow teams to work in parallel, they decided to use API mocks that the API producers created and shared with the API consumers. They created the gRPC API service mocks using an API mocking tool Traffic Parrot.

They also used a handful of manual E2E tests in a pre-production environment to make sure that the microservices would work together in the absence of sufficient automated testing. They would fill the gaps in automated testing on an ongoing basis, which took 12 months of ongoing part-time effort.

To make sure mocks do not get out of date, and to hit their aggressive deadlines, the company decided to test the API mocks by firing a request at both a mock and a real microservice. They compared both responses to a contract definition of expected request/response pairs as defined in a company-specific custom format. This way, the API mocks and the real service was proven to be up to date when compared to the latest definition of the expected behaviour defined in the contract file. This proved to be a good enough solution to remove the bottleneck of testing contacts and allow the teams to focus on delivering the features to customers.

The team contemplated using consumer-driven contact testing but at first glance, it seemed like they would be shooting a fly from a cannon and they have decided to stick to their original good-enough testing plan and revisit the issue in 12-24 months while keeping an eye on the complexity of the existing suite of tests so the maintenance costs do not become significant.

Next steps

At Traffic Parrot we would be happy to help you and your team provide feedback on your transformation strategy. Reach out to us to schedule a call where we walk around your requirements and provide early feedback. If your experience with Consumer-Driven Contract testing was different, please let us know in the comments below!