Tuesday, 9 May 2017

Flaky Tests at Google

Jeff Listfield has published an interesting read on where does tests’ flakiness at Google come from.

The main takeaway from that article is that the larger the binary size and more RAM the test consumed, the more likely it was to be flaky.

So, try to avoid creating large tests, and you should have less flaky tests!

It is worth noting that one way of addressing the flakiness of certain types of tests is using stubbing, mocking or service virtualization. When used properly, it can help stabilize the environment. It has to be assessed on a case-by-case basis if that is the right thing to do. For example, system tests will often be run in isolation, but end to end or smoke tests will require the whole stack to be deployed.

Monday, 1 May 2017

4 tips on "selling" service virtualization inside a large enteprise

What are the next steps if we have done a "POC" and people still don’t want to use service virtualization?

A service virtualization consultant recently asked me the following question.

"We have finished a proof of concept project and demonstrated that service virtualization would work in our company. We have presented our findings to several teams, but none were interested enough to start implementing it in their sprints. What do we do now?" Service Virtualization Specialist at a Fortune 300 company.

Selling service virtualization

There are a few ways this problem can be approached depending on the details and specifics of the organization.

There are four key points to make that should apply to your situation.

1. Selling

You are no longer only a service virtualization specialist once the proof of concept project has finished. Often, you need to become a salesman as well. You need to apply proper sales and marketing practices to win over individuals and teams inside your organization to help them solve their pressing problems.

2. Solving real problems

Good sales people solve people's problems; they do not push solutions. They make people understand what they offer is a good idea for a given situation if there is a big enough problem to solve, or walk away.

When you sell, remember you have to deliver value to the teams by solving their real problems of high priority. You have to build rapport and understand why do they think they do not need SV and what problems do they have that you could help them with. You need to “get a Ph.D.” in their internal and external issues. You also have to understand political issues inside the team that might prevent them from implementing service virtualization. Once you understand the whole environment inside out, you can suggest solutions.

3. Technology adoption lifecycle

Remember that in every large enough organization you will find people that will be very keen to try the new approach and individuals that will resist using it until everybody else is onboard. It’s called the Technology adoption lifecycle. Look for innovators and early majority.

4. Influencers

Some organizations have natural leaders, tech leads, architects or other individuals that developers and testers look up to. You could approach those influencers and work with them to spread service virtualization knowledge.

What are your experiences with “selling” service virtualization in your organization?





Friday, 14 April 2017

80/20 service virtualization

How do we create virtual services for all of our test environments?

I have been asked this question recently by a managing director at a large test consultancy based in the UK. They work with financial institutions that use old test infrastructure that need service virtualization because of the typical problems like third party transaction costs, scheduling time on old mainframe environments and test data setup issues.

When we unpack that question, we realise that the short answer is, in most cases, you don’t to start with virtualizing everything. Instead, you focus on the low hanging fruit, start delivering value to testing teams and later scale from there.

80/20 rule in service virtualization

Try to apply the Pareto principle in service virtualization as often as possible. You could interpret it as “80% of value delivered by using service virtualization comes from 20% of virtual services”. Some managers also say that they would “invest time in picking the low hanging fruit first”.

An example of applying that mindset would be to find easy to virtualize APIs that are causing a lot of problems for the testing teams, virtualize them and let the test teams use those assets. Once that has been done, look for the next most important APIs to virtualize and repeat the cycle again.


Tuesday, 11 April 2017

Introduction to Service Virtualization for software testers

A couple of weeks ago we talked about service virtualization at #TesterGathering in London. There were many interesting questions from the audience. Here is the recording of the talk.

Tuesday, 21 March 2017

3 tips for creating virtual services for an Enterprise Service Bus (ESB)

How much of an approximation virtual services might be?


I have been asked this question recently by a senior technical test infrastructure architect. After drilling down to details, I have found that an example he was interested in was, for example, virtualizing an existing ESB (Enterprise Service Bus) in an environment where there are no stubs or virtual services used yet.

I would rate virtualizing a whole ESB a medium or hard difficulty task, depending on the details. Ideally, I would avoid virtualizing large ESBs as your first service virtualization project if possible and pick other simpler APIs to start with. If you have to do it though, here are my top 3 tips. Please bear in mind there is a possibility they might not apply in your case. Every case is different, especially when visualizing large APIs sets.

Don’t boil the ocean


I have made this mistake myself on one of the projects. We were very confident we will be able to deliver virtual services for the whole ESB in a few months, but after a month we have realised the APIs are so complex that there is no way we could even understand some of them without spending weeks in discussions with the developers.

If you have more than 50 endpoints in your ESB, do not try to virtualise everything in one go and spend months doing it. Find a small subset of 1 to 10 APIs that cause problems for the test team and virtualize them in a week or two. Then let the testing teams use those virtual services.

Do service virtualization in small increments of work, delivering value to testing teams on a regular basis. Don’t make them wait for months; it is too risky.

Use record-replay if possible


I was once on a project where the requests and responses contained a lot of data. A SOAP request or response could have included a hundred of different key-value pairs. The XML structure was simple, but the complexity was in the data.

If you have an existing environment with many requests and responses going through the ESB, it is likely those are quite complex APIs. Creating virtual services from a WSDL file will generate the data in the virtual services for you that is unlikely to be representative. It is very likely to be invalid data. To avoid problems like that, use record-replay to create virtual services, this way you capture the request and response structure as well as the data.

Is virtualizing the ESB a good idea at all?


Ask yourself this question, are you virtualizing in the best place? Maybe it would make sense to virtualise some of the backend systems “below” the ESB instead? Maybe it would be valuable and easier to virtualise somewhere “above” the ESB instead? The answer to this question depends heavily on your environment. Contact us to schedule a free 20-minute consultation to help you understand what is the best plan of action in your case.

What is your experience with virtualizing ESBs?

Sunday, 5 March 2017

Reducing Third Party Test Transaction Costs

If you have never heard about stubbing or service virtualization, this article should be a good basic introduction for you on how to reduce test transaction costs. If you already know about stubbing and service virtualization have a look at Traffic Parrot introduction or browse other posts on our blog instead.

Test transaction costs in banking and insurance

Many banking and insurance systems connect to third party services that charge for usage on a per-transaction basis. This will often include but is not limited to accessing market data or detecting fraud.

Testing and test transaction costs

When software is tested together with a third party service, you have to pay for the requests your systems make. In many cases, those costs become significant reaching more than $1k per test suite run. This results in unhealthy decision dynamics, like reducing the amount of testing done or decreasing the frequency of test cycles which later increases the technical debt accumulation.

Reducing the costs

There is a simple solution to that problem. Instead of testing the system always connected to the third party service you can introduce a new testing phase and test with a test double pretending to be the real system. After the system has been tested in isolation, you can proceed integration testing but do only a minimal amount of it. This should reduce the third party test transaction costs to the absolute minimum. We have seen a reduction in third party transaction of up to 93%.

Other benefits

Service virtualization is a popular practice across the industry. Implementing it will not only reduce third party test transaction costs. It has other advantages as well, for example resolving test data setup and environments uptime issues. For more details have a look at our introduction to service virtualization video.

Next steps

Contact us for a free 20-minute consultation call to help you get started with service virtualization or have a look at our introduction to service virtualization video.

Saturday, 15 October 2016

What is Traffic Parrot?

Traffic Parrot helps IT teams deliver faster and with more predictability. It also helps reduce software testing costs. We do that by implementing a practice growing in popularity called service virtualization.

Service virtualization is like using a wind tunnel to test your software. A new model of an aircraft before going to mass market production will be tested in a wind tunnel in different conditions. You can test software in a wind tunnel as well, by using Traffic Parrot.

Are your testers and developers often blocked by other teams? Do you find at least 5 bugs in production per release, and you would like to instead find them in earlier stages of SDLC to reduce costs? Is one of the systems you depend on available only 2 days a week and you need it 5 days a week for testing your application? Is setting up test data in external systems expensive or time-consuming? Are you paying high third party system transaction fees every time you run the same suite of tests? Are you waiting for an external team to deliver an API you need for functional testing of your system?

Traffic Parrot helps with these problems by simulating backend and third party dependencies. For example, a tester will use Traffic Parrot to create virtual services to simulate a backend server. That will allow her to continue testing even if the backend server is unavailable. It also helps with setting up the test data; it is much easier to do that in a virtual service she has control over. You can manually craft the virtual service based on request/response pairs or create them by doing a recording of the traffic between your application and the backend or third party system.

Saturday, 8 October 2016

Traffic Parrot for Agile testing and development teams

Why is it a good idea to buy Traffic Parrot if you are a manager of an Agile development team?

I have seen it take 3-10 days of development to create an over-the-wire stub with a GUI. Average Java developer pay in the UK is £450pd for contractors or £50k for permanent staff. £50k/252 days in a year is £200pd for a permanent developer.

An average estimate cost of in-house development could be minimum 3*£200=£600 and maximum of 10*£450=£4500. So you could be spending on average between £600 to £4500 in a year on developing a tool like this if only a team needs it.

So, a good guess could be you will spend somewhere between £600 and £4500 on a stub with a GUI, for example, £2500, if you build it in-house. Why not buy Traffic Parrot for Agile teams for only $49 a month, which is around £500 a year?

If there is more than one team developing a tool like this, which I have seen happen in one organization, you will spend a multiple of that number.

If you are doing pair programming, the estimate could probably be two times less to two more depending on the pair.

Please contact us to walk around your situation and run these calculations to see if Traffic Parrot makes sense for your team.

Sunday, 4 September 2016

Five steps on how to implement a service virtualization tool proof of concept project in less than a week

Are you a manager tasked with implementing a service virtualization project? Here is an idea on how to implement a service virtualization proof of concept pilot in less than a week:
  1. Find a small to medium size system or application managed by a maximum of 2-12 people. It will be easier to work with a smaller team and on a less complex system.
  2. Have developers working closely with testers to find an API to virtualize
    1. It should be stateless, transactional APIs are harder to virtualize
    2. It should be simple, for example, an HTTP SOAP request with not more than 20 tags to a third party, instead of an encoded custom TCP protocol call to a mainframe system
    3. Should cause significant problems that can be measured and showcased, for example:
      • The backend system exposing the API is often not available preventing testing
      • It is hard to set up test data in the backend system, the wait time is more than 4 weeks
      • The API has not yet been released to the test environments and the wait time is more than 8 weeks
      • Third party service test transactions are expensive, limiting the amount of testing done
  3. Choose a service virtualization tool that allows for a fast ROI 
    • A tool that can run on existing hardware, to save time requesting new hardware (for example TrafficParrot, Wiremock or Mountebank)
    • A tool that can be used by both developers in an automated way for long-term maintainability and testers for exploratory testing (for example TrafficParrot)
  4. Create a squad team consisting of at least one person from development, testing and environment management and trust them to do the job properly. Ideally, they should sit next to each other to improve the communication channels.
  5. Relax and trust the team will deliver. Wait for a week.
Keep in mind, this is only one of many examples how you could implement the pilot project. The devil is in the details. You can request a free, no obligations 20-minute over the phone service virtualization consultation where we will go through your most pressing problems.


Do you disagree? Leave a comment!


Thursday, 18 August 2016