Friday, 14 April 2017

80/20 service virtualization

How do we create virtual services for all of our test environments?

I have been asked this question recently by a managing director at a large test consultancy based in the UK. They work with financial institutions that use old test infrastructure that need service virtualization because of the typical problems like third party transaction costs, scheduling time on old mainframe environments and test data setup issues.

When we unpack that question, we realise that the short answer is, in most cases, you don’t to start with virtualizing everything. Instead, you focus on the low hanging fruit, start delivering value to testing teams and later scale from there.

80/20 rule in service virtualization

Try to apply the Pareto principle in service virtualization as often as possible. You could interpret it as “80% of value delivered by using service virtualization comes from 20% of virtual services”. Some managers also say that they would “invest time in picking the low hanging fruit first”.

An example of applying that mindset would be to find easy to virtualize APIs that are causing a lot of problems for the testing teams, virtualize them and let the test teams use those assets. Once that has been done, look for the next most important APIs to virtualize and repeat the cycle again.


Tuesday, 11 April 2017

Introduction to Service Virtualization for software testers

A couple of weeks ago we talked about service virtualization at #TesterGathering in London. There were many interesting questions from the audience. Here is the recording of the talk.

Tuesday, 21 March 2017

3 tips for creating virtual services for an Enterprise Service Bus (ESB)

How much of an approximation virtual services might be?


I have been asked this question recently by a senior technical test infrastructure architect. After drilling down to details, I have found that an example he was interested in was, for example, virtualizing an existing ESB (Enterprise Service Bus) in an environment where there are no stubs or virtual services used yet.

I would rate virtualizing a whole ESB a medium or hard difficulty task, depending on the details. Ideally, I would avoid virtualizing large ESBs as your first service virtualization project if possible and pick other simpler APIs to start with. If you have to do it though, here are my top 3 tips. Please bear in mind there is a possibility they might not apply in your case. Every case is different, especially when visualizing large APIs sets.

Don’t boil the ocean


I have made this mistake myself on one of the projects. We were very confident we will be able to deliver virtual services for the whole ESB in a few months, but after a month we have realised the APIs are so complex that there is no way we could even understand some of them without spending weeks in discussions with the developers.

If you have more than 50 endpoints in your ESB, do not try to virtualise everything in one go and spend months doing it. Find a small subset of 1 to 10 APIs that cause problems for the test team and virtualize them in a week or two. Then let the testing teams use those virtual services.

Do service virtualization in small increments of work, delivering value to testing teams on a regular basis. Don’t make them wait for months; it is too risky.

Use record-replay if possible


I was once on a project where the requests and responses contained a lot of data. A SOAP request or response could have included a hundred of different key-value pairs. The XML structure was simple, but the complexity was in the data.

If you have an existing environment with many requests and responses going through the ESB, it is likely those are quite complex APIs. Creating virtual services from a WSDL file will generate the data in the virtual services for you that is unlikely to be representative. It is very likely to be invalid data. To avoid problems like that, use record-replay to create virtual services, this way you capture the request and response structure as well as the data.

Is virtualizing the ESB a good idea at all?


Ask yourself this question, are you virtualizing in the best place? Maybe it would make sense to virtualise some of the backend systems “below” the ESB instead? Maybe it would be valuable and easier to virtualise somewhere “above” the ESB instead? The answer to this question depends heavily on your environment. Contact us to schedule a free 20-minute consultation to help you understand what is the best plan of action in your case.

What is your experience with virtualizing ESBs?

Sunday, 5 March 2017

Reducing Third Party Test Transaction Costs

If you have never heard about stubbing or service virtualization, this article should be a good basic introduction for you on how to reduce test transaction costs. If you already know about stubbing and service virtualization have a look at Traffic Parrot introduction or browse other posts on our blog instead.

Test transaction costs in banking and insurance

Many banking and insurance systems connect to third party services that charge for usage on a per-transaction basis. This will often include but is not limited to accessing market data or detecting fraud.

Testing and test transaction costs

When software is tested together with a third party service, you have to pay for the requests your systems make. In many cases, those costs become significant reaching more than $1k per test suite run. This results in unhealthy decision dynamics, like reducing the amount of testing done or decreasing the frequency of test cycles which later increases the technical debt accumulation.

Reducing the costs

There is a simple solution to that problem. Instead of testing the system always connected to the third party service you can introduce a new testing phase and test with a test double pretending to be the real system. After the system has been tested in isolation, you can proceed integration testing but do only a minimal amount of it. This should reduce the third party test transaction costs to the absolute minimum. We have seen a reduction in third party transaction of up to 93%.

Other benefits

Service virtualization is a popular practice across the industry. Implementing it will not only reduce third party test transaction costs. It has other advantages as well, for example resolving test data setup and environments uptime issues. For more details have a look at our introduction to service virtualization video.

Next steps

Contact us for a free 20-minute consultation call to help you get started with service virtualization or have a look at our introduction to service virtualization video.

Saturday, 15 October 2016

What is Traffic Parrot?

Traffic Parrot helps IT teams deliver faster and with more predictability. It also helps reduce software testing costs. We do that by implementing a practice growing in popularity called service virtualization.

Service virtualization is like using a wind tunnel to test your software. A new model of an aircraft before going to mass market production will be tested in a wind tunnel in different conditions. You can test software in a wind tunnel as well, by using Traffic Parrot.

Are your testers and developers often blocked by other teams? Do you find at least 5 bugs in production per release, and you would like to instead find them in earlier stages of SDLC to reduce costs? Is one of the systems you depend on available only 2 days a week and you need it 5 days a week for testing your application? Is setting up test data in external systems expensive or time-consuming? Are you paying high third party system transaction fees every time you run the same suite of tests? Are you waiting for an external team to deliver an API you need for functional testing of your system?

Traffic Parrot helps with these problems by simulating backend and third party dependencies. For example, a tester will use Traffic Parrot to create virtual services to simulate a backend server. That will allow her to continue testing even if the backend server is unavailable. It also helps with setting up the test data; it is much easier to do that in a virtual service she has control over. You can manually craft the virtual service based on request/response pairs or create them by doing a recording of the traffic between your application and the backend or third party system.

Saturday, 8 October 2016

Traffic Parrot for Agile testing and development teams

Why is it a good idea to buy Traffic Parrot if you are a manager of an Agile development team?

I have seen it take 3-10 days of development to create an over-the-wire stub with a GUI. Average Java developer pay in the UK is £450pd for contractors or £50k for permanent staff. £50k/252 days in a year is £200pd for a permanent developer.

An average estimate cost of in-house development could be minimum 3*£200=£600 and maximum of 10*£450=£4500. So you could be spending on average between £600 to £4500 in a year on developing a tool like this if only a team needs it.

So, a good guess could be you will spend somewhere between £600 and £4500 on a stub with a GUI, for example, £2500, if you build it in-house. Why not buy Traffic Parrot for Agile teams for only $49 a month, which is around £500 a year?

If there is more than one team developing a tool like this, which I have seen happen in one organization, you will spend a multiple of that number.

If you are doing pair programming, the estimate could probably be two times less to two more depending on the pair.

Please contact us to walk around your situation and run these calculations to see if Traffic Parrot makes sense for your team.

Sunday, 4 September 2016

Five steps on how to implement a service virtualization tool proof of concept project in less than a week

Are you a manager tasked with implementing a service virtualization project? Here is an idea on how to implement a service virtualization proof of concept pilot in less than a week:
  1. Find a small to medium size system or application managed by a maximum of 2-12 people. It will be easier to work with a smaller team and on a less complex system.
  2. Have developers working closely with testers to find an API to virtualize
    1. It should be stateless, transactional APIs are harder to virtualize
    2. It should be simple, for example, an HTTP SOAP request with not more than 20 tags to a third party, instead of an encoded custom TCP protocol call to a mainframe system
    3. Should cause significant problems that can be measured and showcased, for example:
      • The backend system exposing the API is often not available preventing testing
      • It is hard to set up test data in the backend system, the wait time is more than 4 weeks
      • The API has not yet been released to the test environments and the wait time is more than 8 weeks
      • Third party service test transactions are expensive, limiting the amount of testing done
  3. Choose a service virtualization tool that allows for a fast ROI 
    • A tool that can run on existing hardware, to save time requesting new hardware (for example TrafficParrot, Wiremock or Mountebank)
    • A tool that can be used by both developers in an automated way for long-term maintainability and testers for exploratory testing (for example TrafficParrot)
  4. Create a squad team consisting of at least one person from development, testing and environment management and trust them to do the job properly. Ideally, they should sit next to each other to improve the communication channels.
  5. Relax and trust the team will deliver. Wait for a week.
Keep in mind, this is only one of many examples how you could implement the pilot project. The devil is in the details. You can request a free, no obligations 20-minute over the phone service virtualization consultation where we will go through your most pressing problems.


Do you disagree? Leave a comment!


Thursday, 18 August 2016

Wednesday, 3 August 2016

My story with service virtualization, CA Lisa and Wiremock





My story with service virtualization

This is my story with service virtualization. If you have ever heard about service virtualization and you would be interested in somebody else's story over a period of five years, please continue reading.

I've been doing software development and testing for more than ten years now, six of which was continuous delivery, continuous integration, TDD, BDD, pair programming, and similar agile practices.

Four companies

This is going to be a story about four companies. I would like you to remember that I'm going to be presenting this story from the continuous delivery point of view. So whenever we're talking about something here, this is from the continuous delivery approach mindset.

Agile 2011-2013

The story begins in 2011-2013 when I was working for organizations that were following the agile principles to a certain extent.

Media company

The first example is a media company, which had 100+ different applications running or different code bases running on more than 200 JVMs. There were 50 people working on those: developers, QAs, database guys, ops guys, support. They were separate functional teams, but they were working very closely together.

When it comes to service virtualization, back then we used to call it stubbing. We would do in-house build stubs. That was because there were no lightweight products available at the time that would fit the continuous integration model that we could use. No open source alternatives or anything like that. Those tools were used by developers and QAs both for functional and performance testing. What I've noticed was it took two to ten days of development to develop such a tool.

Investment bank

Then the story continues and I go to an investment bank developing a risk system. This was a fairly large system running on 1,000 servers in production managed by around 30 people, very cross-functional teams mixed with functional teams. Developers would be doing quality assurance and production support.

In that organization, we also built stubs in-house for developers and it took only two days of development, on average, because they were very specialized. There wasn't much to be built. And, at that time, there still were no lightweight products available that would suit our needs, so we had to build something in-house. We didn't have to cater for QA requirements just because there were no QAs. Developers did everything.

Retail bank

The story continues to 2013-2015 where I was introducing agile to a retail bank. The bank had some issues around delivery. The releases were taking too long, 6 to 12 months. There were a lot of bugs in late stages like UAT testing and production. And it was quite expensive. The complexity of the system was more or less the same as the risk application I was working on in the investment bank I have mentioned, but the retail bank application was 20 times more expensive than that system (sic!).

There were also some architectural challenges. The webapp was a monolith code base with a lot of low-quality Java code. It was connecting to middleware gateways and C++ middleware, and then to 20+ backend systems via different protocols: HTTP, JMS, and custom TCP protocols. There was very little test automation, very long running builds and the reverse testing pyramid in practice. So, almost no unit tests some functional automated testing, some acceptance tests, and then a lot of manual testing on the top.

They had very functional teams, but not working closely together, so developers and QAs did not get along well.

Service virtualization

That was the first time I have heard about service virtualization. In the past, I called this over the wire stubbing or over the wire mocking.

The bank wanted to increase the release schedule predictability and reduce the number of production defects. They hired 15 very experienced and very high profile tech leads to make that change. Those tech leads worked on a few large, strategic projects, like for example modulizing the codebase and moving to a modern source control system. But they were also working on introducing tactical solutions to pick up the low hanging fruit and do something because the strategic solutions would take a long time to implement.

One of the things they came up with was to record the 20+ backend systems, automate testing that way and see what happens. That's how I got hired.

I got hired

I was introduced to CA LISA. CA Service Virtualization is the current name for the system.

My first impression was that it was the first time I had seen this record and replay functionality, which was pretty cool. It was doing templating, so called magic strings. That was cool as well. And it was supporting many protocols, for us, the big win here was custom TCP protocol virtualization that the bank needed to use.

We proceeded to implementation. After six months of trying to do continuous delivery, continuous integration, TDD, and those kinds of things with CA LISA, we found that CA LISA works. We've had several successes with it. We made several teams happier than they were before. They could use a tool to do stubbing or service virtualization. And we had around £300,000 of estimated cost savings.

But we've had some learnings along the way. What we found was that CA LISA didn't play very well with source control systems like GIT. It produces XML files. You can source control them in for example Git, but they're not very easy to merge or compare different versions of those files. If you keep in mind my background and what I have seen in the past, I've seen it done much nicer and much easier.

Also, the centralized deployment infrastructure didn't play very well with our Jenkins build. Ideally what you want to have is everything contained within the build agent and it should do whatever it needs to do to build and test the application. But that didn't play very well with the centralized deployment infrastructure CA LISA provides. Also, because of the licensing model and how heavy the infrastructure was, we couldn't deploy it on every build agent.

Another thing we've noticed is that it was very hard to set up the stubs or virtual services from the test. What you usually do are tests that are split into three logical sections: the "givens section", the "whens section", and the "assertions section". In the givens section, what you're going set up the prerequisites for the test. Then you execute the test. Then, in the last section, you're going to make assertions. And in the givens section, the first section of the test, what you want to do is set up your stubs for the tests to pass. Unfortunately, CA LISA wasn't helping us with that. It was possible, but it didn't look as simple as what we have seen in the past.

The BeanShell extensions were causing us problems as well when scaling to other teams. You couldn't use tools like IntelliJ or Eclipse to write those extensions which was what we have seen in the past. What I have done in the past is write an extension to a stub by simply implementing a Java interface and use code completion provided by your IDE, IntelliJ or Eclipse. But it wasn't possible here. You would have to do it in LISA workstation and it was quite counterintuitive for us. You had to learn a lot of new things before you could do something meaningful.

We had some learnings about the infrastructure as well. Our offshore VMs that were provided to our partners had very little resources and the LISA workstation that was taking a gigabyte of RAM and a lot of disk space. Those VMs had not enough RAM memory to run the LISA workstation in the configuration required by our virtual services. In the past, for the in-house build stubs, we used simple web applications and you could just use the Web browser to connect to that stub. So we felt, that might have been a bit better for us probably at that time.

The infrastructure, separate VSE, separate registry, and separate workstation caused us a few problems as well. For example, the recommended way was to record only in the workstation, and what we would like to do quite often is record actually on the VSEs. Managing the workstations across many VMs and many laptops was a problem also because of the versioning.

We had some learnings around the licensing as well. We wanted to scale to more than 70 teams and ideally keeping the continuous delivery, continuous integration and TDD in mind. We wanted to have an instance of LISA per developer or QA, but that did not look feasible because of the resources it consumed on the system on your VM or your laptop. So we thought, okay, let's have at least one per team. But that still was a problem because of the licensing model, so we had to abandon that idea.

There were a few general user experience learnings we had. We got the feeling there were too many screens, too many controls, and too many components that we just didn't need. It was a very powerful tool, but we didn't need most of the functionality.

Our main takeaway was that LISA works. It supports custom TCP, protocol virtualization, JMS, and HTTP(S). But when we tried to apply it to a continuous delivery environment, it didn't work for us as well as we would hope, keeping in mind my personal experiences with stubbing in agile environments.

It's worth mentioning here that other departments in the bank were using CA LISA as well. Those departments were usually in the maintenance mode, so there was no focus on faster release times, and no focus on things like continuous delivery. As far as I know, CA LISA has been very successful in those environments.

It's also worth mentioning that, in this retail bank, there were two in-house build tools that have been developed to do service virtualization and stubbing (or whatever you want to call it). They were done there to facilitate stubbing of specific components. Unfortunately, neither of those tools were able to record 20-plus backend systems. But what we found was that they fit very well into continuous integration mindset and it took around two to ten days, on average, to develop such a tool.

2015-2016

The story continues to 2015-2016 when I was back to an organization that does a lot of agile. The company had 130+ different applications (different codebases) running on a lot of JVMs in production. There were 50 people managing that working in functional teams very closely together.

Wiremock

By this time the Wiremock platform, an open source stubbing and service virtualization solution, was mature enough that we could start looking at using it.

We have build a service virtualization and over-the-wire stubbing solution on top of Wiremock. It was designed to use by both DEVs and QAs. DEVs would use the Wiremock remote setup programmatic APIs and QAs would use a Web GUI that was built on top of Wiremock to do their exploratory testing and to do performance testing. The QAs were not very tech-savvy in this team and they needed a simple GUI to navigate around the tool. What I found was that it took around 15 days of programming over the last 9 months to do it.

What we found was that Wiremock fits very well into the continuous integration mindset and it's a great tool for developers. It has some bugs, it's still a new tool, but I would use it again if I got the chance. Unfortunately, we had to do some internal development on top of it just because it wasn't designed with non-tech savvy QAs in mind.

Next logical step?

What would be a logical next step for you if you were in my position? Well, for me it was combining those experiences, the traditional and agile methodologies, and the experiences with cross-functional, functional, mixed teams, experiences with CA LISA, using Wiremock as well and building on top of it. Having been in those four companies, I came up with a list of requirements for a tool that would satisfy my needs in all of those circumstances.

New tool requirements

What I wanted to have is a tool that would allow me to deploy anywhere, anytime, where the licensing wouldn't get in the way. What I wanted to have is a lightweight tool that would consume tens or hundreds of megabytes instead of gigabytes of memory. Therefore, since the licensing would be flexible and the tool very lightweight, I could run it on existing hardware.

But that doesn't mean I couldn't run it in a centralized manner. If I wanted to, I could run it both locally and centralized, but the tool shouldn't get in the way of either of those.

I would also like to be able to run this locally or in a centralized environment. And I wouldn't like the tool to get in the way. It doesn't matter if you want to run it centralized on some servers or locally on a laptop. The tool should support both.

I would like the tool to be easy to integrate with CI tools like Jenkins and TeamCity. I would like to run it separately on build agents in an isolated environment to do CI properly. And I would like the tool to be designed with automation in mind, so easy to integrate with things like JBehave, Cucumber, JUnit, etc.

Also, I would like to keep in mind not only Devs but QAs as well. Developers want CI, TDD, etc. What the less tech-savvy QAs want a nice GUI on top of the same service virtualization tool developers use. They also want the same artifacts that the stub produces when used by developers.

I would like to have a tool that supports both functional, cross-functional teams, and I would like a tool that ideally is very simple to use and doesn't need any training, providing great user experience.

Traffic Parrot

And this is how I came to develop Traffic Parrot.

If you look at the tools on the market, there's definitely the option of going open source. And if I was to go open source, I'd probably go for Wiremock today. But if you want to look at other tools and things that are not necessarily Java-related, you want to do some NodeJS or Ruby or something like that, I've compiled a list of 40+ tools.

There's also an option of in-house development. It depends on the environment you're in and how experienced your developers are. That would inform how you would do it. What I have seen, when you've got mid-level to senior guys on your team it's going to take around 5 to 15 days to develop a QA GUI on top of Wiremock. And if you want to start using other protocols like JMS or custom TCP is going to take much longer than that. There's a lot of additional effort required there.

Or, you could go for something like I've already mentioned. An enterprise-ready solution like Traffic Parrot.

What is Traffic Parrot? If you are looking for over-the-wire stubbing or service virtualization tool, and you're doing continuous delivery, continuous integration I would encourage you to go to TrafficParrot.com to learn more about the tool and all the details.

The end

That's my story with service virtualization going through four companies, having different experiences using different types of tools. I hope you enjoyed it. 

Next steps