How should data pipeline teams test their products?

Our client is looking to expand the usage of our tooling at their enterprise, and among many teams, we talked to a data pipeline team.

The advice we have given them was to:
1. Write automated tests for their data pipelines
2. Consider shift-left testing of their data pipelines
3. Use an API design-first approach when integrating with other teams

Thoughtworks has two great articles with an introduction to testing data pipelines and ML quality:

• https://www.thoughtworks.com/en-gb/insights/blog/testing/testing-data-pipelines 
• https://www.thoughtworks.com/insights/blog/machine-learning-and-ai/effective-ml-part-II 

We recommended to this team that they use an API design-first approach as described in the article we published a while back https://www.infoq.com/articles/api-mocking-break-dependencies/ 

API-first and API-mocks will allow their consumer teams to start working on their critical features before the data pipeline and ML code are ready, reducing the time to market for the new critical functionality, which will make the product managers and users very happy!

Contact us for a free-of-charge, no-obligation 30-minute call if you would like to discuss how design-first APIs can help accelerate critical deliverables at your organization.

Traffic Parrot 5.46.7 released, what's new?

We have just released version 5.46.7. Here is a list of the changes that came with the release:

Features

• Added support for sending one or more JMS text messages after an HTTP or gRPC response, configured via mapping JSON:

  "postServeActions" : [ {
      "name" : "send-jms-message",
      "parameters" : {
        "jmsConnectionId" : "(connection id from jms-connections.json)",
        "destination" : {
          "name" : "queue-name",
          "type" : "QUEUE"
        },
        "variables" : {
          "id" : "{{randomValue length=24 type='ALPHANUMERIC'}}"
        },
        "properties" : {
          "id" : "{{variables.id}}"
        },
        "jsonBody" : {
          "id" : "{{variables.id}}",
          "fieldFromRequest" : "{{originalRequest.jsonBody.requestField}}",
          "fieldFromResponse" : "{{originalResponse.jsonBody.responseField}}"
        },
        "delayDistribution" : {
          "type" : "fixed",
          "milliseconds" : 500
        }
      }
  } ]

• Added additional Okta JWT access token verification
• Added gRPC native transport for Apple ARM based processors

Fixes

• Fixed desktop issue where the start and stop scripts would steal focus on launch
• Fixed logging issue where HTTP and HTTPS GUI ports would be printed in the logs even if a port is disabled
• Fixed a DNS resolution failure issue on Mac
• Improved OpenAPI parser compatibility
• Improved error message when script runtime is missing

Changes

• Upgraded bundled JRE from 8u392 to 8u402
• Upgraded gRPC from 1.60.0 to 1.63.0
• This release is compatible with the MQ com.ibm.mq.allclient JAR up to version 9.3.4.1
• When using IBM MQ Advanced for Developers Docker icr.io/ibm-messaging/mq:9.3.4.0-r1 and above, MQ_APP_PASSWORD must be set for the app user, the default lack of password is no longer supported by the container
• IBM MQ username and password are now interpreted as null if set to "" or "null" in the connection JSON configuration
  • To configure a "" value use "<empty>"
  • To configure a "null" value use "<null>"
• Improved JMS message logging for null property values
• Library upgrades to fix OWASP issues