How should data pipeline teams test their products?

Our client is looking to expand the usage of our tooling at their enterprise, and among many teams, we talked to a data pipeline team.

The advice we have given them was to:
1. Write automated tests for their data pipelines
2. Consider shift-left testing of their data pipelines
3. Use an API design-first approach when integrating with other teams

Thoughtworks has two great articles with an introduction to testing data pipelines and ML quality:

• https://www.thoughtworks.com/en-gb/insights/blog/testing/testing-data-pipelines 
• https://www.thoughtworks.com/insights/blog/machine-learning-and-ai/effective-ml-part-II 

We recommended to this team that they use an API design-first approach as described in the article we published a while back https://www.infoq.com/articles/api-mocking-break-dependencies/ 

API-first and API-mocks will allow their consumer teams to start working on their critical features before the data pipeline and ML code are ready, reducing the time to market for the new critical functionality, which will make the product managers and users very happy!

Contact us for a free-of-charge, no-obligation 30-minute call if you would like to discuss how design-first APIs can help accelerate critical deliverables at your organization.