Synthetic Testing: The Art of Faux Testing

5 min readMay 30, 2024

Inspiration

The first time I came across the word “synthetic testing” was a year or two back. I was tasked to implement a simple recurring test to validate the health of a critical endpoint as part of my teams’ production readiness checklist. I did this using the Datadog Synthetic Monitoring functionality. It was quite straight forward & useful but didn't see much into its bigger picture at the time.

I was later inspired to write about this topic when I came across the term again in my journey to get certified as a AWS Solution Architecture Professional. This time I saw the light of how significant it can be for informing & achieving operational objectives in distributed systems. This post is intended to dig deep into it and share sensible defaults on the 3 W’s for its applicability.

What is Synthetic Testing?

Synthetic testing is also known as semantic testing, synthetic monitoring testing, & active/proactive monitoring testing. The popular monitoring approaches like RUM & APM are passive in nature as issues are only raised after it has been experienced by the users. This shortcoming is what synthetic testing sought to alleviate. The main point of synthetic testing is to catch issues before the the user experience them. This is achieved by actively probing your application in a live environment.

Synthetic Monitoring Tests Design Blocks. Taken from Microsoft Engineering Playbook — https://microsoft.github.io/code-with-engineering-playbook/automated-testing/synthetic-monitoring-tests/

As defined by DataDog — “Synthetic testing is a way to identify performance issues with key user journeys by simulating real user traffic in production or production-like environment.”
As defined by Microsoft— “Synthetic tests are a set of functional tests that target a live system in production to verify its health and resilience continuously.”
As defined in MartinFowler.com — “Synthetic tests runs as a subset of an application’s automated tests against the live production system on a regular basis to detect failing business requirements. It relies on adequate monitoring, at the bare minimum, for it to be effective.”

To cut across all of these definitions, effective synthetic testing can be broken down into 4 parts;

Focus Area — Absolute critical paths (key user journeys & business functionalities)

Input Data — Simulated real user data

Where (Live) — Production and/or production-like environment

Outcome —

Detecting failing business requirements
Identify operational issues (functional, performance, …)
Continuously reiterate product’s good health & resilience

Classes of Synthetic Testing

Synthetic testing can be categorised into different flavours based on two different criteria as shown below;

Based on app layer under test —

Browser testing
API testing — Different flavours depending on data exchange protocol used by the app e.g., HTTP, SSL, TCP, DNS, …

Based on aspects of the app under test —

Availability testing — e.g., health checks
Performance testing — e.g, load testing, stress testing, …
Transaction flow testing — ensure reliability of a feature flow which can involve multiple components.
User journey testing — ensure the reliability of specific user journeys which can involve a single component or multiple components.
Smoke testing — Regression or backwards compatibility testing for stability.

Sensible Defaults: 3 W’s — Where, What, & Which

Prod environment — READ ops only

Simple health checks
Monitor uptime of critical functionality

Prod-like environment — READ & WRITE ops

Same as above but with more flexibility
Simulating entering new markets — e.g., Opening up functionality to new line of business
Monitor 3rd party APIs
Integrate into deployment pipeline for quicker & consistent feedback

Sensible default recommendations for the 3 W’s of synthetic test applicability

How To

Below tools can be used to set up synthetic testing with varying level of ease;

API —

Datadog synthetic API testing
Postman (SDK for securely integrating into deployment pipeline)
AWS CloudWatch Synthetics Canary Blueprints
Spring Boot Actuators (Need to be used in conjunction w/ some of the other tools)
JMeter
Wrk

UI —

AWS CloudWatch Synthetics Canary Blueprints
Cypress
Selenium

Benefits

Shifts focus from MTBF (Mean Time Btw Failures) to MTTR (Mean Time To Recovery) — In distributed systems, it is difficult to keep MTBF at a minimum since there are multiple services involved that are probably owned by multiple team that are globally distributed. A mindset change is one of the way to mitigate this by testing through out the development life cycle into production & continuously.

“MTTR > MTBF, for most types of failures” — John Allspaw

Minimises MTTR — With a proactive approach to testing & monitoring, you’ve a greater chance of catching issues earlier. Depending on what kind of issue it is, you can either fix it or roll back even before the user experience it.
Confidently expand to new markets & regions — Effective synthetic monitoring tests gives you the confidence that in a off chance that a bug ever makes it to production, it would be caught almost instantly. This makes it easier to expand your user base into new markets & regions.
Achieve performance objective — Makes it easier to adhere & maintain system OKRs, SLOs, & SLAs.
Informs performance objectives — Remember the AWS Observability motto — “Remember to build, measure, learn, repeat!”. Synthetic testing can help you set sensible expected baseline for app’s current state. Good benchmark metrics aren’t set in stone. In conjunction with passive monitoring, synthetic testing can play a part in informing the evolution of these metrics.
More regular releases — A must have to drive towards daily deploys

Drawbacks

Cost/Effort vs Value — UI/Browser tests can be quite brittle. Upfront effort to set up test might not necessarily yield long term benefits if the UI changes ever so often.

Conclusion

From AWS Serverless badge — “When you’re operating your serverless applications at scale, you can’t afford to fly blind. You need to be able to answer important operational and business questions.”

Combining synthetic testing with passive monitoring allows us to not only gain more insight into these questions but ask better questions. It also re-affirms the operational goals set out for the application.

At different level, most development team have simple synthetic tests like health checks. This can be taken abit further to get the more benefit out of synthetic testing. A service being healthy doesn’t necessarily mean it is functional. Why wait for the user to uncover this for you? Be proactive!