Cypress was our go-to end-to-end (E2E) testing tool, and we were pretty happy with it, up until recently that is. Lately, we've run into a couple of testing scenarios where Cypress support has been limited, notably around multiple domains/tabs and iFrames. This caused us to re-evaluate the available E2E tools and it ultimately came down to two options; keep Cypress or switch to Playwright.
In this article, I compare both tools in the format of an ADR, which should help us decide which tool to go with. Let the battle commence!
Cypress
Good, because it's very easy to learn and get set up with basic tests.
Good, because it has a nice dashboard to view test reports, analytics and recordings.
Good, because it supports Chromium and Firefox.
Good, because it has a very slick Test Runner UI.
Good, because it's built specifically for end-to-end testing.
Good, because you can edit your test code in the browser and instantly see it run as you change the code.
Good, because it's mature and has good community support.
Bad, because it doesn't support multiple domains.
Bad, because authentication requires a lot more setup due to lack of multi-domain support.
Bad, because it doesn't support Webkit (Safari)
Bad, because you cannot run tests against multiple browsers at the same time.
Bad, because iFrame support is limited.
Bad, because there is no "hover" support.
Bad, because the chaining command syntax can quickly get out of hand for more complex tests.
Bad, because you have to pay a premium to get access to some dashboard features (e.g flake detection)
Bad, because to do parallelization well, it requires vendor-locked software.
Playwright
Good, because it supports Chromium, Firefox and Webkit (Safari).
Good, because it supports multiple domains and tabs.
Good, because it supports 5 language bindings (Javascript, Typescript, Java, Python, .NET)
Good, because it's fast.
Good, because you can run tests against multiple browsers at the same time.
Good, because it fully supports parallelization, even locally.
Good, because it supports parallel tests within a single test file.
Good, because it's Javascript first, so feels more natural.
Good, because it has "hover" support.
Good, because iFrames are natively supported.
Good, because it supports reuse of authentication state to speed up tests.
Good, because it lets you choose your test runner (e.g. Jest but the default one is advised)
Good, because signing in is simple, you just fill in the form.
Good, because it's completely free.
Good, because it has few dependencies.
Bad, because it's still quite new, so possibility of a smaller community.
Bad, because 3rd party tutorials are out of date due to changing API.
Bad, because it's a mix between an automation and testing framework.
Bad, because it has a steeper learning curve.
Bad, because it does not have a dedicated dashboard so would be harder to debug tests remotely.
Common features between both
Good documentation
API testing
Point & click test recording
Test debugging tools
Test retries
Automatic waiting
Video and screen capture
Mobile emulation
Regularly updated and well maintained
Run only a subset of tests
Network monitoring
Conclusion
In the end, we decided to go with Playwright, mainly because of its native support for multiple domains, tabs and iFrames. I will say that I found Cypress' debugging to be more developer-friendly and in general "slicker", but that wasn't enough to make us stay put.
Want to see more?
I mainly write about real tech topics I face in my everyday life as a Frontend Developer. If this appeals to you then feel free to follow me on BlueSky. 🦋
Bye for now 👋