docs: parallelism is cheaper than sharding#39128
docs: parallelism is cheaper than sharding#39128Skn0tt wants to merge 8 commits intomicrosoft:mainfrom
Conversation
docs/src/ci.md
Outdated
|
|
||
| We recommend setting [workers](./api/class-testconfig.md#test-config-workers) to "1" in CI environments to prioritize stability and reproducibility. Running tests sequentially ensures each test gets the full system resources, avoiding potential conflicts. However, if you have a powerful self-hosted CI system, you may enable [parallel](./test-parallel.md) tests. For wider parallelization, consider [sharding](./test-parallel.md#shard-tests-between-multiple-machines) - distributing tests across multiple CI jobs. | ||
|
|
||
| If you have a powerful CI environment available, we recommend enabling [parallel](./test-parallel.md) tests and setting [workers](./api/class-testconfig.md#test-config-workers) to `'50%'` in CI environments. |
There was a problem hiding this comment.
50% is the default, let's mention that.
docs/src/test-sharding-js.md
Outdated
|
|
||
| :::info[Before sharding] | ||
| Sharding adds per-shard overhead like checking out the codebase, installing dependencies, and downloading browsers. | ||
| If your test suite can run in parallel without becoming flaky, consider scaling up the machine (more CPU / memory) and increasing the number of [workers](./api/class-testconfig.md#test-config-workers) before introducing sharding. |
There was a problem hiding this comment.
| If your test suite can run in parallel without becoming flaky, consider scaling up the machine (more CPU / memory) and increasing the number of [workers](./api/class-testconfig.md#test-config-workers) before introducing sharding. | |
| Consider scaling up the machine (more CPU / memory) while leaving [workers](./api/class-testconfig.md#test-config-workers) at `50%` before introducing sharding. |
There was a problem hiding this comment.
Before introducing shards, consider scaling up the machine (more CPU and memory). Parallelism is usually constrained by the number of CPU cores.? why do we want to mention 50% at all if it's the default that they are not expected to change?
There was a problem hiding this comment.
Our defaults aren't clear-cut since create-playwright generates workers: CI ? 1 : undefined and fullyParallel: true. Our docs are a little unclear on this.
docs/src/test-sharding-js.md
Outdated
|
|
||
| :::info[Before sharding] | ||
| Sharding adds per-shard overhead like checking out the codebase, installing dependencies, and downloading browsers. | ||
| If your test suite can run in parallel without becoming flaky, consider scaling up the machine (more CPU / memory) and increasing the number of [workers](./api/class-testconfig.md#test-config-workers) before introducing sharding. |
There was a problem hiding this comment.
Before introducing shards, consider scaling up the machine (more CPU and memory). Parallelism is usually constrained by the number of CPU cores.? why do we want to mention 50% at all if it's the default that they are not expected to change?
|
I addressed your feedback and reworked the PR based on what we discussed yesterday. The Sharding doc leads with a paragraph about how sharding might not be necessary now. (I didn't end up renaming it, it looked off after rendering. Most of the doc is still about sharding). PTAL! |
docs/src/ci.md
Outdated
| We recommend setting [workers](./api/class-testconfig.md#test-config-workers) to "1" in CI environments to prioritize stability and reproducibility. Running tests sequentially ensures each test gets the full system resources, avoiding potential conflicts. However, if you have a powerful self-hosted CI system, you may enable [parallel](./test-parallel.md) tests. For wider parallelization, consider [sharding](./test-parallel.md#shard-tests-between-multiple-machines) - distributing tests across multiple CI jobs. | ||
| If you have a powerful CI environment available, we recommend enabling [parallel](./test-parallel.md) tests and removing the limit of [`workers: 1`](./api/class-testconfig.md#test-config-workers) in CI environments, which by default uses 50% of available CPU cores to speed up your test runs. | ||
|
|
||
| In some test suites, this can lead to instability and flakiness due to hidden race conditions and shared state. |
There was a problem hiding this comment.
Split this into 2 separate paragraphs - one about sharding across machine and the other about fighting interdependent tests (move it to the end?)
| To speed up your test runs, often increasing the number of available CPU cores in your CI environment and [not limiting workers](./ci.md#workers) is the best way. | ||
|
|
||
| For some test suites, this can lead to instability and flakiness due to hidden race conditions and shared state. | ||
| If you face this issue and can't resolve it, you can instead run tests on multiple machines simultaneously. |
There was a problem hiding this comment.
I'd not recommend shards as a solution to flakiness. This creates an impression that sharding is a tool for reducing flakiness rather than improving run time.
There was a problem hiding this comment.
The point I want to get across is that sharding is a tool for improving runtime that, compared to parallelism, is less susceptible to introducing flakiness. I think it's an important distinction, but I agree it's hard to express this without creating the wrong impression 🤔
Co-authored-by: Yury Semikhatsky <yurys@chromium.org> Signed-off-by: Simon Knott <info@simonknott.de>
Co-authored-by: Yury Semikhatsky <yurys@chromium.org> Signed-off-by: Simon Knott <info@simonknott.de>
3 participants
No description provided.