State Management Improvements

The Decision

Best will cache the results of each test to a temporary file using mktemp and then print the results at the end.

The Context

Printing the results of each individual test to the terminal is slow. As the number of tests grows, the time it takes Best to run grows linearly.

Trade-offs

• Speed: Frequent terminal output is slow. Temporary file system writes are much faster.
• Memory: The memory difference should be negligable.
• Security: Our assumption is that mktemp has permissions to write files to /tmp/.
• Availability: mktemp should be available on a wide variety of systems, but if we receive user feedback that it's not, then we may need to revisit this mechanism.
• Dependencies: We assume that mktemp is available at runtime.

Alternatives Considered

• Printing each test result to the terminal (this was used previously, but proved to be much slower)
• Storing test results in memory (this hasn't been fully tested and benchmarked yet)

The Consequences

Caching test results until the end and then printing all results at once results in a 59% performance improvement on average, with no observed downsides.

Reevaluate the decision if storing test results in memory proves to be faster or mktemp is discovered to not be as widely available as thought.