-
Notifications
You must be signed in to change notification settings - Fork 10.3k
Isolate the scripting integration tests #19705
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Isolate the scripting integration tests #19705
Conversation
By checking for the expected value directly we can shorten the code, and it simplifies removing the dependencies between the tests in the next commit (by having fewer places to change). Note that this follows the same pattern as PRs mozilla#19192, mozilla#19001 and mozilla#18399 and also helps to remove any further possibilities for intermittent failures.
…on tests The integration tests are order-dependent because they rely on input field state set by a previous test. This commit fixes the issue by updating the values to match the initial state of the document, which makes sure that we don't build upon values from previous tests while still testing the intended logic in the individual tests like before.
To avoid being able to introduce dependencies between tests, and to bring existing dependencies to the surface, this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test. This prevents multiple tests from failing if one of them fails and makes debugging easier by being able to run each test on their own independent of other tests. This commit, combined with the previous ones, is enough to make the scripting integration test suite pass consistently if random mode in Jasmine is enabled, proving that the tests are fully isolated now.
/botio integrationtest |
From: Bot.io (Linux m4)ReceivedCommand cmd_integrationtest from @timvandermeij received. Current queue size: 0 Live output at: http://54.241.84.105:8877/1165e1c46d8bf39/output.txt |
From: Bot.io (Windows)ReceivedCommand cmd_integrationtest from @timvandermeij received. Current queue size: 0 Live output at: http://54.193.163.58:8877/559f2761e4f25ed/output.txt |
From: Bot.io (Linux m4)FailedFull output at http://54.241.84.105:8877/1165e1c46d8bf39/output.txt Total script time: 12.27 mins
|
From: Bot.io (Windows)FailedFull output at http://54.193.163.58:8877/559f2761e4f25ed/output.txt Total script time: 29.97 mins
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
r=me, thank you.
The commit messages contain more details about the individual changes.
Fixes a part of #19065.