Skip to content

Isolate the scripting integration tests #19705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

timvandermeij
Copy link
Contributor

The commit messages contain more details about the individual changes.

Fixes a part of #19065.

By checking for the expected value directly we can shorten the code, and
it simplifies removing the dependencies between the tests in the next
commit (by having fewer places to change). Note that this follows the
same pattern as PRs mozilla#19192, mozilla#19001 and mozilla#18399 and also helps to remove
any further possibilities for intermittent failures.
…on tests

The integration tests are order-dependent because they rely on input
field state set by a previous test. This commit fixes the issue by
updating the values to match the initial state of the document, which
makes sure that we don't build upon values from previous tests while
still testing the intended logic in the individual tests like before.
To avoid being able to introduce dependencies between tests, and to
bring existing dependencies to the surface, this commit makes sure that
we close the document between tests so that we can't accidentally rely
on state set by a previous test. This prevents multiple tests from
failing if one of them fails and makes debugging easier by being able to
run each test on their own independent of other tests.

This commit, combined with the previous ones, is enough to make the
scripting integration test suite pass consistently if random mode in
Jasmine is enabled, proving that the tests are fully isolated now.
@timvandermeij timvandermeij changed the title Integration tests scripting isolate Isolate the scripting integration tests Mar 22, 2025
@timvandermeij
Copy link
Contributor Author

/botio integrationtest

@moz-tools-bot
Copy link
Collaborator

From: Bot.io (Linux m4)


Received

Command cmd_integrationtest from @timvandermeij received. Current queue size: 0

Live output at: http://54.241.84.105:8877/1165e1c46d8bf39/output.txt

@moz-tools-bot
Copy link
Collaborator

From: Bot.io (Windows)


Received

Command cmd_integrationtest from @timvandermeij received. Current queue size: 0

Live output at: http://54.193.163.58:8877/559f2761e4f25ed/output.txt

@moz-tools-bot
Copy link
Collaborator

From: Bot.io (Linux m4)


Failed

Full output at http://54.241.84.105:8877/1165e1c46d8bf39/output.txt

Total script time: 12.27 mins

  • Integration Tests: FAILED

@moz-tools-bot
Copy link
Collaborator

From: Bot.io (Windows)


Failed

Full output at http://54.193.163.58:8877/559f2761e4f25ed/output.txt

Total script time: 29.97 mins

  • Integration Tests: FAILED

Copy link
Collaborator

@Snuffleupagus Snuffleupagus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r=me, thank you.

@timvandermeij timvandermeij merged commit 72212a8 into mozilla:master Mar 22, 2025
7 checks passed
@timvandermeij timvandermeij deleted the integration-tests-scripting-isolate branch March 22, 2025 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants