Customizing test scripts
The test recorder is the easiest way to create test scripts, but it is not the only way. You can create and edit test scripts manually.
The final product would look something like:
# File: ./tests/testthat/test-shinytest2.R
test_that("App initialization is consistent", {
app <- AppDriver$new()
app$expect_values()
})A test script has this basic structure: first, there is an initialization, then the expectations are performed.
In the initialization, the script creates a new
AppDriver object.
Next, define some interactions with the application and makes expectations via snapshots.
app$set_inputs(check_group = c("1", "2"))
app$set_inputs(check_group = c("1", "2", "3"))
app$click("action")
app$expect_values()
app$click("action")
app$expect_values()For customizing a script, the second portion – the interactions and expectations – is the part you will want to modify.
Setting inputs with app$set_inputs() and
app$click()
With app$set_inputs(), you provide the name of one or
more inputs and corresponding values to set them to. Consider this set
of directives:
app$set_inputs(check_group = c("1", "2"))
app$set_inputs(check_group = c("1", "2", "3"))
app$click("action")Notice that we set the value of check_group two times in
a row. When we recorded this test script, it started with the value
"1", and then we checked the "2" and
"3" boxes. The recorded script set the value to
c("1", "2"), and then c("1", "2", "3"). The
c("1", "2") value was simply an intermediate step.
It is possible to simplify and speed up the tests by dropping the intermediate step, which leaves us with this:
Multiple calls to app$set_inputs() should be reduced to
a single line unless the side effects of setting inputs multiple times
is desired. When app$set_inputs() is called, it normally
returns control and moves on to the next step only after the server
sends a response to the client. The reason it waits for a response is so
that a subsequent call to app$snapshot() will be sure to
capture the updated output values. If app$set_inputs() did
not wait for a update, then, if the output update did not happen very
quickly, a snapshot might capture the state of the application before
the outputs are updated.
Making expectations
For an in-depth discussion of expectations, see the Robust testing vignette.
There are a couple standard ways to make expectations from your
AppDriver object.
app$expect_values(): This is the simplest way to make an expectation. It creates a list of values and compares them to the current values of the application. It also takes a debug screenshot and stores it as a snapshot. However, this snapshot will never fail an expectation test. The extra screenshot file is useful for manually determining if UI changes have happened over time while not producing numerous false-positive test failures.app$expect_screenshot(): This method takes a screenshot of the application and compares it to a reference screenshot. If the screenshots are different, it fails the test.
There are few ways to use app$expect_values(). The
simplest way is to call it with no arguments:
The first time this is called in a test script, it will record all
input, output, and export values
from the application, in a file called 001.json. The next
call will save the values in 002.json, and so on.
Each time you call app$expect_values(), it will also
save a debug screenshot of the web browser, in a file
001_.png, 002_.png, and so on. These screen
shots are useful for debugging your tests and inspecting what they’re
doing. You can tell it to not take screen shots, to save space and make
the tests run slightly faster, in the initialization step, with:
If you want to disable screenshots for a single
app$expect_values() call, you can use:
If you want more targeted tests, you can snapshot specific items by
using the input, output, and
export parameters. For example, to capture the value of
just the outputs named "a" and "b", you would
call:
You could also capture specific inputs or exports:
Finally, if you want to snapshot all outputs but no
inputs or exports, you can simply set output to
TRUE as only values that are requested are recorded:
The same can be used to snapshot all input and/or all
export values. To capture all output and
export values, but no input values:
Exported values
In some cases, it’s useful to snapshot some bits of internal state of an application – state that’s not reflected directly in the inputs or outputs. This can be done by exporting values.
Consider this toy example where an application has an internal reactive expression that would be convenient to capture.
shinyApp(
fluidPage(
numericInput("x", "x", 4),
numericInput("y", "y", 10),
numericInput("z", "z", 100),
verbatimTextOutput("result", placeholder = TRUE)
),
function(input, output, session) {
xy <- reactive(input$x * input$y)
yz <- reactive(input$y + input$z)
output$result <- renderText({
xy() / yz()
})
exportTestValues(
xy = {
xy()
},
yz = {
yz()
}
)
}
)
Notice these lines:
This creates exported values called xy and
yz. When a test snapshot is taken, it evaluates the
reactive expression and inserts their values in the snapshot:
{
"input": {
"x": 4,
"y": 10,
"z": 100
},
"output": {
"result": "0.3636364"
},
"export": {
"xy": 40,
"yz": 110
}
}Note that the expression (i.e. xy() and
yz()) is saved when exportTestValues() is
called, not the specific values. Generally, you should keep these
expressions as simple as possible so that test failures yield
informative outputs. Note that any value here has to be serializable to
JSON, so it’s not suitable for testing complex objects.
Adding delays
In some cases, you may need to wait for some amount of time between
steps (e.g. waiting for CSS animations). You can do this by adding
Sys.sleep() in your script. For example:
However, if you are waiting for some Shiny reactives to be
calculated, please read more in the Waiting
for an output (or input) value
section.
Controlling random values
If your application uses randomly-generated values (by calling
functions like rnorm, runif,
sample directly or indirectly), then, in normal operation,
it will produce different results on each run. Because
{shinytest2} works by comparing the current state of an
application to a previous state, these random changes will cause test
failures.
To make such applications exactly repeatable, you can set the random seed. This can be done by specifying a random seed in the recorder. The seed can be any integer value.
If your test script has already been created, you can set the random
seed by editing the test script so that the AppDriver$new()
call has a value for seed. Moreover, if you need these
tests to pass on multiple versions of R with different
RNGkind() defaults (e.g., 3.5 and 3.6), you may want to set
a fixed RNGkind() across your tests (perhaps via
RNGversion("3.5.0")). However, using a
AppDriver$new(variant=) value that contains the R version
can also remedy this situation.
In many cases, instead of setting the random seed, it is more elegant to use an output preprocessor.