Testing

Developing lints for Clippy is a Test-Driven Development (TDD) process because our first task before implementing any logic for a new lint is to write some test cases.

Develop Lints with Tests

When we develop Clippy, we enter a complex and chaotic realm full of programmatic issues, stylistic errors, illogical code and non-adherence to convention. Tests are the first layer of order we can leverage to define when and where we want a new lint to trigger or not.

Moreover, writing tests first help Clippy developers to find a balance for the first iteration of and further enhancements for a lint. With test cases on our side, we will not have to worry about over-engineering a lint on its first version nor missing out some obvious edge cases of the lint. This approach empowers us to iteratively enhance each lint.

Clippy UI Tests

We use UI tests for testing in Clippy. These UI tests check that the output of Clippy is exactly as we expect it to be. Each test is just a plain Rust file that contains the code we want to check.

The output of Clippy is compared against a .stderr file. Note that you don't have to create this file yourself. We'll get to generating the .stderr files with the command cargo bless (seen later on).

Write Test Cases

Let us now think about some tests for our imaginary foo_functions lint. We start by opening the test file tests/ui/foo_functions.rs that was created by cargo dev new_lint.

Update the file with some examples to get started:

#![warn(clippy::foo_functions)] // < Add this, so the lint is guaranteed to be enabled in this file

// Impl methods
struct A;
impl A {
    pub fn fo(&self) {}
    pub fn foo(&self) {} //~ ERROR: function called "foo"
    pub fn food(&self) {}
}

// Default trait methods
trait B {
    fn fo(&self) {}
    fn foo(&self) {} //~ ERROR: function called "foo"
    fn food(&self) {}
}

// Plain functions
fn fo() {}
fn foo() {} //~ ERROR: function called "foo"
fn food() {}

fn main() {
    // We also don't want to lint method calls
    foo();
    let a = A;
    a.foo();
}

Without actual lint logic to emit the lint when we see a foo function name, this test will just pass, because no lint will be emitted. However, we can now run the test with the following command:

$ TESTNAME=foo_functions cargo uitest

Clippy will compile and it will conclude with an ok for the tests:

...Clippy warnings and test outputs...
test compile_test ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.48s

This is normal. After all, we wrote a bunch of Rust code but we haven't really implemented any logic for Clippy to detect foo functions and emit a lint.

As we gradually implement our lint logic, we will keep running this UI test command. Clippy will begin outputting information that allows us to check if the output is turning into what we want it to be.

Example output

As our foo_functions lint is tested, the output would look something like this:

failures:
---- compile_test stdout ----
normalized stderr:
error: function called "foo"
  --> tests/ui/foo_functions.rs:6:12
   |
LL |     pub fn foo(&self) {}
   |            ^^^
   |
   = note: `-D clippy::foo-functions` implied by `-D warnings`
error: function called "foo"
  --> tests/ui/foo_functions.rs:13:8
   |
LL |     fn foo(&self) {}
   |        ^^^
error: function called "foo"
  --> tests/ui/foo_functions.rs:19:4
   |
LL | fn foo() {}
   |    ^^^
error: aborting due to 3 previous errors

Note the failures label at the top of the fragment, we'll get rid of it (saving this output) in the next section.

Note: You can run multiple test files by specifying a comma separated list: TESTNAME=foo_functions,bar_methods,baz_structs.

`cargo bless`

Once we are satisfied with the output, we need to run this command to generate or update the .stderr file for our lint:

$ TESTNAME=foo_functions cargo uibless

This writes the emitted lint suggestions and fixes to the .stderr file, with the reason for the lint, suggested fixes, and line numbers, etc.

Running TESTNAME=foo_functions cargo uitest should pass then. When we commit our lint, we need to commit the generated .stderr files, too.

In general, you should only commit files changed by cargo bless for the specific lint you are creating/editing.

Note: If the generated .stderr, and .fixed files are empty, they should be removed.

`toml` Tests

Some lints can be configured through a clippy.toml file. Those configuration values are tested in tests/ui-toml.

To add a new test there, create a new directory and add the files:

clippy.toml: Put here the configuration value you want to test.
lint_name.rs: A test file where you put the testing code, that should see a different lint behavior according to the configuration set in the clippy.toml file.

The potential .stderr and .fixed files can again be generated with cargo bless.

Cargo Lints

The process of testing is different for Cargo lints in that now we are interested in the Cargo.toml manifest file. In this case, we also need a minimal crate associated with that manifest. Those tests are generated in tests/ui-cargo.

Imagine we have a new example lint that is named foo_categories, we can run:

$ cargo dev new_lint --name=foo_categories --pass=late --category=cargo

After running cargo dev new_lint we will find by default two new crates, each with its manifest file:

tests/ui-cargo/foo_categories/fail/Cargo.toml: this file should cause the new lint to raise an error.
tests/ui-cargo/foo_categories/pass/Cargo.toml: this file should not trigger the lint.

If you need more cases, you can copy one of those crates (under foo_categories) and rename it.

The process of generating the .stderr file is the same as for other lints and prepending the TESTNAME variable to cargo uitest works for Cargo lints too.

Rustfix Tests

If the lint you are working on is making use of structured suggestions, rustfix will apply the suggestions from the lint to the test file code and compare that to the contents of a .fixed file.

Structured suggestions tell a user how to fix or re-write certain code that has been linted with span_lint_and_sugg.

Should span_lint_and_sugg be used to generate a suggestion, but not all suggestions lead to valid code, you can use the //@no-rustfix comment on top of the test file, to not run rustfix on that file.

We'll talk about suggestions more in depth in a later chapter.

Use cargo bless to automatically generate the .fixed file after running the tests.

Testing Manually

Manually testing against an example file can be useful if you have added some println!s and the test suite output becomes unreadable.

To try Clippy with your local modifications, run from the working copy root.

$ cargo dev lint input.rs

Clippy Documentation