JUnit 5 Test Execution Order

In JUnit 5, writing ordered tests is highly discouraged, there may be situations where test execution ordering becomes an important factor for testing the application functionality such as integration testing.

For example, we may want the integration testing for a user management module where the tests should execute in the following manner: create a new user, update the user and then delete the user.

Read More : Test ordering in JUnit 4

1. Test Methods Ordering

JUnit 5 (checked with 5.8.1) supports the following ways to order tests in the interface MethodOrderer:

• MethodName – sorts tests alphanumerically based on their names.
• DisplayName – sorts tests alphanumerically based on their display names.
• OrderAnnotation – sorts tests based on the @Order annotation. Tests with same order will be sorted arbitrarily adjacent to each other. Not ordered tests will appear at the end.
• Random – orders tests pseudo-randomly. Randomness of tests in each execution can be controlled by property junit.jupiter.execution.order.random.seed.
• Custom – orders the test with provided ordering logic.

Another ordering class Alphanumeric has been marked deprecated and will be removed in JUnit 6. So you can avoid using it.

1.1. MethodOrderer.MethodName

An example of sorting the tests based on their method names.

@TestMethodOrder(MethodOrderer.MethodName.class)
public class MethodNameOrderedTests 
{
    @Test
    void testE() {
        assertTrue(true);
    }

    @Test
    void testA() {
        assertTrue(true);
    }

    @Test
    void testD() {
        assertTrue(true);
    }

    @Test
    void testC() {
        assertTrue(true);
    }

    @Test
    void testB() {
        assertTrue(true);
    }
}

testA()
testB()
testC()
testD()
testE()

1.2. MethodOrderer.DisplayName

An example of sorting the tests based on their display names.

import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(MethodOrderer.DisplayName.class)
public class DisplayNameOrderedTests
{
    @DisplayName("5")
    @Test
    void testE() {
        assertTrue(true);
    }

    @DisplayName("3")
    @Test
    void testA() {
        assertTrue(true);
    }

    @DisplayName("1")
    @Test
    void testD() {
        assertTrue(true);
    }

    @DisplayName("2")
    @Test
    void testC() {
        assertTrue(true);
    }

    @DisplayName("4")
    @Test
    void testB() {
        assertTrue(true);
    }
}

1
2
3
4
5

1.3. MethodOrderer.OrderAnnotation

An example of sorting the tests based on their ordering as mentioned in @Order annotation.

import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class OrderAnnotationTests {
    @Order(5)
    @Test
    void testE() {
        assertTrue(true);
    }

    @Order(4)
    @Test
    void testA() {
        assertTrue(true);
    }

    @Order(3)
    @Test
    void testD() {
        assertTrue(true);
    }

    @Order(2)
    @Test
    void testC() {
        assertTrue(true);
    }

    @Order(1)
    @Test
    void testB() {
        assertTrue(true);
    }
}

testB()
testC()
testD()
testA()
testE()

1.4. MethodOrderer.Random

An example of sorting the tests randomly. By default, the random seed used for ordering methods is the System.nanoTime() which is generated during the static initialization of the test class.

We can change the randomness by providing a custom seed through property junit.jupiter.execution.order.random.seed.

junit.jupiter.execution.order.random.seed=9999

@TestMethodOrder(MethodOrderer.Random.class)
public class RandomOrderedTests {
    @Test
    void testE() {
        assertTrue(true);
    }

    @Test
    void testA() {
        assertTrue(true);
    }

    @Test
    void testD() {
        assertTrue(true);
    }

    @Test
    void testC() {
        assertTrue(true);
    }

    @Test
    void testB() {
        assertTrue(true);
    }
}

By default, the output will be different for each execution of the test class. But if we use a fixed seed using the property files, the order of the tests will be fixed and will not change between different executions.

The given output is after we have specified the custom seed 9999.

//Execution 1

testB()
testC()
testE()
testA()
testD()


//Execution 2

testB()
testC()
testE()
testA()
testD()

1.5. Custom Ordering

It is possible to define your own custom ordering by implementing the interface MethodOrderer.

In the given example, we are executing the deprecated tests in the end.

import java.util.Comparator;
import org.junit.jupiter.api.MethodDescriptor;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.MethodOrdererContext;

public class DeprecatedInEndTestOrder implements MethodOrderer {

    private Comparator<MethodDescriptor> comparator = Comparator
                .comparing(md -> md.getMethod().isAnnotationPresent(Deprecated.class));

    @Override
    public void orderMethods(MethodOrdererContext context) {

        context.getMethodDescriptors().sort(comparator);
    }
}

import static org.junit.Assert.assertTrue;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(DeprecatedInEndTestOrder.class)
public class CustomOrderTests {
    @Test
    @Deprecated
    void testC() {
        assertTrue(true);
    }

    @Test
    void testA() {
        assertTrue(true);
    }

    @Test
    void testD() {
        assertTrue(true);
    }
    @Deprecated
    @Test
    void testE() {
        assertTrue(true);
    }

    @Test
    void testB() {
        assertTrue(true);
    }
}

Notice the test output. The deprecated tests have been executed in the last.

testA()
testB()
testD()
testC()
testE()

2. Test Classes Ordering

Reiterating the fact that test classes typically should not rely on the execution order, there are times when it may be needed.

2.1. Supported Ordering Types

JUnit 5 (checked with 5.8.1) supports the following ways to order tests in the interface ClassOrderer. We can use the class order in the same way as method ordering seen in the above sections.

• ClassName – sorts test classes alphanumerically based on their fully qualified class names.
• DisplayName – sorts test classes alphanumerically based on their display names.
• OrderAnnotation – sorts test classes numerically based on values specified via the @Order annotation.
• Random – orders test classes pseudo-randomly. Supports custom seed using the property junit.jupiter.execution.order.random.seed.

To set the order globally for all test classes, use the configuration property junit.jupiter.testclass.order.default.

junit.jupiter.testclass.order.default = org.junit.jupiter.api.ClassOrderer$OrderAnnotation

2.2. Test Class Ordering Example

Given is an example for test class ordering in nested test classes.

@TestClassOrder(ClassOrderer.OrderAnnotation.class)
class OrderedTestClassesExample {

    @Nested
    @Order(1)
    class SetupTests {

        @Test
        void test1() {
        }
    }

    @Nested
    @Order(2)
    class AppFlowTests {

        @Test
        void test2() {
        }
    }
}

SetupTests
    - test1()

AppFlowTests
    - test2()

3. Conclusion

Let me iterate again that forcing an ordering on the test execution order is not recommended and should be avoided. Still, if you come across such a requirement then use above explained techniques to order the test classes and test methods.

I will recommend using @Order annotation with ClassOrderer.OrderAnnotation and MethodOrderer.OrderAnnotation to have full control over the ordering of tests if the test ordering is such important in your case.

Happy Learning !!

Download Source Code