Skip to content

Quick Start

From zero to a running test in four steps: run the example, copy the template, write a wrapper and a test, read the results.

Requirements: JDK 25 and either Gradle (the template ships a 9.8.0 wrapper, so just ./gradlew; 9.1+ works) or Maven 3.9+. Ares 2.2.1, JUnit 6, AssertJ and Byte Buddy come in through the example's build file.

Step 1: Run the example

git clone https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework.git
cd Levenshtein-Testing-Framework/example-gradle
./gradlew test   # the example exercise under Ares 2

Inside the repository the example compiles the framework source of the checkout. ./mvnw -B verify in the repository root builds the framework and the Maven example the same way.

To see the framework's messages, run the same tests against the failing example solution in assignment-failing/src (./mvnw -B verify -Dassignment.dir=assignment-failing for Maven). It fails on purpose, with deviating types, modifiers, parameters and inheritance, some of them combined (all defects: assignment-failing/README.md):

./gradlew test -PassignmentDir=assignment-failing
⚠️ ABWEICHUNG in AbstractVehicle ⚠️
Falls möglich wird der tatsächliche Wert für weitere Tests verwendet.
Erwartet:   public int getYear()
Tatsächlich:    protected long getYear()

❌ FEHLT ❌ im Paket io.github.valentinherrmann.example.vehicles
Erwartet:   public class Car extends AbstractVehicle implements Driveable
Tatsächlich:    public class Car extends AbstractVehicle

Methode:     setPrice(double)
Szenario:    Setter -> Attribut, Attribut price
Eingabe:     setPrice(-1.25)
Erwartet:    Wert abgelehnt (unverändert 2.50 oder Ausnahme)
Tatsächlich: -1.25
(2 von 15 Fällen fehlgeschlagen)

The messages are German by default; LevenshteinSettings.setLanguage(LevenshteinSettings.Language.ENGLISH) switches them to English.

The repository contains three modules:

Module Content
framework/ The published library io.github.valentinherrmann:levenshtein-testing-framework (built with Maven)
example-gradle/ The example exercise as a standalone Gradle project: your exam template
example-maven/ The same exercise as a standalone Maven project: your exam template

Both examples use the Artemis layout: assignment/src = student code, test/ = instructor tests and policy (assignment-failing/src = the failing example solution).

Step 2: Create your exam repository from the template

Copy the whole example-gradle/ (or example-maven/) folder as your exam repository. It runs as it is, so you can check your machine first with ./gradlew test (./mvnw -B verify).

In this walkthrough the student package is org.example.exam and the test package is org.example.tests. The student package must not be a prefix of the test package (org.example.exam.tests would be wrong). Then adapt:

Where What to change
assignment/src/, test/io/, test/sandbox/ Delete the example's student code, tests and sandbox file, including the self-checks SecurityControlTest, TimeoutControlTest and SandboxControl.
Dependencies: build.gradle / pom.xml Remove junit-platform-testkit: only TimeoutControlTest needs it. Add only libraries your course allows.
Reserved-package guard: build.gradle → verifyAresReservedPackages, pom.xml → antrun verify-ares-reserved-packages-v2 Replace io/github/valentinherrmann/example/tests/** with your test package, e.g. org/example/tests/**.
test/SecurityPolicy.yaml theSupervisedCodeUsesTheFollowingPackage: "org.example.exam", theMainClassInsideThisPackageIs, the list theFollowingClassesAreTestClasses (the exact names of all your test and wrapper classes), and regardingFileSystemInteractions: [ ]. Keep the programming language configuration.
Your test classes @Policy(value = "test/SecurityPolicy.yaml", withinPath = "classes/org/example/exam")

The copy takes the newest framework release from Maven Central (levenshteinVersion = latest.release in build.gradle, levenshtein.version = [2000.0.0,) in pom.xml, release candidates included), so it builds on Artemis as it is. Set a fixed version there to keep an exam on one release.

[!IMPORTANT] The reasons for every setting, for both build tools, are in Ares 2 Setup in Detail. Do not remove the reserved-package guard or the weaving: without them the sandbox is silently inactive or can be replaced by the student.

Step 3: Write your first wrapper and test

A wrapper describes one class you expect from the student. The framework finds the student's class and members, even with small typos, and the tests work through the wrapper.

Student code: assignment/src/org/example/exam/Car.java (the expected solution)
package org.example.exam;

public class Car {
    private double price;
    private double speed;

    public Car(double price) {
        this.price = price;
    }

    public void start() {
        speed = 10.0;
    }

    public double getPrice() {
        return price;
    }
}

Wrapper: test/org/example/tests/CarWrapper.java

package org.example.tests;

import io.github.valentinherrmann.levenshtein.*;

/** Expected: public class Car in package org.example.exam. */
public class CarWrapper<T> extends ClassWrapper<T> {
    // One field per expected member (found via reflection).
    private final AttributeWrapper<T, Double> price;
    private final AttributeWrapper<T, Double> speed;
    private final ConstructorWrapper<T> constructor;
    private final MethodWrapper<T, Void> start;
    private final MethodWrapper<T, Double> getPrice;

    public CarWrapper() {
        super("Car", "org.example.exam", "public");
        price = new AttributeWrapper<>(this, "price", double.class, "private");
        speed = new AttributeWrapper<>(this, "speed", double.class, "private");
        constructor = new ConstructorWrapper<>(this,
                new Class<?>[]{double.class}, "public");
        start = new MethodWrapper<>(this, "start", void.class, "public");
        getPrice = new MethodWrapper<>(this, "getPrice", double.class,
                "public");
    }

    // Accessors for the behavioural tests
    public AttributeWrapper<T, Double> price() { return price; }
    public AttributeWrapper<T, Double> speed() { return speed; }
    public MethodWrapper<T, Void> start() { return start; }
    public MethodWrapper<T, Double> getPrice() { return getPrice; }

    // The default instance used by newObj(), getObj() and invoke()
    @Override
    public Object getObj(boolean forceNew) {
        return getObj(forceNew, constructor, 30000.0);
    }
}

Test: test/org/example/tests/CarTest.java

package org.example.tests;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.List;

import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestFactory;

import de.tum.cit.ase.ares.api.Policy;
import io.github.valentinherrmann.levenshtein.LevenshteinSettings;
import io.github.valentinherrmann.levenshtein.LevenshteinTest;
import io.github.valentinherrmann.levenshtein.StructuralLevenshtein;
import io.github.valentinherrmann.levenshtein.StructuralLevenshtein.DetailLevel;

@LevenshteinTest
@Policy(value = "test/SecurityPolicy.yaml",
        withinPath = "classes/org/example/exam")
class CarTest {

    static {
        // feedback language, default: DEUTSCH
        LevenshteinSettings.setLanguage(LevenshteinSettings.Language.ENGLISH);
    }

    static final CarWrapper<?> car = new CarWrapper<>();

    // Structure: Class[Car], Constructors[Car], Attributes[Car], Methods[Car]
    @TestFactory
    List<DynamicTest> structure() {
        return StructuralLevenshtein.structuralTestFactory(
                DetailLevel.ONE_PER_MEMBER_CATEGORY, car);
    }

    // Behaviour: still runs if the student wrote "prize" instead of "price"
    @Test
    void startSetsSpeed() {
        Object c = car.newObj();
        car.start().invokeOnSpecificObject(c);
        assertThat(car.speed().getValue(c)).isEqualTo(10.0);
    }

    @Test
    void getPriceReturnsPrice() {
        car.testGettersSetters(car.price(), car.getPrice());
    }
}

The matching SecurityPolicy.yaml lists the two test classes:

  theSupervisedCodeUsesTheFollowingPackage: "org.example.exam"
  theMainClassInsideThisPackageIs: "Car"
  theFollowingClassesAreTestClasses:
    - "org.example.tests.CarTest"
    - "org.example.tests.CarWrapper"

Step 4: Run it and read the results

./gradlew test        # or, with Maven: ./mvnw -B verify

With the solution above, all 6 tests pass. Every expected element ends up in one of three states:

State Example Structural test Behavioural tests
EXACT ✅ price passes run
DEVIATES ⚠️ prize for price (≤ 20 % of the name may differ), float for double, an extra static fails still run on prize
MISSING ❌ strat() for start() (a swap is 2 edits = 40 %) fails fail: "... is not implemented as expected"

Behavioural failures (a wrong result, an exception of the student's code) show the method, the input, the expected and the actual value, as in the sample output of step 1. The exact matching rules are in How Matching Works.

That's it. Next: describe more classes (Writing Tests), keep tests independent of other student members (Independent Tests and Mocks), and look at the full example in example-maven/test, which covers an interface, an abstract class, inheritance, overloading and Exercise Variants.