Quick Start¶
From zero to a running test in four steps: run the example, copy the template, write a wrapper and a test, read the results.
Requirements: JDK 25 and either Gradle (the template ships a 9.8.0 wrapper, so just ./gradlew;
9.1+ works) or Maven 3.9+. Ares 2.2.1, JUnit 6, AssertJ and Byte Buddy come in through the example's build file.
Step 1: Run the example¶
git clone https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework.git
cd Levenshtein-Testing-Framework/example-gradle
./gradlew test # the example exercise under Ares 2
Inside the repository the example compiles the framework source of the checkout. ./mvnw -B verify in the
repository root builds the framework and the Maven example the same way.
To see the framework's messages, run the same tests against the failing example solution in
assignment-failing/src (./mvnw -B verify -Dassignment.dir=assignment-failing for Maven). It fails on purpose,
with deviating types, modifiers, parameters and inheritance, some of them combined (all defects:
assignment-failing/README.md):
⚠️ ABWEICHUNG in AbstractVehicle ⚠️
Falls möglich wird der tatsächliche Wert für weitere Tests verwendet.
Erwartet: public int getYear()
Tatsächlich: protected long getYear()
❌ FEHLT ❌ im Paket io.github.valentinherrmann.example.vehicles
Erwartet: public class Car extends AbstractVehicle implements Driveable
Tatsächlich: public class Car extends AbstractVehicle
Methode: setPrice(double)
Szenario: Setter -> Attribut, Attribut price
Eingabe: setPrice(-1.25)
Erwartet: Wert abgelehnt (unverändert 2.50 oder Ausnahme)
Tatsächlich: -1.25
(2 von 15 Fällen fehlgeschlagen)
The messages are German by default; LevenshteinSettings.setLanguage(LevenshteinSettings.Language.ENGLISH) switches
them to English.
The repository contains three modules:
| Module | Content |
|---|---|
framework/ |
The published library io.github.valentinherrmann:levenshtein-testing-framework (built with Maven) |
example-gradle/ |
The example exercise as a standalone Gradle project: your exam template |
example-maven/ |
The same exercise as a standalone Maven project: your exam template |
Both examples use the Artemis layout: assignment/src = student code, test/ = instructor tests and policy
(assignment-failing/src = the failing example solution).
Step 2: Create your exam repository from the template¶
Copy the whole example-gradle/ (or example-maven/) folder as
your exam repository. It runs as it is, so you can check your machine first with ./gradlew test
(./mvnw -B verify).
In this walkthrough the student package is org.example.exam and the test package is org.example.tests.
The student package must not be a prefix of the test package (org.example.exam.tests would be wrong).
Then adapt:
| Where | What to change |
|---|---|
assignment/src/, test/io/, test/sandbox/ |
Delete the example's student code, tests and sandbox file, including the self-checks SecurityControlTest, TimeoutControlTest and SandboxControl. |
Dependencies: build.gradle / pom.xml |
Remove junit-platform-testkit: only TimeoutControlTest needs it. Add only libraries your course allows. |
Reserved-package guard: build.gradle → verifyAresReservedPackages, pom.xml → antrun verify-ares-reserved-packages-v2 |
Replace io/github/valentinherrmann/example/tests/** with your test package, e.g. org/example/tests/**. |
test/SecurityPolicy.yaml |
theSupervisedCodeUsesTheFollowingPackage: "org.example.exam", theMainClassInsideThisPackageIs, the list theFollowingClassesAreTestClasses (the exact names of all your test and wrapper classes), and regardingFileSystemInteractions: [ ]. Keep the programming language configuration. |
| Your test classes | @Policy(value = "test/SecurityPolicy.yaml", withinPath = "classes/org/example/exam") |
The copy takes the newest framework release from Maven Central (levenshteinVersion = latest.release in
build.gradle, levenshtein.version = [2000.0.0,) in pom.xml, release candidates included), so it builds on
Artemis as it is. Set a fixed version there to keep an exam on one release.
[!IMPORTANT] The reasons for every setting, for both build tools, are in Ares 2 Setup in Detail. Do not remove the reserved-package guard or the weaving: without them the sandbox is silently inactive or can be replaced by the student.
Step 3: Write your first wrapper and test¶
A wrapper describes one class you expect from the student. The framework finds the student's class and members, even with small typos, and the tests work through the wrapper.
Student code: assignment/src/org/example/exam/Car.java (the expected solution)
Wrapper: test/org/example/tests/CarWrapper.java
package org.example.tests;
import io.github.valentinherrmann.levenshtein.*;
/** Expected: public class Car in package org.example.exam. */
public class CarWrapper<T> extends ClassWrapper<T> {
// One field per expected member (found via reflection).
private final AttributeWrapper<T, Double> price;
private final AttributeWrapper<T, Double> speed;
private final ConstructorWrapper<T> constructor;
private final MethodWrapper<T, Void> start;
private final MethodWrapper<T, Double> getPrice;
public CarWrapper() {
super("Car", "org.example.exam", "public");
price = new AttributeWrapper<>(this, "price", double.class, "private");
speed = new AttributeWrapper<>(this, "speed", double.class, "private");
constructor = new ConstructorWrapper<>(this,
new Class<?>[]{double.class}, "public");
start = new MethodWrapper<>(this, "start", void.class, "public");
getPrice = new MethodWrapper<>(this, "getPrice", double.class,
"public");
}
// Accessors for the behavioural tests
public AttributeWrapper<T, Double> price() { return price; }
public AttributeWrapper<T, Double> speed() { return speed; }
public MethodWrapper<T, Void> start() { return start; }
public MethodWrapper<T, Double> getPrice() { return getPrice; }
// The default instance used by newObj(), getObj() and invoke()
@Override
public Object getObj(boolean forceNew) {
return getObj(forceNew, constructor, 30000.0);
}
}
Test: test/org/example/tests/CarTest.java
package org.example.tests;
import static org.assertj.core.api.Assertions.assertThat;
import java.util.List;
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestFactory;
import de.tum.cit.ase.ares.api.Policy;
import io.github.valentinherrmann.levenshtein.LevenshteinSettings;
import io.github.valentinherrmann.levenshtein.LevenshteinTest;
import io.github.valentinherrmann.levenshtein.StructuralLevenshtein;
import io.github.valentinherrmann.levenshtein.StructuralLevenshtein.DetailLevel;
@LevenshteinTest
@Policy(value = "test/SecurityPolicy.yaml",
withinPath = "classes/org/example/exam")
class CarTest {
static {
// feedback language, default: DEUTSCH
LevenshteinSettings.setLanguage(LevenshteinSettings.Language.ENGLISH);
}
static final CarWrapper<?> car = new CarWrapper<>();
// Structure: Class[Car], Constructors[Car], Attributes[Car], Methods[Car]
@TestFactory
List<DynamicTest> structure() {
return StructuralLevenshtein.structuralTestFactory(
DetailLevel.ONE_PER_MEMBER_CATEGORY, car);
}
// Behaviour: still runs if the student wrote "prize" instead of "price"
@Test
void startSetsSpeed() {
Object c = car.newObj();
car.start().invokeOnSpecificObject(c);
assertThat(car.speed().getValue(c)).isEqualTo(10.0);
}
@Test
void getPriceReturnsPrice() {
car.testGettersSetters(car.price(), car.getPrice());
}
}
The matching SecurityPolicy.yaml lists the two test classes:
theSupervisedCodeUsesTheFollowingPackage: "org.example.exam"
theMainClassInsideThisPackageIs: "Car"
theFollowingClassesAreTestClasses:
- "org.example.tests.CarTest"
- "org.example.tests.CarWrapper"
Step 4: Run it and read the results¶
With the solution above, all 6 tests pass. Every expected element ends up in one of three states:
| State | Example | Structural test | Behavioural tests |
|---|---|---|---|
EXACT ✅ |
price |
passes | run |
DEVIATES ⚠️ |
prize for price (≤ 20 % of the name may differ), float for double, an extra static |
fails | still run on prize |
MISSING ❌ |
strat() for start() (a swap is 2 edits = 40 %) |
fails | fail: "... is not implemented as expected" |
Behavioural failures (a wrong result, an exception of the student's code) show the method, the input, the expected and the actual value, as in the sample output of step 1. The exact matching rules are in How Matching Works.
That's it. Next: describe more classes (Writing Tests), keep tests independent of other
student members (Independent Tests and Mocks), and look at the full example in
example-maven/test, which covers an interface, an
abstract class, inheritance, overloading and Exercise Variants.