# Levenshtein Testing Framework

> Structural and behavioural tests for student code that tolerate small naming deviations, built on Ares 2.

Structural and behavioural tests for Java student code that tolerate small naming deviations (typos, case
slips, int vs. Integer), built on Ares 2 for exams and exercises on Artemis. Wrappers (ClassWrapper,
AttributeWrapper, MethodWrapper, ConstructorWrapper) describe the expected classes; the framework finds the
student's elements with Levenshtein distance and grades them EXACT, DEVIATES or MISSING. The Javadoc is at
https://valentinherrmann.github.io/Levenshtein-Testing-Framework/api/.

# Getting started

# Levenshtein Testing Framework

Structural and behavioural tests for student code that **tolerate small naming deviations** (typos, case slips, `int` vs. `Integer`, ...) while still verifying the structure, built on [Ares 2](<https://github.com/ls1intum/Ares2>) for exams and exercises on Artemis. Exact name matching fails correct solutions because of a typo; this framework stays forgiving about names and strict about the rest.

A student who writes `prize` instead of `price` loses the structural point for that attribute, but every behavioural test still runs against their `prize`. They are not punished twice for one typo. What the student sees:

```text
⚠️ DEVIATION in Car ⚠️
If possible actual will be used for further testing.
Expect: private double price
Actual: private double prize

❌ MISSING in Car ❌
Expect: public void start()
```

## Key features

- **Fuzzy name matching** for classes, methods and attributes; the closest candidate wins
- **Structural verification** of modifiers, types, constructors, superclasses and interfaces; every missing or unexpected modifier is reported, harmless ones can be marked optional (`"final?"`)
- **Behavioural testing** through wrappers that call the student's (possibly misspelled) members
- **One failure format**: every failure in student code, and your own checks via `Failure`, shows Method, Scenario, Input, Expected and Actual, without stack traces
- **Getter/setter tests in one line**: `testGettersSetters` generates values, checks setter conditions and compares arrays by content
- **Independent tests**: objects without running a student constructor, test-owned method overrides and call checks (`mock()`)
- **Student classes as member types**: attributes, return and parameter types may be other student classes, found with the same tolerance
- **Abstract classes and interfaces** instantiated through Byte Buddy subclasses
- **Exercise Variants compatible**: names and types are plain strings and classes
- **Secure by default**: `@LevenshteinTest` runs every test under Ares 2 supervision and resets static state between tests

A failing behavioural test reads like this:

```text
Method:   setPrice(double)
Scenario: setter -> attribute, attribute price
Input:    setPrice(-1.25)
Expected: value rejected (unchanged 2.50 or exception)
Actual:   -1.25
(2 of 15 cases failed)
```

## Get started

You need JDK 25 and Gradle 9.1+ or Maven 3.9+ (details in the [Quick Start](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/quick-start/index.md>)).

```bash
git clone https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework.git
cd Levenshtein-Testing-Framework/example-gradle
./gradlew test   # the example exercise under Ares 2, newest framework release from Maven Central
```

Then follow the **[Quick Start](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/quick-start/index.md>)**: copy the [Gradle](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/tree/main/example-gradle>) (or [Maven](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/tree/main/example-maven>)) template into your exam repository, write your first wrapper and test, and read the results.

## Guides

| Guide | What's inside |
| --- | --- |
| [Quick Start](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/quick-start/index.md>) | From zero to a running test in four steps |
| [Writing Tests](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/index.md>) | Wrapper cheat sheet, structural and behavioural tests, getters and setters, failure format, configuration, annotations |
| [Independent Tests and Mocks](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/independent-tests/index.md>) | Objects without constructor, method overrides and call checks, so one mistake fails one test |
| [Ares 2 Setup in Detail](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/ares-setup/index.md>) | Dependencies, build wiring, reserved-package guard, security policy, Ares 2 pitfalls |
| [How Matching Works](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/index.md>) | How the student's elements are found and graded as `EXACT`, `DEVIATES` or `MISSING`; types and modifiers |
| [Architecture](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/architecture/index.md>) | How the parts fit together, class overview and diagrams |
| [API Reference (Javadoc)](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/api/>) | Every public class and method of the framework |
| [FAQ &amp; Troubleshooting](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/faq/index.md>) | Common problems and known limitations |
| [Versions and Migrating](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/migration/index.md>) | What changed in each release and how to update existing tests |
| [Development and Releasing](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/releasing/index.md>) | Building and testing the framework, mutation testing, publishing a release |

For AI assistants and other tools, the whole documentation is also available as plain text: [`llms.txt`](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/llms.txt>) (index) and [`llms-full.txt`](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/llms-full.txt>) (all pages).

The framework is published on Maven Central as `io.github.valentinherrmann:levenshtein-testing-framework`; the source is on [GitHub](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework>).

# Quick Start

From zero to a running test in four steps: run the example, copy the template, write a wrapper and a test, read the results.

**Requirements:** JDK 25 and either **Gradle** (the template ships a 9.8.0 wrapper, so just `./gradlew`; 9.1+ works) or Maven 3.9+. Ares 2.2.1, JUnit 6, AssertJ and Byte Buddy come in through the example's build file.

## Step 1: Run the example

```bash
git clone https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework.git
cd Levenshtein-Testing-Framework/example-gradle
./gradlew test   # the example exercise under Ares 2
```

Inside the repository the example compiles the framework source of the checkout. `./mvnw -B verify` in the repository root builds the framework and the Maven example the same way.

To see the framework's messages, run the same tests against the failing example solution in `assignment-failing/src` (`./mvnw -B verify -Dassignment.dir=assignment-failing` for Maven). It fails on purpose, with deviating types, modifiers, parameters and inheritance, some of them combined (all defects: [`assignment-failing/README.md`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-gradle/assignment-failing/README.md>)):

```bash
./gradlew test -PassignmentDir=assignment-failing
```

```text
⚠️ ABWEICHUNG in AbstractVehicle ⚠️
Falls möglich wird der tatsächliche Wert für weitere Tests verwendet.
Erwartet:   public int getYear()
Tatsächlich:    protected long getYear()

❌ FEHLT ❌ im Paket io.github.valentinherrmann.example.vehicles
Erwartet:   public class Car extends AbstractVehicle implements Driveable
Tatsächlich:    public class Car extends AbstractVehicle

Methode:     setPrice(double)
Szenario:    Setter -> Attribut, Attribut price
Eingabe:     setPrice(-1.25)
Erwartet:    Wert abgelehnt (unverändert 2.50 oder Ausnahme)
Tatsächlich: -1.25
(2 von 15 Fällen fehlgeschlagen)
```

The messages are German by default; `LevenshteinSettings.setLanguage(LevenshteinSettings.Language.ENGLISH)` switches them to English.

The repository contains three modules:

| Module | Content |
| --- | --- |
| [`framework/`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/tree/main/framework>) | The published library `io.github.valentinherrmann:levenshtein-testing-framework` (built with Maven) |
| [`example-gradle/`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/tree/main/example-gradle>) | The example exercise as a standalone **Gradle** project: your exam template |
| [`example-maven/`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/tree/main/example-maven>) | The same exercise as a standalone **Maven** project: your exam template |

Both examples use the Artemis layout: `assignment/src` = student code, `test/` = instructor tests and policy (`assignment-failing/src` = the failing example solution).

## Step 2: Create your exam repository from the template

Copy the whole [`example-gradle/`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/tree/main/example-gradle>) (or [`example-maven/`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/tree/main/example-maven>)) folder as your exam repository. It runs as it is, so you can check your machine first with `./gradlew test` (`./mvnw -B verify`).

In this walkthrough the student package is `org.example.exam` and the test package is `org.example.tests`. The student package must not be a prefix of the test package (`org.example.exam.tests` would be wrong). Then adapt:

| Where | What to change |
| --- | --- |
| `assignment/src/`, `test/io/`, `test/sandbox/` | Delete the example's student code, tests and sandbox file, including the self-checks `SecurityControlTest`, `TimeoutControlTest` and `SandboxControl`. |
| Dependencies: `build.gradle` / `pom.xml` | Remove `junit-platform-testkit`: only `TimeoutControlTest` needs it. Add only libraries your course allows. |
| Reserved-package guard: `build.gradle` → `verifyAresReservedPackages`, `pom.xml` → antrun `verify-ares-reserved-packages-v2` | Replace `io/github/valentinherrmann/example/tests/**` with your test package, e.g. `org/example/tests/**`. |
| `test/SecurityPolicy.yaml` | `theSupervisedCodeUsesTheFollowingPackage: "org.example.exam"`, `theMainClassInsideThisPackageIs`, the list `theFollowingClassesAreTestClasses` (the **exact** names of all your test and wrapper classes), and `regardingFileSystemInteractions: [ ]`. Keep the programming language configuration. |
| Your test classes | `@Policy(value = "test/SecurityPolicy.yaml", withinPath = "classes/org/example/exam")` |

The copy takes the newest framework release from Maven Central (`levenshteinVersion` = `latest.release` in `build.gradle`, `levenshtein.version` = `[2000.0.0,)` in `pom.xml`, release candidates included), so it builds on Artemis as it is. Set a fixed version there to keep an exam on one release.

> \[!IMPORTANT\] The reasons for every setting, for both build tools, are in [Ares 2 Setup in Detail](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/ares-setup/index.md>). Do not remove the reserved-package guard or the weaving: without them the sandbox is silently inactive or can be replaced by the student.

## Step 3: Write your first wrapper and test

A **wrapper** describes one class you expect from the student. The framework finds the student's class and members, even with small typos, and the tests work through the wrapper.

****Student code**: `assignment/src/org/example/exam/Car.java` (the expected solution)**

```java
package org.example.exam;

public class Car {
    private double price;
    private double speed;

    public Car(double price) {
        this.price = price;
    }

    public void start() {
        speed = 10.0;
    }

    public double getPrice() {
        return price;
    }
}
```

**Wrapper**: `test/org/example/tests/CarWrapper.java`

```java
package org.example.tests;

import io.github.valentinherrmann.levenshtein.*;

/** Expected: public class Car in package org.example.exam. */
public class CarWrapper<T> extends ClassWrapper<T> {
    // One field per expected member (found via reflection).
    private final AttributeWrapper<T, Double> price;
    private final AttributeWrapper<T, Double> speed;
    private final ConstructorWrapper<T> constructor;
    private final MethodWrapper<T, Void> start;
    private final MethodWrapper<T, Double> getPrice;

    public CarWrapper() {
        super("Car", "org.example.exam", "public");
        price = new AttributeWrapper<>(this, "price", double.class, "private");
        speed = new AttributeWrapper<>(this, "speed", double.class, "private");
        constructor = new ConstructorWrapper<>(this,
                new Class<?>[]{double.class}, "public");
        start = new MethodWrapper<>(this, "start", void.class, "public");
        getPrice = new MethodWrapper<>(this, "getPrice", double.class,
                "public");
    }

    // Accessors for the behavioural tests
    public AttributeWrapper<T, Double> price() { return price; }
    public AttributeWrapper<T, Double> speed() { return speed; }
    public MethodWrapper<T, Void> start() { return start; }
    public MethodWrapper<T, Double> getPrice() { return getPrice; }

    // The default instance used by newObj(), getObj() and invoke()
    @Override
    public Object getObj(boolean forceNew) {
        return getObj(forceNew, constructor, 30000.0);
    }
}
```

**Test**: `test/org/example/tests/CarTest.java`

```java
package org.example.tests;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.List;

import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestFactory;

import de.tum.cit.ase.ares.api.Policy;
import io.github.valentinherrmann.levenshtein.LevenshteinSettings;
import io.github.valentinherrmann.levenshtein.LevenshteinTest;
import io.github.valentinherrmann.levenshtein.StructuralLevenshtein;
import io.github.valentinherrmann.levenshtein.StructuralLevenshtein.DetailLevel;

@LevenshteinTest
@Policy(value = "test/SecurityPolicy.yaml",
        withinPath = "classes/org/example/exam")
class CarTest {

    static {
        // feedback language, default: DEUTSCH
        LevenshteinSettings.setLanguage(LevenshteinSettings.Language.ENGLISH);
    }

    static final CarWrapper<?> car = new CarWrapper<>();

    // Structure: Class[Car], Constructors[Car], Attributes[Car], Methods[Car]
    @TestFactory
    List<DynamicTest> structure() {
        return StructuralLevenshtein.structuralTestFactory(
                DetailLevel.ONE_PER_MEMBER_CATEGORY, car);
    }

    // Behaviour: still runs if the student wrote "prize" instead of "price"
    @Test
    void startSetsSpeed() {
        Object c = car.newObj();
        car.start().invokeOnSpecificObject(c);
        assertThat(car.speed().getValue(c)).isEqualTo(10.0);
    }

    @Test
    void getPriceReturnsPrice() {
        car.testGettersSetters(car.price(), car.getPrice());
    }
}
```

The matching `SecurityPolicy.yaml` lists the two test classes:

```yaml
  theSupervisedCodeUsesTheFollowingPackage: "org.example.exam"
  theMainClassInsideThisPackageIs: "Car"
  theFollowingClassesAreTestClasses:
    - "org.example.tests.CarTest"
    - "org.example.tests.CarWrapper"
```

## Step 4: Run it and read the results

```bash
./gradlew test        # or, with Maven: ./mvnw -B verify
```

With the solution above, all 6 tests pass. Every expected element ends up in one of three states:

| State | Example | Structural test | Behavioural tests |
| --- | --- | --- | --- |
| `EXACT` ✅ | `price` | passes | run |
| `DEVIATES` ⚠️ | `prize` for `price` (≤ 20 % of the name may differ), `float` for `double`, an extra `static` | fails | **still run** on `prize` |
| `MISSING` ❌ | `strat()` for `start()` (a swap is 2 edits = 40 %) | fails | fail: "... is not implemented as expected" |

Behavioural failures (a wrong result, an exception of the student's code) show the method, the input, the expected and the actual value, as in the sample output of step 1. The exact matching rules are in [How Matching Works](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/index.md>).

That's it. Next: describe more classes ([Writing Tests](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/index.md>)), keep tests independent of other student members ([Independent Tests and Mocks](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/independent-tests/index.md>)), and look at the full example in [`example-maven/test`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/tree/main/example-maven/test/io/github/valentinherrmann/example/tests>), which covers an interface, an abstract class, inheritance, overloading and Exercise Variants.

# Writing Tests

How to describe the expected classes with wrappers, generate the structural tests, and write behavioural tests that still run when the student deviates slightly. [Quick Start](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/quick-start/index.md>) shows a complete first test; this page is the reference for everything after it.

## Wrapper cheat sheet

| Wrapper | Constructors | Describes |
| --- | --- | --- |
| `ClassWrapper<T>` | `(name, package, modifiers...)`<br>`(name, package, superClassWrapper, interfaceWrappers, modifiers...)` | A class, abstract class, interface, enum or record. `superClassWrapper == null` means "extends `Object`". |
| `AttributeWrapper<T, V>` | `(this, name, type, modifiers...)` | A field, e.g. `(this, "price", double.class, "private")` |
| `MethodWrapper<T, R>` | `(this, name, returnType, modifiers...)`<br>`(this, name, returnType, new Class<?>[]{paramTypes}, modifiers...)`<br>`(this, name, returnType, new Object[]{paramTypes}, modifiers...)` | A method, e.g. `(this, "calculateCost", double.class, new Class<?>[]{int.class}, "public")` |
| `ConstructorWrapper<T>` | `(this, "public")` (no parameters)<br>`(this, new Class<?>[]{paramTypes}, modifiers...)`<br>`(this, new Object[]{paramTypes}, modifiers...)` | A constructor, e.g. `(this, "public")` for `public Car()` |

Where a type is expected (`type`, `returnType`, an entry of `paramTypes`), you can pass a `Class<?>`, the `ClassWrapper` of a student class, or a `Supplier<ClassWrapper<?>>`. A parameter list that contains a wrapper is an `Object[]`; see [Student classes as member types](<#student-classes-as-member-types>).

- **Modifiers** are plain strings: `"public"`, `"private"`, `"protected"`, `"static"`, `"final"`, `"abstract"`, ... A typo throws `IllegalArgumentException` as soon as the wrapper is created. Every modifier of the student's element that you do not list is reported as unexpected; mark harmless extras as optional with a `?` suffix (`"public", "final?"`). See [Modifiers](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/#modifiers>).
- **Member wrappers must be (non-static) fields** of your `ClassWrapper` subclass. That's how the structural tests find them.
- **Every `ClassWrapper` subclass implements `getObj(boolean forceNew)`**, which defines the default instance, usually by delegating to `getObj(forceNew, constructorWrapper, args...)`.
- Nothing is looked up when a wrapper is created. Each wrapper looks up its element **once**, on first use, inside a supervised test.

## Inheritance and interfaces

Pass the wrappers of the expected superclass and interfaces to the subclass wrapper, and create them in dependency order (see `beforeAll()` in [`TestManager`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-maven/test/io/github/valentinherrmann/example/tests/TestManager.java>)):

```java
// interface Driveable
driveable = new DrivableWrapper<>();
// abstract class AbstractVehicle
vehicle = new AbstrWrapper<>();
// class Car extends AbstractVehicle implements Driveable
car = new CarWrapper<>(vehicle, driveable);

// inside CarWrapper:
public CarWrapper(ClassWrapper<?> superClass,
                  ClassWrapper<?>... interfaces) {
    super("Car", "org.example.exam", superClass, interfaces, "public");
    ...
}
```

The superclass and interface checks compare the classes the wrappers *found*, so a deviating superclass name still counts. Inheriting indirectly, or implementing an additional interface, is `DEVIATES`.

## Student classes as member types

If an attribute, return or parameter type is itself a student class, pass its `ClassWrapper` instead of a `Class<?>`. The type is looked up only when the member is checked, so:

- a **missing** student class only makes that type (or the parameters) `MISSING`. The member's name and modifiers are still checked, and nothing throws while the wrappers are built;
- a **misnamed** student class within the threshold is found like any other class, and the member uses it;
- two classes that **refer to each other** work, whichever wrapper is built or checked first.

Messages show the type by its expected simple name (`Drummer`), also if the class is missing.

Because member wrappers are built in the wrapper's constructor, refer to a class whose wrapper does not exist yet in one of two ways:

```java
// 1. by name: a light wrapper without members, found like any other class
ClassWrapper<?> drummerType = ClassWrapper.reference("Drummer", "org.example.band");

// 2. via a supplier of the full wrapper, evaluated when the member is checked
Supplier<ClassWrapper<?>> drummerType = () -> TestManager.drummer;
```

Inside `GuitaristWrapper`'s constructor:

```java
drummer     = new AttributeWrapper<>(this, "drummer", drummerType, "private");
getDrummer  = new MethodWrapper<>(this, "getDrummer", drummerType, "public");
setDrummer  = new MethodWrapper<>(this, "setDrummer", void.class, new Object[]{drummerType}, "public");
constructor = new ConstructorWrapper<>(this, new Object[]{double.class, drummerType}, "public");
```

Parameter lists are `Object[]` here, each entry a `Class<?>`, a `ClassWrapper` or a supplier. A lambda written directly into the array needs a cast, `(Supplier<ClassWrapper<?>>) () -> TestManager.drummer`, so prefer a variable or `ClassWrapper.reference(...)`. A supplier that still returns `null` when the member is checked throws `IllegalStateException` (wrong initialisation order in the test). `getExpectedType()`, `getExpectedReturnType()` and `getParamTypes()` return `null` for a student class that is missing.

## Structural tests

```java
@TestFactory
List<DynamicTest> structure() {
    return StructuralLevenshtein.structuralTestFactory(
            DetailLevel.ONE_PER_MEMBER_CATEGORY, driveable, vehicle, car);
}
```

| Detail level | Generated tests |
| --- | --- |
| `ONE_FOR_EVERYTHING` | `Structural[all]` |
| `ONE_PER_CLASS` | `Structural[Car]`, ... |
| `ONE_PER_MEMBER_CATEGORY` | `Class[Car]`, `Constructors[Car]`, `Attributes[Car]`, `Methods[Car]`, ... |
| `ONE_PER_MEMBER` | `Class[Car]`, `Constructor[public Car(String, int)]`, `Attribute[Car.price]`, `Method[Car.public void start()]`, ... |

Pick the level that matches your grading granularity: each generated test is its own test case (on Artemis, too).

- The order is deterministic; duplicate names get a `#2` suffix instead of replacing each other.
- A group without wrappers gets no test (e.g. no `Attributes[OpenAirStage]` for a subclass without own attributes), because it would always pass, also for an untouched template. `structuralTestFactory(level,   false, wrappers...)` still generates them.
- Student code only runs inside the generated tests, i.e. supervised by Ares.
- If one element cannot be checked, that element is reported and the rest of the test continues.

## Behavioural tests

```java
Object car = carWrapper.newObj();                  // new default instance
carWrapper.start().invokeOnSpecificObject(car);    // student's method
double speed = carWrapper.speed().getValue(car);   // (private) attribute
carWrapper.speed().setValue(car, 0.0);
double cost = carWrapper.calculateCost().invoke(); // on the default instance
Object other = carWrapper.constructor().invoke(25000.0);

// getter and setter (see "Getters and setters" below)
carWrapper.testGettersSetters(carWrapper.price(), carWrapper.getPrice(), carWrapper.setPrice());

// expected exception (a MethodWrapper for "void setYear(int)")
IllegalArgumentException e = carWrapper.setYear()
        .invokeExpectingException(IllegalArgumentException.class, car, -1);
```

Every call goes to the element the wrapper *found*, so a test written for `price` runs against the student's `prize`.

| Call | Target |
| --- | --- |
| `invoke(args...)`, `getValue()`, `setValue(value)` | the cached default instance (`getObj()`), or `null` for static members |
| `invokeOnSpecificObject(obj, args...)`, `getValue(obj)`, `setValue(obj, value)` | `obj` |
| `getObj()` | the cached default instance. The cache is only reused for the same constructor and arguments. |
| `newObj()` / `getObj(true)` / `getObj(true, constructorWrapper, args...)` | always a new instance |
| `setCachedObj(obj)` | pins an object you created yourself as the default instance |

Static members: pass `null` as the object, or use `invoke(...)` / `getValue()`.

### When the student's element is missing or throws

- A `MISSING` element fails the call with "... is not implemented as expected. See structural Tests for details". A `DEVIATES` element (typo, other numeric type, extra modifier, ...) is used as it is.
- An exception thrown by student code fails the test with a [failure block](<#reporting-failures>): the method, the argument values and the exception's type and message as `Actual:`. No stack trace and no cause are attached. A `StackOverflowError` adds a hint on endless recursion.
- Arrays and collections with more than ten elements are shortened in the `Input:` line (`[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, … (+90)]`). They are shown as they were passed, even if the student's code changed them.
- Ares security violations are passed through unchanged, also when a student's `toString()` causes one while the message is built; an endlessly recursing `toString()` is shown as `(toString() threw StackOverflowError)`.
- A failure inside the call (from Ares, a mock or a number conversion) keeps its own message and gets an `Arguments:` line and the text of `invokeWithContext`.

### Expected exceptions

`invokeExpectingException(type, obj, args...)` (methods) and `invokeExpectingException(type, args...)` (constructors) pass if the student's code throws `type` or a subclass, and return the exception for further checks. If nothing or something else is thrown, the test fails with a failure block whose `Expected:` line names the exception type.

```java
IllegalArgumentException e = carWrapper.constructor()
        .invokeExpectingException(IllegalArgumentException.class, -1.0);
assertThat(e.getMessage()).contains("price");
```

### Many generated inputs

`invokeWithContext(() -> "case " + i + ", seed " + seed, args...)` and `invokeOnSpecificObjectWithContext(context, obj, args...)` show the text in the `Scenario:` line of the failure. The supplier is only evaluated if the call fails. For checks of your own results, report with [`Failure`](<#reporting-failures>) and count the failed cases with `.cases(failed, total)`.

### Arrays as arguments

`invoke`, `invokeOnSpecificObject`, `invokeExpectingException`, `getObj(forceNew, constructorWrapper, args...)` and `ConstructorWrapper.invoke` take `Object... args`. Java spreads an object array (`String[]`, `Integer[]`, ...) passed there into several arguments; a primitive array (`boolean[]`) is passed as one. Pass a single array argument as `(Object) array`:

```java
stage.constructor().invoke((Object) spotlights);
```

A wrong number of arguments is reported as `IllegalArgumentException` (an error in the test) with this hint.

### Numbers

Numbers are converted to the student's numeric type: arguments (`invoke(2.5)` for a student's `float` parameter), `setValue` and results. A value that does not fit (`2.5` or `300` for a `byte`, `1e300` for a `float`) fails with a readable message instead of being changed; see [Matching](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/#types>). `Utils.safeCast(value, type)` converts numeric results in the same way (e.g. `int` → `double`) and fails with a readable message instead of producing a `ClassCastException` later.

## Getters and setters

`testGettersSetters` tests the getter and/or setter of an attribute; you only pass the wrappers. Call it in your own test method:

```java
@Test
void priceGettersSetters() {
    car.testGettersSetters(car.price(), car.getPrice(), car.setPrice());          // getter and setter
    car.testGettersSetters(car.price(), car.getPrice());                           // getter only (= testGetter)
    car.testGettersSetters(car.price(), car.setPrice());                           // setter only
    car.testGettersSetters(car.price(), (Double p) -> p >= 0, car.getPrice(), car.setPrice()); // setter condition
    car.testGettersSetters(car.price(), List.of(1.5, 99.0, 0.0), car.setPrice());  // own sample values
}
```

- A method wrapper without parameters is the getter, one with one parameter the setter (any order).
- Up to three subtests run, each on a new instance: **setter -\> attribute** (the setter stores the value), **attribute -\> getter** (a value written into the attribute via reflection is returned; for a `final` attribute only the current value is compared) and **setter -\> getter**. Subtests whose getter or setter was not given are skipped. The cached default instance (`getObj()`) is not changed.
- Sample values are generated for primitives, their wrappers, `String` and enums: five different values (so hard-coded getters and setters ignoring their argument fail), plus `null` for reference types if no primitive is involved. Other types (arrays, own classes, ...) need own sample values.
- A student type with another numeric type (e.g. `float` or `int` instead of `double`) is `DEVIATES`, and the generated values fit it (whole numbers for `int`), so the deviation only costs the structural point. Own sample values are used as they are: a value that does not fit (e.g. `2.5` for an `int`) fails.
- Values are compared by content (`Objects.deepEquals`), so a getter or setter may store or return a copy of an array; failure messages show the elements (`[true, false]`). To require the very same object for reference values, set `LevenshteinSettings.setGettersSettersRequireSameObject(true)` (see [Configuration](<#configuration>)).
- **Setter condition** (`Predicate`, e.g. `v -> v != null`): values that satisfy it must be stored; for other values the setter must keep the old value or throw an exception. A condition that throws itself (e.g. a `NullPointerException` for `null`) marks the value invalid. Only setters are checked; more complex rules need a behavioural test of your own.
- Every subtest runs with every sample value. If a case fails, the test fails with the [failure block](<#reporting-failures>) of the first failing case and the number of failed cases. The `Scenario:` line names the subtest:

```text
Method:   getPrice()
Scenario: attribute -> getter, attribute price
Input:    price = 2.50
Expected: 2.50
Actual:   30000.00
(10 of 15 cases failed)
```

- An exception of the getter or setter is shown as `Actual: IllegalStateException: message`. For an invalid value (setter condition) `Expected:` reads `value rejected (unchanged 2.50 or exception)`.
- A missing getter, setter or attribute fails with the usual short "... is not implemented as expected".
- Wrong use (two getters, a method with two parameters, a condition without setter, no sample values for an unsupported type) throws an `IllegalArgumentException`.

## Reporting failures

Behavioural failures share one format: exceptions of student code, `invokeExpectingException`, `testGettersSetters`, and your own checks through `Failure`. The labels follow `LevenshteinSettings.setLanguage(...)`; lines that are not set are left out:

```text
Method:   calculateLoudness(double, double)
Scenario: generated case 3, seed 2040003
Input:    volume = 12.50, crowdNoise = 3.00
Expected: 31.25
Actual:   27.00
```

Use `Failure` for your own checks, so students see the same format there:

```java
double actual = wrapper.calculateLoudness().invoke(12.5, 3.0);
if (Math.abs(actual - 31.25) > 1e-9) {
    throw Failure.of(wrapper.calculateLoudness())          // Method line: name and parameter types
            .scenario("generated case 3, seed 2040003")
            .input("volume = 12.50, crowdNoise = 3.00")    // or .arguments(12.5, 3.0)
            .expected(31.25).actual(actual)
            .cases(2, 10)                                 // adds "(2 of 10 cases failed)"
            .error();
}
```

- `Failure.of(methodWrapper)`, `Failure.of(constructorWrapper)` or `Failure.of("any text")` start a block.
- `.expected(value)` / `.actual(value)` format values the same way everywhere (`Failure.format`): `double` and `float` with two decimals (`LevenshteinSettings.setDecimals(n)` changes that; the separator is always a dot), strings in double quotes, characters in single quotes, `null`, arrays and lists in brackets. `.expectedText(...)` / `.actualText(...)` show a text as it is.
- `.thrown(exception)` shows an exception as `Actual: IllegalStateException: message`.
- `.error()` returns the `AssertionError` to throw. It never has a cause, so students see no stack trace.
- `Failure.call(() -> ...)` runs student code and turns any exception into a block (`Actual:` only); failures and Ares security violations are passed on unchanged.
- Report the first failing case only and count the others with `.cases(failed, total)`.

Structural failures ("Class X in package Y is not implemented as expected.") keep their short form.

## Abstract classes and interfaces (Byte Buddy)

Abstract classes and interfaces are instantiated through a Byte Buddy subclass, using the actual constructor (including `protected` ones). Concrete classes are **always** created with their own constructor, so `final` classes and `getClass()`-based `equals` work. Calling an abstract method on such an instance throws `AbstractMethodError`; test interface behaviour through an implementing class.

## Independent tests and mocks

A mistake in one student member should fail only the tests of that member. `newObjWithoutConstructor()`, `newObjOverriding(...)` and `mock()` create objects without running a student constructor, replace single student methods with test code and record calls. See [Independent Tests and Mocks](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/independent-tests/index.md>).

## Configuration

Configure at runtime in a static initializer of your test class (or in `@BeforeAll`):

```java
static {
    // feedback language, default: DEUTSCH
    LevenshteinSettings.setLanguage(LevenshteinSettings.Language.ENGLISH);
    // name deviation in percent of the longer name, default: 20 each
    LevenshteinSettings.setClassNameDeviationThreshold(10);
    LevenshteinSettings.setMethodNameDeviationThreshold(20);
    LevenshteinSettings.setAttributeNameDeviationThreshold(20);
    // decimals of double/float values in failure messages, default: 2
    LevenshteinSettings.setDecimals(2);
    // testGettersSetters: equal content (false, default) or the same object (true) for reference values
    LevenshteinSettings.setGettersSettersRequireSameObject(false);
    // modifiers the specification does not list: DEVIATE (default) or IGNORE
    LevenshteinSettings.setUnexpectedModifiers(LevenshteinSettings.UnexpectedModifiers.DEVIATE);
}
```

| Setting | Default | Effect |
| --- | --- | --- |
| `setLanguage` | `DEUTSCH` | Language of every message (`ENGLISH` or `DEUTSCH`) |
| `set...NameDeviationThreshold` | `20` | How much of a name may differ, see [Name matching](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/#name-matching>) |
| `setDecimals` | `2` | Decimals of `double`/`float` values in failure blocks (the separator is always a dot) |
| `setGettersSettersRequireSameObject` | `false` | `testGettersSetters` compares by content, or requires the very same object |
| `setUnexpectedModifiers` | `DEVIATE` | `IGNORE` reports only an unexpected `static` of attributes and methods, and a missing expected `static` makes the element `MISSING`; see [Modifiers](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/#modifiers>) |

> [!WARNING]
>
> **Set them before the first test**
>
> Set the settings in a static initializer or `@BeforeAll`, not in a test method or `@BeforeEach`: `@LevenshteinTest` looks up the attribute wrappers before each test (to reset attributes that are unexpectedly `static`), so a later change does not reach them.

`LevenshteinSettings.reset()` restores the defaults.

## Test annotations and timeouts

| Annotation | Use for | Composed of |
| --- | --- | --- |
| `@LevenshteinTest` | public tests (students see the result) | `@Public` `@StrictTimeout(5)` `@MirrorOutput` `@ExtendWith(StaticStateReset.class)` |
| `@HiddenLevenshteinTest` | hidden tests (results shown after the `@Deadline`) | `@Hidden` `@StrictTimeout(5)` `@MirrorOutput` `@ExtendWith(StaticStateReset.class)` |

Both need your own `@Policy(...)` on the class, and hidden tests also an Ares `@Deadline` (class or method). Without an Ares test-type annotation a test runs unsupervised. A `@StrictTimeout` on the class or a method overrides the 5 seconds. (`regardingTimeouts` in the policy is not enforced by Ares 2.2.1.) A hidden test class looks like a public one:

```java
@HiddenLevenshteinTest
@Deadline("2026-12-01 23:59")
@Policy(value = "test/SecurityPolicy.yaml", withinPath = "classes/org/example/exam")
class CarHiddenTest {
    @Test
    @StrictTimeout(10)   // this test may take longer
    void manyRandomCars() { ... }
}
```

## Exercise Variants

Keep all expected names and types in one place and read them in the wrappers, as in [`example-maven/.../Constants.java`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-maven/test/io/github/valentinherrmann/example/tests/Constants.java>):

```java
public static String concreteClass() {
    return switch (variant) {
        case DEFAULT -> "Car";
    };
}
```

## Organising larger test suites

The example splits the tests into three layers: `TestManager` carries the annotations and the `@Test`/`@TestFactory` methods, `TestAbstr`/`TestImpl`/`TestInterface` hold the test logic as static methods, and `wrappers/*` describe the classes. Only the annotated class needs `@LevenshteinTest` and `@Policy`, but **every** helper class must be listed in `theFollowingClassesAreTestClasses`.

# Independent Tests and Mocks

A mistake in one student member should fail only the tests of that member. This page shows how to create objects without the student's constructor, replace single student methods with test code, and check that a method was called, so failures do not cascade from one member to the next.

## Why tests cascade

Two situations make failures cascade. In the first, the method under test calls another student method: if that method is wrong, the test fails too. In the second, the test needs an object of a student class only as a value, but `newObj()` runs the student's constructor. Three tools remove these dependencies. All three run **no constructor** of the student class, so a missing or broken constructor cannot fail the test.

| Call | Result |
| --- | --- |
| `newObjWithoutConstructor()` | an object of the class whose attributes keep their default values (`0`, `false`, `null`; field initialisers do not run either). Concrete classes, also `final` ones, keep their exact class. Abstract classes and interfaces get a generated subclass. |
| `newObjOverriding(Map.of(method, args -> result, ...))` | an object of a generated subclass in which each given method returns the result of its function |
| `mock()` | a `Mock`: overrides like `newObjOverriding`, plus a record of every call |

## A method under test calls another student method

Example (SE01E03): `isShowReady()` calls `getNumberOfSpotlights()`. Replace the called method, so that only `isShowReady` is tested:

```java
Object stage = stageWrapper.newObjOverriding(Map.of(stageWrapper.getNumberOfSpotlights(), args -> 3));
assertThat(stageWrapper.isShowReady().invokeOnSpecificObject(stage)).isTrue();
```

The student's `isShowReady()` runs unchanged. Its call `this.getNumberOfSpotlights()` (or `getNumberOfSpotlights()`) goes to the test's function.

## The method under test must call a method of another object

Example (SE01E04): `Drummer.calculateLoudness(volume, crowdNoise)` must use `guitarist.calculateLoudness(volume)`. Give the Guitarist a return value the student cannot produce by inlining `volume * 10`, and check the call:

```java
Mock<?> guitarist = guitaristWrapper.mock()
        .override(guitaristWrapper.calculateLoudness(), args -> (int) args[0] * 7);
Object drummer = drummerWrapper.newObjWithoutConstructor();
drummerWrapper.guitarist().setValue(drummer, guitarist.obj());

assertThat(drummerWrapper.calculateLoudness().invokeOnSpecificObject(drummer, 5, 3)).isEqualTo(38);
guitarist.assertCalled(guitaristWrapper.calculateLoudness(), 1,
        () -> "Drummer.calculateLoudness(5, 3) must use calculateLoudness of its guitarist.");
```

## An object is needed only as a value

The Drummer's accessor tests need *some* Guitarist, but not a correct constructor of it:

```java
Object guitarist = guitaristWrapper.newObjWithoutConstructor();  // no Guitarist constructor runs
Object drummer = drummerWrapper.newObj();
drummerWrapper.setGuitarist().invokeOnSpecificObject(drummer, guitarist);
```

## `Mock` in detail

| Call | Effect |
| --- | --- |
| `override(method, args -> result)` | `method` runs the function. |
| `returning(method, value)` | `method` always returns `value`. |
| `othersReturnDefaults()` | Methods without an override return `0`, `false` or `null` instead of running the student's code (a full mock). `equals`, `hashCode` and `toString` without an override behave like those of `Object` (identity), so the object still works in lists, sets and messages. |
| `obj()` | Returns the object. It is created on the first call and stays the same; overrides added later apply to it as well. |
| `callCount(method)` | Returns the number of calls. |
| `calls(method)` | Returns the arguments of the recorded calls, in call order: the first 1000 calls per method (`Mock.MAX_RECORDED_CALLS`). Arrays are copied when the call is recorded; other objects are the ones passed. |
| `assertCalled(method, times[, context])`, `assertNotCalled(method)` | Fails with the expected and actual number of calls and the recorded arguments, e.g. `Calls: (5), (7)`. |
| `resetCalls()` | Forgets all recorded calls, e.g. after preparing the object. |

- Methods are matched by the method the wrapper **actually found**, so a misspelt but tolerated name (`calculateLoudnes`) is still overridden. A `MethodWrapper` of a superclass or interface wrapper overrides the inherited method. Of an overloaded method, only the matching overload is overridden. Package-private methods can be overridden, too: the subclass is defined in the student's own package and class loader.
- The function gets the arguments in the parameter types the wrapper **expects**, e.g. an `Integer` for `int`, even if the student declared `long`. Its result is converted to the student's return type (numeric types are converted, e.g. `args -> 7` for a `double` method). If the function throws, or returns `null` for a primitive return type, the call fails like a call that throws.
- Methods without an override run the student's code, supervised by Ares. An abstract method without an override fails with "The abstract method ... was called, but the test does not override it."
- The functions are test code, so Ares does not restrict them. They are called from student code, though, so keep them to plain computations.
- A missing class or method fails with the usual `MISSING` message. A class or method that cannot be overridden fails with a readable message, e.g. `Guitarist: cannot be overridden (final)`. `calls`, `callCount`, `assertCalled` and `assertNotCalled` check the method in the same way, because calls of a `final`, `static` or `private` method cannot be counted.
- Only `mock()` records calls. The objects of `newObjOverriding` and `newObjWithoutConstructor` do not, since nobody can ask for them. A clone of a mock object (`super.clone()`) shares the mock: its calls are recorded there, and methods without an override run on the clone.

## Limits

These cannot be replaced, and the call fails readably instead:

- `final` classes and methods, records, enums, sealed classes;
- `static` and `private` methods;
- constructors.

Two more limits concern code that bypasses the methods:

- An attribute that student code reads directly is not affected by any override. Set it with `attribute.setValue(obj, value)`.
- A generated subclass is a different class, so a student's `getClass()`-based `equals` returns `false` against a real object.

There are no argument matchers and no call-order checks. A function and `calls(method)` cover these.

# Ares 2 Setup in Detail

The templates [`example-gradle/build.gradle`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-gradle/build.gradle>) (Gradle) and [`example-maven/pom.xml`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-maven/pom.xml>) (Maven), together with the `test/` folder, already contain everything below. This page explains why each part is there, so you know what you must not remove when you adapt a template. Both builds run the same pipeline:

```
flowchart LR
    A["Compile student code<br/>+ weave Ares aspects"] --> B["Reserved-package<br/>guard"]
    B --> C["Compile tests"]
    C --> D["Run tests with<br/>the Ares agent"]
```

Gradle needs version 9.1+ to run on JDK 25 (8.14 cannot even read the build script); the template ships a wrapper for 9.8.0.

## 1\. Dependencies

Ares must be visible to the student sources (the weaver needs it there) *and* to the tests. In Maven that is the scope `provided`, *not* `test`: with test scope AspectJ silently weaves nothing.

****Gradle****

```groovy
repositories {
    mavenCentral()
}

configurations {
    aresAgent { transitive = false }   // just the agent JAR, see section 2
    // Maven's "provided": Ares is also on the test classpath
    testImplementation.extendsFrom compileOnly
}

dependencies {
    compileOnly "de.tum.cit.ase:ares:2.2.1"
    // the aspect library the weaver applies to the student classes
    aspect "de.tum.cit.ase:ares:2.2.1"
    implementation "org.aspectj:aspectjrt:1.9.25.1"
    aresAgent "de.tum.cit.ase:ares:2.2.1:agent"

    testImplementation(
        "io.github.valentinherrmann:levenshtein-testing-framework:latest.release")
    testImplementation platform("org.junit:junit-bom:6.1.3")
    testImplementation 'org.junit.jupiter:junit-jupiter'
    // only the example's TimeoutControlTest needs it: remove it in an exam
    testImplementation 'org.junit.platform:junit-platform-testkit'
    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}
```

****Maven****

```xml
<dependency>
    <groupId>de.tum.cit.ase</groupId>
    <artifactId>ares</artifactId>
    <version>2.2.1</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.aspectj</groupId>
    <artifactId>aspectjrt</artifactId>
    <version>1.9.25.1</version>
</dependency>
<dependency>
    <groupId>io.github.valentinherrmann</groupId>
    <artifactId>levenshtein-testing-framework</artifactId>
    <version>[2000.0.0,)</version>
    <scope>test</scope>
</dependency>
```

## 2\. Ares 2 build wiring

Three things: weave the Ares aspects into the compiled student classes, put the Ares agent where the test JVM can find it, and start the tests with `-javaagent` plus the module flags.

- **Gradle:** the plugin `io.freefair.aspectj.post-compile-weaving` (weaves; it uses the `aspect` dependency above), a `Copy` task for the agent and `aspectjrt`, and `jvmArgs` plus `forkEvery = 1` on the `test` task. Copy these blocks from [`example-gradle/build.gradle`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-gradle/build.gradle>). Two details that are easy to miss:
- `forkEvery = 1` is Gradle's counterpart of Surefire's `reuseForks=false` (see [below](<#ares-2-behaviour-worth-knowing>)).
- `exclude '**/*$*'` keeps nested classes out of the test run. Surefire skips them by default, Gradle does not, and the example's `TimeoutControlTest$TimeoutProbe` deliberately halts its JVM.
- **Maven:** `aspectj-maven-plugin` (weaves), `maven-dependency-plugin` (copies the agent) and Surefire. Copy the plugin blocks from [`example-maven/pom.xml`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-maven/pom.xml>); they follow the Ares guide "[transform an Ares 1 protected project into an Ares 2 protected project](<https://ls1intum.github.io/Ares2/instructor/transform-ares-1-into-ares-2/>)" (Postcompile, Maven).

## 3\. Reserved-package guard (mandatory)

Student classes in the build output come *before* every dependency JAR on the test classpath. A student who submits `io/github/valentinherrmann/levenshtein/StructuralLevenshtein.java` would replace the framework, and with it every structural test. The guard therefore lists the Ares prefixes **plus** `io/github/valentinherrmann/levenshtein/**`, the instructor test package, `org/junit/**`, `org/assertj/**` and `org/opentest4j/**`, and fails the build if the student output contains any of them.

- **Gradle:** the task `verifyAresReservedPackages` in `build.gradle`. It runs right after the student code is compiled and before the tests are compiled, and fails with the offending class files.
- **Maven:** the antrun execution `verify-ares-reserved-packages-v2` in `example-maven/pom.xml`.

## 4\. Security policy and annotations

```java
@LevenshteinTest   // = @Public + @StrictTimeout(5) + @MirrorOutput + StaticStateReset
@Policy(value = "test/SecurityPolicy.yaml",
        withinPath = "classes/org/example/exam")
class ExamTest {
    // @Test, @TestFactory ... (plain JUnit annotations)
}
```

- `@LevenshteinTest` (or `@HiddenLevenshteinTest` plus `@Deadline`) **activates** Ares. A class with `@Policy` but without an Ares test-type annotation runs **completely unsupervised**, silently.
- `@Policy` is exam specific and therefore not part of `@LevenshteinTest`. The nearest `@Policy` wins; policies are never merged.
- `SecurityPolicy.yaml`: see [`example-gradle/test/SecurityPolicy.yaml`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-gradle/test/SecurityPolicy.yaml>) (Gradle) or [`example-maven/test/SecurityPolicy.yaml`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-maven/test/SecurityPolicy.yaml>) (Maven). The two files differ in one line only.
- Use `JAVA_USING_GRADLE_ARCHUNIT_AND_ASPECTJ` with Gradle and `JAVA_USING_MAVEN_ARCHUNIT_AND_ASPECTJ` with Maven. Ares uses it to find the build output. With the Gradle setting Ares reads the `build.gradle` of the **working directory** of the test JVM (the project directory, which is Gradle's default); if it is not there, every test fails with "Gradle was selected but no Gradle descriptor is present".
- `withinPath` is `classes/<student/package/path>` for **both** build tools. A path like `build/classes/java/main/...` is rejected ("should start with classes or test-classes").
- `theFollowingClassesAreTestClasses` must list the **exact fully qualified names** of every instructor test, helper, constant and wrapper class. Never use a package name there, and never a class students can edit.
- For an exam, normally keep all six permission lists empty.
- Use a student package that is not a prefix of your test or wrapper packages.

## Ares 2 behaviour worth knowing

- **Endless loops end the JVM.** On a timeout, Ares 2 interrupts the student code. Student code cannot react to that interrupt (all thread operations, including `Thread.sleep` and `Object.wait`, are blocked), so Ares halts the test JVM with exit code 124. Start a fresh JVM per test class (Surefire `reuseForks=false`, Gradle `forkEvery = 1`, as in the templates), so only the tests of the affected class are lost, and keep tests that might loop in their own classes.
- **Static analysis looks at the whole submission.** A single forbidden call anywhere in the student code (e.g. `Thread.sleep`, file access) fails *every* supervised test, not only the one that runs the code. (If the policy permits some file access, as the example's does for `allowed.txt`, the static analysis leaves file access to the runtime layer; with empty permission lists, as in an exam, it rejects up front.)
- **No negative package rules.** Ares 1's `@BlacklistPackage("java.util.stream.*")` has no Ares 2 counterpart (`java.*` is always permitted). If an exam must forbid streams, write a dedicated structural/ArchUnit test for it.
- **`*_INSTRUMENTATION` modes are not usable with Ares 2.2.1**: the agent fails to transform student classes that declare records.
- The examples contain self-checks that fail if the sandbox is not active (for example when the weaving is missing): [`SecurityControlTest`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-maven/test/io/github/valentinherrmann/example/tests/SecurityControlTest.java>) (permitted/forbidden file read) and [`TimeoutControlTest`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-maven/test/io/github/valentinherrmann/example/tests/TimeoutControlTest.java>) (deadline). They need the `SandboxControl` class in the student sources, so do not copy them into a real exam. Use them to validate your setup once.

# Reference

# How Matching Works

How a wrapper finds the student's element and grades it as `EXACT`, `DEVIATES` or `MISSING`.

**In short**

- A name matches if at most **20 %** (default) of the longer name differs (Levenshtein distance). The closest name wins.
- `int`/`Integer` and two numeric types (`int`/`long`/`double`, ...) **deviate**; other types do not match.
- Methods are found by name **and** parameters; return type and modifiers are only graded afterwards.
- Every missing or unexpected modifier **deviates**; a `?` suffix makes a modifier optional.
- Every part is `EXACT`, `DEVIATES` or `MISSING`; the **worst** part is the verdict.
- `DEVIATES` fails the structural test, but behavioural tests still run on the student's element.

The rest of this page is the reference for each kind of element; use the table of contents to jump to one.

## The big picture

Every wrapper has **parts**. Each part is judged on its own as `EXACT`, `DEVIATES` or `MISSING`, and the verdict of the wrapper is the **worst** part (`MISSING` beats `DEVIATES` beats `EXACT`).

| State | Meaning | Examples |
| --- | --- | --- |
| `EXACT` | Matches the specification | `price` / `price` |
| `DEVIATES` | Found with small differences; the structural test fails, behavioural tests use the actual element | `calculateCost` / `calculateCots`, `int` / `Integer`, `protected` instead of `public`, inherited indirectly, additional interface, any missing or unexpected modifier |
| `MISSING` | Not found or not usable as specified; the structural test and every behavioural test that needs it fail | name too different, `String` vs. `int` |

(`UNCHECKED` is an internal "not looked up yet" state. It never wins an aggregation and is never shown.)

A lookup always has **two phases**, and the order matters:

1. **Find one candidate element.** Only the *identifying* parts take part: the name and, for methods and constructors, the parameter types. If nothing qualifies, the wrapper is `MISSING` and stops here.
2. **Judge the other parts of that candidate.** Return type, modifiers, superclass and so on only *grade* the element that phase 1 picked. They never make the framework pick a different one.

| Wrapper | Identifying (phase 1) | Only graded (phase 2) |
| --- | --- | --- |
| `AttributeWrapper` | name | type, modifiers |
| `MethodWrapper` | name **and** parameter types | return type, modifiers |
| `ConstructorWrapper` | parameter types (a constructor has no name) | modifiers |
| `ClassWrapper` | name (within the expected package) | modifiers, superclass, interfaces |

So a method with the right name and parameters but the wrong return type is found, and reported as a return-type problem. It is not reported as a missing method.

Each wrapper looks up its element **once**, on first use, inside a supervised test. Only members *declared in the class itself* are considered, not inherited ones (see [Methods](<#methods>)).

## Name matching

A name matches when its deviation is within the threshold. The deviation is the Levenshtein distance (number of single-character inserts, deletes and replacements) as a percentage of the **longer** name:

```text
distance * 100 / max(expected.length(), actual.length())  <=  threshold
```

| Expected | Actual | Deviation | At 20 % |
| --- | --- | --- | --- |
| `price` | `pric` | 20.0 % | `DEVIATES` |
| `calculateCost` | `calculateCots` | 15.4 % | `DEVIATES` |
| `myCar` | `MyCar` | 20.0 % | `DEVIATES` |
| `start` | `strat` | 40.0 % (a swap is two edits) | `MISSING` |
| `Car` | `car` | 33.3 % (a case slip is one edit) | `MISSING` |

- The default is 20 %, separately for classes, methods and attributes. Change it with [`LevenshteinSettings`](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/#configuration>).
- Short names tolerate fewer typos: at 20 %, a name of 4 characters or less tolerates no edit at all, and a name needs 5 characters per allowed edit.
- If several candidates qualify, the one with the **smallest distance** wins. A tie is broken by the other identifying parts, then alphabetically (details per element below).
- An exact name always wins and is never compared with the others.

## Types

Types are graded for attributes (their type) and methods (their return type), in this order, first hit wins:

| Expected → actual | Verdict | Example |
| --- | --- | --- |
| the same type | `EXACT` | `double` → `double` |
| primitive ⇄ its wrapper | `DEVIATES` | `int` → `Integer`, `Integer` → `int` |
| the actual type is a supertype of the expected one | `DEVIATES` | `String` → `Object` |
| another numeric type, wider or narrower | `DEVIATES` | `int` → `long`, `long` → `int`, `double` → `int` |
| anything else | `MISSING` | `String` → `int`, `boolean` → `int` |

Numeric types (`byte`, `short`, `char`, `int`, `long`, `float`, `double` and their wrappers) deviate in both directions; `boolean` never mixes with them. Parameter types of methods and constructors follow the [parameter rule](<#how-the-candidate-is-chosen>). Behavioural tests convert values to the student's type: `setValue` writes `2.5` into a `float` field, `invoke(2.5)` passes `2.5f` to a `float` parameter, and `testGettersSetters` generates values that fit it. A value that does not fit fails with a readable message instead of being changed: an integral type (`int`, `long`, `short`, `byte`, `char`) only takes a whole number within its range (`2.5` → `int` and `200` → `byte` fail, `65` → `char` is `'A'`), and a `float` takes every `double` that does not overflow it.

## Modifiers

Modifiers are graded after the element was found. They never decide *which* element is picked. Every difference from the specified modifiers makes the structural test fail, but never makes the element `MISSING`: behavioural tests go on with the student's element.

| Situation | Verdict | Behavioural tests |
| --- | --- | --- |
| all expected modifiers present, nothing else | `EXACT` |  |
| an expected modifier is missing (`static`, `public`, `final`, `abstract`, ...) | `DEVIATES`, "missing modifier: ..." | a method or attribute that should be `static` is used on the default object (or the object passed in) |
| the element has a modifier that is neither expected, optional nor implicit (`static`, `final`, `abstract`, `synchronized`, `transient`, `volatile`, `native`, `interface`) | `DEVIATES`, "unexpected modifier: ..." | run with the element as it is, see below |
| no modifier expected, but the element is `public`, `protected` or `private` | `DEVIATES` (no modifier means package-private) |  |

A different **visibility** is reported as the missing expected visibility only, never as an unexpected modifier. If the specification contains other modifiers (required or optional) but no visibility, the visibility is not checked; only a specification without any modifier means package-private.

**Optional modifiers.** A `?` suffix marks a modifier that may be present or not, e.g. a utility class that may be `final`:

```java
super("SetlistTools", BASE_PACKAGE, "public", "final?");
```

Optional modifiers are not shown in the expected signature. A visibility cannot be optional (`"public?"` throws), and a modifier cannot be both required and optional.

**Implicit modifiers** are never reported as unexpected:

| Element | Implicit |
| --- | --- |
| interface (also annotation) | `abstract`; `static` if nested |
| method of an interface | `abstract` (and `public`); a `static` interface method must be expected `static` |
| attribute of an interface | `static final` (and `public`) |
| enum | `final` (or `abstract` if a constant has a body); `static` if nested |
| enum constant | `static final` (and `public`) |
| record | `final`; `static` if nested. Its component attributes are `final` |
| type declared in an interface | `static` (and `public`) |
| anything | `strictfp` (no effect since Java 17, not detectable) |

A modifier written out explicitly although it is implicit (e.g. `public abstract` on an interface method) compiles to the same class file and cannot be told apart.

**How behavioural tests deal with unexpected modifiers:**

- **`static` attribute** (e.g. SE01E03: `private static boolean[] spotlights` instead of an instance attribute): the state is shared by all objects. Within one test this stays visible on purpose. Before each test, `@LevenshteinTest`/`@HiddenLevenshteinTest` reset the attribute to the value it had after the class was initialised (`StaticStateReset`), so results do not depend on the order of the tests. Arrays are copied deeply; other objects (e.g. lists) are restored by reference, so their content is not reset. `static final` attributes cannot be reset, and an attribute that any wrapper expects to be `static` is never reset. Strictly, the restored value is the one the attribute had when a wrapper first reported it (all wrappers are looked up before the first test); if student code changed it earlier, that value is restored. Because the lookups run before each test, configure `LevenshteinSettings` in a static initializer or `@BeforeAll`, not inside a test method.
- **`static` method:** called without an object.
- **`abstract` class:** objects are created through a generated subclass, as for expected abstract classes.
- **`final` attribute:** `setValue` fails with a message; reading works.
- **nested class with a different `static`:** an inner (non-static) class where a `static` nested class is expected still works: its constructors get an object of the outer class as hidden first argument (one per class wrapper, shared by all its inner objects).

**What the student sees.** A modifier deviation adds one line per kind to the `DEVIATION` message:

```text
⚠️ DEVIATION in FestivalStage ⚠️
If possible actual will be used for further testing.
Expect: private boolean[] spotlights
Actual: private static boolean[] spotlights
unexpected modifier: static
```

In your own code, `wrapper.getMissingModifiers()` and `wrapper.getUnexpectedModifiers()` return the same keywords as lists (after the element was looked up).

**The `IGNORE` mode.** `LevenshteinSettings.setUnexpectedModifiers(UnexpectedModifiers.IGNORE)` relaxes the check: only an unexpected `static` of attributes and methods is reported, and a missing expected `static` makes the element `MISSING`. Implicit modifiers such as the `static` of enum constants, and bits that are no modifiers such as varargs, are not reported in this mode either. Use it for exercises written before modifiers were checked strictly.

An unknown modifier in the specification (a typo) throws `IllegalArgumentException` when the wrapper is created.

## Attributes

1. **Exact:** a field with exactly this name, declared in the class.
2. **Otherwise** the closest field within the attribute threshold. Synthetic fields are ignored, and so are fields whose name another attribute wrapper of the same class expects ([claiming](<#claiming>)).
3. The **type** and the **modifiers** of the found field are graded.

| Expected | Student | Verdict |
| --- | --- | --- |
| `private double price` | `private double prize` | `DEVIATES` (name) |
| `private long count` | `private int count` | `DEVIATES` (numeric) |
| `private String label` | `private Object label` | `DEVIATES` (supertype) |
| `private int count` | `private boolean count` | `MISSING` (type) |

## Methods

A method is identified by its **name and its parameter types together**. This is what makes overloads (`calculateCost()` and `calculateCost(int)`) work.

### How the candidate is chosen

```
flowchart TD
    B{"Exact name and<br/>parameters exist?"} -- yes --> X[/"name EXACT<br/>parameters EXACT"/]
    B -- no --> D{"Candidate found?<br/>(rules below)"}
    D -- no --> M[/"MISSING"/]
    D -- yes --> F[/"closest candidate<br/>DEVIATES"/]
    X --> G["Grade return type and modifiers<br/>Verdict = worst part"]
    F --> G
```

A **candidate** is a method declared in the class that

- is not synthetic or a bridge method,
- is not [claimed](<#claiming>) by another `MethodWrapper` of the class,
- has a name within the method threshold,
- has matching parameters: the same number, in the same order, each one equal, a primitive/wrapper pair or two numeric types:

| Expected | Student | Parameters |
| --- | --- | --- |
| `(int)` | `(int)` | `EXACT` |
| `(int)` | `(Integer)` | `DEVIATES` |
| `(int)` | `(long)` | `DEVIATES` |
| `(int, String)` | `(String, int)` | not a candidate (swapped) |
| `(int, int)` | `(int, int, int)` or `(int)` | not a candidate (count) |

The **closest** candidate wins: smallest name distance, then closest parameters (identical, then primitive/wrapper, then numeric), then alphabetical name.

### Overloads

Overloads are told apart by their parameters, so each overload gets its own wrapper:

```java
// calculateCost()
calculateCost = new MethodWrapper<>(this, "calculateCost", double.class,
        "public");
// calculateCost(int)
calculateCostYears = new MethodWrapper<>(this, "calculateCost", double.class,
        new Class<?>[]{int.class}, "public");
```

Suppose the student wrote `calculateCost()` correctly but misspelled the overload as `calculateCots(int)`:

- The `()` wrapper finds `calculateCost()` exactly: `EXACT`.
- The `(int)` wrapper finds no `calculateCost(int)`, so it looks for a candidate. `calculateCost()` has the wrong number of parameters and is not a candidate. `calculateCots(int)` is (15.4 %). Result: `DEVIATES`.

### Claiming

A method that another `MethodWrapper` of the same class expects **exactly** (same name and parameter types) is *claimed* and is never offered as a candidate to the other wrappers. Without this, a missing method would silently be replaced by a similarly named one:

```text
wrappers:  getMinSpeed()  getMaxSpeed()
student:   getMinSpeed()

getMinSpeed  ->  EXACT
getMaxSpeed  ->  MISSING  (getMinSpeed is only 18.2 % away,
                           but it belongs to the other wrapper)
```

Attribute wrappers claim by name in the same way.

### Return type and modifiers

After the method was picked, its return type is graded like any [type](<#types>) and its modifiers like any other [modifiers](<#modifiers>). `void` is just another return type:

| Expected | Student's method | Verdict |
| --- | --- | --- |
| `public int getCount()` | `public long getCount()` | `DEVIATES` (wider) |
| `public double getPrice()` | `public int getPrice()` | `DEVIATES` (narrower) |
| `public int getCount()` | `public String getCount()` | `MISSING` |
| `public void util()` | `public static void util()` | `DEVIATES` (unexpected `static`) |
| `public static void util()` | `public void util()` | `DEVIATES` (`static` missing; called on the default object) |
| `public void play()` | `public synchronized void play()` | `DEVIATES` (unexpected `synchronized`) |
| `public void packagePrivate()` | `void packagePrivate()` | `DEVIATES` (weaker visibility) |

### Inherited methods are not found

Only methods **declared in the class itself** are looked at. A method the student inherits from a superclass is not found by the wrapper of the subclass:

```text
class Base  { public void inheritedOnly() }
class Garage extends Base

MethodWrapper "inheritedOnly" on the Garage wrapper  ->  MISSING
```

Put the wrapper on the wrapper of the class that declares the method, and let the subclass wrapper point to it as its superclass wrapper (see [Inheritance and interfaces](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/#inheritance-and-interfaces>)). This is also why the superclass is checked separately.

## Constructors

A constructor has no name, so only its **parameter types** identify it. There is no distance measure:

1. **Exact:** the declared constructor with exactly these parameter types (any visibility).
2. **Otherwise** the closest non-synthetic constructor whose parameters match by the [method parameter rule](<#how-the-candidate-is-chosen>) (identical, then primitive/wrapper, then numeric): `DEVIATES`.
3. **Otherwise** `MISSING`.
4. The modifiers of the found constructor are graded (a `protected` constructor where `public` was expected is `DEVIATES`).

Student has `Garage(int, String)` and `Garage(Integer)`:

| Expected | Verdict | Found |
| --- | --- | --- |
| `(int, String)` | `EXACT` | `Garage(int, String)` |
| `(Integer, String)` | `DEVIATES` | `Garage(int, String)` |
| `(int)` | `DEVIATES` | `Garage(Integer)` |
| `(long)` | `DEVIATES` | `Garage(Integer)` |
| `(boolean)`, `(String, int)`, `()` | `MISSING` |  |

Constructors are *not* claimed. If your wrappers expect both `(int)` and `(Integer)` and the student wrote only one of them, both wrappers resolve to that constructor.

## Classes

A class wrapper finds the student's class by name inside the expected package:

1. **Exact:** the class `package.Name` is loaded **without initialisation**, so student static initialisers never run during the structural check.
2. **Otherwise** the closest class within the class threshold (ties alphabetically). Looked at are:
3. the compiled classes of that package,
4. generated spellings of the name: plural/singular, other case of the first letter, one character removed, two neighbouring characters swapped.

Nested classes (`Outer$Inner`) are ignored unless a nested class was expected. A case slip such as `vehicle` for `Vehicle` (14.3 %) is found, `car` for `Car` (33.3 %) is not. 3. The **modifiers**, **superclass** and **interfaces** of the found class are graded.

### Superclass

| Expected | Student's class | Verdict |
| --- | --- | --- |
| the class found by the superclass wrapper | extends exactly that class | `EXACT` |
| the class found by the superclass wrapper | extends it **indirectly** (`Leaf extends Mid extends Base`, expected `Base`) | `DEVIATES` |
| no superclass (`null`) | extends something other than `Object` | `DEVIATES` |
| a superclass | extends something unrelated, or only `Object` | `MISSING` |

A record extends `java.lang.Record` and an enum `java.lang.Enum` implicitly. Like `Object`, these count as **no superclass**, so a record or enum wrapper expects `null` as its superclass.

The superclass is compared by the **class the superclass wrapper found**. If the student misspelled the name of the superclass, the superclass wrapper reports `DEVIATES` for itself, while the subclass is still `EXACT` here.

### Interfaces

The expected interfaces are compared by the classes their wrappers found:

| Situation | Verdict |
| --- | --- |
| exactly the expected interfaces | `EXACT` |
| an expected interface is implemented only indirectly (through the superclass or another interface) | `DEVIATES` |
| the student implements an **additional** interface | `DEVIATES` |
| an expected interface is not implemented at all | `MISSING` |

Extra *members* (fields, methods) in the student's class are ignored; only an extra interface is reported.

A class is usable in behavioural tests as soon as it has been **found**, even if its superclass or interfaces deviate.

## From verdict to tests

| Verdict | Structural test | Behavioural call (`invoke`, `getValue`, ...) |
| --- | --- | --- |
| `EXACT` | passes | works |
| `DEVIATES` | **fails** with a `DEVIATION` message | **works**, on the student's actual element |
| `MISSING` | **fails** with a `MISSING` message | **fails**: "... is not implemented as expected. See structural Tests" |

So a student with one typo loses the structural point once and still gets every behavioural point. A student who is missing a method loses the structural point and every behavioural test that needs that method. See [Writing Tests](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/index.md>) for how to build both kinds of tests.

## Debugging a non-match

If a wrapper reports `MISSING` although the student "obviously" has the element:

1. **Name.** Compute `distance * 100 / maxLength` and compare it with the threshold. Remember that a swap of two letters is two edits.
2. **Parameters (methods and constructors).** Same number of parameters? Same order? A `String` where you expect an `int` makes it a different method (`long` for `int` only deviates).
3. **Claiming.** Does another wrapper of the same class expect this exact signature (or, for attributes, this exact name)? Then it is not available to this wrapper.
4. **Declared in this class?** An inherited method belongs to the wrapper of the class that declares it.
5. **Modifiers.** Modifiers never make an element `MISSING` (unless `UnexpectedModifiers.IGNORE` is set: then a missing `static` does).
6. **Type.** Numeric types deviate into each other; `boolean`, `String` and other unrelated types are `MISSING`.

# Architecture

How the framework, an exam's tests and the student code fit together, which class does what, and how a wrapper gets from "expected" to "found". For the exact signatures of every public class and method see the [API Reference (Javadoc)](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/api/>); the diagrams on this page show the parts you work with, not every method.

## Overview

```
flowchart TB
    subgraph EX["Example exercise (test/)"]
        direction LR
        TM["TestManager<br/>@LevenshteinTest + @Policy"]
        POL[("SecurityPolicy.yaml")]
        subgraph WR["wrappers"]
            direction TB
            DW[DrivableWrapper]
            AW[AbstrWrapper]
            CRW[CarWrapper]
        end
        TM -- uses --> WR
        TM -. "@Policy" .-> POL
    end

    subgraph FW["Framework (framework/, published)"]
        direction LR
        LT["@LevenshteinTest<br/>@HiddenLevenshteinTest"]
        SL[StructuralLevenshtein]
        CW["ClassWrapper&lt;T&gt;"]
        MW["AttributeWrapper<br/>MethodWrapper<br/>ConstructorWrapper"]
        W["Wrapper&lt;T&gt;"]
        WP["WrapperProperty&lt;T&gt;"]
        U["Failure · Mock<br/>LevenshteinSettings · Messages · Utils"]
        CW --> W
        MW --> W
        W --> WP
    end

    subgraph ST["Student code (assignment/src), supervised by Ares 2"]
        direction LR
        D["«interface»<br/>Driveable"]
        AV["«abstract»<br/>AbstractVehicle"]
        C[Car]
        C -- extends --> AV
        C -. implements .-> D
    end

    TM -. annotated with .-> LT
    TM -- generates tests --> SL
    WR -- extend --> CW
    WR == "verify (fuzzy)" ==> ST

    classDef fw fill:#2C3E50,stroke:#1a252f,color:#ffffff
    classDef ex fill:#27AE60,stroke:#1e8449,color:#ffffff
    classDef tests fill:#F39C12,stroke:#b9770e,color:#000000
    classDef student fill:#E74C3C,stroke:#a93226,color:#ffffff
    class LT,SL,W,CW,MW,WP,U fw
    class DW,AW,CRW ex
    class TM,POL tests
    class D,AV,C student
```

Every expected class of the student code has a wrapper (a `ClassWrapper` subclass) that describes it. The wrapper finds the student's actual class and members, even with small naming deviations. `TestManager` generates the structural tests from the wrappers and uses the wrappers for the behavioural tests. Every test runs under Ares 2 supervision (`@LevenshteinTest` + `@Policy`).

## What happens in a test run

1. **Building the wrappers** (static field or `@BeforeAll`): each wrapper stores only what is *expected*: names, types, modifiers. Nothing is looked up yet, and no student code runs.
2. **First use, inside a supervised test**: the wrapper looks up its element once ([How Matching Works](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/index.md>)) and stores each part as a `WrapperProperty` (expected, actual, `EXACT`/`DEVIATES`/`MISSING`). The worst part is the wrapper's verdict.
3. **Structural tests** (`StructuralLevenshtein.structuralTestFactory`) fail for every wrapper that is not `EXACT` and print its `DEVIATION`/`MISSING` message.
4. **Behavioural tests** call the student's element through the wrapper (`invoke`, `getValue`, `newObj`, ...). A `DEVIATES` element is used as it is; a `MISSING` one fails the call. Exceptions of student code become a [failure block](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/#reporting-failures>) (`Failure`).
5. **Between tests**, `StaticStateReset` (part of `@LevenshteinTest`) resets attributes that are `static` although the test expects instance attributes, so tests do not influence each other.

## Packages and classes

**`io.github.valentinherrmann.levenshtein`** (framework, published)

| Class | Role |
| --- | --- |
| `Wrapper<T>` | Base of all wrappers: expected name and modifiers, verdict, messages, missing/unexpected modifiers |
| `ClassWrapper<T>` | A class, abstract class or interface: superclass, interfaces, instantiation (`newObj`, `newObjWithoutConstructor`, `newObjOverriding`, `mock`), `testGettersSetters`, `ClassWrapper.reference(name, package)` |
| `AttributeWrapper<T,V>`, `MethodWrapper<T,R>`, `ConstructorWrapper<T>` | Members: lookup, `getValue`/`setValue`, `invoke`, `invokeExpectingException` |
| `GenericClassWrapper<T>` | Wraps a class that is already loaded; used for the student's *actual* superclass and interfaces, not for expected classes |
| `WrapperProperty<T>` | Expected vs. actual value of one part plus its `Existence` |
| `StructuralLevenshtein` | JUnit `DynamicTest` factory for the structural tests (`DetailLevel`) |
| `Failure` | The structured failure block (`Method:`, `Scenario:`, `Input:`, `Expected:`, `Actual:`), also for your own checks |
| `Mock<T>` | Test-owned overrides and call records on an object of a student class |
| `LevenshteinTest` / `HiddenLevenshteinTest` | Composed annotations: `@Public`/`@Hidden` + `@StrictTimeout(5)` + `@MirrorOutput` + `@ExtendWith(StaticStateReset.class)` |
| `StaticStateReset` | JUnit extension that resets unexpectedly `static` attributes before each test |
| `LevenshteinSettings` | Runtime configuration: language, thresholds, decimals, getter/setter comparison, unexpected modifiers |
| `Messages` | German/English feedback texts |
| `Utils` | Levenshtein distance, type compatibility, `safeCast` |
| `TypeRef`, `Invocations`, `GettersSettersTest`, `MockClasses` (package-private) | Member types given as `ClassWrapper`, calling student code, getter/setter test, generated subclasses |

**`example-maven/`** and **`example-gradle/`** (not published): the same example exercise as two standalone projects, one per build tool. Their `assignment/` and `test/` folders are identical (CI checks this) except for one line in `SecurityPolicy.yaml`.

- `assignment/src/.../vehicles`: the "student" solution (`Car`, `AbstractVehicle`, `Driveable`), explained in [`SIMPLIFIED_CONCEPTS.md`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-maven/SIMPLIFIED_CONCEPTS.md>)
- `assignment-failing/src`: a solution with deliberate defects that shows the framework's messages
- `test/.../tests`: `TestManager` (tests), `TestAbstr`/`TestImpl`/`TestInterface` (test logic), `Constants` (Exercise Variants), `wrappers/*`, `SecurityPolicy.yaml`

## Class diagrams

The diagrams are [Mermaid](<https://mermaid.js.org/>) and render on GitHub and on the docs site. They show the public API you use in tests; keep them in sync when it changes.

### Wrappers

Every wrapper describes one expected element of the student code. `ClassWrapper` holds the member wrappers of its class; each member wrapper knows its parent `ClassWrapper`.

```
classDiagram
    direction TB
    class Wrapper~T~ {
        <<abstract>>
        +getOverallExistence() Existence
        +getExpectedName() String
        +getActualName() String
        +getActualModifiers() String
        +getMissingModifiers() List~String~
        +getUnexpectedModifiers() List~String~
        +getParentClassWrapper() ClassWrapper~T~
        +verifyExistence(boolean throwAssertion)
    }
    class ClassWrapper~T~ {
        <<abstract>>
        +reference(String name, String package)$ ClassWrapper
        +getClazz() Class~T~
        +getObj(boolean forceNew)* Object
        +getObj() Object
        +newObj() Object
        +getObj(boolean forceNew, ConstructorWrapper ctor, Object... args) T
        +setCachedObj(Object obj)
        +newObjWithoutConstructor() Object
        +newObjOverriding(Map overrides) Object
        +mock() Mock~T~
        +testGettersSetters(AttributeWrapper attribute, ...)
    }
    class GenericClassWrapper~T~ {
        +GenericClassWrapper(Class~T~ clz)
    }
    class AttributeWrapper["AttributeWrapper&lt;T, V&gt;"] {
        +getExpectedType() Class
        +getValue() V
        +getValue(Object obj) V
        +setValue(Object value)
        +setValue(Object obj, Object value)
    }
    class MethodWrapper["MethodWrapper&lt;T, R&gt;"] {
        +invoke(Object... params) R
        +invokeOnSpecificObject(Object obj, Object... params) R
        +invokeWithContext(Supplier context, Object... params) R
        +invokeExpectingException(Class expected, Object obj, Object... params) X
        +getParamTypes() Class[]
        +getActualParamTypes() Class[]
        +getExpectedReturnType() Class
    }
    class ConstructorWrapper~T~ {
        +invoke(Object... args) T
        +invokeExpectingException(Class expected, Object... args) X
        +getParamTypes() Class[]
        +getActualParamTypes() Class[]
    }

    ClassWrapper --|> Wrapper
    GenericClassWrapper --|> ClassWrapper
    AttributeWrapper --|> Wrapper
    MethodWrapper --|> Wrapper
    ConstructorWrapper --|> Wrapper
    ClassWrapper o-- AttributeWrapper : fields of the subclass
    ClassWrapper o-- MethodWrapper
    ClassWrapper o-- ConstructorWrapper
```

### Verdicts and member types

Each part of a wrapper (name, modifiers, type, parameters, superclass, ...) is a `WrapperProperty` with its own `Existence`; the worst one is the wrapper's verdict. A member type that is a student class is a `TypeRef`, resolved through its `ClassWrapper` when the member is checked.

```
classDiagram
    direction LR
    class Wrapper~T~ {
        <<abstract>>
    }
    class WrapperProperty~T~ {
        +getExpected() T
        +getActual() T
        +getExistence() Existence
    }
    class Existence {
        <<enumeration>>
        UNCHECKED
        EXACT
        DEVIATES
        MISSING
        +worst(Existence a, Existence b)$ Existence
    }
    class TypeRef {
        <<interface, package-private>>
        +resolve() Class
        +expectedSimpleName() String
    }
    class ClassWrapper~T~ {
        <<abstract>>
    }

    Wrapper *-- WrapperProperty : name, modifiers, type, params, ...
    WrapperProperty --> Existence
    Wrapper *-- TypeRef : attribute, return and parameter types
    TypeRef --> ClassWrapper : a fixed Class or a student class
```

### Failures, mocks and settings

```
classDiagram
    direction LR
    class Failure {
        <<final>>
        +of(MethodWrapper method)$ Failure
        +of(ConstructorWrapper ctor)$ Failure
        +of(String member)$ Failure
        +scenario(String text) Failure
        +input(String text) Failure
        +arguments(Object... args) Failure
        +expected(Object value) Failure
        +actual(Object value) Failure
        +thrown(Throwable t) Failure
        +cases(int failed, int total) Failure
        +error() AssertionError
        +format(Object value)$ String
        +call(ThrowingSupplier code)$ R
    }
    class Mock~T~ {
        <<final>>
        +override(MethodWrapper method, Function function) Mock~T~
        +returning(MethodWrapper method, Object value) Mock~T~
        +othersReturnDefaults() Mock~T~
        +obj() Object
        +callCount(MethodWrapper method) long
        +calls(MethodWrapper method) List~Object[]~
        +assertCalled(MethodWrapper method, int times)
        +assertNotCalled(MethodWrapper method)
        +resetCalls()
    }
    class LevenshteinSettings {
        <<final>>
        +setLanguage(Language language)$
        +setClassNameDeviationThreshold(int percent)$
        +setMethodNameDeviationThreshold(int percent)$
        +setAttributeNameDeviationThreshold(int percent)$
        +setDecimals(int decimals)$
        +setGettersSettersRequireSameObject(boolean same)$
        +setUnexpectedModifiers(UnexpectedModifiers mode)$
        +reset()$
    }
    class Language {
        <<enumeration>>
        DEUTSCH
        ENGLISH
    }
    class UnexpectedModifiers {
        <<enumeration>>
        DEVIATE
        IGNORE
    }
    class ClassWrapper~T~ {
        <<abstract>>
        +mock() Mock~T~
    }
    class MethodWrapper["MethodWrapper&lt;T, R&gt;"]

    ClassWrapper ..> Mock : creates
    Mock --> MethodWrapper : overrides, records calls
    Failure ..> MethodWrapper : Method line
    Failure ..> LevenshteinSettings : language, decimals
    LevenshteinSettings ..> Language
    LevenshteinSettings ..> UnexpectedModifiers
```

### Test generation and annotations

```
classDiagram
    direction LR
    class StructuralLevenshtein {
        <<final>>
        +structuralTestFactory(DetailLevel level, ClassWrapper... wrappers)$ List~DynamicTest~
        +structuralTestFactory(DetailLevel level, boolean skipEmpty, ClassWrapper... wrappers)$ List~DynamicTest~
    }
    class DetailLevel {
        <<enumeration>>
        ONE_FOR_EVERYTHING
        ONE_PER_CLASS
        ONE_PER_MEMBER_CATEGORY
        ONE_PER_MEMBER
    }
    class LevenshteinTest {
        <<annotation>>
    }
    class HiddenLevenshteinTest {
        <<annotation>>
    }
    class StaticStateReset {
        <<JUnit extension>>
    }

    StructuralLevenshtein ..> DetailLevel
    LevenshteinTest ..> StaticStateReset : @ExtendWith
    HiddenLevenshteinTest ..> StaticStateReset : @ExtendWith

    note for LevenshteinTest "= @Public + @StrictTimeout(5) + @MirrorOutput\n+ @ExtendWith(StaticStateReset.class)"
    note for HiddenLevenshteinTest "= @Hidden + @StrictTimeout(5) + @MirrorOutput\n+ @ExtendWith(StaticStateReset.class)"
```

### Example exercise: wrappers

One wrapper per expected class of the student code (`io.github.valentinherrmann.example.tests.wrappers`). Each wrapper declares its members as `AttributeWrapper`, `MethodWrapper` and `ConstructorWrapper` fields (found by `ClassWrapper` via reflection for the structural tests) and exposes them through accessors for the behavioural tests. Expected names and types come from `Constants` (Exercise Variants pattern).

```
classDiagram
    direction BT
    class ClassWrapper~T~ {
        <<abstract, framework>>
        +getObj(boolean forceNew)* Object
    }
    class DrivableWrapper~T~ {
        +maxSpeed() AttributeWrapper
        +startMethod() MethodWrapper
        +getSpeedMethod() MethodWrapper
        +getObj(boolean forceNew) Object
    }
    class AbstrWrapper~T~ {
        +manufacturer() AttributeWrapper
        +year() AttributeWrapper
        +constructor() ConstructorWrapper
        +getManufacturer() MethodWrapper
        +getYear() MethodWrapper
        +calculateCost() MethodWrapper
        +getInfo() MethodWrapper
        +getObj(boolean forceNew) T
    }
    class CarWrapper~T~ {
        +CarWrapper(ClassWrapper superClass, ClassWrapper... interfaces)
        +price() AttributeWrapper
        +speed() AttributeWrapper
        +constructor_full() ConstructorWrapper
        +constructor_default() ConstructorWrapper
        +getPrice() MethodWrapper
        +setPrice() MethodWrapper
        +startMethod() MethodWrapper
        +getSpeed() MethodWrapper
        +calculateCost() MethodWrapper
        +calculateCostYears() MethodWrapper
        +getInfo() MethodWrapper
        +getObj(boolean forceNew) Object
    }

    DrivableWrapper --|> ClassWrapper
    AbstrWrapper --|> ClassWrapper
    CarWrapper --|> ClassWrapper
    CarWrapper ..> AbstrWrapper : superClassWrapper
    CarWrapper ..> DrivableWrapper : interfaceWrappers

    note for DrivableWrapper "interface Driveable:\nMAX_SPEED, start(), getSpeed()"
    note for AbstrWrapper "abstract class AbstractVehicle:\nmanufacturer, year, abstract calculateCost(), getInfo()"
    note for CarWrapper "Car extends AbstractVehicle implements Driveable:\noverloaded constructors and calculateCost"
```

# Help

# FAQ &amp; Troubleshooting

Answers to common questions and fixes for typical problems, grouped by topic. Each question is a heading, so you can link to it directly.

## Setup and build

### Does it work with Gradle?

Yes, and Gradle is the primary path of the [Quick Start](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/quick-start/index.md>). The template is [`example-gradle/`](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/tree/main/example-gradle>) and CI runs it next to the Maven build. The framework itself is built and published with Maven, but it is an ordinary Maven Central artifact that Gradle consumes like any other. See [Ares 2 Setup in Detail](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/ares-setup/index.md>) for the Gradle specifics.

### Which framework version do I get?

The templates take the newest release from Maven Central, release candidates included (`levenshteinVersion` = `latest.release` in `build.gradle`, `levenshtein.version` = `[2000.0.0,)` in `pom.xml`). Set a fixed version there to keep an exam on one release. These docs describe the newest release; changes between versions are listed in [Versions and Migrating](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/migration/index.md>).

### Gradle: "Gradle was selected but no Gradle descriptor is present"

The policy says `JAVA_USING_GRADLE_...`, but the test JVM's working directory has no `build.gradle`. Use the project directory (Gradle's default) as working directory.

### Gradle: the test JVM exits with code 124 right at the start

Nested classes are run as stand-alone tests (Gradle, unlike Surefire, does not skip them). Add `exclude '**/*$*'` to the `test` task.

## Ares and the sandbox

### The policy seems to have no effect

Check that the class carries `@LevenshteinTest` (or another Ares test-type annotation), that `withinPath` is `classes/<student/package/path>`, that Ares is `provided` (Maven) or on `compileOnly` and `aspect` (Gradle), and that `SecurityControlTest` of the example passes in your setup.

### All tests fail with "... was blocked by Ares"

The static analysis found a forbidden call somewhere in the student code (see the message). This is intended. If the exercise legitimately needs the operation, grant exactly that in the policy.

### The test JVM crashed with exit code 124

A test timed out in student code. Ares halts the JVM in that case, see [Ares 2 behaviour worth knowing](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/ares-setup/#ares-2-behaviour-worth-knowing>).

## Matching and structural tests

### A similar name is not found

Work through the checklist in [Debugging a non-match](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/#debugging-a-non-match>): the name percentage, the parameters of a method (same number and order), claiming by other wrappers, and inherited vs. declared methods.

### A structural test fails although the behavioural tests pass

Intended: the element `DEVIATES`. The student loses the structural point, not the behavioural ones.

### A structural test fails with "unexpected modifier: ..."

The student's element has a modifier your wrapper does not list, e.g. `static` on an attribute or `final` on a class. Every modifier that is neither expected, optional nor implicit is a deviation. If the extra modifier is harmless, mark it as optional with a `?` suffix (`"public", "final?"`). See [Modifiers](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/#modifiers>).

### Extra members in the student code?

They are ignored. Only an additional *interface* and additional *modifiers* are reported as `DEVIATES`.

### There is no `Attributes[...]` test for one of my classes

A group without wrappers gets no structural test, because it would always pass. `structuralTestFactory(level, false, wrappers...)` still generates such tests. See [Structural tests](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/#structural-tests>).

### Generic classes?

Type parameters are not part of the specification; use the erased types.

## Behavioural tests

### The feedback is in German

That is the default. Call `LevenshteinSettings.setLanguage(LevenshteinSettings.Language.ENGLISH)`, see [Configuration](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/#configuration>).

### A setting has no effect

Set `LevenshteinSettings` in a static initializer or `@BeforeAll`, not in a test method or `@BeforeEach`: `@LevenshteinTest` looks up the attribute wrappers before each test, so a later change does not reach them.

### An array argument arrives as several arguments

`invoke`, `getObj(forceNew, constructorWrapper, args...)` and the other `Object...` methods are varargs. Pass a single array argument as `(Object) array`, see [Arrays as arguments](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/#arrays-as-arguments>).

### A getter/setter test fails for an array attribute although the content is right

Arrays are compared by content. If the test fails, the content differs; the message shows the elements. If you want to require the same object instead, set `LevenshteinSettings.setGettersSettersRequireSameObject(true)`.

### How do my own checks get the same failure format?

Throw `Failure.of(methodWrapper)...error()` instead of an AssertJ assertion, see [Reporting failures](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/#reporting-failures>).

### Results depend on the order of the tests

Usually a student attribute that is `static` although an instance attribute is expected. `@LevenshteinTest` resets such attributes before each test, but not `static final` ones and not the *content* of a list or map; see [Modifiers](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/#modifiers>).

## Limitations

- Thresholds are global per element kind, not per wrapper.
- Records and enums are checked structurally like classes (their implicit superclass is ignored). An enum cannot be instantiated, so the `getObj` of its wrapper returns `null` or one of its constants. Record components, enum constants, `sealed`/`permits`, generic type parameters and annotations are not verified.
- **JavaFX / EOS exercises are untested.** The framework needs Ares 2 on the test classpath: `@LevenshteinTest` composes Ares 2 annotations, and the wrappers call student code through Ares 2's `ReflectionTestUtils`. EOS 1.1.0 bundles a fork of Ares 1, and whether it works together with Ares 2 has not been checked.
- Method parameter types must match (up to primitive ⇄ wrapper or numeric type); reordered parameters are `MISSING`.
- Mocks (`mock()`, `newObjOverriding`) cannot replace `final` classes or methods, records, enums, sealed classes, `static` or `private` methods, or constructors, and an attribute read directly by student code bypasses every override. See [Independent Tests and Mocks](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/independent-tests/#limits>).

# Versions and Migrating

What changed between releases and how to update existing tests. If you start a new exam, you do not need this page: the other guides describe the current release.

| Version | Highlights |
| --- | --- |
| **2026.11.1** | One failure format for behaviour failures (`Failure`), unexpected modifiers reported, student classes as member types, objects without constructor and mocks, docs site |
| 2026.11.0 | `testGettersSetters` (getter/setter tests with setter conditions and own sample values), failing example solution |
| 2000.0.0 | First release on Ares 2 and Java 25 |
| 1000.0.0 | Last release on Ares 1 (`0260.9.0` was published by accident instead of `0.9.0`; `1000.0.0` acts as `1.0.0`) |

## Upgrading to 2026.11.1

The changes since 2000.0.0, for test suites written against 2000.0.0 or 2026.11.0. Coming from 1000.x, first apply [the 2000.0.0 changes](<#migrating-from-1000x-to-2000>).

### What can turn existing tests red

- **Unexpected modifiers deviate** (\#35). A modifier of the student's element that is neither expected, optional nor implicit (e.g. `private static boolean[] spotlights` where `private` is expected, an extra `abstract`, `final` or `synchronized`) fails the structural test with "unexpected modifier: ...". Mark harmless extras as optional with a `?` suffix, e.g. `"public", "final?"` for a utility class. See [Modifiers](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/matching/#modifiers>).
- A wrapper of a `static` nested class must now expect `static`.
- To keep the old behaviour, set `LevenshteinSettings.setUnexpectedModifiers(UnexpectedModifiers.IGNORE)`.
- **A missing expected `static` is a deviation instead of `MISSING`**: behavioural tests go on and call the method or read the attribute on the default object.
- **Messages of behaviour failures changed** (\#34). Exceptions of student code (`invoke`, `newObj`, `ConstructorWrapper.invoke`), `invokeExpectingException` and `testGettersSetters` report with one block of `Method:` / `Scenario:` / `Input:` / `Expected:` / `Actual:` lines (German: `Methode:`, `Szenario:`, `Eingabe:`, `Erwartet:`, `Tatsächlich:`). Tests that assert on message texts need updating:
- "Calling method ... threw an exception: ...", "... should throw X, but: ..." and "Getter/setter test for attribute ...: ✅/❌ ..." are gone. An exception thrown by a student constructor is reported in the block, too.
- `Arguments: (...)` became the `Input:` line; the text of `invokeWithContext` is the `Scenario:` line.
- `testGettersSetters` shows only the first failing case plus "(n of m cases failed)" instead of one line per subtest. A missing getter or setter fails with the short structural message.
- `double`/`float` values are shown with two decimals (`2.50`, `LevenshteinSettings.setDecimals`).
- The `AssertionError` no longer has the student's exception as cause (no stack trace, no `Caused by`). Read the exception from `invokeExpectingException` if a test needs it.
- A `StackOverflowError` of student code is a normal failure with a hint on endless recursion instead of being rethrown.
- The old message constants are kept but `@Deprecated`.
- **No structural tests for empty groups.** `structuralTestFactory` no longer generates tests such as `Attributes[X]` for a class without attribute wrappers or `Constructors[X]` for an interface. They always passed, also for an untouched template. If your Artemis exercise gives points for one of them, remove it there, or keep the old tests with `structuralTestFactory(level, false, wrappers...)`.
- **Numbers are no longer changed silently.** `Utils.safeCast`, `setValue` and arguments for a deviating numeric parameter type fail with a readable message if the value does not fit the student's type (e.g. `2.5` → `int`, `200` → `byte`). Before, `safeCast` and `setValue` truncated or wrapped such values. Arguments are now converted to a deviating numeric parameter type (`2.5` → `float`); before, the call failed.
- **A wrong number of arguments** for `invoke(...)`, `ConstructorWrapper.invoke(...)` or `getObj(forceNew,   constructorWrapper, args...)` is an `IllegalArgumentException` (an error in the test) instead of a failed test. Pass a single array argument as `(Object) array`.
- **Getters and setters compare arrays by content** (`Objects.deepEquals`). To require the same object, set `LevenshteinSettings.setGettersSettersRequireSameObject(true)`.

### API changes

- A literal `null` as return type (`new MethodWrapper<>(this, "m", null, ...)`) is ambiguous now; write `(Class<Object>) null`.
- A `null` entry in a parameter type array throws `IllegalArgumentException` when the wrapper is built.
- With `@LevenshteinTest`, attribute wrappers are looked up before each test. Configure `LevenshteinSettings` in a static initializer or `@BeforeAll`; changes inside a test method come too late for them.

### New features

- **`Failure`** for your own checks, so students see the same format there. See [Reporting failures](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/#reporting-failures>).
- **Optional modifiers** with a `?` suffix, and `getMissingModifiers()` / `getUnexpectedModifiers()` on every wrapper.
- **`StaticStateReset`**: `@LevenshteinTest` and `@HiddenLevenshteinTest` reset attributes that are unexpectedly `static` before each test, so results do not depend on the order of the tests.
- **Inner instead of nested class**: an inner class where a `static` nested class is expected can still be instantiated (the outer instance is created automatically).
- **Student classes as member types** (\#33). `AttributeWrapper`, `MethodWrapper` and `ConstructorWrapper` accept a `ClassWrapper` (e.g. `ClassWrapper.reference("Drummer", pkg)`) or a supplier of one as type; parameter lists can be an `Object[]` that mixes them with `Class<?>`. Replace helpers like a copied `TypeWrapper.resolve(name)` with Byte Buddy placeholder classes by these. See [Student classes as member types](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/writing-tests/#student-classes-as-member-types>).
- **Independent tests** (\#30): `ClassWrapper.newObjWithoutConstructor()`, `newObjOverriding(...)` and `mock()`. See [Independent Tests and Mocks](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/independent-tests/index.md>).
- An `AssertionError` from inside a call (from Ares, a mock or a conversion) gets the arguments and the text of `invokeWithContext` as well.
- **Docs site** with search and the Javadoc: [https://valentinherrmann.github.io/Levenshtein-Testing-Framework/](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/>).

## Migrating from 1000.x to 2000

**`2000.0.0`** is the first release of the Ares 2 / Java 25 line and contains these breaking changes:

| 1000.x | 2000.0.0 |
| --- | --- |
| Ares 1 (`de.tum.in.ase:artemis-java-test-sandbox`) | Ares 2 (`de.tum.cit.ase:ares`, provided), Java 25 |
| `@LevenshteinTest` = Ares 1 security annotations; **the sandbox was never active** because no `@Public`/`@Hidden` was included | `@LevenshteinTest` = `@Public @StrictTimeout(5) @MirrorOutput` (2026.11.1 adds `StaticStateReset`); plus `@Policy` + `SecurityPolicy.yaml` in the exam |
| `io.github.valentinherrmann.test.TestSettings` constants (inlined at compile time, not changeable) | `io.github.valentinherrmann.levenshtein.LevenshteinSettings` setters |
| `io.github.valentinherrmann.test.Messages.X` (String) | `io.github.valentinherrmann.levenshtein.Messages.X.get()` / `.format(...)` |
| `TestSettings.BASE_PACKAGE`, `variant` | in your own `Constants` |
| `getObj(boolean forceNew, boolean useByteBuddy)`, `useByteBuddy=false` for private members | `getObj(boolean forceNew)`; wrappers delegate to `getObj(forceNew, constructorWrapper, args...)`. Abstract classes and interfaces always use Byte Buddy, concrete classes never |
| `getObj(boolean useByteBuddy)` (returned the **cached** instance) | `getObj()` (cached) or `newObj()` / `getObj(true)` (new) |
| `getDynamicSubclassObj(Class<?>[] paramTypes, args...)` | `getDynamicSubclassObj(constructorWrapper, args...)` |
| `Utils.saveCast(...)` | `Utils.safeCast(...)` |
| `getMethodsWrappers()` | `getMethodWrappers()` |
| `setObj` writes `obj` directly | `setCachedObj(obj)` |
| Exceptions from student code: generic failure | type and message in the failure, `invokeExpectingException(...)` |

**Gradle or Maven?** Both work. The coordinates are unchanged (`io.github.valentinherrmann:levenshtein-testing-framework`), but a 1000.x exam repository also needs the Ares 2 build wiring (weaving, agent, reserved-package guard). Start from the [Gradle](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-gradle/build.gradle>) or [Maven](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/example-maven/pom.xml>) template and see [Ares 2 Setup in Detail](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/ares-setup/index.md>).

# Development and Releasing

For maintainers of the framework: how the repository is built and tested, and how a release reaches Maven Central. Exam authors do not need this page.

## Building and testing

```bash
# framework unit tests + Maven example under Ares 2, against the framework source of this checkout
./mvnw -B verify

# Gradle example under Ares 2, against the framework source of this checkout
(cd example-gradle && ./gradlew test)

# the failing example solution (fails on purpose; shows the students' messages)
./mvnw -B verify -Dassignment.dir=assignment-failing
(cd example-gradle && ./gradlew test -PassignmentDir=assignment-failing)

# + sources, javadoc, GPG signature (needs a key)
./mvnw -B -P release -pl framework verify

# mutation testing of the framework with PIT (report: framework/target/pit-reports/index.html)
./mvnw -B -f framework/pom.xml -P mutation verify
```

Inside this repository both examples compile the framework source (`../framework/src/main/java`) together with the tests, so the framework can be debugged directly. The examples are exam templates: a copy without `../framework` uses the newest framework release from Maven Central, release candidates included (`levenshteinVersion` in `build.gradle`, `levenshtein.version` in `pom.xml`, both overridable). CI installs the SNAPSHOT and passes it to standalone copies of both examples.

The framework is built and published with Maven only (publishing with Gradle is more trouble than it is worth). The Gradle module is a consumer of it, like an exam repository.

### Mutation testing (PIT)

The framework's own tests are checked with [PIT](<https://pitest.org>) mutation testing: PIT changes the framework code in small ways (e.g. `<` to `<=`, a condition to `true`) and reruns the tests; a change no test notices ("surviving mutant") points to a missing test or a bug. Only framework classes are mutated, never the examples. CI's `mutation` job fails below `mutationThreshold` in the `mutation` profile of `framework/pom.xml` and uploads the HTML report as the `pit-report` artifact. Raise the threshold when new tests kill more mutants; never lower it to make a change pass.

## Releases

A version on Maven Central can never be changed or removed, so every release goes through a release candidate first. The `Publish` workflow builds and deploys the framework when a GitHub release is published; the version comes from the tag (`2026.11.1` or `2026.11.1-RC1`, as with the existing tags; a leading `v` would be stripped). `framework/pom.xml` is the only place that holds the framework version.

### The version number

The only version in the code is `<version>` in `framework/pom.xml`. It always holds a `-SNAPSHOT` of the **next** release (e.g. `2026.11.1-SNAPSHOT` for the release `2026.11.1`).

- **Before a release: change nothing.** Do not remove `-SNAPSHOT` and do not commit the release version. `Publish` takes the version from the tag and sets it in the build only; CI's `release-dry-run` job tests the SNAPSHOT without the suffix. Choose the tag so that it equals the SNAPSHOT without `-SNAPSHOT` (or deliberately choose a different version, then the next step sets it).
- **After the final release: set `<version>` to the next `-SNAPSHOT`** (e.g. `2026.11.1` released, then `2026.12.0-SNAPSHOT`) and commit it to `main`. This is the last step of the checklist. Do not bump after an RC: the RC belongs to the same SNAPSHOT.
- **Never change:** the version ranges in the examples (`levenshtein.version` = `[2000.0.0,)` in `example-maven/pom.xml`, `levenshteinVersion` = `latest.release` in `example-gradle/build.gradle`), and the `0-SNAPSHOT` of the root `pom.xml` and `example-maven/pom.xml`. They always take the newest release from Maven Central.
- **Only on a new major line** (a breaking change like 2000.0.0): raise the lower bound `2000.0.0` of the example ranges in `example-maven/pom.xml` and `docs/ares-setup.md`, and write the breaking changes into [Migration](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/migration/index.md>).

### Checklist

1. [ ] `main` is green: CI passes, including the `release-dry-run` job (it builds sources, javadoc and the version change like `publish.yml`, without GPG and without the upload).
2. [ ] Docs match the code: [Versions and Migrating](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/migration/index.md>) lists the new version and its changes; the guides, the [README](<https://github.com/ValentinHerrmann/Levenshtein-Testing-Framework/blob/main/README.md>) and the examples show current usage (see "Keeping the docs up to date" below).
3. [ ] Check that the secrets `GPG_PRIVATE_KEY`, `GPG_PASSPHRASE`, `OSSRH_USER` and `OSSRH_TOKEN` are set and still valid.
4. [ ] Create a **pre-release** on GitHub with the tag `<version>-RC1` on `main`. This starts `Publish`.
5. [ ] Wait for `Publish` to succeed and for the artifact to appear on Maven Central (this can take some minutes).
6. [ ] Run the `Smoke test` workflow with the version `<version>-RC1`: it runs both example templates (Gradle and Maven) against Maven Central.
7. [ ] If a problem shows up: fix it on `main` and publish `-RC2` (go to step 4). Never re-use a tag.
8. [ ] Create the **release** on GitHub with the tag `<version>` on the commit that passed the smoke test.
9. [ ] Wait for `Publish` and run `Smoke test` once more without an input (it takes the newest version on Maven Central).
10. [ ] Set `<version>` in `framework/pom.xml` to the next `-SNAPSHOT` and commit it to `main`. This is the only commit that changes the version.

## Documentation site

The docs site (`docs/` and the Javadoc of `framework/src/main`) is rebuilt and deployed to GitHub Pages by the `Docs` workflow on every push to `main` that touches them; nothing to do for a release. It shows the state of `main` (or of the branch of the last manual run: *Actions \> Docs \> Run workflow* deploys any branch, the next push to `main` restores it), so merge doc changes for a new version together with the code. Preview it locally with `pip install zensical && zensical serve`.

### Keeping the docs up to date

Every change that users can notice (public API, messages, settings, behaviour, build templates) updates the docs in the same pull request:

- the guide that explains the feature (`docs/*.md`), written as the current behaviour, without "since version X" or "before version Y";
- [Versions and Migrating](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/migration/index.md>): what changed and how to update existing tests;
- the class diagrams in [Architecture](<https://valentinherrmann.github.io/Levenshtein-Testing-Framework/architecture/index.md>) if public classes or methods change;
- new pages in `nav` of `zensical.toml`.

`zensical build --clean --strict` must pass (the `Docs` workflow runs it for every pull request). `docs/plans/` holds internal implementation plans and is not published.