ICCMA 2027 Competition Tracks

NEW FOR ICCMA 2027:

  • Weak admissibility makes its ICCMA debut: the new task DC-WAD (credulous acceptance under weakly admissible semantics) joins the main track.
  • A new scoring system for the heuristics track, aligned with established evaluation practices in related fields such as machine learning. See here for more information.

Just like the last competition, ICCMA 2027 will once again consist of four tracks:

  • main track
  • heuristics track
  • dynamic track
  • ABA track

Each track is composed of multiple sub-tracks, defined by a combination of a reasoning problem and an argumentation semantics.

For each (sub-)track (except No-Limits Track), an argumentation system can additionally take part in a fixed-SAT solver comparison. A participating solver is then ranked also in a separate list with only other solvers taking part in this comparison (in addition to the normal ranking). Note the special requirements for taking part in this comparison. We encourage all participants using a SAT solver to take part in this comparison.

Note: Argumentation systems (solvers) can be submitted for evaluation into any choice of sub-tracks, i.e., there is no requirement to support e.g. all semantics for a specific reasoning problem, or all reasoning problems for a specific semantics.

Main Track

Note: The focus of the Main Track is to evaluate sequential core argumentation reasoning engines available in open source. Systems combining different core reasoning engines e.g. via portfolio-style techniques, systems employing parallel computations via the use of multiple processor cores, as well as systems which will not be made available in open source are invited to the special No-Limits Track which consists of the same subtracks as the Main Track. The organizers reserve the right to move Main Track submissions to No-Limits based on case-by-case analysis.

Problem Setting

We recall the definition of Dung's Abstract Argumentation Frameworks (AFs) [Dung 95] and its semantics.

An AF is a directed graph F = (A,R) where A is a set of arguments, and R ⊆ A × A is the attack relation. For a,b ∈ A, we say that a attacks b when (a,b) ∈ R. If in addition b attacks c ∈ A, then a defends c against b. The same concepts are extended to sets of arguments: S ⊆ A attacks (respectively defends) an argument b ∈ A if there is some a ∈ S that attacks (respectively defends) b.

The relevant semantics of AFs are defined as follows. Let the range of S ⊆ A be S ∪ {a ∈ A | S attacks a}, and let the reduct FS be the AF obtained from F by removing all arguments in the range of S. A set S ⊆ A is

  • conflict-free (S ∈ CF(F)) if S does not attack any of its elements;
  • admissible (S ∈ AD(F)) if S ∈ CF(F) and S defends all its elements against all their attackers;
  • weakly admissible (S ∈ WAD(F)) if S ∈ CF(F) and no attacker of S belongs to any weakly admissible set of FS;
  • a complete extension (S ∈ CO(F)) if S ∈ AD(F) and S contains all the arguments that it defends;
  • a preferred extension (S ∈ PR(F)) if S is a maximal (w.r.t. set inclusion) admissible extension;
  • a stable extension (S ∈ ST(F)) if S ∈ CF(F) and S attacks each argument not in S;
  • a semi-stable extension (S ∈ SST(F)) if S is complete and has a maximal range (w.r.t. set inclusion) among the complete extensions;
  • an ideal extension (S ∈ ID(F)) if S is the maximal (w.r.t. set inclusion) admissible extension that is a subset of each preferred extension.

Subtracks

The semantics included in the Main Track are weakly admissible (WAD), complete (CO), preferred (PR), stable (ST), semi-stable (SST), and ideal (ID).

The reasoning problems included in the Main Track are

  • DC-σ: Given F=(A,R) and a ∈ A, decide whether a is credulously accepted under σ and if it is, report a σ-extension that contains a. An argument a ∈ A is credulously accepted under σ if a is in at least one σ-extension.
  • DS-σ: Given F=(A,R) and a ∈ A, decide whether a is skeptically accepted under σ and if it is not, report a σ-extension that does not contain a. An argument a ∈ A is skeptically accepted under σ if a is in all σ-extensions.
  • SE-σ: Given F=(A,R), return a σ-extension of F.

Each of the following combinations of semantics and reasoning mode is a subtrack:

  • DC-{WAD|CO|ST|SST}
  • DS-{PR|ST|SST}
  • SE-{PR|ST|SST|ID}

Scoring

Each subtrack (i.e. semantics and reasoning mode combination) is ranked separately. The time limit is 1200 seconds CPU time per instance, and PAR-2 scoring is used. That is, the score for a given solver on an instance is 2 * 1200 if the solver timed out on this instance, and otherwise the CPU running time of the solver on this instance in seconds. The score for a solver on a subtrack is the sum of the solver’s scores over all instances of the subtrack. The winner is the solver with the lowest score.

No-Limits Track

The No-Limits Track is a more permissible version of the Main Track and has the same problems as subtracks. In particular, solvers are allowed to run on multiple cores and can be portfolio-based (i.e. combine the usage of several solvers). The Scoring is otherwise the same as for the Main Track, but wall-clock time is used instead of CPU time.

Heuristics Track

Problem Setting

Abstract argumentation, as defined above. The time limit for the track is lower than for the other tracks, namely 60 seconds CPU time per instance.

Subtracks

Semantics: CO, PR, ST, SST, and ID. Reasoning modes: DC-σ and DS-σ.

Each of the following combinations of semantics and reasoning mode is a subtrack:

  • DC-{CO|ST|SST|ID}
  • DS-{PR|ST|SST}

Scoring

Up to ICCMA 2025, solvers in the heuristics track were ranked by the number of correctly answered instances. Since every instance of the heuristics track is a binary decision problem, a solver can be regarded as a binary classifier that predicts YES or NO. The evaluation of binary classifiers is a well-studied problem in the machine learning community. Standard metrics from this field are therefore natural candidates for a scoring system of the heuristics track.

An important property of the benchmark data in this respect is its class distribution. Table 1 shows that in ICCMA 2025, some subtracks were almost perfectly balanced (DC-CO, DS-ST), while in others only 12–15% of the instances had the answer YES (DC-ID, DS-PR, DS-SST). On such imbalanced subtracks, a solver that always answers NO without any computation already answers most instances correctly.

For ICCMA 2027, we aim to make the instance set of every subtrack as balanced as possible. Nevertheless, it is sensible to use a scoring system that accounts for heterogeneous class distributions. We therefore base our scoring system on the well-established Matthews correlation coefficient (MCC), which takes the class distribution into account. It is described in detail below.

Table 1: Class distribution of the ICCMA 2025 heuristics track.
SubtrackYESNOShare of YES
DC-CO16015451.0%
DC-ST11320136.0%
DC-SST14916447.6%
DC-ID3827012.3%
DS-PR3827512.1%
DS-ST15416049.0%
DS-SST4626614.7%

Notation

Let I={1,…,n} be the set of instances of a subtrack. For each instance i∈I:

  • yi∈{0,1} denotes the correct answer, where 1 stands for YES and 0 for NO.
  • oi∈{0,1,⊥} denotes the output of the solver, where ⊥ stands for a timeout (more than 60 seconds CPU time), a crash, or an output that cannot be parsed as YES or NO.

A missing answer is counted as a misclassification. The effective prediction y^i of a solver on instance i is

y^i = { oi if oi∈{0,1}, 1−yi if oi=⊥.

The confusion matrix of a solver is given by

TP =|{i∈I∣y^i=1∧yi=1}|, FN =|{i∈I∣y^i=0∧yi=1}|, TN =|{i∈I∣y^i=0∧yi=0}|, FP =|{i∈I∣y^i=1∧yi=0}|,

with TP+TN+FP+FN=n.

Matthews Correlation Coefficient

The Matthews correlation coefficient (MCC) measures the correlation between the predicted and the correct answers. It is defined as

MCC = TP·TN−FP·FN (TP+FP) (TP+FN) (TN+FP) (TN+FN) ,

where MCC=0 if the denominator is 0. It holds that MCC∈[−1,1]. A value of 1 corresponds to a perfect prediction, 0 to a constant answer or a prediction that is on average no better than random guessing, and −1 to a prediction that is wrong on every instance.

Ranking

In each subtrack, solvers are ranked by their MCC in descending order. Ties are broken by the cumulative CPU time over all instances of the subtrack (lower is better), where each instance with output ⊥ due to a timeout is counted with 60 seconds. The solver ranked first wins the subtrack.

Dynamic Track

Problem Setting

Abstract argumentation, as defined above. Dynamic changes to an initial AF and acceptance queries are issued by different applications via IPAFAIR, an API for incremental reasoning in abstract argumentation. Please see ipafair.py in the repository for more details.

Subtracks

The subtracks in the Dynamic Track are DC-CO, DS-PR, DC-ST, and DS-ST. Each subtrack consists of applications which call an AF solver which implements the interface, applying changes to the underlying AF between solver calls. The name of the subtrack determines the allowed reasoning task and semantics. For example, in the DS-PR subtrack, the AF solver is initialized using the preferred semantics, and only solve_skept() calls are allowed with a single query argument. Both the query argument and the underlying AF may change between these calls.

Scoring

Each subtrack (i.e. semantics and reasoning mode combination) is ranked separately. The time limit is 1200 seconds CPU time per instance, and PAR-2 scoring is used. That is, the score for a given solver on an instance is 2 * 1200 if the solver timed out on this instance, and otherwise the CPU running time of the solver on this instance in seconds. The score for a solver on a subtrack is the sum of the solver’s scores over all instances of the subtrack. The winner is the solver with the lowest score.

ABA Track

Problem Setting

Assumption-based Argumentation (ABA) [Bondarenko et al 97] and the corresponding semantics are defined as follows.

An ABA framework is a tuple F = (L,R,A,‾) where

  • L is a set of literals
  • R is a set of rules
  • A ⊆ L is a (non-empty) set of assumptions
  • ‾ is a mapping from assumptions to literals, that represents a notion of contrariness

A rule h ← b1,...,bn is made of literals h, b1,... bn ∈ L, and n ≥ 0. We call h the head of the rule and b1,... bn the body. A rule without a body (i.e. h ←, or alternatively h ←⊤) is a fact. We consider only flat ABA frameworks, i.e. ABA frameworks where the head of a rule is never an assumption.

A sentence a ∈ L is derivable from a set X ⊆ A via rules R, denoted by X ⊢ a, if a ∈ X or there is a sequence of rules (r1,...,rn) such that head(rn) = a and for each rule ri we have ri ∈ R and each sentence in the body of ri is derived from rules earlier in the sequence or in X. A set of assumptions A1 attacks a set of assumptions A2 if the contrary of some a ∈ A2 is derivable from A1. A set of assumptions A1 defends an assumption a if A1 attacks every set of assumption that attacks a.

The semantics for an ABA framework F can be defined as follows. Given a set of assumptions X ⊆ A,

  • X ∈ CF(F) if it does not attack any of its elements;
  • X ∈ AD(F) if X ∈ CF(F) and X defends all its elements;
  • X ∈ CO(F) if X ∈ AD(F) and X contains every assumption it defends;
  • X ∈ PR(F) if it is a ⊆-maximal admissible set of F;
  • X ∈ ST(F) if X ∈ CF(F) and X attacks every assumption not in X.

We call a set of assumptions in σ(F) a σ-extension for semantics σ ∈ {CO, PR, ST} (complete, preferred and stable, respectively).

Subtracks

Semantics: CO, PR, ST.

The reasoning tasks for the ABA Track are

  • DC-σ: Given ABF F=(L,R,A,‾) and s ∈ L, decide whether s is credulously accepted under σ.
  • DS-σ: Given ABF F=(L,R,A,‾) and s ∈ L, decide whether s is skeptically accepted under σ.
  • SE-σ: Given ABF F=(L,R,A,‾), return a σ-extension of F.

Each of the following combinations of semantics and reasoning mode is a subtrack:

  • DC-{CO|ST}
  • DS-{PR|ST}
  • SE-{PR|ST}

Scoring

Each subtrack (i.e. semantics and reasoning mode combination) is ranked separately. The time limit is 1200 seconds CPU time per instance, and PAR-2 scoring is used. That is, the score for a given solver on an instance is 2 * 1200 if the solver timed out on this instance, and otherwise the CPU running time of the solver on this instance in seconds. The score for a solver on a subtrack is the sum of the solver’s scores over all instances of the subtrack. The winner is the solver with the lowest score.

Fixed-SAT-Solver Comparison

We encourage all solvers making use of SAT solvers to participate in this comparison. Any solver can be marked for taking part in the Fixed-SAT-Solver comparison. Then the solver will be evaluated normally and, additionally, for each (sub-)track (except No-Limits) all participating solvers in the Fixed-SAT-Solver comparison will be compared separately. This is meant to evaluate solvers relying heavily on SAT solvers in a more uniform comparison. Otherwise, the choice of different SAT solvers might lead to differences in performance, making fair comparisons of such solvers difficult.

If a solver takes part in the Fixed-SAT-Solver comparison, the solver must implement the IPASIR interface for incremental SAT solvers. For examples, we provide a simple example, and also see the IPASIR repository. The solver will then interact with a SAT solver only with the IPASIR interface and be compiled with a specific SAT solver for ICCMA 2027. Taking part in the Fixed-SAT-Solver comparison excludes then use of any other search engine similar to SAT solvers (like answer set programming). Usage of a SAT solver is limited to usage of the IPASIR interface (i.e., the solver may only use the interface to delegate computationally intensive computations). The organizers reserve the right to remove submissions to the Fixed-SAT-Solver comparison based on a case-by-case analysis.

References

[Dung 95] P. M. Dung, On the Acceptability of Arguments and its Fundamental Role in Nonmonotonic Reasoning, Logic Programming and n-Person Games. Artif. Intell. 77(2): 321-358 (1995)

[Bondarenko et al 97] A. Bondarenko, P. M. Dung, R. Kowalski, F. Toni, An Abstract, Argumentation-Theoretic Approach to Default Reasoning. Artif. Intell. 93: 63-101 (1997)