The First International Competition on Computational Models of Argumentation (ICCMA'15) will be conducted in the first half of 2015. The results will be presented at the workshop "Theory and Applications of Formal Argument" (TAFA'15) which will be co-located with the 24th International Joint Conference on Artificial Intelligence (IJCAI'15).
ICCMA'15 will focus on reasoning tasks in abstract argumentation frameworks, cf. [Dung:1995]. Submitted solvers will be tested on several artificially generated argumentation frameworks and some frameworks formalizing real-world problems.
For the first instance of the competition we will evaluate solvers based on their performance in solving the following computational tasks:
The above computational tasks are to be solved with respect to the following standard semantics:
Developers of solvers may decide to only provide support for a subset of the above computational tasks and/or semantics. For each category (task+semantics) we will provide a separate ranking of the submitted solvers.
The competition will be conducted using the open source benchmark suite probo.
2014/12/22: As an early christmas present, we uploaded a second set of benchmark examples. Our benchmark generator has been tweaked a bit and the new set contains more interesting examples than the first set (e.g. more and different extensions).
We also provide the ground truth for each argumentation graph in the archive under the folder "solutions". For each semantics (CO, PR, GR, ST) and each argument graph, there is one file in "solutions" containing the set of all extensions wrt. the given semantics. We also implemented a solver that reads the solutions from these files in order to compare the output directly in probo.
You can download the archive with the new benchmarks and their solutions here (the first number of the filename is an indicator for the hardness of the graph: the larger the harder). We also uploaded the solutions to the first benchmark set at (only solutions to the "rdm1000" test cases are missing).
2014/11/21: We uploaded an archive containing some example graphs in TGF and APX format to the competition website (zip).
We invite all potential participants to try out these examples with their solver. The graphs are drawn from different probabilistic models with different sizes and there are also some graphs taken from real-world debates. For now, we will not disclose the exact parameters on how the random graphs are generated in order to avoid an optimisation of solvers towards specific graph models. However, in terms of structure these example graphs are representative for the graphs that will actually be used in the competition.
As for the size of the graphs, we included in the archive graphs ranging from 20 arguments to more than 1000 arguments. As ICCMA'15 will be the first of its kind, we would be very happy if you can tell us a bit how your solvers perform on the different sizes of graphs. We currently estimate that we will set a time-out of 10-30 minutes for each computational problem on each graph (e.g. computing all preferred extensions on one graph). It would be very good if you can give us some feedback about how large the graphs can be while still being solved in the given time-out (please send your feedback to email@example.com).
As for the file formats, we currently only support TGF and APX (see iccma15notes_v3.pdf for a description) and removed support of the CNF format as there are ambiguities when converting CNF to the other formats. If you wish to have another format supported, please do not hesitate to express your wish on our mailinglist firstname.lastname@example.org. We can easily add support for any other format.