The International Planning Competition is an important event for the Planning and Scheduling Community. As such, it receives a lot of attention and, as a matter of fact, it has received an outstanding number of submissions in the Seventh edition.
While the Competition is intended, in the very end, to declare both a winner and a runner-up per track, its very important role is to gather data and disseminate it to all researchers. Besides, it has been also used to promote or to review standards and to foster (sometimes vivid) discussions. At last, but not least, the problems and domains used at the various editions of the International Planning Competition have been used as a reference in most experiments by the vast majority of practicioners and the practice of choice to show progress with respect to a particular planner is to show the performance for the same set of problems of the last Competition.
Therefore, while it is mandatory to provide the names of the research team and the code of the top performers at the different tracks, there are various advantages in distributing also the code used to evaluate them. Precisely, the target of this documentation pages is to serve as a reference point for using and extending the code used at the Seventh International Planning Competition.
Thanks to a common repository which shares the code for researchers to make experiments and for the organizers of any International Planning Competition to evaluate every planner:
All competitors can test their software before submitting it to the International Planning Competition —or IPC.
This would significantly improve the overall running time of an International Planning Competition. Common errors are scripts without a shebang line, executables without the right permissions set, compilation errors (either due to the assumption that the target machine have a number of libraries available or just by mistaking whether the target architectures are 32/64 bits, etc.) and a long etcetera. However, if the organizers of an IPC set up a SVN repository and allow competitors to enter their computers before the submission deadline, they can actually test their software just by executing one script. The script produces a log file which can then be used as a ticket for registering.
Everyone can easily run tests on his planner to make comparisons with other planners
This sets up a common framework for everyone to make experiments making comparisons fairer. Of course, planners under development are expected to do not be exposed to the general public. Therefore, one of the main drivers of the software of the Seventh International Planning Competition was the ability to work with respect to any SVN. This way, all research teams can create their own private repostiories so that private experiments can be conducted. Then, if desired, the results can be made available to the public by importing them into a svn repository.
The results of all the experiments are permanently stored in a repository publicly available
By providing additional tools to allow everyone to examine the available data there is no need to be repeating experiments over and over. Instead, they are performed only once and then permanently stored at the repository. Of course, the same principle applies to private repositories created by anyone. This also means that the results of many of these scripts can be deleted from the local computer once the experimentation is over since the planners/domains and results are stored in the SVN.
In particular, this principle is essential to the organizers of the International Planning Competition. Allowing everyone to inspect the SVN repository, the evaluation becomes more transparent. As a matter of fact, the contents of the working SVN repository used during the Seventh International Planning Competition was dumped to a new location preserving all revisions.
The idea of creating software for automating the tasks necessary for evaluating planners (from automatically compiling planners, creating testsets, validating results and even generating pdf documents with an overall view of the performance of each entry) has been already considered in other editions of the IPC.
In particular, the organizers of the Sixth International Planning Competition developed a lot of code for automating all these tasks. The architecture devised at that time consisted of three facilities:
1. A SVN repository which stores all the necessary information. Not only domains, planners, and scripts necessary for running the competition, but also other folders which store and share private data of the organizers
2. A wiki which is used to publish and share information among the organizers and the community. Thanks to the Access Control Lists the wiki allows different levels for sharing information among users.
3. A cluster or, in general, the hardware premises where the competition is actually run. This can be either a single computer or a cluster or a network of clusters.
The software of the Seventh International Planning Competition is strongly influenced from all the ideas developed at the precedent edition of the IPC. In fact, the first chapter is devoted to a discussion of these parts and how they relate to each other.