GBEA Competition PPSN 2020

Participate in our competition, it’s a WIN-WIN situation

Why?
  • Participants match up with other researchers who think they have the best optimisation algorithm available
  • Organisers collect data from test suites to further improve the benchmark and push real-world benchmarking a step forward

And there are more reasons: Games are a very interesting topic that motivates a lot of research and have repeatedly been suggested as testbeds for AI algorithms. Key features of games are controllability, safety and repeatability. Moreover, they feature the ability to simulate properties of real-world problems such as measurement noise, uncertainty and multiple objectives.

A popular topic within the area are game optimisation problems, where the task is to find solutions representing game content that optimise a given objective function. That’s why we compiled the Game Benchmark for Evolutionary Algorithms (GBEA), which provides you with the chance to try your algorithm in real-world-like scenarios.

Competition Details

What is the competition about?

  • Single and bi-objective suites from two interesting games (TopTrumps and Mario)
  • Continuous and integer search spaces, depending on game
=> 2 x 2 = 4 combinations (tracks) all together
You are of course free to choose one or multiple tracks that interest you most!

What exactly is a contribution? What do you need to submit?

  • COCO Logs
    Yep, we use COCO to compare results. GBEA is integrated in COCO as four stand-alone problem suites. This allows for an easy comparison of algorithms.
  • Access to the code
    We would like access (link to a repository, the code sent, or at least a description of it) to reproduce or at least retrace the results to determine the winners.

When?
  • Submission deadline: 07. August 2020
  • Winner announcement: 05. September 2020

Evaluation Criteria

There are several usecases for (game) optimisation algorithms that we intend to capture with our benchmark. However, for the competition, we are interested in finding algorithms that are best in specific usecases defined below. These usecases dictate how we evaluate the submitted entries. For different iterations of the competition, we will vary these usecases in order to identify strengths and weaknesses of different algorithms and to use the benchmark to its full potential.

For our PPSN 2020 competition, we will be focusing on the following usecase.

General target usecase

We will compute the average runtime (aRT) for each function to reach specified targets, averaged across instances. We then divide each aRT value by the search space dimension and take the average across all functions and dimensions. We use the resulting value to find the winner of the competition, but will provide details on achieved aRTs per function and dimension for all entries.

The motivation for this usecase is as follows. If an algorithm for generating content (such as a Mario level or a Top Trumps deck) is integrated in a game, the goal is usually to provide replay value by varying the content. In this context, it is not necessary to find the single solution that optimises the designer's objectives, but instead, it is important that a "good enough" solution can be found as fast as possible. "Good enough" in this case can be defined in relation to the values achieved by a baseline algorithm. Additionally, ideally, the same algorithm can find solutions across different objectives.

Based on this usecase, we define the following task: Find solutions of sufficient quality (as specified by the target) as quickly as possible. The target for each function is the best value found by random search after 1000 evaluations per dimensions. This has proven to be reasonably difficult in previous experiments. The algorithm that is able to reach these targets the earliest across all functions will win this usecase. Winners will be determined independently for each suite (rw-top-trumps, rw-top-trumps-biobj, rw-mario-gan, rw-mario-gan-biobj).

The nitty-gritty details

  • Follow the GBEA installation instructions using your preferred language (Python, C/C++, Java or Matlab/Octave). Make sure you are using the correct branch of the repository (gbea). At the end of the instructions, all GBEA suites will be installed and you will have run an example experiment with the default algorithm. Post-processing is not required for the competition.
  • Modify the example experiment implementation and add the algorithm you want to enter the competition with. Replace the call to the random search optimizer in the example experiment file (in folder coce-experiments/build/[chosen-language]) by a call to your algorithm. Update the output result_folder, the algorithm_name and algorithm_info of the observer options in the example experiment file.
  • Check the obtained results in [result_folder]. If you want, you can post-process the results in order to gain a better understanding of your algorithm's performance. To do that, install and execute the COCO post-processing tool following these instructions. Do not worry if the curves look flat, some targets are very difficult or even impossible to reach.
  • Submit to the competition by sending your entry to gamesbench.info@gmail.com. Your entry will automatically be considered for all tracks of the competitions you submit the logs for. It should consist of:
    • The generated COCO logs for the GBEA suites (rw-top-trumps, rw-top-trumps-biobjs, rw-mario-gan, rw-mario-gan-biobj) relevant for the track. This means all output produced by COCO using the "bbob" / "bbbob-biobj" logger.
    • A short description of your algorithm
    • If possible, a link to your repository, the source code or a binary of your algorithm so we can verify the results