How to add new problem?

Below we present a short tutorial for everybody who wants to add a new problem to OPTIL.io platform.

Let us know about your problem

Send as an e-mail with short description of the problem, including:

  • your username on OPTIL.io platform,
  • one to three sentences describing your problem,
  • number of test instances that you want to publish,
  • time and memory limits that you want to set for solutions,
  • any additional requirements that you have, for example availability of third-party libraries or specific programming languages.

Usually it is possible to publish any optimization problem. However, we can help you to improve its definition to better integrate it with OPTIL.io platform.

After we receive information about your problem we will grant you the author role, so that you can add problem definition using the administration interface.

Prepare the problem description

Describe the problem that you want to publish. It is much easier for our users if all problems are described in the similar way, that is why we prepared the template of the problem description. Use this template! In a template you can use Markdown format to apply formatting and LaTeX to format mathematical formulas. If you are new to Markdown there is a nice tuturial. For each problem you should define its standard name and short, abbreviated name consisting of several upper case letters and numbers. The short name is used in some views that does not allow to present longer names.

Together with the problem description you can provide HTML code presenting authors of the problem and their affiliations. The affiliations will be presented on the problem description in a tooltip visible after pointing authors names. Please use the following format to provide authors names:

Szymon Wasik<sup>1,2</sup>, Maciej Antczak<sup>1</sup>

and the following for affiliations:

<sup>1</sup>Institute of Computing Science, Poznan University of Technology<br/><sup>2</sup>Institute of Bioorganic Chemistry, Polish Academy of Sciences

If you want to add some pictures to the problem description you have to upload them first. From the author menu select Pictures and upload your files. After pictures are uploaded a Pictures panel presents the path that should be used as a path to the image in the problem description.

Define submit interval (waiting period)

For each problem you can define the waiting period that the user has to wait before submitting two consecutive solutions of the problem. Setting such interval is a good idea for problems that have many test instances with long time limits. You can define it in minutes using appropriate field in the edit description form. If you do not want to set this interval leave the field empty.

Prepare test instances

Prepare test instances describing test cases that will be solved by users' optimization algorithms. For each problem we recommend to prepare 30 to 50 instances that should satisfy the following conditions:

  1. Each instance should be saved in a single, separate file.
  2. Instances should be presented in simple, text format, to make it easy to use and parse them. We do not recommend using XML format. Take a look at several problems published at OPTIL.io to see some examples.
  3. For each instance the time limit for solving it should be defined. We recommend limits between 10 and 600 seconds, however, longer limits are also possible. After exceeding the time limit a solution solving this instance is killed. You should also define shorter, soft time limit after which the solution receives Linux SIGTERM signal. Users solving your problem can use this signal to know that this is a right time to stop computation and save results. For this purpose it should be 3 to 10 seconds shorter than the hard time limit.
  4. For each instance the memory limit should be defined. We recommend limits that are not higher than 2 GB, however, larger limits are also possible.
  5. For each instance the output limit should be defined. It is the maximal number of bytes that the user's submission is allowed to output. Usually it should be a bit larger than the possible size of a program output. It is used to kill wrong solutions, that output too many data, for example, because of infinite loop.

You can add instances and edit their parameters using the Edit instances form. You can also add many instances at once using Add pack option. All instances should be packed in one zip archive. It can be convenient if you have many instances and you do not want do add them one by one.

For each instance you have to define input and output file. Input file will be redirected to the standard input of the user's solutions. Output file will be available for the judge program that will evaluate the solution and can store some additional data, for example optimal solutions. If you do not need any additional data during the assessment process you can add empty output file.

Instances visibility

There are two types of instances that you can add - private and public instances. By default all instances are public, and it is a behavior recommended for most of the problem authors.

You can use both, public instances together with private ones to organize contests with some instances hidden from the participants. Beware: there is no point in adding private instances if you do not plan to organize the contest. The flow of such a contest is following:

  1. You add both, public and private instances.
  2. Participants can see evaluation results only for public instances.
  3. You can see evaluation results for both, public and private instances. To see results for private instances you have to log in using problem author's account, navigate to the problem description and check the private standing tab.
  4. After the contest ends, you can click the reveal private instances button to hide public instances and display the standing based on private instances.
  5. After revealing private instances, the problem will be evaluated only using private instances, and it will not be possible to see standing based on public instances anymore.

Prepare a judge

The judge is a program that evaluates the user's result and outputs single, double precision number being the value of the objective function. The judge should be written in any compilable language. For you convenience we attach a template of the judge program implemented in the C++. You should compile the judge as a static Linux binary and add it using the Add judge form.

The judge will be executed using the following command:

judge instance-input user-output instance-output usertime perftime

The meaning of the arguments passed to the judge is following:

  • instance-input - the path to the input file that was provided for the test instance by the problem author,
  • user-output - the path to the output file that was generated by the program submitted by the user,
  • instance-output - the path to the output file that was provided for the test instance by the problem author,
  • usertime - usertime in milliseconds used by the user's submission,
  • perftime - number of CPU instructions executed measured using Linux Performance Counter - much more stable measure than usertime.

Beware - do not check if the number of arguments is equal to 5, because additional statistics can be added in the future.

Submit the problem

After you provide all information described above you can send the problem for moderation. To do this press a Review button on the problems list. You will have a chance to add optional comment describing your problem. After sending the problem for moderation you will not be able to modify it. After moderation we will publish your problem and you will see it on the list of your submitted problems. However, we do not allow to modify any publicly available problem. To modify it you have to contact us and explain why you want to change it. Then we will add you such a possibility.

Is it free?

YES! Publishing problems on OPTIL.io is free for non-commercial use. If you want to use it to solve some industrial problem, sponsor some contest or use it commercially in other way, please contact us.