MLcomp version alpha Statistics: 2040 users, 637 datasets, 389 programs, 18661 runs (0 queued, 2 running), 0 workers

Not registered? Sign up for a free MLcomp account.
Forgot password?

1. How does MLcomp work?
   • You upload programs and/or datasets.
   • Programs are run on datasets.
   • Various performance metrics are reported on these runs.
2. What is the goal of MLcomp?
   MLcomp aims to improve research in the following ways:

   • Transparency: all metrics (error, running time, etc.) of all programs and datasets are shown side-by-side for comparison.
   • Reproducibility: one can download any program and dataset, which are stored with their exact settings/formats, to verify results.
   • Collaboration: anyone can upload programs and/or datasets which can be run on and compared with existing programs and datasets.
3. Is there a forum or mailing list?
   Yes.
4. How are programs and datasets rated?
   Intuitively, good programs have low error rates and good datasets distinguish programs. Specifically, the rating of a dataset is related to the standard deviation of the error rates of programs run on that dataset. For each dataset, a Gaussian is fit on the error rates of the programs, resulting a percentile (0 to 100) for each program. The rating of a program is the average percentile across all the datasets on which it is run. This is essentially a first cut at rating programs and we intend to improve it. Suggestions are welcome.
5. What is a domain?
   A domain defines a particular machine learning task, for example, binary classification or collaborative filtering. Each domain is equipped with the following:

   • a particular dataset format,
   • a particular program interface, and
   • standard evaluation metrics.

   For example, in the BinaryClassification domain, datasets are in SVMlight format, and program interface must read these datasets and output binary labels, and the evaluation metric is misclassification error rate.
6. What is a run?
   A run is the execution of a program on a dataset. For example, for the binary classification domain, a run includes both training, testing, and evaluation. Each run produces a log file and a set of statistics (time, memory, error rate, etc.), which can be viewed on the appropriate page of the run.
7. How do I create a run?
   There are two ways:

   1. Go to the page for a program. At the bottom of the screen, choose the dataset to run on.
   2. Go to the page for a dataset. At the bottom of the screen, choose the program to run it on.