Demo of using the CDE auto-packaging tool with MLcomp

This zip file contains a simple demo of how to use the CDE auto-packaging tool with MLcomp. CDE allows you to easily run your programs on MLcomp even when they use languages, extensions, or libraries that are not installed on the default Linux worker machines. Here are the basic steps involved:

  1. Run your program under CDE supervision on your x86-Linux machine to create a self-contained package within the cde-package/ sub-directory. This package contains all of the dependencies (e.g., shared libraries, language run-times) required to re-execute your program on any contemporary x86-Linux machine.
  2. Create a run wrapper script that copies input files into your CDE package, invokes your program, and then copies the output files out of the package. This wrapper is necessary because CDE-packaged programs can only access files within their own package.
  3. Create a metadata file like this one.
  4. Zip up all of your files and upload to MLcomp. Use 'zip -ry' to preserve symbolic links, since CDE packages might break if symlinks get converted into regular files.

For this demo, I've created a simple Python script located in: cde-package/cde-root/home/pgbovine/python-mlcomp-classifier/

This script implements the dumbest possible classifier (always returning '1' regardless of input). However, the crucial line that we care about is: import networkx

This line tells Python to import the NetworkX graph library. When this script is run natively on the MLcomp worker machine, it will crash since NetworkX is not installed. However, since NetworkX is installed on my personal Linux machine, I can run on my machine with CDE to package up all of its dependencies, most notably the version of Python with NetworkX installed for it.

To execute the program within the CDE package, we must first change into this working directory: cde-package/cde-root/home/pgbovine/python-mlcomp-classifier/

and then execute ./python.cde, which will execute the script using the version of Python that resides within the package. Note that running python will not work on the MLcomp worker machine, since that will execute the native version of Python, which does not have NetworkX.

The final required component is the top-level run script required by the MLcomp program interface. For this demo, we have created a BASH script that wraps around ./python.cde Here is the tricky part about writing a run script to work with CDE: CDE-packaged programs can only access files within the package. This means that run must properly copy the input files from their original locations into the package, execute your CDE-packaged program, and then copy the resulting output files back out of the package into the location expected by MLcomp. Please see the run script within this directory for an example of how this is done. Fortunately, it's pretty simple once you understand this concept.

Note that although this demo is for Python, CDE is language-agnostic; it will work with any set of programs on your x86-Linux machine. Please email me at if you need help getting CDE working with MLcomp.

Login required!

Run this program on a dataset Arrow_right

Existing runs on cde-mlcomp-demo 1-12 of 12   Action_refresh_blue
ID Program Dataset Tuned hyper. User Updated << Status Total time Memory Error
Run #28790 cde-mlcomp-demo Sca Motion no andresvila 4y40d ago done 6s 38M 0.893
Run #21445 cde-mlcomp-demo ralign-random no internal 4y325d ago done 0.503
Run #17504 cde-mlcomp-demo songsAndEmotions#3 no internal 4y325d ago done 0
Run #17503 cde-mlcomp-demo songsAndEmotions#6 no internal 4y325d ago done 1
Run #16884 cde-mlcomp-demo songsAndEmotions#1 no internal 4y325d ago done 1
Run #16240 cde-mlcomp-demo DAS_cui_hr_bi_lab no linflyer 5y53d ago done 5s 56M 0.601
Run #14615 cde-mlcomp-demo Synthetic 10% Density, Small, Few Labels no internal 5y316d ago done 11s 38M 0.817
Run #14614 cde-mlcomp-demo Synthetic 10% Density, Small, Many Labels no internal 5y316d ago done 29s 38M 0.952
Run #14611 cde-mlcomp-demo annealing no internal 5y316d ago done 1m7s 38M 0.533
Run #14613 cde-mlcomp-demo statlog-shuttle no internal 5y316d ago done 33s 38M 0.208
Run #14612 cde-mlcomp-demo mushroom no internal 5y316d ago done 49s 38M 0.485
Run #13860 cde-mlcomp-demo multiclass-sample no pgbovine 5y364d ago done 7s 44M 0.500

Program contents Arrow_right


Must be logged in to post comments.