MICCAI 2020 Combined Radiology and Pathology Classification Challenge

Organized by bbearce - Current server time: Aug. 12, 2020, 3:31 a.m. UTC

Previous

Training
April 22, 2020, midnight UTC

Current

Validation
July 9, 2020, midnight UTC

Next

Test
Aug. 24, 2020, midnight UTC

Official Miccai Website

Challenge Design Document

NOTE: Please see the Terms and Conditions section for updated timeline for the validation and test phases and the short paper submission instructions. 

Brain cancer is a fatal and complex disease. Diagnosis and grading of brain tumors is traditionally done by pathologists, who examine tissue sections fixed on glass slides under a light microscope. While this process continues to be widely applied in clinical setting, it is not scalable to translational and clinical research studies involving hundreds or thousands of tissue specimens. Computer-aided classification has the potential to improve tumor diagnosis and grading process, as well as to enable quantitative studies of the mechanisms underlying disease onset and progression.

The goal of CPM-RadPath 2020 is to assess automated brain tumor (glioma) classification algorithms, when data from both radiology (MRI) and histopathology (digital pathology) imaging are used. The algorithmic performance will be evaluated based on a retrospective cohort of three types of gliomas, i.e., glioblastoma, oligodendroglioma, and astrocytoma. Participants are asked to classify a cohort of brain tumor cases into three sub-types: Glioblastoma, Oligodendroglioma, and Astrocytoma.

The significance of CPM-RadPath 2020 challenge is the integrated use of two different types of imaging, at different spatial resolutions, both of which are key in routine clinical diagnosis and management of brain tumor patients. The selection and number of features from each imaging type will be left to the participants. However, they are required to use at least one feature from each imaging data type in their algorithms, but the decision about how information from the two imaging types is integrated is left to the participants. The challenge will consist of three phases: a training phase, a validation (or fine-tuning) phase, and a final test phase. Please see the Evaluation section for details. 

Organizing Committee

  • Keyvan Farahani, Center for Biomedical Informatics and Information Technology, National Cancer Institute, National Institutes of Health
  • Tahsin Kurc, Stony Brook Cancer Center
  • Spyridon Bakas, Center for Biomedical Image Computing and Analytics, University of Pennsylvania
  • Benjamin Aaron Bearce, Massachusetts General Hospital
  • Jayashree Kalpathy-Cramer, MGH, Harvard University
  • John Freymann, Fredrick National Lab for Cancer Research
  • MacLean P. Nasrallah, University of Pennsylvania
  • Joel Saltz, Stony Brook Cancer Center
  • Eric Stahlberg, Fredrick National Lab for Cancer Research
  • George Zaki, Fredrick National Lab for Cancer Research
  • Russell T. Shinohara, University of Pennsylvania

 

The challenge will consist of three phases: a training phase, a validation (or fine-tuning) phase, and a final test phase.  

Training Phase: 

In the training phase, participants will be provided paired radiology-pathology image data and ground truth labels to train their models. Please follow this link to access the training dataset. 

Validation Phase: 

An independent set of paired radiology-pathology data will be made available to the participants with the intention to allow them assess the generalizability of their methods in unseen data. Multiple submissions of classification results are allowed to the challenge platform.

At the end of the validation phase, participants will have to evaluate their methods on the training and validation datasets, and submit a short paper (~8 LNCS pages; see Terms and Conditions for the paper formatting and submission instructions) describing their method and results. The short paper must include the name of the participating team as registered on the challenge platform. Classification results submitted from this team in the test phase will be used for scoring and ranking. Participants who do not submit a short paper at the end of the validation phase will not be eligible for the test phase and final ranking. 

Test Phase: 

A test dataset of paired radiology-pathology images will made available to each participating team that submitted a short paper. The participants will analyze the images using their local computing infrastructure and will have to submit their classification results to the online evaluation portal. Each participating team will be allowed to submit their results to the platform only once. 

Scoring and Ranking: 

Three metrics will be used to score and rank participants. 

1. F1-Score
2. Cohen's Kappa
3. Balanced Accuracy

These metrics were chosen after extensive consultation with biostatisticians and data scientists in the organizing team and because they are widely used in evaluating performance of algorithms and manual raters.

- As a multi-class classifier, the F1 score is the harmonic mean of the precision and recall, where an F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0.

- Cohen's Kappa addresses classification by chance. A kappa of 1 indicates perfect agreement, whereas a kappa of 0 indicates agreement equivalent to chance. A limitation of kappa is that it is affected by the prevalence of the finding under observation.

- Balanced Accuracy is calculated as the average of the proportion corrects of each class individually.

Participants will be ranked for each metric, and then the average of these ranks will be computed towards the final ranking of the participants. For example, if participant A is ranked 1st in the F1 score, 3rd in the Cohen's Kappa score, and 5th in Balanced Accuracy score, the rank of participant A will be (1+3+5)/3=3. 

Terms and Conditions

Update: (New Short Paper Instructions and Timeline)

Dear Participants,

Due to covid19 the MICCAI 2020 challenge will be pushed back a month. We are hoping this eases some of the stress created by the virus and hope that it doesn't create more. The new due dates are as follows:

[19 Aug]
Provide the submission of the short papers.

[24 Aug - 4 Sept]
Test Phase: Release of testing data

[4 Sept - 16 Sept]
Docker Evaluation Phase: Collect docker images from the top 6 scoring participants.

[16 Sept]
Top 3 ranking participants will be contacted for preparing slides for oral presentation.

The test data will be made available for the full duration of 24 Aug - 4 Sept and will be released at midnight. For the top scoring participants, the docker submission runtime after the test phase will have a 48hr time limit to perform the same inference that generated your test phase submission (on our system). Late submissions, occurring after 4 Sept UTC midnight, will not be accepted for the test phase.

PS: Regarding the test phase data itself; we realize that there is a concern about unfairness as the test data was briefly released last year. Unfortunately there is not much we can do about it. We planned to have additional test data but covid19 has forced us to scale down our plans. There are several ways a participating team can cheat. We expect a certain level of academic integrity from the participating teams.

By participating in this challenge, each participant agrees to

  1. Submit a short paper (details regarding page limit, format, and other submission requirements are forthcoming) describing their algorithm by the end of the validation phase. The short paper must include the name of the participating team as registered on the challenge platform. Classification results submitted from this team in the test phase will be used for scoring and ranking. Participants who do not submit a short paper at the end of the validation phase will not be eligible for the test phase and final ranking.  

    The participants will submit a short paper (~8 LNCS pages) together with the LNCS Consent to Publish (link1) describing their method and results, to the BrainLes CMT submission system (link2). The submissions should choose CPM-RadPath as the "Track". All paper submissions should use the LNCS template, available both in LaTeX (link3) and in MS Word (link4) format, directly from Springer (link5).

    Updated date for submission: 19 Aug

  2. Provide a docker image via DockerHub. At the end of the test phase, the organizers will contact the top scoring teams and ask them to provide a docker image containing their algorithm. The docker image will be used to verify the test scores of the participant. If a participant does not wish to provide a docker image, the participant will not be included in the final ranking. 

    Update date for submission: 4 Sept - 16 Sept

  3. Present their algorithm and challenge results at the MICCAI BrainLes 2020 workshop at MICCAI 2020, if the participant's entry is selected for presentation.  
  4. DO NOT register a team more than once. The submission limits are per team and it is considered cheating if you have multiple accounts for your instituition. Penalties include disqualification. 

Training

Start: April 22, 2020, midnight

Validation

Start: July 9, 2020, midnight

Docker Submission

Start: Sept. 4, 2020, midnight

Description: Participants asked to participate will be asked to submit a docker image for the final validation of their model

Test

Start: Aug. 24, 2020, midnight

Competition Ends

Sept. 16, 2020, midnight

You must be logged in to participate in competitions.

Sign In