NOTE: Please see the Terms and Conditions section for updated timeline for the validation and test phases and the short paper submission instructions.
Brain cancer is a fatal and complex disease. Diagnosis and grading of brain tumors is traditionally done by pathologists, who examine tissue sections fixed on glass slides under a light microscope. While this process continues to be widely applied in clinical setting, it is not scalable to translational and clinical research studies involving hundreds or thousands of tissue specimens. Computer-aided classification has the potential to improve tumor diagnosis and grading process, as well as to enable quantitative studies of the mechanisms underlying disease onset and progression.
The goal of CPM-RadPath 2020 is to assess automated brain tumor (glioma) classification algorithms, when data from both radiology (MRI) and histopathology (digital pathology) imaging are used. The algorithmic performance will be evaluated based on a retrospective cohort of three types of gliomas, i.e., glioblastoma, oligodendroglioma, and astrocytoma. Participants are asked to classify a cohort of brain tumor cases into three sub-types: Glioblastoma, Oligodendroglioma, and Astrocytoma.
The significance of CPM-RadPath 2020 challenge is the integrated use of two different types of imaging, at different spatial resolutions, both of which are key in routine clinical diagnosis and management of brain tumor patients. The selection and number of features from each imaging type will be left to the participants. However, they are required to use at least one feature from each imaging data type in their algorithms, but the decision about how information from the two imaging types is integrated is left to the participants. The challenge will consist of three phases: a training phase, a validation (or fine-tuning) phase, and a final test phase. Please see the Evaluation section for details.
The challenge will consist of three phases: a training phase, a validation (or fine-tuning) phase, and a final test phase.
In the training phase, participants will be provided paired radiology-pathology image data and ground truth labels to train their models. Please follow this link to access the training dataset.
An independent set of paired radiology-pathology data will be made available to the participants with the intention to allow them assess the generalizability of their methods in unseen data. Multiple submissions of classification results are allowed to the challenge platform.
At the end of the validation phase, participants will have to evaluate their methods on the training and validation datasets, and submit a short paper (~8 LNCS pages; see Terms and Conditions for the paper formatting and submission instructions) describing their method and results. The short paper must include the name of the participating team as registered on the challenge platform. Classification results submitted from this team in the test phase will be used for scoring and ranking. Participants who do not submit a short paper at the end of the validation phase will not be eligible for the test phase and final ranking.
A test dataset of paired radiology-pathology images will made available to each participating team that submitted a short paper. The participants will analyze the images using their local computing infrastructure and will have to submit their classification results to the online evaluation portal. Each participating team will be allowed to submit their results to the platform only once.
Scoring and Ranking:
Three metrics will be used to score and rank participants.
2. Cohen's Kappa
3. Balanced Accuracy
These metrics were chosen after extensive consultation with biostatisticians and data scientists in the organizing team and because they are widely used in evaluating performance of algorithms and manual raters.
- As a multi-class classifier, the F1 score is the harmonic mean of the precision and recall, where an F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0.
- Cohen's Kappa addresses classification by chance. A kappa of 1 indicates perfect agreement, whereas a kappa of 0 indicates agreement equivalent to chance. A limitation of kappa is that it is affected by the prevalence of the finding under observation.
- Balanced Accuracy is calculated as the average of the proportion corrects of each class individually.
Participants will be ranked for each metric, and then the average of these ranks will be computed towards the final ranking of the participants. For example, if participant A is ranked 1st in the F1 score, 3rd in the Cohen's Kappa score, and 5th in Balanced Accuracy score, the rank of participant A will be (1+3+5)/3=3.
Data Usage Agreement / Citations
You are free to use and/or refer to the CPM-RadPath datasets in your own research, provided that you always cite the following manuscript:
T. Kurc, S. Bakas, X. Ren, A. Bagari, A. Momeni, Y. Huang, L. Zhang, A. Kumar, M. Thibault, Q. Qi, Q. Wang, A. Kori, O. Gevaert, Y. Zhang, D. Shen, M. Khened, X. Ding, G. Krishnamurthi, J. Kalpathy-Cramer, J. Davis, T. Zhao, R. Gupta, J. Saltz, K. Farahani. "Segmentation and classification in digital pathology for glioma research: challenges and deep learning approaches". Frontiers in neuroscience, p.27, 2020.
Due to covid19 the MICCAI 2020 challenge will be pushed back a month. We are hoping this eases some of the stress created by the virus and hope that it doesn't create more. The new due dates are as follows:
Provide the submission of the short papers.
[24 Aug - 4 Sept]
Test Phase: Release of testing data
[4 Sept - 16 Sept]
Docker Evaluation Phase: Collect docker images from the top 6 scoring participants.
Top 3 ranking participants will be contacted for preparing slides for oral presentation.
The test data will be made available for the full duration of 24 Aug - 4 Sept and will be released at midnight. For the top scoring participants, the docker submission runtime after the test phase will have a 48hr time limit to perform the same inference that generated your test phase submission (on our system). Late submissions, occurring after 4 Sept UTC midnight, will not be accepted for the test phase.
PS: Regarding the test phase data itself; we realize that there is a concern about unfairness as the test data was briefly released last year. Unfortunately there is not much we can do about it. We planned to have additional test data but covid19 has forced us to scale down our plans. There are several ways a participating team can cheat. We expect a certain level of academic integrity from the participating teams.
By participating in this challenge, each participant agrees to
The participants will submit a short paper (~8 LNCS pages) together with the LNCS Consent to Publish (link1) describing their method and results, to the BrainLes CMT submission system (link2). The submissions should choose CPM-RadPath as the "Track". All paper submissions should use the LNCS template, available both in LaTeX (link3) and in MS Word (link4) format, directly from Springer (link5).
Start: April 22, 2020, midnight
Start: July 9, 2020, midnight
Start: Aug. 20, 2020, midnight
Start: Sept. 1, 2020, midnight
Description: Participants asked to participate will be asked to submit a docker image for the final validation of their model
Start: April 1, 2021, 10:18 a.m.
Description: Open Ended Phase
Sept. 16, 2020, midnight
You must be logged in to participate in competitions.Sign In