Instructions for submitting your Track 2 system

Prepare your ASR and diarization results for the dev and eval sets

  • Prepare the RTTM and Kaldi transcription files. If you are using Kaldi, they will be obtained by using local/ and local/, respectively.
  • For participants who don’t use Kaldi, please also strictly follow the Kaldi format transcriptions for the submission. We only accept the Kaldi format transcriptions as follows:
    $ head exp/chain_train_worn_simu_u400k_cleaned_rvb/tdnn1b_sp/decode_dev_beamformit_dereverb_diarized_2stage/scoring_kaldi/penalty_0.5/11.txt
    S02_U06.ENH-1-008054-008144 um
    S02_U06.ENH-1-008916-009050 to the other previous
    S02_U06.ENH-1-009064-009326 so it was like
    S02_U06.ENH-1-009838-010085 i think i g- is only to go right
    S02_U06.ENH-1-011824-012011 and if not she knows for everyone
    S02_U06.ENH-1-012580-013280 you can be done sometime previous between and a small bowl
    S02_U06.ENH-1-013285-013397 they probably
    S02_U06.ENH-1-013576-013717 i don't actually
    S02_U06.ENH-1-013957-014063 i dunno
    $ head exp/chain_train_worn_simu_u400k_cleaned_rvb/tdnn1b_sp/decode_eval_beamformit_dereverb_diarized_2stage/scoring_kaldi/penalty_0.5/11.txt
    S01_U06.ENH-1-002923-003095 do we have
    S01_U06.ENH-1-003657-003732 you want
    S01_U06.ENH-1-006512-006755 yeah the wok moline here
    S01_U06.ENH-1-012895-012970 or just
    S01_U06.ENH-1-013195-013495 yeah i put the sauce in a while cuz it's
    S01_U06.ENH-1-024428-024503 yeah
  • Please make sure to keep the utterance ID format as <session ID>_<array ID etc.>-<speaker ID>-<start time>-<end-time>. You can change <array ID etc.>, <start time>, and <end-time> with your own IDs or times, but do not change <session ID> and <speaker ID>.
  • We only accept the RTTM files as follows:
    $ head exp/dev_beamformit_dereverb_stats_seg_diarization/rttm
    SPEAKER S02_U06.ENH 1   2.170   1.125 <NA> <NA> 1 <NA> <NA>
    SPEAKER S02_U06.ENH 1   3.295   1.285 <NA> <NA> 4 <NA> <NA>
    SPEAKER S02_U06.ENH 1   4.600   1.125 <NA> <NA> 4 <NA> <NA>
    SPEAKER S02_U06.ENH 1   5.725   0.445 <NA> <NA> 2 <NA> <NA>
    SPEAKER S02_U06.ENH 1   6.460   4.420 <NA> <NA> 4 <NA> <NA>
    SPEAKER S02_U06.ENH 1  11.560   6.010 <NA> <NA> 4 <NA> <NA>
    SPEAKER S02_U06.ENH 1  18.700   3.760 <NA> <NA> 3 <NA> <NA>
    SPEAKER S02_U06.ENH 1  23.590   4.600 <NA> <NA> 3 <NA> <NA>
    SPEAKER S02_U06.ENH 1  28.360   2.620 <NA> <NA> 3 <NA> <NA>
    SPEAKER S02_U06.ENH 1  31.030   3.880 <NA> <NA> 3 <NA> <NA>
    $ head exp/eval_beamformit_dereverb_stats_seg_diarization/rttm
    SPEAKER S01_U06.ENH 1   0.000   3.375 <NA> <NA> 4 <NA> <NA>
    SPEAKER S01_U06.ENH 1   3.375   7.590 <NA> <NA> 2 <NA> <NA>
    SPEAKER S01_U06.ENH 1  10.965   2.250 <NA> <NA> 4 <NA> <NA>
    SPEAKER S01_U06.ENH 1  13.215   0.750 <NA> <NA> 3 <NA> <NA>
    SPEAKER S01_U06.ENH 1  13.965   3.000 <NA> <NA> 2 <NA> <NA>
    SPEAKER S01_U06.ENH 1  16.965   0.815 <NA> <NA> 4 <NA> <NA>
    SPEAKER S01_U06.ENH 1  17.920   1.480 <NA> <NA> 2 <NA> <NA>
    SPEAKER S01_U06.ENH 1  19.480   5.170 <NA> <NA> 2 <NA> <NA>
    SPEAKER S01_U06.ENH 1  24.850   2.740 <NA> <NA> 2 <NA> <NA>
    SPEAKER S01_U06.ENH 1  27.590   0.900 <NA> <NA> 1 <NA> <NA>
  • Make sure to use the same format. The start time (4th column), duration (5th), and speaker ID (8th) must remain in the same columns. Also, please keep the session ID (e.g., S01, S21) in the 2nd column..
  • Provide the single best result tuned with the development set. For example, see local/ about how to tune the language model weight, insertion penalty, and array.
  • Submit the RTTM file used in your ASR evaluation above. Our intention is to have an analysis of the relationship between RTTM and ASR results, and we would not offiicaly rank the result only with DER.

Submit your results via the following Google forms

  • Track 2 Category A
  • Track 2 Category B

    • Please first submit a challenge abstract paper by following and get the CMT paper ID, which is required to finish the form.
    • This form requires a Google account. Please contact us if you have difficulties on access to the form due to the account issue.
    • In the form, we ask you to fill out the team profile, brief technical descriptions and so on for our challenge analysis. It can be updated anytime before the challenge submission deadline (11:59pm (Pacific time), April 10th).
    • If you want to only submit the diarization results, please contact us in advance.