Results
We would like to thank all participants and congratulate them for having all significantly improved over our baseline. Thank you for accepting this challenge and for pushing the technical field of meeting transcription further !
We encountered many difficulties in organizing this Task and we are glad at the end we had so many participants ! It was not expected due to this Task being really arduos to approach.
Hereafter we summarize the results of the CHiME-7 DASR Task, as presented during the 2023 CHiME-7 Workshop in Dublin (25th August).
A pre-print of the Task and Baseline description is now available on arXiv.
Along with the results of the CHiME-7 DASR Main Track Results and Acoustic Robustness Sub-Track we report the results on the previous CHiME-6 Challenge data.
If you are interested, all the outputs of the scoring script are made available here as .csv files.
For any question, please reach us via CHiME Slack Workspace.
Main Track Results
Systems Ranking
Hereafter we report the main track final ranking in terms of macro-averaged DA-WER (across the three CHiME-6, DiPCo and Mixer 6) scenarios. For each team we report the best ranked system (best out of the 3 each team could submit).
See submission and rules for more info about the ranking.
Note that some teams did not use the original split for the development set but re-arranged or trained on it. See additional plots to see which teams.
| Rank | Team Name | System Tag | Tech Report | macro DER (%) (eval) | macro JER (%) (eval) | macro DA-WER (%) (eval) | macro DER (%) (dev) | macro JER (%) (dev) | macro DA-WER (%) (dev) |
|---|
Supplementary Metrics
| Rank | Team Name | System Tag | Tech Report | macro DER (%) (eval) | chime6 DER (%) (eval) | dipco DER (%) (eval) | mixer6 DER (%) (eval) | macro JER (%) (eval) | chime6 JER (%) (eval) | dipco JER (%) (eval) | mixer6 JER (%) (eval) | macro DA-WER (%) (eval) | chime6 DA-WER (%) (eval) | dipco DA-WER (%) (eval) | mixer6 DA-WER (%) (eval) | macro DER (%) (dev) | chime6 DER (%) (dev) | dipco DER (%) (dev) | mixer6 DER (%) (dev) | macro JER (%) (dev) | chime6 JER (%) (dev) | dipco JER (%) (dev) | mixer6 JER (%) (dev) | macro DA-WER (%) (dev) | chime6 DA-WER (%) (dev) | dipco DA-WER (%) (dev) | mixer6 DA-WER (%) (dev) |
|---|
Additional Plots
Evaluation Set
Development Set
Acoustic Robustness Sub-Track Results
Systems Ranking
Hereafter we report the sub track final ranking in terms of macro-averaged DA-WER (across the three CHiME-6, DiPCo and Mixer 6) scenarios. For each team we report the best ranked system (best out of the 3 each team could submit).
See submission and rules for more info about the ranking.
In this track the participants were allowed to use oracle diarization.
Note, in the following plot we report only the eval set as some teams did not provide the predictions of their best performing system on the dev set.
Note that some teams did not use the original split for the development set but re-arranged or trained on it. See additional plots to see which teams.
| Rank | Team Name | System Tag | Tech Report | macro DA-WER (%) (eval) |
|---|
Supplementary Metrics
| Rank | Team Name | System Tag | Tech Report | macro DA-WER (%) (eval) | chime6 DA-WER (%) (eval) | dipco DA-WER (%) (eval) | mixer6 DA-WER (%) (eval) | macro DA-WER (%) (dev) | chime6 DA-WER (%) (dev) | dipco DA-WER (%) (dev) | mixer6 DA-WER (%) (dev) |
|---|
Additional Plots
Evaluation Set
Development Set
Bonus Track: Original CHiME-6 Challenge
The evaluation portion of CHiME-6 scenario in the DASR Task includes the previous S01 and S21 evaluation sessions.
Note however that we perform additional text normalization and exclude the enrollment compared to the previous CHiME-6 Challenge.
Nonetheless the two are reasonably comparable in our opinion.
Again, here we report only the best systems (out of the 3 submitted by participants) according to best macro DA-WER on each corresponding DASR track (main and sub).
More details are available in Data page and in the DASR Task Description paper.
Track 1 Unconstrained Language Model (same as DASR Sub Track)
| Rank | Team Name | System Tag | Tech Report | cpWER (%) (eval) |
|---|
Track 2 Unconstrained Language Model (same as DASR Main Track)
| Rank | Team Name | System Tag | Tech Report | DER (%) (eval) | JER (%) (eval) | cpWER (%) (eval) |
|---|