Results
🙌 We would like to thank all participants for their significant effort given this year’s challenge difficulty.
Designing a meeting transcription system that is robust and can generalize to multiple scenarios, as humans are capable of, is a key machine listening problem and certainly not an easy feat !
This year’s DASR task did not have many submissions. In CHiME-8 this year there were three different ASR tasks and participants split amongst the three different tasks.
Nonetheless, we have already some findings we can share, also thanks to the coordination with NOTSOFAR-1 Task 2:
- In the challenge design stage we were interested in particular in how a “generalist” meeting transcription system (DASR Task 1) fares against a specialized one (NOTSOFAR-1 Task 2).
- Contrary to our expectation, STCON (1st place) and NTT (2nd place) DASR systems technically would place 2nd and 3rd also in Task 2 multi-channel track (see NOTSOFAR-1 Task 2 results).
- Despite the fact that this year DASR task was significantly complicated by addition of the NOTSOFAR-1 scenario (which is highly different from the others), the top-2 ranked teams (STCON and NTT) were able to push further the performance compared to previous year best results.
- In CHiME-7+8 System Ranking Section we summarize results for last year and this year submissions all together on CHiME, DiPCo and Mixer 6 scenarios.
📈 Hereafter we summarize the results of the CHiME-8 DASR Task.
- All participants that submitted a technical description paper must present their work (exceptions are made for teams that are unable to participate due to unexpected reasons) at the CHiME-8 Workshop.
- 📌 We did have only one submission for the unconstrained LM track by STCON. It is available in Supplementary Metrics plot below and is labeled as sys2 (unconstrained LM).
Constrained LM Track Results
Teams Ranking
| Rank | Team Name | System Tag | macro DER (%) (eval) | macro tcpWER (%) (eval) | macro cpWER (%) (eval) | macro DER (%) (dev) | macro tcpWER (%) (dev) | macro cpWER (%) (dev) |
|---|
Supplementary Metrics
We report in this scatter plot each submitted system DER and tcpWER for each scenario and partition (dev or eval).
As mentioned, STCON sys2 (unconstrained LM) refers to the only submission to the unconstrained LM track.
| Rank | Team Name | System Tag | macro DER (%) (eval) | chime6 DER (%) (eval) | dipco DER (%) (eval) | mixer6 DER (%) (eval) | notsofar1 DER (%) (eval) | macro tcpWER (%) (eval) | chime6 tcpWER (%) (eval) | dipco tcpWER (%) (eval) | mixer6 tcpWER (%) (eval) | notsofar1 tcpWER (%) (eval) | macro cpWER (%) (eval) | chime6 cpWER (%) (eval) | dipco cpWER (%) (eval) | mixer6 cpWER (%) (eval) | notsofar1 cpWER (%) (eval) | macro DER (%) (dev) | chime6 DER (%) (dev) | dipco DER (%) (dev) | mixer6 DER (%) (dev) | notsofar1 DER (%) (dev) | macro tcpWER (%) (dev) | chime6 tcpWER (%) (dev) | dipco tcpWER (%) (dev) | mixer6 tcpWER (%) (dev) | notsofar1 tcpWER (%) (dev) | macro cpWER (%) (dev) | chime6 cpWER (%) (dev) | dipco cpWER (%) (dev) | mixer6 cpWER (%) (dev) | notsofar1 cpWER (%) (dev) |
|---|
Per Scenario Metrics
Here we report using plotly tcpWER (%) for each team best system on the 4 different scenarios plus its macro average.
Evaluation Set
Development Set
CHiME-7+8 System Ranking
We report here tcpWER, cpWER and DER as obtained by this year and last year submissions on CHiME-6, DiPCo and Mixer 6 scenarios.
As said, this year top-2 submissions are able to improve further even in these scenarios despite having to deal also with NOTSOFAR-1.
This year best system, STCON (C8), improves the macro tcpWER by approximately 4% compared to last year’s best system.
| Rank | Team Name + (challenge: C8 or C7) | System Tag | Challenge Edition | macro DER (%) (eval) | chime6 DER (%) (eval) | dipco DER (%) (eval) | mixer6 DER (%) (eval) | macro tcpWER (%) (eval) | chime6 tcpWER (%) (eval) | dipco tcpWER (%) (eval) | mixer6 tcpWER (%) (eval) | macro cpWER (%) (eval) | chime6 cpWER (%) (eval) | dipco cpWER (%) (eval) | mixer6 cpWER (%) (eval) | macro DER (%) (dev) | chime6 DER (%) (dev) | dipco DER (%) (dev) | mixer6 DER (%) (dev) | macro tcpWER (%) (dev) | chime6 tcpWER (%) (dev) | dipco tcpWER (%) (dev) | mixer6 tcpWER (%) (dev) | macro cpWER (%) (dev) | chime6 cpWER (%) (dev) | dipco cpWER (%) (dev) | mixer6 cpWER (%) (dev) |
|---|
📩 Contact
For any question you can reach the organizers via CHiME Google Group or via CHiME Slack Workspace.