Large efficiency gains signaled mixture-of-experts would matter

Large efficiency gains signaled mixture-of-experts would matter

Dean says the researchers recognized mixture-of-experts as an important idea because experiments showed roughly a tenfold improvement in training-compute-to-quality ratios compared with dense models.