Large pre-trained code generation models, such as OpenAI Codex, can generate syntax-and function-correct code, making the coding of programmers more productive. In this paper, we introduce CodeGeeX, a multilingual model with 13 billion parameters for code generation. CodeGeeX is pre-trained on 850 billion tokens of 23 programming languages as of June 2022. Our extensive experiments suggest that CodeGeeX outperforms multilingual code models of similar scale for both the tasks of code generation and translation on HumanEval-X. Building upon HumanEval (Python only), we develop the HumanEval-X benchmark for evaluating multilingual models by hand-writing the solutions in C++, Java, JavaScript, and Go. In addition, we build CodeGeeX-based extensions on Visual Studio Code, JetBrains, and Cloud Studio, generating 8 billion tokens for tens of thousands of active users per week. Our user study demonstrates that CodeGeeX can help to increase coding efficiency for 83.4% of its users. Finally, CodeGeeX is publicly accessible since Sep. 2022, we open-sourced its code, model weights, API, extensions, and HumanEval-X at https://github.com/THUDM/CodeGeeX.

Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, Jie Tang. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Massive open online courses (MOOCs) have developed rapidly in recent years, and have attracted millions of online users. However, a central challenge is the extremely high dropout rate — recent reports show that the completion rate in MOOCs is below 5% (Onah, Sinclair, and Boyatt 2014; Kizilcec, Piech, and Schneider 2013; Seaton et al. 2014).What are the major factors that cause the users to drop out?What are the major motivations for the users to study in MOOCs? In this paper, employing a dataset from XuetangX1, one of the largest MOOCs in China, we conduct a systematical study for the dropout problem in MOOCs. We found that the users’ learning behavior can be clustered into several distinct categories. Our statistics also reveal high correlation between dropouts of different courses and strong influence between friends’ dropout behaviors. Based on the gained insights, we propose a Context-aware Feature Interaction Network (CFIN) to model and to predict users’ dropout behavior. CFIN utilizes context-smoothing technique to smooth feature values with different context, and use attention mechanism to combine user and course information into the modeling framework. Experiments on two large datasets show that the proposed method achieves better performance than several state-of-the-art methods. The proposed method model has been deployed on a real system to help improve user retention.

Background. Papillary thyroid microcarcinoma (PTMC) has a high incidence and a good prognosis. Surgical operation for all PTMC might be an overtreatment. The objective of our study was to evaluate the efficacy and safety of ultrasound (US)-guided radiofrequency ablation (RFA) for treating low risk PTMC. Methods. 98 PTMC in 92 patients were included in our study. US and contrast-enhanced ultrasound (CEUS) examinations were performed before ablation. RFA was performed using the moving shot technique. The ablation area exceeded the tumor edge to prevent marginal residue and recurrence. Patients were followed at one, three, six, and twelve months and every six months thereafter. US and CEUS examinations were used to evaluate the ablation area. At 3 months after ablation, US-guided core needle biopsy (CNB) was performed in the center, at the edge of the ablation area, and in the surrounding thyroid parenchyma, respectively, to exclude recurrence. Results. The mean tumor volume was 118.8±106.9 mm3. The mean volume reduction ratio was 0.47±0.27, 0.19±0.16, 0.08±0.11, 0.04±0.10 and 0 at one month, 3 months, 6 months, 12 months and 18 month after RFA respectively. Significant differences of the volume reduction ratio were found between every two follow-up times before 6 months (P<0.01), and no significant differences of the volume reduction ratio were found between 6 months and after 12 months (P=0.42). Of all the nodules, 10 (41.7%) resolved in 6 months and 23 (95.8%) resolved in 12 months. No residual or recurrent tumor tissue was detected in RFA area or in residual thyroid tissue during follow up. No suspicious metastatic lymph nodes were detected. The histological pathology results of US-guided CNB confirmed no recurrent tumor. No major complications were encountered. Conclusions. RFA can effectively eliminate low risk PTMC with a very small complication rate. RFA may be an alternative strategy for the therapy of PTMC.

Graph self-supervised learning (SSL), including contrastive and generative approaches, offers great potential to address the fundamental challenge of label scarcity in real-world graph data. Among both sets of graph SSL techniques, the masked graph autoencoders (e.g., GraphMAE)—one type of generative methods—have recently produced promising results. The idea behind this is to reconstruct the node features (or structures)—that are randomly masked from the input—with the autoencoder architecture. However, the performance of masked feature reconstruction naturally relies on the discriminability of the input features and is usually vulnerable to disturbance in the features. In this paper, we present a masked self-supervised learning framework1 GraphMAE2 with the goal of overcoming this issue. The idea is to impose regularization on feature reconstruction for graph SSL. Specifically, we design the strategies of multi-view random re-mask decoding and latent representation prediction to regularize the feature reconstruction. The multi-view random re-mask decoding is to introduce randomness into reconstruction in the feature space, while the latent representation prediction is to enforce the reconstruction in the embedding space. Extensive experiments show that GraphMAE2 can consistently generate top results on various public datasets, including at least 2.45% improvements over state-of-the-art baselines on ogbn-Papers100M with 111M nodes and 1.6B edges.