The collaborative construction of large-scale, diverse datasets is crucial for developing high-performance machine learning models. However, this collaboration faces significant challenges, including ensuring data security, protecting participant privacy, maintaining high dataset quality, and aligning economic incentives among multiple stakeholders. Effective risk management strategies are essential to systematically identify, assess, and mitigate potential risks associated with data collaboration. This study proposes a federated blockchain-based framework designed to manage multiparty dataset collaborations securely and transparently, explicitly incorporating comprehensive risk management practices. The proposed framework involves six core entities—key distribution center (KDC), researcher (RA), data owner (DO), consortium blockchain, dataset evaluation platform, and the orchestrating model itself—to ensure secure, privacy-preserving and high-quality dataset collaboration. In addition, the framework uses blockchain technology to guarantee the traceability and immutability of data transactions, integrating token-based incentives to encourage data contributors to provide high-quality datasets. To systematically mitigate dataset quality risks, we introduced an innovative categorical dataset quality assessment method leveraging label reordering to robustly evaluate datasets. We validated this quality assessment approach using both publicly available (UCI) and privately constructed datasets. Furthermore, our research implemented the proposed blockchain-based management system within a consortium blockchain infrastructure, benchmarking its performance against existing methods to demonstrate enhanced security, reliability, risk mitigation effectiveness, and incentive alignment in dataset collaboration.