In stacked object scenes, effectively perceiving the positional relationships between objects is crucial for enabling robots to grasp safely. However, most existing methods rely on single-scale features with limited representational capability, ignoring the benefits of multi-scale features. In this paper, we propose a multi-task grasp detection model for stacked object scenes based on multi-scale feature fusion. We design a Multi-Scale Feature Aggregation (MSFA) module to fuse features, enhancing the feature maps' ability to represent objects of varying sizes. Additionally, we leverage features from multiple views to predict object positional relationships. In particular, the local Intersection Region Feature (IRF) significantly enhances the model's capability to identify positional relationships between objects while improving computational efficiency. Experimental results on the Visual Manipulation Relationship Dataset (VMRD) demonstrate that our model surpasses current state-of-the-art methods. Furthermore, we conducted grasping experiments in real-world stacking scenarios, which verified the model's effectiveness and generalization capability.