Effective supply chain inventory management is crucial for large-scale manufacturing industries such as civil aircraft and automobile manufacturing to ensure efficient manufacturing. Generally, the main manufacturer makes the annual inventory management plan, and contacts with suppliers when some material is approaching critical inventory level according to the actual production schedule, which increases the difficulty of inventory management. In recent years, many researchers have focused on using reinforcement learning method to study inventory management problems. Current approaches were mainly designed for the supply chain with single-node multi-material or multi-node single-material mode, which are not suitable to the civil aircraft manufacturing supply chain with multi-node multi-material mode. To deal with this problem, we formulated the problem as a partially observable Markov decision process (POMDP) model and proposed a multi-agent reinforcement learning method for supply chain inventory management, in which the dual-policy and information transmission mechanism was designed to help the supply chain participant improve the global information utilization efficiency of the supply chain and the coordination efficiency with other participants. The experiment results show that our method has about 45% performance improvement on efficiency compared with current reinforcement learning-based methods.