This paper presents new results on multistability of networks when neurons undergo self-excitation and second-order synaptic connectivity. Due to self-excitation of neurons, we split state space into invariant regions and establish new criteria of coexistence of equilibria (periodic orbits) which are exponentially stable. It is shown that high-order synaptic connectivity and external inputs play an important role on the number of equilibria and their convergent dynamics. As a consequence, our results refute traditional viewpoint: high-order interactions of neurons have faster convergence rate and greater storage capacity than first-order ones. Finally, numerical simulations will illustrate our new and interesting results.