A comprehensive graph theoretical and finite geometrical study of the commutation relations between the generalized Pauli operators of $N$-qudits is performed in which vertices/points correspond to the operators and edges/lines join commuting pairs of them. As per two-qubits, all basic properties and partitionings of the corresponding {\it Pauli graph} are embodied in the geometry of the generalized quadrangle of order two. Here, one identifies the operators with the points of the quadrangle and groups of maximally commuting subsets of the operators with the lines of the quadrangle. The three basic partitionings are (a) a pencil of lines and a cube, (b) a Mermin's array and a bipartite-part and (c) a maximum independent set and the Petersen graph. These factorizations stem naturally from the existence of three distinct geometric hyperplanes of the quadrangle, namely a set of points collinear with a given point, a grid and an ovoid, which answer to three distinguished subsets of the Pauli graph, namely a set of six operators commuting with a given one, a Mermin's square, and set of five mutually non-commuting operators, respectively. The generalized Pauli graph for multiple qubits is found to follow from symplectic polar spaces of order two, where maximal totally isotropic subspaces stand for maximal subsets of mutually commuting operators. The substructure of the (strongly regular) $N$-qubit Pauli graph is shown to be pseudo-geometric, i.\,e., isomorphic to a graph of a partial geometry. Finally, the (not strongly regular) Pauli graph of a two-qutrit system is introduced; here it turns out more convenient to deal with its dual in order to see all the parallels with the two-qubit case and its surmised relation with the generalized quadrangle $Q(4,3)$, the dual of $W(3)$.