Emerging studies have reported that functional brain networks change with increasing age. Graph theory is applied to understand the age-related differences in brain behavior and function, and functional connectivity between the regions is examined using electroencephalography (EEG). The effect of normal aging on functional networks and inter-regional synchronization during the working memory (WM) state is not well known. In this study, we applied graph theory to investigate the effect of aging on network topology in a resting state and during performing a visual WM task to classify aging EEG signals. We recorded EEGs from 20 healthy middle-aged and 20 healthy elderly subjects with their eyes open, eyes closed, and during a visual WM task. EEG signals were used to construct the functional network; nodes are represented by EEG electrodes; and edges denote the functional connectivity. Graph theory matrices including global efficiency, local efficiency, clustering coefficient, characteristic path length, node strength, node betweenness centrality, and assortativity were calculated to analyze the networks. We applied the three classifiers of K-nearest neighbor (KNN), a support vector machine (SVM), and random forest (RF) to classify both groups. The analyses showed the significantly reduced network topology features in the elderly group. Local efficiency, global efficiency, and clustering coefficient were significantly lower in the elderly group with the eyes-open, eyes-closed, and visual WM task states. KNN achieved its highest accuracy of 98.89% during the visual WM task and depicted better classification performance than other classifiers. Our analysis of functional network connectivity and topological characteristics can be used as an appropriate technique to explore normal age-related changes in the human brain.