This paper reviews energy storage systems, in general, and for specific applications in low-cost micro-energy harvesting (MEH) systems, low-cost microelectronic devices, and wireless sensor networks (WSNs). With the development of electronic gadgets, low-cost microelectronic devices and WSNs, the need for an efficient, light and reliable energy storage device is increased. The current energy storage systems (ESS) have the disadvantages of self-discharging, energy density, life cycles, and cost. The ambient energy resources are the best option as an energy source, but the main challenge in harvesting energy from ambient sources is the instability of the source of energy. Due to the explosion of lithium batteries in many cases, and the pros associated with them, the design of an efficient device, which is more reliable and efficient than conventional batteries, is important. This review paper focused on the issues of the reliability and performance of electrical ESS, and, especially, discussed the technical challenges and suggested solutions for ESS (batteries, supercapacitors, and for a hybrid combination of supercapacitors and batteries) in detail. Nowadays, the main market of batteries is WSNs, but in the last decade, the world’s attention has turned toward supercapacitors as a good alternative of batteries. The main advantages of supercapacitors are their light weight, volume, greater life cycle, turbo charging/discharging, high energy density and power density, low cost, easy maintenance, and no pollution. This study reviews supercapacitors as a better alternative of batteries in low-cost electronic devices, WSNs, and MEH systems.