Stochastic Computing has attracted extensive attention in the deployment of neural networks at the edge due to its low hardware cost and high fault tolerance. However, traditional stochastic computing requires a long random bit stream to achieve sufficient numerical precision. The long bit stream, in turn, increases the network inference time, hardware cost, and power consumption, which limits its application in executing tasks such as handwritten recognition, speech recognition, image processing, and image classification at the near-sensor end. To realize high-energy-efficiency and low-cost hardware neural networks at the near-sensor end, a hardware optimization design of convolutional neural networks based on the hybrid encoding of deterministic encoding and binary encoding is proposed. By transforming the output signals from the sensor into deterministic encoding and co-optimizing the network training process, a low-cost and high-energy-efficiency convolution operation network is achieved with a shorter bit stream input. This network can achieve good recognition performance with an extremely short bit stream, significantly reducing the system’s latency and energy consumption. Compared with traditional stochastic computing networks, this network shortens the bit stream length by 64 times without affecting the recognition rate, achieving a recognition rate of 99% with a 2-bit input. Compared with the traditional 2-bit stochastic computing scheme, the area is reduced by 44.98%, the power consumption is reduced by 60.47%, and the energy efficiency is increased by 12 times. Compared with the traditional 256-bit stochastic computing scheme, the area is reduced by 82.87%, and the energy efficiency is increased by 1947 times. These comparative results demonstrate that this work has significant advantages in executing tasks such as image classification at the near-sensor end and edge devices.