Robots fabricated with soft materials can provide higher flexibility and, thus, better safety while interacting in unpredictable situations. However, the usage of soft material makes it challenging to predict the deformation of a continuum body under actuation and, therefore, brings difficulty to the kinematic control of its movement. In this article, we present a geometry-based framework for computing the deformation of soft robots within the range of linear material elasticity. After formulating both manipulators and actuators as geometry elements, deformation can be efficiently computed by solving a constrained optimization problem. Because of its efficiency, forward and inverse kinematics for soft manipulators can be solved by an iterative algorithm with a low computational cost. Meanwhile, components with multiple materials can also be geometrically modeled in our framework with the help of a simple calibration. Numerical and physical experimental tests are conducted on soft manipulators driven by different actuators with large deformation to demonstrate the performance of our approach.