We introduce a mathematical framework designed to enable a simple image-to-simulation workflow for solids of complex geometries in the geometrically nonlinear regime. While the material point method is used to circumvent the mesh distortion issues commonly exhibited in Lagrangian meshes, a shifted domain technique originated from Main and Scovazzi (J Comput Phys 372:972–995, 2018 ) is used to represent the boundary conditions implicitly via a level set or signed distance function. Consequently, this method completely bypasses the need to generate high-quality conformal mesh to represent complex geometries and therefore allows modelers to select the space of the interpolation function without the constraints due to the geometric need. This important simplification enables us to simulate deformation of complex geometries inferred from voxel images. Verification examples on deformable body subjected to finite rotation have shown that the new shifted domain material point method is able to generate frame-indifferent results. Meanwhile, simulations using micro-CT images of a Hostun sand have demonstrated that this method is able to reproduce the quasi-brittle damage mechanisms of single grain without the excessively concentrated nodes commonly displayed in conformal meshes that represent 3D objects with local fine details.