With a circular economy in the spotlight, wood waste has emerged as an important secondary raw material. Bearing this in mind, a comprehensive experimental study was carried out to evaluate the feasibility of using concrete compositions containing wood chips and sawdust for structural and non-structural building applications. First, the mineral and wood aggregates used in the composite design were fully characterized. Twelve compositions containing varying types of wood particles in different amounts were then produced and characterized in terms of physical and mechanical performance (e.g., mass density, compressive strength, modulus of elasticity, and flexural strength). Subsequently, two compositions with optimized features (mass density below 2125 kg/m³, compressive strength above 25 MPa, and maximum volume content of wood) were selected to undergo additional experimental tests. These included microstructural characterization, as well as the evaluation of relevant durability (e.g., wetting–drying, freeze–thaw, and thermal shock cycles) and hygrothermal (e.g., thermal conductivity, water absorption, and shrinkage and expansion) properties. All compositions showed compressive strength above 30 MPa. The durability assessment of selected compositions further showed that compressive strength after relevant artificial aging was still higher than the predefined criteria. Promising hygrothermal properties (minimal water absorption and low thermal conductivity) were also recorded.