We consider a special class of compact complex nilmanifolds, which we call compact nilmanifolds with nilpotent complex structure. It is shown that if $\Gamma \backslash G$ is a compact nilmanifold with nilpotent complex structure, then the Dolbeault cohomology $H^{\ast ,\ast }_{\bar {\partial } }(\Gamma \backslash G)$ is canonically isomorphic to the $\bar {\partial }$ –cohomology $H^{\ast ,\ast }_{\bar {\partial } }(\mathfrak {g}^{\mathbb {C} })$ of the bigraded complex $(\Lambda ^{\ast ,\ast } (\mathfrak {g} ^{\mathbb {C} })^{\ast }, \bar {\partial } )$ of complex valued left invariant differential forms on the nilpotent Lie group $G$ .

We determine the 6D solvmanifolds admitting an invariant complex structure with holomorphically trivial canonical bundle. Such complex structures are classified up to isomorphism, and the existence of strong Kähler with torsion, generalized Gauduchon, balanced and strongly Gauduchon metrics is studied. As an application, we construct a holomorphic family $(M,J_a)$ of compact complex manifolds such that $(M,J_a)$ satisfies the $\partial {\bar {\partial }}$ -lemma and admits a balanced metric for any $a\not =0$ , but the central limit neither satisfies the $\partial {\bar {\partial }}$ -lemma nor admits balanced metrics.

The invariant balanced Hermitian geometry of nilmanifolds of dimension 6 is described. We prove that the (restricted) holonomy group of the associated Bismut connection reduces to a proper subgroup of SU(3) if and only if the complex structure is abelian. As an application we show that if J is abelian, then any invariant balanced J-Hermitian structure provides solutions of the Strominger system.