A speckle interferometer which can measure whole field three-dimensional displacements continuously and dynamically has been built. Three different wavelength lasers are used to produce the speckle interferograms of the two in-plane displacements (displacements in the x- and y-direction) and one out-of-plane displacement (displacement in the z-direction), respectively. One color CCD camera is employed to collect these mixed speckle interferograms simultaneously. The mixed interferograms are separated by the Red, Green and Blue channels of the color CCD camera, and then are processed by the wavelet transform technique to extract the phase information of the measured object. The preliminary experiment is carried out to demonstrate the performance of this new device.