Hyperspectral images contain both spectral and spatial image information and were investigated to characterize the freshness of fish. However, most studies of this application have focused on spectral signals rather than image features. The goal of this work was to investigate the ability of spectral and image textural variables for predicting the chemical and physical qualities of fish, respectively, and to optimize the variables for the specific quality determination. The chemical (total volatile basic nitrogen, TVB-N) and physical (texture profile analysis, TPA) properties were investigated. Partial least square (PLS) was applied to develop fish quality prediction models with the spectral and textural variables from the hyperspectral images. The results showed that the TVB-N content of fish fillets was accurately predicted using the spectra. Meanwhile, the TPA parameters were determined through the image textural features with high accuracy, which indicated image textural features were highly related with the TPA parameters. Moreover, spectral and textural features were also extracted from fish eyes and gills and were further used to predict the intact fish quality, taking advantage of the freshness sensitivity of the eyes and gills. The results illustrate that spectra from fish eyes and gills are a potential tool to predict the TVB-N content and TPA parameters for intact fish.