To achieve a non-destructive and rapid detection of oyster freshness, an intelligent method using deep learning fused with malondialdehyde (MDA) and total sulfhydryl groups (SH) information was proposed. In this study, an “MDA-SH-storage days” polynomial fitting model and oyster meat image dataset were first built. AleNet-MDA and AlxNet-SH classification models were then constructed to automatically identify and classify four levels of oyster meat images with overall accuracies of 92.72% and 94.06%, respectively. Next, the outputs of the two models were used as the inputs to “MDA-SH-storage days” model, which ultimately succeeded in predicting the corresponding MDA content, SH content and storage day for an oyster image within 0.03 ms. Furthermore, the interpretability of the two models for oyster meat image were also investigated by feature visualization and strongest activations techniques. Thus, this study brings new thoughts on oyster freshness prediction from the perspective of computer vision and artificial intelligence.