Deep learning (DL) technology has shown outstanding performance in various fields such as object recognition and classification, speech recognition, and natural language processing. However, it is well known that DL models are vulnerable to data poisoning attacks, where adversaries modify or inject data samples maliciously during the training phase, leading to degraded classification accuracy or misclassification. Since data poisoning attacks keep evolving to avoid existing defense methods, security researchers thoroughly examine data poisoning attack models and devise more reliable and effective detection methods accordingly. In particular, data poisoning attacks can be realistic in an adversarial situation where we retrain a DL model with a new dataset obtained from an external source during transfer learning. By this motivation, we propose a novel defense method that partitions and inspects the new dataset and then removes malicious sub-datasets. Specifically, our proposed method first divides a new dataset into n sub-datasets either evenly or randomly, inspects them by using the clean DL model as a poisoned dataset detector, and finally removes malicious sub-datasets classified by the detector. For partition and inspection, we design two dynamic defensive algorithms: the Sequential Partitioning and Inspection Algorithm (SPIA) and the Randomized Partitioning and Inspection Algorithm (RPIA). With this approach, a resulting cleaned dataset can be used reliably for retraining a DL model. In addition, we conducted two experiments in the Python and DL environment to show that our proposed methods effectively defend against two data poisoning attack models (concentrated poisoning attacks and random poisoning attacks) in terms of various evaluation metrics such as removed poison rate (RPR), attack success rate (ASR), and classification accuracy (ACC). Specifically, the SPIA completely removed all poisoned data under concentrated poisoning attacks in both Python and DL environments. In addition, the RPIA removed up to 91.1% and 99.1% of poisoned data under random poisoning attacks in Python and DL environments, respectively.