Growing obesity has been a worldwide issue for several decades. This is the outcome of common nutritional disorders which results in obese individuals who are prone to many diseases. Managing diet while simultaneously dealing with the obligations of a working adult can be difficult. This paper presents the design and development of a smartphone-based diet-tracking application, Smart Diet Diary, to assist obese people as well as patients to manage their dietary intake for a healthier life. The proposed system uses deep learning to recognize a food item and calculate its nutritional value in terms of calorie count. The dataset used comprises 16,000 images of food items belonging to 14 different categories to train a multi-label classifier. We applied a pre-trained faster R-CNN model for classification and achieved an overall accuracy of approximately 80.1% and an average calorie computation within 10% of the real calorie value.