Travel behavior understanding is a long-standing and critically important topic in the area of smart cities. Big volumes of various GPS-based travel data can be easily collected, among which the taxi GPS trajectory data is a typical example. However, in GPS trajectory data, there is usually little information on travelers' activities, thereby they can only support limited applications. Quite a few studies have been focused on enriching the semantic meaning for raw data, such as travel mode/purpose inferring. Unfortunately, trip purpose imputation receives relatively less attention and requires no real-time response. To narrow the gap, we propose a probabilistic two-phase framework named TripImputor, for making the real-time taxi trip purpose imputation and recommending services to passengers at their dropoff points. Specifically, in the first phase, we propose a two-stage clustering algorithm to identify candidate activity areas (CAAs) in the urban space. Then, we extract fine-granularity spatial and temporal patterns of human behaviors inside the CAAs from foursquare check-in data to approximate the priori probability for each activity, and compute the posterior probabilities (i.e., infer the trip purposes) using Bayes' theorem. In the second phase, we take a sophisticated procedure that clusters historical dropoff points and matches the dropoff clusters and CAAs to immerse the real-time response. Finally, we evaluate the effectiveness and efficiency of the proposed two-phase framework using real-world data sets, which consist of road network, check-in data generated by over 38,000 users in one year, and the large-scale taxi trip data generated by over 19,000 taxis in a month in Manhattan, the New York City, USA. Experimental results demonstrate that the system is able to infer the trip purpose accurately, and can provide recommendation results to passengers within 1.6 s in Manhattan on average, just using a single normal PC.