Research and development around indoor positioning and navigation is capturing the attention of an increasing number of research groups and labs around the world. Among the several techniques being proposed for indoor positioning, solutions based on Wi-Fi fingerprinting are the most popular since they exploit existing WLAN infrastructures to support software-only positioning, tracking and navigation applications. Despite the enormous research efforts in this domain, and despite the existence of some commercial products based on Wi-Fi fingerprinting, it is still difficult to compare the performance, in the real world, of the several existing solutions. The EvAAL competition, hosted by the IPIN 2015 conference, contributed to fill this gap. This paper describes the experience of the RTLS@UM team in participating in track 3 of that competition.

This paper presents the analysis and discussion of the off-site localization competition track, which took place during the Seventh International Conference on Indoor Positioning and Indoor Navigation (IPIN 2016). Five international teams proposed different strategies for smartphone-based indoor positioning using the same reference data. The competitors were provided with several smartphone-collected signal datasets, some of which were used for training (known trajectories), and others for evaluating (unknown trajectories). The competition permits a coherent evaluation method of the competitors’ estimations, where inside information to fine-tune their systems is not offered, and thus provides, in our opinion, a good starting point to introduce a fair comparison between the smartphone-based systems found in the literature. The methodology, experience, feedback from competitors and future working lines are described.

The development of indoor positioning solutions using smartphones is a growing activity with an enormous potential for everyday life and professional applications. The research activities on this topic concentrate on the development of new positioning solutions that are tested in specific environments under their own evaluation metrics. To explore the real positioning quality of smartphone-based solutions and their capabilities for seamlessly adapting to different scenarios, it is needed to find fair evaluation frameworks. The design of competitions using extensive pre-recorded datasets is a valid way to generate open data for comparing the different solutions created by research teams. In this paper, we discuss the details of the 2017 IPIN indoor localization competition, the different datasets created, the teams participating in the event, and the results they obtained. We compare these results with other competition-based approaches (Microsoft and Perf-loc) and on-line evaluation web sites. The lessons learned by organising these competitions and the benefits for the community are addressed along the paper. Our analysis paves the way for future developments on the standardization of evaluations and for creating a widely-adopted benchmark strategy for researchers and companies in the field.

Every year, for ten years now, the IPIN competition has aimed at evaluating real-world indoor localisation systems by testing them in a realistic environment, with realistic movement, using the EvAAL framework. The competition provided a unique overview of the state-of-the-art of systems, technologies, and methods for indoor positioning and navigation purposes. Through fair comparison of the performance achieved by each system, the competition was able to identify the most promising approaches and to pinpoint the most critical working conditions. In 2020, the competition included 5 diverse off-site off-site Tracks, each resembling real use cases and challenges for indoor positioning. The results in terms of participation and accuracy of the proposed systems have been encouraging. The best performing competitors obtained a third quartile of error of 1 m for the Smartphone Track and 0.5 m for the Foot-mounted IMU Track. While not running on physical systems, but only as algorithms, these results represent impressive achievements.

IPIN 2019 Competition, sixth in a series of IPIN competitions, was held at the CNR Research Area of Pisa (IT), integrated into the program of the IPIN 2019 Conference. It included two on-site real-time Tracks and three off-site Tracks. The four Tracks presented in this paper were set in the same environment, made of two buildings close together for a total usable area of 1000 m 2 outdoors and and 6000 m 2 indoors over three floors, with a total path length exceeding 500 m. IPIN competitions, based on the EvAAL framework, have aimed at comparing the accuracy performance of personal positioning systems in fair and realistic conditions: past editions of the competition were carried in big conference settings, university campuses and a shopping mall. Positioning accuracy is computed while the person carrying the system under test walks at normal walking speed, uses lifts and goes up and down stairs or briefly stops at given points. Results presented here are a showcase of state-of-the-art systems tested side by side in real-world settings as part of the on-site real-time competition Tracks. Results for off-site Tracks allow a detailed and reproducible comparison of the most recent positioning and tracking algorithms in the same environment as the on-site Tracks.

Fingerprint is one of the most widely used methods for locating devices in indoor wireless environments and we have witnessed the emergence of several positioning systems aimed for indoor environments based on this approach. However, additional efforts are required in order to improve the performance of these systems so that applications that are highly dependent on user location can provide better services to its users. In this work we discuss some improvements to the positioning accuracy of the fingerprint-based systems. Our algorithm ranks the information about the location in a hierarchical way by identifying the building, the floor, the room and the geometric position. The proposed fingerprint method uses a previously stored map of the signal strength at several positions and determines the position using similarity functions and majority rules. In particular, we compare different similarity functions to understand their impact on the accuracy of the positioning system. The experimental results confirm the possibility of correctly determining the building, the floor and the room where the persons or the objects are at with high rates, and with an average error around 3 meters. Moreover, detailed statistics about the errors are provided, showing that the average error metric, often used by many authors, hides many aspects on the system performance.