The latest mobile alarm apps provide wake-up tasks (e.g., solving math problems) to dismiss the alarm, and many users willingly accept such an inconvenience in return for successfully waking up on time. However, there have been no studies that investigate how the wake-up tasks are used and their effects from a human–computer interaction perspective. This study aims to deepen our understanding of how users engage and utilize the task-based alarm app by (1) examining the characteristics of different wake-up tasks and (2) extracting usage factors of hard tasks which involve physical or cognitive task loads over a certain level. We developed and deployed Alarmy, which is a task-based mobile alarm app with four wake-up task features: touching a button, taking a picture, shaking the device, and solving math problems. We collected 42.9 million in situ usage data from 211,273 US users for five months. Their alarm app usage behaviors were measured in two folds: eight alarm-set variables and five alarm-dismiss variables. Our statistical test results reveal the significant differences in alarm usage behaviors depending on the wake-up task, and the multiple regression analysis results show key usage patterns that affect the frequent uses of hard tasks, which are late alarm hours, many snoozes, and relatively more use on weekends. Our study results provide theoretical implications on behavior change as well as practical implications for designing task-based mobile alarm.