Scene text recognition (STR) has benefited from vision–language modeling and multi-modal in-context learning, which enables models to exploit annotated demonstrations during inference without parameter updating. However, existing in-context STR methods mainly select demonstrations according to global visual similarity. Such image-level representations jointly encode textual content, font appearance, background, degradation, and geometric deformation, making it difficult to distinguish appearance compatibility from character-level relevance. As a result, globally similar samples may fail to provide effective evidence for ambiguous characters. To address this limitation, we propose Scene-to-Glyph, a hierarchical coarse-to-fine retrieval framework for multi-modal in-context scene text recognition. At the coarse-grained stage, an appearance-oriented scene representation is extracted from visual features to characterize shared conditions such as font style, degradation, deformation, and background appearance while reducing the influence of textual content. Scene-level similarity is then used to retrieve a compact set of appearance-compatible contextual candidates. At the fine-grained stage, uncertain character positions are identified from the recognition output, and decoder–vision correspondence is exploited to derive localized glyph representations. These representations are matched with character-level features from the candidate demonstrations, enabling the model to select demonstrations that are both scene-compatible and relevant to ambiguous characters. Experiments on conventional and challenging STR benchmarks validate the effectiveness of Scene-to-Glyph and the complementary roles of scene-level and glyph-level retrieval.