Problem solving encompasses the broad domain of human, goal-directed behaviors. Though we may attempt to measure problem solving using tightly controlled and decontextualized tasks, it is inextricably embedded in both reasoners’ experiences and their contexts. Without situating problem solvers, problem contexts, and our own experiential partialities as researchers, we risk intertwining the research of information relevance with our own confirmatory biases about people, environments, and ourselves. We review each of these ecological facets of information relevance in problem solving, and we suggest a framework to guide its measurement. We ground this framework with concrete examples of ecologically valid, culturally relevant measurement of problem solving.

The digital assessment community aims to align assessment design and measurement selection with theory, but many digital assessments fail to establish these theoretical links. In this paper, we introduce Fraction Ball: Exactly (FBE), a digital game-based assessment for planning in a mathematics problem-solving context. Using gameplay learning analytics, we create distinct measurement models for adult and child players. We demonstrate convergent validity of FBE both with the Shallice Tower of London and with mathematical knowledge. We share our approaches to theoretically ground our assessment design, measure selection, and measurement model validation. By theoretically grounding each process, we outline ways in which digital assessment developers can integrate educational theory with assessment design and validation and theorize for themselves.

Little is known about mismatches between the language of mathematics testing instruments and the rich linguistic repertoires that African American children develop at home and in the community, in part because research paradigms with African American English (AAE) dialect speakers face complex challenges in measurement, historical exclusion, and other social, economic, cultural, and linguistic confounds. The current study aims to provide a proof of concept and novel explanatory item response design that uses error analysis to investigate the relationship between AAE child language and children's mathematics assessment outcomes. Here, we illustrate 2^nd and 3^rd grade children's qualitative patterns of performance on arithmetic tasks in relation to their AAE dialect use and elaborate a unified framework for examining child and item level linguistic characteristics. Results suggest that children draw upon their emerging (bi)dialectal repertoire with arithmetic problems when selecting appropriate problem-solving strategies on language-formatted problems. The mismatch of assessment language formatting with children's repertoires may disadvantage AAE speakers' strategy selections and result in a language-based performance disadvantage unrelated to mathematical ability. Research designs that look beyond correct/incorrect scoring to examine qualitative patterns of performance in AAE speaking children can provide valuable and oft-overlooked evidence when considering equity in mathematics assessment formats.

The Fraction Ball has demonstrated positive impacts on rational number learning. fractions and decimals through basketball. We explored activity features that sparked math talk discussions. We found that 6th graders engaged in minimal math discussions that were mostly preceded by teacher prompting. Future designs might leverage student prior knowledge by incorporating increasingly difficult questions to support peer-led discussions.