Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Generative artificial intelligence (AI) has shown rapid advancement and increasing applications in various domains, including healthcare. Previous studies have evaluated AI performance on medical license exams, primarily focusing on ChatGPT. However, the availability of new online chat-based large language models (OC-LLMs) and their potential utility in pharmacy licensing exams remain underexplored. Given that pharmacists require a broad range of expertise in physics, chemistry, biology, and pharmacology, there is a need to verify the knowledge base and problem-solving abilities of these newer models in Japanese pharmacy examinations. Objective: This study aimed to assess the performance of 18 OC-LLM models released in 2024 in the 107th Japanese National License Examination for Pharmacists (JNLEP), comparing their accuracy and identifying areas of improvement relative to earlier models. Methods: The 107th JNLEP, comprising 345 questions in Japanese, was used as the benchmark. Each OC-LLM was prompted with the original text-based questions, and images were uploaded where permitted. No additional prompt engineering or English translation was performed. For questions that included diagrams or chemical structures, models incapable of image input were considered incorrect. Model outputs were compared with publicly available correct answers. Overall accuracy rates were calculated by subject area (pharmacology and chemistry) and question type (text-only, diagram-based, calculation, and chemical structure). Fleiss’ kappa was used to measure answer consistency among the top-performing models. Results: Four flagship models—ChatGPT o1, Gemini 2.0 Flash, Claude 3.5 Sonnet (New), and Perplexity Pro—achieved 80% accuracy, surpassing the official passing threshold and average examinee score. A significant improvement in the overall accuracy was observed between the early and latest 2024 models. Marked improvements were noted in text-only and diagram-based questions compared with those of earlier versions. However, accuracy for chemistry-related and chemical structure questions remained relatively low. Fleiss’ kappa among the four flagship models was 0.334, suggesting moderate consistency, but highlighting variability in more complex questions. Conclusions: OC-LLMs have substantially improved their capacity to handle Japanese pharmacist examination content, with several newer models achieving accuracy rates of over 80%. Despite these advancements, even the best-performing models exhibited an error rate exceeding 10%, underscoring the ongoing need for careful human oversight in clinical settings. The 107th JNLEP serves as a valuable benchmark for current and future generative AI evaluations in pharmacy licensing examinations.

With the recent shift in community pharmacist services toward in-person services and the growing need for centralized and continuous medication management/monitoring, pharmacist–patient information sharing is crucial. This study investigated the pharmacist–patient gap in the recognition of pharmacists’ behaviors regarding information sharing and assessed the potential impact of such recognition on patient trust and willingness to self-disclose. This cross-sectional study included 600 patients (aged 21–85 years) using pharmacy services (surveyed online in December 2020) and 591 community pharmacists with ≥1 year of experience (surveyed from September to November 2021). Both groups responded to items on the recognition of pharmacists’ behaviors regarding information sharing. There were patient-specific items on trust in community pharmacists and willingness to self-disclose. For all items on the recognition of pharmacists’ behaviors, patients’ scores were significantly lower (4–5) than pharmacists’ own scores (≥5), revealing a notable perception gap. Patients’ recognition had a positive, direct effect on trust and willingness, and trust had a positive, direct effect on willingness. Patients’ recognition and trust positively influenced their willingness to self-disclose. Pharmacist communication with clear intent is important to bridge the gaps in pharmacist–patient recognition and foster effective patient–pharmacist relationships.

This review introduces two sets of research results, one regarding patients’ and consumers’ perceptions of the pharmacist profession and pharmacy function, and the other regarding factors that influence patients’ medication-taking behavior. First, as an example of what was examined from patients’ perspectives regarding the pharmacist profession and pharmacy function, an analysis of patient response data before the introduction of the family pharmacist/pharmacy system is presented. The results clarified that the quality of medication instruction influences patients’ evaluations of pharmacists and further affects their evaluation of pharmacies, and that the main factor influencing patients’ evaluation of pharmacies is their evaluation of the pharmacists. In particular, patients who continuously used the same pharmacist and the same pharmacy were highly statistically related, demonstrating the significance of recommending a family pharmacist/pharmacy. Next, regarding patients’ medication-taking behavior, an empirical study of Japanese patients regarding the two-dimensional influencing factors was conducted. These factors, which have been the focus of recent overseas studies, included intentional nonadherence, such as skipping a dose, and unintentional nonadherence, such as forgetting to take a dose. The main influences were found to be from unintentional nonadherence and intentional non-adherence, with excessive information seeking and motivation to acquire knowledge potentially exacerbating intentional nonadherence in particular, as well as differences across diseases. These findings may contribute to supporting medication adherence in patients. In collaboration with graduate and postgraduate research students, future studies will continue to examine patients’ and consumers’ perspectives on medication adherence.

Objective: Small language models (SLMs) suitable for local deployment may enable integration of generative AI into electronic health record (EHR) systems; however, their performance in non-English pharmaceutical domains remains unclear. Methods: Fifty-two open-source SLMs (≤20B parameters) were evaluated using 283 text-based questions from the 107th Japanese National License Examination for Pharmacists. Each model was tested in three independent runs. Performance was assessed using accuracy and response consistency metrics, including the Any Correct Rate, Full Correct Rate, and Consistency Gap. Results: Three models (GPT-OSS:20b, Qwen3:14b, and Qwen3:8b) achieved high accuracy comparable to frontier LLMs, whereas medical-specialized SLMs performed poorly. Consistency Gap showed no association with parameter size but demonstrated a nonlinear relationship with accuracy. Conclusions: Selected locally deployable SLMs achieved strong performance in a Japanese pharmaceutical benchmark, supporting their potential as components of EHR-compatible decisionsupport systems, provided that their reliability is further validated in clinical contexts.

Purpose: Conventional single-run accuracy may be insufficient for evaluating frontier generative artificial intelligence (AI) models when safety-related refusals occur. This 4-model benchmark examined the need for repeated, refusal-aware evaluation using the Japanese National License Examination for Pharmacists (JNLEP).Methods: ChatGPT GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8, and Claude Fable 5 were evaluated using all 345 questions from the 107th JNLEP. The original Japanese questions, including image-containing items, were submitted through application programming interfaces (APIs) in 3 independent runs. Refusals were treated as incorrect when overall accuracy was calculated. For Fable 5, accuracy excluding refusals, refusal rate, refusal consistency across runs, the subject-wise distribution of refusals, and system-assigned refusal categories were also evaluated.Results: The mean overall accuracies were 98.7% for GPT-5.5, 98.3% for Gemini 3.5 Flash, 96.1% for Claude Opus 4.8, and 70.3% for Claude Fable 5. Fable 5 had a mean refusal rate of 29.0%, whereas its mean accuracy excluding refusals was 99.0%. Among the 345 items, 93 were refused in all 3 runs, 14 were refused inconsistently across runs, and 238 were never refused. All 300 refusal responses were assigned to the bio category. Refusals were most frequent in Biology (90.0%) and Pharmacology (69.2%) but uncommon in Practice (2.5%).Conclusion: Near-saturation benchmark performance coexisted with frequent and partly run-dependent refusals in a safeguard-equipped model. Overall accuracy, accuracy excluding refusals, refusal rate, and refusal consistency describe complementary aspects of performance; therefore, repeated, refusal-aware evaluation is needed to interpret frontier AI models in pharmacy education.