In content-based recommendation, a second-stage (L2) reranker decides which of roughly a hundred retrieved items fill the first screen of results. The standard L2 model is a cross-encoder trained to score relevance. Decision models, general models that answer a typed question about a text with a probability, could offer an alternative through a single plain-language question. Whether they can is unknown: after retrieval most candidates are already on-topic, so the reranker must separate exact matches from close substitutes, a distinction decision models were not trained for, and sparse relevance labels make the comparison hard to measure. We cast a decision model as a pointwise reranker that asks, for each candidate, whether it exactly matches the request, and uses the answer probability as its score. We evaluate this under a pre-registered protocol: every model reranks the same frozen candidate lists, and a gain counts only if it holds both on the realistic retrieved lists and on fully labelled lists. On Amazon ESCI (1,000 requests) and Wayfair WANDS (299), the commercial Jev decision model, used zero-shot, improves P@10 over embedding order by 0.046 to 0.054 and significantly outperforms the Qwen3-Reranker cross-encoder on ESCI. The open decision model CLM-v0.1-8B ranks no better than random. Jev costs about four times more per request than the cross-encoder run on a rented cloud GPU, so the result is best read as evidence that some decision models can serve as high-precision L2 rerankers when accuracy at the top of the list matters more than serving cost.
Large language models (LLMs) face a growing spectrum of adversarial jailbreak inputs that bypass safety training and elicit harmful outputs. Existing red-teaming efforts are ad hoc, using incompatible metrics and omitting guardrail-layer evaluation. We introduce GuardBench, a pre-deployment evaluation framework comprising 4,200 jailbreak prompts across seven attack categories, a four-stage automated pipeline, and a multi-metric scoring protocol covering Attack Success Rate (ASR), False Positive Rate (FPR), and P95 latency with 95% confidence intervals. We evaluate two guardrail configurations against GPT-4o-mini: a rule-based keyword filter (G1) and Llama Guard 2, an 8B-parameter LLM classifier (G3). Evaluated via static replay of archival 2023-era prompts, the undefended baseline achieves 5.8% ASR—far below prior-work rates of 60–80% obtained with live adaptive generation; the gap reflects prompt staleness, alignment generalisation, or both. Against this baseline, G1 reduces ASR by 0.32 pp with negligible latency; G3 achieves a 2.4 pp reduction but scores below the unguarded baseline on the Composite Safety Score due to 5,799 ms P95 latency under our evaluation hardware (Apple M3 Pro, llama.cpp, Q4_K_M quantisation). Token Manipulation attacks carry a 38.8% residual ASR under both configurations—the dominant open problem for next-generation guardrail design. GuardBench includes a CI/CD gate with configurable thresholds for evidence-based guardrail selection. Datasets, code, and results will be released upon acceptance.
In content-based recommendation, a second-stage (L2) reranker decides which retrieved items fill the first screen. Decision models, which answer a typed question about a text with a probability, are an alternative to trained cross-encoders, but whether they separate exact matches from close substitutes is unknown. We cast a decision model as a pointwise reranker that asks whether each candidate exactly matches the request. Under a pre-registered protocol, a gain must hold on both retrieved and fully labelled lists. On Amazon ESCI (1000 requests) and Wayfair WANDS (299), the commercial Jev model, used zero-shot, improves P@10 over embedding order by 0.046 to 0.054 and beats the 0.6B Qwen3-Reranker by 0.02 to 0.03 on ESCI, but not significantly on WANDS. Larger trained cross-encoders (Qwen3-Reranker-4B and 8B, bge-reranker-v2-m3) erase most of this lead: Jev is ahead only on fully labelled ESCI lists. A prompted Qwen3-8B stays behind Jev everywhere. The open decision model CLM-v0.1-8B ranks at random level when scoring a whole list but matches embedding order one product at a time. Jev costs about as much as the 4B model and 13 times bge-reranker-v2-m3. A decision model can serve as an L2 reranker but not demonstrably better than trained cross-encoders.