Does Slightly Mean Somewhat? Measuring Vague Intensity Words in LLM Numeric Actions
Do language models preserve the ordinal meaning of intensity words when those words must produce numeric actions?
Paper
Code and data available upon request.
Summary
This study asks what happens when vague human language must become a precise LLM action. I built a controlled resource-allocation environment where Claude Haiku receives a natural-language instruction containing an intensity word (e.g., “slightly increase,” “dramatically increase”), produces a numeric allocation via a structured tool call, and a deterministic backend converts that allocation into a measurable outcome. The only variable that changes between runs is the intensity word or the starting system state.
Across 6,620 runs at two temperature settings, three patterns emerge.
Compression: 10 Words, 5 Outputs
The model compresses 10 intensity words into 5 distinct median outputs. Four lower-tier words (slightly, marginally, somewhat, mildly) all map to the same value (0.50), while stronger words break into higher regimes. Spearman $\rho = 0.845$ ($p < 0.001$) confirms coarse ordinal structure, but several distinctions a user might expect to matter are collapsed into identical numeric output.
Context Dominates Word Choice
When the current system state is supplied, separate Kruskal-Wallis tests show that grouping by starting allocation captures roughly 10 times more rank-based variance than grouping by word ($\varepsilon^2_{\text{baseline}} = 0.782$ vs. $\varepsilon^2_{\text{word}} = 0.079$). The same word means different things at different starting points. Word choice matters most when the system has headroom and becomes irrelevant as it approaches capacity.
Boundary Behavior: Hedge, Act, Abstain
At the 89% starting allocation, the model does not merely scale actions downward. It sometimes stops acting entirely. Lower-tier words always act with small adjustments. Several booster-class words (considerably, substantially, significantly) abstain in all 30 runs. The word drastically behaves differently: it pushes to the local ceiling. Near the boundary, the LLM makes a categorical decision about whether to act at all, and that decision depends on which word appeared in the prompt.
What I Built
- Experiment platform: Python harness with deep-copied session state, checkpoint/resume, deterministic solver backend, structured JSONL output
- Measurement environment: Synthetic MILP resource-allocation optimizer (30 factories, 3 machine types, 4 task categories), used as a controlled instrument
- Agent layer: Anthropic SDK tool-use agent with structured tool calls, provider-abstracted LLM client, session isolation (no memory between runs)
- Analysis: Nonparametric statistics throughout (Kruskal-Wallis, Spearman, effect sizes), no distributional assumptions
- Scale: 6,620 API calls across two temperature sweeps, 10 intensity words, 10 starting allocations, 30 runs per cell
Robustness
The core patterns persist across temperature. Raising $T$ from 0.0 to 0.7 broadens distributions but does not restore ordinal distinctions between words. The 10:1 context-over-word variance ratio holds. Abstention patterns shift slightly but the three-mode structure remains.
References
- Cliff, N. (1959). Adverbs as multipliers. Psychological Review, 66(1), 27-44.
- Quirk, R., Greenbaum, S., Leech, G., & Svartvik, J. (1985). A Comprehensive Grammar of the English Language. Longman.
- Mosteller, F., & Youtz, C. (1990). Quantifying probabilistic expressions. Statistical Science, 5(1), 2-12.
- Ramotowska, S., Szarvas, F., & Szymanik, J. (2024). Quantifier comprehension across languages. Semantics and Pragmatics.
- Zhang, R., et al. (2024). Calibrating LLM probability expressions. arXiv preprint.
