Most sentiment tooling returns a label and no way to check it. This returns numbers on a named psychological model, against a written schema, and reports its own accuracy — including where it is weak and where it fails outright.
Valence vs human · glm-5.2
+0.96
Pearson r, 95% CI [+0.92, +0.98]
human ceiling +0.88
human ceiling +0.88
Arousal vs human · glm-5.2
+0.49
Pearson r, 95% CI [+0.16, +0.72]
human ceiling +0.71
human ceiling +0.71
Hedging vs human · glm-5.2
+0.81
Pearson r, 95% CI [+0.64, +0.91]
human ceiling +0.31 (low)
human ceiling +0.31 (low)
30 responses rated independently by 2 human raters
(Ashwin Srivastava and Prashant); every response was scored by three methods — the language model
glm-5.2, Jev
typesafe-ai/jev, and the lexicon control
nrc-vad-1. Each model correlation is shown
against the inter-rater ceiling — how well the two humans agree with each
other on that axis. Where the ceiling is low, the axis is not measurable at this sample
size whatever the model scores, and it says so below.What is measured
Valence and arousal are the two axes of Russell's Circumplex Model of Affect (Russell, 1980), scored independently — a calm, content response is pleasant with low arousal; fury is unpleasant with high arousal. Hedging is measured separately as qualifying language in the response itself.
| Field | Range | Meaning |
|---|---|---|
| valence | −1.00 to +1.00 | unpleasant to pleasant |
| arousal | 0.00 to 1.00 | calm and flat to intense and activated |
| hedging_score | 0.00 to 1.00 | direct and certain to heavily qualified |
| response_latency_ms | integer or null | passthrough, measured by the calling application |
| method | string | which scoring method produced the result: llm_v1, lexicon_v1 or jev_v1 |
| model | string | the exact model id, so results stay attributable |
| sentiment_category | string or null | the dominant tone as a label — scored by jev_v1 only, null from the other methods |
How it works
Grounded, not vibesBoth axes are defined by a named,
citable model before any scoring happens, so "how positive is this" is not the model's
private judgment.
Structured output, forcedThe model must return the schema via
a forced tool call. Nothing parses free text, so a malformed score cannot silently pass.
Code-mixed, nativelyHindi, English and Hinglish are scored as
written. No translation step — translation is where tone is destroyed.
What was measured
Each response was scored by all three methods and compared against the per-item mean of 2 human ratings using Pearson correlation, with a 95% confidence interval (Fisher z-transform) and a two-sided p-value. The ceiling column is the humans' agreement with each other.
| Method | Axis | r | 95% CI | p | n | ceiling |
|---|---|---|---|---|---|---|
| Language model (glm-5.2) | valence | +0.96 | [+0.92, +0.98] | <0.0001 | 30 | +0.88 |
| Language model (glm-5.2) | arousal | +0.49 | [+0.16, +0.72] | 0.0061 | 30 | +0.71 |
| Language model (glm-5.2) | hedging | +0.81 | [+0.64, +0.91] | <0.0001 | 30 | +0.31 |
| Jev — typed decision model (typesafe-ai/jev) | valence | +0.96 | [+0.92, +0.98] | <0.0001 | 30 | +0.88 |
| Jev — typed decision model (typesafe-ai/jev) | arousal | +0.54 | [+0.22, +0.75] | 0.0022 | 30 | +0.71 |
| Jev — typed decision model (typesafe-ai/jev) | hedging | +0.70 | [+0.45, +0.84] | <0.0001 | 30 | +0.31 |
| NRC-VAD lexicon (nrc-vad-1) | valence | +0.17 | [-0.20, +0.50] | 0.3696 | 30 | +0.88 |
| NRC-VAD lexicon (nrc-vad-1) | arousal | +0.18 | [-0.20, +0.50] | 0.3527 | 30 | +0.71 |
| NRC-VAD lexicon (nrc-vad-1) | hedging | not measured | not available | not available | — | +0.31 |
Not measured, and why. NRC-VAD lexicon (nrc-vad-1), hedging: hedging is unavailable from lexicon_v1 - NRC-VAD has no hedging dimension. These cells carry no number on purpose: a zero or a bare dash would read like a result.
Ratings a rater left blank are excluded from that axis only and never imputed (1 left blank on valence, 1 left blank on arousal, 3 left blank on hedging). At n = 30 the intervals are wide: this is
an indicative measurement, not proof.
Three scoring methods, run in parallel
Every response is scored by each of these, and the harness measures all three against the same human ratings. A method that failed on an item is reported as a failure for that item, never imputed, so a partial pass is visible above rather than smoothed away.
| Method | What it is | valence r | arousal r | hedging r |
|---|---|---|---|---|
| Language model (glm-5.2) | A prompt plus a forced tool call: the model has to return the three scores through the written schema, so nothing parses free text. | +0.96 | +0.49 | +0.81 |
| Jev — typed decision model (typesafe-ai/jev) | Jev by TypeSafe AI, a typed-decision model reached through the Vercel AI Gateway. It is handed the response plus declared questions and returns typed values with a confidence — about 1.2 s and $0.042 per 1M input tokens (output billed at zero) per response. It does not hold a conversation; it only scores. | +0.96 | +0.54 | +0.70 |
| NRC-VAD lexicon (nrc-vad-1) | An offline cross-check against the NRC-VAD affect lexicon. No model, no network, deterministic — and English-annotated, which is why both its coverage and its correlations fall over on code-mixed responses. | +0.17 | +0.18 | not measured |
Why three. Not to crown a winner up front: the point is to let this harness decide which method is accurate enough to trust at scale, because a cheaper or faster method only matters once its accuracy clears the bar. On this sample the two model-based methods are not separable: valence reads +0.96 for the language model against +0.96 for Jev, and arousal +0.49 against +0.54, with the 95% intervals overlapping on both. The language model is ahead on hedging (+0.81 against +0.70), which is the axis whose human ceiling is lowest, so neither number settles much. The defensible claim is parity on the two axes where the raters agree, not a win: at this sample size the intervals cannot separate 0.96 from 0.96.
Read Jev with its caveats attached. It is a typed-decision model, not a reasoning model, so a schema-valid score is not automatically a correct one — the correlations above are the only evidence that it can be trusted, and nothing here justifies using it beyond triage without them. Its own vendor benchmark reports roughly 68% accuracy on their internal workflow evals: a starting expectation, not a guarantee for this task. It scores by anchor probability, so its scale is stepped at about 0.05 rather than continuous, which caps how high its r can climb at this sample size. And the gateway key observed a five-requests-per-minute limit, which is why the harness paces its calls at 12 s and retries transient failures instead of assuming one call per response.
Read Jev with its caveats attached. It is a typed-decision model, not a reasoning model, so a schema-valid score is not automatically a correct one — the correlations above are the only evidence that it can be trusted, and nothing here justifies using it beyond triage without them. Its own vendor benchmark reports roughly 68% accuracy on their internal workflow evals: a starting expectation, not a guarantee for this task. It scores by anchor probability, so its scale is stepped at about 0.05 rather than continuous, which caps how high its r can climb at this sample size. And the gateway key observed a five-requests-per-minute limit, which is why the harness paces its calls at 12 s and retries transient failures instead of assuming one call per response.
Do the humans agree on each axis?
| Axis | inter-rater r | overlap n | reading |
|---|---|---|---|
| Valence | +0.88 | 29 | strong — a fair yardstick |
| Arousal | +0.71 | 29 | moderate — interpret with care |
| Hedging | +0.31 | 27 | low — the raters do not agree on this axis, so no model r against it is interpretable |
The ceiling is the number that makes each model r interpretable. A model
cannot be judged against an axis the raters themselves score differently, and a model r that
exceeds a low ceiling is not evidence of skill — it is evidence that the axis has no
stable human answer to hit. With 2 raters there is a single pair, so this ceiling
is itself a noisy estimate; three or four raters would tighten it.
How each rater used the scale
| Axis | Ashwin Srivastava | Prashant | mean of raters, sd | ||||
|---|---|---|---|---|---|---|---|
| mean | sd | range | mean | sd | range | ||
| Valence | -0.01 | 0.851 | -1.00…+1.00 | -0.07 | 0.702 | -0.97…+0.97 | 0.755 |
| Arousal | +0.45 | 0.364 | +0.00…+1.00 | +0.69 | 0.173 | +0.24…+1.00 | 0.267 |
| Hedging | +0.45 | 0.433 | +0.00…+1.00 | +0.32 | 0.265 | +0.01…+0.77 | 0.288 |
Scale habits differ. Ashwin Srivastava used an exact endpoint on 18 of 29 valence ratings and 9 of 28 hedging ratings. Endpoint parking compresses the spread a correlation needs and pulls the inter-rater ceiling down mechanically, so the ceiling below is a lower bound on what the axis could support. The last column is the spread of the value the model is
actually correlated against: a compressed criterion cannot support a large r however good
the model is.
Worked sample — agreements and disagreements
Six responses. Human is the mean of the 2 ratings; the middle column of each axis pair is the absolute gap between human mean and model: <0.20 0.20–0.39 ≥0.40.
| id | response | human V | model V | gap | human A | model A | gap | human H | model H | gap |
|---|---|---|---|---|---|---|---|---|---|---|
| E13 | Nothing works here. Nothing has ever worked here. I am tired. | -0.98 | -0.80 | 0.18 | 0.29 | 0.40 | 0.11 | 0.01 | 0.00 | 0.01 |
| E03 | I can't believe it actually worked! The pension came through this month, first time in two years! | +0.97 | +0.90 | 0.07 | 0.96 | 0.80 | 0.16 | 0.52 | 0.00 | 0.52 |
| E11 | Delighted! My son got the placement letter yesterday, whole family is celebrating. | +0.97 | +0.90 | 0.07 | 0.99 | 0.80 | 0.19 | 0.01 | 0.00 | 0.01 |
| D01 | हमें योजना के बारे में कुछ नहीं पता। कोई बताता ही नहीं है। | -0.84 | -0.50 | 0.34 | 0.90 | 0.40 | 0.50 | 0.04 | 0.20 | 0.16 |
| H01 | Shayad next month ka form bhar sakte hain, but pata nahi kab tak. | +0.07 | +0.10 | 0.03 | 0.49 | 0.30 | 0.20 | 0.78 | 0.90 | 0.12 |
| E08 | It's okay I suppose. I mean, the water comes, mostly. Some days not. What can you do. | -0.26 | -0.20 | 0.05 | 0.12 | 0.20 | 0.08 | 0.81 | 0.70 | 0.11 |
The same six responses, rater by rater — where the two sheets disagree is where the ceiling comes from.
| id | Ashwin Srivastava | Prashant | ||||
|---|---|---|---|---|---|---|
| V | A | H | V | A | H | |
| E13 | -1.00 | +0.00 | +0.00 | -0.97 | +0.58 | +0.02 |
| E03 | +1.00 | +1.00 | +0.30 | +0.94 | +0.92 | +0.73 |
| E11 | +1.00 | +1.00 | +0.00 | +0.94 | +0.99 | +0.01 |
| D01 | -1.00 | +1.00 | +0.00 | -0.69 | +0.80 | +0.09 |
| H01 | +0.00 | +0.33 | +1.00 | +0.14 | +0.66 | +0.57 |
| E08 | -0.20 | +0.00 | +1.00 | -0.31 | +0.24 | +0.63 |
2 of the six contain at least one gap of 0.40 or more against the human
mean. They are shown deliberately: an instrument that reports only its good examples is not
an instrument.
The lexicon control is the honest test of whether a cheap deterministic
method would do. It does not: median coverage of only
0.366 of tokens (minimum 0.071) across
30 scored responses, because the lexicon is
English-annotated and most Indian conversational responses are code-mixed or Romanised, and
neither valence nor arousal correlation is distinguishable from zero. On 14 response(s) it returned the opposite sign to the human mean (E04, E05, E06, E08, D01, E13, H03, D02, E17, E18, E19, H05, E21, E22).
What this does not prove yet
- Two raters is the minimum for a ceiling, not the target. The language model's valence now reads +0.96 and Jev's +0.96, against a ceiling of +0.88, which is a defensible number for this module. Every other axis needs more raters before its ceiling stops being one noisy pair.
- Hedging has no usable human ground truth here. The raters agree at +0.31 — below the 0.50 threshold — so the model's +0.81 on that axis is not evidence it can measure hedging. It is evidence that "how much does this person hedge" is being read differently by different humans. Fixing the rubric is the next task, not tuning the model.
- Arousal is the weak axis, and now that is measurable. +0.49 against a ceiling of +0.71: the model sits clearly below what the humans can agree on, so this is model error rather than ambiguity in the measure.
- Jev is a triage instrument until its numbers say otherwise. It is a typed-decision model, not a reasoning model, so a schema-valid score is not automatically a correct one. On this sample its valence and arousal sit level with the language model's (+0.96 and +0.54 against +0.96 and +0.49) — enough to justify a cheap first pass at scale, not enough to replace the language model where the decision matters. Its own vendor benchmark reports roughly 68% on their internal evals, and its anchor-based scale is stepped rather than continuous; both facts hold its numbers down, and neither is a reason to trust it further than the correlations above.
- A model r above the ceiling is not proof. The model is correlated against the mean of the ratings, which is more reliable than either single rater, while the ceiling is computed between single raters. The two numbers are therefore not strictly comparable, and the ceiling is a floor on the axis's reliability, not a target the model has beaten.
- The model is not deterministic. Two runs over the same 30 responses differed on roughly half the items (mean absolute difference 0.06 on valence, up to 0.50 on hedging), moving the reported correlation by about 0.05. Valence was stable; averaging repeated samples is the fix.
- The lexicon cannot ship. NRC-VAD is licensed for research use; commercial use requires written consent from NRC, and redistribution is prohibited. It is used here as a research control only.
- This validates a module, not a business. The claim above is about scoring one response. It says nothing yet about scale, cost per response, or behaviour on real deployment traffic.
Why it matters
Any team can call a model and print a sentiment score. The defensible asset is the discipline around it: a fixed schema, a citable psychological model, two further methods as controls, a blind rating pack scored by more than one person, a measured result with its ceiling attached, and a written account of where it fails. That is what makes a capability claim auditable — by a client, a partner, or an investor's technical advisor. The same harness is what will train a domain-specific model later, because it already produces labelled data in a fixed format.