Add JEV-CPU: CPU port of SemIf + web UI

README.md

@@ -1,17 +1,3 @@
----
-license: mit
-base_model:
-- Qwen/Qwen3-0.6B
-tags:
-- semantic-if
-- decision
-- semif
-- cpu
-- zero-shot-classification
-pipeline_tag: zero-shot-classification
-library_name: transformers
----
-
 # JEV-CPU
 
 <div align="center">
@@ -270,7 +256,7 @@ MODEL = "openbmb/MiniCPM5-2B"   # or "Qwen/Qwen3.5-4B"
 
 That is exactly what fixes the slip in the loan demo: `Qwen3-0.6B` flags *high risk* correctly but still leans *approve*; a 2B/4B brain keeps the secondary decision consistent. The trade-off is resources — a 4B model needs `transformers`' native Qwen3.5 support and more RAM/compute than this 8 GB CPU box; a GPU (SemIf's target) makes it comfortable.
 
-**Takeaway:** JEV-CPU shows the method runs anywhere; **accuracy scales with the model you point it at.**
+**Takeaway:** JEV-CPU shows the method runs anywhere; **accuracy scales with the model you point it at.** And a larger open-weight model **served on a GPU** relaxes the CPU latency wall too — lower latency, far larger inputs (toward the model's 40,960-token context), and many decisions per second via batching + shared-state reuse. That's the shape of a **production JEV** — see the report's [Outlook: GPU serving & a production JEV](https://leesk212.github.io/JEV-CPU/#outlook).
 
 ---
 

docs/index.html

@@ -99,9 +99,11 @@
   <a href="#domains">5. Qualitative results across domains</a>
   <a href="#latency">6. Latency and input-length limits</a>
   <a href="#scaling">7. Accuracy scaling with model size</a>
+  <a href="#outlook">7.1 Outlook: GPU serving &amp; a production JEV</a>
   <a href="#limitations">8. Limitations</a>
   <a href="#conclusion">9. Conclusion</a>
   <a href="#repro">10. Reproducibility</a>
+  <a href="#appendix">Appendix A. Code-level walkthrough</a>
   <a href="#refs">References</a>
 </nav>
 
@@ -239,6 +241,36 @@ accuracy floor, not the ceiling. From SemIf's own evaluation:</p>
 compute that exceed this CPU box (a 4B model wants a GPU, SemIf's target). The takeaway: <strong>JEV-CPU shows
 the method runs anywhere; accuracy scales with the model you point it at.</strong></p>
 
+<h3 id="outlook">7.1 Outlook — GPU serving and a production JEV</h3>
+<p>The two axes of this report — accuracy (§7) and CPU latency (§6) — are usually assumed to trade off, but a
+GPU relaxes <em>both at once</em>. On CPU the wall is prefill: a single 7,697-token decision took ~117&nbsp;s
+(§6). A GPU changes the regime on three fronts:</p>
+<ul>
+  <li><strong>Latency collapses.</strong> Prefill is exactly the workload GPUs are built for. SemIf reports a
+  4B model on a single RTX&nbsp;3090 returning <strong>21 binary decisions in a median 1.02&nbsp;s</strong>,
+  versus 5.3&nbsp;s to generate the equivalent JSON array — the same logit-readout path, just not latency-bound.</li>
+  <li><strong>Input size grows.</strong> With the latency wall gone, the practical input rises from our
+  ~7,700-token / ~120&nbsp;s CPU ceiling toward the model's full <strong>40,960-token</strong> context — long
+  documents, transcripts, or code diffs become decidable in one pass.</li>
+  <li><strong>Throughput multiplies.</strong> Because a decision is one forward pass reading fixed positions,
+  decisions <em>batch</em> trivially, and shared-state prefix reuse amortizes one long state across many
+  criteria. SemIf measures <strong>~20 decisions/s</strong> (777 decisions in 38.8&nbsp;s) with parallel
+  suffixes on one 3090.</li>
+</ul>
+<p>Put together, a larger open-weight model (4B+) served on a GPU is <strong>simultaneously more accurate,
+accepts far larger inputs, and answers many decisions per second</strong> — all with typed, auditable outputs
+and no generation to parse. That combination is the shape of a <strong>production, potentially commercial,
+JEV</strong>: semantic-if offered as a low-latency, high-throughput hosted service, with per-tenant models and
+batched shared-state decisions. In that framing, <strong>JEV-CPU is the floor</strong> — proof the method is
+portable to any machine — and a <strong>GPU-served larger model is the ceiling</strong> that turns the same
+engine into a product.</p>
+<div class="note">
+  <strong>Scope.</strong> We did not run GPU experiments in this report. The latency and throughput figures
+  above are SemIf's published single-GPU (RTX 3090, 4B) measurements together with projections from our own
+  CPU curve (§6) — offered as an implication, not as measured results of this work. Quantifying a GPU-served
+  JEV (tokens/s, decisions/s, cost per million decisions, batching and concurrency) is the natural next study.
+</div>
+
 <h2 id="limitations">8. Limitations</h2>
 <ul>
   <li><strong>No new method.</strong> The logit-readout mechanism is prior art; this is a port and measurement.</li>
@@ -256,7 +288,9 @@ the method runs anywhere; accuracy scales with the model you point it at.</stron
 SemIf's engine runs on a commodity CPU with a 0.6B model, decides across many domains at ~1&nbsp;s each, and
 stays memory-stable well past its default token cap. On this hardware the honest limit is latency: inputs above
 ~7,700 tokens cross the ~120&nbsp;s mark and are impractical, and small-model accuracy — not context or memory —
-is what improves by scaling the model up. JEV-CPU is offered as a reproducible, minimal demonstration of that.</p>
+is what improves by scaling the model up. JEV-CPU is offered as a reproducible, minimal demonstration of that
+floor; the same engine, given a larger open-weight model on a GPU, points toward the ceiling (§7.1) — an
+accurate, high-throughput, low-latency semantic-if service, i.e. a production JEV.</p>
 
 <h2 id="repro">10. Reproducibility</h2>
 <p>Code, the web UI, the exact scripts behind §5–§6, and all demo GIFs are public:</p>
@@ -270,6 +304,157 @@ pip install transformers accelerate
 python semif_cpu.py      # CLI: typed option probabilities
 python server.py         # web UI on http://localhost:8080</code></pre>
 
+<hr>
+
+<h2 id="appendix">Appendix A. Code-level walkthrough: turning an open-weight model into a decision engine</h2>
+<p>A "JEV" / decision engine is <strong>not a fine-tune and not new weights</strong> — it is a way of
+<em>calling</em> an ordinary open-weight causal language model so that a typed decision falls out of a single
+forward pass. This appendix walks the full pipeline function by function, as it runs in JEV-CPU. The scoring
+code is <a href="https://github.com/TheoLeeCJ/SemIf">SemIf</a>'s, reproduced here verbatim for the report;
+JEV-CPU changes only the model <em>loader</em> (<a href="#a7">A.7</a>). All snippets are from
+<code>src/semif_phase1/{core,direct}.py</code>.</p>
+
+<h3 id="a1">A.1 — The record and its validation</h3>
+<p>A decision is one plain record: a <code>state</code> (evidence, string / JSON object / array), a
+<code>question</code> (criterion), and 2–16 <code>options</code>, each with a stable <code>id</code> and a
+human <code>description</code>. Validation is strict so nothing is silently coerced:</p>
+<pre><code>def validate_row(row):
+    required = {"id", "state", "question", "options"}
+    if not required &lt;= row.keys(): raise ValueError(...)
+    # state must be a nonempty, finite, JSON-serializable string/object/array
+    json.dumps(state, ensure_ascii=False, allow_nan=False)
+    # 2..16 options, each {id: str, description: str}, ids unique
+    if not isinstance(options, list) or not 2 &lt;= len(options) &lt;= len(LETTERS): raise ValueError(...)</code></pre>
+
+<h3 id="a2">A.2 — Building a letter-choice prompt</h3>
+<p>Each option is assigned an uppercase letter and the record is serialized into a two-turn chat. The system
+turn constrains the model to answer with a single letter and nothing else — this is what makes the <em>next</em>
+token the entire decision:</p>
+<pre><code>LETTERS = "ABCDEFGHIJKLMNOP"
+DIRECT_SYSTEM = ("Apply the supplied criterion to the supplied evidence. "
+                 "Choose exactly one listed option. Respond with only its "
+                 "uppercase letter, with no explanation or reasoning.")
+
+def direct_messages(row):
+    payload = {
+        "evidence":  row["state"],
+        "criterion": row["question"],
+        "options":   [{"letter": LETTERS[i], "description": o["description"]}
+                      for i, o in enumerate(row["options"])],
+    }
+    return [{"role": "system", "content": DIRECT_SYSTEM},
+            {"role": "user",   "content": json.dumps(payload, ensure_ascii=False)}]</code></pre>
+
+<h3 id="a3">A.3 — Pinning each option to exactly one token</h3>
+<p>The readout compares the model's probability of each answer letter, so every letter must map to a
+<strong>single, clean vocabulary slot</strong>. Each letter is checked to encode to exactly one token that
+round-trips, with no collisions between options:</p>
+<pre><code>def _slot_ids(tokenizer, count):
+    result = []
+    for letter in LETTERS[:count]:
+        encoded = tokenizer.encode(letter, add_special_tokens=False)
+        if len(encoded) != 1 or tokenizer.decode(encoded) != letter:
+            raise ValueError(f"Answer slot {letter!r} is not one exact round-trip token")
+        result.append(encoded[0])
+    if len(result) != len(set(result)):
+        raise ValueError("Answer-slot tokens collide")
+    return result            # e.g. token ids for "A", "B", "C"</code></pre>
+
+<h3 id="a4">A.4 — Encoding and verifying the answer boundary</h3>
+<p>The chat template is applied with the generation prompt on and thinking disabled, then two invariants are
+asserted: the prompt fits the token budget (no silent truncation), and appending any answer letter extends the
+tokenization by exactly that one slot token — i.e. the boundary between prompt and answer is stable:</p>
+<pre><code>def encode_prompt(tokenizer, row, max_tokens):
+    prompt = tokenizer.apply_chat_template(
+        direct_messages(row), tokenize=False,
+        add_generation_prompt=True, enable_thinking=False)
+    ids = tokenizer.encode(prompt, add_special_tokens=False)
+    if not ids or len(ids) &gt; max_tokens:
+        raise ValueError("input tokens exceed limit; no truncation allowed")
+    slots = _slot_ids(tokenizer, len(row["options"]))
+    for letter, token in zip(LETTERS, slots):
+        if tokenizer.encode(prompt + letter, add_special_tokens=False) != ids + [token]:
+            raise ValueError(f"Answer boundary changes tokenization for slot {letter}")
+    return ids, slots, digest(prompt)   # digest = sha256 for auditability</code></pre>
+
+<h3 id="a5">A.5 — One forward pass, last-position logits</h3>
+<p>The whole model is run <strong>once</strong>. We keep only the final position's logits — the distribution
+over the next token. <code>logits_to_keep=1</code> (when the model supports it) tells the model to compute
+just that row, avoiding a full-sequence logit tensor. There is no sampling and no decode loop:</p>
+<pre><code>def _forward(model, inputs):
+    params = inspect.signature(model.forward).parameters
+    kwargs = dict(inputs, use_cache=False, return_dict=True)
+    if "logits_to_keep" in params:
+        kwargs["logits_to_keep"] = 1
+    return model(**kwargs).logits[:, -1, :]     # shape (batch, vocab)</code></pre>
+
+<h3 id="a6">A.6 — Gather the slots, softmax, decide</h3>
+<p>From the full-vocabulary logit vector we index the option-letter tokens and softmax over <em>only</em>
+those, giving a probability per option conditional on the declared set. Note the two device lines: the code
+reads whatever device the model is on and only synchronizes under CUDA — the property that makes it run
+unchanged on CPU:</p>
+<pre><code>def score(model, tokenizer, row, metadata, max_tokens=4096):
+    import torch
+    ids, slots, prompt_hash = encode_prompt(tokenizer, row, max_tokens)
+    device = next(model.parameters()).device          # &lt;- follow the model
+    inputs = {"input_ids":      torch.tensor([ids], dtype=torch.long, device=device),
+              "attention_mask": torch.ones((1, len(ids)), dtype=torch.long, device=device)}
+    if device.type == "cuda": torch.cuda.synchronize(device)   # &lt;- no-op on CPU
+    with torch.inference_mode():
+        vocabulary = _forward(model, inputs)[0].float()        # (vocab,)
+    selected = vocabulary[slots].cpu().tolist()   # logits at A, B, C, ...
+    return {
+        "option_ids":   [o["id"] for o in row["options"]],
+        "option_logits": selected,
+        "probabilities": softmax(selected),        # winner = argmax
+        "input_tokens":  len(ids),
+        "readout": "native full-vocabulary last-position logits restricted to declared answer slots",
+        "probability_status": "conditional option score; uncalibrated as decision confidence",
+        ...
+    }</code></pre>
+<p>That is the entire "JEV-ification": a stock <code>AutoModelForCausalLM</code> is never asked to generate;
+it is asked once for its next-token logits, and the decision is the arg-max over the option slots.</p>
+
+<h3 id="a7">A.7 — The only change to run on CPU</h3>
+<p>SemIf's loader hard-codes a single CUDA device and <code>bfloat16</code>. JEV-CPU replaces just this
+function; every function above is untouched:</p>
+<div class="tablewrap">
+<table>
+  <thead><tr><th>SemIf — <code>core.load_causal_model</code> (GPU)</th><th>JEV-CPU — <code>load_causal_model_cpu</code></th></tr></thead>
+  <tbody><tr>
+  <td><pre style="margin:0"><code>if not torch.cuda.is_available() \
+   or torch.cuda.device_count() != 1:
+    raise ValueError("Expose exactly "
+        "one CUDA GPU ...")
+...
+model = cls.from_pretrained(
+    source, config=config,
+    dtype=torch.bfloat16,
+    device_map={"": "cuda:0"},
+    low_cpu_mem_usage=True, **common)</code></pre></td>
+  <td><pre style="margin:0"><code># no CUDA check, no device_map
+model = AutoModelForCausalLM.from_pretrained(
+    source, config=config,
+    dtype=torch.float32,   # CPU-stable
+    low_cpu_mem_usage=True, **common)
+model.eval()</code></pre></td>
+  </tr></tbody>
+</table>
+</div>
+<p>Because <code>score()</code> and <code>score_shared()</code> derive their device from the model object,
+loading on CPU is sufficient — the same scoring code then runs with <code>torch.cuda.synchronize</code> skipped.
+Swapping the brain model is the one other knob: change <code>MODEL = "Qwen/Qwen3-0.6B"</code> to any causal
+LM whose answer letters are single tokens.</p>
+
+<h3 id="a8">A.8 — Reusing one state across many criteria (shared mode)</h3>
+<p>When many criteria judge the same <code>state</code>, <code>shared.py</code> prefills that state once into a
+native key–value cache, replicates the cache across branches with <code>cache.reorder_cache(...)</code>, and
+evaluates every criterion's answer position in one batched forward using a vector
+<code>logits_to_keep</code>. One expensive prefill, many cheap decisions — and, like the direct path, it
+synchronizes only under CUDA, so it too runs unchanged on CPU.</p>
+
+<hr>
+
 <h2 id="refs">References</h2>
 <ol class="refs">
   <li>T. Lee (TheoLeeCJ). <em>SemIf — Semantic ifs from open models.</em> <a href="https://github.com/TheoLeeCJ/SemIf">github.com/TheoLeeCJ/SemIf</a>. Browser demo: <a href="https://openjev.com/">openjev.com</a>.</li>