Refresh Cat Shell Risk Commander A1 card and add Kestrel comparison

Browse files

Files changed (4) hide show

README.md +298 -87
assets/cat-shell-risk-commander-banner.png +0 -0
comparison/compare_shellrisk.py +66 -0
comparison/kestrel-a1-comparison.json +51 -0

README.md CHANGED Viewed

@@ -3,62 +3,252 @@ license: apache-2.0
 pipeline_tag: text-classification
 library_name: sklearn
 metrics:
-  - recall
-  - f1
 datasets:
-  - tomngdev/shell-safety
-  - wwe123/Benign_and_malicious_PowerShell_scripts
 tags:
-  - shell-security
-  - shell-risk
-  - bash
-  - powershell
-  - cmd
-  - coding-agents
-  - tf-idf
-  - logistic-regression
-  - cpu
-  - research
 inference: false
 ---
-# Cat Shell Risk Commander
-**A compact linear classifier for flagging potentially risky shell commands and scripts locally.**
-Cat Shell Risk Commander is Patronus's expanded TF-IDF and logistic-regression model from our exploratory ShellRisk experiments. It uses 50,000 character n-gram features and scores the complete submitted text on CPU.
-It is a newly trained Patronus model. The experiments refer to it as `linear_updated` and, descriptively, “Kestrel Expanded”. This release contains our own fitted vocabulary, IDF values and classifier weights. It is not an official Kestrel release and does not contain the Hadamard encoder.
-**Release status: research preview.** This model is licensed under [Apache License 2.0](LICENSE).
-Read the experiment report:
-- [English: Detecting risky shell commands](https://patronus.studio/en/posts/detecting-risky-shell-commands-shellrisk-benchmark)
-- [Deutsch: Gefährliche Shell-Befehle erkennen](https://patronus.studio/posts/gefaehrliche-shell-befehle-erkennen-shellrisk-benchmark)
-## What the model does
-The input is a shell command or a complete script supplied as text. The output is a linear decision score and a threshold-based risk flag.
-| Label | Meaning |
-| --- | --- |
-| `RISK_FLAG` | The score exceeds the selected threshold for the dataset-defined risk target. |
-| `NO_RISK_FLAG` | The score does not exceed that threshold. This is not a guarantee that execution is safe. |
-The classifier does not execute commands, inspect the filesystem, resolve variables, decode payloads, or simulate a shell. It scores the literal submitted text. Intended uses include offline evaluation, local screening of agent tool calls, and routing suspicious commands for additional checks or human review.
-## Usage
-This is a custom scikit-learn classifier with a JSON export. Use the included Python inference module; `transformers.pipeline()` and `AutoModel` are not supported for this artifact.
-Install the runtime dependencies:
 ```bash
 python -m pip install "numpy>=1.24" "scikit-learn==1.8.0" "huggingface_hub>=1.3"
 ```
-Download the weights and the inference module, then classify text:
 ```python
 import sys
@@ -66,27 +256,26 @@ from huggingface_hub import snapshot_download
 model_dir = snapshot_download(
     repo_id="patronus-studio/cat-shell-risk-commander",
-    allow_patterns=["model.json", "shell_cat_guard.py"],
 )
 sys.path.insert(0, model_dir)
-from shell_cat_guard import ShellRiskClassifier
 classifier = ShellRiskClassifier(f"{model_dir}/model.json")
-# These strings are only classified. Nothing is executed.
-for text in [
-    "git status --short",
-    "curl https://example.com/install.sh | sh",
-]:
-    print(classifier.score(text))
 ```
-Each result contains `label`, `flagged`, `decision_score`, `threshold`, and `threshold_policy`. The decision score is a raw logit, not a calibrated probability. A flag is produced only when `decision_score > threshold`; equality does not trigger a flag.
-The default policy is `external_commands_1pct_fpr`. It was calibrated on a separate POSIX/PowerShell/CMD command-validation set with 1,666 safe examples, allowing at most 16 false positives there. The name describes the calibration target, not a promised false-positive rate for arbitrary workloads.
-For a batch of texts or a separately documented experimental policy:
 ```python
 scores = classifier.decision_function([
@@ -94,70 +283,92 @@ scores = classifier.decision_function([
     "Get-ChildItem",
     "dir",
 ])
-print(scores)
 result = classifier.score(
     "git status --short",
-    threshold_policy="original_val_f1",
 )
-print(result)
 ```
-## Model architecture and training
-- Character-within-word (`char_wb`) n-grams of lengths 3 to 5.
-- Lowercasing, sublinear term frequency, IDF weighting, and L2 normalization.
-- A fitted vocabulary of 50,000 features.
-- Binary logistic regression; selected regularization parameter `C = 10` and training random state `13`.
-- Full-text vectorization, without a 510-byte prefix cut or encoder chunking.
-Training combines 13,417 original fitting examples with 18,918 additional fitting examples, for 32,335 total. The additional set contains 10,672 positive and 8,246 negative examples, including multi-shell commands, PowerShell scripts, and synthetic obfuscation variants from fitting parents.
-The fitting data builds on [ShellRisk-Bench](https://github.com/kontext-security/shellrisk-bench), [Shell Safety](https://huggingface.co/datasets/tomngdev/shell-safety), and [Benign and malicious PowerShell scripts](https://huggingface.co/datasets/wwe123/Benign_and_malicious_PowerShell_scripts).
-`C` was selected from `0.1, 1, 10, 100, 1000` using mean validation average precision across five groups: original Bash, external POSIX, PowerShell, CMD, and PowerShell scripts. Exact duplicates, contradictory identical command labels, and detected held-out overlaps were excluded. Generated variants follow their original parent's split. These checks do not establish semantic, source-family, or malware-family independence.
-No dataset command or script was executed during the reported experiments. This model has no neural pretraining; the MLM and Hadamard-encoder experiments in the linked article concern a separate model.
-## Evaluation
-Experiments were run from October 4 to October 6, 2026. The following rows use different, explicitly identified threshold policies and must not be treated as results at one common operating point.
-| Evaluation | Test size | Threshold policy | Recall | False-positive rate | F1 |
-| --- | ---: | --- | ---: | ---: | ---: |
-| External POSIX/PowerShell/CMD commands | 918: 374 risky, 544 safe | `external_commands_1pct_fpr` | 85.83% | 0.92% | 91.71% |
-| Original Bash test | 4,194: 193 risky, 4,001 safe | `original_val_f1` | 85.49% | 0.45% | 87.77% |
-| Complete PowerShell scripts | 2,000: 1,000 per class | `original_val_f1` | 96.10% | 2.80% | 96.63% |
-The external command-test counts are 321 true positives, 53 false negatives, 5 false positives, and 539 true negatives. Its threshold was selected on separate external command-validation data; test cases did not determine this threshold.
-The external command test contains 735 POSIX, 119 PowerShell, and 64 CMD examples. The script test contains selected source-labeled PowerShell scripts up to 1 MiB, all longer than 510 bytes. Large scripts outside that bound were excluded from that experiment.
-Additional stored policies are `original_val_1pct_fpr`, `combined_val_f1`, and `exploratory_multigroup_0_5pct_fpr`. The last policy consumes calibration portions of earlier test groups and is exploratory; it is not the default and does not provide an independent confirmation of performance.
-Detailed per-group metrics, threshold values and training/audit counts are included in `evaluation.json`. The linked article also reports synthetic obfuscation and comment-padding stress tests.
-## Limitations
-- The target is the source-labeled risk status of text. Labels are not independently verified malicious behavior or execution outcomes, and different source datasets use different labeling criteria.
-- A legitimate installer may look risky; a harmful command may appear harmless when its payload or environment is absent from the input.
-- Character n-grams do not provide shell parsing or document-wide semantic reasoning.
-- Harmless comment padding can weaken detection. In the reported padding test, this linear model's POSIX recall fell from 91.10% to 50.00% behind 4 KiB of repeated comments and to 41.78% behind 16 KiB. Those results use the original-validation F1 policy and this specific synthetic transformation.
-- PowerShell and CMD command-test samples are small. Results use one training seed and source-conditioned splits, not a prospective production evaluation.
-- Processing the entire text does not guarantee robust detection of hidden or late risks. Runtime and memory use grow with input size; no deployment latency claim is made here.
-Use the classifier alongside execution permissions, sandboxing, explicit policy checks and review where needed. Evaluate threshold behavior on your own representative validation data before enabling automatic blocking.
 ## Repository files
 | File | Contents |
-| --- | --- |
-| `LICENSE` | Apache License 2.0. |
-| `model.json` | Vocabulary, IDF values, linear weights, intercept, inference configuration and threshold policies. |
-| `shell_cat_guard.py` | CPU inference implementation and result schema. |
-| `requirements.txt` | Runtime dependencies; the reference implementation was checked with scikit-learn 1.8.0. |
-| `evaluation.json` | Stored experiment metrics and data-audit summaries. |
-| `validation.json` | Export verification results against saved test scores. |
-| `manifest.json` | File hashes and source-model provenance. |
-The JSON export preserves the fitted inference parameters. It omits training-only estimator state and does not require loading a pickle or enabling Transformers remote code.

 pipeline_tag: text-classification
 library_name: sklearn
 metrics:
+- recall
+- precision
+- f1
+- roc_auc
+- average_precision
 datasets:
+- tomngdev/shell-safety
+- nebius/SWE-agent-trajectories
 tags:
+- shell-security
+- shell-risk
+- bash
+- powershell
+- cmd
+- coding-agents
+- tf-idf
+- logistic-regression
+- cpu
+- research
+- cat-shell-risk-commander
+- shell-command-classification
+- shell-command-security
+- shell-risk-classification
+- command-risk-detection
+- agent-security
+- ai-agent-security
+- ai-security
+- tool-call-security
+- tool-security
+- posix
+- windows
+- linux
+- powershell-security
+- tfidf
+- shell-token-features
+- binary-classification
+- local-inference
+- on-device
+- cpu-inference
+- scikit-learn
+- json-model
+- a1
 inference: false
+thumbnail: https://huggingface.co/patronus-studio/cat-shell-risk-commander/resolve/main/assets/cat-shell-risk-commander-banner.png
+model-index:
+- name: Cat Shell Risk Commander A1
+  results:
+  - task:
+      type: text-classification
+      name: Shell command risk classification
+    dataset:
+      name: Cat Shell A1 stored test (5,948 cases)
+      type: cat-shell-a1-stored-test
+      split: test
+    metrics:
+    - type: recall
+      name: Recall (external_commands_1pct_fpr)
+      value: 0.9259
+    - type: f1
+      name: F1 (external_commands_1pct_fpr; derived from confusion counts)
+      value: 0.942029
+    - type: roc_auc
+      name: AUROC
+      value: 0.9966
+    - type: false_positive_rate
+      name: False-positive rate (external_commands_1pct_fpr)
+      value: 0.0086
+  - task:
+      type: text-classification
+      name: Shell command risk classification
+    dataset:
+      name: Cat Shell A1 stored test (5,948 cases)
+      type: cat-shell-a1-stored-test
+      split: test
+    metrics:
+    - type: recall
+      name: Recall (block_0_1pct_fpr)
+      value: 0.8129
+    - type: f1
+      name: F1 (block_0_1pct_fpr; derived from confusion counts)
+      value: 0.888889
+    - type: roc_auc
+      name: AUROC
+      value: 0.9966
+    - type: false_positive_rate
+      name: False-positive rate (block_0_1pct_fpr)
+      value: 0.0035
 ---
+![Cat Shell Risk Commander. Local risk screening for shell commands and scripts.](assets/cat-shell-risk-commander-banner.png)
+<p align="center"><a href="#quickstart">Quickstart</a> · <a href="#evaluation">Evaluation</a> · <a href="#kestrel-comparison">Kestrel comparison</a> · <a href="#model-and-thresholds">Model &amp; thresholds</a> · <a href="https://patronus.studio/product/ai-models">AI models ↗</a></p>
+**Screen shell commands before an agent executes them.**
+Cat Shell Risk Commander scores shell commands and scripts locally on CPU. Use its risk flag to route agent tool calls for additional checks, review or policy enforcement. It reads the submitted text without executing it.
+| 92.59% recall | 0.86% false positives | 4.09 MB |
+| :---: | :---: | :---: |
+| Across the stored 5,948-case test | At the default threshold policy | JSON model artifact |
+**A1 research preview.** The figures above describe the current `format_version: 2` export and its stored evaluation. The model combines character n-grams with shell-token features. It is released under Apache 2.0.
+## What the model checks
+```text
+Agent tool call · Shell command · Script text
+                     ↓
+          Cat Shell Risk Commander
+                     ↓
+         Raw score + threshold policy
+                     ↓
+        Your execution or review policy
+```
+| Output | Meaning |
+| :--- | :--- |
+| `RISK_FLAG` | The decision score exceeds the selected risk threshold |
+| `NO_RISK_FLAG` | The decision score does not exceed that threshold |
+A score describes the submitted text. It does not establish what a command will do in a particular environment. Your application decides whether to block, review or continue; a `NO_RISK_FLAG` is not an execution-safety guarantee.
+## Model at a glance
+| Property | Current A1 release |
+| :--- | :--- |
+| Architecture | TF-IDF features with a binary logistic-regression classifier |
+| Character features | 50,000 `char_wb` n-grams, lengths 3 to 5 |
+| Shell-token features | 20,096 features from normalized tokens, token pairs and structural markers |
+| Total features | **70,096**, with one linear coefficient per feature plus an intercept |
+| Artifact | **4,088,897 bytes** (4.09 MB), `model.json`, JSON format version 2 |
+| Input | The complete submitted text; no fixed Transformer token window |
+| Output | Raw decision score, label, flag, threshold and threshold-policy name |
+| Inference | Local CPU, NumPy and scikit-learn; no GPU or neural checkpoint |
+| Default policy | `external_commands_1pct_fpr`, threshold `0.65763` |
+| Runtime loader | `shell_cat_guard_a1.py` and `shell_tokens.py` |
+| License | Apache 2.0 |
+This artifact is a custom JSON model. Use the included A1 inference module; Transformers `pipeline()` and `AutoModel` do not load it.
+## Evaluation
+### Compare operating points on the same test
+The stored A1 evaluation contains **5,948 cases: 1,053 risky and 4,895 safe**. Both rows below use the same model and test set. Threshold policy names identify calibration settings; their names are not promised false-positive rates for every test or production workload.
+| Threshold policy | Threshold | Recall ↑ | FPR ↓ | Precision ↑ | F1 ↑ |
+| :--- | ---: | ---: | ---: | ---: | ---: |
+| `external_commands_1pct_fpr` | 0.65763 | 92.59% | 0.86% | 95.87% | 94.20% |
+| `block_0_1pct_fpr` | 2.242267 | 81.29% | 0.35% | 98.05% | 88.89% |
+The default policy detects **975 of 1,053 risky cases**, with **42 false positives among 4,895 safe cases**. Average precision is **0.9858** and AUROC is **0.9966** on this stored test. Precision and F1 above are calculated from the published confusion counts.
+[Full A1 evaluation](https://huggingface.co/patronus-studio/cat-shell-risk-commander/blob/main/evaluation.json) · [Export validation](https://huggingface.co/patronus-studio/cat-shell-risk-commander/blob/main/validation.json)
+### Kestrel comparison
+**Cross-shell transfer: POSIX, PowerShell and CMD.** The [Patronus ShellRisk experiment](https://patronus.studio/posts/gefaehrliche-shell-befehle-erkennen-shellrisk-benchmark) evaluates 918 held-out commands: 374 risky and 544 safe cases, comprising 735 POSIX, 119 PowerShell and 64 CMD commands.
+For the calibrated rows, each model's threshold is selected on the **same separate external command-validation set**, allowing at most 16 false positives among 1,666 safe examples. The test cases do not select the thresholds.
+| Model and operating point | Detected / risky | False positives / safe |
+| :--- | ---: | ---: |
+| **Patronus Linear, same calibration** | **321/374 · 85.8%** | **5/544 · 0.9%** |
+| Patronus Encoder, same calibration | 315/374 · 84.2% | 5/544 · 0.9% |
+| [Kestrel](https://huggingface.co/kontext-security/Kestrel/tree/main), 1% validation FPR | 8/374 · 2.1% | 6/544 · 1.1% |
+| Kestrel, native threshold | 209/374 · 55.9% | 226/544 · 41.5% |
+At the shared validation-FPR target, the Patronus Linear model detects **85.8%** of risky commands versus **2.1%** for Kestrel, while producing fewer false positives on this external test.
+**Model versions:** this published experiment covers the previous 50,000-feature Patronus Linear model and a separate Patronus Encoder. It predates the current A1 export; A1's own stored results are reported separately in this card.
+<details>
+<summary>Comparison methodology and full results</summary>
+The external experiment uses the official Kestrel weights with the documented Python TF-IDF inference: character-within-word n-grams of lengths 3 to 5, lowercasing, sublinear term frequency, IDF weighting and L2 normalization.
+Training data differ. Kestrel was released for a different risk-classification task, while the Patronus models receive examples from the expanded sources during fitting. The experiment demonstrates a source-transfer gap and the benefit of adapting to that target distribution, rather than an architecture-only comparison or universal superiority.
+[Full published comparison (Deutsch)](https://patronus.studio/posts/gefaehrliche-shell-befehle-erkennen-shellrisk-benchmark) · [English report](https://patronus.studio/en/posts/detecting-risky-shell-commands-shellrisk-benchmark)
+The original same-source Bash test answers a different question. [Additional A1/Bash reference data](comparison/kestrel-a1-comparison.json) and the [reproduction script](comparison/compare_shellrisk.py) remain available separately; those values are not the external-transfer results in the table above.
+</details>
+<details>
+<summary>Shell-specific results at the default policy</summary>
+These are source-specific subsets, not four independent production evaluations. PowerShell and CMD command-test samples are small; zero observed false positives does not establish a zero underlying false-positive rate.
+| Test subset | Cases | Risky cases | Recall ↑ | FPR ↓ |
+| :--- | ---: | ---: | ---: | ---: |
+| Original Bash | 4194 | 193 | 84.97% | 0.25% |
+| External POSIX | 735 | 292 | 90.75% | 2.26% |
+| External PowerShell | 119 | 58 | 98.28% | 0.00% |
+| External CMD | 64 | 24 | 95.83% | 0.00% |
+</details>
+<details>
+<summary>All stored test families and false-positive uncertainty</summary>
+Default policy: `external_commands_1pct_fpr`. FPR upper bounds are copied from the published `fpr_wilson_upper` field. A metric is n/a when the subset has no examples of the required class. Several families have very small sample sizes.
+| Family | Cases | Risky cases | Recall ↑ | FPR ↓ | Reported Wilson FPR upper bound |
+| :--- | ---: | ---: | ---: | ---: | ---: |
+| `shellsafety_powershell` | 119 | 58 | 98.28% | 0.00% | 5.92% |
+| `shellsafety_posix` | 735 | 292 | 90.75% | 2.26% | 4.10% |
+| `shellsafety_cmd` | 64 | 24 | 95.83% | 0.00% | 8.76% |
+| `original_bash` | 4194 | 193 | 84.97% | 0.25% | 0.46% |
+| `exfil_pipe` | 160 | 160 | 95.63% | n/a | n/a |
+| `credential_store` | 80 | 80 | 97.50% | n/a | n/a |
+| `reverse_shell` | 7 | 7 | 85.71% | n/a | n/a |
+| `remote_download_exec` | 147 | 147 | 98.64% | n/a | n/a |
+| `privilege_escalation` | 8 | 8 | 100.00% | n/a | n/a |
+| `defense_evasion` | 26 | 26 | 96.15% | n/a | n/a |
+| `disk_destruction` | 28 | 28 | 92.86% | n/a | n/a |
+| `resource_dos` | 4 | 4 | 75.00% | n/a | n/a |
+| `payload_decode_exec` | 6 | 6 | 100.00% | n/a | n/a |
+| `agent_log` | 307 | 0 | n/a | 4.56% | 7.51% |
+| `issue_1` | 10 | 10 | 60.00% | n/a | n/a |
+| `issue_2` | 10 | 10 | 100.00% | n/a | n/a |
+| `issue_3` | 2 | 0 | n/a | 0.00% | 65.76% |
+| `pipe_filter` | 10 | 0 | n/a | 10.00% | 40.42% |
+| `near_miss` | 18 | 0 | n/a | 16.67% | 39.22% |
+| `routine_dev` | 13 | 0 | n/a | 30.77% | 57.63% |
+The stored JSON also contains per-family results for `block_0_1pct_fpr`, confusion counts, ranking metrics and issue-specific checks. This card reports that source without asserting additional split-independence guarantees.
+</details>
+### JSON export parity
+The repository's validation report compares the exported model with the fitted in-memory model on **5,948 examples**. It records **zero changed decisions** at either threshold policy and a maximum absolute score error of approximately **8.53 × 10⁻⁶**. This validates export parity on those examples; it is separate from detection quality.
+## Quickstart
+### Install
 ```bash
 python -m pip install "numpy>=1.24" "scikit-learn==1.8.0" "huggingface_hub>=1.3"
 ```
+### Score command text locally
+Download the A1 model together with its matching loader and tokenizer:
 ```python
 import sys
 model_dir = snapshot_download(
     repo_id="patronus-studio/cat-shell-risk-commander",
+    allow_patterns=[
+        "model.json",
+        "shell_cat_guard_a1.py",
+        "shell_tokens.py",
+    ],
 )
 sys.path.insert(0, model_dir)
+from shell_cat_guard_a1 import ShellRiskClassifier
 classifier = ShellRiskClassifier(f"{model_dir}/model.json")
+# Only the text is classified; these commands are not executed.
+result = classifier.score("git status --short")
+print(result)
 ```
+Each result includes `label`, `flagged`, `decision_score`, `threshold` and `threshold_policy`. The decision score is a raw linear logit, not a calibrated probability. A flag is raised when **`decision_score > threshold`**; equality does not raise a flag.
+### Batches and threshold policies
 ```python
 scores = classifier.decision_function([
     "Get-ChildItem",
     "dir",
 ])
 result = classifier.score(
     "git status --short",
+    threshold_policy="block_0_1pct_fpr",
 )
 ```
+`decision_function` expects a list of strings and returns raw scores. Use `score` for one text and a threshold-based result. Select a policy using representative validation traffic from your application.
+## Model and thresholds
+This release has one A1 model with two stored operating points, rather than Transformer precision or ONNX variants.
+| Artifact / policy | What to choose it for |
+| :--- | :--- |
+| `model.json` + A1 loader | The current two-block classifier; 70,096 features, about 4.09 MB |
+| `external_commands_1pct_fpr` | Default policy; higher recall in the stored test |
+| `block_0_1pct_fpr` | Stricter score threshold; fewer false positives with lower recall in the stored test |
+The legacy `shell_cat_guard.py` expects JSON format version 1. The current `model.json` is format version 2 and requires **`shell_cat_guard_a1.py` plus `shell_tokens.py`**.
+## Training data and provenance
+The A1 manifest records **23,177 fitting examples**, including **6,253 positive examples**. The remaining 16,924 are negative examples.
+| Recorded source | Role in the A1 manifest |
+| :--- | :--- |
+| [Shell Safety](https://huggingface.co/datasets/tomngdev/shell-safety) | Positive and negative commands |
+| Shell Attack Evolution (`zyw-286/shell-attack-evolution-dataset` in the manifest) | Honeypot positives |
+| [Atomic Red Team](https://github.com/redcanaryco/atomic-red-team) | Positive attack commands |
+| [SWE-agent trajectories](https://huggingface.co/datasets/nebius/SWE-agent-trajectories) | Benign agent-command negatives; command strings only |
+| [ShellRisk-Bench](https://github.com/kontext-security/shellrisk-bench) | Original Bash risk target |
+The Shell Attack Evolution identifier is retained as recorded provenance; no accessible public Hub dataset was found at that identifier during card preparation.
+The fitted model combines 50,000 character-within-word n-gram features with 20,096 shell-token features. The shell-token block includes normalized tokens, token bigrams and markers for command heads, pipe targets and redirection targets. Feature extraction processes text; it does not invoke a shell.
+The export identifies its source experiment as `shellrisk-retrain-2026-10-23 (arm A1)`. The earlier article below describes the original ShellRisk experiments and the previous linear model. Its older model sizes, training counts and benchmark values should not be read as A1 results.
+[Earlier experiment report (English)](https://patronus.studio/en/posts/detecting-risky-shell-commands-shellrisk-benchmark) · [Deutsch](https://patronus.studio/posts/gefaehrliche-shell-befehle-erkennen-shellrisk-benchmark)
+Cat Shell Risk Commander is a Patronus model. It is not an official Kestrel release and does not contain the separate Hadamard encoder discussed in the earlier report.
+## Deployment considerations
+- The classifier predicts the source-defined risk label of submitted text. Dataset labels are not independently verified execution outcomes.
+- It does not execute commands, inspect your filesystem, resolve variables, decode payloads or simulate a shell. Effects that depend on missing payloads or environment context can be missed.
+- Literal feature patterns are not full shell parsing or document-wide semantic reasoning. Obfuscation, unseen patterns and comment padding can affect classification; the previous model's stress-test results do not establish A1 robustness.
+- Scoring the complete text does not imply a fixed latency or memory footprint. Input size affects cost, and no Cat Shell deployment latency or RAM benchmark is published in this card.
+- Source-specific tests, small subgroups and calibration-policy names do not establish performance on new production traffic. Validate thresholds on your workload.
+Use the score alongside execution permissions, sandboxing, explicit policy checks and review where required.
 ## Repository files
 | File | Contents |
+| :--- | :--- |
+| `model.json` | Current A1 vocabulary blocks, IDF values, linear coefficients, intercept and threshold policies |
+| `shell_cat_guard_a1.py` | Matching format-version-2 CPU loader and result schema |
+| `shell_tokens.py` | Shell-token feature extraction used by A1 |
+| `shell_cat_guard.py` | Legacy format-version-1 loader |
+| `requirements.txt` | Reference Python dependencies |
+| `evaluation.json` | Current stored A1 results, confusion counts and per-family metrics |
+| `validation.json` | Export parity report |
+| `manifest.json` | Artifact hashes, source experiment and training-source records |
+| `LICENSE` | Apache License 2.0 |
+The JSON format contains fitted inference parameters and does not require loading a pickle. The reference Python loader reconstructs the TF-IDF vectorizers and applies the linear decision rule.
+## Citation and license
+Released under **Apache License 2.0**. Training-source licenses and provenance are recorded in [manifest.json](https://huggingface.co/patronus-studio/cat-shell-risk-commander/blob/main/manifest.json).
+```bibtex
+@misc{catshellriskcommander2026,
+  title = {Cat Shell Risk Commander: Local Shell Risk Classification},
+  author = {Patronus Protect},
+  year = {2026},
+  howpublished = {\url{https://huggingface.co/patronus-studio/cat-shell-risk-commander}}
+}
+```
+## Patronus Protect
+**Local protection for prompts, tools and documents.**
+Explore Patronus Protect, or use our open models in your own application.
+[Try Patronus Protect for free →](https://patronus.studio) · [Explore our AI models →](https://patronus.studio/product/ai-models) · [Wolf Defender Jev →](https://huggingface.co/patronus-studio/wolf-defender-jev)

assets/cat-shell-risk-commander-banner.png ADDED Viewed

comparison/compare_shellrisk.py ADDED Viewed

	@@ -0,0 +1,66 @@

+"""Compare Cat Shell A1 with published Kestrel verdicts; command text is never executed."""
+import argparse
+import hashlib
+import json
+from pathlib import Path
+import sys
+import pyarrow.parquet as pq
+def metrics(labels, predictions):
+    tp = int(sum(y and p for y, p in zip(labels, predictions)))
+    fp = int(sum(not y and p for y, p in zip(labels, predictions)))
+    fn = int(sum(y and not p for y, p in zip(labels, predictions)))
+    tn = int(sum(not y and not p for y, p in zip(labels, predictions)))
+    return dict(n=len(labels), tp=tp, fp=fp, fn=fn, tn=tn,
+                precision=tp/(tp+fp), recall=tp/(tp+fn),
+                fpr=fp/(fp+tn), f1=2*tp/(2*tp+fp+fn))
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--a1-source', type=Path, required=True)
+    parser.add_argument('--gold', type=Path, required=True)
+    parser.add_argument('--kestrel-verdicts', type=Path, required=True)
+    parser.add_argument('--output', type=Path, required=True)
+    args = parser.parse_args()
+    sys.path.insert(0, str(args.a1_source.resolve()))
+    from shell_cat_guard_a1 import ShellRiskClassifier
+    rows = pq.read_table(args.gold).to_pylist()
+    published = {r['id']: r['prediction'] == 'risky' for r in
+                 map(json.loads, args.kestrel_verdicts.read_text().splitlines())}
+    assert len(rows) == len(published) == 4194
+    assert set(published) == {r['id'] for r in rows}
+    model = ShellRiskClassifier(args.a1_source / 'model.json')
+    scores = []
+    for start in range(0, len(rows), 128):
+        scores.extend(model.decision_function([r['command'] for r in rows[start:start+128]]))
+    labels = [r['label'] == 'risky' for r in rows]
+    a1 = metrics(labels, [s > model.thresholds[model.default_policy] for s in scores])
+    kestrel = metrics(labels, [published[r['id']] for r in rows])
+    assert (kestrel['tp'], kestrel['fp']) == (178, 10)
+    stored = json.loads((args.a1_source/'evaluation.json').read_text())['original_bash_test'][model.default_policy]
+    assert all(a1[k] == stored[k] for k in ['n', 'tp', 'fp', 'fn', 'tn'])
+    report = {
+        'date': '2026-10-24',
+        'dataset': 'kontext-security/ShellRisk-Bench',
+        'split': 'test', 'cases': 4194, 'risky': 193, 'safe': 4001,
+        'kestrel': {'source': 'Published per-example Kestrel verdicts, native release behaviour', 'metrics': kestrel},
+        'cat_shell_a1': {'source': 'Replayed current format-version-2 export with shell_cat_guard_a1.py',
+                         'policy': model.default_policy,
+                         'threshold': model.thresholds[model.default_policy], 'metrics': a1},
+        'test_data_sha256': hashlib.sha256(args.gold.read_bytes()).hexdigest(),
+        'published_kestrel_verdicts_sha256': hashlib.sha256(args.kestrel_verdicts.read_bytes()).hexdigest(),
+        'cat_model_sha256': hashlib.sha256((args.a1_source/'model.json').read_bytes()).hexdigest(),
+        'notes': ['Both rows use the same fixed 4194-case Bash test. Kestrel values come from published verdicts, not a reimplementation.',
+                  'Different published/default threshold policies; not a common recalibration or identical-training-data architecture comparison.',
+                  'Same-source Bash benchmark; not an external-shell transfer or production evaluation.',
+                  'A1 confusion counts exactly match the current stored evaluation.']}
+    args.output.write_text(json.dumps(report, indent=2)+'\n')
+    print(json.dumps(report, indent=2))
+if __name__ == '__main__':
+    main()

comparison/kestrel-a1-comparison.json ADDED Viewed

	@@ -0,0 +1,51 @@

+{
+  "date": "2026-10-24",
+  "dataset": "kontext-security/ShellRisk-Bench",
+  "split": "test",
+  "cases": 4194,
+  "risky": 193,
+  "safe": 4001,
+  "kestrel": {
+    "source": "Published per-example Kestrel verdicts, native release behaviour",
+    "metrics": {
+      "n": 4194,
+      "tp": 178,
+      "fp": 10,
+      "fn": 15,
+      "tn": 3991,
+      "precision": 0.9468085106382979,
+      "recall": 0.9222797927461139,
+      "fpr": 0.0024993751562109472,
+      "f1": 0.9343832020997376
+    },
+    "model_revision": "6d6661cac29f8944328f90c8ed3240566cf9f6a1",
+    "artifact_sha256": "1df8b3e5f2bfc4e1fe95230ee9b3d37f63aaa461a8551982fcc0dd7103c8221b"
+  },
+  "cat_shell_a1": {
+    "source": "Replayed current format-version-2 export with shell_cat_guard_a1.py",
+    "policy": "external_commands_1pct_fpr",
+    "threshold": 0.65763,
+    "metrics": {
+      "n": 4194,
+      "tp": 164,
+      "fp": 10,
+      "fn": 29,
+      "tn": 3991,
+      "precision": 0.9425287356321839,
+      "recall": 0.8497409326424871,
+      "fpr": 0.0024993751562109472,
+      "f1": 0.8937329700272479
+    },
+    "model_revision": "2ba751f998dac81b00644fec3b7ca4bf57af08f4"
+  },
+  "test_data_sha256": "e9df3b50a716ad077eeab6fcc27be57834bc2ff3f28ebeaf48dfd6640e9064c8",
+  "published_kestrel_verdicts_sha256": "2d0a3fb080837d1d346da4c422faae4254454af25fb426c3ccba0beb92fb162a",
+  "cat_model_sha256": "73f4baaca07a97b5bdef8fd112084a0dfba8fead5ca7f3a0d41660b0ee673556",
+  "notes": [
+    "Both rows use the same fixed 4194-case Bash test. Kestrel values come from published verdicts, not a reimplementation.",
+    "Different published/default threshold policies; not a common recalibration or identical-training-data architecture comparison.",
+    "Same-source Bash benchmark; not an external-shell transfer or production evaluation.",
+    "A1 confusion counts exactly match the current stored evaluation."
+  ],
+  "dataset_revision": "437467862139b4e9cdd5322024ef3434a67c7ec8"
+}