We evaluate nine open-weight language models — spanning abliterated, uncensored-fine-tuned, purpose-trained offensive-security, and censored baselines — on a battery of representative red-team tasks (reverse-shell generation, SQL-injection auth bypass, attack-path reasoning from nmap output, CVE factual recall, and pentest report writing). Each model is benchmarked under a fixed neutral prompt and a per-model tuned prompt, with responses graded by Claude Haiku 4.5 against task-specific rubrics.
Three findings contradict common deployment assumptions. First, the purpose-trained offensive-security model (WhiteRabbitNeo 8B) ranks seventh of nine, behind multiple generalist abliterated models 3–4× its size; continued-pretraining on cybersecurity corpora does not beat activation-space refusal surgery when the underlying base model is larger. Second, the best censored model (Gemma4 26B) scored 4.04/5 under our baseline prompt — higher than every uncensored model — but dropped to 3.36/5 when we applied its recommended system prompt, which triggered a cliff refusal on the reverse-shell task. Third, prompt sensitivity (baseline→tuned delta) is small for abliterated models (|Δ|≤0.25) and large for SFT-uncensored Dolphin variants (|Δ|≥0.33), reflecting that abliteration operates on representations rather than policy, and so generalizes across prompts.
The winning model for interactive deployment is Qwen2.5-coder-abliterate (32B, abliterated), scoring 3.40/5 tuned with stable cross-prompt behavior. We release the full harness for independent replication and future-model evaluation.
| Rank | Model | Method | Score | TPS |
|---|---|---|---|---|
| 1 | qwen2.5-coder-abliterate (32B dense) | Abliterated | 3.40 | 2.1 |
| 2 | gemma4 (26B) | Censored | 3.36 | 17.6 |
| 3 | qwen3-coder-abliterated (30B MoE) | Abliterated | 2.92 | 24.4 |
| 4 | qwq-abliterated (32B reasoning) | Abliterated | 2.70 | 5.1 |
| 5 | qwen2.5-coder (7B) | Censored | 2.53 | 101.3 |
| 6 | dolphin-llama3 (8B) | Dolphin SFT | 2.09 | 96.2 |
| 7 | WhiteRabbitNeo (8B) | Purpose-trained offsec | 1.94 | 8.7 |
| 8 | dolphin-mistral (7B) | Dolphin SFT | 1.70 | 101.1 |
| 9 | wizardlm-uncensored (13B) | Old uncensored SFT | 1.20 | 10.1 |
Judge: Claude Haiku 4.5. Mean across 5 tasks, 1 seed per cell. See PDF for full methodology, per-task breakdown, and limitations.
| Models tested | 9 (7 uncensored/abliterated, 2 censored baselines) |
|---|---|
| Task bank | 5 tasks: reverse shell, SQL injection, nmap reasoning, CVE recall, pentest writeup |
| Prompt variants | Baseline (fixed, comparable) and Tuned (per-model, ceiling) |
| Judge | Claude Haiku 4.5 with task-specific rubrics |
| Metrics per cell | TTFT, tokens/sec, completion tokens, VRAM peak, refusal flag, pattern score, judge score |
| Runtime | ~90 minutes total, 90 cells, single-host run |
| Hardware | AMD Ryzen 5 9600X + RX 9070 XT (16 GB VRAM), ROCm 6.3, Ollama 0.20.2 |
Full harness source (task YAMLs, per-model tuned prompts, graders, runner, aggregator) is included as Appendices C and D of the paper. All nine model digests are published in §3.2 to enable exact replication. The full raw run.jsonl is retained internally and is available on request to researchers conducting replication or meta-analysis — contact Gray Owl Research.