Jev vs. Machine Learning
An empirical benchmark comparing Jev with conventional machine-learning pipelines across text and tabular classification tasks.
Where does Jev differ from classical ML?
Balanced accuracy on the fixed test holdout under Protocol 3.0.1. Inspect both raw default rules and calibrated policy thresholds.
Performance Range Comparison
Balanced Accuracy (%) on test holdout · One row per dataset · Direct signed Δ displacement
Per-Dataset Signed Difference (Δ)
Zero-centered displacement (Jev zero-shot minus best classical pipeline in percentage points)
AG News
Banking77
SMS Spam
IMDb
Bank Marketing
Online Shoppers
Breast Cancer
Iris
| Dataset ↕ | Domain ↕ | Classes ↕ | Test N ↕ | Jev Zero-shot ↕ | Jev Few-shot ↕ | Best Classical ↕ | Δ vs Best ↕ | Inspect |
|---|---|---|---|---|---|---|---|---|
| AG News | Text | 4 | 1,000 | 87.5% ±0.0 | 86.3% ±0.6 | 88.4% (SVM) | -0.9% | inspect → |
| Banking77 | Text | 77 | 1,500 | 78.9% ±0.0 | 81.9% ±1.7 | 89.7% (SVM) | -10.8% | inspect → |
| SMS Spam | Text | 2 | 1,000 | 96.1% ±0.0 | 95.6% ±0.9 | 95.0% (Naive Bayes) | +1.1% | inspect → |
| IMDb | Text | 2 | 1,000 | 96.3% ±0.0 | 95.9% ±0.5 | 88.4% (Logistic regression) | +7.9% | inspect → |
| Bank Marketing | Tabular | 2 | 1,000 | 53.4% ±0.0 | 55.3% ±3.1 | 71.8% (SVM) | -18.4% | inspect → |
| Online Shoppers | Tabular | 2 | 1,000 | 51.4% ±0.0 | 54.7% ±9.5 | 69.1% (SVM) | -17.7% | inspect → |
| Breast Cancer | Tabular | 2 | 114 | 61.0% ±0.0 | 88.8% ±5.5 | 100.0% (SVM, k-NN) | -39.0% | inspect → |
| Iris | Tabular | 3 | 30 | 97.0% ±0.0 | 94.5% ±4.8 | 100.0% (Logistic regression, SVM, Extra trees, Naive Bayes, Hist gradient boost, Voting ensemble) | -3.0% | inspect → |
Key Empirical Observations
Data-grounded findings across domain types, policy thresholds, and conditioning strategies.
IMDb Sentiment Standout
On IMDb, zero-shot Jev reaches 96.3% balanced accuracy vs. 88.4% for TF-IDF Logistic Regression (+7.9 pp lead). The advantage persists under policy calibration (96.1% vs 88.3%).
Thresholding Changes SMS Spam
Under raw decisions, Jev leads Naive Bayes (96.1% vs 95.0%). Following policy threshold optimization, Naive Bayes reaches 96.3% against Jev's 95.9%.
Tabular Business Tasks
On Bank Marketing, adjusted zero-shot Jev reaches 59.7% vs. 73.3% for the voting ensemble. On Online Shoppers, Jev scores 51.8% vs. 71.2%.
Selective Few-Shot Efficacy
One example per class lifts raw Breast Cancer from 61.0% to 88.8% and Banking77 from 78.9% to 81.9%, while slightly reducing means on AG News and IMDb.
Interactive Methodology Pipeline
Protocol 3.0.1 strictly isolates candidate selection, decision-threshold calibration, and final holdout evaluation. Click any stage to inspect.
01 · Data Split & Holdout Isolation
A dedicated test holdout is isolated before any model exploration (Seed 20260920). It remains completely untouched during feature extraction, hyperparameter tuning, and threshold selection.
Model Architecture Matrix
11 classical model families, 2 Jev prompt formulations, and a majority-class reference baseline.
| Model Family | Feature Representation | Backend Runtime | GPU Support | Tuning Budget |
|---|---|---|---|---|
| Jev Zero-Shot | Structured text / feature prompts | Jev API (jev-1.13.0) |
● (Cloud) | 1 config |
| Jev Few-Shot | In-context 1-example per class | Jev API (jev-1.13.0) |
● (Cloud) | 1 config |
| Logistic Regression | TF-IDF / StandardScaler | cuML / scikit-learn | ● | 4 candidates |
| Linear / RBF SVM | TF-IDF / Dense Scaled | cuML / scikit-learn | ● | 4 candidates |
| Random Forest | Dense / Sparse Scaled | cuML / scikit-learn | ● | 4 candidates (max 150 trees) |
| Extra Trees | Dense / Sparse Scaled | scikit-learn | — | 4 candidates (max 150 trees) |
| k-Nearest Neighbors | TruncatedSVD (300d) / Dense | cuML / scikit-learn | ● | 4 candidates |
| Naive Bayes | Count / TF-IDF / Gaussian | scikit-learn | — | 4 candidates |
| Hist Gradient Boosting | Binned Numerical / Dense | scikit-learn | — | 4 candidates (max 60 iters) |
| XGBoost | Tree Features / DMatrix | CUDA / CPU | ● | 4 candidates |
| CatBoost | Native Categorical & Numerical | CUDA / CPU | ● | 4 candidates |
| Voting Ensemble | Probability Aggregation | CPU / Mixed | — | Composite soft voting |
| Majority Baseline | Constant class assignment | Reference | — | 1 config |
Verification & Execution
Execute offline validation tests or reproduce the benchmark in a CUDA GPU environment.
Analytical Boundaries
Explicit documentation of boundary conditions, statistical assumptions, and non-claims.
Statistical Inference
Reported ± values represent sample standard deviations across 3 random training seeds on the same test set. They do not constitute formal confidence intervals.
Cached Zero-Shot Calls
Identical successful API requests are cached and reused across seeds. Zero variance reflects deterministic prompt caching rather than independent API replications.
Banking77 API Failures
Requests yielding unparseable or out-of-vocabulary label predictions were assigned failure code -1 and counted as incorrect.
Sample Size Constraints
Iris (n=30) and Breast Cancer (n=114) feature compact holdouts. Near-perfect scores on these samples should not be generalized to larger distributions.
Classical Baseline Scope
Classical pipelines operate under a strict computational budget (max 150 trees, 60 hist iters). They do not represent fine-tuned deep transformers.
Latency & Cost Scope
The benchmark logged 38,922 request attempts with an estimated ~$4.19 input cost. This is an empirical run estimate, not an SLA or billing quote.
Download Data & Notebooks
All tables, source code, and frozen notebook outputs are open-source.