Instructions to use getSTEAV/system-one-fdb-fakejob with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Laya
How to use getSTEAV/system-one-fdb-fakejob with Laya:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
System One fraud detector for FDB fakejob: Fake Job Postings (EMSCAD)
This model detects job postings that are scams rather than genuine vacancies. It is a two-stage stack: a gradient-boosted tree model scores each posting from engineered features, and System One, a typed-decision model (Laya ModernBERT-large encoder with a LoRA adapter and a decision head), reads that score, a few readable engineered fields and the original fields (long text cut to its first 700 characters) as JSON evidence, and returns a calibrated probability that the posting is fake.
On FDB's official test split (3,576 postings, 187 fake) it scores 0.9985 AUROC and catches 97.3% of fake postings at a 1% false-positive rate, against the best published 0.998 AUROC (AutoGluon) and 92.5% recall (AutoGluon).
It was trained for the Fraud Dataset Benchmark (FDB) and is a benchmark model, not a production fraud system; read Limitations first.
Results
Official FDB test split, scored once per model. AUROC and recall at 1% FPR follow FDB's own evaluation (recall is np.interp(0.01, fpr, tpr) on the test ROC). 95% confidence intervals: class-stratified bootstrap, 1,000 resamples.
| Model | AUROC [95% CI] | Recall at 1% FPR [95% CI] | Average precision | Recall at 0.1% FPR |
|---|---|---|---|---|
| This model: System One | 0.9985 [0.9970, 0.9996] | 97.3% [94.7, 99.5] | 0.9837 | 86.6% |
| Tree alone (System One's input) | 0.9974 [0.9949, 0.9993] | 96.3% [93.6, 98.9] | 0.9787 | 87.7% |
| Blend fitted on System One's holdout split (not released) | 0.9984 [0.9968, 0.9995] | 96.8% [94.1, 98.9] | 0.9828 | 87.7% |
| Best published AUROC: AutoGluon | 0.998 | |||
| Best published recall at 1% FPR: AutoGluon | 92.5% |
AUROC leaders are from the FDB paper (arXiv 2208.14417 v3). The paper reports no recall, so recall leaders are from the results table in FDB's README at commit 54cdefa211; later versions of the README keep that table inside an HTML comment, so it no longer renders. With 187 fake postings in the test set, each one is worth 0.53 points of recall.
How the released configuration was chosen. All models were trained and calibrated without test data, and each test set was scored once per candidate. After the test results were known, one model per benchmark was released: System One on every set (for a uniform, calibrated decision interface), and on this set it is also the strongest candidate on both headline metrics. Choosing among candidates after seeing test results can flatter the headline slightly, so the table shows every candidate.
Model details
- Developed by: STEAV (steav.io), trained with CID Model Studio.
- Model type: stacked binary classifier: gradient-boosted trees, then System One with offset stacking (System One adds a learned correction to the tree's log-odds).
- System One base: convaiinnovations/laya at revision
7b928d828b7b(Apache-2.0), a ModernBERT-large encoder (about 395M parameters) with Laya's typed decision head. The base weights are downloaded from the Hub at that pinned revision and checked against their sha256; they are not redistributed here. - Trained parameters: a LoRA adapter (rank 16, alpha 32, on the attention
Wqkv/Woand MLPWoprojections of all 28 layers; 4.39M parameters) and the decision head (26.2M parameters, initialised from Laya's head; its 0.26M-parameter action sub-head stays frozen). - Question: "Is this job posting fraudulent, meaning it is a fake posting rather than a genuine vacancy?" (a yes/no decision; the output is P(yes)).
- License: Apache-2.0 for the weights and code; some data files carry their own terms (see License and attribution).
- Version: 1.0.0.
- Contact: questions and issues go to this repository's Community tab.
Uses
Intended: benchmark research and comparison on FDB; a reference implementation of stacking a typed-decision model on a tree model's score; a starting point for fine-tuning on your own data.
Out of scope: automatically rejecting real job applications or employers without human review; any automated decision about a person without human review; use on data from a different distribution without retraining and validation.
How to use
Python 3.12 with the pinned versions in requirements.txt. On the reference GPU (NVIDIA GB10 in a DGX Spark, torch 2.11.0, bfloat16) the code reproduced the evaluated scores bit for bit; other GPUs and CPUs give slightly different scores (see Reproducibility). On CPU (Apple M4 Max, macOS arm64, float32) System One takes about 0.77 s per posting, about 0.8 h for the 3,576-row test set.
hf download getSTEAV/system-one-fdb-fakejob --local-dir system-one-fdb-fakejob
cd system-one-fdb-fakejob && pip install -r requirements.txt
# run from the repository folder
from system_one_fraud import FraudPipeline
from system_one_fraud.fdb import load_split
pipe = FraudPipeline.from_pretrained(".") # the tree, the adapter and the pinned Laya base
train, test, labels = load_split("fakejob") # FDB's loader (needs Kaggle API credentials)
p_fraud = pipe.predict_proba(test) # one probability per row
predict_proba returns, per row, the probability that the posting is fake. output="tree" returns the tree model's probability alone. states, tree_out = pipe.states(rows) returns the exact JSON evidence System One reads and the tree model's outputs. examples/quickstart.py runs the same steps on a few test rows.
The adapter does not load with PeftModel.from_pretrained alone: Laya's root config.json is a stub that transformers cannot instantiate, and the decision head and evidence format live in the bundled system_one_fraud package. On macOS the tree model runs in a helper Python process, because PyTorch and the gradient-boosting libraries load two OpenMP runtimes that crash in one process; results are identical. System One's bfloat16 GPU scores depend slightly on how rows are batched (padding); the reference scores used batch size 64 in file order.
Input contract. Each row needs the posting fields title, company_profile, description, requirements, benefits, location, department, salary_range, employment_type, required_experience, required_education, industry, function, telecommuting, has_company_logo and has_questions. Every other column (including Kaggle's fraudulent label and job_id) is ignored, and a missing field raises an error. Every row is scored against the bundled training reference, so scoring postings that were in the training data is meaningless: they match themselves (on 1,000 training postings the tree scores 1.0000 AUROC). 14,304 of the 17,880 postings in the public EMSCAD/Kaggle file are in that reference. Batching rows differently changes the tree score only in the last floating-point bit (about 1e-16); the reference scores come from scoring the 3,576 test rows in one call, in file order.
Training data
- Source: The Employment Scam Aegean Dataset (EMSCAD), 17,880 real job ads published through the Workable applicant-tracking system in 2012-2014, of which 866 are fraudulent (Vidros et al., 2017), as loaded by FDB from the Kaggle copy
shivamb/real-or-fake-fake-jobposting-prediction(version 1). - Benchmark split: FDB's random 80/20 split (fixed seed). Training: 14,304 postings (679 fake, 4.75%). Test: 3,576 postings (187 fake).
- Licence: EMSCAD was released publicly by the Laboratory of Information and Communication Systems Security, University of the Aegean, for research and for training anti-scam classifiers (its paper expects applicant-tracking vendors to train classifiers on it). The creators never attached a formal licence; the Kaggle copy is labelled CC0 by its re-uploader. No posting text ships in readable form, but the near-copy reference vectors and lookup tables let much of the training text be reconstructed (see Embedded data).
Training procedure
Stage 1, tree model. An equal-thirds blend of (1) five histogram gradient-boosting stackers (seeds 0-4), (2) three LightGBM stackers (seeds 0-2), both over 87 features computed from the full posting text, and (3) the version-1 stack, a histogram gradient-boosting stacker over the same 87-feature recipe computed from text cut to 700 characters (120 for short fields). Each component is Platt-scaled to log-odds; the blend gets a final Platt and isotonic calibration. The 87 features are posting metadata, seven categorical fields, five class-balanced logistic-regression text models over word and character n-grams, seven nearest-neighbour "near copy of a known fake" features, and out-of-fold target encodings (smoothed fake rate and count) of 11 keys: seven field values (title, location, city, department, salary range, industry, function) and hashes of the case- and whitespace-normalised company profile, description, requirements and benefits. All fitted statistics come from training rows only; training rows get out-of-fold values. The blend was chosen from a small, pre-declared candidate set (the version-1 recipe on full text, extra features, a LightGBM stacker, seed bags, blends with the version-1 stack) by the mean over three repeats of stratified 5-fold cross-validation on all 14,304 training postings of pooled AUROC, average precision and recall at 1% FPR. Because the choice was made on the same out-of-fold scores, those development figures are slightly optimistic; the test set played no part.
Training rows carry out-of-fold tree scores: the models that scored a training row never saw it, so System One never learns from a score fit on its own row. Only the final Platt and isotonic calibration maps were fitted on all training rows' out-of-fold scores and labels.
Stage 2, System One. Each training row is rendered as a JSON state: the tree's out-of-fold score (Tree_fraud_score), its percentile among all out-of-fold training scores (Tree_percentile), readable engineered fields (Context_*) and the original fields. As in training, System One is given only the first 700 characters of each long text field (company profile, description, requirements, benefits) and the first 120 of the other fields; on the official test set this shortens the description of 68.4%, the company profile of 46.1%, the requirements of 37.6% and the benefits of 12.9% of postings that have the field. System One trained on the 14,304 FDB training postings at their natural fake rate; CID Model Studio removed 55 rows whose evidence duplicated another row, leaving 14,249. The rows were split 11,399 / 1,424 / 1,426 (train / validation / holdout). System One reads at most 1,024 tokens and cuts the end of the evidence (keys are sorted) when it does not fit: on the official test set this happens on 7 of 3,576 rows (0.2%). It drops the end of description on 0.1% and requirements on 0.2% of the rows that have the field; the tree's own fields always fit.
- LoRA r = 16, alpha = 32, dropout 0.05; learning rate 1e-4 (head 5e-5), weight decay 0.01, 5% warm-up, batch 8 with 4-step gradient accumulation (effective 32); bfloat16; maximum sequence length 1,024 tokens; seed 42.
- 2 supervised epochs with soft cross-entropy, then one RLCD epoch (reinforcement learning on the decision distribution with proper-scoring-rule rewards, following Laya's method).
- 1,071 optimiser steps, 19.2M training tokens, 2.0 h on one NVIDIA GB10 (DGX Spark).
- Calibration: one temperature fitted on the validation split (L-BFGS).
- Validation: accuracy 0.9930, expected calibration error 0.0032.
Evaluation
- Protocol. FDB's official split; the test set was never used for training, feature selection or calibration, and labels were read only to compute the metrics.
- Calibration on test (10 equal-width bins, released output): expected calibration error 0.0047, Brier score 0.00573.
- Cost of high detection (released output): catching 95% of fake postings requires a false-positive rate of 0.44%, and catching 99% requires 4.04%.
- Reproduce:
python eval/reproduce_fdb.pyrebuilds the split with FDB's own loader, checks it against the frames used here, scores it with the released pipeline and compares the metrics witheval/test_metrics.json.
Limitations
- The benchmark rewards memorisation. FDB's split is random, and near-duplicate postings straddle it: 78 official test postings are exact copies of training postings (107 are identical in the first 700/120 characters of every field), and in cross-validation about 240 of the 679 training fakes had a near-copy (cosine >= 0.9) among the fakes in other folds. The near-copy and exact-hash features exploit this: in repeated 5-fold cross-validation on the training postings the released tree recipe is near perfect on postings with a near-copy and scores 0.991 AUROC and 88.3% recall at 1% FPR on postings without one. There is no such figure for System One, and genuinely new scam styles will score lower than the headline.
- Do not score the training postings. Every posting is compared with the bundled training reference, so a posting from the training data matches itself: on 1,000 training postings the tree alone scores 1.0000 AUROC. 14,304 of the 17,880 postings in the public EMSCAD/Kaggle file are in that reference, so evaluate only on postings outside it.
- Era and source. The ads are from 2012-2014 and from a single applicant-tracking platform; scam language, platforms and labour markets have changed since.
- English only. The text models and System One's inputs are English.
Bias, risks and ethical considerations
Fraud models make errors in both directions: false positives block or delay legitimate customers, and their burden can fall unevenly across groups. No fairness evaluation was done for this model; the benchmark data carries no protected attributes we could audit. Use the score as one input to a reviewed decision, monitor error rates on your own population, and retrain when the data drifts.
Reproducibility
These checks passed before release:
- Tree model. Refit from scratch, the tree model in
tree/reproduces the evaluated tree scores bit for bit on all 3,576 test rows (Apple M4 Max, macOS arm64, the pinned versions), when the test rows are scored in one call; other batchings differ by about 1e-16. On other platforms (other BLAS or OpenMP builds) the last bits can differ, which can occasionally change theTree_fraud_scoretext System One reads. - Evidence. The state builder reproduces the evaluated System One inputs exactly on all 3,576 test rows.
- System One. The inference code in
system_one_fraud/reproduces the evaluated scores bit for bit on all 3,576 test rows on the reference GPU (NVIDIA GB10 in a DGX Spark, torch 2.11.0, bfloat16, batch size 64 in file order, base downloaded from the Hub), run in the training environment with the same pinned versions rather than a fresh install. - Fresh install. A new virtual environment built from
requirements.txt(Apple M4 Max, macOS arm64, pandas 3.0.6, torch 2.11.0) reproduces the tree scores bit for bit and the System One inputs exactly. - Fresh download. FDB's loader, run fresh with the pinned pandas, rebuilds the evaluated frames exactly (content hashes in
eval/fdb_split_hashes.json) apart from the identifier and timestamp columns FDB randomises on every load, which the models never read. - End to end on CPU.
eval/reproduce_fdb.py(fresh download, released pipeline, Apple M4 Max, macOS arm64, float32) gives 0.9985 AUROC and 97.3% recall at 1% FPR, against the reported 0.9985 and 97.3% (identical to 4 decimals).
Measured, not gated: CPU drift. On CPU in float32, System One's scores differ slightly from the bfloat16 GPU reference. On 313 holdout postings (57 fake) the mean absolute difference in probability is 4.6e-04 and the largest 2.3e-02; decisions at 0.5 agree on 100.0% of rows, and AUROC on that sample is 0.9968 on the GPU and 0.9968 on CPU. The largest difference in any of the 3 CPU comparisons was 3.3e-02, on the first 256 test postings, in the clean-install check.
Environmental impact
System One training used one NVIDIA GB10 (DGX Spark, on premises) for 2.0 h, roughly 0.5 kWh at the system's 240 W rating (an upper-bound estimate). The tree models trained in minutes on a laptop CPU.
Files
| Path | Contents |
|---|---|
adapter_model.safetensors, adapter_config.json |
LoRA adapter |
system_one/head.safetensors |
decision head |
system_one/config.json, system_one/calibration.json, release.json |
head configuration, temperature, release manifest (question, pinned base, priors) |
system_one/percentile_reference.npy |
sorted out-of-fold training tree scores, for the Tree_percentile evidence field |
tree/ |
the nine blended components with their own feature pipelines: five scikit-learn histogram gradient-boosting stackers and the version-1 stack (skops; each file trusts only the types it contains, from a fixed allowlist), three LightGBM stackers (model-text format), TF-IDF vectorisers over hashed vocabularies, a hashed character 2-5-gram idf table and class-balanced logistic-regression text models (skops), the near-copy reference vectors, target-encoding and ordinal tables, Platt and isotonic calibration maps (JSON), and manifest.json with every file's size and sha256 (checked on load; unlisted files are refused) |
system_one_fraud/ |
inference code: pipeline.py (end to end), state.py (JSON evidence), system_one.py (System One), tree_fakejob.py (tree model), tree_runner.py (macOS helper process), fdb.py (FDB loader) |
eval/ |
reproduce_fdb.py, the split hashes and the test metrics |
examples/quickstart.py |
load, score and print a few test rows |
LICENSE, NOTICE |
licence texts and attributions |
requirements.txt |
pinned dependencies |
checksums.sha256 |
sha256 of every file except this README |
Embedded data. No posting text ships in readable form: an independent scan of all 290 MB of files, with the skops archives unpacked (2,996 training-text probes in several encodings), found no match. But much of the training text can be reconstructed. Vocabulary terms, category values and lookup keys are stored as keyed 64-bit BLAKE2b digests whose key ships with the model, so anyone can hash a guessed word and test it, and the vocabulary columns are in alphabetical order of the hidden terms. The near-copy features need the TF-IDF vector of each of the 14,304 training postings over word 1-2-grams, with its fake/genuine label (tree/new/; tree/old/ holds a second set over the 700/120-character text). Term counts are exactly recoverable from these vectors. An independent test using only this repository and a stock word list recovered, for 300 random training postings, a median of 68% of each posting's words with exact counts, 41% of adjacent word pairs, and runs of about 11 consecutive words, and rebuilt 1,126 exact training titles, which reveals the label of each posting whose title is unique (397 of them). These are lower bounds: a larger word list would recover more. The target-encoding tables hold, per hashed value (a title, location or normalised description), how many training postings had it and how many were fake, so anyone who holds or guesses such a value learns its training label.
Citation
@misc{steav2026systemonefdb,
title = {System One fraud detectors for the Fraud Dataset Benchmark},
author = {{STEAV}},
year = {2026},
howpublished = {\url{https://huggingface.co/getSTEAV/system-one-fdb-fakejob}}
}
@misc{grover2023fraud,
title = {Fraud Dataset Benchmark and Applications},
author = {Prince Grover and Julia Xu and Justin Tittelfitz and Anqi Cheng and Zheng Li and Jakub Zablocki and Jianbo Liu and Hao Zhou},
year = {2023},
eprint = {2208.14417},
archivePrefix = {arXiv},
primaryClass = {cs.LG}
}
@misc{convai2026laya,
title = {Laya: a non-autoregressive System 1 decision model},
author = {{Convai Innovations}},
year = {2026},
howpublished = {\url{https://huggingface.co/convaiinnovations/laya}},
note = {Revision 7b928d828b7b0e022f929d9bd2e44165aa270148}
}
@misc{warner2024modernbert,
title = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
author = {Benjamin Warner and Antoine Chaffin and Benjamin Clavi{\'e} and Orion Weller and Oskar Hallstr{\"o}m and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
year = {2024},
eprint = {2412.13663},
archivePrefix = {arXiv},
primaryClass = {cs.CL}
}
@article{vidros2017emscad,
title = {Automatic Detection of Online Recruitment Frauds: Characteristics, Methods, and a Public Dataset},
author = {Vidros, Sokratis and Kolias, Constantinos and Kambourakis, Georgios and Akoglu, Leman},
journal = {Future Internet},
volume = {9},
number = {1},
pages = {6},
year = {2017}
}
License and attribution
| Files | Terms |
|---|---|
tree/new/ and tree/old/: knn_ref.*, te.*, ordinal.*, vec.*.digests.npy, vec.*.tfidf.skops and char_idf.skops |
derived from EMSCAD postings, which carry no formal licence (see Training data); Apache-2.0 covers only STEAV's contribution |
| everything else (weights, code, configuration, model files) | Apache-2.0 (LICENSE) |
Attributions are collected in NOTICE.
- Data: Trained on the Employment Scam Aegean Dataset (EMSCAD), Laboratory of Information and Communication Systems Security, University of the Aegean (S. Vidros, C. Kolias, G. Kambourakis, L. Akoglu, "Automatic Detection of Online Recruitment Frauds: Characteristics, Methods, and a Public Dataset", Future Internet 9(1):6, 2017), via the Kaggle copy shivamb/real-or-fake-fake-jobposting-prediction. Files derived from the postings (tree/new/ and tree/old/ reference vectors, lookup tables and digests) carry no licence from the dataset's creators; the Apache License covers only STEAV's contribution to them.
- Base model: Laya by Convai Innovations (Apache-2.0), built on ModernBERT-large by Answer.AI and LightOn (Apache-2.0). The sequence layout and the decision-head architecture follow Laya's reference code.
- Benchmark: Fraud Dataset Benchmark, Grover et al. 2023 (MIT; Copyright (c) 2021-2022 Prince Grover and Zheng Li, and (c) 2022 Jianbo Liu, Jakub Zablocki, Hao Zhou, Julia Xu and Anqi Cheng).
- Downloads last month
- -
Model tree for getSTEAV/system-one-fdb-fakejob
Base model
convaiinnovations/laya