Building PaperCheck: the detector, the plumbing, and everything that broke

October 2026. I built this over about a week, mostly on free tiers and student credit. Code and model are private. The numbers below are measured unless I say "estimate".

Why I built it

Everyone in my college has the same fear in the last week of a project: will the report get flagged by Turnitin. People paste their work into three different detectors, get 3%, 40% and 70%, and have no idea which one to believe. I wanted something that did one honest job: take a report, score every sentence, show exactly which lines look machine-written and why, so you can rewrite them yourself before you submit.

It also checks plagiarism against arXiv, because a lot of project reports lean on papers they cite and sometimes lean a bit too hard.

I used AI coding assistants a lot while building it (Claude Code mostly), and I reviewed and tested what they wrote. Most of what follows is what happened when real students started using it.

What it looks like from the outside

You sign in with Google, upload a Word, PDF or text file, and about a minute later you get a score, a highlighted copy of your paper, the reason for each flagged line, and an email with the link. Five free checks a month, up to 20,000 words each. There's also an MCP server, so Claude or ChatGPT can run the check on your own document from inside a chat.

Underneath it is a Cloudflare Worker (login, limits, payments, a D1 database) in front of a Python engine on Azure that does the heavy work. The diagrams are in the architecture page; here I'll stick to what was hard.

Getting the detector to behave

This took most of the week and most of the failed experiments.

The first version ran two open detectors over 5-sentence windows (each sentence plus the two before and two after) and reported the share of words in flagged sentences. I then compared it with real Turnitin reports for a handful of papers. It was about 27 points off on average (46 vs 60, 4 vs 33, 45 vs 28, 80 vs 32). No fixed multiplier fixed it; the leave-one-out error stayed at 35 to 40 points. I deleted the "Turnitin estimate" and stopped promising to match it.

That was the first real decision: judge the detector on text where I know who wrote it, not on whether it agrees with Turnitin. Turnitin's per-sentence labels on one paper were basically unlearnable (AUROC 0.42 to 0.47 for every open detector I tried), and Turnitin isn't ground truth anyway.

Then I trained my own. I fine-tuned a RoBERTa-base model, because I can't legally ship a fine-tune of the copyleft detector I'd started with. Each round taught me something I should have known already:

• Round 1 flagged real student papers at 100%. It had never seen a human student write anything. Fix: add human student text as negatives.
• Round 2 looked great on AI text and then flagged 73% of unseen human text. I had only measured how much AI it caught. Now every model is judged on held-out human writing from several domains first (academic papers, encyclopedia text, non-native English, learner essays), and catch rate comes second.
• A 3-class DeBERTa model (human, polished, AI) got 99% validation accuracy and caught only 28% of AI text on held-out data. The validation chunks came from the same papers as training. It had learned the genre, not the writing.
• A later model looked like 85.7% catch, until I cut every chunk to the same length. It dropped to 57%. The AI training text was chunked shorter than the human text, and the model had found the shortcut.
• A candidate with a great benchmark score turned out to be almost blind to Claude-style essays. I'd only tested it on older generators.

The rules I ended up with: split by document, never by chunk; choose the threshold on validation documents; report false positives per domain; match lengths; and withdraw any number I can't reproduce (I did, once, after finding my reconstruction of someone else's model was wrong).

What's live: the fine-tuned RoBERTa-base ("round 5"), a sentence counts as flagged above a high score cut-off, and a second view groups text into 250-word chunks like institutional tools do (a chunk counts if 28% of its words are flagged). On held-out human documents it flags 0% to 2.1% across five domains. The price is recall: it catches roughly 44% to 59% of newer-model text depending on the cut-off. I chose few false accusations over a high catch rate, and the page says plainly that the score is an estimate.

I also removed a "humanizer" I'd added, because a tool that helps text slip past checkers is the opposite of what this is for.

Making it fast

The queue. One engine had to serve many people. Each job lives in a folder; a worker claims a part of a job by creating a file with O_EXCL, which is atomic, so two servers can never run the same part. Workers touch the claim file every 30 seconds as a heartbeat, and a claim that goes quiet for three minutes is taken over. That let the engine restart in the middle of a check without losing it. The AI part and the plagiarism part (which mostly waits on arXiv) run side by side.

The bug that mattered most. Checks took 7 minutes at the median and up to an hour. Inside a container, os.cpu_count() returns the host's core count, not your 4 vCPUs, and I was passing it straight to the thread setting of the AI library. Dozens of threads fighting for 4 cores. On the same box a 4,000-word paper went from over 10 minutes to about a minute once I set the thread count explicitly. I had been staring at the models, not the container.

Four models, one that mattered. Each check ran four detectors. Only one sets the number users see (I timed them: 46%, 25%, 15% and 14% of model time). I kept the one that counts. The scores are identical to four decimals, because I score the same windows the same way.

Stuck checks. One check that hung held a worker slot for the full one-hour timeout. Twenty-one did that in a day, which is a fifth of my capacity. Now a stuck check is killed at a fixed limit, the user is refunded, and the slot is free.

Keeping it cheap and keeping it alive

My Azure bill went from about 60 rupees a day to 6,600 in about a week. Three causes, none of them the AI work:

1. The engine re-read about 900 old job folders on a billed file share every 30 seconds. Cloud file storage bills per operation, and that single loop cost thousands of rupees. Remembering finished jobs cut the operations by 90%, measured on a mock share.
2. Old server versions stayed running, each scaling up on its own.
3. The thread bug above made every check burn far more CPU than it needed.

Then Cloudflare's free database warned me at 75% of its daily write quota. I asked the database which queries wrote the most: one INSERT OR REPLACE that rewrote a device row on every progress poll was 94% of all writes. Writing it once and refreshing every six hours fixed it.

Then the student Azure account hit its spending cap and switched itself off, with real users on it. I moved the engine to another account. The new platform has no persistent disk, so I started saving finished reports into the Cloudflare database, compressed, for seven days. Before I switched, I re-scored a real production paper on the new host and compared it with what production had returned: identical on all 341 sentences. I also found that Azure's own image builder is blocked on student plans, so GitHub Actions now builds the image and pushes it to the registry.

Money and trust

Anything that touches money gets tested harder than the model does. A failed or cancelled check returns the user's words exactly once. "Exactly once" is enforced by inserting a claim row in the database inside the same transaction as the refund, so a double click or a retry can't pay twice. Opening an old finished report must never look like a lost check, which was easy to get wrong once reports stopped living on the engine. I wrote a fake database and a fake engine and ran about 40 cases: report served after the engine sleeps, lost check refunded once, failed check not refunded twice, retry back-off for emails.

One missed function name had stopped every result email for hours before I noticed. Every finished check now sends exactly one email, and there's a test for it.

How it grew

I launched on 15 October with Google login and a free tier, put up some guide pages and a machine-readable description of the site, and listed the MCP server in the public registries. Checks per day:

That's 311 accounts and a peak of about a million words in 24 hours. About 40% of the people who ran a check ran more than one, which told me the line-by-line report was worth more than a single number.

What I'd do differently

• Run and time it in a production-like container on day one, not on my laptop.
• Put cost and quota alarms in before launch.
• Never put a write in a path that runs on every poll.
• Keep state out of the container from the start.
• Next: keep the model loaded between checks (saves 10 to 15 seconds each), try int8 after a proper accuracy check, and move the engine to a permanent free host so a spending cap can't switch it off.

Limits

It's a one-person project. The score is an estimate, not proof, and other checkers will disagree with it. Plagiarism is checked against arXiv only, not the whole web. The model catches less of the newest generators than I'd like. The code and the trained model stay private, because the base models and some of the data come with licence limits.

The system is drawn out in full on the architecture page (also as a PDF).