HackerRank's open-source AI resume screener scored one resume from 66 to 99

HackerRank open-sourced its applicant-tracking system, a GitHub project called hiring-agent that racked up thousands of likes on LinkedIn and Reddit. When the author ran the same resume through it a hundred times with no changes, scores ranged from 66 to 99. Against a typical 85-point cutoff, that identical resume would be rejected 65% of the time — pure luck deciding who advances. The pipeline parses a PDF, calls an LLM six times to extract structured fields, scrapes the candidate’s GitHub, then grades everything in one pass using gemma3:4b at temperature 0.1.

The variance isn’t random across the board — it tracks what LLMs can and can’t do. Technical skills, essentially a checklist of whether you know React or Python, scored 8/10 in 98 of 100 runs. Projects, which require a subjective quality judgment, swung wildly, with the model alternately praising ‘real-world deployment’ and dinging ‘architectural complexity’ for the same work. Dropping to temperature 0 didn’t fix it, and a GitHub issue from October documented the same non-determinism — making this a design flaw, not a tuning problem. Swapping in Gemini tightened the spread (48–64) but still failed the resume 28% of the time at a 60-point cutoff. The experience category, by contrast, was perfectly consistent and perfectly useless: a single internship and a decade of distributed-systems work both earn 25/25, because the prompt is two lines with no rubric or anchors — and the underlying template even hardcodes ‘Software Intern.’

The broader point is a warning about AI resume screening. LLMs are genuinely good at parsing documents and matching skill checklists, but asking one whether someone’s experience is worth 18 or 24 points yields a vibe-check — exactly the inconsistent gatekeeping HR has spent decades trying to eliminate. Weighting open source and personal projects at 65% of the score compounds the problem, structurally penalizing strong engineers whose best work never landed on a public GitHub. A screener that can’t reliably distinguish quality isn’t filtering for it; it’s just filtering, and rejection becomes a function of bad luck rather than fit.