An HR analyst is asked whether the company pays its data engineers in line with the market. The salary survey she has is a year old and lumps the role into a broad job family; the job boards hold hundreds of advertised ranges for that exact title, posted this month. Which competitors hire for the same skills is answered in the same place: job postings scattered over career pages, boards and aggregators.
This post explains what job posting data is, who builds or buys it and where it comes from, ranked from official statistics and public career-page endpoints down to scraped board pages; it then covers normalization, the quality traps, the legal frame, where proxies fit, and six short answers.
What is job posting data?
Job posting data is the content of online job advertisements captured as structured fields. A posting is written for a candidate; the dataset rewrites it for a query, so that "senior data engineer, Berlin, hybrid" becomes a row that can be counted, compared and tracked.
The core fields are the same whatever the source:
- Title as written, plus a normalized occupation
- Employer name, and whether the poster is the employer or an agency
- Location: city, country, and a remote or hybrid flag
- Salary: minimum, maximum, currency and period, where disclosed
- Seniority and contract type
- Skills and requirements extracted from the description
- Dates: first seen, last seen, closing or removal
- Source: board or career page, and a stable URL
Two fields carry most of the value and are the hardest to get right: the dates, which turn a pile of postings into a time series, and the normalized title, which counts "Growth Hacker" and "Digital Marketing Specialist" as one occupation.
Who uses job posting data, and for what?
Four groups build or buy it, each for a different question.
Compensation and HR teams benchmark pay. Where employers publish ranges, advertised salaries give a current read on market rates by title, city and seniority, and show which skills competitors ask for.
Labour-market researchers read postings as a demand indicator. Indeed's Hiring Lab, the job board's research arm, publishes analysis built on Indeed's own postings; Lightcast sells labour market intelligence built on a very large postings archive.
Strategy and sales teams treat hiring as a roadmap signal. A rival opening twenty roles that mention a new cloud platform, market or language is announcing a plan without a press release, and a sales team reads the same signal to find companies hiring in a function it sells to.
Recruiters, aggregators and investors. Aggregators collect postings to list them, recruiting firms watch where vacancies open, and investment analysts count hiring volumes as one of the web-based types of alternative data.
Where does job posting data come from?
Sources rank by reliability and permissibility, the reverse of how most teams start.
1. Official vacancy statistics. National statistics offices measure labour demand directly. In the US, the Bureau of Labor Statistics' JOLTS survey produces monthly estimates of job openings, hires and separations; its 3 November 2026 release put openings at 7.5 million for September. Eurostat's job vacancy statistics report each quarter the share of posts that are vacant (2.1 % in the euro area in the second quarter of 2026). In Türkiye, TurkStat publishes monthly labour force statistics, and the monthly bulletin of ISKUR, the public employment agency, counts the open positions employers register with it, by occupation and sector. They are free and citable, but aggregate: no titles, skills or employers.
2. Employers' career pages through their ATS. Many companies host their career page on an applicant tracking system (ATS), which publishes their postings as public JSON. Greenhouse documents a Job Board API whose GET endpoints need no authentication: https://boards-api.greenhouse.io/v1/boards/{board_token}/jobs?content=true returns every open job with title, location, updated_at, description, departments and offices, and pay_transparency=true adds disclosed pay ranges.
Lever's Postings API works the same way: https://api.lever.co/v0/postings/{site}?mode=json lists published postings with categories (location, team, commitment), workplaceType, descriptionPlain and an optional salaryRange. For named employers these endpoints are the cleanest source there is.
3. Aggregator APIs. Adzuna's API, for example, exposes job search, categories and salary histograms per country behind an app_id and app_key you register for. Aggregator data covers many employers at once but inherits the aggregator's coverage, de-duplication and licence terms. LinkedIn is the exception people ask about most: its User Agreement (section 8.2, "Don'ts") forbids scraping or copying the service, including with crawlers and browser plugins, so its postings are read through its own job search, not collected.
4. Public board pages. Everything else is a web page without a feed. Collecting them is web scraping, gated by the site's terms of use and its robots.txt; where both allow it, the data is current and detailed but unstructured, so the normalization work below falls on you.
How is job posting data collected and normalized?
A working pipeline has eight steps; collection is the smallest.
- Define the schema first. Fix the fields and types before touching a source; add
source,source_url,first_seenandlast_seento every record. - List the sources and their access route. Official series, ATS endpoint, licensed API or public page; the terms that apply and the polling frequency.
- Set the cadence. Postings stay online for weeks, some are renewed, a few vanish within days; a daily pass captures the dates, a weekly pass is enough for volume trends.
- Normalize titles to an occupation taxonomy. Map free-text titles to ESCO, the European Commission's classification of 3,039 occupations and 13,939 skills in 28 languages, free and with an API; to O*NET, the US Department of Labor's database of more than 900 occupations; or to the ILO's ISCO-08 with its ten major groups. A dictionary of known titles handles the first pass, a text classifier the rest.
- Parse salary. Extract minimum, maximum, currency and period, convert hourly and monthly figures to a yearly range, and store "not disclosed" as a value, because its frequency is itself a signal.
- De-duplicate. The same vacancy appears on the career page, two boards and an aggregator, often under different titles; match on employer, normalized title, location and a description fingerprint, and keep one record with all source URLs.
- Derive signals. New postings per week by employer and occupation, median days online (a stand-in for time to fill), salary disclosure share, skill frequency.
- QA every run. Row counts per source, share of records missing salary or location, ten random records checked against the live page.
A normalized record for a fictional employer looks like this:
{
"posting_id": "example-ltd-2026-11-0417",
"title_raw": "Sr. Data Engineer (Hybrid)",
"occupation": "Data engineer",
"employer": "Example Ltd",
"poster_type": "employer",
"location": { "city": "Berlin", "country": "DE", "workplace": "hybrid" },
"salary": { "min": 85000, "max": 100000, "currency": "EUR", "period": "year", "disclosed": true },
"seniority": "senior",
"contract": "full-time",
"skills": ["Spark", "Airflow", "SQL", "Python"],
"first_seen": "2026-11-02",
"last_seen": "2026-11-09",
"closed_at": null,
"sources": ["https://example.com/careers/417", "https://jobs.example.net/view/88213"]
}Job posting data sources compared
| Source | Coverage | Freshness | Constraint |
|---|---|---|---|
| Official statistics (JOLTS, Eurostat, ISKUR) | Whole economy, aggregated | Monthly or quarterly | No titles, skills or employers |
| ATS public endpoints (Greenhouse, Lever) | Named employers, every open role | Live | One board per company; you build the list |
| Aggregator APIs | Many employers per country | Hours to days | Licence terms, the vendor's de-duplication |
| Public board pages | Anything published | As often as you poll | Terms of use, robots.txt, parsing work |
What goes wrong with job posting data?
Postings are advertisements, not vacancies; analysis fails in the gap between the two.
Ghost and evergreen postings. A 2025 primer by the US Congressional Research Service defines "ghost" job postings as postings for positions that do not exist or that the employer does not plan to fill immediately, notes that no official statistics measure them, and warns that the surveys quoting percentages mostly come from firms selling hiring tools. In the data they are postings open for months, reappearing on a fixed cycle, or advertised everywhere at once.
Repostings and agency duplicates. Boards reset the date when an employer renews an ad, so "posted 2 days ago" can be a three-month-old vacancy; agencies post the same client role under their own name, sometimes several at once. Keep your own first_seen, flag agency posters and count a vacancy once.
Missing salary. Many postings disclose no pay, and those that do are not a random sample. Report the disclosure rate next to every salary statistic.
Title inflation and drift. The same work is advertised as engineer, developer, architect or consultant depending on market and year; without a taxonomy, a company that renamed its developers looks like one that stopped hiring them.
Is collecting job posting data legal?
Three layers apply: personal data, terms of use and the law of the country you operate in. None of this is legal advice.
Personal data. A posting is about a job, but it often carries a recruiter's name, e-mail address or phone number. Those are personal data, and both the GDPR (Article 6) and Türkiye's KVKK allow processing only on a legal basis; a job-market dataset has no need for them, so never collect them and strip them from descriptions on ingest. Personal data in scraped datasets covers how to mask what slips through.
Terms of use. Many boards forbid automated access in their terms, LinkedIn explicitly so. The hiQ Labs v. LinkedIn decision (US Court of Appeals for the Ninth Circuit, 2022) concluded that the Computer Fraud and Abuse Act's "without authorization" concept does not apply to public websites; it concerned a US criminal statute, not a site's terms, which remain a separate question.
Local law. Database rights, unfair competition rules and employment-advertising law differ by country. Before a job-data product ships, have a lawyer review the sources and the terms that apply to each.
Where does a proxy fit in job data collection?
A proxy decides which network address a request leaves from; two situations in job data make that matter.
First, boards localize: many show the visitor's country by default, and some change currency, language or salary display with location. To record what a candidate in each market sees, the request has to come from that market; Residential Proxy addresses let you choose the country and city per request, so a series for Germany is not mixed with one for Türkiye.
Second, volume is spread thin: a thousand career pages mean a thousand small, polite jobs rather than one heavy one. Static JSON endpoints such as the ATS feeds above suit Datacenter Proxy addresses, whose traffic is unmetered; sites that vary by visitor country belong on residential addresses. A proxy does not change which sources you may read, how often a site lets you poll, or what its terms say.
If you would rather receive normalized rows than run the pipeline, a managed extraction service does the parsing and de-duplication for you.
Common mistakes with job posting data
- Counting postings as hires or vacancies; calibrate volumes against official series.
- Starting with scraping when ATS endpoints and licensed APIs cover the same employers.
- Trusting the board's posting date instead of your own
first_seenandlast_seen. - Skipping de-duplication, so one vacancy counts three times.
- Keeping raw titles only, so trends disappear into renamings.
- Collecting recruiter contact details because they were on the page.
Decision guide
| Need | Route |
|---|---|
| Labour demand for a country or sector | Official statistics (JOLTS, Eurostat, ISKUR) |
| Hiring plans of fifty named competitors | Their ATS endpoints, polled daily |
| Salary ranges by title and city | Licensed aggregator API plus ATS pay fields |
| One board as candidates see it in each country | Residential proxies per country, within its terms |
| Thousands of small career pages without feeds | Datacenter proxies at a polite rate, or a managed feed |
| Rows today, no pipeline team | Managed extraction with a normalized schema |
Frequently asked questions
Is job scraping legal?
Collecting public job postings is not unlawful in itself in most countries, but a project becomes unlawful when it processes personal data without a legal basis, breaches a site's terms of use, or ignores local rules such as database rights. Official statistics and public ATS endpoints carry none of these risks.
Where can I get job posting data for free?
From official statistics (BLS JOLTS, Eurostat's job vacancy statistics, ISKUR's monthly bulletin) for aggregates, and from employers' public ATS endpoints such as Greenhouse's Job Board API and Lever's Postings API for individual postings; neither needs an API key. Free datasets on data-science platforms are snapshots, fine for practice but not for a time series.
How do you detect ghost jobs?
By behaviour over time rather than by the text: a posting open for months, the same posting re-created on a fixed cycle, one role advertised in every location at once, and employers whose posting count never moves while headcount does. None is proof alone; combine them into a score and report it next to the counts.
How often do job postings change?
Daily, if you count individual postings: new ones appear every day and others close or are renewed. Weekly collection is enough for volumes by occupation or employer; daily polling is needed for accurate first-seen and last-seen dates, which drive the time-to-fill signal.
Can I scrape LinkedIn jobs?
No. LinkedIn's User Agreement forbids scraping or copying the service, including with crawlers and browser extensions, so a dataset built that way breaches the terms from the first request. To see how listings differ by country, use its own search filters; How to Search LinkedIn Jobs in Another Country shows how.
Job posting dataset or live scraping: which is better?
A purchased dataset gives you history and a vendor's de-duplication on day one; live collection gives you the fields, sources and cadence you choose, starting today. Teams needing backfill buy; teams needing a specific schema or country build.
Summary
Job posting data turns advertisements into a structured, dated record of labour demand: who is hiring, for which occupation, where, at what pay and since when. Its best sources are also its most legitimate ones, official vacancy statistics for the aggregate picture and employers' public ATS endpoints for the detail. When the collection has to see each market as a local candidate does, Proxynet's market research proxies page covers the residential and datacenter exits, and the Web Scraping API, available through the sales team, is the managed route for teams that want the rows rather than the pipeline.
