An analyst wants to know whether asking prices in a city's central districts began to fall before the official index says so. The index arrives monthly, with a lag, for the whole city; the listing portals change every day as flats get price cuts, disappear and return under new adverts. That gap between slow official figures and fast public listings is why companies collect real estate data from the web.
This post covers what the data is, which fields and signals matter, who uses it and where it should come from, then the pipeline, the quality traps, the legal limits and where proxies and managed scraping APIs fit. No code and no portal recipes.
What is real estate data scraping?
Real estate data scraping is the automated, repeated collection of publicly visible property listings and their transformation into one clean table of records. A listing page is written for a buyer; a dataset needs the same information as typed fields with units, so the work is less about downloading pages than about turning text into comparable numbers and tracking the same property over time.
Which listing fields and derived signals matter?
The useful fields are the ones that make one home comparable with another and let you follow it over time:
- Asking price and currency, rent or sale.
- Size as net or gross floor area, in square metres or square feet.
- Rooms in the local notation: "2+1" in Türkiye, "3 Zimmer" in Germany, bedrooms in the US.
- Floor, building floors and building age.
- Location at the level the portal shows: city, district, neighbourhood.
- Listing date, seller type (owner, agency, developer) and photo count.
From these a model derives what the page never states: price per square metre; days on market, from the first to the last day you saw the listing; price cuts, from the asking-price history you recorded; delisting, which means sold, rented or withdrawn, and the page cannot tell you which; and re-listing, the same property under a new ID, often to reset its age.
A normalised record (fictional values):
{
"listing_id": "example.com:48213977",
"first_seen": "2026-10-02",
"last_seen": "2026-11-09",
"asking_price": 4250000,
"currency": "TRY",
"area_net_m2": 95,
"area_gross_m2": 118,
"price_per_m2_net": 44737,
"rooms": "2+1",
"floor": 3,
"building_age_years": 12,
"district": "Bornova",
"seller_type": "agency",
"photo_count": 14,
"price_history": [
{ "date": "2026-10-02", "price": 4450000 },
{ "date": "2026-10-24", "price": 4250000 }
],
"days_on_market": 38,
"duplicate_group": "dg_7f3a"
}It holds no owner name, phone or e-mail, deliberately; the legal section explains why.
Who uses real estate data, and for what?
Investors and funds read listing data as an early indicator: inventory, price cuts and days on market move weeks before a transaction index. For them it is one more type of alternative data, with the same demand for clean history and documented sourcing.
Proptech companies feed the records into automated valuation models, banks check mortgage collateral against them, agencies and developers price new instructions against the district, and researchers and municipalities study rents, supply and neighbourhood change.
Where should real estate data come from?
Official statistics first, licensed feeds second, public listing pages last. The order follows reliability and legal clarity, and serious datasets combine all three.
Official statistics. For Türkiye, the central bank publishes the Konut Fiyat Endeksi (housing price index) as a monthly time series, and the Turkish Statistical Institute (TurkStat) publishes a monthly bulletin of housing and workplace sales with first-hand and second-hand counts by province. These are the figures a model is tested against.
For the EU, Eurostat's house price index tracks the transaction prices of dwellings bought by households, new and existing, every quarter, next to a house sales indicator. Built from actual transactions, it shows what homes sold for, not what sellers asked.
For the US, the FHFA House Price Index is a repeat-sales index built from mortgage transactions on single-family homes bought or securitised by Fannie Mae or Freddie Mac, published monthly and quarterly down to ZIP-code level.
Licensed feeds and APIs. In the US, listings live in multiple listing services (MLS) and reach licensed users through the RESO Web API, a standard of the Real Estate Standards Organization; each local MLS grants access under its own data licence. Some portals run data programmes or publish aggregate series. A licensed route delivers structured records with status changes.
Public listing pages. The scraping tier has two gates: the portal's terms of use and its robots.txt. Most large portals forbid automated access in their terms, so this tier is for cases where permission exists or the terms allow it. It is also the only tier with asking prices and listing-level detail across every portal at once.
| Source | What you get | Freshness | Constraint |
|---|---|---|---|
| Official statistics | Price indices, sales counts by region | Monthly or quarterly, with a lag | Aggregates only, no single listings |
| Licensed feeds (MLS, portal programmes) | Structured listings with status changes | Near real time | Licence, fees, use restrictions |
| Public listing pages | Asking prices, attributes, dates, photo counts | As often as you collect | Terms of use, robots.txt, personal data, layout changes |
| Managed scraping API | Normalised records delivered to you | On your schedule | Vendor terms; compliance stays yours |
How does a real estate data pipeline work?
Seven steps, and the first is the one most teams skip:
- Define the schema. Every field, type and unit before collecting anything. The record above is a schema; a folder of HTML files is not.
- Pick the geography. A city, a set of districts or a country; it decides which result pages you need.
- Collect on a schedule. Daily for active sale markets, weekly for slow ones. Every run records what it saw and when; days on market and price history come from those timestamps.
- Normalise. Square feet to square metres, net and gross area in separate fields, currency stored with the price, room notation parsed into a number.
- De-duplicate. The same flat appears through two agencies, on three portals or under a new ID. Match on location, size, floor, age and photo count, and group the matches so one property counts once.
- Derive signals. Price cuts from history, days on market from first and last seen, delisting after a set number of missed runs, re-listing when a grouped property reappears.
- QA. Row counts per run, share of empty fields, distribution of price per square metre. A sudden drop in rows usually means a layout change, not a market crash.
Which data-quality pitfalls distort housing data?
Asking prices are not transaction prices: sellers anchor high, buyers negotiate, and the gap varies by market and season. The other traps are mechanical:
- Re-listing to reset age. Without duplicate grouping, days on market comes out too short.
- Placeholder prices. A price of 1, 0 or a round number far from the market means "call us"; filter on price per square metre.
- Duplicates across portals and agencies, which inflate inventory and bias averages.
- Furnished and unfurnished rents mixed, which shifts a district's rent per square metre.
- Mixed units: net versus gross area, square feet versus metres, prices with or without fees and VAT.
- Delisting read as a sale, when it may be a rental, a withdrawal or a move to another agency.
Is real estate data scraping legal?
Collecting public listing data is not illegal in itself, but three rules apply to every project: personal data, the portal's terms of use and database rights.
Personal data. The owner's name, phone number and e-mail on a listing are personal data under the GDPR and Türkiye's KVKK; publishing them on a portal does not remove the need for a legal basis. Never collect those fields; our guide to personal data in scraped datasets shows how to filter them at collection time.
Terms of use. Most large portals forbid automated access in their terms, often alongside a ban on reproducing listing information elsewhere; how far such terms bind a visitor who never registered differs by country.
Database rights. In the EU, Directive 96/9/EC gives the maker of a database that shows a substantial investment in obtaining, verifying or presenting its contents the right to prevent extraction or re-utilisation of the whole or a substantial part of it for fifteen years, and forbids the repeated and systematic extraction of insubstantial parts that conflicts with its normal exploitation. Daily collection of a large share of a portal's listings is the act the right was written for. Rules outside the EU differ.
Where does a proxy fit in real estate data collection?
Some portals adapt the page to the visitor's location: default region, currency, language, sometimes which listings appear first, so a collector in a cloud region abroad may record a different page from the one a local buyer sees. A Residential Proxy lets you choose the country and city the request exits from, so the recorded page matches the market you measure.
Paging through a result list is one sequence of requests whose sort order, filters and page number a portal may tie to the visitor. A Sticky Proxy session keeps the same exit address for a set number of minutes, so one crawl of one district runs from one consistent identity.
A proxy changes where a request comes from and nothing else: terms, robots.txt and rate limits apply to every address, and a verification page is an answer to respect, not an obstacle to route around.
When does a managed scraping API make more sense?
When the team's job is the model, not the parser. A managed scraping API returns normalised records instead of pages, so schema, unit conversion and layout changes are handled on the vendor's side and your analysts start at step five. It also fits teams that need several countries and portals at once.
The compliance questions stay with you: terms, personal data rules and database rights apply to the records you receive. Proxynet offers a Web Scraping API for managed page extraction through its sales team; ask any vendor about its field schema, refresh schedule and sources.
Common mistakes in real estate data projects
- Collecting before defining the schema. Saved pages cannot answer a question about price per square metre next quarter.
- Keeping owner phone numbers "just in case". They add risk and no signal.
- Collecting at a rate the portal would notice. Daily snapshots at a polite pace are enough.
- Storing only the current price. Without history, price cuts are invisible.
Decision guide
| Need | Route |
|---|---|
| Monthly price trend for a country or region | Official statistics: TCMB, Eurostat, FHFA or the national equivalent |
| Transaction-level prices for valuation | Licensed feeds or a country's public transaction records |
| One city's asking prices this week | Portal pages, after a terms and robots.txt check, at a low rate |
| Listings across many countries and portals | Managed scraping API returning normalised records |
| Own scraper, pages differ by visitor location | Residential exits in the target city, one sticky session per crawl |
| Any dataset with owner names or phones | Drop the fields at collection; they are personal data |
Frequently asked questions
Is it legal to scrape real estate listing portals?
It depends on the portal's terms, the fields you take and the country. Most large portals forbid automated access in their terms; data about a property is not personal data, but the owner's contact details are; and in the EU a portal can invoke database rights against the extraction of a substantial part of its listings. Official statistics cover price trends without any scraping.
Is scraping Zillow legal?
Zillow's Terms of Use, updated October 28, 2025, list "automated queries (including screen and database scraping, spiders, robots, crawlers, bypassing 'captcha' or similar precautions, or any other automated activity with the purpose of obtaining information from the Services)" among the things users agree not to do, and separately forbid reproducing listing information on other websites except where explicitly permitted. For US price trends, the FHFA index and licensed MLS feeds raise none of these questions.
What is the difference between asking price and sale price?
The asking price is what the seller publishes; the sale price is what the buyer pays, usually less after negotiation. Portal data only ever contains asking prices, so it measures seller expectations and how they change. Transaction indices from statistics offices measure what homes actually sold for.
How often should real estate data be refreshed?
Daily for sale and rental listings in active markets, weekly where listings stay up for months. Days on market and price-cut detection depend on the gap between runs: a weekly schedule cannot see a cut that was reversed after three days.
How do you detect duplicate property listings?
Match listings that share the district, a floor area within a small tolerance, floor, building age and room count, then compare photo counts and the first words of the description. Matches are grouped as one property. A new listing ID that matches an older group is a re-listing, and its days on market continue from the first appearance.
Which real estate listing fields matter most?
Asking price with currency, floor area with its unit and net or gross basis, room count, floor, building age, district, and the dates you first and last saw the listing. Those eight fields produce price per square metre, days on market and price history. Photo count and seller type help with de-duplication.
Summary
Real estate data scraping turns public listings into a normalised, time-stamped table from which price per square metre, days on market, price cuts and re-listings are derived. Official statistics come first and remain the reference, licensed feeds second, listing pages last, behind the portal's terms, its robots.txt and the rule that owner contact details are never collected. If your team is sizing a market this way, our market research page shows how the collection side is set up.
