Public Data Aggregation Automation

n8n-driven automation that gathers public source information, normalizes key fields, and delivers consistent datasets for market intelligence.

Public Data Aggregation Platform

Overview

An automatic aggregation layer for real estate market monitoring that periodically collects narratives from curated public data sources, cleans HTML, extracts key fields using CSS selectors, and structures data through AI modules, then optionally delivers datasets to a backend API.

The workflow includes consolidating source listings, filtering by target market, fetching detailed content, parsing dates, merging streams, and producing unified output with leasing/sales/construction context ready for analytics.

Technology Stack

  • n8n Workflow Automation: Schedule Trigger, HTTP Request, HTML Extract, Merge, IF, Code (Node.js)
  • AI Parsing: gpt-4o-mini via n8n LangChain nodes, Structured Output Parser for consistent JSON
  • CSS Selectors: Configurable selectors per source group for lists, detail pages, dates, titles, and content
  • Backend Integration: Secure HTTP batching into internal aggregation endpoints

Key Features

  • Data harmonization from multiple public sources: fetching links, dates, locations, and content, then merging entries
  • Market filtering: regex-based link checks and location guards to maintain regional relevance
  • AI structuring: rigid JSON schemas for Property Type, Leasing, Sales, Construction, Certificates, and Market
  • Date normalization: conversions and time filters in Code nodes for rolling publication windows

Results & Outcomes

  • Consistent JSON for analysis: each record has title, date, link, category, market, and Leasing/Sales/Construction sections
  • Reduction of manual work: automation of public data processing and metadata extraction ready for CRM/Notion/DB
  • Stable pipeline: harmonization of selectors and parsers minimizes errors when layout changes occur

Visualizations

n8n Workflow Diagram — Public Data Aggregation

Complete workflow showing monitored source branches, merge, enrichment, and output stages

n8n Workflow Diagram - Public Data Aggregation

← Back to Projects

Related Projects