No description
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
yuuhikaze bc96b13a2b docs: rewrite README with full usage, output schema, and architecture
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-10 23:27:44 -05:00
resources Cleaned committed files 2024-05-05 15:22:31 -05:00
src feat: migrate scraper to PVGIS v5.3 basic output 2026-05-04 18:50:18 -05:00
.gitignore Cleaned committed files 2024-05-05 14:15:56 -05:00
Cargo.lock feat: remove interactivity, opt for CLI interaction instead 2026-04-21 12:47:34 -05:00
Cargo.toml feat: remove interactivity, opt for CLI interaction instead 2026-04-21 12:47:34 -05:00
LICENSE Create LICENSE 2024-05-19 21:41:05 -05:00
README.md docs: rewrite README with full usage, output schema, and architecture 2026-05-10 23:27:44 -05:00

title
PVGIS Scraper

Concurrent data acquisition tool for the European Commission's PVGIS hourly solar simulation database. Generates random geographic coordinates guaranteed to fall on land, queries the PVGIS REST API for each, and writes the resulting timeseries CSVs to disk.

Built in Rust for memory safety under concurrency and predictable performance over long-running scraping sessions (300+ sequential API calls with retry logic).

What it produces

Each successful query downloads a CSV with hourly data for a 1 kWp PV system at the sampled coordinate, covering 2005–2020 (140,256 rows per station):

Column Description
P PV power output (W) at 1 kWp, 14% system losses
G(i) Global irradiance on the inclined plane (W/m²)
Gb(i) Direct (beam) irradiance (W/m²)
Gd(i) Diffuse irradiance (W/m²)
Gr(i) Ground-reflected irradiance (W/m²)
H_sun Sun elevation angle (°)
T2m Air temperature at 2m (°C)
WS10m Wind speed at 10m (m/s)

Output filenames encode the coordinates: timeseries_{lat}_{lon}.csv.

How it works

resources/map.png  ←  black/white land/ocean bitmap
       ↓
 sample random pixel, reject if not black (ocean)
       ↓
 convert pixel (x, y) → (latitude, longitude)
       ↓
 query PVGIS API v5.3 with optimal angles
       ↓
 write CSV to --path, retry up to 3× on failure
       ↓
 repeat N times

The bitmap approach guarantees every query hits a valid land coordinate, avoiding wasted API calls on ocean points. The same bitmap can be edited to restrict sampling to specific regions: paint unwanted areas white.

Usage

cargo run --release -- --n 300 --path /mnt/aether/PVGIS/csv
  • --n — number of stations to fetch
  • --path — output directory for CSV files (default: /tmp)

The scraper throttles requests with a 500ms delay between successful downloads and a 2s backoff between retries to respect the PVGIS API rate limits.

Project structure

├── src/
│   ├── main.rs                        # CLI entry point (clap)
│   ├── geographic_bitmap_analysis.rs  # Bitmap coordinate sampling
│   └── curl_module.rs                 # HTTP download with retry logic
├── resources/
│   └── map.png                        # Land/ocean bitmap (black = land)
└── Cargo.toml

Dependencies

  • clap — CLI argument parsing
  • curl — HTTP client (libcurl bindings)
  • image — bitmap pixel analysis
  • rand — coordinate randomization