- Rust 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
|
||
| resources | ||
| src | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| LICENSE | ||
| README.md | ||
| title |
|---|
| PVGIS Scraper |
Concurrent data acquisition tool for the European Commission's PVGIS hourly solar simulation database. Generates random geographic coordinates guaranteed to fall on land, queries the PVGIS REST API for each, and writes the resulting timeseries CSVs to disk.
Built in Rust for memory safety under concurrency and predictable performance over long-running scraping sessions (300+ sequential API calls with retry logic).
What it produces
Each successful query downloads a CSV with hourly data for a 1 kWp PV system at the sampled coordinate, covering 2005–2020 (140,256 rows per station):
| Column | Description |
|---|---|
P |
PV power output (W) at 1 kWp, 14% system losses |
G(i) |
Global irradiance on the inclined plane (W/m²) |
Gb(i) |
Direct (beam) irradiance (W/m²) |
Gd(i) |
Diffuse irradiance (W/m²) |
Gr(i) |
Ground-reflected irradiance (W/m²) |
H_sun |
Sun elevation angle (°) |
T2m |
Air temperature at 2m (°C) |
WS10m |
Wind speed at 10m (m/s) |
Output filenames encode the coordinates: timeseries_{lat}_{lon}.csv.
How it works
resources/map.png ← black/white land/ocean bitmap
↓
sample random pixel, reject if not black (ocean)
↓
convert pixel (x, y) → (latitude, longitude)
↓
query PVGIS API v5.3 with optimal angles
↓
write CSV to --path, retry up to 3× on failure
↓
repeat N times
The bitmap approach guarantees every query hits a valid land coordinate, avoiding wasted API calls on ocean points. The same bitmap can be edited to restrict sampling to specific regions: paint unwanted areas white.
Usage
cargo run --release -- --n 300 --path /mnt/aether/PVGIS/csv
--n— number of stations to fetch--path— output directory for CSV files (default:/tmp)
The scraper throttles requests with a 500ms delay between successful downloads and a 2s backoff between retries to respect the PVGIS API rate limits.
Project structure
├── src/
│ ├── main.rs # CLI entry point (clap)
│ ├── geographic_bitmap_analysis.rs # Bitmap coordinate sampling
│ └── curl_module.rs # HTTP download with retry logic
├── resources/
│ └── map.png # Land/ocean bitmap (black = land)
└── Cargo.toml