dtc_super_resolution.rcm_emulator.pre_processing_module.preprocessing
Preprocessing pipeline for the RCM emulator.
Functions
|
Generate ERA5 inputs for the model based on the tiles metadata. |
|
Load ERA5 dataset from the given path. |
|
Load RCM dataset from the given path. |
|
Run the preprocessing pipeline for the RCM emulator. |
|
Run the preprocessing pipeline for the RCM emulator. |
|
Validate the ERA5 dataset to ensure it contains each of the required layers. |
- generate_era5_patches(era5: Dataset, tiles: list) tuple[ndarray, list]
Generate ERA5 inputs for the model based on the tiles metadata.
Skip tiles that contain only sea values.
- Parameters:
era5 (xr.Dataset) – The ERA5 dataset.
tiles (list) – A list of dictionaries containing metadata for each tile.
- Returns:
A tuple containing the ERA5 patches (numpy arrays) and their corresponding metadata (list of dicts).
- Return type:
tuple[np.ndarray, list]
- load_era5_dataset(era5_path: str) Dataset
Load ERA5 dataset from the given path.
- Parameters:
era5_path (str) – Path to the ERA5 dataset.
- Returns:
Loaded ERA5 dataset.
- Return type:
xr.Dataset
- load_rcm_dataset(rcm_path: str) Dataset
Load RCM dataset from the given path.
- Parameters:
rcm_path (str) – Path to the RCM dataset.
- Returns:
Loaded RCM dataset.
- Return type:
xr.Dataset
- run_preprocessing(gcm_zarr_path: str, rcm_zarr_path: str) tuple[ndarray, list, list, Dataset, Dataset]
Run the preprocessing pipeline for the RCM emulator.
- Parameters:
gcm_zarr_path (str) – Path to the GCM data.
rcm_zarr_path (str) – Path to the RCM data.
- Returns:
- A tuple containing:
- input_patches: np.ndarray
A numpy array containing the preprocessed ERA5 patches ready for model input.
- metadata: list
A list of dictionaries containing metadata for each ERA5 patch.
- tiles: list
A list of dictionaries containing metadata for each tile.
- rcm_ds: xarray.Dataset
The RCM dataset.
- era5_xy: xarray.Dataset
The ERA5 dataset reprojected to the RCM grid.
- Return type:
tuple[np.ndarray, list, list, xr.Dataset, xr.Dataset]
- run_preprocessing_loc(gcm_zarr_path: str, rcm_zarr_path: str, location: tuple[float, float]) tuple[ndarray, list, dict, Dataset, Dataset]
Run the preprocessing pipeline for the RCM emulator.
- Parameters:
gcm_zarr_path (str) – Path to the GCM data.
rcm_zarr_path (str) – Path to the RCM data.
location (tuple[float, float]) – The latitude and longitude of the location for which to run the preprocessing.
- Returns:
- A tuple containing:
- input_patches: np.ndarray
A numpy array containing the preprocessed ERA5 patches ready for model input.
- patch_metadata: list
A list (one entry per month) of single-item lists of dictionaries containing per-patch metadata (e.g. “time_idx”), as expected by the prediction pipeline.
- tile_meta: dict
Metadata about the single tile location (slices, center coords, etc.), as expected by the location-based postprocessing step.
- rcm_ds: xarray.Dataset
The RCM dataset.
- era5_xy: xarray.Dataset
The ERA5 dataset reprojected to the RCM grid.
- Return type:
tuple[np.ndarray, list, dict, xr.Dataset, xr.Dataset]
- validate_era5_dataset(era5: Dataset) None
Validate the ERA5 dataset to ensure it contains each of the required layers.
- Parameters:
era5 (xr.Dataset) – The ERA5 dataset to validate.
- Raises:
ValueError – If any of the required variables are missing from the dataset.