Update files
#1
by
mchantry
- opened
.gitattributes
CHANGED
|
@@ -36,4 +36,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 36 |
aifs_diagram.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
decoder_graph.jpeg filter=lfs diff=lfs merge=lfs -text
|
| 38 |
encoder_graph.jpeg filter=lfs diff=lfs merge=lfs -text
|
| 39 |
-
assets/aifs-crps-schematic.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
| 36 |
aifs_diagram.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
decoder_graph.jpeg filter=lfs diff=lfs merge=lfs -text
|
| 38 |
encoder_graph.jpeg filter=lfs diff=lfs merge=lfs -text
|
|
|
README.md
CHANGED
|
@@ -12,34 +12,33 @@ library_name: anemoi
|
|
| 12 |
|
| 13 |
Here, we introduce the Artificial Intelligence Forecasting System (AIFS) ENS,
|
| 14 |
an ensemble-based probablistic data driven forecast model developed by the
|
| 15 |
-
|
|
|
|
| 16 |
|
| 17 |
AIFS produces highly skilled forecasts for upper-air variables, surface weather parameters and
|
| 18 |
tropical cyclone tracks. AIFS ENS is run four times daily alongside ECMWF’s physics-based NWP model and forecasts
|
| 19 |
-
are available to the public under ECMWF’s
|
| 20 |
Note that due to the non-determinism of GPUs, users will be unable to exactly reproduce an official AIFS ENS forecast
|
| 21 |
when running AIFS ENS themselves.
|
| 22 |
|
| 23 |
-
For more details please refer to
|
| 24 |
|
| 25 |
## Data Details
|
| 26 |
|
| 27 |
### Data parameters
|
| 28 |
|
| 29 |
|
| 30 |
-
**Note**
|
| 31 |
Regarding precipitation units, it's worth noting that AIFS model was trained on \\(m^{3}/m^{2}\\) and will therefore produce precip in that units.
|
| 32 |
If one wants to retrieve precipitation from Open data, the units will be \\(mm\\).
|
| 33 |
|
| 34 |
|
| 35 |
## Model Details
|
| 36 |
|
| 37 |
-
### Description
|
| 38 |
|
| 39 |
<!-- Provide a longer summary of what this model is. -->
|
| 40 |
|
| 41 |
-
For full details of the AIFS ENS, please see [Lang et al. 2024](https://arxiv.org/abs/2412.15832).
|
| 42 |
-
|
| 43 |
AIFS is based on a graph neural network (GNN) encoder and decoder, and a sliding window transformer processor,
|
| 44 |
and is trained on ECMWF’s ERA5 re-analysis and ECMWF’s operational numerical weather prediction (NWP) analyses.
|
| 45 |
|
|
@@ -52,67 +51,55 @@ It has a flexible and modular design and supports several levels of parallelism
|
|
| 52 |
high resolution input data. AIFS forecast skill is assessed by comparing its forecasts to NWP analyses
|
| 53 |
and direct observational data.
|
| 54 |
|
| 55 |
-
AIFS ENS was trained to minimise the Continuous Ranked Probability Score (CRPS) calculated over
|
| 56 |
-
a small ensemble.
|
| 57 |
-
<div style="display: flex; justify-content: center;">
|
| 58 |
-
<img src="assets/aifs-crps-schematic.png" alt="Training schematic" style="width: 75%;"/>
|
| 59 |
-
</div>
|
| 60 |
-
|
| 61 |
- **Developed by:** ECMWF
|
| 62 |
- **Model type:** Encoder-processor-decoder model
|
| 63 |
- **License:** These model weights are published under a Creative Commons Attribution 4.0 International (CC BY 4.0).
|
| 64 |
-
To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0
|
| 65 |
-
The notebooks and other script files are published under an Apache 2.0 licence, to view a copy of this license, visit https://www.apache.org/licenses/LICENSE-2.0.txt.
|
| 66 |
|
| 67 |
-
###
|
| 68 |
|
| 69 |
| | Component | Horizontal Resolution [kms] | Vertical Resolution [levels] |
|
| 70 |
|---|:---:|:---:|:---:|
|
| 71 |
| Atmosphere | AIFS ENS v1.0 | ~ 31 | 13 |
|
| 72 |
|
| 73 |
|
| 74 |
-
### Sources
|
| 75 |
|
| 76 |
<!-- Provide the basic links for the model. -->
|
| 77 |
|
| 78 |
- **Repository:** [Anemoi](https://anemoi.readthedocs.io/en/latest/index.html) is an open-source framework for
|
| 79 |
-
creating machine learning (ML) weather forecasting systems,
|
| 80 |
-
services across Europe.
|
| 81 |
- **Paper:** https://arxiv.org/abs/2412.15832
|
| 82 |
|
| 83 |
-
## Get
|
| 84 |
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
To generate a new forecast using AIFS ENS, you can use [anemoi-inference](https://github.com/ecmwf/anemoi-inference). In the [example notebook](run_AIFS_ENS_v1.ipynb), a
|
| 90 |
-
step-by-step workflow is specified to run AIFS ENS using the HuggingFace model:
|
| 91 |
|
| 92 |
1. **Install Required Packages and Imports**
|
| 93 |
-
- `torch==2.5.0 anemoi-inference[huggingface]==0.6.0 anemoi-models==0.6.0 anemoi-graphs==0.6.0 anemoi-datasets==0.5.23`
|
| 94 |
-
- `earthkit-regrid==0.4.0 'ecmwf-opendata>=0.3.19'`
|
| 95 |
-
- `flash_attn`
|
| 96 |
2. **Retrieve Initial Conditions from ECMWF Open Data**
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
3. **Load the Model and Run the Forecast**
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
4. **Inspect the generated forecast**
|
| 108 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
|
| 110 |
-
🚨 **Note** we train AIFS using [`flash_attention`](https://github.com/Dao-AILab/flash-attention).
|
| 111 |
-
The use of the 'Flash Attention' package also imposes certain requirements in terms of software and hardware, which can be found under
|
| 112 |
-
[Installation and Features](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#installation-and-features) on the GitHub repo.
|
| 113 |
|
| 114 |
-
🚨 **Note** the `aifs_ens_v1.0.ckpt` checkpoint just contains the model’s weights. It does not contain any information about the optimizer states,
|
| 115 |
-
lr-scheduler states, etc.
|
| 116 |
|
| 117 |
## Citation
|
| 118 |
|
|
|
|
| 12 |
|
| 13 |
Here, we introduce the Artificial Intelligence Forecasting System (AIFS) ENS,
|
| 14 |
an ensemble-based probablistic data driven forecast model developed by the
|
| 15 |
+
European Centre for Medium-Range Weather Forecasts (ECMWF).
|
| 16 |
+
|
| 17 |
|
| 18 |
AIFS produces highly skilled forecasts for upper-air variables, surface weather parameters and
|
| 19 |
tropical cyclone tracks. AIFS ENS is run four times daily alongside ECMWF’s physics-based NWP model and forecasts
|
| 20 |
+
are available to the public under ECMWF’s open data policy (https://www.ecmwf.int/en/forecasts/datasets/open-data).
|
| 21 |
Note that due to the non-determinism of GPUs, users will be unable to exactly reproduce an official AIFS ENS forecast
|
| 22 |
when running AIFS ENS themselves.
|
| 23 |
|
| 24 |
+
For more details please refer to https://confluence.ecmwf.int/display/FCST/Implementation+of+AIFS+ENS+v1
|
| 25 |
|
| 26 |
## Data Details
|
| 27 |
|
| 28 |
### Data parameters
|
| 29 |
|
| 30 |
|
| 31 |
+
**Note**
|
| 32 |
Regarding precipitation units, it's worth noting that AIFS model was trained on \\(m^{3}/m^{2}\\) and will therefore produce precip in that units.
|
| 33 |
If one wants to retrieve precipitation from Open data, the units will be \\(mm\\).
|
| 34 |
|
| 35 |
|
| 36 |
## Model Details
|
| 37 |
|
| 38 |
+
### Model Description
|
| 39 |
|
| 40 |
<!-- Provide a longer summary of what this model is. -->
|
| 41 |
|
|
|
|
|
|
|
| 42 |
AIFS is based on a graph neural network (GNN) encoder and decoder, and a sliding window transformer processor,
|
| 43 |
and is trained on ECMWF’s ERA5 re-analysis and ECMWF’s operational numerical weather prediction (NWP) analyses.
|
| 44 |
|
|
|
|
| 51 |
high resolution input data. AIFS forecast skill is assessed by comparing its forecasts to NWP analyses
|
| 52 |
and direct observational data.
|
| 53 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
- **Developed by:** ECMWF
|
| 55 |
- **Model type:** Encoder-processor-decoder model
|
| 56 |
- **License:** These model weights are published under a Creative Commons Attribution 4.0 International (CC BY 4.0).
|
| 57 |
+
To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/
|
|
|
|
| 58 |
|
| 59 |
+
### Model resolution
|
| 60 |
|
| 61 |
| | Component | Horizontal Resolution [kms] | Vertical Resolution [levels] |
|
| 62 |
|---|:---:|:---:|:---:|
|
| 63 |
| Atmosphere | AIFS ENS v1.0 | ~ 31 | 13 |
|
| 64 |
|
| 65 |
|
| 66 |
+
### Model Sources
|
| 67 |
|
| 68 |
<!-- Provide the basic links for the model. -->
|
| 69 |
|
| 70 |
- **Repository:** [Anemoi](https://anemoi.readthedocs.io/en/latest/index.html) is an open-source framework for
|
| 71 |
+
creating machine learning (ML) weather forecasting systems, which ECMWF and a range of national meteorological
|
| 72 |
+
services across Europe have co-developed.
|
| 73 |
- **Paper:** https://arxiv.org/abs/2412.15832
|
| 74 |
|
| 75 |
+
## How to Get Started with the Model
|
| 76 |
|
| 77 |
+
To generate a new forecast using AIFS, you can use [anemoi-inference](https://github.com/ecmwf/anemoi-inference). In the [following notebook](run_AIFS_ENS_v1.ipynb), a
|
| 78 |
+
step-by-step workflow is specified to run the AIFS using the HuggingFace model:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
|
| 80 |
1. **Install Required Packages and Imports**
|
|
|
|
|
|
|
|
|
|
| 81 |
2. **Retrieve Initial Conditions from ECMWF Open Data**
|
| 82 |
+
- Select a date
|
| 83 |
+
- Get the data from the [ECMWF Open Data API](https://www.ecmwf.int/en/forecasts/datasets/open-data)
|
| 84 |
+
- Get input fields
|
| 85 |
+
- Add the single levels fields and pressure levels fields
|
| 86 |
+
- Convert geopotential height into geopotential
|
| 87 |
+
- Create the initial state
|
| 88 |
3. **Load the Model and Run the Forecast**
|
| 89 |
+
- Download the Model's Checkpoint from Hugging Face
|
| 90 |
+
- Create a runner
|
| 91 |
+
- Run the forecast using anemoi-inference
|
| 92 |
4. **Inspect the generated forecast**
|
| 93 |
+
- Plot a field
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
🚨 **Note** we train AIFS using `flash_attention` (https://github.com/Dao-AILab/flash-attention).
|
| 97 |
+
The use of 'Flash Attention' package also imposes certain requirements in terms of software and hardware. Those can be found under #Installation and Features in https://github.com/Dao-AILab/flash-attention
|
| 98 |
+
|
| 99 |
+
🚨 **Note** the `aifs_ens_v1.0.ckpt` checkpoint just contains the model’s weights.
|
| 100 |
+
That file does not contain any information about the optimizer states, lr-scheduler states, etc.
|
| 101 |
|
|
|
|
|
|
|
|
|
|
| 102 |
|
|
|
|
|
|
|
| 103 |
|
| 104 |
## Citation
|
| 105 |
|
aifs-ens-crps-1.0.ckpt
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:5f176e2e018f2c8f54cbb8927f7ee64a948bfdb6960a4867645304e1f6e3134a
|
| 3 |
-
size 921591109
|
|
|
|
|
|
|
|
|
|
|
|
assets/aifs-crps-schematic.png → aifs_diagram.png
RENAMED
|
File without changes
|
assets/decoder_graph.jpeg → decoder_graph.jpeg
RENAMED
|
File without changes
|
assets/encoder_graph.jpeg → encoder_graph.jpeg
RENAMED
|
File without changes
|
run_AIFS_ENS_v1.ipynb
DELETED
|
The diff for this file is too large to render.
See raw diff
|
|
|