Graph Machine Learning
AnemoI
English
.gitattributes CHANGED
@@ -36,4 +36,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
36
  aifs_diagram.png filter=lfs diff=lfs merge=lfs -text
37
  decoder_graph.jpeg filter=lfs diff=lfs merge=lfs -text
38
  encoder_graph.jpeg filter=lfs diff=lfs merge=lfs -text
39
- assets/aifs-crps-schematic.png filter=lfs diff=lfs merge=lfs -text
 
36
  aifs_diagram.png filter=lfs diff=lfs merge=lfs -text
37
  decoder_graph.jpeg filter=lfs diff=lfs merge=lfs -text
38
  encoder_graph.jpeg filter=lfs diff=lfs merge=lfs -text
 
README.md CHANGED
@@ -12,34 +12,33 @@ library_name: anemoi
12
 
13
  Here, we introduce the Artificial Intelligence Forecasting System (AIFS) ENS,
14
  an ensemble-based probablistic data driven forecast model developed by the
15
- [European Centre for Medium-Range Weather Forecasts (ECMWF)](https://www.ecmwf.int/).
 
16
 
17
  AIFS produces highly skilled forecasts for upper-air variables, surface weather parameters and
18
  tropical cyclone tracks. AIFS ENS is run four times daily alongside ECMWF’s physics-based NWP model and forecasts
19
- are available to the public under ECMWF’s [open data policy](https://www.ecmwf.int/en/forecasts/datasets/open-data).
20
  Note that due to the non-determinism of GPUs, users will be unable to exactly reproduce an official AIFS ENS forecast
21
  when running AIFS ENS themselves.
22
 
23
- For more details please refer to our [documentation](https://confluence.ecmwf.int/display/FCST/Implementation+of+AIFS+ENS+v1).
24
 
25
  ## Data Details
26
 
27
  ### Data parameters
28
 
29
 
30
- **Note**
31
  Regarding precipitation units, it's worth noting that AIFS model was trained on \\(m^{3}/m^{2}\\) and will therefore produce precip in that units.
32
  If one wants to retrieve precipitation from Open data, the units will be \\(mm\\).
33
 
34
 
35
  ## Model Details
36
 
37
- ### Description
38
 
39
  <!-- Provide a longer summary of what this model is. -->
40
 
41
- For full details of the AIFS ENS, please see [Lang et al. 2024](https://arxiv.org/abs/2412.15832).
42
-
43
  AIFS is based on a graph neural network (GNN) encoder and decoder, and a sliding window transformer processor,
44
  and is trained on ECMWF’s ERA5 re-analysis and ECMWF’s operational numerical weather prediction (NWP) analyses.
45
 
@@ -52,67 +51,55 @@ It has a flexible and modular design and supports several levels of parallelism
52
  high resolution input data. AIFS forecast skill is assessed by comparing its forecasts to NWP analyses
53
  and direct observational data.
54
 
55
- AIFS ENS was trained to minimise the Continuous Ranked Probability Score (CRPS) calculated over
56
- a small ensemble.
57
- <div style="display: flex; justify-content: center;">
58
- <img src="assets/aifs-crps-schematic.png" alt="Training schematic" style="width: 75%;"/>
59
- </div>
60
-
61
  - **Developed by:** ECMWF
62
  - **Model type:** Encoder-processor-decoder model
63
  - **License:** These model weights are published under a Creative Commons Attribution 4.0 International (CC BY 4.0).
64
- To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/.
65
- The notebooks and other script files are published under an Apache 2.0 licence, to view a copy of this license, visit https://www.apache.org/licenses/LICENSE-2.0.txt.
66
 
67
- ### Resolution
68
 
69
  | | Component | Horizontal Resolution [kms] | Vertical Resolution [levels] |
70
  |---|:---:|:---:|:---:|
71
  | Atmosphere | AIFS ENS v1.0 | ~ 31 | 13 |
72
 
73
 
74
- ### Sources
75
 
76
  <!-- Provide the basic links for the model. -->
77
 
78
  - **Repository:** [Anemoi](https://anemoi.readthedocs.io/en/latest/index.html) is an open-source framework for
79
- creating machine learning (ML) weather forecasting systems, co-developed by ECMWF and a range of national meteorological
80
- services across Europe.
81
  - **Paper:** https://arxiv.org/abs/2412.15832
82
 
83
- ## Get started
84
 
85
- 🚨 **Note** `aifs-ens-1.0` requires `38GB` of GPU memory by default, it is possible to reduce this by setting `ANEMOI_INFERENCE_NUM_CHUNKS` to a multiple of 2.
86
- For example setting it to `16` reduces the memory requirements to `24GB` at the cost of throughput. Therefore issues may arise with lower end GPUs.
87
-
88
-
89
- To generate a new forecast using AIFS ENS, you can use [anemoi-inference](https://github.com/ecmwf/anemoi-inference). In the [example notebook](run_AIFS_ENS_v1.ipynb), a
90
- step-by-step workflow is specified to run AIFS ENS using the HuggingFace model:
91
 
92
  1. **Install Required Packages and Imports**
93
- - `torch==2.5.0 anemoi-inference[huggingface]==0.6.0 anemoi-models==0.6.0 anemoi-graphs==0.6.0 anemoi-datasets==0.5.23`
94
- - `earthkit-regrid==0.4.0 'ecmwf-opendata>=0.3.19'`
95
- - `flash_attn`
96
  2. **Retrieve Initial Conditions from ECMWF Open Data**
97
- - Select a date
98
- - Get the data from the [ECMWF Open Data API](https://www.ecmwf.int/en/forecasts/datasets/open-data)
99
- - Get input fields
100
- - Add the single levels fields and pressure levels fields
101
- - Convert geopotential height into geopotential
102
- - Create the initial state
103
  3. **Load the Model and Run the Forecast**
104
- - Download the Model's Checkpoint from Hugging Face
105
- - Create a runner
106
- - Run the forecast using anemoi-inference
107
  4. **Inspect the generated forecast**
108
- - Plot a field
 
 
 
 
 
 
 
109
 
110
- 🚨 **Note** we train AIFS using [`flash_attention`](https://github.com/Dao-AILab/flash-attention).
111
- The use of the 'Flash Attention' package also imposes certain requirements in terms of software and hardware, which can be found under
112
- [Installation and Features](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#installation-and-features) on the GitHub repo.
113
 
114
- 🚨 **Note** the `aifs_ens_v1.0.ckpt` checkpoint just contains the model’s weights. It does not contain any information about the optimizer states,
115
- lr-scheduler states, etc.
116
 
117
  ## Citation
118
 
 
12
 
13
  Here, we introduce the Artificial Intelligence Forecasting System (AIFS) ENS,
14
  an ensemble-based probablistic data driven forecast model developed by the
15
+ European Centre for Medium-Range Weather Forecasts (ECMWF).
16
+
17
 
18
  AIFS produces highly skilled forecasts for upper-air variables, surface weather parameters and
19
  tropical cyclone tracks. AIFS ENS is run four times daily alongside ECMWF’s physics-based NWP model and forecasts
20
+ are available to the public under ECMWF’s open data policy (https://www.ecmwf.int/en/forecasts/datasets/open-data).
21
  Note that due to the non-determinism of GPUs, users will be unable to exactly reproduce an official AIFS ENS forecast
22
  when running AIFS ENS themselves.
23
 
24
+ For more details please refer to https://confluence.ecmwf.int/display/FCST/Implementation+of+AIFS+ENS+v1
25
 
26
  ## Data Details
27
 
28
  ### Data parameters
29
 
30
 
31
+ **Note**
32
  Regarding precipitation units, it's worth noting that AIFS model was trained on \\(m^{3}/m^{2}\\) and will therefore produce precip in that units.
33
  If one wants to retrieve precipitation from Open data, the units will be \\(mm\\).
34
 
35
 
36
  ## Model Details
37
 
38
+ ### Model Description
39
 
40
  <!-- Provide a longer summary of what this model is. -->
41
 
 
 
42
  AIFS is based on a graph neural network (GNN) encoder and decoder, and a sliding window transformer processor,
43
  and is trained on ECMWF’s ERA5 re-analysis and ECMWF’s operational numerical weather prediction (NWP) analyses.
44
 
 
51
  high resolution input data. AIFS forecast skill is assessed by comparing its forecasts to NWP analyses
52
  and direct observational data.
53
 
 
 
 
 
 
 
54
  - **Developed by:** ECMWF
55
  - **Model type:** Encoder-processor-decoder model
56
  - **License:** These model weights are published under a Creative Commons Attribution 4.0 International (CC BY 4.0).
57
+ To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/
 
58
 
59
+ ### Model resolution
60
 
61
  | | Component | Horizontal Resolution [kms] | Vertical Resolution [levels] |
62
  |---|:---:|:---:|:---:|
63
  | Atmosphere | AIFS ENS v1.0 | ~ 31 | 13 |
64
 
65
 
66
+ ### Model Sources
67
 
68
  <!-- Provide the basic links for the model. -->
69
 
70
  - **Repository:** [Anemoi](https://anemoi.readthedocs.io/en/latest/index.html) is an open-source framework for
71
+ creating machine learning (ML) weather forecasting systems, which ECMWF and a range of national meteorological
72
+ services across Europe have co-developed.
73
  - **Paper:** https://arxiv.org/abs/2412.15832
74
 
75
+ ## How to Get Started with the Model
76
 
77
+ To generate a new forecast using AIFS, you can use [anemoi-inference](https://github.com/ecmwf/anemoi-inference). In the [following notebook](run_AIFS_ENS_v1.ipynb), a
78
+ step-by-step workflow is specified to run the AIFS using the HuggingFace model:
 
 
 
 
79
 
80
  1. **Install Required Packages and Imports**
 
 
 
81
  2. **Retrieve Initial Conditions from ECMWF Open Data**
82
+ - Select a date
83
+ - Get the data from the [ECMWF Open Data API](https://www.ecmwf.int/en/forecasts/datasets/open-data)
84
+ - Get input fields
85
+ - Add the single levels fields and pressure levels fields
86
+ - Convert geopotential height into geopotential
87
+ - Create the initial state
88
  3. **Load the Model and Run the Forecast**
89
+ - Download the Model's Checkpoint from Hugging Face
90
+ - Create a runner
91
+ - Run the forecast using anemoi-inference
92
  4. **Inspect the generated forecast**
93
+ - Plot a field
94
+
95
+
96
+ 🚨 **Note** we train AIFS using `flash_attention` (https://github.com/Dao-AILab/flash-attention).
97
+ The use of 'Flash Attention' package also imposes certain requirements in terms of software and hardware. Those can be found under #Installation and Features in https://github.com/Dao-AILab/flash-attention
98
+
99
+ 🚨 **Note** the `aifs_ens_v1.0.ckpt` checkpoint just contains the model’s weights.
100
+ That file does not contain any information about the optimizer states, lr-scheduler states, etc.
101
 
 
 
 
102
 
 
 
103
 
104
  ## Citation
105
 
aifs-ens-crps-1.0.ckpt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:5f176e2e018f2c8f54cbb8927f7ee64a948bfdb6960a4867645304e1f6e3134a
3
- size 921591109
 
 
 
 
assets/aifs-crps-schematic.png → aifs_diagram.png RENAMED
File without changes
assets/decoder_graph.jpeg → decoder_graph.jpeg RENAMED
File without changes
assets/encoder_graph.jpeg → encoder_graph.jpeg RENAMED
File without changes
run_AIFS_ENS_v1.ipynb DELETED
The diff for this file is too large to render. See raw diff