Upload folder using huggingface_hub

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_2_lora/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ./weights/Bunny-v1_1-Llama-3-8B-V
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_2_lora/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "gate_proj",
+    "down_proj",
+    "q_proj",
+    "v_proj",
+    "k_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_2_lora/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6471582fd50057e84dfb08f495660314c24a16a94f8ac9426da1edcd17b1603
+size 671150064

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_2_lora/config.json

@@ -0,0 +1,45 @@
+{
+  "_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "architectures": [
+    "BunnyLlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_bunny_llama.BunnyLlamaConfig",
+    "AutoModelForCausalLM": "modeling_bunny_llama.BunnyLlamaForCausalLM"
+  },
+  "bos_token_id": 128000,
+  "continuous_training": false,
+  "eos_token_id": 128001,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect_ratio": "pad",
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 8192,
+  "mm_hidden_size": 3456,
+  "mm_projector_lr": null,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_vision_tower": "./weights/siglip-so400m-patch14-384",
+  "model_type": "bunny-llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 2048,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "float16",
+  "transformers_version": "4.41.2",
+  "tune_mm_mlp_adapter": false,
+  "unfreeze_vision_tower": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_s2": true,
+  "vocab_size": 128256
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_2_lora/non_lora_trainables.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:582698d24359f6be8e4df167566915dd311ba98e0ec3ef337f6141b3712b3e53
+size 918507402

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_4_lora/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ./weights/Bunny-v1_1-Llama-3-8B-V
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_4_lora/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "up_proj",
+    "q_proj",
+    "k_proj",
+    "o_proj",
+    "v_proj",
+    "down_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_4_lora/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:509f4fe725368c830f31a07e9dfa637e7409475215bca031f12ab92ad28fc1d5
+size 671150064

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_4_lora/config.json

@@ -0,0 +1,45 @@
+{
+  "_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "architectures": [
+    "BunnyLlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_bunny_llama.BunnyLlamaConfig",
+    "AutoModelForCausalLM": "modeling_bunny_llama.BunnyLlamaForCausalLM"
+  },
+  "bos_token_id": 128000,
+  "continuous_training": false,
+  "eos_token_id": 128001,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect_ratio": "pad",
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 8192,
+  "mm_hidden_size": 3456,
+  "mm_projector_lr": null,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_vision_tower": "./weights/siglip-so400m-patch14-384",
+  "model_type": "bunny-llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 2048,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "float16",
+  "transformers_version": "4.41.2",
+  "tune_mm_mlp_adapter": false,
+  "unfreeze_vision_tower": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_s2": true,
+  "vocab_size": 128256
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_4_lora/non_lora_trainables.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1cfa623b92512d18b2427e22625319dbb311b37891dcfc42a982b687758f3ac6
+size 918507402

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_1_epochs_1_GA_4_lora/trainer_state.json

@@ -0,0 +1,2226 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9984,
+  "eval_steps": 500,
+  "global_step": 312,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0032,
+      "grad_norm": 0.8230906866528206,
+      "learning_rate": 2e-05,
+      "loss": 1.3718,
+      "step": 1
+    },
+    {
+      "epoch": 0.0064,
+      "grad_norm": 0.8011482349224612,
+      "learning_rate": 4e-05,
+      "loss": 1.3884,
+      "step": 2
+    },
+    {
+      "epoch": 0.0096,
+      "grad_norm": 0.6777465654408708,
+      "learning_rate": 6e-05,
+      "loss": 1.2942,
+      "step": 3
+    },
+    {
+      "epoch": 0.0128,
+      "grad_norm": 0.6985842469244755,
+      "learning_rate": 8e-05,
+      "loss": 1.3011,
+      "step": 4
+    },
+    {
+      "epoch": 0.016,
+      "grad_norm": 0.6358810285065617,
+      "learning_rate": 0.0001,
+      "loss": 1.2236,
+      "step": 5
+    },
+    {
+      "epoch": 0.0192,
+      "grad_norm": 0.8043615850550121,
+      "learning_rate": 0.00012,
+      "loss": 1.1222,
+      "step": 6
+    },
+    {
+      "epoch": 0.0224,
+      "grad_norm": 0.8766371190581345,
+      "learning_rate": 0.00014,
+      "loss": 1.0481,
+      "step": 7
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 0.5896469063055397,
+      "learning_rate": 0.00016,
+      "loss": 1.0115,
+      "step": 8
+    },
+    {
+      "epoch": 0.0288,
+      "grad_norm": 0.5140042859696677,
+      "learning_rate": 0.00018,
+      "loss": 1.0032,
+      "step": 9
+    },
+    {
+      "epoch": 0.032,
+      "grad_norm": 0.47047377131530216,
+      "learning_rate": 0.0002,
+      "loss": 0.9718,
+      "step": 10
+    },
+    {
+      "epoch": 0.0352,
+      "grad_norm": 0.4689357680369753,
+      "learning_rate": 0.00019999458931878073,
+      "loss": 1.0164,
+      "step": 11
+    },
+    {
+      "epoch": 0.0384,
+      "grad_norm": 0.49055471890156765,
+      "learning_rate": 0.0001999783578606323,
+      "loss": 1.0478,
+      "step": 12
+    },
+    {
+      "epoch": 0.0416,
+      "grad_norm": 0.48360570699467204,
+      "learning_rate": 0.00019995130738201966,
+      "loss": 1.0179,
+      "step": 13
+    },
+    {
+      "epoch": 0.0448,
+      "grad_norm": 0.4385882347361944,
+      "learning_rate": 0.0001999134408101731,
+      "loss": 0.8839,
+      "step": 14
+    },
+    {
+      "epoch": 0.048,
+      "grad_norm": 0.5842432990089418,
+      "learning_rate": 0.00019986476224277165,
+      "loss": 1.0378,
+      "step": 15
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 0.4798987625762095,
+      "learning_rate": 0.00019980527694749952,
+      "loss": 0.9014,
+      "step": 16
+    },
+    {
+      "epoch": 0.0544,
+      "grad_norm": 0.43385099220728984,
+      "learning_rate": 0.00019973499136147606,
+      "loss": 0.9908,
+      "step": 17
+    },
+    {
+      "epoch": 0.0576,
+      "grad_norm": 0.4363177117395925,
+      "learning_rate": 0.0001996539130905593,
+      "loss": 0.9302,
+      "step": 18
+    },
+    {
+      "epoch": 0.0608,
+      "grad_norm": 0.4184869797333112,
+      "learning_rate": 0.0001995620509085228,
+      "loss": 0.9156,
+      "step": 19
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 0.4093043649081553,
+      "learning_rate": 0.00019945941475610623,
+      "loss": 0.9582,
+      "step": 20
+    },
+    {
+      "epoch": 0.0672,
+      "grad_norm": 0.40265123100906836,
+      "learning_rate": 0.0001993460157399396,
+      "loss": 0.8491,
+      "step": 21
+    },
+    {
+      "epoch": 0.0704,
+      "grad_norm": 0.40866258779980846,
+      "learning_rate": 0.0001992218661313415,
+      "loss": 0.9515,
+      "step": 22
+    },
+    {
+      "epoch": 0.0736,
+      "grad_norm": 0.47714968174592676,
+      "learning_rate": 0.00019908697936499103,
+      "loss": 0.983,
+      "step": 23
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 0.3725347901708648,
+      "learning_rate": 0.00019894137003747403,
+      "loss": 0.9136,
+      "step": 24
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 0.48768461879424546,
+      "learning_rate": 0.00019878505390570362,
+      "loss": 0.9486,
+      "step": 25
+    },
+    {
+      "epoch": 0.0832,
+      "grad_norm": 0.46702127014841616,
+      "learning_rate": 0.00019861804788521493,
+      "loss": 0.9692,
+      "step": 26
+    },
+    {
+      "epoch": 0.0864,
+      "grad_norm": 0.39034215128065525,
+      "learning_rate": 0.00019844037004833473,
+      "loss": 0.8706,
+      "step": 27
+    },
+    {
+      "epoch": 0.0896,
+      "grad_norm": 0.4730253707882492,
+      "learning_rate": 0.00019825203962222572,
+      "loss": 0.9078,
+      "step": 28
+    },
+    {
+      "epoch": 0.0928,
+      "grad_norm": 0.3566887338551931,
+      "learning_rate": 0.0001980530769868059,
+      "loss": 0.8921,
+      "step": 29
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 0.37413026403570376,
+      "learning_rate": 0.00019784350367254322,
+      "loss": 0.8723,
+      "step": 30
+    },
+    {
+      "epoch": 0.0992,
+      "grad_norm": 0.4494091415776092,
+      "learning_rate": 0.0001976233423581255,
+      "loss": 0.9454,
+      "step": 31
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 0.4380782092404729,
+      "learning_rate": 0.0001973926168680066,
+      "loss": 0.9703,
+      "step": 32
+    },
+    {
+      "epoch": 0.1056,
+      "grad_norm": 0.4024207233737277,
+      "learning_rate": 0.00019715135216982798,
+      "loss": 0.8925,
+      "step": 33
+    },
+    {
+      "epoch": 0.1088,
+      "grad_norm": 0.40406197533093235,
+      "learning_rate": 0.0001968995743717171,
+      "loss": 0.9361,
+      "step": 34
+    },
+    {
+      "epoch": 0.112,
+      "grad_norm": 0.3618341982376051,
+      "learning_rate": 0.00019663731071946206,
+      "loss": 0.8614,
+      "step": 35
+    },
+    {
+      "epoch": 0.1152,
+      "grad_norm": 0.3658614124884964,
+      "learning_rate": 0.00019636458959356316,
+      "loss": 0.8498,
+      "step": 36
+    },
+    {
+      "epoch": 0.1184,
+      "grad_norm": 0.5573838774300858,
+      "learning_rate": 0.0001960814405061619,
+      "loss": 0.8736,
+      "step": 37
+    },
+    {
+      "epoch": 0.1216,
+      "grad_norm": 0.38557296694847226,
+      "learning_rate": 0.00019578789409784727,
+      "loss": 0.8929,
+      "step": 38
+    },
+    {
+      "epoch": 0.1248,
+      "grad_norm": 0.40079642763917983,
+      "learning_rate": 0.00019548398213434007,
+      "loss": 0.8423,
+      "step": 39
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 0.38961763082023404,
+      "learning_rate": 0.00019516973750305532,
+      "loss": 0.8835,
+      "step": 40
+    },
+    {
+      "epoch": 0.1312,
+      "grad_norm": 0.4099214309122936,
+      "learning_rate": 0.00019484519420954354,
+      "loss": 0.9405,
+      "step": 41
+    },
+    {
+      "epoch": 0.1344,
+      "grad_norm": 0.39275472621957447,
+      "learning_rate": 0.00019451038737381077,
+      "loss": 0.8549,
+      "step": 42
+    },
+    {
+      "epoch": 0.1376,
+      "grad_norm": 0.37890491998796727,
+      "learning_rate": 0.00019416535322651818,
+      "loss": 0.9153,
+      "step": 43
+    },
+    {
+      "epoch": 0.1408,
+      "grad_norm": 0.3677543896244243,
+      "learning_rate": 0.00019381012910506146,
+      "loss": 0.8453,
+      "step": 44
+    },
+    {
+      "epoch": 0.144,
+      "grad_norm": 0.3798879355149729,
+      "learning_rate": 0.00019344475344953012,
+      "loss": 0.8136,
+      "step": 45
+    },
+    {
+      "epoch": 0.1472,
+      "grad_norm": 0.39541256509109524,
+      "learning_rate": 0.00019306926579854821,
+      "loss": 0.8917,
+      "step": 46
+    },
+    {
+      "epoch": 0.1504,
+      "grad_norm": 0.3933918528877009,
+      "learning_rate": 0.00019268370678499533,
+      "loss": 0.8155,
+      "step": 47
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 0.40698599023324944,
+      "learning_rate": 0.0001922881181316097,
+      "loss": 0.8666,
+      "step": 48
+    },
+    {
+      "epoch": 0.1568,
+      "grad_norm": 0.40042005946240466,
+      "learning_rate": 0.00019188254264647337,
+      "loss": 0.8412,
+      "step": 49
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.4607168760067279,
+      "learning_rate": 0.0001914670242183795,
+      "loss": 0.8868,
+      "step": 50
+    },
+    {
+      "epoch": 0.1632,
+      "grad_norm": 0.39972125720735685,
+      "learning_rate": 0.0001910416078120832,
+      "loss": 0.8918,
+      "step": 51
+    },
+    {
+      "epoch": 0.1664,
+      "grad_norm": 0.4188280311405362,
+      "learning_rate": 0.0001906063394634356,
+      "loss": 0.902,
+      "step": 52
+    },
+    {
+      "epoch": 0.1696,
+      "grad_norm": 0.42940697765165986,
+      "learning_rate": 0.00019016126627440237,
+      "loss": 0.8641,
+      "step": 53
+    },
+    {
+      "epoch": 0.1728,
+      "grad_norm": 0.3545329739920464,
+      "learning_rate": 0.00018970643640796642,
+      "loss": 0.8735,
+      "step": 54
+    },
+    {
+      "epoch": 0.176,
+      "grad_norm": 0.39694907744821073,
+      "learning_rate": 0.000189241899082916,
+      "loss": 0.7712,
+      "step": 55
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 0.3873462097880066,
+      "learning_rate": 0.00018876770456851877,
+      "loss": 0.8613,
+      "step": 56
+    },
+    {
+      "epoch": 0.1824,
+      "grad_norm": 0.39944251069469355,
+      "learning_rate": 0.0001882839041790818,
+      "loss": 0.8342,
+      "step": 57
+    },
+    {
+      "epoch": 0.1856,
+      "grad_norm": 0.38873644333253415,
+      "learning_rate": 0.00018779055026839868,
+      "loss": 0.8602,
+      "step": 58
+    },
+    {
+      "epoch": 0.1888,
+      "grad_norm": 0.41970967893409644,
+      "learning_rate": 0.00018728769622408423,
+      "loss": 0.8577,
+      "step": 59
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 0.3660391396843375,
+      "learning_rate": 0.00018677539646179707,
+      "loss": 0.8452,
+      "step": 60
+    },
+    {
+      "epoch": 0.1952,
+      "grad_norm": 0.4567680983316884,
+      "learning_rate": 0.00018625370641935129,
+      "loss": 0.9641,
+      "step": 61
+    },
+    {
+      "epoch": 0.1984,
+      "grad_norm": 0.4347858356915908,
+      "learning_rate": 0.00018572268255071718,
+      "loss": 0.9423,
+      "step": 62
+    },
+    {
+      "epoch": 0.2016,
+      "grad_norm": 0.3337070071383876,
+      "learning_rate": 0.00018518238231991218,
+      "loss": 0.8076,
+      "step": 63
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 0.4286830947909843,
+      "learning_rate": 0.00018463286419478255,
+      "loss": 0.9527,
+      "step": 64
+    },
+    {
+      "epoch": 0.208,
+      "grad_norm": 0.35295999710631726,
+      "learning_rate": 0.00018407418764067627,
+      "loss": 0.8146,
+      "step": 65
+    },
+    {
+      "epoch": 0.2112,
+      "grad_norm": 0.40069838158976223,
+      "learning_rate": 0.00018350641311400812,
+      "loss": 0.9182,
+      "step": 66
+    },
+    {
+      "epoch": 0.2144,
+      "grad_norm": 0.428778831044889,
+      "learning_rate": 0.0001829296020557174,
+      "loss": 0.9332,
+      "step": 67
+    },
+    {
+      "epoch": 0.2176,
+      "grad_norm": 0.35155177053946185,
+      "learning_rate": 0.00018234381688461942,
+      "loss": 0.8472,
+      "step": 68
+    },
+    {
+      "epoch": 0.2208,
+      "grad_norm": 0.3393523853315711,
+      "learning_rate": 0.0001817491209906506,
+      "loss": 0.8116,
+      "step": 69
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 0.39467587805954535,
+      "learning_rate": 0.00018114557872800905,
+      "loss": 0.8882,
+      "step": 70
+    },
+    {
+      "epoch": 0.2272,
+      "grad_norm": 0.45738467437649594,
+      "learning_rate": 0.00018053325540819045,
+      "loss": 0.9899,
+      "step": 71
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 0.40860575562909096,
+      "learning_rate": 0.0001799122172929206,
+      "loss": 0.8518,
+      "step": 72
+    },
+    {
+      "epoch": 0.2336,
+      "grad_norm": 0.37358375585659853,
+      "learning_rate": 0.00017928253158698473,
+      "loss": 0.8671,
+      "step": 73
+    },
+    {
+      "epoch": 0.2368,
+      "grad_norm": 0.3614438681293151,
+      "learning_rate": 0.0001786442664309554,
+      "loss": 0.7581,
+      "step": 74
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.39185167616797406,
+      "learning_rate": 0.0001779974908938184,
+      "loss": 0.8273,
+      "step": 75
+    },
+    {
+      "epoch": 0.2432,
+      "grad_norm": 0.3725033489438709,
+      "learning_rate": 0.0001773422749654988,
+      "loss": 0.7593,
+      "step": 76
+    },
+    {
+      "epoch": 0.2464,
+      "grad_norm": 0.3879109255323267,
+      "learning_rate": 0.00017667868954928694,
+      "loss": 0.8456,
+      "step": 77
+    },
+    {
+      "epoch": 0.2496,
+      "grad_norm": 0.3449533879291914,
+      "learning_rate": 0.00017600680645416583,
+      "loss": 0.8246,
+      "step": 78
+    },
+    {
+      "epoch": 0.2528,
+      "grad_norm": 0.3557304717641571,
+      "learning_rate": 0.00017532669838704035,
+      "loss": 0.8002,
+      "step": 79
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 0.39624711741320795,
+      "learning_rate": 0.00017463843894486937,
+      "loss": 0.845,
+      "step": 80
+    },
+    {
+      "epoch": 0.2592,
+      "grad_norm": 0.4040277144139735,
+      "learning_rate": 0.0001739421026067017,
+      "loss": 0.9255,
+      "step": 81
+    },
+    {
+      "epoch": 0.2624,
+      "grad_norm": 0.3573834496881195,
+      "learning_rate": 0.00017323776472561627,
+      "loss": 0.8664,
+      "step": 82
+    },
+    {
+      "epoch": 0.2656,
+      "grad_norm": 0.39861349237455607,
+      "learning_rate": 0.00017252550152056795,
+      "loss": 0.869,
+      "step": 83
+    },
+    {
+      "epoch": 0.2688,
+      "grad_norm": 0.4122338593577007,
+      "learning_rate": 0.0001718053900681397,
+      "loss": 0.8542,
+      "step": 84
+    },
+    {
+      "epoch": 0.272,
+      "grad_norm": 0.38711562232847824,
+      "learning_rate": 0.00017107750829420176,
+      "loss": 0.8265,
+      "step": 85
+    },
+    {
+      "epoch": 0.2752,
+      "grad_norm": 0.39947214892388133,
+      "learning_rate": 0.00017034193496547902,
+      "loss": 0.9227,
+      "step": 86
+    },
+    {
+      "epoch": 0.2784,
+      "grad_norm": 0.36818193645695974,
+      "learning_rate": 0.00016959874968102735,
+      "loss": 0.8025,
+      "step": 87
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 0.3563867965150087,
+      "learning_rate": 0.00016884803286362,
+      "loss": 0.799,
+      "step": 88
+    },
+    {
+      "epoch": 0.2848,
+      "grad_norm": 0.3863926930438656,
+      "learning_rate": 0.00016808986575104465,
+      "loss": 0.9039,
+      "step": 89
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 0.35592980299837496,
+      "learning_rate": 0.00016732433038731242,
+      "loss": 0.8311,
+      "step": 90
+    },
+    {
+      "epoch": 0.2912,
+      "grad_norm": 0.4204313739806602,
+      "learning_rate": 0.0001665515096137797,
+      "loss": 0.9226,
+      "step": 91
+    },
+    {
+      "epoch": 0.2944,
+      "grad_norm": 0.3723535694546385,
+      "learning_rate": 0.00016577148706018328,
+      "loss": 0.8311,
+      "step": 92
+    },
+    {
+      "epoch": 0.2976,
+      "grad_norm": 0.41234782362203953,
+      "learning_rate": 0.00016498434713559088,
+      "loss": 0.8979,
+      "step": 93
+    },
+    {
+      "epoch": 0.3008,
+      "grad_norm": 0.4053425895286585,
+      "learning_rate": 0.00016419017501926656,
+      "loss": 0.8344,
+      "step": 94
+    },
+    {
+      "epoch": 0.304,
+      "grad_norm": 0.36505412209597476,
+      "learning_rate": 0.0001633890566514535,
+      "loss": 0.8345,
+      "step": 95
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 0.382611792068928,
+      "learning_rate": 0.00016258107872407375,
+      "loss": 0.8749,
+      "step": 96
+    },
+    {
+      "epoch": 0.3104,
+      "grad_norm": 0.37741695374713863,
+      "learning_rate": 0.0001617663286713474,
+      "loss": 0.8493,
+      "step": 97
+    },
+    {
+      "epoch": 0.3136,
+      "grad_norm": 0.4676602590760305,
+      "learning_rate": 0.00016094489466033043,
+      "loss": 0.8432,
+      "step": 98
+    },
+    {
+      "epoch": 0.3168,
+      "grad_norm": 0.34327944895308454,
+      "learning_rate": 0.00016011686558137448,
+      "loss": 0.7974,
+      "step": 99
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 0.3556557830313786,
+      "learning_rate": 0.0001592823310385073,
+      "loss": 0.8677,
+      "step": 100
+    },
+    {
+      "epoch": 0.3232,
+      "grad_norm": 0.4186166790777886,
+      "learning_rate": 0.0001584413813397364,
+      "loss": 0.8586,
+      "step": 101
+    },
+    {
+      "epoch": 0.3264,
+      "grad_norm": 0.37112136732584244,
+      "learning_rate": 0.00015759410748727662,
+      "loss": 0.8209,
+      "step": 102
+    },
+    {
+      "epoch": 0.3296,
+      "grad_norm": 0.38959686649650355,
+      "learning_rate": 0.00015674060116770236,
+      "loss": 0.8077,
+      "step": 103
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 0.36221559570247797,
+      "learning_rate": 0.00015588095474202595,
+      "loss": 0.8142,
+      "step": 104
+    },
+    {
+      "epoch": 0.336,
+      "grad_norm": 0.3408395021986205,
+      "learning_rate": 0.00015501526123570277,
+      "loss": 0.825,
+      "step": 105
+    },
+    {
+      "epoch": 0.3392,
+      "grad_norm": 0.46256890237184806,
+      "learning_rate": 0.00015414361432856475,
+      "loss": 0.9299,
+      "step": 106
+    },
+    {
+      "epoch": 0.3424,
+      "grad_norm": 0.4279687120096611,
+      "learning_rate": 0.0001532661083446829,
+      "loss": 0.9313,
+      "step": 107
+    },
+    {
+      "epoch": 0.3456,
+      "grad_norm": 0.499080817408835,
+      "learning_rate": 0.00015238283824216015,
+      "loss": 0.9257,
+      "step": 108
+    },
+    {
+      "epoch": 0.3488,
+      "grad_norm": 0.3677888976110564,
+      "learning_rate": 0.00015149389960285558,
+      "loss": 0.8234,
+      "step": 109
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 0.40300933145036566,
+      "learning_rate": 0.00015059938862204127,
+      "loss": 0.8661,
+      "step": 110
+    },
+    {
+      "epoch": 0.3552,
+      "grad_norm": 0.35272019193019816,
+      "learning_rate": 0.00014969940209799248,
+      "loss": 0.7685,
+      "step": 111
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 0.42072850665932426,
+      "learning_rate": 0.00014879403742151283,
+      "loss": 0.838,
+      "step": 112
+    },
+    {
+      "epoch": 0.3616,
+      "grad_norm": 0.3843947444096128,
+      "learning_rate": 0.00014788339256539544,
+      "loss": 0.7295,
+      "step": 113
+    },
+    {
+      "epoch": 0.3648,
+      "grad_norm": 0.43191105712976785,
+      "learning_rate": 0.0001469675660738206,
+      "loss": 0.8535,
+      "step": 114
+    },
+    {
+      "epoch": 0.368,
+      "grad_norm": 0.3584146254614676,
+      "learning_rate": 0.00014604665705169237,
+      "loss": 0.8303,
+      "step": 115
+    },
+    {
+      "epoch": 0.3712,
+      "grad_norm": 0.4020587727121328,
+      "learning_rate": 0.00014512076515391375,
+      "loss": 0.8827,
+      "step": 116
+    },
+    {
+      "epoch": 0.3744,
+      "grad_norm": 0.42244806005338637,
+      "learning_rate": 0.00014418999057460276,
+      "loss": 0.8637,
+      "step": 117
+    },
+    {
+      "epoch": 0.3776,
+      "grad_norm": 0.3490560388254176,
+      "learning_rate": 0.0001432544340362501,
+      "loss": 0.7685,
+      "step": 118
+    },
+    {
+      "epoch": 0.3808,
+      "grad_norm": 0.3610932978019899,
+      "learning_rate": 0.00014231419677881966,
+      "loss": 0.7689,
+      "step": 119
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 0.35696651622274633,
+      "learning_rate": 0.00014136938054879283,
+      "loss": 0.8692,
+      "step": 120
+    },
+    {
+      "epoch": 0.3872,
+      "grad_norm": 0.37982273980822234,
+      "learning_rate": 0.00014042008758815818,
+      "loss": 0.8633,
+      "step": 121
+    },
+    {
+      "epoch": 0.3904,
+      "grad_norm": 0.49338348781038344,
+      "learning_rate": 0.00013946642062334766,
+      "loss": 0.7882,
+      "step": 122
+    },
+    {
+      "epoch": 0.3936,
+      "grad_norm": 0.41407314429068576,
+      "learning_rate": 0.00013850848285411994,
+      "loss": 0.8352,
+      "step": 123
+    },
+    {
+      "epoch": 0.3968,
+      "grad_norm": 0.35863668034543733,
+      "learning_rate": 0.000137546377942393,
+      "loss": 0.824,
+      "step": 124
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.34638170296412163,
+      "learning_rate": 0.00013658021000102636,
+      "loss": 0.7332,
+      "step": 125
+    },
+    {
+      "epoch": 0.4032,
+      "grad_norm": 0.33192770455483456,
+      "learning_rate": 0.00013561008358255468,
+      "loss": 0.8012,
+      "step": 126
+    },
+    {
+      "epoch": 0.4064,
+      "grad_norm": 0.4018840350246986,
+      "learning_rate": 0.00013463610366787392,
+      "loss": 0.8274,
+      "step": 127
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 0.46935625670519754,
+      "learning_rate": 0.00013365837565488064,
+      "loss": 0.9137,
+      "step": 128
+    },
+    {
+      "epoch": 0.4128,
+      "grad_norm": 0.3467457829185042,
+      "learning_rate": 0.0001326770053470668,
+      "loss": 0.8057,
+      "step": 129
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 0.3713608562339026,
+      "learning_rate": 0.0001316920989420703,
+      "loss": 0.778,
+      "step": 130
+    },
+    {
+      "epoch": 0.4192,
+      "grad_norm": 0.2953063312947437,
+      "learning_rate": 0.00013070376302018287,
+      "loss": 0.7741,
+      "step": 131
+    },
+    {
+      "epoch": 0.4224,
+      "grad_norm": 0.315778283756295,
+      "learning_rate": 0.00012971210453281674,
+      "loss": 0.7968,
+      "step": 132
+    },
+    {
+      "epoch": 0.4256,
+      "grad_norm": 0.3777293731399955,
+      "learning_rate": 0.000128717230790931,
+      "loss": 0.8476,
+      "step": 133
+    },
+    {
+      "epoch": 0.4288,
+      "grad_norm": 0.36565481085206214,
+      "learning_rate": 0.00012771924945341906,
+      "loss": 0.7424,
+      "step": 134
+    },
+    {
+      "epoch": 0.432,
+      "grad_norm": 0.40097648303660766,
+      "learning_rate": 0.00012671826851545851,
+      "loss": 0.8961,
+      "step": 135
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 0.4335469508570514,
+      "learning_rate": 0.0001257143962968246,
+      "loss": 0.9337,
+      "step": 136
+    },
+    {
+      "epoch": 0.4384,
+      "grad_norm": 0.3571253000150674,
+      "learning_rate": 0.00012470774143016853,
+      "loss": 0.7772,
+      "step": 137
+    },
+    {
+      "epoch": 0.4416,
+      "grad_norm": 0.4175995621863514,
+      "learning_rate": 0.00012369841284926188,
+      "loss": 0.7899,
+      "step": 138
+    },
+    {
+      "epoch": 0.4448,
+      "grad_norm": 0.3555584844534414,
+      "learning_rate": 0.00012268651977720866,
+      "loss": 0.8058,
+      "step": 139
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 0.4478506698284743,
+      "learning_rate": 0.00012167217171462566,
+      "loss": 0.8602,
+      "step": 140
+    },
+    {
+      "epoch": 0.4512,
+      "grad_norm": 0.3924154628163152,
+      "learning_rate": 0.0001206554784277931,
+      "loss": 0.8566,
+      "step": 141
+    },
+    {
+      "epoch": 0.4544,
+      "grad_norm": 0.3413680221828951,
+      "learning_rate": 0.00011963654993677645,
+      "loss": 0.8442,
+      "step": 142
+    },
+    {
+      "epoch": 0.4576,
+      "grad_norm": 0.37597687671024504,
+      "learning_rate": 0.00011861549650352069,
+      "loss": 0.7896,
+      "step": 143
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 0.35508126757199976,
+      "learning_rate": 0.00011759242861991855,
+      "loss": 0.7702,
+      "step": 144
+    },
+    {
+      "epoch": 0.464,
+      "grad_norm": 0.35676098581829857,
+      "learning_rate": 0.00011656745699585371,
+      "loss": 0.7463,
+      "step": 145
+    },
+    {
+      "epoch": 0.4672,
+      "grad_norm": 0.34262057715324995,
+      "learning_rate": 0.00011554069254722051,
+      "loss": 0.8023,
+      "step": 146
+    },
+    {
+      "epoch": 0.4704,
+      "grad_norm": 0.3160775424152765,
+      "learning_rate": 0.00011451224638392129,
+      "loss": 0.753,
+      "step": 147
+    },
+    {
+      "epoch": 0.4736,
+      "grad_norm": 0.41408430747051145,
+      "learning_rate": 0.00011348222979784289,
+      "loss": 0.8722,
+      "step": 148
+    },
+    {
+      "epoch": 0.4768,
+      "grad_norm": 0.3807991104363706,
+      "learning_rate": 0.00011245075425081328,
+      "loss": 0.7902,
+      "step": 149
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.3444267731692557,
+      "learning_rate": 0.00011141793136253986,
+      "loss": 0.7729,
+      "step": 150
+    },
+    {
+      "epoch": 0.4832,
+      "grad_norm": 0.3198788705719864,
+      "learning_rate": 0.0001103838728985307,
+      "loss": 0.7863,
+      "step": 151
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 0.41645947104870057,
+      "learning_rate": 0.000109348690758,
+      "loss": 0.8442,
+      "step": 152
+    },
+    {
+      "epoch": 0.4896,
+      "grad_norm": 0.3473642274949473,
+      "learning_rate": 0.00010831249696175918,
+      "loss": 0.7755,
+      "step": 153
+    },
+    {
+      "epoch": 0.4928,
+      "grad_norm": 0.351233087536269,
+      "learning_rate": 0.0001072754036400944,
+      "loss": 0.7799,
+      "step": 154
+    },
+    {
+      "epoch": 0.496,
+      "grad_norm": 0.39062438800982935,
+      "learning_rate": 0.00010623752302063283,
+      "loss": 0.8262,
+      "step": 155
+    },
+    {
+      "epoch": 0.4992,
+      "grad_norm": 0.4661216475962772,
+      "learning_rate": 0.00010519896741619803,
+      "loss": 0.8754,
+      "step": 156
+    },
+    {
+      "epoch": 0.5024,
+      "grad_norm": 0.3680543907349618,
+      "learning_rate": 0.00010415984921265609,
+      "loss": 0.8511,
+      "step": 157
+    },
+    {
+      "epoch": 0.5056,
+      "grad_norm": 0.3554554750391017,
+      "learning_rate": 0.00010312028085675391,
+      "loss": 0.7905,
+      "step": 158
+    },
+    {
+      "epoch": 0.5088,
+      "grad_norm": 0.34003331830378203,
+      "learning_rate": 0.00010208037484395114,
+      "loss": 0.8518,
+      "step": 159
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 0.35691497758794416,
+      "learning_rate": 0.00010104024370624644,
+      "loss": 0.8042,
+      "step": 160
+    },
+    {
+      "epoch": 0.5152,
+      "grad_norm": 0.3680532317532297,
+      "learning_rate": 0.0001,
+      "loss": 0.826,
+      "step": 161
+    },
+    {
+      "epoch": 0.5184,
+      "grad_norm": 0.4306874440922751,
+      "learning_rate": 9.895975629375359e-05,
+      "loss": 0.8114,
+      "step": 162
+    },
+    {
+      "epoch": 0.5216,
+      "grad_norm": 0.3927827556296759,
+      "learning_rate": 9.791962515604887e-05,
+      "loss": 0.8477,
+      "step": 163
+    },
+    {
+      "epoch": 0.5248,
+      "grad_norm": 0.3688233147013582,
+      "learning_rate": 9.687971914324607e-05,
+      "loss": 0.787,
+      "step": 164
+    },
+    {
+      "epoch": 0.528,
+      "grad_norm": 0.3460381638988584,
+      "learning_rate": 9.584015078734395e-05,
+      "loss": 0.7861,
+      "step": 165
+    },
+    {
+      "epoch": 0.5312,
+      "grad_norm": 0.3293873896538617,
+      "learning_rate": 9.480103258380198e-05,
+      "loss": 0.7599,
+      "step": 166
+    },
+    {
+      "epoch": 0.5344,
+      "grad_norm": 0.33067678398241424,
+      "learning_rate": 9.376247697936719e-05,
+      "loss": 0.7788,
+      "step": 167
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 0.30272346196435956,
+      "learning_rate": 9.272459635990562e-05,
+      "loss": 0.7594,
+      "step": 168
+    },
+    {
+      "epoch": 0.5408,
+      "grad_norm": 0.35694173144440655,
+      "learning_rate": 9.168750303824084e-05,
+      "loss": 0.8066,
+      "step": 169
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 0.36394962483417836,
+      "learning_rate": 9.065130924199998e-05,
+      "loss": 0.6855,
+      "step": 170
+    },
+    {
+      "epoch": 0.5472,
+      "grad_norm": 0.4052142574936814,
+      "learning_rate": 8.961612710146934e-05,
+      "loss": 0.8238,
+      "step": 171
+    },
+    {
+      "epoch": 0.5504,
+      "grad_norm": 0.4466361193230218,
+      "learning_rate": 8.858206863746018e-05,
+      "loss": 0.7949,
+      "step": 172
+    },
+    {
+      "epoch": 0.5536,
+      "grad_norm": 0.3691146825930911,
+      "learning_rate": 8.754924574918675e-05,
+      "loss": 0.7591,
+      "step": 173
+    },
+    {
+      "epoch": 0.5568,
+      "grad_norm": 0.3884324905017403,
+      "learning_rate": 8.651777020215712e-05,
+      "loss": 0.7671,
+      "step": 174
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.37235345511558177,
+      "learning_rate": 8.548775361607872e-05,
+      "loss": 0.7954,
+      "step": 175
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 0.43342144747289674,
+      "learning_rate": 8.445930745277953e-05,
+      "loss": 0.7674,
+      "step": 176
+    },
+    {
+      "epoch": 0.5664,
+      "grad_norm": 0.3601056959148259,
+      "learning_rate": 8.343254300414628e-05,
+      "loss": 0.6814,
+      "step": 177
+    },
+    {
+      "epoch": 0.5696,
+      "grad_norm": 0.34765672643846884,
+      "learning_rate": 8.240757138008149e-05,
+      "loss": 0.8115,
+      "step": 178
+    },
+    {
+      "epoch": 0.5728,
+      "grad_norm": 0.3542918688839561,
+      "learning_rate": 8.138450349647936e-05,
+      "loss": 0.7751,
+      "step": 179
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 0.43405703341062163,
+      "learning_rate": 8.036345006322359e-05,
+      "loss": 0.883,
+      "step": 180
+    },
+    {
+      "epoch": 0.5792,
+      "grad_norm": 0.35990538609349737,
+      "learning_rate": 7.934452157220694e-05,
+      "loss": 0.8274,
+      "step": 181
+    },
+    {
+      "epoch": 0.5824,
+      "grad_norm": 0.36566849513888866,
+      "learning_rate": 7.832782828537437e-05,
+      "loss": 0.8316,
+      "step": 182
+    },
+    {
+      "epoch": 0.5856,
+      "grad_norm": 0.3880712586476513,
+      "learning_rate": 7.731348022279134e-05,
+      "loss": 0.7823,
+      "step": 183
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 0.40070345592700185,
+      "learning_rate": 7.630158715073813e-05,
+      "loss": 0.8682,
+      "step": 184
+    },
+    {
+      "epoch": 0.592,
+      "grad_norm": 0.452126500075801,
+      "learning_rate": 7.52922585698315e-05,
+      "loss": 0.7349,
+      "step": 185
+    },
+    {
+      "epoch": 0.5952,
+      "grad_norm": 0.39133686156881714,
+      "learning_rate": 7.428560370317542e-05,
+      "loss": 0.8166,
+      "step": 186
+    },
+    {
+      "epoch": 0.5984,
+      "grad_norm": 0.30433046880932474,
+      "learning_rate": 7.328173148454151e-05,
+      "loss": 0.7799,
+      "step": 187
+    },
+    {
+      "epoch": 0.6016,
+      "grad_norm": 0.38912324970709505,
+      "learning_rate": 7.228075054658096e-05,
+      "loss": 0.874,
+      "step": 188
+    },
+    {
+      "epoch": 0.6048,
+      "grad_norm": 0.34571755403615634,
+      "learning_rate": 7.1282769209069e-05,
+      "loss": 0.7972,
+      "step": 189
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 0.3759979893898139,
+      "learning_rate": 7.028789546718326e-05,
+      "loss": 0.7436,
+      "step": 190
+    },
+    {
+      "epoch": 0.6112,
+      "grad_norm": 0.3396623833976361,
+      "learning_rate": 6.929623697981718e-05,
+      "loss": 0.771,
+      "step": 191
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 0.3930329153264238,
+      "learning_rate": 6.830790105792973e-05,
+      "loss": 0.7294,
+      "step": 192
+    },
+    {
+      "epoch": 0.6176,
+      "grad_norm": 0.384939784498545,
+      "learning_rate": 6.732299465293322e-05,
+      "loss": 0.8072,
+      "step": 193
+    },
+    {
+      "epoch": 0.6208,
+      "grad_norm": 0.3670712491930503,
+      "learning_rate": 6.63416243451194e-05,
+      "loss": 0.8139,
+      "step": 194
+    },
+    {
+      "epoch": 0.624,
+      "grad_norm": 0.37471193485584176,
+      "learning_rate": 6.536389633212609e-05,
+      "loss": 0.8026,
+      "step": 195
+    },
+    {
+      "epoch": 0.6272,
+      "grad_norm": 0.3466481941938704,
+      "learning_rate": 6.43899164174453e-05,
+      "loss": 0.7856,
+      "step": 196
+    },
+    {
+      "epoch": 0.6304,
+      "grad_norm": 0.334681278560714,
+      "learning_rate": 6.341978999897365e-05,
+      "loss": 0.7457,
+      "step": 197
+    },
+    {
+      "epoch": 0.6336,
+      "grad_norm": 0.3580934295092547,
+      "learning_rate": 6.245362205760704e-05,
+      "loss": 0.8062,
+      "step": 198
+    },
+    {
+      "epoch": 0.6368,
+      "grad_norm": 0.41942104569534644,
+      "learning_rate": 6.149151714588009e-05,
+      "loss": 0.8307,
+      "step": 199
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.3044902604615677,
+      "learning_rate": 6.053357937665237e-05,
+      "loss": 0.6707,
+      "step": 200
+    },
+    {
+      "epoch": 0.6432,
+      "grad_norm": 0.37144444775815594,
+      "learning_rate": 5.957991241184184e-05,
+      "loss": 0.7813,
+      "step": 201
+    },
+    {
+      "epoch": 0.6464,
+      "grad_norm": 0.34248491706444256,
+      "learning_rate": 5.863061945120719e-05,
+      "loss": 0.8051,
+      "step": 202
+    },
+    {
+      "epoch": 0.6496,
+      "grad_norm": 0.34748171044370163,
+      "learning_rate": 5.768580322118034e-05,
+      "loss": 0.7627,
+      "step": 203
+    },
+    {
+      "epoch": 0.6528,
+      "grad_norm": 0.38512453359767645,
+      "learning_rate": 5.6745565963749925e-05,
+      "loss": 0.8318,
+      "step": 204
+    },
+    {
+      "epoch": 0.656,
+      "grad_norm": 0.36707414698679175,
+      "learning_rate": 5.5810009425397294e-05,
+      "loss": 0.8305,
+      "step": 205
+    },
+    {
+      "epoch": 0.6592,
+      "grad_norm": 0.48217984342912773,
+      "learning_rate": 5.487923484608629e-05,
+      "loss": 0.9014,
+      "step": 206
+    },
+    {
+      "epoch": 0.6624,
+      "grad_norm": 0.38863801931906744,
+      "learning_rate": 5.395334294830765e-05,
+      "loss": 0.8525,
+      "step": 207
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 0.35518873432323794,
+      "learning_rate": 5.3032433926179395e-05,
+      "loss": 0.7778,
+      "step": 208
+    },
+    {
+      "epoch": 0.6688,
+      "grad_norm": 0.34123260040379993,
+      "learning_rate": 5.211660743460458e-05,
+      "loss": 0.775,
+      "step": 209
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 0.4001757666899788,
+      "learning_rate": 5.1205962578487155e-05,
+      "loss": 0.7731,
+      "step": 210
+    },
+    {
+      "epoch": 0.6752,
+      "grad_norm": 0.3315496723721597,
+      "learning_rate": 5.030059790200756e-05,
+      "loss": 0.7922,
+      "step": 211
+    },
+    {
+      "epoch": 0.6784,
+      "grad_norm": 0.3258037685568246,
+      "learning_rate": 4.940061137795876e-05,
+      "loss": 0.699,
+      "step": 212
+    },
+    {
+      "epoch": 0.6816,
+      "grad_norm": 0.3884518590202934,
+      "learning_rate": 4.850610039714444e-05,
+      "loss": 0.7866,
+      "step": 213
+    },
+    {
+      "epoch": 0.6848,
+      "grad_norm": 0.7104627436824442,
+      "learning_rate": 4.761716175783989e-05,
+      "loss": 0.8089,
+      "step": 214
+    },
+    {
+      "epoch": 0.688,
+      "grad_norm": 0.3013941102836104,
+      "learning_rate": 4.673389165531714e-05,
+      "loss": 0.7054,
+      "step": 215
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 0.3731169801442719,
+      "learning_rate": 4.585638567143529e-05,
+      "loss": 0.7737,
+      "step": 216
+    },
+    {
+      "epoch": 0.6944,
+      "grad_norm": 0.30263267234838137,
+      "learning_rate": 4.498473876429726e-05,
+      "loss": 0.7029,
+      "step": 217
+    },
+    {
+      "epoch": 0.6976,
+      "grad_norm": 0.3659751206161679,
+      "learning_rate": 4.411904525797408e-05,
+      "loss": 0.8146,
+      "step": 218
+    },
+    {
+      "epoch": 0.7008,
+      "grad_norm": 0.36421909907458466,
+      "learning_rate": 4.325939883229766e-05,
+      "loss": 0.7769,
+      "step": 219
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 0.3403495561782968,
+      "learning_rate": 4.240589251272342e-05,
+      "loss": 0.8074,
+      "step": 220
+    },
+    {
+      "epoch": 0.7072,
+      "grad_norm": 0.4116298314841219,
+      "learning_rate": 4.155861866026364e-05,
+      "loss": 0.7561,
+      "step": 221
+    },
+    {
+      "epoch": 0.7104,
+      "grad_norm": 0.4017877316418641,
+      "learning_rate": 4.071766896149273e-05,
+      "loss": 0.7802,
+      "step": 222
+    },
+    {
+      "epoch": 0.7136,
+      "grad_norm": 0.3933048903616485,
+      "learning_rate": 3.988313441862553e-05,
+      "loss": 0.8302,
+      "step": 223
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 0.4053820034136759,
+      "learning_rate": 3.9055105339669595e-05,
+      "loss": 0.7537,
+      "step": 224
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.3422570001344906,
+      "learning_rate": 3.823367132865265e-05,
+      "loss": 0.7513,
+      "step": 225
+    },
+    {
+      "epoch": 0.7232,
+      "grad_norm": 0.3672321719467038,
+      "learning_rate": 3.741892127592625e-05,
+      "loss": 0.7552,
+      "step": 226
+    },
+    {
+      "epoch": 0.7264,
+      "grad_norm": 0.38655052693729053,
+      "learning_rate": 3.6610943348546526e-05,
+      "loss": 0.8076,
+      "step": 227
+    },
+    {
+      "epoch": 0.7296,
+      "grad_norm": 0.32046006517959386,
+      "learning_rate": 3.580982498073344e-05,
+      "loss": 0.7549,
+      "step": 228
+    },
+    {
+      "epoch": 0.7328,
+      "grad_norm": 0.45524335147940526,
+      "learning_rate": 3.501565286440914e-05,
+      "loss": 0.8619,
+      "step": 229
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 0.29405543478538154,
+      "learning_rate": 3.422851293981676e-05,
+      "loss": 0.7585,
+      "step": 230
+    },
+    {
+      "epoch": 0.7392,
+      "grad_norm": 0.38285886702055727,
+      "learning_rate": 3.3448490386220355e-05,
+      "loss": 0.8357,
+      "step": 231
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 0.3801889952884557,
+      "learning_rate": 3.2675669612687565e-05,
+      "loss": 0.8901,
+      "step": 232
+    },
+    {
+      "epoch": 0.7456,
+      "grad_norm": 0.35279011141194516,
+      "learning_rate": 3.191013424895536e-05,
+      "loss": 0.7481,
+      "step": 233
+    },
+    {
+      "epoch": 0.7488,
+      "grad_norm": 0.3471907142593491,
+      "learning_rate": 3.115196713638e-05,
+      "loss": 0.7855,
+      "step": 234
+    },
+    {
+      "epoch": 0.752,
+      "grad_norm": 0.36619187308979667,
+      "learning_rate": 3.040125031897264e-05,
+      "loss": 0.7625,
+      "step": 235
+    },
+    {
+      "epoch": 0.7552,
+      "grad_norm": 0.3762428412156853,
+      "learning_rate": 2.9658065034520978e-05,
+      "loss": 0.8157,
+      "step": 236
+    },
+    {
+      "epoch": 0.7584,
+      "grad_norm": 0.3171161168873978,
+      "learning_rate": 2.892249170579826e-05,
+      "loss": 0.7799,
+      "step": 237
+    },
+    {
+      "epoch": 0.7616,
+      "grad_norm": 0.3592730455514312,
+      "learning_rate": 2.8194609931860316e-05,
+      "loss": 0.7478,
+      "step": 238
+    },
+    {
+      "epoch": 0.7648,
+      "grad_norm": 0.4134931519699558,
+      "learning_rate": 2.7474498479432087e-05,
+      "loss": 0.8437,
+      "step": 239
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 0.30546020641842614,
+      "learning_rate": 2.6762235274383772e-05,
+      "loss": 0.7224,
+      "step": 240
+    },
+    {
+      "epoch": 0.7712,
+      "grad_norm": 0.3454408457555957,
+      "learning_rate": 2.6057897393298324e-05,
+      "loss": 0.7727,
+      "step": 241
+    },
+    {
+      "epoch": 0.7744,
+      "grad_norm": 0.3680677506255253,
+      "learning_rate": 2.536156105513062e-05,
+      "loss": 0.7745,
+      "step": 242
+    },
+    {
+      "epoch": 0.7776,
+      "grad_norm": 0.36274461235683025,
+      "learning_rate": 2.4673301612959654e-05,
+      "loss": 0.8305,
+      "step": 243
+    },
+    {
+      "epoch": 0.7808,
+      "grad_norm": 0.4459754472676995,
+      "learning_rate": 2.399319354583418e-05,
+      "loss": 0.8917,
+      "step": 244
+    },
+    {
+      "epoch": 0.784,
+      "grad_norm": 0.34453176080889125,
+      "learning_rate": 2.3321310450713062e-05,
+      "loss": 0.7476,
+      "step": 245
+    },
+    {
+      "epoch": 0.7872,
+      "grad_norm": 0.31196539201987367,
+      "learning_rate": 2.265772503450122e-05,
+      "loss": 0.7468,
+      "step": 246
+    },
+    {
+      "epoch": 0.7904,
+      "grad_norm": 0.3184608885787424,
+      "learning_rate": 2.2002509106181624e-05,
+      "loss": 0.7402,
+      "step": 247
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 0.36443373379957883,
+      "learning_rate": 2.1355733569044635e-05,
+      "loss": 0.7567,
+      "step": 248
+    },
+    {
+      "epoch": 0.7968,
+      "grad_norm": 0.4417924711660864,
+      "learning_rate": 2.0717468413015283e-05,
+      "loss": 0.823,
+      "step": 249
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.3236759284366048,
+      "learning_rate": 2.008778270707944e-05,
+      "loss": 0.6885,
+      "step": 250
+    },
+    {
+      "epoch": 0.8032,
+      "grad_norm": 0.39242767874833534,
+      "learning_rate": 1.946674459180955e-05,
+      "loss": 0.9275,
+      "step": 251
+    },
+    {
+      "epoch": 0.8064,
+      "grad_norm": 0.3427482665469463,
+      "learning_rate": 1.8854421271990964e-05,
+      "loss": 0.7372,
+      "step": 252
+    },
+    {
+      "epoch": 0.8096,
+      "grad_norm": 0.3626863241157415,
+      "learning_rate": 1.8250879009349398e-05,
+      "loss": 0.8017,
+      "step": 253
+    },
+    {
+      "epoch": 0.8128,
+      "grad_norm": 0.34605782828987924,
+      "learning_rate": 1.7656183115380577e-05,
+      "loss": 0.8484,
+      "step": 254
+    },
+    {
+      "epoch": 0.816,
+      "grad_norm": 0.4383327443511539,
+      "learning_rate": 1.707039794428259e-05,
+      "loss": 0.8186,
+      "step": 255
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 0.3447222581311919,
+      "learning_rate": 1.649358688599191e-05,
+      "loss": 0.7213,
+      "step": 256
+    },
+    {
+      "epoch": 0.8224,
+      "grad_norm": 0.31970536995245263,
+      "learning_rate": 1.5925812359323745e-05,
+      "loss": 0.7434,
+      "step": 257
+    },
+    {
+      "epoch": 0.8256,
+      "grad_norm": 0.3656450810361097,
+      "learning_rate": 1.5367135805217458e-05,
+      "loss": 0.7676,
+      "step": 258
+    },
+    {
+      "epoch": 0.8288,
+      "grad_norm": 0.36298269369872477,
+      "learning_rate": 1.4817617680087825e-05,
+      "loss": 0.7487,
+      "step": 259
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 0.42152653173895593,
+      "learning_rate": 1.4277317449282834e-05,
+      "loss": 0.8926,
+      "step": 260
+    },
+    {
+      "epoch": 0.8352,
+      "grad_norm": 0.38946242385235036,
+      "learning_rate": 1.3746293580648717e-05,
+      "loss": 0.8176,
+      "step": 261
+    },
+    {
+      "epoch": 0.8384,
+      "grad_norm": 0.41804903127540954,
+      "learning_rate": 1.3224603538202929e-05,
+      "loss": 0.7996,
+      "step": 262
+    },
+    {
+      "epoch": 0.8416,
+      "grad_norm": 0.31093486947817817,
+      "learning_rate": 1.2712303775915802e-05,
+      "loss": 0.7247,
+      "step": 263
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 0.3187140163567531,
+      "learning_rate": 1.220944973160133e-05,
+      "loss": 0.7473,
+      "step": 264
+    },
+    {
+      "epoch": 0.848,
+      "grad_norm": 0.3872254324451249,
+      "learning_rate": 1.1716095820918216e-05,
+      "loss": 0.8821,
+      "step": 265
+    },
+    {
+      "epoch": 0.8512,
+      "grad_norm": 0.4051984951107359,
+      "learning_rate": 1.1232295431481222e-05,
+      "loss": 0.7603,
+      "step": 266
+    },
+    {
+      "epoch": 0.8544,
+      "grad_norm": 0.40122800893302063,
+      "learning_rate": 1.0758100917083991e-05,
+      "loss": 0.8414,
+      "step": 267
+    },
+    {
+      "epoch": 0.8576,
+      "grad_norm": 0.35538652933955217,
+      "learning_rate": 1.0293563592033595e-05,
+      "loss": 0.7676,
+      "step": 268
+    },
+    {
+      "epoch": 0.8608,
+      "grad_norm": 0.3137491139738254,
+      "learning_rate": 9.838733725597615e-06,
+      "loss": 0.7501,
+      "step": 269
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 0.45963025353581044,
+      "learning_rate": 9.393660536564408e-06,
+      "loss": 0.7936,
+      "step": 270
+    },
+    {
+      "epoch": 0.8672,
+      "grad_norm": 0.3348526950988372,
+      "learning_rate": 8.958392187916841e-06,
+      "loss": 0.736,
+      "step": 271
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 0.39896905950270944,
+      "learning_rate": 8.532975781620512e-06,
+      "loss": 0.7654,
+      "step": 272
+    },
+    {
+      "epoch": 0.8736,
+      "grad_norm": 0.37722430967890513,
+      "learning_rate": 8.117457353526625e-06,
+      "loss": 0.7327,
+      "step": 273
+    },
+    {
+      "epoch": 0.8768,
+      "grad_norm": 0.31478006761399024,
+      "learning_rate": 7.711881868390291e-06,
+      "loss": 0.7615,
+      "step": 274
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 0.4188263703262344,
+      "learning_rate": 7.3162932150046885e-06,
+      "loss": 0.8783,
+      "step": 275
+    },
+    {
+      "epoch": 0.8832,
+      "grad_norm": 0.31942616334359136,
+      "learning_rate": 6.930734201451816e-06,
+      "loss": 0.725,
+      "step": 276
+    },
+    {
+      "epoch": 0.8864,
+      "grad_norm": 0.2997679274265762,
+      "learning_rate": 6.555246550469907e-06,
+      "loss": 0.7615,
+      "step": 277
+    },
+    {
+      "epoch": 0.8896,
+      "grad_norm": 0.36551094625988517,
+      "learning_rate": 6.189870894938587e-06,
+      "loss": 0.7969,
+      "step": 278
+    },
+    {
+      "epoch": 0.8928,
+      "grad_norm": 0.34379292762124014,
+      "learning_rate": 5.834646773481811e-06,
+      "loss": 0.8058,
+      "step": 279
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 0.35790166821440883,
+      "learning_rate": 5.489612626189245e-06,
+      "loss": 0.7782,
+      "step": 280
+    },
+    {
+      "epoch": 0.8992,
+      "grad_norm": 0.33879760718838586,
+      "learning_rate": 5.154805790456485e-06,
+      "loss": 0.7636,
+      "step": 281
+    },
+    {
+      "epoch": 0.9024,
+      "grad_norm": 0.31407783216446544,
+      "learning_rate": 4.830262496944693e-06,
+      "loss": 0.7323,
+      "step": 282
+    },
+    {
+      "epoch": 0.9056,
+      "grad_norm": 0.38757334898188933,
+      "learning_rate": 4.516017865659949e-06,
+      "loss": 0.8443,
+      "step": 283
+    },
+    {
+      "epoch": 0.9088,
+      "grad_norm": 0.39258218690616686,
+      "learning_rate": 4.21210590215273e-06,
+      "loss": 0.8223,
+      "step": 284
+    },
+    {
+      "epoch": 0.912,
+      "grad_norm": 0.3313155165113673,
+      "learning_rate": 3.918559493838114e-06,
+      "loss": 0.7951,
+      "step": 285
+    },
+    {
+      "epoch": 0.9152,
+      "grad_norm": 0.32605323503196637,
+      "learning_rate": 3.6354104064368566e-06,
+      "loss": 0.799,
+      "step": 286
+    },
+    {
+      "epoch": 0.9184,
+      "grad_norm": 0.393570523991794,
+      "learning_rate": 3.3626892805379562e-06,
+      "loss": 0.741,
+      "step": 287
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 0.37378520088314604,
+      "learning_rate": 3.100425628282899e-06,
+      "loss": 0.7429,
+      "step": 288
+    },
+    {
+      "epoch": 0.9248,
+      "grad_norm": 0.3179898573134195,
+      "learning_rate": 2.848647830172024e-06,
+      "loss": 0.7852,
+      "step": 289
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 0.33638562625723384,
+      "learning_rate": 2.607383131993424e-06,
+      "loss": 0.7518,
+      "step": 290
+    },
+    {
+      "epoch": 0.9312,
+      "grad_norm": 0.36906155605346674,
+      "learning_rate": 2.3766576418745022e-06,
+      "loss": 0.7948,
+      "step": 291
+    },
+    {
+      "epoch": 0.9344,
+      "grad_norm": 0.5865294985065448,
+      "learning_rate": 2.1564963274568027e-06,
+      "loss": 0.7865,
+      "step": 292
+    },
+    {
+      "epoch": 0.9376,
+      "grad_norm": 0.3357869205531904,
+      "learning_rate": 1.9469230131940907e-06,
+      "loss": 0.8067,
+      "step": 293
+    },
+    {
+      "epoch": 0.9408,
+      "grad_norm": 0.3214550435563662,
+      "learning_rate": 1.7479603777742938e-06,
+      "loss": 0.7578,
+      "step": 294
+    },
+    {
+      "epoch": 0.944,
+      "grad_norm": 0.3617697253780003,
+      "learning_rate": 1.559629951665298e-06,
+      "loss": 0.7783,
+      "step": 295
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 0.3851342510588961,
+      "learning_rate": 1.3819521147851123e-06,
+      "loss": 0.7932,
+      "step": 296
+    },
+    {
+      "epoch": 0.9504,
+      "grad_norm": 0.34426702777854773,
+      "learning_rate": 1.2149460942964098e-06,
+      "loss": 0.7609,
+      "step": 297
+    },
+    {
+      "epoch": 0.9536,
+      "grad_norm": 0.36253585088408663,
+      "learning_rate": 1.05862996252597e-06,
+      "loss": 0.7984,
+      "step": 298
+    },
+    {
+      "epoch": 0.9568,
+      "grad_norm": 0.4054570274935403,
+      "learning_rate": 9.130206350089765e-07,
+      "loss": 0.8,
+      "step": 299
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 0.34938837916613863,
+      "learning_rate": 7.781338686584927e-07,
+      "loss": 0.8201,
+      "step": 300
+    },
+    {
+      "epoch": 0.9632,
+      "grad_norm": 0.37777115892854124,
+      "learning_rate": 6.539842600603918e-07,
+      "loss": 0.8213,
+      "step": 301
+    },
+    {
+      "epoch": 0.9664,
+      "grad_norm": 0.37766797195751295,
+      "learning_rate": 5.405852438937764e-07,
+      "loss": 0.7736,
+      "step": 302
+    },
+    {
+      "epoch": 0.9696,
+      "grad_norm": 0.34012917606966536,
+      "learning_rate": 4.3794909147720773e-07,
+      "loss": 0.7852,
+      "step": 303
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 0.39195816403535333,
+      "learning_rate": 3.4608690944071263e-07,
+      "loss": 0.7284,
+      "step": 304
+    },
+    {
+      "epoch": 0.976,
+      "grad_norm": 0.38813789832520107,
+      "learning_rate": 2.6500863852395584e-07,
+      "loss": 0.8645,
+      "step": 305
+    },
+    {
+      "epoch": 0.9792,
+      "grad_norm": 0.32915922567617034,
+      "learning_rate": 1.947230525005006e-07,
+      "loss": 0.7437,
+      "step": 306
+    },
+    {
+      "epoch": 0.9824,
+      "grad_norm": 0.3351156902277233,
+      "learning_rate": 1.3523775722834587e-07,
+      "loss": 0.7069,
+      "step": 307
+    },
+    {
+      "epoch": 0.9856,
+      "grad_norm": 0.32461016981926744,
+      "learning_rate": 8.655918982689581e-08,
+      "loss": 0.7698,
+      "step": 308
+    },
+    {
+      "epoch": 0.9888,
+      "grad_norm": 0.34243368327755425,
+      "learning_rate": 4.8692617980350406e-08,
+      "loss": 0.8092,
+      "step": 309
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 0.35449923757603197,
+      "learning_rate": 2.164213936770576e-08,
+      "loss": 0.7537,
+      "step": 310
+    },
+    {
+      "epoch": 0.9952,
+      "grad_norm": 0.37595905935200546,
+      "learning_rate": 5.410681219286673e-09,
+      "loss": 0.7974,
+      "step": 311
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 0.3296041419119035,
+      "learning_rate": 0.0,
+      "loss": 0.7362,
+      "step": 312
+    },
+    {
+      "epoch": 0.9984,
+      "step": 312,
+      "total_flos": 730661894488064.0,
+      "train_loss": 0.8330854674180349,
+      "train_runtime": 9213.2243,
+      "train_samples_per_second": 1.085,
+      "train_steps_per_second": 0.034
+    }
+  ],
+  "logging_steps": 1.0,
+  "max_steps": 312,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": false,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 730661894488064.0,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_2_lora/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ./weights/Bunny-v1_1-Llama-3-8B-V
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_2_lora/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "up_proj",
+    "o_proj",
+    "q_proj",
+    "down_proj",
+    "v_proj",
+    "k_proj",
+    "gate_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_2_lora/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb223c3820e4d3ba11a4de32d2e8233430f8ee290e3539d12aefaa3e3db8d9e5
+size 671150064

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_2_lora/config.json

@@ -0,0 +1,45 @@
+{
+  "_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "architectures": [
+    "BunnyLlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_bunny_llama.BunnyLlamaConfig",
+    "AutoModelForCausalLM": "modeling_bunny_llama.BunnyLlamaForCausalLM"
+  },
+  "bos_token_id": 128000,
+  "continuous_training": false,
+  "eos_token_id": 128001,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect_ratio": "pad",
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 8192,
+  "mm_hidden_size": 3456,
+  "mm_projector_lr": null,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_vision_tower": "./weights/siglip-so400m-patch14-384",
+  "model_type": "bunny-llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 2048,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "float16",
+  "transformers_version": "4.41.2",
+  "tune_mm_mlp_adapter": false,
+  "unfreeze_vision_tower": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_s2": true,
+  "vocab_size": 128256
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_2_lora/non_lora_trainables.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a0dab0cefd97719e632dbe34186b207b24721cd439d644851337f06d4b9e2aa
+size 918507402

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_4_lora/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ./weights/Bunny-v1_1-Llama-3-8B-V
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_4_lora/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "up_proj",
+    "k_proj",
+    "o_proj",
+    "v_proj",
+    "down_proj",
+    "gate_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_4_lora/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b97c5d7de491d336dcc2b1c913eec57bc2ade51a9e0214178292b044b106f4bf
+size 671150064

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_4_lora/config.json

@@ -0,0 +1,45 @@
+{
+  "_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "architectures": [
+    "BunnyLlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_bunny_llama.BunnyLlamaConfig",
+    "AutoModelForCausalLM": "modeling_bunny_llama.BunnyLlamaForCausalLM"
+  },
+  "bos_token_id": 128000,
+  "continuous_training": false,
+  "eos_token_id": 128001,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect_ratio": "pad",
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 8192,
+  "mm_hidden_size": 3456,
+  "mm_projector_lr": null,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_vision_tower": "./weights/siglip-so400m-patch14-384",
+  "model_type": "bunny-llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 2048,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "float16",
+  "transformers_version": "4.41.2",
+  "tune_mm_mlp_adapter": false,
+  "unfreeze_vision_tower": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_s2": true,
+  "vocab_size": 128256
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_4_lora/non_lora_trainables.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:510b7354cbdb44580b8fcff4c0cba0358329aad7b5f7e48dcb45928b558fe28f
+size 918507402

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_2_epochs_1_GA_4_lora/trainer_state.json

@@ -0,0 +1,2226 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9984,
+  "eval_steps": 500,
+  "global_step": 312,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0032,
+      "grad_norm": 0.8210982159609682,
+      "learning_rate": 2e-05,
+      "loss": 1.3329,
+      "step": 1
+    },
+    {
+      "epoch": 0.0064,
+      "grad_norm": 0.7751787218518346,
+      "learning_rate": 4e-05,
+      "loss": 1.3721,
+      "step": 2
+    },
+    {
+      "epoch": 0.0096,
+      "grad_norm": 0.6646539221769208,
+      "learning_rate": 6e-05,
+      "loss": 1.3088,
+      "step": 3
+    },
+    {
+      "epoch": 0.0128,
+      "grad_norm": 0.7036195246304789,
+      "learning_rate": 8e-05,
+      "loss": 1.3377,
+      "step": 4
+    },
+    {
+      "epoch": 0.016,
+      "grad_norm": 0.6168233543536914,
+      "learning_rate": 0.0001,
+      "loss": 1.1919,
+      "step": 5
+    },
+    {
+      "epoch": 0.0192,
+      "grad_norm": 0.8492868526153039,
+      "learning_rate": 0.00012,
+      "loss": 1.2512,
+      "step": 6
+    },
+    {
+      "epoch": 0.0224,
+      "grad_norm": 0.7936611575617178,
+      "learning_rate": 0.00014,
+      "loss": 1.0697,
+      "step": 7
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 0.5862560941923312,
+      "learning_rate": 0.00016,
+      "loss": 1.0911,
+      "step": 8
+    },
+    {
+      "epoch": 0.0288,
+      "grad_norm": 0.49328071163938675,
+      "learning_rate": 0.00018,
+      "loss": 0.9762,
+      "step": 9
+    },
+    {
+      "epoch": 0.032,
+      "grad_norm": 0.4796953108328627,
+      "learning_rate": 0.0002,
+      "loss": 0.9269,
+      "step": 10
+    },
+    {
+      "epoch": 0.0352,
+      "grad_norm": 0.47431010608538954,
+      "learning_rate": 0.00019999458931878073,
+      "loss": 1.032,
+      "step": 11
+    },
+    {
+      "epoch": 0.0384,
+      "grad_norm": 0.4707062597281157,
+      "learning_rate": 0.0001999783578606323,
+      "loss": 0.9748,
+      "step": 12
+    },
+    {
+      "epoch": 0.0416,
+      "grad_norm": 0.4880961566615289,
+      "learning_rate": 0.00019995130738201966,
+      "loss": 0.9767,
+      "step": 13
+    },
+    {
+      "epoch": 0.0448,
+      "grad_norm": 0.4658440438656818,
+      "learning_rate": 0.0001999134408101731,
+      "loss": 0.9179,
+      "step": 14
+    },
+    {
+      "epoch": 0.048,
+      "grad_norm": 0.5344941086238363,
+      "learning_rate": 0.00019986476224277165,
+      "loss": 1.0129,
+      "step": 15
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 0.5007192463357719,
+      "learning_rate": 0.00019980527694749952,
+      "loss": 0.9953,
+      "step": 16
+    },
+    {
+      "epoch": 0.0544,
+      "grad_norm": 0.44861347739015633,
+      "learning_rate": 0.00019973499136147606,
+      "loss": 0.8987,
+      "step": 17
+    },
+    {
+      "epoch": 0.0576,
+      "grad_norm": 0.40432515099975497,
+      "learning_rate": 0.0001996539130905593,
+      "loss": 0.9949,
+      "step": 18
+    },
+    {
+      "epoch": 0.0608,
+      "grad_norm": 0.42728242685352247,
+      "learning_rate": 0.0001995620509085228,
+      "loss": 0.9649,
+      "step": 19
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 0.4641229091681979,
+      "learning_rate": 0.00019945941475610623,
+      "loss": 0.9609,
+      "step": 20
+    },
+    {
+      "epoch": 0.0672,
+      "grad_norm": 0.3913009193318844,
+      "learning_rate": 0.0001993460157399396,
+      "loss": 0.8997,
+      "step": 21
+    },
+    {
+      "epoch": 0.0704,
+      "grad_norm": 0.3960221279711468,
+      "learning_rate": 0.0001992218661313415,
+      "loss": 0.9046,
+      "step": 22
+    },
+    {
+      "epoch": 0.0736,
+      "grad_norm": 0.4893549306818581,
+      "learning_rate": 0.00019908697936499103,
+      "loss": 0.9538,
+      "step": 23
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 0.3943436800759236,
+      "learning_rate": 0.00019894137003747403,
+      "loss": 0.85,
+      "step": 24
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 0.49281687959276665,
+      "learning_rate": 0.00019878505390570362,
+      "loss": 0.8904,
+      "step": 25
+    },
+    {
+      "epoch": 0.0832,
+      "grad_norm": 0.45164645594942804,
+      "learning_rate": 0.00019861804788521493,
+      "loss": 0.9793,
+      "step": 26
+    },
+    {
+      "epoch": 0.0864,
+      "grad_norm": 0.3884822532677797,
+      "learning_rate": 0.00019844037004833473,
+      "loss": 0.8047,
+      "step": 27
+    },
+    {
+      "epoch": 0.0896,
+      "grad_norm": 0.41101776336511836,
+      "learning_rate": 0.00019825203962222572,
+      "loss": 0.9077,
+      "step": 28
+    },
+    {
+      "epoch": 0.0928,
+      "grad_norm": 1.3086028478450786,
+      "learning_rate": 0.0001980530769868059,
+      "loss": 0.9363,
+      "step": 29
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 0.3375555003941169,
+      "learning_rate": 0.00019784350367254322,
+      "loss": 0.8523,
+      "step": 30
+    },
+    {
+      "epoch": 0.0992,
+      "grad_norm": 0.4356929720245981,
+      "learning_rate": 0.0001976233423581255,
+      "loss": 0.9504,
+      "step": 31
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 0.4558256010915256,
+      "learning_rate": 0.0001973926168680066,
+      "loss": 0.9619,
+      "step": 32
+    },
+    {
+      "epoch": 0.1056,
+      "grad_norm": 0.4182066475020067,
+      "learning_rate": 0.00019715135216982798,
+      "loss": 0.853,
+      "step": 33
+    },
+    {
+      "epoch": 0.1088,
+      "grad_norm": 0.42581671871982196,
+      "learning_rate": 0.0001968995743717171,
+      "loss": 0.9528,
+      "step": 34
+    },
+    {
+      "epoch": 0.112,
+      "grad_norm": 0.39591765873363666,
+      "learning_rate": 0.00019663731071946206,
+      "loss": 0.9005,
+      "step": 35
+    },
+    {
+      "epoch": 0.1152,
+      "grad_norm": 0.39119354380322896,
+      "learning_rate": 0.00019636458959356316,
+      "loss": 0.8988,
+      "step": 36
+    },
+    {
+      "epoch": 0.1184,
+      "grad_norm": 0.40090914965423335,
+      "learning_rate": 0.0001960814405061619,
+      "loss": 0.8758,
+      "step": 37
+    },
+    {
+      "epoch": 0.1216,
+      "grad_norm": 0.3945361491838484,
+      "learning_rate": 0.00019578789409784727,
+      "loss": 0.9393,
+      "step": 38
+    },
+    {
+      "epoch": 0.1248,
+      "grad_norm": 0.4394341724704068,
+      "learning_rate": 0.00019548398213434007,
+      "loss": 0.9428,
+      "step": 39
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 0.3727456121947463,
+      "learning_rate": 0.00019516973750305532,
+      "loss": 0.8748,
+      "step": 40
+    },
+    {
+      "epoch": 0.1312,
+      "grad_norm": 0.37234199586270694,
+      "learning_rate": 0.00019484519420954354,
+      "loss": 0.8736,
+      "step": 41
+    },
+    {
+      "epoch": 0.1344,
+      "grad_norm": 0.39842513634466276,
+      "learning_rate": 0.00019451038737381077,
+      "loss": 0.9022,
+      "step": 42
+    },
+    {
+      "epoch": 0.1376,
+      "grad_norm": 0.38128602501461806,
+      "learning_rate": 0.00019416535322651818,
+      "loss": 0.8208,
+      "step": 43
+    },
+    {
+      "epoch": 0.1408,
+      "grad_norm": 0.38025302062603994,
+      "learning_rate": 0.00019381012910506146,
+      "loss": 0.8832,
+      "step": 44
+    },
+    {
+      "epoch": 0.144,
+      "grad_norm": 0.34279246176384087,
+      "learning_rate": 0.00019344475344953012,
+      "loss": 0.79,
+      "step": 45
+    },
+    {
+      "epoch": 0.1472,
+      "grad_norm": 0.39891015144038994,
+      "learning_rate": 0.00019306926579854821,
+      "loss": 0.8637,
+      "step": 46
+    },
+    {
+      "epoch": 0.1504,
+      "grad_norm": 0.33599563185761405,
+      "learning_rate": 0.00019268370678499533,
+      "loss": 0.7904,
+      "step": 47
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 0.405639307195739,
+      "learning_rate": 0.0001922881181316097,
+      "loss": 0.8904,
+      "step": 48
+    },
+    {
+      "epoch": 0.1568,
+      "grad_norm": 0.347008173492179,
+      "learning_rate": 0.00019188254264647337,
+      "loss": 0.8188,
+      "step": 49
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.4053373760955314,
+      "learning_rate": 0.0001914670242183795,
+      "loss": 0.8767,
+      "step": 50
+    },
+    {
+      "epoch": 0.1632,
+      "grad_norm": 0.4071076068532092,
+      "learning_rate": 0.0001910416078120832,
+      "loss": 0.8753,
+      "step": 51
+    },
+    {
+      "epoch": 0.1664,
+      "grad_norm": 0.3947573279192353,
+      "learning_rate": 0.0001906063394634356,
+      "loss": 0.9461,
+      "step": 52
+    },
+    {
+      "epoch": 0.1696,
+      "grad_norm": 0.41593394777293125,
+      "learning_rate": 0.00019016126627440237,
+      "loss": 0.8543,
+      "step": 53
+    },
+    {
+      "epoch": 0.1728,
+      "grad_norm": 0.34556265770047007,
+      "learning_rate": 0.00018970643640796642,
+      "loss": 0.8457,
+      "step": 54
+    },
+    {
+      "epoch": 0.176,
+      "grad_norm": 0.4135992491712443,
+      "learning_rate": 0.000189241899082916,
+      "loss": 0.8471,
+      "step": 55
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 0.36656186910259925,
+      "learning_rate": 0.00018876770456851877,
+      "loss": 0.8315,
+      "step": 56
+    },
+    {
+      "epoch": 0.1824,
+      "grad_norm": 0.41888747530592474,
+      "learning_rate": 0.0001882839041790818,
+      "loss": 0.9118,
+      "step": 57
+    },
+    {
+      "epoch": 0.1856,
+      "grad_norm": 0.36251863953206737,
+      "learning_rate": 0.00018779055026839868,
+      "loss": 0.8626,
+      "step": 58
+    },
+    {
+      "epoch": 0.1888,
+      "grad_norm": 0.3748079918860433,
+      "learning_rate": 0.00018728769622408423,
+      "loss": 0.9207,
+      "step": 59
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 0.34699582513061866,
+      "learning_rate": 0.00018677539646179707,
+      "loss": 0.7823,
+      "step": 60
+    },
+    {
+      "epoch": 0.1952,
+      "grad_norm": 0.4324436996517203,
+      "learning_rate": 0.00018625370641935129,
+      "loss": 0.9012,
+      "step": 61
+    },
+    {
+      "epoch": 0.1984,
+      "grad_norm": 0.40830183945729825,
+      "learning_rate": 0.00018572268255071718,
+      "loss": 0.8866,
+      "step": 62
+    },
+    {
+      "epoch": 0.2016,
+      "grad_norm": 0.35645475792133624,
+      "learning_rate": 0.00018518238231991218,
+      "loss": 0.8246,
+      "step": 63
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 0.39501413519318324,
+      "learning_rate": 0.00018463286419478255,
+      "loss": 0.8903,
+      "step": 64
+    },
+    {
+      "epoch": 0.208,
+      "grad_norm": 0.3403250694314037,
+      "learning_rate": 0.00018407418764067627,
+      "loss": 0.8002,
+      "step": 65
+    },
+    {
+      "epoch": 0.2112,
+      "grad_norm": 0.38578492015884985,
+      "learning_rate": 0.00018350641311400812,
+      "loss": 0.8823,
+      "step": 66
+    },
+    {
+      "epoch": 0.2144,
+      "grad_norm": 0.42912996214481597,
+      "learning_rate": 0.0001829296020557174,
+      "loss": 0.9337,
+      "step": 67
+    },
+    {
+      "epoch": 0.2176,
+      "grad_norm": 0.33840309416398107,
+      "learning_rate": 0.00018234381688461942,
+      "loss": 0.8152,
+      "step": 68
+    },
+    {
+      "epoch": 0.2208,
+      "grad_norm": 0.36261675945692257,
+      "learning_rate": 0.0001817491209906506,
+      "loss": 0.8119,
+      "step": 69
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 0.3978630764800173,
+      "learning_rate": 0.00018114557872800905,
+      "loss": 0.8884,
+      "step": 70
+    },
+    {
+      "epoch": 0.2272,
+      "grad_norm": 0.4067690308983075,
+      "learning_rate": 0.00018053325540819045,
+      "loss": 0.888,
+      "step": 71
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 0.39761946908218565,
+      "learning_rate": 0.0001799122172929206,
+      "loss": 0.8734,
+      "step": 72
+    },
+    {
+      "epoch": 0.2336,
+      "grad_norm": 0.416514669804446,
+      "learning_rate": 0.00017928253158698473,
+      "loss": 0.8813,
+      "step": 73
+    },
+    {
+      "epoch": 0.2368,
+      "grad_norm": 0.3831820092966067,
+      "learning_rate": 0.0001786442664309554,
+      "loss": 0.8015,
+      "step": 74
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.40531349166429315,
+      "learning_rate": 0.0001779974908938184,
+      "loss": 0.8021,
+      "step": 75
+    },
+    {
+      "epoch": 0.2432,
+      "grad_norm": 0.3757725806826467,
+      "learning_rate": 0.0001773422749654988,
+      "loss": 0.7819,
+      "step": 76
+    },
+    {
+      "epoch": 0.2464,
+      "grad_norm": 0.3973182683957758,
+      "learning_rate": 0.00017667868954928694,
+      "loss": 0.8807,
+      "step": 77
+    },
+    {
+      "epoch": 0.2496,
+      "grad_norm": 0.3251584668348043,
+      "learning_rate": 0.00017600680645416583,
+      "loss": 0.7627,
+      "step": 78
+    },
+    {
+      "epoch": 0.2528,
+      "grad_norm": 0.41127123846488717,
+      "learning_rate": 0.00017532669838704035,
+      "loss": 0.8387,
+      "step": 79
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 0.37661663005517965,
+      "learning_rate": 0.00017463843894486937,
+      "loss": 0.8423,
+      "step": 80
+    },
+    {
+      "epoch": 0.2592,
+      "grad_norm": 0.5786386678521719,
+      "learning_rate": 0.0001739421026067017,
+      "loss": 0.8949,
+      "step": 81
+    },
+    {
+      "epoch": 0.2624,
+      "grad_norm": 0.362252109033277,
+      "learning_rate": 0.00017323776472561627,
+      "loss": 0.8644,
+      "step": 82
+    },
+    {
+      "epoch": 0.2656,
+      "grad_norm": 0.3786074251460458,
+      "learning_rate": 0.00017252550152056795,
+      "loss": 0.8161,
+      "step": 83
+    },
+    {
+      "epoch": 0.2688,
+      "grad_norm": 0.38704773419959004,
+      "learning_rate": 0.0001718053900681397,
+      "loss": 0.8054,
+      "step": 84
+    },
+    {
+      "epoch": 0.272,
+      "grad_norm": 0.36687924172159914,
+      "learning_rate": 0.00017107750829420176,
+      "loss": 0.8053,
+      "step": 85
+    },
+    {
+      "epoch": 0.2752,
+      "grad_norm": 0.4532905914552837,
+      "learning_rate": 0.00017034193496547902,
+      "loss": 0.8496,
+      "step": 86
+    },
+    {
+      "epoch": 0.2784,
+      "grad_norm": 0.3868107415971005,
+      "learning_rate": 0.00016959874968102735,
+      "loss": 0.8292,
+      "step": 87
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 0.45004745878752395,
+      "learning_rate": 0.00016884803286362,
+      "loss": 0.8386,
+      "step": 88
+    },
+    {
+      "epoch": 0.2848,
+      "grad_norm": 0.3601950439013075,
+      "learning_rate": 0.00016808986575104465,
+      "loss": 0.8144,
+      "step": 89
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 0.39414164051104805,
+      "learning_rate": 0.00016732433038731242,
+      "loss": 0.7563,
+      "step": 90
+    },
+    {
+      "epoch": 0.2912,
+      "grad_norm": 0.4162252396854745,
+      "learning_rate": 0.0001665515096137797,
+      "loss": 0.79,
+      "step": 91
+    },
+    {
+      "epoch": 0.2944,
+      "grad_norm": 0.37960017989400824,
+      "learning_rate": 0.00016577148706018328,
+      "loss": 0.8201,
+      "step": 92
+    },
+    {
+      "epoch": 0.2976,
+      "grad_norm": 0.3838649671026415,
+      "learning_rate": 0.00016498434713559088,
+      "loss": 0.8362,
+      "step": 93
+    },
+    {
+      "epoch": 0.3008,
+      "grad_norm": 0.43789279418131016,
+      "learning_rate": 0.00016419017501926656,
+      "loss": 0.8057,
+      "step": 94
+    },
+    {
+      "epoch": 0.304,
+      "grad_norm": 0.33824523290379066,
+      "learning_rate": 0.0001633890566514535,
+      "loss": 0.7832,
+      "step": 95
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 0.38508464229871187,
+      "learning_rate": 0.00016258107872407375,
+      "loss": 0.8825,
+      "step": 96
+    },
+    {
+      "epoch": 0.3104,
+      "grad_norm": 0.3872280278580832,
+      "learning_rate": 0.0001617663286713474,
+      "loss": 0.8568,
+      "step": 97
+    },
+    {
+      "epoch": 0.3136,
+      "grad_norm": 0.3710845404668928,
+      "learning_rate": 0.00016094489466033043,
+      "loss": 0.8433,
+      "step": 98
+    },
+    {
+      "epoch": 0.3168,
+      "grad_norm": 0.3601850097818311,
+      "learning_rate": 0.00016011686558137448,
+      "loss": 0.8426,
+      "step": 99
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 0.36336268281281814,
+      "learning_rate": 0.0001592823310385073,
+      "loss": 0.8459,
+      "step": 100
+    },
+    {
+      "epoch": 0.3232,
+      "grad_norm": 0.4007311050026622,
+      "learning_rate": 0.0001584413813397364,
+      "loss": 0.8332,
+      "step": 101
+    },
+    {
+      "epoch": 0.3264,
+      "grad_norm": 0.3279742689663737,
+      "learning_rate": 0.00015759410748727662,
+      "loss": 0.7898,
+      "step": 102
+    },
+    {
+      "epoch": 0.3296,
+      "grad_norm": 0.3937884919475543,
+      "learning_rate": 0.00015674060116770236,
+      "loss": 0.8439,
+      "step": 103
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 0.4143083432212905,
+      "learning_rate": 0.00015588095474202595,
+      "loss": 0.8246,
+      "step": 104
+    },
+    {
+      "epoch": 0.336,
+      "grad_norm": 0.35676836384855826,
+      "learning_rate": 0.00015501526123570277,
+      "loss": 0.8377,
+      "step": 105
+    },
+    {
+      "epoch": 0.3392,
+      "grad_norm": 0.4418839450505997,
+      "learning_rate": 0.00015414361432856475,
+      "loss": 0.8546,
+      "step": 106
+    },
+    {
+      "epoch": 0.3424,
+      "grad_norm": 0.38722861928111846,
+      "learning_rate": 0.0001532661083446829,
+      "loss": 0.8938,
+      "step": 107
+    },
+    {
+      "epoch": 0.3456,
+      "grad_norm": 0.4798810748731012,
+      "learning_rate": 0.00015238283824216015,
+      "loss": 0.933,
+      "step": 108
+    },
+    {
+      "epoch": 0.3488,
+      "grad_norm": 0.4192610184794806,
+      "learning_rate": 0.00015149389960285558,
+      "loss": 0.8314,
+      "step": 109
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 0.37992908787234797,
+      "learning_rate": 0.00015059938862204127,
+      "loss": 0.8502,
+      "step": 110
+    },
+    {
+      "epoch": 0.3552,
+      "grad_norm": 0.3537079585046425,
+      "learning_rate": 0.00014969940209799248,
+      "loss": 0.837,
+      "step": 111
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 0.4947879619869325,
+      "learning_rate": 0.00014879403742151283,
+      "loss": 0.8655,
+      "step": 112
+    },
+    {
+      "epoch": 0.3616,
+      "grad_norm": 0.4144273342987794,
+      "learning_rate": 0.00014788339256539544,
+      "loss": 0.8024,
+      "step": 113
+    },
+    {
+      "epoch": 0.3648,
+      "grad_norm": 0.42810201253462243,
+      "learning_rate": 0.0001469675660738206,
+      "loss": 0.8646,
+      "step": 114
+    },
+    {
+      "epoch": 0.368,
+      "grad_norm": 0.43495581513946346,
+      "learning_rate": 0.00014604665705169237,
+      "loss": 0.8019,
+      "step": 115
+    },
+    {
+      "epoch": 0.3712,
+      "grad_norm": 0.40514388555481956,
+      "learning_rate": 0.00014512076515391375,
+      "loss": 0.8307,
+      "step": 116
+    },
+    {
+      "epoch": 0.3744,
+      "grad_norm": 0.5344793506502119,
+      "learning_rate": 0.00014418999057460276,
+      "loss": 0.8583,
+      "step": 117
+    },
+    {
+      "epoch": 0.3776,
+      "grad_norm": 0.3591362856868415,
+      "learning_rate": 0.0001432544340362501,
+      "loss": 0.8422,
+      "step": 118
+    },
+    {
+      "epoch": 0.3808,
+      "grad_norm": 0.43800340514585073,
+      "learning_rate": 0.00014231419677881966,
+      "loss": 0.829,
+      "step": 119
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 0.3546231000833001,
+      "learning_rate": 0.00014136938054879283,
+      "loss": 0.8502,
+      "step": 120
+    },
+    {
+      "epoch": 0.3872,
+      "grad_norm": 0.3824038358260862,
+      "learning_rate": 0.00014042008758815818,
+      "loss": 0.8601,
+      "step": 121
+    },
+    {
+      "epoch": 0.3904,
+      "grad_norm": 0.3773025369439645,
+      "learning_rate": 0.00013946642062334766,
+      "loss": 0.7483,
+      "step": 122
+    },
+    {
+      "epoch": 0.3936,
+      "grad_norm": 0.40841295198664107,
+      "learning_rate": 0.00013850848285411994,
+      "loss": 0.8981,
+      "step": 123
+    },
+    {
+      "epoch": 0.3968,
+      "grad_norm": 0.32297152126914114,
+      "learning_rate": 0.000137546377942393,
+      "loss": 0.7604,
+      "step": 124
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.38908595736780166,
+      "learning_rate": 0.00013658021000102636,
+      "loss": 0.8377,
+      "step": 125
+    },
+    {
+      "epoch": 0.4032,
+      "grad_norm": 0.30749072492921053,
+      "learning_rate": 0.00013561008358255468,
+      "loss": 0.6953,
+      "step": 126
+    },
+    {
+      "epoch": 0.4064,
+      "grad_norm": 0.42279066677539,
+      "learning_rate": 0.00013463610366787392,
+      "loss": 0.8381,
+      "step": 127
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 0.46835688525644387,
+      "learning_rate": 0.00013365837565488064,
+      "loss": 0.8801,
+      "step": 128
+    },
+    {
+      "epoch": 0.4128,
+      "grad_norm": 0.38170959986063163,
+      "learning_rate": 0.0001326770053470668,
+      "loss": 0.8177,
+      "step": 129
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 0.3748992249270009,
+      "learning_rate": 0.0001316920989420703,
+      "loss": 0.7318,
+      "step": 130
+    },
+    {
+      "epoch": 0.4192,
+      "grad_norm": 0.32117252663093104,
+      "learning_rate": 0.00013070376302018287,
+      "loss": 0.7911,
+      "step": 131
+    },
+    {
+      "epoch": 0.4224,
+      "grad_norm": 0.32643252918835663,
+      "learning_rate": 0.00012971210453281674,
+      "loss": 0.7565,
+      "step": 132
+    },
+    {
+      "epoch": 0.4256,
+      "grad_norm": 0.34707082290937796,
+      "learning_rate": 0.000128717230790931,
+      "loss": 0.8179,
+      "step": 133
+    },
+    {
+      "epoch": 0.4288,
+      "grad_norm": 0.37747643787259155,
+      "learning_rate": 0.00012771924945341906,
+      "loss": 0.7647,
+      "step": 134
+    },
+    {
+      "epoch": 0.432,
+      "grad_norm": 0.4042549495002011,
+      "learning_rate": 0.00012671826851545851,
+      "loss": 0.8638,
+      "step": 135
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 0.40682592869078416,
+      "learning_rate": 0.0001257143962968246,
+      "loss": 0.9235,
+      "step": 136
+    },
+    {
+      "epoch": 0.4384,
+      "grad_norm": 0.3394181019470551,
+      "learning_rate": 0.00012470774143016853,
+      "loss": 0.8029,
+      "step": 137
+    },
+    {
+      "epoch": 0.4416,
+      "grad_norm": 0.37379683333378905,
+      "learning_rate": 0.00012369841284926188,
+      "loss": 0.8183,
+      "step": 138
+    },
+    {
+      "epoch": 0.4448,
+      "grad_norm": 0.37760782532653564,
+      "learning_rate": 0.00012268651977720866,
+      "loss": 0.732,
+      "step": 139
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 0.42926915905667945,
+      "learning_rate": 0.00012167217171462566,
+      "loss": 0.8645,
+      "step": 140
+    },
+    {
+      "epoch": 0.4512,
+      "grad_norm": 0.3353578654728262,
+      "learning_rate": 0.0001206554784277931,
+      "loss": 0.7573,
+      "step": 141
+    },
+    {
+      "epoch": 0.4544,
+      "grad_norm": 0.36702732140537525,
+      "learning_rate": 0.00011963654993677645,
+      "loss": 0.7855,
+      "step": 142
+    },
+    {
+      "epoch": 0.4576,
+      "grad_norm": 0.3945829288474564,
+      "learning_rate": 0.00011861549650352069,
+      "loss": 0.9015,
+      "step": 143
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 0.3384991496539021,
+      "learning_rate": 0.00011759242861991855,
+      "loss": 0.7672,
+      "step": 144
+    },
+    {
+      "epoch": 0.464,
+      "grad_norm": 0.36235446616831335,
+      "learning_rate": 0.00011656745699585371,
+      "loss": 0.7298,
+      "step": 145
+    },
+    {
+      "epoch": 0.4672,
+      "grad_norm": 0.37275162386027966,
+      "learning_rate": 0.00011554069254722051,
+      "loss": 0.8231,
+      "step": 146
+    },
+    {
+      "epoch": 0.4704,
+      "grad_norm": 0.3351601172488379,
+      "learning_rate": 0.00011451224638392129,
+      "loss": 0.78,
+      "step": 147
+    },
+    {
+      "epoch": 0.4736,
+      "grad_norm": 0.4177215304468739,
+      "learning_rate": 0.00011348222979784289,
+      "loss": 0.8148,
+      "step": 148
+    },
+    {
+      "epoch": 0.4768,
+      "grad_norm": 0.31882281813401364,
+      "learning_rate": 0.00011245075425081328,
+      "loss": 0.8256,
+      "step": 149
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.34927524739208016,
+      "learning_rate": 0.00011141793136253986,
+      "loss": 0.7206,
+      "step": 150
+    },
+    {
+      "epoch": 0.4832,
+      "grad_norm": 0.3272143865687314,
+      "learning_rate": 0.0001103838728985307,
+      "loss": 0.8422,
+      "step": 151
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 0.38082561425167516,
+      "learning_rate": 0.000109348690758,
+      "loss": 0.801,
+      "step": 152
+    },
+    {
+      "epoch": 0.4896,
+      "grad_norm": 0.3683618334975772,
+      "learning_rate": 0.00010831249696175918,
+      "loss": 0.754,
+      "step": 153
+    },
+    {
+      "epoch": 0.4928,
+      "grad_norm": 0.3460529686559446,
+      "learning_rate": 0.0001072754036400944,
+      "loss": 0.7703,
+      "step": 154
+    },
+    {
+      "epoch": 0.496,
+      "grad_norm": 0.3906059155144969,
+      "learning_rate": 0.00010623752302063283,
+      "loss": 0.8635,
+      "step": 155
+    },
+    {
+      "epoch": 0.4992,
+      "grad_norm": 0.4498300529094466,
+      "learning_rate": 0.00010519896741619803,
+      "loss": 0.887,
+      "step": 156
+    },
+    {
+      "epoch": 0.5024,
+      "grad_norm": 0.34890284462378934,
+      "learning_rate": 0.00010415984921265609,
+      "loss": 0.8415,
+      "step": 157
+    },
+    {
+      "epoch": 0.5056,
+      "grad_norm": 0.35283761184137985,
+      "learning_rate": 0.00010312028085675391,
+      "loss": 0.7897,
+      "step": 158
+    },
+    {
+      "epoch": 0.5088,
+      "grad_norm": 0.3383893586354043,
+      "learning_rate": 0.00010208037484395114,
+      "loss": 0.7699,
+      "step": 159
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 0.37944587951981884,
+      "learning_rate": 0.00010104024370624644,
+      "loss": 0.8168,
+      "step": 160
+    },
+    {
+      "epoch": 0.5152,
+      "grad_norm": 0.35361574612426283,
+      "learning_rate": 0.0001,
+      "loss": 0.8237,
+      "step": 161
+    },
+    {
+      "epoch": 0.5184,
+      "grad_norm": 0.49601519414630724,
+      "learning_rate": 9.895975629375359e-05,
+      "loss": 0.8944,
+      "step": 162
+    },
+    {
+      "epoch": 0.5216,
+      "grad_norm": 0.4329854234900553,
+      "learning_rate": 9.791962515604887e-05,
+      "loss": 0.8409,
+      "step": 163
+    },
+    {
+      "epoch": 0.5248,
+      "grad_norm": 0.33753405466010045,
+      "learning_rate": 9.687971914324607e-05,
+      "loss": 0.6823,
+      "step": 164
+    },
+    {
+      "epoch": 0.528,
+      "grad_norm": 0.3641968735452542,
+      "learning_rate": 9.584015078734395e-05,
+      "loss": 0.7999,
+      "step": 165
+    },
+    {
+      "epoch": 0.5312,
+      "grad_norm": 0.3412252144703986,
+      "learning_rate": 9.480103258380198e-05,
+      "loss": 0.7485,
+      "step": 166
+    },
+    {
+      "epoch": 0.5344,
+      "grad_norm": 0.36557961816073514,
+      "learning_rate": 9.376247697936719e-05,
+      "loss": 0.8047,
+      "step": 167
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 0.3143242351094908,
+      "learning_rate": 9.272459635990562e-05,
+      "loss": 0.7417,
+      "step": 168
+    },
+    {
+      "epoch": 0.5408,
+      "grad_norm": 0.37740381793108085,
+      "learning_rate": 9.168750303824084e-05,
+      "loss": 0.7847,
+      "step": 169
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 0.3756001432572221,
+      "learning_rate": 9.065130924199998e-05,
+      "loss": 0.8045,
+      "step": 170
+    },
+    {
+      "epoch": 0.5472,
+      "grad_norm": 0.3886531783560117,
+      "learning_rate": 8.961612710146934e-05,
+      "loss": 0.7739,
+      "step": 171
+    },
+    {
+      "epoch": 0.5504,
+      "grad_norm": 0.35961818663851197,
+      "learning_rate": 8.858206863746018e-05,
+      "loss": 0.7682,
+      "step": 172
+    },
+    {
+      "epoch": 0.5536,
+      "grad_norm": 0.34291998146787495,
+      "learning_rate": 8.754924574918675e-05,
+      "loss": 0.8154,
+      "step": 173
+    },
+    {
+      "epoch": 0.5568,
+      "grad_norm": 0.365623213677616,
+      "learning_rate": 8.651777020215712e-05,
+      "loss": 0.7723,
+      "step": 174
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.4154693198085254,
+      "learning_rate": 8.548775361607872e-05,
+      "loss": 0.8059,
+      "step": 175
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 0.41853675355977066,
+      "learning_rate": 8.445930745277953e-05,
+      "loss": 0.7936,
+      "step": 176
+    },
+    {
+      "epoch": 0.5664,
+      "grad_norm": 0.3029988536971098,
+      "learning_rate": 8.343254300414628e-05,
+      "loss": 0.7538,
+      "step": 177
+    },
+    {
+      "epoch": 0.5696,
+      "grad_norm": 0.35015284331125907,
+      "learning_rate": 8.240757138008149e-05,
+      "loss": 0.784,
+      "step": 178
+    },
+    {
+      "epoch": 0.5728,
+      "grad_norm": 0.36899119805196373,
+      "learning_rate": 8.138450349647936e-05,
+      "loss": 0.7712,
+      "step": 179
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 0.4635909482238713,
+      "learning_rate": 8.036345006322359e-05,
+      "loss": 0.8151,
+      "step": 180
+    },
+    {
+      "epoch": 0.5792,
+      "grad_norm": 0.39324815381124656,
+      "learning_rate": 7.934452157220694e-05,
+      "loss": 0.8479,
+      "step": 181
+    },
+    {
+      "epoch": 0.5824,
+      "grad_norm": 0.3790446343866871,
+      "learning_rate": 7.832782828537437e-05,
+      "loss": 0.7849,
+      "step": 182
+    },
+    {
+      "epoch": 0.5856,
+      "grad_norm": 0.38585647860183997,
+      "learning_rate": 7.731348022279134e-05,
+      "loss": 0.8062,
+      "step": 183
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 0.3851371620272845,
+      "learning_rate": 7.630158715073813e-05,
+      "loss": 0.8341,
+      "step": 184
+    },
+    {
+      "epoch": 0.592,
+      "grad_norm": 0.33007054542330805,
+      "learning_rate": 7.52922585698315e-05,
+      "loss": 0.7197,
+      "step": 185
+    },
+    {
+      "epoch": 0.5952,
+      "grad_norm": 0.42174584227102946,
+      "learning_rate": 7.428560370317542e-05,
+      "loss": 0.8017,
+      "step": 186
+    },
+    {
+      "epoch": 0.5984,
+      "grad_norm": 0.33067818645649066,
+      "learning_rate": 7.328173148454151e-05,
+      "loss": 0.7886,
+      "step": 187
+    },
+    {
+      "epoch": 0.6016,
+      "grad_norm": 0.42265554001971417,
+      "learning_rate": 7.228075054658096e-05,
+      "loss": 0.8238,
+      "step": 188
+    },
+    {
+      "epoch": 0.6048,
+      "grad_norm": 0.34908037692220767,
+      "learning_rate": 7.1282769209069e-05,
+      "loss": 0.7928,
+      "step": 189
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 0.3700495310823468,
+      "learning_rate": 7.028789546718326e-05,
+      "loss": 0.7491,
+      "step": 190
+    },
+    {
+      "epoch": 0.6112,
+      "grad_norm": 0.3535456489525407,
+      "learning_rate": 6.929623697981718e-05,
+      "loss": 0.8094,
+      "step": 191
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 0.35992816828147584,
+      "learning_rate": 6.830790105792973e-05,
+      "loss": 0.747,
+      "step": 192
+    },
+    {
+      "epoch": 0.6176,
+      "grad_norm": 0.34357552011988995,
+      "learning_rate": 6.732299465293322e-05,
+      "loss": 0.7329,
+      "step": 193
+    },
+    {
+      "epoch": 0.6208,
+      "grad_norm": 0.4017091190478587,
+      "learning_rate": 6.63416243451194e-05,
+      "loss": 0.807,
+      "step": 194
+    },
+    {
+      "epoch": 0.624,
+      "grad_norm": 0.3092472351482309,
+      "learning_rate": 6.536389633212609e-05,
+      "loss": 0.7309,
+      "step": 195
+    },
+    {
+      "epoch": 0.6272,
+      "grad_norm": 0.3737027071808558,
+      "learning_rate": 6.43899164174453e-05,
+      "loss": 0.8191,
+      "step": 196
+    },
+    {
+      "epoch": 0.6304,
+      "grad_norm": 0.36607238095868505,
+      "learning_rate": 6.341978999897365e-05,
+      "loss": 0.7998,
+      "step": 197
+    },
+    {
+      "epoch": 0.6336,
+      "grad_norm": 0.4042496034179054,
+      "learning_rate": 6.245362205760704e-05,
+      "loss": 0.773,
+      "step": 198
+    },
+    {
+      "epoch": 0.6368,
+      "grad_norm": 0.37764639494375285,
+      "learning_rate": 6.149151714588009e-05,
+      "loss": 0.8055,
+      "step": 199
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.3348761239337774,
+      "learning_rate": 6.053357937665237e-05,
+      "loss": 0.7443,
+      "step": 200
+    },
+    {
+      "epoch": 0.6432,
+      "grad_norm": 0.37626992146366794,
+      "learning_rate": 5.957991241184184e-05,
+      "loss": 0.7826,
+      "step": 201
+    },
+    {
+      "epoch": 0.6464,
+      "grad_norm": 0.3665662199313722,
+      "learning_rate": 5.863061945120719e-05,
+      "loss": 0.8335,
+      "step": 202
+    },
+    {
+      "epoch": 0.6496,
+      "grad_norm": 0.33785931924439677,
+      "learning_rate": 5.768580322118034e-05,
+      "loss": 0.7972,
+      "step": 203
+    },
+    {
+      "epoch": 0.6528,
+      "grad_norm": 0.3880113242436853,
+      "learning_rate": 5.6745565963749925e-05,
+      "loss": 0.8896,
+      "step": 204
+    },
+    {
+      "epoch": 0.656,
+      "grad_norm": 0.3721493819848271,
+      "learning_rate": 5.5810009425397294e-05,
+      "loss": 0.8183,
+      "step": 205
+    },
+    {
+      "epoch": 0.6592,
+      "grad_norm": 0.5198646512996709,
+      "learning_rate": 5.487923484608629e-05,
+      "loss": 0.8566,
+      "step": 206
+    },
+    {
+      "epoch": 0.6624,
+      "grad_norm": 0.4065329950447661,
+      "learning_rate": 5.395334294830765e-05,
+      "loss": 0.9147,
+      "step": 207
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 0.37859745509497567,
+      "learning_rate": 5.3032433926179395e-05,
+      "loss": 0.8042,
+      "step": 208
+    },
+    {
+      "epoch": 0.6688,
+      "grad_norm": 0.3607765773764099,
+      "learning_rate": 5.211660743460458e-05,
+      "loss": 0.7938,
+      "step": 209
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 0.4092950714526401,
+      "learning_rate": 5.1205962578487155e-05,
+      "loss": 0.806,
+      "step": 210
+    },
+    {
+      "epoch": 0.6752,
+      "grad_norm": 0.30432228901089703,
+      "learning_rate": 5.030059790200756e-05,
+      "loss": 0.7262,
+      "step": 211
+    },
+    {
+      "epoch": 0.6784,
+      "grad_norm": 0.37088903131596684,
+      "learning_rate": 4.940061137795876e-05,
+      "loss": 0.7169,
+      "step": 212
+    },
+    {
+      "epoch": 0.6816,
+      "grad_norm": 0.3904681317398879,
+      "learning_rate": 4.850610039714444e-05,
+      "loss": 0.8104,
+      "step": 213
+    },
+    {
+      "epoch": 0.6848,
+      "grad_norm": 0.46898322895625755,
+      "learning_rate": 4.761716175783989e-05,
+      "loss": 0.8688,
+      "step": 214
+    },
+    {
+      "epoch": 0.688,
+      "grad_norm": 0.3006038153490141,
+      "learning_rate": 4.673389165531714e-05,
+      "loss": 0.7169,
+      "step": 215
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 0.37810250308131216,
+      "learning_rate": 4.585638567143529e-05,
+      "loss": 0.789,
+      "step": 216
+    },
+    {
+      "epoch": 0.6944,
+      "grad_norm": 0.2989111481815409,
+      "learning_rate": 4.498473876429726e-05,
+      "loss": 0.7303,
+      "step": 217
+    },
+    {
+      "epoch": 0.6976,
+      "grad_norm": 0.32911143249763547,
+      "learning_rate": 4.411904525797408e-05,
+      "loss": 0.7592,
+      "step": 218
+    },
+    {
+      "epoch": 0.7008,
+      "grad_norm": 0.35549746840897417,
+      "learning_rate": 4.325939883229766e-05,
+      "loss": 0.8175,
+      "step": 219
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 0.31577828848903333,
+      "learning_rate": 4.240589251272342e-05,
+      "loss": 0.7581,
+      "step": 220
+    },
+    {
+      "epoch": 0.7072,
+      "grad_norm": 0.38150936709065336,
+      "learning_rate": 4.155861866026364e-05,
+      "loss": 0.7383,
+      "step": 221
+    },
+    {
+      "epoch": 0.7104,
+      "grad_norm": 0.352873952674236,
+      "learning_rate": 4.071766896149273e-05,
+      "loss": 0.7899,
+      "step": 222
+    },
+    {
+      "epoch": 0.7136,
+      "grad_norm": 0.3729395002008322,
+      "learning_rate": 3.988313441862553e-05,
+      "loss": 0.8267,
+      "step": 223
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 0.38612103435747785,
+      "learning_rate": 3.9055105339669595e-05,
+      "loss": 0.8438,
+      "step": 224
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.3343773300394658,
+      "learning_rate": 3.823367132865265e-05,
+      "loss": 0.7766,
+      "step": 225
+    },
+    {
+      "epoch": 0.7232,
+      "grad_norm": 0.34821484213233017,
+      "learning_rate": 3.741892127592625e-05,
+      "loss": 0.7869,
+      "step": 226
+    },
+    {
+      "epoch": 0.7264,
+      "grad_norm": 0.34452187805524936,
+      "learning_rate": 3.6610943348546526e-05,
+      "loss": 0.8142,
+      "step": 227
+    },
+    {
+      "epoch": 0.7296,
+      "grad_norm": 0.389690433062319,
+      "learning_rate": 3.580982498073344e-05,
+      "loss": 0.75,
+      "step": 228
+    },
+    {
+      "epoch": 0.7328,
+      "grad_norm": 0.4645823764551201,
+      "learning_rate": 3.501565286440914e-05,
+      "loss": 0.9086,
+      "step": 229
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 0.3374398934171241,
+      "learning_rate": 3.422851293981676e-05,
+      "loss": 0.7671,
+      "step": 230
+    },
+    {
+      "epoch": 0.7392,
+      "grad_norm": 0.4111564099874274,
+      "learning_rate": 3.3448490386220355e-05,
+      "loss": 0.83,
+      "step": 231
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 0.4082259005237418,
+      "learning_rate": 3.2675669612687565e-05,
+      "loss": 0.8499,
+      "step": 232
+    },
+    {
+      "epoch": 0.7456,
+      "grad_norm": 0.35262858393716395,
+      "learning_rate": 3.191013424895536e-05,
+      "loss": 0.8093,
+      "step": 233
+    },
+    {
+      "epoch": 0.7488,
+      "grad_norm": 0.41355266235382226,
+      "learning_rate": 3.115196713638e-05,
+      "loss": 0.8087,
+      "step": 234
+    },
+    {
+      "epoch": 0.752,
+      "grad_norm": 0.41241448924179946,
+      "learning_rate": 3.040125031897264e-05,
+      "loss": 0.7911,
+      "step": 235
+    },
+    {
+      "epoch": 0.7552,
+      "grad_norm": 0.3279954208792422,
+      "learning_rate": 2.9658065034520978e-05,
+      "loss": 0.7616,
+      "step": 236
+    },
+    {
+      "epoch": 0.7584,
+      "grad_norm": 0.3710790783951858,
+      "learning_rate": 2.892249170579826e-05,
+      "loss": 0.7417,
+      "step": 237
+    },
+    {
+      "epoch": 0.7616,
+      "grad_norm": 0.3638956295243486,
+      "learning_rate": 2.8194609931860316e-05,
+      "loss": 0.7734,
+      "step": 238
+    },
+    {
+      "epoch": 0.7648,
+      "grad_norm": 0.45340861210629174,
+      "learning_rate": 2.7474498479432087e-05,
+      "loss": 0.845,
+      "step": 239
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 0.32432848532039027,
+      "learning_rate": 2.6762235274383772e-05,
+      "loss": 0.7292,
+      "step": 240
+    },
+    {
+      "epoch": 0.7712,
+      "grad_norm": 0.33285800890422707,
+      "learning_rate": 2.6057897393298324e-05,
+      "loss": 0.6951,
+      "step": 241
+    },
+    {
+      "epoch": 0.7744,
+      "grad_norm": 0.36168512959628546,
+      "learning_rate": 2.536156105513062e-05,
+      "loss": 0.7777,
+      "step": 242
+    },
+    {
+      "epoch": 0.7776,
+      "grad_norm": 0.3755110018820757,
+      "learning_rate": 2.4673301612959654e-05,
+      "loss": 0.7907,
+      "step": 243
+    },
+    {
+      "epoch": 0.7808,
+      "grad_norm": 0.4746969071550545,
+      "learning_rate": 2.399319354583418e-05,
+      "loss": 0.8731,
+      "step": 244
+    },
+    {
+      "epoch": 0.784,
+      "grad_norm": 0.3485421714780257,
+      "learning_rate": 2.3321310450713062e-05,
+      "loss": 0.7017,
+      "step": 245
+    },
+    {
+      "epoch": 0.7872,
+      "grad_norm": 0.3522895092881672,
+      "learning_rate": 2.265772503450122e-05,
+      "loss": 0.7341,
+      "step": 246
+    },
+    {
+      "epoch": 0.7904,
+      "grad_norm": 0.3171714951862727,
+      "learning_rate": 2.2002509106181624e-05,
+      "loss": 0.778,
+      "step": 247
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 0.30854770168466894,
+      "learning_rate": 2.1355733569044635e-05,
+      "loss": 0.7326,
+      "step": 248
+    },
+    {
+      "epoch": 0.7968,
+      "grad_norm": 0.4491439626115905,
+      "learning_rate": 2.0717468413015283e-05,
+      "loss": 0.8296,
+      "step": 249
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.37090037131233355,
+      "learning_rate": 2.008778270707944e-05,
+      "loss": 0.7953,
+      "step": 250
+    },
+    {
+      "epoch": 0.8032,
+      "grad_norm": 0.3834261067678261,
+      "learning_rate": 1.946674459180955e-05,
+      "loss": 0.9055,
+      "step": 251
+    },
+    {
+      "epoch": 0.8064,
+      "grad_norm": 0.3255432432056942,
+      "learning_rate": 1.8854421271990964e-05,
+      "loss": 0.7592,
+      "step": 252
+    },
+    {
+      "epoch": 0.8096,
+      "grad_norm": 0.38151180353083647,
+      "learning_rate": 1.8250879009349398e-05,
+      "loss": 0.7883,
+      "step": 253
+    },
+    {
+      "epoch": 0.8128,
+      "grad_norm": 0.43340018294551996,
+      "learning_rate": 1.7656183115380577e-05,
+      "loss": 0.9072,
+      "step": 254
+    },
+    {
+      "epoch": 0.816,
+      "grad_norm": 0.4692009486980658,
+      "learning_rate": 1.707039794428259e-05,
+      "loss": 0.8892,
+      "step": 255
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 0.34906122041207605,
+      "learning_rate": 1.649358688599191e-05,
+      "loss": 0.7177,
+      "step": 256
+    },
+    {
+      "epoch": 0.8224,
+      "grad_norm": 0.357075923035099,
+      "learning_rate": 1.5925812359323745e-05,
+      "loss": 0.7832,
+      "step": 257
+    },
+    {
+      "epoch": 0.8256,
+      "grad_norm": 0.3853853663460861,
+      "learning_rate": 1.5367135805217458e-05,
+      "loss": 0.7875,
+      "step": 258
+    },
+    {
+      "epoch": 0.8288,
+      "grad_norm": 0.3162997323586251,
+      "learning_rate": 1.4817617680087825e-05,
+      "loss": 0.7652,
+      "step": 259
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 0.5058370501510021,
+      "learning_rate": 1.4277317449282834e-05,
+      "loss": 0.788,
+      "step": 260
+    },
+    {
+      "epoch": 0.8352,
+      "grad_norm": 0.38934979473108083,
+      "learning_rate": 1.3746293580648717e-05,
+      "loss": 0.8643,
+      "step": 261
+    },
+    {
+      "epoch": 0.8384,
+      "grad_norm": 0.36714032393813667,
+      "learning_rate": 1.3224603538202929e-05,
+      "loss": 0.7469,
+      "step": 262
+    },
+    {
+      "epoch": 0.8416,
+      "grad_norm": 0.3488248102724531,
+      "learning_rate": 1.2712303775915802e-05,
+      "loss": 0.7318,
+      "step": 263
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 0.294650221806305,
+      "learning_rate": 1.220944973160133e-05,
+      "loss": 0.7344,
+      "step": 264
+    },
+    {
+      "epoch": 0.848,
+      "grad_norm": 0.3540730290394561,
+      "learning_rate": 1.1716095820918216e-05,
+      "loss": 0.7354,
+      "step": 265
+    },
+    {
+      "epoch": 0.8512,
+      "grad_norm": 0.4226203562757057,
+      "learning_rate": 1.1232295431481222e-05,
+      "loss": 0.808,
+      "step": 266
+    },
+    {
+      "epoch": 0.8544,
+      "grad_norm": 0.4014616869239377,
+      "learning_rate": 1.0758100917083991e-05,
+      "loss": 0.7749,
+      "step": 267
+    },
+    {
+      "epoch": 0.8576,
+      "grad_norm": 0.3934215620886321,
+      "learning_rate": 1.0293563592033595e-05,
+      "loss": 0.8003,
+      "step": 268
+    },
+    {
+      "epoch": 0.8608,
+      "grad_norm": 0.3353659014526165,
+      "learning_rate": 9.838733725597615e-06,
+      "loss": 0.7851,
+      "step": 269
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 0.3700720016155327,
+      "learning_rate": 9.393660536564408e-06,
+      "loss": 0.7941,
+      "step": 270
+    },
+    {
+      "epoch": 0.8672,
+      "grad_norm": 0.31100504311573046,
+      "learning_rate": 8.958392187916841e-06,
+      "loss": 0.7298,
+      "step": 271
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 0.4172712953645561,
+      "learning_rate": 8.532975781620512e-06,
+      "loss": 0.7686,
+      "step": 272
+    },
+    {
+      "epoch": 0.8736,
+      "grad_norm": 0.34667441316708114,
+      "learning_rate": 8.117457353526625e-06,
+      "loss": 0.7106,
+      "step": 273
+    },
+    {
+      "epoch": 0.8768,
+      "grad_norm": 0.3004518824396042,
+      "learning_rate": 7.711881868390291e-06,
+      "loss": 0.6887,
+      "step": 274
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 0.38133602184118887,
+      "learning_rate": 7.3162932150046885e-06,
+      "loss": 0.8172,
+      "step": 275
+    },
+    {
+      "epoch": 0.8832,
+      "grad_norm": 0.33116090594086295,
+      "learning_rate": 6.930734201451816e-06,
+      "loss": 0.7005,
+      "step": 276
+    },
+    {
+      "epoch": 0.8864,
+      "grad_norm": 0.3188298094332972,
+      "learning_rate": 6.555246550469907e-06,
+      "loss": 0.7534,
+      "step": 277
+    },
+    {
+      "epoch": 0.8896,
+      "grad_norm": 0.4403999231950908,
+      "learning_rate": 6.189870894938587e-06,
+      "loss": 0.8213,
+      "step": 278
+    },
+    {
+      "epoch": 0.8928,
+      "grad_norm": 0.3608087635662593,
+      "learning_rate": 5.834646773481811e-06,
+      "loss": 0.7844,
+      "step": 279
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 0.40474388045501003,
+      "learning_rate": 5.489612626189245e-06,
+      "loss": 0.8244,
+      "step": 280
+    },
+    {
+      "epoch": 0.8992,
+      "grad_norm": 0.3389637770265784,
+      "learning_rate": 5.154805790456485e-06,
+      "loss": 0.7018,
+      "step": 281
+    },
+    {
+      "epoch": 0.9024,
+      "grad_norm": 0.30643971186231883,
+      "learning_rate": 4.830262496944693e-06,
+      "loss": 0.7619,
+      "step": 282
+    },
+    {
+      "epoch": 0.9056,
+      "grad_norm": 0.3661184834317572,
+      "learning_rate": 4.516017865659949e-06,
+      "loss": 0.703,
+      "step": 283
+    },
+    {
+      "epoch": 0.9088,
+      "grad_norm": 0.3671129154556767,
+      "learning_rate": 4.21210590215273e-06,
+      "loss": 0.7824,
+      "step": 284
+    },
+    {
+      "epoch": 0.912,
+      "grad_norm": 0.34883381196841734,
+      "learning_rate": 3.918559493838114e-06,
+      "loss": 0.8148,
+      "step": 285
+    },
+    {
+      "epoch": 0.9152,
+      "grad_norm": 0.32439361737968153,
+      "learning_rate": 3.6354104064368566e-06,
+      "loss": 0.7575,
+      "step": 286
+    },
+    {
+      "epoch": 0.9184,
+      "grad_norm": 0.3866839936796524,
+      "learning_rate": 3.3626892805379562e-06,
+      "loss": 0.8134,
+      "step": 287
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 0.3825547694903366,
+      "learning_rate": 3.100425628282899e-06,
+      "loss": 0.7618,
+      "step": 288
+    },
+    {
+      "epoch": 0.9248,
+      "grad_norm": 0.31827546790247363,
+      "learning_rate": 2.848647830172024e-06,
+      "loss": 0.7321,
+      "step": 289
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 0.29623028045498556,
+      "learning_rate": 2.607383131993424e-06,
+      "loss": 0.6688,
+      "step": 290
+    },
+    {
+      "epoch": 0.9312,
+      "grad_norm": 0.40429655628990896,
+      "learning_rate": 2.3766576418745022e-06,
+      "loss": 0.8106,
+      "step": 291
+    },
+    {
+      "epoch": 0.9344,
+      "grad_norm": 0.3650008962489634,
+      "learning_rate": 2.1564963274568027e-06,
+      "loss": 0.7758,
+      "step": 292
+    },
+    {
+      "epoch": 0.9376,
+      "grad_norm": 0.34654671660631003,
+      "learning_rate": 1.9469230131940907e-06,
+      "loss": 0.7488,
+      "step": 293
+    },
+    {
+      "epoch": 0.9408,
+      "grad_norm": 0.3417601789633415,
+      "learning_rate": 1.7479603777742938e-06,
+      "loss": 0.7376,
+      "step": 294
+    },
+    {
+      "epoch": 0.944,
+      "grad_norm": 0.4766922892989375,
+      "learning_rate": 1.559629951665298e-06,
+      "loss": 0.8448,
+      "step": 295
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 0.3764754677513324,
+      "learning_rate": 1.3819521147851123e-06,
+      "loss": 0.756,
+      "step": 296
+    },
+    {
+      "epoch": 0.9504,
+      "grad_norm": 0.3744144624463219,
+      "learning_rate": 1.2149460942964098e-06,
+      "loss": 0.7769,
+      "step": 297
+    },
+    {
+      "epoch": 0.9536,
+      "grad_norm": 0.41513981451373816,
+      "learning_rate": 1.05862996252597e-06,
+      "loss": 0.7734,
+      "step": 298
+    },
+    {
+      "epoch": 0.9568,
+      "grad_norm": 0.36679556444218603,
+      "learning_rate": 9.130206350089765e-07,
+      "loss": 0.7664,
+      "step": 299
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 0.3859580111548229,
+      "learning_rate": 7.781338686584927e-07,
+      "loss": 0.7639,
+      "step": 300
+    },
+    {
+      "epoch": 0.9632,
+      "grad_norm": 0.34312029909834285,
+      "learning_rate": 6.539842600603918e-07,
+      "loss": 0.8412,
+      "step": 301
+    },
+    {
+      "epoch": 0.9664,
+      "grad_norm": 0.4680442538250724,
+      "learning_rate": 5.405852438937764e-07,
+      "loss": 0.8093,
+      "step": 302
+    },
+    {
+      "epoch": 0.9696,
+      "grad_norm": 0.3160450323270329,
+      "learning_rate": 4.3794909147720773e-07,
+      "loss": 0.709,
+      "step": 303
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 0.45915080458361,
+      "learning_rate": 3.4608690944071263e-07,
+      "loss": 0.8064,
+      "step": 304
+    },
+    {
+      "epoch": 0.976,
+      "grad_norm": 0.3737969883986542,
+      "learning_rate": 2.6500863852395584e-07,
+      "loss": 0.8277,
+      "step": 305
+    },
+    {
+      "epoch": 0.9792,
+      "grad_norm": 0.3789680381062684,
+      "learning_rate": 1.947230525005006e-07,
+      "loss": 0.8404,
+      "step": 306
+    },
+    {
+      "epoch": 0.9824,
+      "grad_norm": 0.35233074082938165,
+      "learning_rate": 1.3523775722834587e-07,
+      "loss": 0.7446,
+      "step": 307
+    },
+    {
+      "epoch": 0.9856,
+      "grad_norm": 0.3337846639484774,
+      "learning_rate": 8.655918982689581e-08,
+      "loss": 0.7811,
+      "step": 308
+    },
+    {
+      "epoch": 0.9888,
+      "grad_norm": 0.35005314677512966,
+      "learning_rate": 4.8692617980350406e-08,
+      "loss": 0.8336,
+      "step": 309
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 0.38107841486064986,
+      "learning_rate": 2.164213936770576e-08,
+      "loss": 0.768,
+      "step": 310
+    },
+    {
+      "epoch": 0.9952,
+      "grad_norm": 0.37188047064663365,
+      "learning_rate": 5.410681219286673e-09,
+      "loss": 0.7323,
+      "step": 311
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 0.3528962453383034,
+      "learning_rate": 0.0,
+      "loss": 0.7924,
+      "step": 312
+    },
+    {
+      "epoch": 0.9984,
+      "step": 312,
+      "total_flos": 736497759813632.0,
+      "train_loss": 0.8298983348485751,
+      "train_runtime": 9149.0946,
+      "train_samples_per_second": 1.093,
+      "train_steps_per_second": 0.034
+    }
+  ],
+  "logging_steps": 1.0,
+  "max_steps": 312,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": false,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 736497759813632.0,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_2_lora/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ./weights/Bunny-v1_1-Llama-3-8B-V
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_2_lora/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "up_proj",
+    "k_proj",
+    "down_proj",
+    "o_proj",
+    "gate_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_2_lora/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61659b7548812b91a870a85aa48869803be2f9ca23408f3b764320bcca5a24cc
+size 671150064

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_2_lora/config.json

@@ -0,0 +1,45 @@
+{
+  "_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "architectures": [
+    "BunnyLlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_bunny_llama.BunnyLlamaConfig",
+    "AutoModelForCausalLM": "modeling_bunny_llama.BunnyLlamaForCausalLM"
+  },
+  "bos_token_id": 128000,
+  "continuous_training": false,
+  "eos_token_id": 128001,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect_ratio": "pad",
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 8192,
+  "mm_hidden_size": 3456,
+  "mm_projector_lr": null,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_vision_tower": "./weights/siglip-so400m-patch14-384",
+  "model_type": "bunny-llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 2048,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "float16",
+  "transformers_version": "4.41.2",
+  "tune_mm_mlp_adapter": false,
+  "unfreeze_vision_tower": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_s2": true,
+  "vocab_size": 128256
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_2_lora/non_lora_trainables.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf545e27d23f76953d9bdd19703edae1ee5f1f56973e863e6006b8a69c875de6
+size 918507402

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_4_lora/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ./weights/Bunny-v1_1-Llama-3-8B-V
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_4_lora/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "up_proj",
+    "q_proj",
+    "o_proj",
+    "v_proj",
+    "down_proj",
+    "gate_proj",
+    "k_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_4_lora/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60e78440e8ad5c216e1bbb491649fb73d8e9423283d6831d816658f013ddea60
+size 671150064

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_4_lora/config.json

@@ -0,0 +1,45 @@
+{
+  "_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "architectures": [
+    "BunnyLlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_bunny_llama.BunnyLlamaConfig",
+    "AutoModelForCausalLM": "modeling_bunny_llama.BunnyLlamaForCausalLM"
+  },
+  "bos_token_id": 128000,
+  "continuous_training": false,
+  "eos_token_id": 128001,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect_ratio": "pad",
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 8192,
+  "mm_hidden_size": 3456,
+  "mm_projector_lr": null,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_vision_tower": "./weights/siglip-so400m-patch14-384",
+  "model_type": "bunny-llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 2048,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "float16",
+  "transformers_version": "4.41.2",
+  "tune_mm_mlp_adapter": false,
+  "unfreeze_vision_tower": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_s2": true,
+  "vocab_size": 128256
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_4_lora/non_lora_trainables.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d89c8f41acde27e42e9a8a3b4e1df3f1d80d5199b474eab8c9904c815c8b6d0
+size 918507402

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_10000_repeat_3_epochs_1_GA_4_lora/trainer_state.json

@@ -0,0 +1,2226 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9984,
+  "eval_steps": 500,
+  "global_step": 312,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0032,
+      "grad_norm": 0.7954520844133007,
+      "learning_rate": 2e-05,
+      "loss": 1.3544,
+      "step": 1
+    },
+    {
+      "epoch": 0.0064,
+      "grad_norm": 0.8486736733468345,
+      "learning_rate": 4e-05,
+      "loss": 1.4258,
+      "step": 2
+    },
+    {
+      "epoch": 0.0096,
+      "grad_norm": 0.7231129434225938,
+      "learning_rate": 6e-05,
+      "loss": 1.2565,
+      "step": 3
+    },
+    {
+      "epoch": 0.0128,
+      "grad_norm": 0.6549525568483539,
+      "learning_rate": 8e-05,
+      "loss": 1.301,
+      "step": 4
+    },
+    {
+      "epoch": 0.016,
+      "grad_norm": 0.6713937782391836,
+      "learning_rate": 0.0001,
+      "loss": 1.1783,
+      "step": 5
+    },
+    {
+      "epoch": 0.0192,
+      "grad_norm": 0.7775279811022765,
+      "learning_rate": 0.00012,
+      "loss": 1.2018,
+      "step": 6
+    },
+    {
+      "epoch": 0.0224,
+      "grad_norm": 0.8210500566974728,
+      "learning_rate": 0.00014,
+      "loss": 1.0322,
+      "step": 7
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 0.582072473996548,
+      "learning_rate": 0.00016,
+      "loss": 1.0719,
+      "step": 8
+    },
+    {
+      "epoch": 0.0288,
+      "grad_norm": 0.5332524469921555,
+      "learning_rate": 0.00018,
+      "loss": 0.9937,
+      "step": 9
+    },
+    {
+      "epoch": 0.032,
+      "grad_norm": 0.4652344399947701,
+      "learning_rate": 0.0002,
+      "loss": 0.9582,
+      "step": 10
+    },
+    {
+      "epoch": 0.0352,
+      "grad_norm": 0.528479459724629,
+      "learning_rate": 0.00019999458931878073,
+      "loss": 1.0025,
+      "step": 11
+    },
+    {
+      "epoch": 0.0384,
+      "grad_norm": 0.48862298453038183,
+      "learning_rate": 0.0001999783578606323,
+      "loss": 1.0045,
+      "step": 12
+    },
+    {
+      "epoch": 0.0416,
+      "grad_norm": 0.4788245246121141,
+      "learning_rate": 0.00019995130738201966,
+      "loss": 0.9681,
+      "step": 13
+    },
+    {
+      "epoch": 0.0448,
+      "grad_norm": 0.44036289873008344,
+      "learning_rate": 0.0001999134408101731,
+      "loss": 0.9215,
+      "step": 14
+    },
+    {
+      "epoch": 0.048,
+      "grad_norm": 0.5449869090192805,
+      "learning_rate": 0.00019986476224277165,
+      "loss": 0.9593,
+      "step": 15
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 0.4237521722332184,
+      "learning_rate": 0.00019980527694749952,
+      "loss": 0.8736,
+      "step": 16
+    },
+    {
+      "epoch": 0.0544,
+      "grad_norm": 0.4168713872622836,
+      "learning_rate": 0.00019973499136147606,
+      "loss": 0.8863,
+      "step": 17
+    },
+    {
+      "epoch": 0.0576,
+      "grad_norm": 0.45264982727147834,
+      "learning_rate": 0.0001996539130905593,
+      "loss": 0.8696,
+      "step": 18
+    },
+    {
+      "epoch": 0.0608,
+      "grad_norm": 0.3929576698684051,
+      "learning_rate": 0.0001995620509085228,
+      "loss": 0.916,
+      "step": 19
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 0.4415540009570294,
+      "learning_rate": 0.00019945941475610623,
+      "loss": 0.9198,
+      "step": 20
+    },
+    {
+      "epoch": 0.0672,
+      "grad_norm": 0.3812265501303348,
+      "learning_rate": 0.0001993460157399396,
+      "loss": 0.8966,
+      "step": 21
+    },
+    {
+      "epoch": 0.0704,
+      "grad_norm": 0.4047740643644689,
+      "learning_rate": 0.0001992218661313415,
+      "loss": 0.9509,
+      "step": 22
+    },
+    {
+      "epoch": 0.0736,
+      "grad_norm": 0.394670597027155,
+      "learning_rate": 0.00019908697936499103,
+      "loss": 0.8582,
+      "step": 23
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 0.40883800080500177,
+      "learning_rate": 0.00019894137003747403,
+      "loss": 0.8873,
+      "step": 24
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 0.4945237937729557,
+      "learning_rate": 0.00019878505390570362,
+      "loss": 0.9728,
+      "step": 25
+    },
+    {
+      "epoch": 0.0832,
+      "grad_norm": 0.4621016693300622,
+      "learning_rate": 0.00019861804788521493,
+      "loss": 0.9232,
+      "step": 26
+    },
+    {
+      "epoch": 0.0864,
+      "grad_norm": 0.6732021018661585,
+      "learning_rate": 0.00019844037004833473,
+      "loss": 0.8132,
+      "step": 27
+    },
+    {
+      "epoch": 0.0896,
+      "grad_norm": 0.4422468298010891,
+      "learning_rate": 0.00019825203962222572,
+      "loss": 0.9266,
+      "step": 28
+    },
+    {
+      "epoch": 0.0928,
+      "grad_norm": 0.40220173065598824,
+      "learning_rate": 0.0001980530769868059,
+      "loss": 0.8905,
+      "step": 29
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 0.36918169793645084,
+      "learning_rate": 0.00019784350367254322,
+      "loss": 0.8733,
+      "step": 30
+    },
+    {
+      "epoch": 0.0992,
+      "grad_norm": 0.5597457386366598,
+      "learning_rate": 0.0001976233423581255,
+      "loss": 0.9705,
+      "step": 31
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 0.5595809993867294,
+      "learning_rate": 0.0001973926168680066,
+      "loss": 0.9519,
+      "step": 32
+    },
+    {
+      "epoch": 0.1056,
+      "grad_norm": 0.4605324530129531,
+      "learning_rate": 0.00019715135216982798,
+      "loss": 0.895,
+      "step": 33
+    },
+    {
+      "epoch": 0.1088,
+      "grad_norm": 0.42815914241954167,
+      "learning_rate": 0.0001968995743717171,
+      "loss": 0.9781,
+      "step": 34
+    },
+    {
+      "epoch": 0.112,
+      "grad_norm": 0.4124008245937204,
+      "learning_rate": 0.00019663731071946206,
+      "loss": 0.8654,
+      "step": 35
+    },
+    {
+      "epoch": 0.1152,
+      "grad_norm": 0.4388119798381529,
+      "learning_rate": 0.00019636458959356316,
+      "loss": 0.9486,
+      "step": 36
+    },
+    {
+      "epoch": 0.1184,
+      "grad_norm": 0.41876874470531383,
+      "learning_rate": 0.0001960814405061619,
+      "loss": 0.8528,
+      "step": 37
+    },
+    {
+      "epoch": 0.1216,
+      "grad_norm": 0.4063934253065074,
+      "learning_rate": 0.00019578789409784727,
+      "loss": 0.9519,
+      "step": 38
+    },
+    {
+      "epoch": 0.1248,
+      "grad_norm": 0.4281987478958129,
+      "learning_rate": 0.00019548398213434007,
+      "loss": 0.931,
+      "step": 39
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 0.3995519502824548,
+      "learning_rate": 0.00019516973750305532,
+      "loss": 0.8706,
+      "step": 40
+    },
+    {
+      "epoch": 0.1312,
+      "grad_norm": 0.39938546863455104,
+      "learning_rate": 0.00019484519420954354,
+      "loss": 0.9081,
+      "step": 41
+    },
+    {
+      "epoch": 0.1344,
+      "grad_norm": 0.41849656692135495,
+      "learning_rate": 0.00019451038737381077,
+      "loss": 0.9316,
+      "step": 42
+    },
+    {
+      "epoch": 0.1376,
+      "grad_norm": 0.38704750788751674,
+      "learning_rate": 0.00019416535322651818,
+      "loss": 0.9122,
+      "step": 43
+    },
+    {
+      "epoch": 0.1408,
+      "grad_norm": 0.36713908717782284,
+      "learning_rate": 0.00019381012910506146,
+      "loss": 0.8173,
+      "step": 44
+    },
+    {
+      "epoch": 0.144,
+      "grad_norm": 0.386332629442072,
+      "learning_rate": 0.00019344475344953012,
+      "loss": 0.8054,
+      "step": 45
+    },
+    {
+      "epoch": 0.1472,
+      "grad_norm": 0.40175758635877834,
+      "learning_rate": 0.00019306926579854821,
+      "loss": 0.871,
+      "step": 46
+    },
+    {
+      "epoch": 0.1504,
+      "grad_norm": 0.3980262653132001,
+      "learning_rate": 0.00019268370678499533,
+      "loss": 0.864,
+      "step": 47
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 0.4340588852915137,
+      "learning_rate": 0.0001922881181316097,
+      "loss": 0.9131,
+      "step": 48
+    },
+    {
+      "epoch": 0.1568,
+      "grad_norm": 0.424408064468781,
+      "learning_rate": 0.00019188254264647337,
+      "loss": 0.8575,
+      "step": 49
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.42662287831550505,
+      "learning_rate": 0.0001914670242183795,
+      "loss": 0.8401,
+      "step": 50
+    },
+    {
+      "epoch": 0.1632,
+      "grad_norm": 0.4018922149491412,
+      "learning_rate": 0.0001910416078120832,
+      "loss": 0.8477,
+      "step": 51
+    },
+    {
+      "epoch": 0.1664,
+      "grad_norm": 0.4547104194477866,
+      "learning_rate": 0.0001906063394634356,
+      "loss": 0.9216,
+      "step": 52
+    },
+    {
+      "epoch": 0.1696,
+      "grad_norm": 0.46878889945327834,
+      "learning_rate": 0.00019016126627440237,
+      "loss": 0.8866,
+      "step": 53
+    },
+    {
+      "epoch": 0.1728,
+      "grad_norm": 0.3901101154924772,
+      "learning_rate": 0.00018970643640796642,
+      "loss": 0.8071,
+      "step": 54
+    },
+    {
+      "epoch": 0.176,
+      "grad_norm": 0.3786177626239962,
+      "learning_rate": 0.000189241899082916,
+      "loss": 0.8155,
+      "step": 55
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 0.37399747556550167,
+      "learning_rate": 0.00018876770456851877,
+      "loss": 0.8673,
+      "step": 56
+    },
+    {
+      "epoch": 0.1824,
+      "grad_norm": 0.4476713311081534,
+      "learning_rate": 0.0001882839041790818,
+      "loss": 0.8913,
+      "step": 57
+    },
+    {
+      "epoch": 0.1856,
+      "grad_norm": 0.3695813169482114,
+      "learning_rate": 0.00018779055026839868,
+      "loss": 0.8532,
+      "step": 58
+    },
+    {
+      "epoch": 0.1888,
+      "grad_norm": 0.3740852677452882,
+      "learning_rate": 0.00018728769622408423,
+      "loss": 0.851,
+      "step": 59
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 0.37805895790025723,
+      "learning_rate": 0.00018677539646179707,
+      "loss": 0.8893,
+      "step": 60
+    },
+    {
+      "epoch": 0.1952,
+      "grad_norm": 0.48550367409643636,
+      "learning_rate": 0.00018625370641935129,
+      "loss": 0.9271,
+      "step": 61
+    },
+    {
+      "epoch": 0.1984,
+      "grad_norm": 0.4261985087419851,
+      "learning_rate": 0.00018572268255071718,
+      "loss": 0.9485,
+      "step": 62
+    },
+    {
+      "epoch": 0.2016,
+      "grad_norm": 0.3506821138816665,
+      "learning_rate": 0.00018518238231991218,
+      "loss": 0.7993,
+      "step": 63
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 0.39926954240428475,
+      "learning_rate": 0.00018463286419478255,
+      "loss": 0.9094,
+      "step": 64
+    },
+    {
+      "epoch": 0.208,
+      "grad_norm": 0.369474669842929,
+      "learning_rate": 0.00018407418764067627,
+      "loss": 0.8639,
+      "step": 65
+    },
+    {
+      "epoch": 0.2112,
+      "grad_norm": 0.38412612507425553,
+      "learning_rate": 0.00018350641311400812,
+      "loss": 0.92,
+      "step": 66
+    },
+    {
+      "epoch": 0.2144,
+      "grad_norm": 0.6236483911477445,
+      "learning_rate": 0.0001829296020557174,
+      "loss": 0.9896,
+      "step": 67
+    },
+    {
+      "epoch": 0.2176,
+      "grad_norm": 0.40709892702353573,
+      "learning_rate": 0.00018234381688461942,
+      "loss": 0.7737,
+      "step": 68
+    },
+    {
+      "epoch": 0.2208,
+      "grad_norm": 0.38384691117683545,
+      "learning_rate": 0.0001817491209906506,
+      "loss": 0.8189,
+      "step": 69
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 0.419002960060546,
+      "learning_rate": 0.00018114557872800905,
+      "loss": 0.8892,
+      "step": 70
+    },
+    {
+      "epoch": 0.2272,
+      "grad_norm": 0.4317304658707876,
+      "learning_rate": 0.00018053325540819045,
+      "loss": 0.9425,
+      "step": 71
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 0.3605359976609917,
+      "learning_rate": 0.0001799122172929206,
+      "loss": 0.8432,
+      "step": 72
+    },
+    {
+      "epoch": 0.2336,
+      "grad_norm": 0.38387195033721394,
+      "learning_rate": 0.00017928253158698473,
+      "loss": 0.8127,
+      "step": 73
+    },
+    {
+      "epoch": 0.2368,
+      "grad_norm": 0.4115773141922566,
+      "learning_rate": 0.0001786442664309554,
+      "loss": 0.835,
+      "step": 74
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.41682944760695084,
+      "learning_rate": 0.0001779974908938184,
+      "loss": 0.8527,
+      "step": 75
+    },
+    {
+      "epoch": 0.2432,
+      "grad_norm": 0.44021561768781686,
+      "learning_rate": 0.0001773422749654988,
+      "loss": 0.8634,
+      "step": 76
+    },
+    {
+      "epoch": 0.2464,
+      "grad_norm": 0.3788641404846892,
+      "learning_rate": 0.00017667868954928694,
+      "loss": 0.8657,
+      "step": 77
+    },
+    {
+      "epoch": 0.2496,
+      "grad_norm": 0.35062726715426834,
+      "learning_rate": 0.00017600680645416583,
+      "loss": 0.776,
+      "step": 78
+    },
+    {
+      "epoch": 0.2528,
+      "grad_norm": 0.3640559161150146,
+      "learning_rate": 0.00017532669838704035,
+      "loss": 0.8121,
+      "step": 79
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 0.35673860083372294,
+      "learning_rate": 0.00017463843894486937,
+      "loss": 0.8013,
+      "step": 80
+    },
+    {
+      "epoch": 0.2592,
+      "grad_norm": 0.36719254075722085,
+      "learning_rate": 0.0001739421026067017,
+      "loss": 0.864,
+      "step": 81
+    },
+    {
+      "epoch": 0.2624,
+      "grad_norm": 0.39931819574163646,
+      "learning_rate": 0.00017323776472561627,
+      "loss": 0.8932,
+      "step": 82
+    },
+    {
+      "epoch": 0.2656,
+      "grad_norm": 0.3982022670626305,
+      "learning_rate": 0.00017252550152056795,
+      "loss": 0.8254,
+      "step": 83
+    },
+    {
+      "epoch": 0.2688,
+      "grad_norm": 0.3997476767736355,
+      "learning_rate": 0.0001718053900681397,
+      "loss": 0.8817,
+      "step": 84
+    },
+    {
+      "epoch": 0.272,
+      "grad_norm": 0.3463507690011195,
+      "learning_rate": 0.00017107750829420176,
+      "loss": 0.8348,
+      "step": 85
+    },
+    {
+      "epoch": 0.2752,
+      "grad_norm": 0.4200196272874818,
+      "learning_rate": 0.00017034193496547902,
+      "loss": 0.9056,
+      "step": 86
+    },
+    {
+      "epoch": 0.2784,
+      "grad_norm": 0.3755782720326349,
+      "learning_rate": 0.00016959874968102735,
+      "loss": 0.8419,
+      "step": 87
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 0.3981270184947585,
+      "learning_rate": 0.00016884803286362,
+      "loss": 0.8491,
+      "step": 88
+    },
+    {
+      "epoch": 0.2848,
+      "grad_norm": 0.37218427069658755,
+      "learning_rate": 0.00016808986575104465,
+      "loss": 0.9352,
+      "step": 89
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 0.3395579879759133,
+      "learning_rate": 0.00016732433038731242,
+      "loss": 0.7546,
+      "step": 90
+    },
+    {
+      "epoch": 0.2912,
+      "grad_norm": 0.3673793454918801,
+      "learning_rate": 0.0001665515096137797,
+      "loss": 0.833,
+      "step": 91
+    },
+    {
+      "epoch": 0.2944,
+      "grad_norm": 0.34780686537520417,
+      "learning_rate": 0.00016577148706018328,
+      "loss": 0.8021,
+      "step": 92
+    },
+    {
+      "epoch": 0.2976,
+      "grad_norm": 0.38425962561813065,
+      "learning_rate": 0.00016498434713559088,
+      "loss": 0.8679,
+      "step": 93
+    },
+    {
+      "epoch": 0.3008,
+      "grad_norm": 0.42936546302655365,
+      "learning_rate": 0.00016419017501926656,
+      "loss": 0.8377,
+      "step": 94
+    },
+    {
+      "epoch": 0.304,
+      "grad_norm": 0.4267179181570991,
+      "learning_rate": 0.0001633890566514535,
+      "loss": 0.8357,
+      "step": 95
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 0.36916905077663476,
+      "learning_rate": 0.00016258107872407375,
+      "loss": 0.8232,
+      "step": 96
+    },
+    {
+      "epoch": 0.3104,
+      "grad_norm": 0.38729223078060027,
+      "learning_rate": 0.0001617663286713474,
+      "loss": 0.8608,
+      "step": 97
+    },
+    {
+      "epoch": 0.3136,
+      "grad_norm": 0.4065510182299363,
+      "learning_rate": 0.00016094489466033043,
+      "loss": 0.8215,
+      "step": 98
+    },
+    {
+      "epoch": 0.3168,
+      "grad_norm": 0.35357551053351133,
+      "learning_rate": 0.00016011686558137448,
+      "loss": 0.8115,
+      "step": 99
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 1.0001838046966431,
+      "learning_rate": 0.0001592823310385073,
+      "loss": 0.8702,
+      "step": 100
+    },
+    {
+      "epoch": 0.3232,
+      "grad_norm": 0.40193375582947716,
+      "learning_rate": 0.0001584413813397364,
+      "loss": 0.8128,
+      "step": 101
+    },
+    {
+      "epoch": 0.3264,
+      "grad_norm": 0.3935235182421146,
+      "learning_rate": 0.00015759410748727662,
+      "loss": 0.7955,
+      "step": 102
+    },
+    {
+      "epoch": 0.3296,
+      "grad_norm": 0.6013285238613454,
+      "learning_rate": 0.00015674060116770236,
+      "loss": 0.8225,
+      "step": 103
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 0.3787137979507313,
+      "learning_rate": 0.00015588095474202595,
+      "loss": 0.8257,
+      "step": 104
+    },
+    {
+      "epoch": 0.336,
+      "grad_norm": 0.38425572159674526,
+      "learning_rate": 0.00015501526123570277,
+      "loss": 0.8475,
+      "step": 105
+    },
+    {
+      "epoch": 0.3392,
+      "grad_norm": 0.4006328412662351,
+      "learning_rate": 0.00015414361432856475,
+      "loss": 0.8648,
+      "step": 106
+    },
+    {
+      "epoch": 0.3424,
+      "grad_norm": 0.4086678609977253,
+      "learning_rate": 0.0001532661083446829,
+      "loss": 0.8863,
+      "step": 107
+    },
+    {
+      "epoch": 0.3456,
+      "grad_norm": 0.4881485864286447,
+      "learning_rate": 0.00015238283824216015,
+      "loss": 0.9356,
+      "step": 108
+    },
+    {
+      "epoch": 0.3488,
+      "grad_norm": 0.4025668507825546,
+      "learning_rate": 0.00015149389960285558,
+      "loss": 0.8127,
+      "step": 109
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 0.3710129691466552,
+      "learning_rate": 0.00015059938862204127,
+      "loss": 0.8428,
+      "step": 110
+    },
+    {
+      "epoch": 0.3552,
+      "grad_norm": 0.4003824576144442,
+      "learning_rate": 0.00014969940209799248,
+      "loss": 0.8425,
+      "step": 111
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 0.5018583001311245,
+      "learning_rate": 0.00014879403742151283,
+      "loss": 0.8779,
+      "step": 112
+    },
+    {
+      "epoch": 0.3616,
+      "grad_norm": 0.36121263099811374,
+      "learning_rate": 0.00014788339256539544,
+      "loss": 0.7713,
+      "step": 113
+    },
+    {
+      "epoch": 0.3648,
+      "grad_norm": 0.4251687064188239,
+      "learning_rate": 0.0001469675660738206,
+      "loss": 0.7802,
+      "step": 114
+    },
+    {
+      "epoch": 0.368,
+      "grad_norm": 0.3569667148399426,
+      "learning_rate": 0.00014604665705169237,
+      "loss": 0.8449,
+      "step": 115
+    },
+    {
+      "epoch": 0.3712,
+      "grad_norm": 0.4066427039698594,
+      "learning_rate": 0.00014512076515391375,
+      "loss": 0.8426,
+      "step": 116
+    },
+    {
+      "epoch": 0.3744,
+      "grad_norm": 0.44110171219803324,
+      "learning_rate": 0.00014418999057460276,
+      "loss": 0.8606,
+      "step": 117
+    },
+    {
+      "epoch": 0.3776,
+      "grad_norm": 0.37113662406191406,
+      "learning_rate": 0.0001432544340362501,
+      "loss": 0.8036,
+      "step": 118
+    },
+    {
+      "epoch": 0.3808,
+      "grad_norm": 0.36927548404036525,
+      "learning_rate": 0.00014231419677881966,
+      "loss": 0.7892,
+      "step": 119
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 0.3628801931909292,
+      "learning_rate": 0.00014136938054879283,
+      "loss": 0.8945,
+      "step": 120
+    },
+    {
+      "epoch": 0.3872,
+      "grad_norm": 0.3881852305317355,
+      "learning_rate": 0.00014042008758815818,
+      "loss": 0.8457,
+      "step": 121
+    },
+    {
+      "epoch": 0.3904,
+      "grad_norm": 0.3493063856843603,
+      "learning_rate": 0.00013946642062334766,
+      "loss": 0.8096,
+      "step": 122
+    },
+    {
+      "epoch": 0.3936,
+      "grad_norm": 0.4238637282608415,
+      "learning_rate": 0.00013850848285411994,
+      "loss": 0.8239,
+      "step": 123
+    },
+    {
+      "epoch": 0.3968,
+      "grad_norm": 0.35268751443414337,
+      "learning_rate": 0.000137546377942393,
+      "loss": 0.8348,
+      "step": 124
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.39291285524686237,
+      "learning_rate": 0.00013658021000102636,
+      "loss": 0.8241,
+      "step": 125
+    },
+    {
+      "epoch": 0.4032,
+      "grad_norm": 0.34506213306561334,
+      "learning_rate": 0.00013561008358255468,
+      "loss": 0.7677,
+      "step": 126
+    },
+    {
+      "epoch": 0.4064,
+      "grad_norm": 0.4203851880192952,
+      "learning_rate": 0.00013463610366787392,
+      "loss": 0.8461,
+      "step": 127
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 0.4474038709406084,
+      "learning_rate": 0.00013365837565488064,
+      "loss": 0.8768,
+      "step": 128
+    },
+    {
+      "epoch": 0.4128,
+      "grad_norm": 0.3778914426775528,
+      "learning_rate": 0.0001326770053470668,
+      "loss": 0.8762,
+      "step": 129
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 0.3583604676991779,
+      "learning_rate": 0.0001316920989420703,
+      "loss": 0.749,
+      "step": 130
+    },
+    {
+      "epoch": 0.4192,
+      "grad_norm": 0.3201977553502979,
+      "learning_rate": 0.00013070376302018287,
+      "loss": 0.771,
+      "step": 131
+    },
+    {
+      "epoch": 0.4224,
+      "grad_norm": 0.327874635613636,
+      "learning_rate": 0.00012971210453281674,
+      "loss": 0.7247,
+      "step": 132
+    },
+    {
+      "epoch": 0.4256,
+      "grad_norm": 0.36654127787678314,
+      "learning_rate": 0.000128717230790931,
+      "loss": 0.8057,
+      "step": 133
+    },
+    {
+      "epoch": 0.4288,
+      "grad_norm": 0.37665903879585244,
+      "learning_rate": 0.00012771924945341906,
+      "loss": 0.7767,
+      "step": 134
+    },
+    {
+      "epoch": 0.432,
+      "grad_norm": 0.4034764628456416,
+      "learning_rate": 0.00012671826851545851,
+      "loss": 0.8348,
+      "step": 135
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 0.3907539802126158,
+      "learning_rate": 0.0001257143962968246,
+      "loss": 0.8872,
+      "step": 136
+    },
+    {
+      "epoch": 0.4384,
+      "grad_norm": 0.3569846944905879,
+      "learning_rate": 0.00012470774143016853,
+      "loss": 0.8112,
+      "step": 137
+    },
+    {
+      "epoch": 0.4416,
+      "grad_norm": 0.36188126942655424,
+      "learning_rate": 0.00012369841284926188,
+      "loss": 0.8019,
+      "step": 138
+    },
+    {
+      "epoch": 0.4448,
+      "grad_norm": 0.379019802148045,
+      "learning_rate": 0.00012268651977720866,
+      "loss": 0.7994,
+      "step": 139
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 0.41234904215035534,
+      "learning_rate": 0.00012167217171462566,
+      "loss": 0.8809,
+      "step": 140
+    },
+    {
+      "epoch": 0.4512,
+      "grad_norm": 0.38868434007296393,
+      "learning_rate": 0.0001206554784277931,
+      "loss": 0.7578,
+      "step": 141
+    },
+    {
+      "epoch": 0.4544,
+      "grad_norm": 0.350346834606368,
+      "learning_rate": 0.00011963654993677645,
+      "loss": 0.8003,
+      "step": 142
+    },
+    {
+      "epoch": 0.4576,
+      "grad_norm": 0.4135182041369236,
+      "learning_rate": 0.00011861549650352069,
+      "loss": 0.7891,
+      "step": 143
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 0.3447428610535347,
+      "learning_rate": 0.00011759242861991855,
+      "loss": 0.7747,
+      "step": 144
+    },
+    {
+      "epoch": 0.464,
+      "grad_norm": 0.3659043229276162,
+      "learning_rate": 0.00011656745699585371,
+      "loss": 0.7847,
+      "step": 145
+    },
+    {
+      "epoch": 0.4672,
+      "grad_norm": 0.3627598928666497,
+      "learning_rate": 0.00011554069254722051,
+      "loss": 0.8729,
+      "step": 146
+    },
+    {
+      "epoch": 0.4704,
+      "grad_norm": 0.3316910349747776,
+      "learning_rate": 0.00011451224638392129,
+      "loss": 0.7812,
+      "step": 147
+    },
+    {
+      "epoch": 0.4736,
+      "grad_norm": 0.3946570922891007,
+      "learning_rate": 0.00011348222979784289,
+      "loss": 0.7865,
+      "step": 148
+    },
+    {
+      "epoch": 0.4768,
+      "grad_norm": 0.3359636799382807,
+      "learning_rate": 0.00011245075425081328,
+      "loss": 0.7965,
+      "step": 149
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.37137439376427284,
+      "learning_rate": 0.00011141793136253986,
+      "loss": 0.849,
+      "step": 150
+    },
+    {
+      "epoch": 0.4832,
+      "grad_norm": 0.3574031642165924,
+      "learning_rate": 0.0001103838728985307,
+      "loss": 0.825,
+      "step": 151
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 0.38556543326751924,
+      "learning_rate": 0.000109348690758,
+      "loss": 0.8379,
+      "step": 152
+    },
+    {
+      "epoch": 0.4896,
+      "grad_norm": 0.38805213476282924,
+      "learning_rate": 0.00010831249696175918,
+      "loss": 0.8097,
+      "step": 153
+    },
+    {
+      "epoch": 0.4928,
+      "grad_norm": 0.3574217944877886,
+      "learning_rate": 0.0001072754036400944,
+      "loss": 0.7886,
+      "step": 154
+    },
+    {
+      "epoch": 0.496,
+      "grad_norm": 0.39271223057544774,
+      "learning_rate": 0.00010623752302063283,
+      "loss": 0.8542,
+      "step": 155
+    },
+    {
+      "epoch": 0.4992,
+      "grad_norm": 0.4324299524551468,
+      "learning_rate": 0.00010519896741619803,
+      "loss": 0.8882,
+      "step": 156
+    },
+    {
+      "epoch": 0.5024,
+      "grad_norm": 0.32450275667877587,
+      "learning_rate": 0.00010415984921265609,
+      "loss": 0.7915,
+      "step": 157
+    },
+    {
+      "epoch": 0.5056,
+      "grad_norm": 0.37732713298950105,
+      "learning_rate": 0.00010312028085675391,
+      "loss": 0.7915,
+      "step": 158
+    },
+    {
+      "epoch": 0.5088,
+      "grad_norm": 0.3406928749347663,
+      "learning_rate": 0.00010208037484395114,
+      "loss": 0.817,
+      "step": 159
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 0.3751715626743408,
+      "learning_rate": 0.00010104024370624644,
+      "loss": 0.8437,
+      "step": 160
+    },
+    {
+      "epoch": 0.5152,
+      "grad_norm": 0.33757211373718954,
+      "learning_rate": 0.0001,
+      "loss": 0.7485,
+      "step": 161
+    },
+    {
+      "epoch": 0.5184,
+      "grad_norm": 0.46079705496359785,
+      "learning_rate": 9.895975629375359e-05,
+      "loss": 0.8788,
+      "step": 162
+    },
+    {
+      "epoch": 0.5216,
+      "grad_norm": 0.4271610568124271,
+      "learning_rate": 9.791962515604887e-05,
+      "loss": 0.8616,
+      "step": 163
+    },
+    {
+      "epoch": 0.5248,
+      "grad_norm": 0.3730244116708226,
+      "learning_rate": 9.687971914324607e-05,
+      "loss": 0.713,
+      "step": 164
+    },
+    {
+      "epoch": 0.528,
+      "grad_norm": 0.32767683574358164,
+      "learning_rate": 9.584015078734395e-05,
+      "loss": 0.7503,
+      "step": 165
+    },
+    {
+      "epoch": 0.5312,
+      "grad_norm": 0.32244567576195554,
+      "learning_rate": 9.480103258380198e-05,
+      "loss": 0.7239,
+      "step": 166
+    },
+    {
+      "epoch": 0.5344,
+      "grad_norm": 0.3397126044271277,
+      "learning_rate": 9.376247697936719e-05,
+      "loss": 0.7984,
+      "step": 167
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 0.3012960591552935,
+      "learning_rate": 9.272459635990562e-05,
+      "loss": 0.7531,
+      "step": 168
+    },
+    {
+      "epoch": 0.5408,
+      "grad_norm": 0.37938450909301,
+      "learning_rate": 9.168750303824084e-05,
+      "loss": 0.7915,
+      "step": 169
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 0.38701381132864615,
+      "learning_rate": 9.065130924199998e-05,
+      "loss": 0.7544,
+      "step": 170
+    },
+    {
+      "epoch": 0.5472,
+      "grad_norm": 0.38690509598881456,
+      "learning_rate": 8.961612710146934e-05,
+      "loss": 0.8261,
+      "step": 171
+    },
+    {
+      "epoch": 0.5504,
+      "grad_norm": 0.3662178194553636,
+      "learning_rate": 8.858206863746018e-05,
+      "loss": 0.7712,
+      "step": 172
+    },
+    {
+      "epoch": 0.5536,
+      "grad_norm": 0.325465438950959,
+      "learning_rate": 8.754924574918675e-05,
+      "loss": 0.7461,
+      "step": 173
+    },
+    {
+      "epoch": 0.5568,
+      "grad_norm": 0.36372519207365633,
+      "learning_rate": 8.651777020215712e-05,
+      "loss": 0.7577,
+      "step": 174
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.39791618344605173,
+      "learning_rate": 8.548775361607872e-05,
+      "loss": 0.8952,
+      "step": 175
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 0.39106877210397273,
+      "learning_rate": 8.445930745277953e-05,
+      "loss": 0.8067,
+      "step": 176
+    },
+    {
+      "epoch": 0.5664,
+      "grad_norm": 0.32165292969442727,
+      "learning_rate": 8.343254300414628e-05,
+      "loss": 0.704,
+      "step": 177
+    },
+    {
+      "epoch": 0.5696,
+      "grad_norm": 0.36154222910864736,
+      "learning_rate": 8.240757138008149e-05,
+      "loss": 0.8184,
+      "step": 178
+    },
+    {
+      "epoch": 0.5728,
+      "grad_norm": 0.42719982445728705,
+      "learning_rate": 8.138450349647936e-05,
+      "loss": 0.8165,
+      "step": 179
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 0.5246656865774878,
+      "learning_rate": 8.036345006322359e-05,
+      "loss": 0.9235,
+      "step": 180
+    },
+    {
+      "epoch": 0.5792,
+      "grad_norm": 0.3889988384699533,
+      "learning_rate": 7.934452157220694e-05,
+      "loss": 0.8564,
+      "step": 181
+    },
+    {
+      "epoch": 0.5824,
+      "grad_norm": 0.34293279991160774,
+      "learning_rate": 7.832782828537437e-05,
+      "loss": 0.7917,
+      "step": 182
+    },
+    {
+      "epoch": 0.5856,
+      "grad_norm": 0.43448804859625373,
+      "learning_rate": 7.731348022279134e-05,
+      "loss": 0.8397,
+      "step": 183
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 0.37332421671048194,
+      "learning_rate": 7.630158715073813e-05,
+      "loss": 0.847,
+      "step": 184
+    },
+    {
+      "epoch": 0.592,
+      "grad_norm": 0.2847408908726539,
+      "learning_rate": 7.52922585698315e-05,
+      "loss": 0.7008,
+      "step": 185
+    },
+    {
+      "epoch": 0.5952,
+      "grad_norm": 0.3793630586511894,
+      "learning_rate": 7.428560370317542e-05,
+      "loss": 0.8073,
+      "step": 186
+    },
+    {
+      "epoch": 0.5984,
+      "grad_norm": 0.3314978495730383,
+      "learning_rate": 7.328173148454151e-05,
+      "loss": 0.7501,
+      "step": 187
+    },
+    {
+      "epoch": 0.6016,
+      "grad_norm": 0.38192244839668066,
+      "learning_rate": 7.228075054658096e-05,
+      "loss": 0.8806,
+      "step": 188
+    },
+    {
+      "epoch": 0.6048,
+      "grad_norm": 0.35144673379944147,
+      "learning_rate": 7.1282769209069e-05,
+      "loss": 0.7983,
+      "step": 189
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 0.3320272570958794,
+      "learning_rate": 7.028789546718326e-05,
+      "loss": 0.7778,
+      "step": 190
+    },
+    {
+      "epoch": 0.6112,
+      "grad_norm": 0.3245100936187699,
+      "learning_rate": 6.929623697981718e-05,
+      "loss": 0.7638,
+      "step": 191
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 0.3487129086320174,
+      "learning_rate": 6.830790105792973e-05,
+      "loss": 0.771,
+      "step": 192
+    },
+    {
+      "epoch": 0.6176,
+      "grad_norm": 0.35931606259571913,
+      "learning_rate": 6.732299465293322e-05,
+      "loss": 0.7663,
+      "step": 193
+    },
+    {
+      "epoch": 0.6208,
+      "grad_norm": 0.3367390991318867,
+      "learning_rate": 6.63416243451194e-05,
+      "loss": 0.7966,
+      "step": 194
+    },
+    {
+      "epoch": 0.624,
+      "grad_norm": 0.33183255910992066,
+      "learning_rate": 6.536389633212609e-05,
+      "loss": 0.78,
+      "step": 195
+    },
+    {
+      "epoch": 0.6272,
+      "grad_norm": 0.3264528270630262,
+      "learning_rate": 6.43899164174453e-05,
+      "loss": 0.7772,
+      "step": 196
+    },
+    {
+      "epoch": 0.6304,
+      "grad_norm": 0.3517378205790495,
+      "learning_rate": 6.341978999897365e-05,
+      "loss": 0.7642,
+      "step": 197
+    },
+    {
+      "epoch": 0.6336,
+      "grad_norm": 0.431338962220006,
+      "learning_rate": 6.245362205760704e-05,
+      "loss": 0.7927,
+      "step": 198
+    },
+    {
+      "epoch": 0.6368,
+      "grad_norm": 0.3731631592093385,
+      "learning_rate": 6.149151714588009e-05,
+      "loss": 0.8503,
+      "step": 199
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.33909105312590193,
+      "learning_rate": 6.053357937665237e-05,
+      "loss": 0.7154,
+      "step": 200
+    },
+    {
+      "epoch": 0.6432,
+      "grad_norm": 0.36200420260594113,
+      "learning_rate": 5.957991241184184e-05,
+      "loss": 0.7569,
+      "step": 201
+    },
+    {
+      "epoch": 0.6464,
+      "grad_norm": 0.4149004455820632,
+      "learning_rate": 5.863061945120719e-05,
+      "loss": 0.8286,
+      "step": 202
+    },
+    {
+      "epoch": 0.6496,
+      "grad_norm": 0.32292850370856624,
+      "learning_rate": 5.768580322118034e-05,
+      "loss": 0.8015,
+      "step": 203
+    },
+    {
+      "epoch": 0.6528,
+      "grad_norm": 0.3771935747682809,
+      "learning_rate": 5.6745565963749925e-05,
+      "loss": 0.803,
+      "step": 204
+    },
+    {
+      "epoch": 0.656,
+      "grad_norm": 0.3541848583253109,
+      "learning_rate": 5.5810009425397294e-05,
+      "loss": 0.8545,
+      "step": 205
+    },
+    {
+      "epoch": 0.6592,
+      "grad_norm": 0.44832715565151354,
+      "learning_rate": 5.487923484608629e-05,
+      "loss": 0.8812,
+      "step": 206
+    },
+    {
+      "epoch": 0.6624,
+      "grad_norm": 0.37472326339826867,
+      "learning_rate": 5.395334294830765e-05,
+      "loss": 0.9129,
+      "step": 207
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 0.35228666839882417,
+      "learning_rate": 5.3032433926179395e-05,
+      "loss": 0.7832,
+      "step": 208
+    },
+    {
+      "epoch": 0.6688,
+      "grad_norm": 0.3350939709150189,
+      "learning_rate": 5.211660743460458e-05,
+      "loss": 0.7491,
+      "step": 209
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 0.38948109383274765,
+      "learning_rate": 5.1205962578487155e-05,
+      "loss": 0.7651,
+      "step": 210
+    },
+    {
+      "epoch": 0.6752,
+      "grad_norm": 0.32094235443298985,
+      "learning_rate": 5.030059790200756e-05,
+      "loss": 0.769,
+      "step": 211
+    },
+    {
+      "epoch": 0.6784,
+      "grad_norm": 0.30333546694103175,
+      "learning_rate": 4.940061137795876e-05,
+      "loss": 0.6676,
+      "step": 212
+    },
+    {
+      "epoch": 0.6816,
+      "grad_norm": 0.36011644742045107,
+      "learning_rate": 4.850610039714444e-05,
+      "loss": 0.7948,
+      "step": 213
+    },
+    {
+      "epoch": 0.6848,
+      "grad_norm": 0.4501270149702274,
+      "learning_rate": 4.761716175783989e-05,
+      "loss": 0.8075,
+      "step": 214
+    },
+    {
+      "epoch": 0.688,
+      "grad_norm": 0.32443270170477734,
+      "learning_rate": 4.673389165531714e-05,
+      "loss": 0.7327,
+      "step": 215
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 0.35437103043306173,
+      "learning_rate": 4.585638567143529e-05,
+      "loss": 0.7306,
+      "step": 216
+    },
+    {
+      "epoch": 0.6944,
+      "grad_norm": 0.32026538320410597,
+      "learning_rate": 4.498473876429726e-05,
+      "loss": 0.7387,
+      "step": 217
+    },
+    {
+      "epoch": 0.6976,
+      "grad_norm": 0.3565566202307532,
+      "learning_rate": 4.411904525797408e-05,
+      "loss": 0.7722,
+      "step": 218
+    },
+    {
+      "epoch": 0.7008,
+      "grad_norm": 0.3770279902011318,
+      "learning_rate": 4.325939883229766e-05,
+      "loss": 0.8262,
+      "step": 219
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 0.3225427774272655,
+      "learning_rate": 4.240589251272342e-05,
+      "loss": 0.7808,
+      "step": 220
+    },
+    {
+      "epoch": 0.7072,
+      "grad_norm": 0.3643305874741898,
+      "learning_rate": 4.155861866026364e-05,
+      "loss": 0.6751,
+      "step": 221
+    },
+    {
+      "epoch": 0.7104,
+      "grad_norm": 0.3495365472341216,
+      "learning_rate": 4.071766896149273e-05,
+      "loss": 0.7634,
+      "step": 222
+    },
+    {
+      "epoch": 0.7136,
+      "grad_norm": 0.3865725013932627,
+      "learning_rate": 3.988313441862553e-05,
+      "loss": 0.8201,
+      "step": 223
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 0.39371326689707414,
+      "learning_rate": 3.9055105339669595e-05,
+      "loss": 0.783,
+      "step": 224
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.3343399455415386,
+      "learning_rate": 3.823367132865265e-05,
+      "loss": 0.7558,
+      "step": 225
+    },
+    {
+      "epoch": 0.7232,
+      "grad_norm": 0.37018368726419704,
+      "learning_rate": 3.741892127592625e-05,
+      "loss": 0.8034,
+      "step": 226
+    },
+    {
+      "epoch": 0.7264,
+      "grad_norm": 0.2917671574400929,
+      "learning_rate": 3.6610943348546526e-05,
+      "loss": 0.7608,
+      "step": 227
+    },
+    {
+      "epoch": 0.7296,
+      "grad_norm": 0.3149055123541856,
+      "learning_rate": 3.580982498073344e-05,
+      "loss": 0.7376,
+      "step": 228
+    },
+    {
+      "epoch": 0.7328,
+      "grad_norm": 0.48194732886134795,
+      "learning_rate": 3.501565286440914e-05,
+      "loss": 0.8543,
+      "step": 229
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 0.3201762925229535,
+      "learning_rate": 3.422851293981676e-05,
+      "loss": 0.7311,
+      "step": 230
+    },
+    {
+      "epoch": 0.7392,
+      "grad_norm": 0.3832969139432672,
+      "learning_rate": 3.3448490386220355e-05,
+      "loss": 0.8361,
+      "step": 231
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 0.3591778802980467,
+      "learning_rate": 3.2675669612687565e-05,
+      "loss": 0.8246,
+      "step": 232
+    },
+    {
+      "epoch": 0.7456,
+      "grad_norm": 0.33711687357568193,
+      "learning_rate": 3.191013424895536e-05,
+      "loss": 0.7911,
+      "step": 233
+    },
+    {
+      "epoch": 0.7488,
+      "grad_norm": 0.3992050715809478,
+      "learning_rate": 3.115196713638e-05,
+      "loss": 0.784,
+      "step": 234
+    },
+    {
+      "epoch": 0.752,
+      "grad_norm": 0.3717791655175765,
+      "learning_rate": 3.040125031897264e-05,
+      "loss": 0.8544,
+      "step": 235
+    },
+    {
+      "epoch": 0.7552,
+      "grad_norm": 0.3266902771814692,
+      "learning_rate": 2.9658065034520978e-05,
+      "loss": 0.758,
+      "step": 236
+    },
+    {
+      "epoch": 0.7584,
+      "grad_norm": 0.35563021204215095,
+      "learning_rate": 2.892249170579826e-05,
+      "loss": 0.7699,
+      "step": 237
+    },
+    {
+      "epoch": 0.7616,
+      "grad_norm": 0.379714101588036,
+      "learning_rate": 2.8194609931860316e-05,
+      "loss": 0.8409,
+      "step": 238
+    },
+    {
+      "epoch": 0.7648,
+      "grad_norm": 0.47451149824579747,
+      "learning_rate": 2.7474498479432087e-05,
+      "loss": 0.8289,
+      "step": 239
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 0.3188599450071592,
+      "learning_rate": 2.6762235274383772e-05,
+      "loss": 0.7431,
+      "step": 240
+    },
+    {
+      "epoch": 0.7712,
+      "grad_norm": 0.3537473485993502,
+      "learning_rate": 2.6057897393298324e-05,
+      "loss": 0.7519,
+      "step": 241
+    },
+    {
+      "epoch": 0.7744,
+      "grad_norm": 0.35260123842359997,
+      "learning_rate": 2.536156105513062e-05,
+      "loss": 0.8052,
+      "step": 242
+    },
+    {
+      "epoch": 0.7776,
+      "grad_norm": 0.3782134918689213,
+      "learning_rate": 2.4673301612959654e-05,
+      "loss": 0.8498,
+      "step": 243
+    },
+    {
+      "epoch": 0.7808,
+      "grad_norm": 0.4240797008066036,
+      "learning_rate": 2.399319354583418e-05,
+      "loss": 0.8504,
+      "step": 244
+    },
+    {
+      "epoch": 0.784,
+      "grad_norm": 0.3493922027357962,
+      "learning_rate": 2.3321310450713062e-05,
+      "loss": 0.748,
+      "step": 245
+    },
+    {
+      "epoch": 0.7872,
+      "grad_norm": 0.3109975196467063,
+      "learning_rate": 2.265772503450122e-05,
+      "loss": 0.7332,
+      "step": 246
+    },
+    {
+      "epoch": 0.7904,
+      "grad_norm": 0.3490386685549296,
+      "learning_rate": 2.2002509106181624e-05,
+      "loss": 0.7745,
+      "step": 247
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 0.36175624431520176,
+      "learning_rate": 2.1355733569044635e-05,
+      "loss": 0.7388,
+      "step": 248
+    },
+    {
+      "epoch": 0.7968,
+      "grad_norm": 0.4367153475876239,
+      "learning_rate": 2.0717468413015283e-05,
+      "loss": 0.8179,
+      "step": 249
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.39966660383957925,
+      "learning_rate": 2.008778270707944e-05,
+      "loss": 0.7862,
+      "step": 250
+    },
+    {
+      "epoch": 0.8032,
+      "grad_norm": 0.39454280119505963,
+      "learning_rate": 1.946674459180955e-05,
+      "loss": 0.7943,
+      "step": 251
+    },
+    {
+      "epoch": 0.8064,
+      "grad_norm": 0.34240338202982135,
+      "learning_rate": 1.8854421271990964e-05,
+      "loss": 0.7764,
+      "step": 252
+    },
+    {
+      "epoch": 0.8096,
+      "grad_norm": 0.3836219267710939,
+      "learning_rate": 1.8250879009349398e-05,
+      "loss": 0.7249,
+      "step": 253
+    },
+    {
+      "epoch": 0.8128,
+      "grad_norm": 0.3545049008632808,
+      "learning_rate": 1.7656183115380577e-05,
+      "loss": 0.745,
+      "step": 254
+    },
+    {
+      "epoch": 0.816,
+      "grad_norm": 0.4782319117565856,
+      "learning_rate": 1.707039794428259e-05,
+      "loss": 0.9546,
+      "step": 255
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 0.30912276688779844,
+      "learning_rate": 1.649358688599191e-05,
+      "loss": 0.7098,
+      "step": 256
+    },
+    {
+      "epoch": 0.8224,
+      "grad_norm": 0.3564793276918867,
+      "learning_rate": 1.5925812359323745e-05,
+      "loss": 0.7553,
+      "step": 257
+    },
+    {
+      "epoch": 0.8256,
+      "grad_norm": 0.34525121870000597,
+      "learning_rate": 1.5367135805217458e-05,
+      "loss": 0.7499,
+      "step": 258
+    },
+    {
+      "epoch": 0.8288,
+      "grad_norm": 0.3349420082155695,
+      "learning_rate": 1.4817617680087825e-05,
+      "loss": 0.805,
+      "step": 259
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 0.433540567668089,
+      "learning_rate": 1.4277317449282834e-05,
+      "loss": 0.877,
+      "step": 260
+    },
+    {
+      "epoch": 0.8352,
+      "grad_norm": 0.3801150070687427,
+      "learning_rate": 1.3746293580648717e-05,
+      "loss": 0.8694,
+      "step": 261
+    },
+    {
+      "epoch": 0.8384,
+      "grad_norm": 0.3738768456908641,
+      "learning_rate": 1.3224603538202929e-05,
+      "loss": 0.7737,
+      "step": 262
+    },
+    {
+      "epoch": 0.8416,
+      "grad_norm": 0.3801506817792511,
+      "learning_rate": 1.2712303775915802e-05,
+      "loss": 0.7256,
+      "step": 263
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 0.29294064483194054,
+      "learning_rate": 1.220944973160133e-05,
+      "loss": 0.7278,
+      "step": 264
+    },
+    {
+      "epoch": 0.848,
+      "grad_norm": 0.3574084890505441,
+      "learning_rate": 1.1716095820918216e-05,
+      "loss": 0.8017,
+      "step": 265
+    },
+    {
+      "epoch": 0.8512,
+      "grad_norm": 0.44325322418927066,
+      "learning_rate": 1.1232295431481222e-05,
+      "loss": 0.8188,
+      "step": 266
+    },
+    {
+      "epoch": 0.8544,
+      "grad_norm": 0.33465511889302607,
+      "learning_rate": 1.0758100917083991e-05,
+      "loss": 0.7404,
+      "step": 267
+    },
+    {
+      "epoch": 0.8576,
+      "grad_norm": 0.35582143701746155,
+      "learning_rate": 1.0293563592033595e-05,
+      "loss": 0.828,
+      "step": 268
+    },
+    {
+      "epoch": 0.8608,
+      "grad_norm": 0.32908366694518915,
+      "learning_rate": 9.838733725597615e-06,
+      "loss": 0.7496,
+      "step": 269
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 0.3923286422615176,
+      "learning_rate": 9.393660536564408e-06,
+      "loss": 0.7605,
+      "step": 270
+    },
+    {
+      "epoch": 0.8672,
+      "grad_norm": 0.32871379992022076,
+      "learning_rate": 8.958392187916841e-06,
+      "loss": 0.7252,
+      "step": 271
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 0.4464878317171815,
+      "learning_rate": 8.532975781620512e-06,
+      "loss": 0.7487,
+      "step": 272
+    },
+    {
+      "epoch": 0.8736,
+      "grad_norm": 0.37431391710968503,
+      "learning_rate": 8.117457353526625e-06,
+      "loss": 0.7393,
+      "step": 273
+    },
+    {
+      "epoch": 0.8768,
+      "grad_norm": 0.31383965433569194,
+      "learning_rate": 7.711881868390291e-06,
+      "loss": 0.7417,
+      "step": 274
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 0.40622594368739234,
+      "learning_rate": 7.3162932150046885e-06,
+      "loss": 0.8212,
+      "step": 275
+    },
+    {
+      "epoch": 0.8832,
+      "grad_norm": 0.312688075077189,
+      "learning_rate": 6.930734201451816e-06,
+      "loss": 0.6931,
+      "step": 276
+    },
+    {
+      "epoch": 0.8864,
+      "grad_norm": 0.2951795837647385,
+      "learning_rate": 6.555246550469907e-06,
+      "loss": 0.7062,
+      "step": 277
+    },
+    {
+      "epoch": 0.8896,
+      "grad_norm": 0.3906241449752177,
+      "learning_rate": 6.189870894938587e-06,
+      "loss": 0.8502,
+      "step": 278
+    },
+    {
+      "epoch": 0.8928,
+      "grad_norm": 0.33976393181898756,
+      "learning_rate": 5.834646773481811e-06,
+      "loss": 0.7466,
+      "step": 279
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 0.3645631566569671,
+      "learning_rate": 5.489612626189245e-06,
+      "loss": 0.8331,
+      "step": 280
+    },
+    {
+      "epoch": 0.8992,
+      "grad_norm": 0.32171725161543063,
+      "learning_rate": 5.154805790456485e-06,
+      "loss": 0.7301,
+      "step": 281
+    },
+    {
+      "epoch": 0.9024,
+      "grad_norm": 0.3215222867159588,
+      "learning_rate": 4.830262496944693e-06,
+      "loss": 0.7803,
+      "step": 282
+    },
+    {
+      "epoch": 0.9056,
+      "grad_norm": 0.3930936220455362,
+      "learning_rate": 4.516017865659949e-06,
+      "loss": 0.8015,
+      "step": 283
+    },
+    {
+      "epoch": 0.9088,
+      "grad_norm": 0.4009757739317676,
+      "learning_rate": 4.21210590215273e-06,
+      "loss": 0.7702,
+      "step": 284
+    },
+    {
+      "epoch": 0.912,
+      "grad_norm": 0.3509902652942518,
+      "learning_rate": 3.918559493838114e-06,
+      "loss": 0.8313,
+      "step": 285
+    },
+    {
+      "epoch": 0.9152,
+      "grad_norm": 0.4823527769540213,
+      "learning_rate": 3.6354104064368566e-06,
+      "loss": 0.7737,
+      "step": 286
+    },
+    {
+      "epoch": 0.9184,
+      "grad_norm": 0.37189019955946206,
+      "learning_rate": 3.3626892805379562e-06,
+      "loss": 0.7485,
+      "step": 287
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 0.3929380712488443,
+      "learning_rate": 3.100425628282899e-06,
+      "loss": 0.7541,
+      "step": 288
+    },
+    {
+      "epoch": 0.9248,
+      "grad_norm": 0.3230172315650001,
+      "learning_rate": 2.848647830172024e-06,
+      "loss": 0.7514,
+      "step": 289
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 0.331298596116422,
+      "learning_rate": 2.607383131993424e-06,
+      "loss": 0.7476,
+      "step": 290
+    },
+    {
+      "epoch": 0.9312,
+      "grad_norm": 0.41208820254007095,
+      "learning_rate": 2.3766576418745022e-06,
+      "loss": 0.8253,
+      "step": 291
+    },
+    {
+      "epoch": 0.9344,
+      "grad_norm": 0.3540797011234827,
+      "learning_rate": 2.1564963274568027e-06,
+      "loss": 0.7754,
+      "step": 292
+    },
+    {
+      "epoch": 0.9376,
+      "grad_norm": 0.40367747859127645,
+      "learning_rate": 1.9469230131940907e-06,
+      "loss": 0.8121,
+      "step": 293
+    },
+    {
+      "epoch": 0.9408,
+      "grad_norm": 0.36985043876259127,
+      "learning_rate": 1.7479603777742938e-06,
+      "loss": 0.7875,
+      "step": 294
+    },
+    {
+      "epoch": 0.944,
+      "grad_norm": 0.3757213402313114,
+      "learning_rate": 1.559629951665298e-06,
+      "loss": 0.7626,
+      "step": 295
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 0.3519319526157233,
+      "learning_rate": 1.3819521147851123e-06,
+      "loss": 0.8127,
+      "step": 296
+    },
+    {
+      "epoch": 0.9504,
+      "grad_norm": 0.3733908580196815,
+      "learning_rate": 1.2149460942964098e-06,
+      "loss": 0.7547,
+      "step": 297
+    },
+    {
+      "epoch": 0.9536,
+      "grad_norm": 0.36965768774397506,
+      "learning_rate": 1.05862996252597e-06,
+      "loss": 0.7968,
+      "step": 298
+    },
+    {
+      "epoch": 0.9568,
+      "grad_norm": 0.392672840473129,
+      "learning_rate": 9.130206350089765e-07,
+      "loss": 0.7971,
+      "step": 299
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 0.35145171210243764,
+      "learning_rate": 7.781338686584927e-07,
+      "loss": 0.7712,
+      "step": 300
+    },
+    {
+      "epoch": 0.9632,
+      "grad_norm": 0.3557470819008264,
+      "learning_rate": 6.539842600603918e-07,
+      "loss": 0.8122,
+      "step": 301
+    },
+    {
+      "epoch": 0.9664,
+      "grad_norm": 0.39875493221618996,
+      "learning_rate": 5.405852438937764e-07,
+      "loss": 0.7222,
+      "step": 302
+    },
+    {
+      "epoch": 0.9696,
+      "grad_norm": 0.3398025671483795,
+      "learning_rate": 4.3794909147720773e-07,
+      "loss": 0.7241,
+      "step": 303
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 0.44734223609044316,
+      "learning_rate": 3.4608690944071263e-07,
+      "loss": 0.8064,
+      "step": 304
+    },
+    {
+      "epoch": 0.976,
+      "grad_norm": 0.36261141044117756,
+      "learning_rate": 2.6500863852395584e-07,
+      "loss": 0.7815,
+      "step": 305
+    },
+    {
+      "epoch": 0.9792,
+      "grad_norm": 1.1171234591773465,
+      "learning_rate": 1.947230525005006e-07,
+      "loss": 0.7992,
+      "step": 306
+    },
+    {
+      "epoch": 0.9824,
+      "grad_norm": 0.3255664161802146,
+      "learning_rate": 1.3523775722834587e-07,
+      "loss": 0.7498,
+      "step": 307
+    },
+    {
+      "epoch": 0.9856,
+      "grad_norm": 0.34989009087457607,
+      "learning_rate": 8.655918982689581e-08,
+      "loss": 0.8037,
+      "step": 308
+    },
+    {
+      "epoch": 0.9888,
+      "grad_norm": 0.33217965843871977,
+      "learning_rate": 4.8692617980350406e-08,
+      "loss": 0.8156,
+      "step": 309
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 0.3603261858382364,
+      "learning_rate": 2.164213936770576e-08,
+      "loss": 0.7071,
+      "step": 310
+    },
+    {
+      "epoch": 0.9952,
+      "grad_norm": 0.3558631704357258,
+      "learning_rate": 5.410681219286673e-09,
+      "loss": 0.742,
+      "step": 311
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 0.3200726372631764,
+      "learning_rate": 0.0,
+      "loss": 0.7306,
+      "step": 312
+    },
+    {
+      "epoch": 0.9984,
+      "step": 312,
+      "total_flos": 739882583326720.0,
+      "train_loss": 0.8309416449986972,
+      "train_runtime": 9163.8195,
+      "train_samples_per_second": 1.091,
+      "train_steps_per_second": 0.034
+    }
+  ],
+  "logging_steps": 1.0,
+  "max_steps": 312,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": false,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 739882583326720.0,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_1_epochs_1_GA_2_lora/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ./weights/Bunny-v1_1-Llama-3-8B-V
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_1_epochs_1_GA_2_lora/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj",
+    "gate_proj",
+    "k_proj",
+    "o_proj",
+    "down_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_1_epochs_1_GA_2_lora/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:807df09410a8bac8441b10edd939efedcbdf49e2206c5d73aed6963a3087c65e
+size 671150064

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_1_epochs_1_GA_2_lora/config.json

@@ -0,0 +1,45 @@
+{
+  "_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "architectures": [
+    "BunnyLlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_bunny_llama.BunnyLlamaConfig",
+    "AutoModelForCausalLM": "modeling_bunny_llama.BunnyLlamaForCausalLM"
+  },
+  "bos_token_id": 128000,
+  "continuous_training": false,
+  "eos_token_id": 128001,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect_ratio": "pad",
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 8192,
+  "mm_hidden_size": 3456,
+  "mm_projector_lr": null,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_vision_tower": "./weights/siglip-so400m-patch14-384",
+  "model_type": "bunny-llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 2048,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "float16",
+  "transformers_version": "4.41.2",
+  "tune_mm_mlp_adapter": false,
+  "unfreeze_vision_tower": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_s2": true,
+  "vocab_size": 128256
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_1_epochs_1_GA_2_lora/non_lora_trainables.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a957a55853852e70c4e4559e96f956dcefc8993421eea81bbc558164c8b426a
+size 918507402

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_2_epochs_1_GA_2_lora/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ./weights/Bunny-v1_1-Llama-3-8B-V
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_2_epochs_1_GA_2_lora/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "q_proj",
+    "down_proj",
+    "up_proj",
+    "o_proj",
+    "v_proj",
+    "gate_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_2_epochs_1_GA_2_lora/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:222d6020a814fe4a9d38c7187c52bcc5c8fab80c418dd58f37f19789d5d3258e
+size 671150064

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_2_epochs_1_GA_2_lora/config.json

@@ -0,0 +1,45 @@
+{
+  "_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "architectures": [
+    "BunnyLlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_bunny_llama.BunnyLlamaConfig",
+    "AutoModelForCausalLM": "modeling_bunny_llama.BunnyLlamaForCausalLM"
+  },
+  "bos_token_id": 128000,
+  "continuous_training": false,
+  "eos_token_id": 128001,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect_ratio": "pad",
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 8192,
+  "mm_hidden_size": 3456,
+  "mm_projector_lr": null,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_vision_tower": "./weights/siglip-so400m-patch14-384",
+  "model_type": "bunny-llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 2048,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "float16",
+  "transformers_version": "4.41.2",
+  "tune_mm_mlp_adapter": false,
+  "unfreeze_vision_tower": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_s2": true,
+  "vocab_size": 128256
+}

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_2_epochs_1_GA_2_lora/non_lora_trainables.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48d577f56281a55d9ebce8783da6a9f998306862aa87ce27b012361b7d4c3a07
+size 918507402

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_3_epochs_1_GA_2_lora/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ./weights/Bunny-v1_1-Llama-3-8B-V
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

mixing_strategies/Equal/bugsBunny-v1_1-Llama-3-8B-V-Equal_dataset_20000_repeat_3_epochs_1_GA_2_lora/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "./weights/Bunny-v1_1-Llama-3-8B-V",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "up_proj",
+    "v_proj",
+    "q_proj",
+    "o_proj",
+    "gate_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}