Why adaptor_model.bin becomes much larger than llama familes?
#34
by andreaKIM - opened
Using peft in llama models, average size of adapter_model.bin is about 100MB ~ 400MB with lora_rank 64~128. (# of trainable parameter 33M ~ 67M)
What is the main reason that mistralai model has so many trainable parameters than llama models?(330M with rank 128)
andreaKIM changed discussion status to closed