hari106/sd-1.5-yoga-lora

Model Card for Model ID

This model is a finetuned LoRA on top of Stable Diffusion which aims to enhance accuracy of yoga poses. The finetuned model is trained on 107 classes of yoga poses spread over 7k+ images.

Model Description

• Developed by: [hari106]
• Model type: [Adapter]
• License: [openrail]

Uses

Generates better and more accurate yoga poses than base SD 1.5

Out-of-Scope Use

While the dataset covered a plethora of popular poses, there is a chance some of them might be missed out - esoteric or obscure yoga poses.

Bias, Risks, and Limitations

The model performs well on standard and simple poses, but can get jittery with complex inversions or spinal twists such as "yoganidrasana". The captioning quality is also not the best in this case, and could need improvements. Simple keywords like "Adho Mukha Svanasana, Downward facing dog, yoga pose, full body" helps get better results.

Training Details

Training Data

This model was trained on Shruti Saxena's yoga classification dataset: [https://www.kaggle.com/datasets/shrutisaxena/yoga-pose-image-classification-dataset]

Training Hyperparameters

• Training regime: fp16 mixed precision
• Epochs: 10
• Optimizer: AdamW
• Learning Rate: 1e-4
• UNet Learning Rate: 1e-4
• Scheduler:: cosine_with_restarts
• Network Module: networks.lora
• Network Dimensions: 32
• Network Alpha: 16

Metrics

Rather than FID and CLIP which was not really the evaluation criteria, the images were evaluated on the basis of joint angle statistics. Namely:

• Object Keypoint Similarity (OKS)
• Percentage of Correct Keypoints (PCK)
• Joint Angle Error (JAE)

Results

The evaluation CSV could be found in the repo. Check [summary_geometry.csv]

Summary

The model performs quite well with JAE metric compared to the base model. Roughly 81 classes or yoga poses had better JAE score as compared to the reference images than the base model.

Model Card Contact

Email: [cosmic.waves2001@gmail.com]