---
title: Tini-Lad
emoji: ⚡
colorFrom: pink
colorTo: red
sdk: gradio
sdk_version: 5.33.0
app_file: app.py
pinned: false
license: other
short_description: DLM
---

# 💬 Diffusion Language Model Demo

Note: Paper coming out soon; if anyone is interested in discussing the model, please contact me.

This is an interactive demo of a **diffusion-style language model**, which generates text through iterative refinement. 
Inspired by diffusion processes in vision models, the system gradually improves a corrupted text sequence until convergence. 

This implementation has several benefits:
- **Noiseless convergence**: A unique feature of this implementation is its ability to convergence **without intermediate noising**.
- *Scalable test time compute*: By increasing the number of iterations, the answer quality improves.
- *Reduced inference time*: Most questions can be answered with less iterations then the number of tokens generated!
- *Greatly reduced training time*: By LoRA-based finetuning of an autoregressive model, this model can be trained within several hours on a single GPU.

---

## 🔧 Settings
- **Disable Intermediate Noising**: Speeds up convergence by skipping the noising step between iterations. Works best for short, factual questions.
- **Iterations**: Number of refinement steps. More iterations means more time to refine the answer.
- **Pause Between Steps**: Slows down the process so you can visually follow the changes.

---

## 🖍️ Visualization
- **Red tokens**: Masked (noised) tokens that will be regenerated.
- **Green tokens**: Newly generated tokens compared to the previous step.

---

## 🧪 Example Prompt
For noiseless diffusion, try short questions like:
> What's the capital of France?

For more in-depth questions, enable intermediate noising. Increasing the number of iterations generally improves answer quality.
> What do you know about Amsterdam?

See how low you can go with the number of iterations while still receiving adequate answers!

---

More technical details (architecture, training, and evaluation) can be found in the accompanying blog post:  

📘 [Read the blog post here](https://example.com/diffusion-language-model-blog)

For a more tweakable version that includes all inference parameters, check out this version:

🎛️ [Explore the model here](https://huggingface.co/spaces/Ruurd/tini)

Paper coming out soon! If you already want to cite this model, please refer to the blogpost