nextgenomni/ngo-media-assets

Reelquill Presenter Pack 2

The open models behind the presenter in Reelquill AI: a short video of a person (someone who agreed to it, or one of the app's ready-made presenters: AI-generated people who don't exist), whose lips are made again to say your narration. They come as one file, ngo-media-assets-v2.tar (1,042,657,280 bytes), a plain uncompressed tar. It is the same bytes as reelquill-presenter-pack-v2.tar, the name the app builds it under.

Every model in it was made by others and published under the MIT License or the Apache License 2.0. NextGenOmni LLC, the maker of Reelquill AI, exported four of the networks to ONNX in smaller number formats (float16, 8-bit) and saved Whisper's mel filters as a file. The two MediaPipe files are unchanged. Nothing was retrained. Each file keeps its own licence: see Licences.

Please read the use policy before you use it.

Pack 1 (ngo-media-assets-v1.tar, for the presenter made from one photo) stays in this repo for older versions of the app: see Pack 1.

What's in it

File	What it does	Origin	Licence	Converted
unet_int8.onnx	Makes the lips: from a frame's face with its lower half blanked, the same face whole, and the speech around that moment, it makes the face with a new mouth (as a latent), in one step	MuseTalk 1.5's UNet (Tencent Music Entertainment Group, Lyra Lab): musetalkV15/unet.pth	MIT	Yes, by NextGenOmni LLC: exported to ONNX (opset 17, its timestep, 0, built in), float16 (its Resize scales kept in float32); its Conv, MatMul and Gemm weights stored as 8-bit integers, one scale per output channel, read through DequantizeLinear and turned back into float16 when it loads
vae_encoder_fp16.onnx	Turns a 256 x 256 crop of the face into the latent the UNet works on (once per presenter video, when it is prepared)	sd-vae-ft-mse's encoder (Stability AI)	MIT	Yes, by NextGenOmni LLC: exported to ONNX (opset 17; the latent's mean out, scaled as MuseTalk scales it), float16
vae_decoder_fp16.onnx	Draws the 256 x 256 face from the UNet's latent	sd-vae-ft-mse's decoder (Stability AI)	MIT	Yes, by NextGenOmni LLC: exported to ONNX (opset 17; the scaled latent in), float16 (its Resize scales kept in float32)
whisper_tiny_encoder_fp16.onnx	Hears the narration: turns its log-mel into speech features, 50 a second, for the UNet	Whisper tiny's encoder (OpenAI): openai/whisper-tiny	MIT (OpenAI's release; the Hugging Face copy is labelled Apache 2.0)	Yes, by NextGenOmni LLC: exported to ONNX (opset 17) with every layer's hidden states stacked, as MuseTalk uses them, float16
mel_filters_201x80.f32	Whisper's 80 mel filters for a 400-point FFT: the app turns the narration's sound into the log-mel Whisper hears with them	Computed by Hugging Face transformers' WhisperFeatureExtractor, for Whisper	MIT (Whisper) / Apache 2.0 (transformers)	Made by NextGenOmni LLC: the filter bank as WhisperFeatureExtractor computes it, saved as raw float32 (201 x 80, little-endian, no header)
blaze_face.onnx	Finds the face in each frame of the presenter's video	MediaPipe (Google): BlazeFace, short range, from the Face Landmarker; ONNX as published with Ditto (Ant Group)	Apache 2.0	No (the same file as in Pack 1)
face_mesh.onnx	Finds 478 points on the face: to turn it upright, and to make the mask of the lower face that is pasted back	MediaPipe (Google): face landmarks, from the Face Landmarker; ONNX as published with Ditto	Apache 2.0	No (the same file as in Pack 1)

Every network has float32 inputs and outputs; only its weights are stored smaller.

Also in the pack:

• pack.json: every other file's size and SHA-256.
• README.md: a short description of the pack (this page is the long one).
• NOTICE: how each file was made, and from what. Next to the pack in this repo it is NOTICE-pack-v2.txt, the same bytes.
• LICENSE-MIT and LICENSE-Apache-2.0: the licences in full. Next to the pack they are LICENSE-MIT-pack-v2.txt and LICENSE-Apache-2.0.txt, the same bytes (the Apache text is also the one Pack 1 links).

The app keeps NOTICE, LICENSE-MIT and LICENSE-Apache-2.0 next to the models.

Where the originals are

Each file was made from these, the same bytes as at the commit linked (their SHA-256 is the one Hugging Face lists):

Our file	Made from	SHA-256 of the original
unet_int8.onnx	musetalkV15/unet.pth (with its musetalk.json) at TMElyralab/MuseTalk, commit 2bcb936	7ebf6c98c181e20838e4c0054e96e944ac60d5d692cc01db42839fe11b787007
vae_encoder_fp16.onnx, vae_decoder_fp16.onnx	diffusion_pytorch_model.bin (with its config.json) at stabilityai/sd-vae-ft-mse, commit 31f26fd	1b4889b6b1d4ce7ae320a02dedaeff1780ad77d415ea0d744b476155c6377ddc
whisper_tiny_encoder_fp16.onnx, mel_filters_201x80.f32	pytorch_model.bin (with its config.json and preprocessor_config.json) at openai/whisper-tiny, commit 169d4a4	9607f98a2b22d9e229ae43c52ecea79dcede9e0c5cfae67e8da6eda86d8aac1d
blaze_face.onnx, face_mesh.onnx	the same files, byte for byte, in ditto_onnx/ at digital-avatar/ditto-talkinghead, commit e4a2f60	as in Verify it

How Reelquill AI uses it

Reelquill AI is a screen recorder and video editor for Windows. Its presenter is a short video of a person talking to the camera (someone who agreed to it, or one of the app's ready-made presenters: AI-generated people who don't exist), put in a bubble on your video: the app makes the person's lips again, frame by frame, to say your narration. The rest of them (the head, the eyes, the shoulders) moves as in their own video.

• Once per presenter video, the app prepares it: blaze_face.onnx and face_mesh.onnx find the face on every frame; the face is turned upright and cropped to 256 x 256; vae_encoder_fp16.onnx turns each crop into latents (once with its lower half blanked, once whole); and the face mesh's points give the mask of the lower face that is pasted back.
• For each narration, its sound goes through Whisper's log-mel (mel_filters_201x80.f32) and whisper_tiny_encoder_fp16.onnx. Then, frame by frame, unet_int8.onnx makes the face with its new mouth from the frame's two latents and the speech around it, vae_decoder_fp16.onnx draws it, and it is pasted back through the mask, turned back to the frame's angle.
• The files come with the app's installer. When one is missing or damaged, the app downloads just that file from this repo: it knows where each file's bytes are inside the pack and asks for only those (HTTP range requests), so there is nothing to unpack and no second copy on your drive. It asks for the pack at the commit it was uploaded in, so a later change to this repo never changes what it downloads.
• Where the app can't reach the internet (a firewall), the pack can be downloaded in a web browser and chosen in the app's Presenter card (Import model pack from file…). The app checks the whole file's SHA-256, then each file's, and keeps only the files it was missing.
• It checks every file against its SHA-256 before using it, and refuses one that doesn't match.
• The networks run on your PC with ONNX Runtime, on your graphics chip (DirectML). The UNet's 8-bit weights are turned back into float16 once, as it loads. Your videos, your voice and your narration are not uploaded.
• In one test on a 10 s clip, the UNet with its weights rounded to 8 bits the same way (one scale per output channel) drew the face at 49 dB PSNR from the float16 UNet's, on the face's area: no difference to the eye. On a laptop with Intel Iris Xe graphics the lips take about 1.2 to 1.3 s a frame: about half an hour of making for a minute of video, at 25 frames a second.

Verify it

• The pack, ngo-media-assets-v2.tar: 1,042,657,280 bytes, SHA-256 335d732630c86afc2ebf8ce0c9ffd85ff194a32c345ba4f45fb9b06e12d81e0a. Hugging Face shows the same on the file's own page: click it under Files and versions, and read the line labelled SHA256 (not the Xet hash).
• ngo-media-assets-v2.json, next to the pack in this repo, is its manifest: the same bytes as the manifest the app carries (pack-v2.json). It gives the pack's size and SHA-256, and for every file inside it, where its bytes start (offset), its size and its SHA-256. (Its file field keeps the build name, reelquill-presenter-pack-v2.tar.)
• pack.json, inside the pack, lists every other file's size and SHA-256.
• Every file in the pack, in its order:

File	Offset	Size (bytes)	SHA-256
pack.json	512	2,026	89c397e70bb1788ae3c441be3fa88662e262af0127119868781d0f03b3f16ef2
README.md	3,072	1,239	aca05ff507920b2b85ececd2f95dc6f89a95c80127380e928f3d7aeee2a00388
unet_int8.onnx	5,120	853,211,160	02de46ed47a2863c77030af12ee874b139fab09e8a669380f601dc58122f37e6
vae_decoder_fp16.onnx	853,217,280	99,094,635	e292698d6e5f68dcf98afaa0bb0c481b547fcab24391e7e3ab3381c83abc9f66
LICENSE-Apache-2.0	952,312,832	11,356	43070e2d4e532684de521b885f385d0841030efa2b1a20bafb76133a5e1379c1
LICENSE-MIT	952,325,120	1,479	c1412a7f3da8bd026cc51ba0f8fa5044e7e0868f661c7abb6c7f190f964469a4
NOTICE	952,327,168	1,528	9aa22d92791311c5ae417692fc771b924271d4f921c229e009ab6bb30063305e
blaze_face.onnx	952,329,216	418,456	9b217128f6b66f7e11dc3673c22be250d72f68f507430817d0754ef1451eb609
face_mesh.onnx	952,748,544	4,958,394	bbbb49fac007b7cde79b478c680d171969f23f53ae6cdaf6a3b4264fa18c8479
mel_filters_201x80.f32	957,707,776	64,320	039ce818842793b3547f84ca19dbf32488538b14016d3cb259342ed6667903c4
vae_encoder_fp16.onnx	957,772,800	68,427,649	56c9eef60daa97388b83321680f3cb48128016b735eb40e1d1349c4a535503be
whisper_tiny_encoder_fp16.onnx	1,026,201,088	16,454,911	fd0fd8d357a62e6a1c0a1a333f4b0acc5e139e858edc9d2f7f2de380856f0c38

• Check a file on Windows with Get-FileHash <file> -Algorithm SHA256 (PowerShell) or certutil -hashfile <file> SHA256 (Command Prompt); with sha256sum <file> on Linux, and shasum -a 256 <file> on macOS.
• tar -tvf ngo-media-assets-v2.tar lists what is inside, and tar -xf ngo-media-assets-v2.tar unpacks it into the current folder (tar comes with Windows 10 and 11). Every entry carries the same fixed date and owner, so the pack is built byte for byte the same every time.
• One file alone can be fetched by its bytes, as the app does: from its offset to its offset + size - 1. For blaze_face.onnx: curl -L -r 952329216-952747671 -o blaze_face.onnx https://huggingface.co/nextgenomni/ngo-media-assets/resolve/main/ngo-media-assets-v2.tar
• The two MediaPipe files can be compared with Ditto's copies, and the others' originals are linked above.

Use policy

This is a request from NextGenOmni LLC, not a licence term. The files in this pack are licensed under the MIT License and the Apache License 2.0, and those licences decide what you may legally do with them. This policy adds no condition to them and takes away none of the rights they give. We ask it because a video of a real person saying words they never said can be used to hurt people, and we want this pack never to be. Laws apply on their own, whatever this policy says: item 3 names one that is a requirement, not a request.

Whoever you are, and whatever app you use it in, please:

1. Only use videos of people who agreed. Yourself, or someone who gave you clear permission to have their lips made to say other words (for a child: a parent or guardian). Reelquill AI asks you to confirm this for every video you add, and keeps your note of who it is with it.
2. Don't impersonate or deceive. Don't pass a video made with this pack off as a real recording of what a real person said, and don't use one to put words in someone's mouth, fake an endorsement, scam, defraud, harass, or mislead people about what someone said or did.
3. Label AI-made video. In places it is the law: the EU AI Act (Article 50(4)) requires anyone who uses an AI system to make a deep fake (image, audio or video that resembles real people, objects, places or events and would falsely seem authentic), other than in a purely personal, non-professional activity, to disclose that it was artificially made or manipulated. Other places have similar rules. When in doubt, label it. Reelquill AI marks every video with its presenter as AI-generated inside the file, and can also show an "AI-generated" label on the picture.
4. Nothing illegal. No use that breaks the law where you are or where the video is seen: privacy and image rights, defamation, fraud, election rules, intimate images made without consent.

To report misuse, or a problem with these files, open a discussion in this repo's Community tab.

Licences

The Hugging Face licence field holds one licence, so it says "other": the pack is a mix, and each file keeps its own licence (the table above).

• MIT License: LICENSE-MIT-pack-v2.txt, with the three copyright lines it applies to.
  • MuseTalk 1.5's UNet: Copyright (c) 2024 Tencent Music Entertainment Group. MuseTalk's code is under the MIT License, and its README says its trained models may be used for any purpose, commercial use included. unet_int8.onnx is a modified copy, as NOTICE-pack-v2.txt says.
  • sd-vae-ft-mse: Copyright (c) 2022 Stability AI, MIT on its model card. The two VAE files are modified copies.
  • Whisper: Copyright (c) 2022 OpenAI, which releases Whisper's code and model weights under the MIT License. The Hugging Face copy, openai/whisper-tiny, is labelled Apache 2.0, whose text is here too. whisper_tiny_encoder_fp16.onnx is a modified copy.
• Apache License 2.0: MediaPipe's BlazeFace and face mesh (Copyright The MediaPipe Authors; ONNX as Ant Group published them with Ditto, unchanged), and the mel filters as Hugging Face transformers computes them: LICENSE-Apache-2.0.txt.
• NextGenOmni LLC's changes (four networks exported to ONNX and stored as float16 or 8-bit, Whisper's mel filters saved as a file, nothing retrained) are listed in NOTICE-pack-v2.txt. They are offered under the licence of the file each was made from, and add no terms.

Not included:

• MuseTalk's face-parsing network (BiSeNet, face-parse-bisent), which MuseTalk uses for the mask of the face. Its weights come from a dataset licensed for non-commercial research only, so the mask is made from face_mesh.onnx's points instead.
• The other models MuseTalk uses to train or to find its own face points (SyncNet, VGG, DWPose, S3FD), and MuseTalk's test data, which is for non-commercial research only. The face's points here are MediaPipe's.

Credits

• MuseTalk: MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling. Yue Zhang, Zhizhou Zhong, Minhao Liu, Zhaokang Chen, Bin Wu, Yubin Zeng, Chao Zhan, Yingjie He, Junxin Huang, Wenjiang Zhou. Lyra Lab, Tencent Music Entertainment. Technical report · code · models
• sd-vae-ft-mse: Stability AI's fine-tune of the kl-f8 autoencoder from Latent Diffusion (CompVis), the VAE MuseTalk works in. model
• Whisper: Robust Speech Recognition via Large-Scale Weak Supervision. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever. OpenAI. Paper · code · model
• transformers: Hugging Face. Its WhisperFeatureExtractor computed the mel filters. code
• MediaPipe: Google. Face Landmarker · code
• Ditto (Ant Group): its release has the ONNX copies of MediaPipe's two models used here. models

The people and companies named here are credited for their work. They don't endorse Reelquill AI or this pack, and their names and marks are not licensed for any other use.

Citations

@article{musetalk,
  title   = {MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling},
  author  = {Zhang, Yue and Zhong, Zhizhou and Liu, Minhao and Chen, Zhaokang and Wu, Bin and Zeng, Yubin and Zhan, Chao and He, Yingjie and Huang, Junxin and Zhou, Wenjiang},
  journal = {arXiv preprint arXiv:2410.10122},
  year    = {2025}
}

@article{radford2022whisper,
  title   = {Robust Speech Recognition via Large-Scale Weak Supervision},
  author  = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal = {arXiv preprint arXiv:2212.04356},
  year    = {2022}
}

Pack 1, for older versions of the app

ngo-media-assets-v1.tar and its manifest ngo-media-assets-v1.json stay in this repo, unchanged, for the versions of Reelquill AI whose presenter is made from one photo (Ditto, LivePortrait, HuBERT and MediaPipe; MIT and Apache 2.0). Its own page, with what is in it, its credits and its licences: Pack 1's page (commit ed59509). Its notice and licence texts stay here too: NOTICE-Reelquill.txt, NOTICE-voxta-ditto-talkinghead-onnx.txt, LICENSE-MIT-LivePortrait.txt, LICENSE-MIT-fairseq.txt and LICENSE-Apache-2.0.txt. The use policy above applies to it too. Of the model files, only blaze_face.onnx and face_mesh.onnx are in both packs.

Made by

NextGenOmni LLC, the maker of Reelquill AI. nextgenomni.com · Reelquill AI