SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification
Paper
• 2305.09781 • Published
• 4
Papers - TensorRT-LLM - FasterTransformer - deprecated
Note https://github.com/NVIDIA/FasterTransformer readme states: FasterTransformer development has transitioned to TensorRT-LLM. All developers are encouraged to leverage TensorRT-LLM to get the latest improvements on LLM Inference. The NVIDIA/FasterTransformer repo will stay up, but will not have further development.