# TVP [[tvp]] ## 개요 [[overview]] Text-Visual Prompting(TVP) 프레임워크는 Yimeng Zhang, Xin Chen, Jinghan Jia, Sijia Liu, Ke Ding이 발표한 논문 [Text-Visual Prompting for Efficient 2D Temporal Video Grounding](https://huggingface.co/papers/2303.04995)에서 제안되었습니다. 논문의 초록은 다음과 같습니다: *본 논문에서는 길고, 편집되지 않은 비디오에서 문장으로 설명된 순간의 시작/종료 시점을 예측하는 것을 목표로 하는 Temporal Video Grounding(TVG) 문제를 다룹니다. 세밀한 3D 시각적 특징 덕분에 TVG 기술은 최근 몇 년 동안 놀라운 발전을 이뤘습니다. 하지만 3D 합성곱 신경망(CNN)의 높은 복잡성으로 인해 밀도 높은 3D 시각적 특징을 추출하는 데 시간이 오래 걸리고 그만큼 많은 메모리와 연산 자원을 필요로 합니다. 효율적인 TVG를 위해, 본 논문에서는 TVG 모델의 시각적 입력과 텍스트 특징 모두에 최적화된 교란 패턴('프롬프트'라고 부름)을 통합하는 새로운 Text-Visual Prompting(TVP) 프레임워크를 제안합니다. 3D CNN과 뚜렷이 대비되게 TVP가 2D TVG 모델에서 비전 인코더와 언어 인코더를 효과적으로 공동 학습할 수 있게 하고, 낮은 복잡도의 희소한 2D 시각적 특징만을 사용하여 크로스 모달 특징 융합의 성능을 향상시킵니다. 더 나아가, TVG의 효율적인 학습을 위해 Temporal-Distance IoU(TDIoU) 손실 함수를 제안합니다. 두 개의 벤치마크 데이터 세트인 Charades-STA와 ActivityNet Captions 데이터셋에 대한 실험을 통해, 제안된 TVP가 2D TVG의 성능을 크게 향상시키고(예: Charades-STA에서 9.79% 향상, ActivityNet Captions에서 30.77% 향상) 3D 시각적 특징을 사용하는 TVG에 비해 5배의 추론 가속을 달성함을 실험적으로 입증합니다.* 이 연구는 Temporal Video Grounding(TVG)을 다룹니다. TVG는 문장으로 설명된 특정 이벤트의 시작 및 종료 시점을 긴 비디오에서 정확히 찾아내는 과정입니다. TVG 성능을 향상시키기 위해 Text-Visual Prompting(TVP)이 제안되었습니다. TVP는 '프롬프트'라고 알려진 특별히 설계된 패턴을 TVG 모델의 시각적(이미지 기반) 및 텍스트(단어 기반) 입력 구성 요소 모두에 통합하는 것을 방식입니다. 이 프롬프트는 추가적인 시공간적 컨텍스트를 제공함으로써 모델이 비디오 내 이벤트 시점의 예측 정확도를 높입니다. 이 접근 방식은 3D 시각적 입력 대신 2D 입력을 사용합니다. 3D 입력은 보다 풍부한 시공간적 세부 정보를 제공하지만 처리하는 데 시간이 더 많이 걸립니다. 따라서 프롬프팅 메소드와 함께 2D 입력을 사용하여 이와 유사한 수준의 컨텍스트와 정확도를 더 효율적으로 제공하는 것을 목표로 합니다. TVP 아키텍처. 원본 논문에서 발췌. 이 모델은 [Jiqing Feng](https://huggingface.co/Jiqing)님이 기여했습니다. 원본 코드는 [이 곳](https://github.com/intel/TVP)에서 찾을 수 있습니다. ## 사용 팁 및 예시 [[usage-tips-and-examples]] 프롬프트는 최적화된 교란 패턴으로 입력 비디오 프레임이나 텍스트 특징에 추가되는 패턴입니다. 범용 세트란 모든 입력에 대해 동일한 프롬프트 세트를 사용하는 것을 말합니다. 즉, 입력 내용과 관계없이 모든 비디오 프레임과 텍스트 특징에 이 프롬프트들을 일관적으로 추가합니다. TVP는 시각 인코더와 크로스 모달 인코더로 구성됩니다. 범용 시각 프롬프트와 텍스트 프롬프트 세트가 각각 샘플링된 비디오 프레임과 텍스트 특징에 통합됩니다. 특히, 서로 다른 시각 프롬프트 세트가 편집되지 않은 한 비디오에서 균일하게 샘플링된 프레임에 순서대로 적용됩니다. 이 모델의 목표는 학습 가능한 프롬프트를 시각적 입력과 텍스트 특징 모두에 통합하여 Temporal Video Grounding(TVG) 문제를 해결하는 것입니다. 원칙적으로, 제안된 아키텍처에는 어떤 시각 인코더나 크로스 모달 인코더라도 적용할 수 있습니다. [TvpProcessor]는 [BertTokenizer]와 [TvpImageProcessor]를 단일 인스턴스로 래핑하여 텍스트를 인코딩하고 이미지를 각각 준비합니다. 다음 예시는 [TvpProcessor]와 [TvpForVideoGrounding]을 사용하여 TVG를 실행하는 방법을 보여줍니다. ```python import av import cv2 import numpy as np import torch from huggingface_hub import hf_hub_download from transformers import AutoProcessor, TvpForVideoGrounding def pyav_decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps): ''' 원본 fps의 비디오를 지정한 fps(target_fps)로 변환하고 PyAV 디코더로 비디오를 디코딩합니다. Args: container (container): pyav 컨테이너 객체입니다. sampling_rate (int): 프레임 샘플링 속도입니다.(샘플링된 두개의 프레임 사이의 간격을 말합니다) num_frames (int): 샘플링할 프레임 수입니다. clip_idx (int): clip_idx가 -1이면 시간 축에서 무작위 샘플링을 수행합니다. clip_idx가 -1보다 크면 비디오를 num_clips 개로 균등 분할한 후 clip_idx번째 비디오 클립을 선택합니다. num_clips (int): 주어진 비디오에서 균일하게 샘플링할 전체 클립 수입니다. target_fps (int): 입력 비디오의 fps가 다를 수 있으므로, 샘플링 전에 지정한 fps로 변환합니다 Returns: frames (tensor): 비디오에서 디코딩된 프레임입니다. 비디오 스트림을 찾을 수 없는 경우 None을 반환합니다. fps (float): 비디오의 초당 프레임 수입니다. ''' video = container.streams.video[0] fps = float(video.average_rate) clip_size = sampling_rate * num_frames / target_fps * fps delta = max(num_frames - clip_size, 0) start_idx = delta * clip_idx / num_clips end_idx = start_idx + clip_size - 1 timebase = video.duration / num_frames video_start_pts = int(start_idx * timebase) video_end_pts = int(end_idx * timebase) seek_offset = max(video_start_pts - 1024, 0) container.seek(seek_offset, any_frame=False, backward=True, stream=video) frames = {} for frame in container.decode(video=0): if frame.pts video_end_pts: break frames = [frames[pts] for pts in sorted(frames)] return frames, fps def decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps): ''' 비디오를 디코딩하고 시간 축 샘플링을 수행합니다. Args: container (container): pyav 컨테이너 객체입니다. sampling_rate (int): 프레임 샘플링 속도입니다.(샘플링된 두개의 프레임 사이의 간격을 말합니다) num_frames (int): 샘플링할 프레임 수입니다. clip_idx (int): clip_idx가 -1이면 시간 축에서 무작위 샘플링을 수행합니다. clip_idx가 -1보다 크면 비디오를 num_clips 개로 균등 분할한 후 clip_idx번째 비디오 클립을 선택합니다. num_clips (int): 주어진 비디오에서 균일하게 샘플링할 전체 클립 수입니다. target_fps (int): 입력 비디오의 fps가 다를 수 있으므로, 샘플링 전에 지정한 fps로 변환합니다 Returns: frames (tensor): 비디오에서 디코딩된 프레임입니다. ''' assert clip_idx >= -2, "Not a valid clip_idx {}".format(clip_idx) frames, fps = pyav_decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps) clip_size = sampling_rate * num_frames / target_fps * fps index = np.linspace(0, clip_size - 1, num_frames) index = np.clip(index, 0, len(frames) - 1).astype(np.int64) frames = np.array([frames[idx].to_rgb().to_ndarray() for idx in index]) frames = frames.transpose(0, 3, 1, 2) return frames file = hf_hub_download(repo_id="Intel/tvp_demo", filename="AK2KG.mp4", repo_type="dataset") model = TvpForVideoGrounding.from_pretrained("Intel/tvp-base") decoder_kwargs = dict( container=av.open(file, metadata_errors="ignore"), sampling_rate=1, num_frames=model.config.num_frames, clip_idx=0, num_clips=1, target_fps=3, ) raw_sampled_frms = decode(**decoder_kwargs) text = "a person is sitting on a bed." processor = AutoProcessor.from_pretrained("Intel/tvp-base") model_inputs = processor( text=[text], videos=list(raw_sampled_frms), return_tensors="pt", max_text_length=100#, size=size ) model_inputs["pixel_values"] = model_inputs["pixel_values"].to(model.dtype) output = model(**model_inputs) def get_video_duration(filename): cap = cv2.VideoCapture(filename) if cap.isOpened(): rate = cap.get(5) frame_num = cap.get(7) duration = frame_num/rate return duration return -1 duration = get_video_duration(file) start, end = processor.post_process_video_grounding(output.logits, duration) print(f"The time slot of the video corresponding to the text \"{text}\" is from {start}s to {end}s") ``` 팁: - 이 TVP 구현은 텍스트 임베딩을 생성하기 위해 [BertTokenizer]를 사용하고, 시각적 임베딩을 계산하기 위해 Resnet-50 모델을 사용합니다. - 사전 학습된 [tvp-base](https://huggingface.co/Intel/tvp-base)의 체크포인트가 공개되어 있습니다. - 시간적 비디오 그라운딩 작업에 대한 TVP의 성능은 [표 2](https://huggingface.co/papers/2303.04995)를 참고하세요. ## TvpConfig [[transformers.TvpConfig]][[transformers.TvpConfig]] #### transformers.TvpConfig[[transformers.TvpConfig]] [Source](https://github.com/huggingface/transformers/blob/v5.0.0rc2/src/transformers/models/tvp/configuration_tvp.py#L26) This is the configuration class to store the configuration of a [TvpModel](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpModel). It is used to instantiate an Tvp model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Tvp [Intel/tvp-base](https://huggingface.co/Intel/tvp-base) architecture. Configuration objects inherit from [PreTrainedConfig](/docs/transformers/v5.0.0rc2/ko/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the documentation from [PreTrainedConfig](/docs/transformers/v5.0.0rc2/ko/main_classes/configuration#transformers.PreTrainedConfig) for more information. **Parameters:** backbone_config (`Union[dict, "PreTrainedConfig"]`, *optional*, defaults to `ResNetConfig()`) : The configuration of the backbone model. backbone (`str`, *optional*) : Name of backbone to use when `backbone_config` is `None`. If `use_pretrained_backbone` is `True`, this will load the corresponding pretrained weights from the timm or transformers library. If `use_pretrained_backbone` is `False`, this loads the backbone's config and uses that to initialize the backbone with random weights. use_pretrained_backbone (`bool`, *optional*, defaults to `False`) : Whether to use pretrained weights for the backbone. use_timm_backbone (`bool`, *optional*, defaults to `False`) : Whether to load `backbone` from the timm library. If `False`, the backbone is loaded from the transformers library. backbone_kwargs (`dict`, *optional*) : Keyword arguments to be passed to AutoBackbone when loading from a checkpoint e.g. `{'out_indices': (0, 1, 2, 3)}`. Cannot be specified if `backbone_config` is set. distance_loss_weight (`float`, *optional*, defaults to 1.0) : The weight of distance loss. duration_loss_weight (`float`, *optional*, defaults to 0.1) : The weight of duration loss. visual_prompter_type (`str`, *optional*, defaults to `"framepad"`) : Visual prompt type. The type of padding. Framepad means padding on each frame. Should be one of "framepad" or "framedownpad" visual_prompter_apply (`str`, *optional*, defaults to `"replace"`) : The way of applying visual prompt. Replace means use the value of prompt to change the original value in visual inputs. Should be one of "replace", or "add", or "remove". visual_prompt_size (`int`, *optional*, defaults to 96) : The size of visual prompt. max_img_size (`int`, *optional*, defaults to 448) : The maximum size of frame. num_frames (`int`, *optional*, defaults to 48) : The number of frames extracted from a video. vocab_size (`int`, *optional*, defaults to 30522) : Vocabulary size of the Tvp text model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [TvpModel](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpModel). type_vocab_size (`int`, *optional*, defaults to 2) : The vocabulary size of the `token_type_ids` passed when calling [TvpModel](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpModel). hidden_size (`int`, *optional*, defaults to 768) : Dimensionality of the encoder layers. intermediate_size (`int`, *optional*, defaults to 3072) : Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. num_hidden_layers (`int`, *optional*, defaults to 12) : Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 12) : Number of attention heads for each attention layer in the Transformer encoder. max_position_embeddings (`int`, *optional*, defaults to 512) : The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). max_grid_col_position_embeddings (`int`, *optional*, defaults to 100) : The largest number of horizontal patches from a video frame. max_grid_row_position_embeddings (`int`, *optional*, defaults to 100) : The largest number of vertical patches from a video frame. hidden_dropout_prob (`float`, *optional*, defaults to 0.1) : The dropout probability of hidden layers. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`) : The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported. layer_norm_eps (`float`, *optional*, defaults to 1e-12) : The epsilon used by the layer normalization layers. initializer_range (`float`, *optional*, defaults to 0.02) : The standard deviation of the truncated_normal_initializer for initializing all weight matrices. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1) : The dropout probability of attention layers. ## TvpImageProcessor [[transformers.TvpImageProcessor]][[transformers.TvpImageProcessor]] #### transformers.TvpImageProcessor[[transformers.TvpImageProcessor]] [Source](https://github.com/huggingface/transformers/blob/v5.0.0rc2/src/transformers/models/tvp/image_processing_tvp.py#L101) Constructs a Tvp image processor. preprocesstransformers.TvpImageProcessor.preprocesshttps://github.com/huggingface/transformers/blob/v5.0.0rc2/src/transformers/models/tvp/image_processing_tvp.py#L355[{"name": "videos", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], list[typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]], list[list[typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]]]]"}, {"name": "do_resize", "val": ": typing.Optional[bool] = None"}, {"name": "size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "resample", "val": ": typing.Optional[PIL.Image.Resampling] = None"}, {"name": "do_center_crop", "val": ": typing.Optional[bool] = None"}, {"name": "crop_size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "do_rescale", "val": ": typing.Optional[bool] = None"}, {"name": "rescale_factor", "val": ": typing.Optional[float] = None"}, {"name": "do_pad", "val": ": typing.Optional[bool] = None"}, {"name": "pad_size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "constant_values", "val": ": typing.Union[float, collections.abc.Iterable[float], NoneType] = None"}, {"name": "pad_mode", "val": ": typing.Optional[transformers.image_transforms.PaddingMode] = None"}, {"name": "do_normalize", "val": ": typing.Optional[bool] = None"}, {"name": "do_flip_channel_order", "val": ": typing.Optional[bool] = None"}, {"name": "image_mean", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "image_std", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "return_tensors", "val": ": typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None"}, {"name": "data_format", "val": ": ChannelDimension = "}, {"name": "input_data_format", "val": ": typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None"}]- **videos** (`ImageInput` or `list[ImageInput]` or `list[list[ImageInput]]`) -- Frames to preprocess. - **do_resize** (`bool`, *optional*, defaults to `self.do_resize`) -- Whether to resize the image. - **size** (`dict[str, int]`, *optional*, defaults to `self.size`) -- Size of the image after applying resize. - **resample** (`PILImageResampling`, *optional*, defaults to `self.resample`) -- Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`, Only has an effect if `do_resize` is set to `True`. - **do_center_crop** (`bool`, *optional*, defaults to `self.do_centre_crop`) -- Whether to centre crop the image. - **crop_size** (`dict[str, int]`, *optional*, defaults to `self.crop_size`) -- Size of the image after applying the centre crop. - **do_rescale** (`bool`, *optional*, defaults to `self.do_rescale`) -- Whether to rescale the image values between [0 - 1]. - **rescale_factor** (`float`, *optional*, defaults to `self.rescale_factor`) -- Rescale factor to rescale the image by if `do_rescale` is set to `True`. - **do_pad** (`bool`, *optional*, defaults to `True`) -- Whether to pad the image. Can be overridden by the `do_pad` parameter in the `preprocess` method. - **pad_size** (`dict[str, int]`, *optional*, defaults to `{"height" -- 448, "width": 448}`): Size of the image after applying the padding. Can be overridden by the `pad_size` parameter in the `preprocess` method. - **constant_values** (`Union[float, Iterable[float]]`, *optional*, defaults to 0) -- The fill value to use when padding the image. - **pad_mode** (`PaddingMode`, *optional*, defaults to "PaddingMode.CONSTANT") -- Use what kind of mode in padding. - **do_normalize** (`bool`, *optional*, defaults to `self.do_normalize`) -- Whether to normalize the image. - **do_flip_channel_order** (`bool`, *optional*, defaults to `self.do_flip_channel_order`) -- Whether to flip the channel order of the image. - **image_mean** (`float` or `list[float]`, *optional*, defaults to `self.image_mean`) -- Image mean. - **image_std** (`float` or `list[float]`, *optional*, defaults to `self.image_std`) -- Image standard deviation. - **return_tensors** (`str` or `TensorType`, *optional*) -- The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - **data_format** (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`) -- The channel dimension format for the output image. Can be one of: - `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the inferred channel dimension format of the input image. - **input_data_format** (`ChannelDimension` or `str`, *optional*) -- The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.0 Preprocess an image or batch of images. **Parameters:** do_resize (`bool`, *optional*, defaults to `True`) : Whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by the `do_resize` parameter in the `preprocess` method. size (`dict[str, int]` *optional*, defaults to `{"longest_edge" : 448}`): Size of the output image after resizing. The longest edge of the image will be resized to `size["longest_edge"]` while maintaining the aspect ratio of the original image. Can be overridden by `size` in the `preprocess` method. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BILINEAR`) : Resampling filter to use if resizing the image. Can be overridden by the `resample` parameter in the `preprocess` method. do_center_crop (`bool`, *optional*, defaults to `True`) : Whether to center crop the image to the specified `crop_size`. Can be overridden by the `do_center_crop` parameter in the `preprocess` method. crop_size (`dict[str, int]`, *optional*, defaults to `{"height" : 448, "width": 448}`): Size of the image after applying the center crop. Can be overridden by the `crop_size` parameter in the `preprocess` method. do_rescale (`bool`, *optional*, defaults to `True`) : Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by the `do_rescale` parameter in the `preprocess` method. rescale_factor (`int` or `float`, *optional*, defaults to `1/255`) : Defines the scale factor to use if rescaling the image. Can be overridden by the `rescale_factor` parameter in the `preprocess` method. do_pad (`bool`, *optional*, defaults to `True`) : Whether to pad the image. Can be overridden by the `do_pad` parameter in the `preprocess` method. pad_size (`dict[str, int]`, *optional*, defaults to `{"height" : 448, "width": 448}`): Size of the image after applying the padding. Can be overridden by the `pad_size` parameter in the `preprocess` method. constant_values (`Union[float, Iterable[float]]`, *optional*, defaults to 0) : The fill value to use when padding the image. pad_mode (`PaddingMode`, *optional*, defaults to `PaddingMode.CONSTANT`) : Use what kind of mode in padding. do_normalize (`bool`, *optional*, defaults to `True`) : Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess` method. do_flip_channel_order (`bool`, *optional*, defaults to `True`) : Whether to flip the color channels from RGB to BGR. Can be overridden by the `do_flip_channel_order` parameter in the `preprocess` method. image_mean (`float` or `list[float]`, *optional*, defaults to `IMAGENET_STANDARD_MEAN`) : Mean to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method. image_std (`float` or `list[float]`, *optional*, defaults to `IMAGENET_STANDARD_STD`) : Standard deviation to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method. ## TvpProcessor [[transformers.TvpProcessor]][[transformers.TvpProcessor]] #### transformers.TvpProcessor[[transformers.TvpProcessor]] [Source](https://github.com/huggingface/transformers/blob/v5.0.0rc2/src/transformers/models/tvp/processing_tvp.py#L33) Constructs an TVP processor which wraps a TVP image processor and a Bert tokenizer into a single processor. [TvpProcessor](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpProcessor) offers all the functionalities of [TvpImageProcessor](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpImageProcessor) and [BertTokenizerFast](/docs/transformers/v5.0.0rc2/ko/model_doc/electra#transformers.BertTokenizer). See the [__call__()](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpProcessor.__call__) and [decode()](/docs/transformers/v5.0.0rc2/ko/main_classes/processors#transformers.ProcessorMixin.decode) for more information. __call__transformers.TvpProcessor.__call__https://github.com/huggingface/transformers/blob/v5.0.0rc2/src/transformers/processing_utils.py#L617[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = None"}, {"name": "text", "val": ": typing.Union[str, list[str], list[list[str]], NoneType] = None"}, {"name": "videos", "val": ": typing.Union[list['PIL.Image.Image'], numpy.ndarray, ForwardRef('torch.Tensor'), list[numpy.ndarray], list['torch.Tensor'], list[list['PIL.Image.Image']], list[list[numpy.ndarray]], list[list['torch.Tensor']], transformers.video_utils.URL, list[transformers.video_utils.URL], list[list[transformers.video_utils.URL]], transformers.video_utils.Path, list[transformers.video_utils.Path], list[list[transformers.video_utils.Path]], NoneType] = None"}, {"name": "audio", "val": ": typing.Union[numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor'], NoneType] = None"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.processing_utils.ProcessingKwargs]"}]- **images** (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`) -- The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. Both channels-first and channels-last formats are supported. - **text** (`TextInput`, `PreTokenizedInput`, `list[TextInput]`, `list[PreTokenizedInput]`, *optional*) -- The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). - **videos** (`np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`) -- The video or batch of videos to be prepared. Each video can be a 4D NumPy array or PyTorch tensor, or a nested list of 3D frames. Both channels-first and channels-last formats are supported. - **audio** (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`) -- The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch tensor. - **return_tensors** (`str` or [TensorType](/docs/transformers/v5.0.0rc2/ko/internal/file_utils#transformers.TensorType), *optional*) -- If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects.0[BatchFeature](/docs/transformers/v5.0.0rc2/ko/main_classes/feature_extractor#transformers.BatchFeature)A [BatchFeature](/docs/transformers/v5.0.0rc2/ko/main_classes/feature_extractor#transformers.BatchFeature) object with processed inputs in a dict format. Main method to prepare for model inputs. This method forwards the each modality argument to its own processor along with `kwargs`. Please refer to the docstring of the each processor attributes for more information. **Parameters:** image_processor ([TvpImageProcessor](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpImageProcessor), *optional*) : The image processor is a required input. tokenizer ([BertTokenizerFast](/docs/transformers/v5.0.0rc2/ko/model_doc/electra#transformers.BertTokenizer), *optional*) : The tokenizer is a required input. **Returns:** `[BatchFeature](/docs/transformers/v5.0.0rc2/ko/main_classes/feature_extractor#transformers.BatchFeature)` A [BatchFeature](/docs/transformers/v5.0.0rc2/ko/main_classes/feature_extractor#transformers.BatchFeature) object with processed inputs in a dict format. ## TvpModel [[transformers.TvpModel]][[transformers.TvpModel]] #### transformers.TvpModel[[transformers.TvpModel]] [Source](https://github.com/huggingface/transformers/blob/v5.0.0rc2/src/transformers/models/tvp/modeling_tvp.py#L691) The bare Tvp Model transformer outputting BaseModelOutputWithPooling object without any specific head on top. This model inherits from [PreTrainedModel](/docs/transformers/v5.0.0rc2/ko/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. forwardtransformers.TvpModel.forwardhttps://github.com/huggingface/transformers/blob/v5.0.0rc2/src/transformers/models/tvp/modeling_tvp.py#L714[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}, {"name": "interpolate_pos_encoding", "val": ": bool = False"}, {"name": "**kwargs", "val": ""}]- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. Indices can be obtained using [AutoTokenizer](/docs/transformers/v5.0.0rc2/ko/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v5.0.0rc2/ko/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.__call__()](/docs/transformers/v5.0.0rc2/ko/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. [What are input IDs?](../glossary#input-ids) - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- The tensors corresponding to the input images. Pixel values can be obtained using [TvpImageProcessor](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpImageProcessor). See `TvpImageProcessor.__call__()` for details ([TvpProcessor](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpProcessor) uses [TvpImageProcessor](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpImageProcessor) for processing images). - **attention_mask** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) - **output_attentions** (`bool`, *optional*) -- Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. - **output_hidden_states** (`bool`, *optional*) -- Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. - **return_dict** (`bool`, *optional*) -- Whether or not to return a [ModelOutput](/docs/transformers/v5.0.0rc2/ko/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple. - **interpolate_pos_encoding** (`bool`, *optional*, defaults to `False`) -- Whether to interpolate the pre-trained position encodings.0[transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/v5.0.0rc2/ko/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`A [transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/v5.0.0rc2/ko/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([TvpConfig](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpConfig)) and inputs. - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining. - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. The [TvpModel](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpModel) forward method, overrides the `__call__` special method. Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them. Examples: ```python >>> import torch >>> from transformers import AutoConfig, AutoTokenizer, TvpModel >>> model = TvpModel.from_pretrained("Jiqing/tiny-random-tvp") >>> tokenizer = AutoTokenizer.from_pretrained("Jiqing/tiny-random-tvp") >>> pixel_values = torch.rand(1, 1, 3, 448, 448) >>> text_inputs = tokenizer("This is an example input", return_tensors="pt") >>> output = model(text_inputs.input_ids, pixel_values, text_inputs.attention_mask) ``` **Parameters:** config ([TvpModel](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpModel)) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.0.0rc2/ko/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. **Returns:** `[transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/v5.0.0rc2/ko/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`` A [transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/v5.0.0rc2/ko/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([TvpConfig](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpConfig)) and inputs. - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining. - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. ## TvpForVideoGrounding [[transformers.TvpForVideoGrounding]][[transformers.TvpForVideoGrounding]] #### transformers.TvpForVideoGrounding[[transformers.TvpForVideoGrounding]] [Source](https://github.com/huggingface/transformers/blob/v5.0.0rc2/src/transformers/models/tvp/modeling_tvp.py#L806) Tvp Model with a video grounding head on top computing IoU, distance, and duration loss. This model inherits from [PreTrainedModel](/docs/transformers/v5.0.0rc2/ko/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. forwardtransformers.TvpForVideoGrounding.forwardhttps://github.com/huggingface/transformers/blob/v5.0.0rc2/src/transformers/models/tvp/modeling_tvp.py#L815[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "labels", "val": ": typing.Optional[tuple[torch.Tensor]] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}, {"name": "interpolate_pos_encoding", "val": ": bool = False"}, {"name": "**kwargs", "val": ""}]- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. Indices can be obtained using [AutoTokenizer](/docs/transformers/v5.0.0rc2/ko/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v5.0.0rc2/ko/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.__call__()](/docs/transformers/v5.0.0rc2/ko/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. [What are input IDs?](../glossary#input-ids) - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- The tensors corresponding to the input images. Pixel values can be obtained using [TvpImageProcessor](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpImageProcessor). See `TvpImageProcessor.__call__()` for details ([TvpProcessor](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpProcessor) uses [TvpImageProcessor](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpImageProcessor) for processing images). - **attention_mask** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) - **labels** (`torch.FloatTensor` of shape `(batch_size, 3)`, *optional*) -- The labels contains duration, start time, and end time of the video corresponding to the text. - **output_attentions** (`bool`, *optional*) -- Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. - **output_hidden_states** (`bool`, *optional*) -- Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. - **return_dict** (`bool`, *optional*) -- Whether or not to return a [ModelOutput](/docs/transformers/v5.0.0rc2/ko/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple. - **interpolate_pos_encoding** (`bool`, *optional*, defaults to `False`) -- Whether to interpolate the pre-trained position encodings.0`transformers.models.tvp.modeling_tvp.TvpVideoGroundingOutput` or `tuple(torch.FloatTensor)`A `transformers.models.tvp.modeling_tvp.TvpVideoGroundingOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([TvpConfig](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpConfig)) and inputs. - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`) -- Temporal-Distance IoU loss for video grounding. - **logits** (`torch.FloatTensor` of shape `(batch_size, 2)`) -- Contains start_time/duration and end_time/duration. It is the time slot of the videos corresponding to the input texts. - **hidden_states** (`tuple[torch.FloatTensor, ...]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. The [TvpForVideoGrounding](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpForVideoGrounding) forward method, overrides the `__call__` special method. Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them. Examples: ```python >>> import torch >>> from transformers import AutoConfig, AutoTokenizer, TvpForVideoGrounding >>> model = TvpForVideoGrounding.from_pretrained("Jiqing/tiny-random-tvp") >>> tokenizer = AutoTokenizer.from_pretrained("Jiqing/tiny-random-tvp") >>> pixel_values = torch.rand(1, 1, 3, 448, 448) >>> text_inputs = tokenizer("This is an example input", return_tensors="pt") >>> output = model(text_inputs.input_ids, pixel_values, text_inputs.attention_mask) ``` **Parameters:** config ([TvpForVideoGrounding](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpForVideoGrounding)) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.0.0rc2/ko/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. **Returns:** ``transformers.models.tvp.modeling_tvp.TvpVideoGroundingOutput` or `tuple(torch.FloatTensor)`` A `transformers.models.tvp.modeling_tvp.TvpVideoGroundingOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([TvpConfig](/docs/transformers/v5.0.0rc2/ko/model_doc/tvp#transformers.TvpConfig)) and inputs. - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`) -- Temporal-Distance IoU loss for video grounding. - **logits** (`torch.FloatTensor` of shape `(batch_size, 2)`) -- Contains start_time/duration and end_time/duration. It is the time slot of the videos corresponding to the input texts. - **hidden_states** (`tuple[torch.FloatTensor, ...]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.