HMTV: Hierarchical Multimodal Transformer for Video Highlight Query on Baseball

doi:10.1007/s00530-024-01479-6

淡江大學機構典藏 > 工學院 > 資訊工程學系暨研究所 > 期刊論文 > Item 987654321/126772

jsp.display-item.identifier=請使用永久網址來引用或連結此文件: https://tkuir.lib.tku.edu.tw/dspace/handle/987654321/126772

题名:	HMTV: Hierarchical Multimodal Transformer for Video Highlight Query on Baseball
作者:	Zhang, Qiaoyun;Chang, Chih-Yung;Su, Ming-Yang;Chang, Hsiang-Chuan;Roy, Diptendu Sinha
关键词:	分層多模態 Transformer;BERT;反白顯示查詢
日期:	2024-09-23
上传时间:	2025-03-20 09:24:10 (UTC+8)
摘要:	With the increasing popularity of watching baseball videos, there is a growing desire among fans to enjoy the highlights of these videos. However, the extraction of the highlights from lengthy baseball videos faces a significant challenge due to its time-consuming and labor-intensive nature. To address this challenge, this paper proposes a novel mechanism, called Hierarchical Multimodal Transformer for Video query (HMTV). The proposed HMTV incorporates a two-phase involving Coarse-Grained clipping for candidate videos and Fine-Grained identification for highlights. In the Coarse-Grained phase, a pitching detection model is employed to extract relevant candidate videos from baseball videos, encompassing the features of pitch deliveries and pitching. In the Fine-Grained phase, Transformer encoder and pre-trained Bidirectional Encoder Representations from Transformers (BERT) are utilized to capture relationship features between frames of candidate videos and words from users’ questions, respectively. These relationship features are then fed into the Video Query (VideoQ) model, implemented by the Text Video Attention (TVA). The VideoQ model identifies the start and end positions of the highlights mentioned in the query within the candidate videos. Simulation results demonstrate that the proposed HMTV significantly improves accuracy of highlights identification in terms of precision, recall, and F1-score.
關聯:	Multimedia Systems 30(285)， p. 1-18
DOI:	10.1007/s00530-024-01479-6
显示于类别:	[資訊工程學系暨研究所] 期刊論文

文件中的档案:

档案	描述	大小	格式	浏览次数
index.html		0Kb	HTML	14	检视/开启

在機構典藏中所有的数据项都受到原著作权保护.

TAIR相关文章

数据加载中.....