HMTV: Hierarchical Multimodal Transformer for Video Highlight Query on Baseball

doi:10.1007/s00530-024-01479-6

淡江大學機構典藏 > 工學院 > 資訊工程學系暨研究所 > 期刊論文 > Item 987654321/126772

請使用永久網址來引用或連結此文件: https://tkuir.lib.tku.edu.tw/dspace/handle/987654321/126772

題名:	HMTV: Hierarchical Multimodal Transformer for Video Highlight Query on Baseball
作者:	Zhang, Qiaoyun;Chang, Chih-Yung;Su, Ming-Yang;Chang, Hsiang-Chuan;Roy, Diptendu Sinha
關鍵詞:	分層多模態 Transformer;BERT;反白顯示查詢
日期:	2024-09-23
上傳時間:	2025-03-20 09:24:10 (UTC+8)
摘要:	With the increasing popularity of watching baseball videos, there is a growing desire among fans to enjoy the highlights of these videos. However, the extraction of the highlights from lengthy baseball videos faces a significant challenge due to its time-consuming and labor-intensive nature. To address this challenge, this paper proposes a novel mechanism, called Hierarchical Multimodal Transformer for Video query (HMTV). The proposed HMTV incorporates a two-phase involving Coarse-Grained clipping for candidate videos and Fine-Grained identification for highlights. In the Coarse-Grained phase, a pitching detection model is employed to extract relevant candidate videos from baseball videos, encompassing the features of pitch deliveries and pitching. In the Fine-Grained phase, Transformer encoder and pre-trained Bidirectional Encoder Representations from Transformers (BERT) are utilized to capture relationship features between frames of candidate videos and words from users’ questions, respectively. These relationship features are then fed into the Video Query (VideoQ) model, implemented by the Text Video Attention (TVA). The VideoQ model identifies the start and end positions of the highlights mentioned in the query within the candidate videos. Simulation results demonstrate that the proposed HMTV significantly improves accuracy of highlights identification in terms of precision, recall, and F1-score.
關聯:	Multimedia Systems 30(285)， p. 1-18
DOI:	10.1007/s00530-024-01479-6
顯示於類別:	[資訊工程學系暨研究所] 期刊論文

文件中的檔案:

檔案	描述	大小	格式	瀏覽次數
index.html		0Kb	HTML	18	檢視/開啟

在機構典藏中所有的資料項目都受到原著作權保護.

TAIR相關文章

資料載入中.....