ChartStamp: Robust Chart Embedding for Real-World Applications Jiayun Fu ∗ fujiayun@hust.edu.cn Huazhong University of Science and Technology Wuhan, China Bin B. Zhu 2 Haidong Zhang binzhu@microsoft.com haizhang@microsoft.com Microsoft Research Asia Beijing, China Yayi Zou ∗ yayizou@163.com Huazhong University of Science and Technology Beijing, China Song Ge Weiwei Cui songge@microsoft.com weiweicu@microsoft.com Microsoft Research Asia Beijing, China Yun Wang Dongmei Zhang wangyun@microsoft.com dongmeiz@microsoft.com Microsoft Research Asia Beijing, China Xiaojing Ma Hai Jin lindahust@hust.edu.cn hjin@hust.edu.cn Huazhong University of Science and Technology Wuhan, China ABSTRACT Deep learning-based image embedding methods are typically de- signed for natural images and may not work for chart images due to their homogeneous regions, which lack variations to hide data both robustly and imperceptibly. In this paper, we propose Chart- Stamp, the frst chart embedding method that is robust to real-world printing and displaying (printed on paper and displayed on screen, respectively, and then captured with a camera) while maintaining a good perceptual quality. ChartStamp hides 100, 1,000, or 10,000 raw bits into a chart image, depending on the designated robustness to printing, displaying, or JPEG. To ensure perceptual quality, it introduces a new perceptual model to guide embedding to insen- sitive regions of a chart image and a smoothness loss to ensure smoothness of the embedding residual in homogeneous regions. ChartStamp applies a distortion layer approximating designated real-world manipulations to train a model robust to these manip- ulations. Our experimental evaluation indicates that ChartStamp achieves the robustness and embedding capacity on chart images similar to their state-of-the-art counterparts on natural images. Our ∗ Jiayun Fu and Xiaojing Ma are with National Engineering Research Center for Big Data Technology and System, Services Computing Technology and System Lab, Hubei Engineering Research Center on Big Data Security, Hubei key Laboratory of Distributed System Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology. Hai Jin is with National Engineering Research Center for Big Data Technology and System, Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology. This work was done when Yayi Zou was an intern at Microsoft Research Asia. 2 Corresponding author: Bin B. Zhu (binzhu@microsoft.com). Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specifc permission and/or a fee. Request permissions from permissions@acm.org. MM ’22, October 10–14, 2022, Lisboa, Portugal © 2022 Association for Computing Machinery. ACM ISBN 978-1-4503-9203-7/22/10. . . $15.00 https://doi.org/10.1145/3503161.3548286 user studies indicate that ChartStamp achieves better perceptual quality than existing robust chart embedding methods and that our perceptual model outperforms the existing perceptual model. CCS CONCEPTS · Computing methodologies → Machine learning algorithms. KEYWORDS Chart embedding; image embedding; data hiding; perceptual model ACM Reference Format: Jiayun Fu, Bin B. Zhu, Haidong Zhang, Yayi Zou, Song Ge, Weiwei Cui, Yun Wang, Dongmei Zhang, Xiaojing Ma, and Hai Jin. 2022. ChartStamp: Robust Chart Embedding for Real-World Applications. In Proceedings of the 30th ACM International Conference on Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal. ACM, New York, NY, USA, 10 pages. https: //doi.org/10.1145/3503161.3548286 1 INTRODUCTION Charts are widely used as an efective means to convey quantitative information [25]. They are typically disseminated as bitmap images. Chart image embedding or simply chart embedding hides data into a host chart image by subtly modifying the chart image. The hidden data can be a unique hyperlink, chart data or information to facilitate recovering the chart from its bitmap image, or arbitrary information. When a chart is converted to a bitmap image, its chart information and visual style are unreadable to computers. Chart embedding provides a viable solution to the challenging problem to recover a chart from its bitmap image. Accurate recovery with recognition-based techniques is still difcult due to complexity and diversity of charts. A future augmented reality system can also utilize the hidden data or linked data via the hidden hyperlink to visually overlay the recovered or retrieved information with the chart to provide an enhanced user experience. Image embedding has been extensively studied for natural im- ages [7, 8]. In addition to traditional approaches, deep learning has been adopted recently to train an encoder-decoder network 2786