The best thing is to avoid them in order to save. Sometimes, you don&#x27;t even realize the harm you&#x27;re doing.

Three mistakes you make when washing clothes - you're wasting time and money

<strong>TEMPO</strong>.CO, Jakarta - The Meteorology, Climatology, and Geophysics Agency or BMKG released the weather forecast in Greater Jakarta today, Monday, April 29, 2024. In the morning, the entirety of Jakarta will be greeted by a partly cloudy sky. The weather will last throughout the afternoon in Central ...

BMKG Forecasts Partly Cloudy Morning in Greater Jakarta - Sci En.tempo.co

The latest findings support the existence of the mysterious Planet 9, based on detailed analysis of trans-Neptunian object clustering

A group of astronomers finds new evidence of the possible existence of the mysterious Planet 9 in the solar system

Tempo

tempo

Authors:
(1) Yuwei Guo, The Chinese University of Hong Kong;
(2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author;
(3) Anyi Rao, Stanford University;
(4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory;
(5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory;
(6) Yu Qiao, Shanghai Artificial Intelligence Laboratory;
(7) Maneesh Agrawala, Stanford University;
(8) Dahua Lin, Shanghai Artificial Intelligence Laboratory;
(9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction 2 Work Related 3 Preliminary AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.3 Ablative Study 5.4 Controllable Generation 6 Conclusion 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References 4.2 LEARN MOTION PRIORS WITH MOTION MODULE To model motion dynamics along the temporal dimension on top of a pre-trained T2I, we must 1) inflate the 2-dimensional diffusion model to deal with 3-dimensional video data and 2) design a sub-module to enable efficient information exchange along the temporal axis. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Authors: Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Work Related 2 Work Related 3 Preliminary 3 Preliminary AnimateDiff AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.2 Qualitative Comparison 5.3 Ablative Study 5.3 Ablative Study 5.4 Controllable Generation 5.4 Controllable Generation 6 Conclusion 6 Conclusion 7 Ethics Statement 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References 8 Reproducibility Statement, Acknowledgement and References 4.2 LEARN MOTION PRIORS WITH MOTION MODULE To model motion dynamics along the temporal dimension on top of a pre-trained T2I, we must 1) inflate the 2-dimensional diffusion model to deal with 3-dimensional video data and 2) design a sub-module to enable efficient information exchange along the temporal axis. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Mastering Motion Dynamics in Animation with Temporal Transformers

(1) Feng Liang, The University of Texas at Austin and Work partially done during an internship at Meta GenAI (Email: jeffliang@utexas.edu);
(2) Bichen Wu, Meta GenAI and Corresponding author;
(3) Jialiang Wang, Meta GenAI;
(4) Licheng Yu, Meta GenAI;
(5) Kunpeng Li, Meta GenAI;
(6) Yinan Zhao, Meta GenAI;
(7) Ishan Misra, Meta GenAI;
(8) Jia-Bin Huang, Meta GenAI;
(9) Peizhao Zhang, Meta GenAI (Email: stzpz@meta.com);
(10) Peter Vajda, Meta GenAI (Email: vajdap@meta.com);
(11) Diana Marculescu, The University of Texas at Austin (Email: dianam@utexas.edu). Table of Links Abstract and Introduction
2. Related Work
3. Preliminary
4. FlowVid
4.1. Inflating image U-Net to accommodate video
4.2. Training with joint spatial-temporal conditions
4.3. Generation: edit the first frame then propagate


Experiments


5.1. Settings
5.2. Qualitative results
5.3. Quantitative results
5.4. Ablation study and 5.5. Limitations
Conclusion, Acknowledgments and References
A. Webpage Demo and B. Quantitative comparisons 4.2. Training with joint spatial-temporal conditions Upon expanding the image model, a straightforward method might be to train the video model using paired depth-video data. Yet, our empirical analysis indicates that this leads to sub-optimal results, as detailed in the ablation study in Section 5.4. We hypothesize that this method neglects the temporal clue within the video, making the frame consistency hard to maintain. While some studies, such as Rerender [49] and CoDeF [32], incorporate optical flow in video synthesis, they typically apply it as a rigid constraint. In contrast, our approach uses flow as a soft condition, allowing us to manage the imperfections commonly found in flow estimation. This paper is available on arxiv under CC 4.0 license. (1) Feng Liang, The University of Texas at Austin and Work partially done during an internship at Meta GenAI (Email: jeffliang@utexas.edu); (2) Bichen Wu, Meta GenAI and Corresponding author; (3) Jialiang Wang, Meta GenAI; (4) Licheng Yu, Meta GenAI; (5) Kunpeng Li, Meta GenAI; (6) Yinan Zhao, Meta GenAI; (7) Ishan Misra, Meta GenAI; (8) Jia-Bin Huang, Meta GenAI; (9) Peizhao Zhang, Meta GenAI (Email: stzpz@meta.com); (10) Peter Vajda, Meta GenAI (Email: vajdap@meta.com); (11) Diana Marculescu, The University of Texas at Austin (Email: dianam@utexas.edu). (1) Feng Liang, The University of Texas at Austin and Work partially done during an internship at Meta GenAI (Email: jeffliang@utexas.edu); (2) Bichen Wu, Meta GenAI and Corresponding author; (3) Jialiang Wang, Meta GenAI; (4) Licheng Yu, Meta GenAI; (5) Kunpeng Li, Meta GenAI; (6) Yinan Zhao, Meta GenAI; (7) Ishan Misra, Meta GenAI; (8) Jia-Bin Huang, Meta GenAI; (9) Peizhao Zhang, Meta GenAI (Email: stzpz@meta.com); (10) Peter Vajda, Meta GenAI (Email: vajdap@meta.com); (11) Diana Marculescu, The University of Texas at Austin (Email: dianam@utexas.edu). Table of Links Abstract and Introduction 2. Related Work 3. Preliminary 4. FlowVid 4.1. Inflating image U-Net to accommodate video 4.2. Training with joint spatial-temporal conditions 4.3. Generation: edit the first frame then propagate Experiments 5.1. Settings 5.2. Qualitative results 5.3. Quantitative results 5.4. Ablation study and 5.5. Limitations Conclusion, Acknowledgments and References A. Webpage Demo and B. Quantitative comparisons Abstract and Introduction Abstract and Introduction 2. Related Work 2. Related Work 3. Preliminary 3. Preliminary 4. FlowVid 4. FlowVid 4 .1. Inflating image U-Net to accommodate video .1. Inflating image U-Net to accommodate video 4.2. Training with joint spatial-temporal conditions 4.2. Training with joint spatial-temporal conditions 4.3. Generation: edit the first frame then propagate 4.3. Generation: edit the first frame then propagate Experiments Experiments Experiments 5.1. Settings 5.1. Settings 5.2. Qualitative results 5.2. Qualitative results 5.3. Quantitative results 5.3. Quantitative results 5.4. Ablation study and 5.5. Limitations 5.4. Ablation study and 5.5. Limitations Conclusion, Acknowledgments and References Conclusion, Acknowledgments and References A. Webpage Demo and B. Quantitative comparisons A. Webpage Demo and B. Quantitative comparisons 4.2. Training with joint spatial-temporal conditions Upon expanding the image model, a straightforward method might be to train the video model using paired depth-video data. Yet, our empirical analysis indicates that this leads to sub-optimal results, as detailed in the ablation study in Section 5.4. We hypothesize that this method neglects the temporal clue within the video, making the frame consistency hard to maintain. While some studies, such as Rerender [49] and CoDeF [32], incorporate optical flow in video synthesis, they typically apply it as a rigid constraint. In contrast, our approach uses flow as a soft condition, allowing us to manage the imperfections commonly found in flow estimation. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

FlowVid: Taming Imperfect Optical Flows: Training With Joint Spatial-temporal Conditions

Authors:
(1) Xiaohan Ding, Department of Computer Science, Virginia Tech, (e-mail: xiaohan@vt.edu);
(2) Mike Horning, Department of Communication, Virginia Tech, (e-mail: mhorning@vt.edu);
(3) Eugenia H. Rho, Department of Computer Science, Virginia Tech, (e-mail: eugenia@vt.edu ). Table of Links Abstract and Introduction Related Work Study 1: Evolution of Semantic Polarity in Broadcast Media Language (2010-2020) Study 2: Words that Characterize Semantic Polarity between Fox News & CNN in 2020 Study 3: How Semantic Polarization in Broadcast Media Language Forecasts Semantic Polarity in Social Media Discourse Discussion and Ethics Statement Appendix and References Related Work Temporal Dynamics in Linguistic Polarization Scholars have measured linguistic polarization through stance (Dash et al. 2022), toxicity (Sap et al. 2019), sentiment detection (Yang et al. 2017), topic modeling (Ebeling et al. 2022), and lexicon-dictionaries (Polignano et al. 2022). Such methods typically capture an aggregated snapshot of polarization across large textual corpora, providing a static quantification of media bias based on preset criteria. As a result, such approaches seldom capture temporal fluctuations in semantic polarity. Only a few research to date quantify linguistic polarization over time. For example, researchers have measured partisan trends in congressional speech by defining polarization as the expected posterior probability that a neutral observer would infer a speaker’s political party from a randomly selected utterance (Gentzkow, Shapiro, and Taddy 2019) across various issues. Demszky et al. use this Bayesian approach to capture how social media discussions on mass shootings become increasingly divisive (2020). Such methods, however, capture linguistic polarization as a general metric across nearly all aggregated words within a textual corpus. Given the powerful salience of certain topical keywords that are inseparable from American politics, unlike prior work, we focus on how specific keywords polarize over time. We depart from previous approaches by devising a framework that captures semantic polarity as a function of how two different entities semantically diverge across time in their contextual use of identical words that are central to the American public discourse. Specifically, we measure linguistic polarization by capturing the diachronic shifts in how two news stations use the same topical words on their news programs over an 11-year period. Further, in effort to understand the linguistic drivers behind how and why semantic polarization evolves, we aim to identify the source of diachronic shifts based on how two different entities use identical, politically contentious keywords. Few research has endeavored to detect causal factors underlying how the meaning of words changes over time, which remains an open challenge in NLP scholarship (Kutuzov et al. 2018). Hamilton et al. show how word meanings change between consecutive decades due to cultural shifts (e.g., change in the meaning of “cell” driven by technological advancement: prison cell vs. cell phone) (2016). Yet such research typically makes a priori assumptions that the language between two textual corpora of comparison are significantly related without statistically demonstrating how so. On the other hand, research that do provide evidence of statistical relations between two separate textual data (Dutta, Ma, and Choudhury 2018), generally do not explain what words drive the direction of semantic influence from one corpus to another. We overcome these limitations: first, we statistically validate the existence of a significant semantic relationship between two textual corpora, TV news language and social media discourse, through Grangercausality (Study 3). Then, we strategically separate our data by lag-times associated with significant Granger-causal relations and apply a deep learning interpretation technique to demonstrate which contextual words from TV news language influences Twitter discussions around specific topical keywords (and vice-versa) over time. Who Influences Who? Setting a Media Agenda On any given day, newsrooms select news on a wide range of topics that cover a variety of issues. According to Agenda Setting Theory, this selection process leads audiences to see certain issues as more significant than others (McCombs 1997). Agenda Setting was particularly useful in decades past when media was much more consolidated. In the U.S. for example, three major networks (NBC, ABC, and CBS) provided the bulk of broadcast news content, and so their news agendas had broad influence on what audiences saw as important. However, today’s media landscape is much more fragmented. People receive their news from any number of sources including television, newspapers, radio, online websites, and social media. In such an environment, Agenda Setting has certain limitations as the theory assumes that audiences are relatively passive and simply receive information from the media. This was true when news flowed one-way from the media to the mass public. By contrast nowadays, the two-way form of communication afforded by the internet and social media makes it possible for the public to both influence media agendas and be influenced by them (Papacharissi 2009; Barnard 2018). As a result, new scholarship has argued through Intermedia Agenda Setting Theory that media agendas do not come entirely from within their own organizations, but from two other sources: other media and the mass audience. In the latter case, for example, audiences themselves can influence media agendas through online affordances (e.g., retweeting, commenting, sharing), to raise certain issues to prominence online (Rogstad 2016). In this work, we explore these new emerging dynamics with an interest in not only understanding how news agendas are constructed, but how the possible adoption of different agendas might influence agenda dynamics as media interact with the public online. This paper is available on arxiv under CC 4.0 license. Authors: (1) Xiaohan Ding, Department of Computer Science, Virginia Tech, (e-mail: xiaohan@vt.edu); (2) Mike Horning, Department of Communication, Virginia Tech, (e-mail: mhorning@vt.edu); (3) Eugenia H. Rho, Department of Computer Science, Virginia Tech, (e-mail: eugenia@vt.edu ). Authors: Authors: (1) Xiaohan Ding, Department of Computer Science, Virginia Tech, (e-mail: xiaohan@vt.edu); (2) Mike Horning, Department of Communication, Virginia Tech, (e-mail: mhorning@vt.edu); (3) Eugenia H. Rho, Department of Computer Science, Virginia Tech, (e-mail: eugenia@vt.edu ). Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Study 1: Evolution of Semantic Polarity in Broadcast Media Language (2010-2020) Study 1: Evolution of Semantic Polarity in Broadcast Media Language (2010-2020) Study 2: Words that Characterize Semantic Polarity between Fox News & CNN in 2020 Study 2: Words that Characterize Semantic Polarity between Fox News & CNN in 2020 Study 3: How Semantic Polarization in Broadcast Media Language Forecasts Semantic Polarity in Social Media Discourse Study 3: How Semantic Polarization in Broadcast Media Language Forecasts Semantic Polarity in Social Media Discourse Discussion and Ethics Statement Discussion and Ethics Statement Appendix and References Appendix and References Related Work Temporal Dynamics in Linguistic Polarization Scholars have measured linguistic polarization through stance (Dash et al. 2022), toxicity (Sap et al. 2019), sentiment detection (Yang et al. 2017), topic modeling (Ebeling et al. 2022), and lexicon-dictionaries (Polignano et al. 2022). Such methods typically capture an aggregated snapshot of polarization across large textual corpora, providing a static quantification of media bias based on preset criteria. As a result, such approaches seldom capture temporal fluctuations in semantic polarity. Only a few research to date quantify linguistic polarization over time. For example, researchers have measured partisan trends in congressional speech by defining polarization as the expected posterior probability that a neutral observer would infer a speaker’s political party from a randomly selected utterance (Gentzkow, Shapiro, and Taddy 2019) across various issues. Demszky et al. use this Bayesian approach to capture how social media discussions on mass shootings become increasingly divisive (2020). Such methods, however, capture linguistic polarization as a general metric across nearly all aggregated words within a textual corpus. Given the powerful salience of certain topical keywords that are inseparable from American politics, unlike prior work, we focus on how specific keywords polarize over time. We depart from previous approaches by devising a framework that captures semantic polarity as a function of how two different entities semantically diverge across time in their contextual use of identical words that are central to the American public discourse. Specifically, we measure linguistic polarization by capturing the diachronic shifts in how two news stations use the same topical words on their news programs over an 11-year period. Further, in effort to understand the linguistic drivers behind how and why semantic polarization evolves, we aim to identify the source of diachronic shifts based on how two different entities use identical, politically contentious keywords. Few research has endeavored to detect causal factors underlying how the meaning of words changes over time, which remains an open challenge in NLP scholarship (Kutuzov et al. 2018). Hamilton et al. show how word meanings change between consecutive decades due to cultural shifts (e.g., change in the meaning of “cell” driven by technological advancement: prison cell vs. cell phone) (2016). Yet such research typically makes a priori assumptions that the language between two textual corpora of comparison are significantly related without statistically demonstrating how so. On the other hand, research that do provide evidence of statistical relations between two separate textual data (Dutta, Ma, and Choudhury 2018), generally do not explain what words drive the direction of semantic influence from one corpus to another. We overcome these limitations: first, we statistically validate the existence of a significant semantic relationship between two textual corpora, TV news language and social media discourse, through Grangercausality (Study 3). Then, we strategically separate our data by lag-times associated with significant Granger-causal relations and apply a deep learning interpretation technique to demonstrate which contextual words from TV news language influences Twitter discussions around specific topical keywords (and vice-versa) over time. Who Influences Who? Setting a Media Agenda On any given day, newsrooms select news on a wide range of topics that cover a variety of issues. According to Agenda Setting Theory, this selection process leads audiences to see certain issues as more significant than others (McCombs 1997). Agenda Setting was particularly useful in decades past when media was much more consolidated. In the U.S. for example, three major networks (NBC, ABC, and CBS) provided the bulk of broadcast news content, and so their news agendas had broad influence on what audiences saw as important. However, today’s media landscape is much more fragmented. People receive their news from any number of sources including television, newspapers, radio, online websites, and social media. In such an environment, Agenda Setting has certain limitations as the theory assumes that audiences are relatively passive and simply receive information from the media. This was true when news flowed one-way from the media to the mass public. By contrast nowadays, the two-way form of communication afforded by the internet and social media makes it possible for the public to both influence media agendas and be influenced by them (Papacharissi 2009; Barnard 2018). As a result, new scholarship has argued through Intermedia Agenda Setting Theory that media agendas do not come entirely from within their own organizations, but from two other sources: other media and the mass audience. In the latter case, for example, audiences themselves can influence media agendas through online affordances (e.g., retweeting, commenting, sharing), to raise certain issues to prominence online (Rogstad 2016). In this work, we explore these new emerging dynamics with an interest in not only understanding how news agendas are constructed, but how the possible adoption of different agendas might influence agenda dynamics as media interact with the public online. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Temporal Dynamics in Linguistic Polarization

Temporary Lifetime Extension: Complicated Cases

Author:
(1) Yigit Ege Bayiz, Electrical and Computer Engineering The University of Texas at Austin Austin, Texas, USA (Email: egebayiz@utexas.edu);
(2) Ufuk Topcu, Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, Texas, USA (Email: utopcu@utexas.edu). Table of Links Abstract and Introduction Related Works Preliminaries Optimal Prebunking Problem Deterministic Baseline Policies Temporally Equidistant Prebunking Numerical Results Conclusion and References VI. TEMPORALLY EQUIDISTANT PREBUNKING Thus, by Corollary 3, the optimal policy delivers prebunks evenly across all time steps with equal time gaps in-between. Intuitively, this means that the optimal policy tries to maximize the time gap between each consecutive prebunk delivery. We emphasize that Corollary 3 only holds exactly when we have the restricted problem 24, without the additional constraints. We can approximate this equal time gap policy in the general problem 12. The goal of the policy π in this case is to determine which Pk to issue at time t. Clearly, it is not optimal to provide a prebunk Pk to user c more than once, since doing so does not change the feasibility of problem and cannot reduce the maximal cost C(π). Thus, we can assume without loss of optimality that each Pk is in at most one At ∈ A. Hence, in its decision, the policy π only needs to take into account the misinformations Mk, for which it has not delivered any prebunk Pk yet. Without loss of generality, we assume that this set of misinformations is the finite set {M1, . . . , MK} as Assumption 2 guarantees that at each time t there are finitely many Mk for which Ik ̸= ∅. In our experiments, we use a Monte Carlo method and estimate Xk by simulating misinformation propagations in the local network Gc, and obtaining a frequentist estimation of how long it takes for the misinformation to reach from any node in the network to the central user c. Clearly, this estimation method does not scale to large networks. However, in our tests, for small choices of m the local network around user c remains small enough to permit this approach. It is possible that there are other estimation methods that admit more accurate estimations or scale better to large networks, which might provide better results. This paper is available on arxiv under CC 4.0 license. Author: (1) Yigit Ege Bayiz, Electrical and Computer Engineering The University of Texas at Austin Austin, Texas, USA (Email: egebayiz@utexas.edu); (2) Ufuk Topcu, Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, Texas, USA (Email: utopcu@utexas.edu). Author: Author: (1) Yigit Ege Bayiz, Electrical and Computer Engineering The University of Texas at Austin Austin, Texas, USA (Email: egebayiz@utexas.edu); (2) Ufuk Topcu, Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, Texas, USA (Email: utopcu@utexas.edu). Table of Links Abstract and Introduction Abstract and Introduction Related Works Related Works Preliminaries Preliminaries Optimal Prebunking Problem Optimal Prebunking Problem Deterministic Baseline Policies Deterministic Baseline Policies Temporally Equidistant Prebunking Temporally Equidistant Prebunking Numerical Results Numerical Results Conclusion and References Conclusion and References VI. TEMPORALLY EQUIDISTANT PREBUNKING Thus, by Corollary 3, the optimal policy delivers prebunks evenly across all time steps with equal time gaps in-between. Intuitively, this means that the optimal policy tries to maximize the time gap between each consecutive prebunk delivery. We emphasize that Corollary 3 only holds exactly when we have the restricted problem 24, without the additional constraints. We can approximate this equal time gap policy in the general problem 12. The goal of the policy π in this case is to determine which Pk to issue at time t. Clearly, it is not optimal to provide a prebunk Pk to user c more than once, since doing so does not change the feasibility of problem and cannot reduce the maximal cost C(π). Thus, we can assume without loss of optimality that each Pk is in at most one At ∈ A. Hence, in its decision, the policy π only needs to take into account the misinformations Mk, for which it has not delivered any prebunk Pk yet. Without loss of generality, we assume that this set of misinformations is the finite set {M1, . . . , MK} as Assumption 2 guarantees that at each time t there are finitely many Mk for which Ik ̸= ∅. In our experiments, we use a Monte Carlo method and estimate Xk by simulating misinformation propagations in the local network Gc, and obtaining a frequentist estimation of how long it takes for the misinformation to reach from any node in the network to the central user c. Clearly, this estimation method does not scale to large networks. However, in our tests, for small choices of m the local network around user c remains small enough to permit this approach. It is possible that there are other estimation methods that admit more accurate estimations or scale better to large networks, which might provide better results. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Temporally Equidistant Prebunking

Temporary Lifetime Extension: Mistakes and Solutions

Let's explore how temporary tables work in MariaDB. First, we have to connect to the server. For example (use your own connection details): mariadb -h 127.0.0.1 -u root -p"RootPassword!" --database demo Now, just to point something out, let's create a standard (permanent) table. Here's how: CREATE TABLE t (
    c INT
); This table, t, will persist in the database even after we exit the client: exit When we reconnect and check the existing tables using SHOW TABLES;, the table t will still be listed: mariadb -h 127.0.0.1 -u root -p"RootPassword!" --database demo SHOW TABLES; +----------------+
| Tables_in_demo |
+----------------+
| t              |
+----------------+ All this is pretty obvious, but now, let's recreate this table and try something different: CREATE OR REPLACE TEMPORARY TABLE t (
    c INT
); Notice the TEMPORARY keyword. After creating this table, if we run SHOW TABLES;, it appears in the list. We can insert data into it, query it, and join it with other tables. It behaves like a normal table during the current session. However, if we exit the client, then reconnect, and perform a SHOW TABLES; again, the temporary table t will not be listed. A temporary table only exists for the duration of the session in which it was created and other sessions won't be able to see it. Use Case for Temporary Tables Temporary tables are quite useful for transient data operations. For instance, consider a table called products in our database: CREATE TABLE products (
  id INT NOT NULL AUTO_INCREMENT,
  code VARCHAR(100) NOT NULL,
  name VARCHAR(250) NOT NULL,
  description TEXT DEFAULT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY code (code)
) We can create a temporary table that mimics the structure of products: CREATE TEMPORARY TABLE t LIKE products; We can confirm this by running: DESCRIBE t; +-------------+--------------+------+-----+---------+----------------+
| Field       | Type         | Null | Key | Default | Extra          |
+-------------+--------------+------+-----+---------+----------------+
| id          | int(11)      | NO   | PRI | NULL    | auto_increment |
| code        | varchar(100) | NO   | UNI | NULL    |                |
| name        | varchar(250) | NO   |     | NULL    |                |
| description | text         | YES  |     | NULL    |                |
+-------------+--------------+------+-----+---------+----------------+ Initially, t will be empty. However, suppose we want to transfer some data from products to t. Let’s assume we only want to include products that contain the number 0 in their code: INSERT INTO t
SELECT * FROM products
WHERE code LIKE '%0%'; After running this command, if we query the temporary table t: SELECT * FROM t; +----+--------+------------------+---------------------------------------------------+
| id | code   | name             | description                                       |
+----+--------+------------------+---------------------------------------------------+
|  1 | BG2024 | BugBlaster       | Eradicates software bugs with a single scan.      |
|  3 | FW001  | FireWhale        | An oversized, comprehensive firewall solution.    |
|  4 | CLD404 | CloudNine Finder | Find your way back from cloud outages and errors. |
+----+--------+------------------+---------------------------------------------------+ We see the filtered data. Conclusion Temporary tables offer a powerful way to handle data for temporary processing without affecting the persistent data store. They are particularly useful in scenarios where data needs to be manipulated or transformed temporarily. You can use permanent tables for this kind of data manipulation but temporary tables are useful when you need automatic cleanup, reduced risk of naming conflicts, isolation and security, and resource management for query performance. Let's explore how temporary tables work in MariaDB. First, we have to connect to the server. For example (use your own connection details): temporary tables mariadb -h 127.0.0.1 -u root -p"RootPassword!" --database demo mariadb -h 127.0.0.1 -u root -p"RootPassword!" --database demo Now, just to point something out, let's create a standard (permanent) table. Here's how: CREATE TABLE t (
    c INT
); CREATE TABLE t (
    c INT
); This table, t , will persist in the database even after we exit the client: t exit exit When we reconnect and check the existing tables using SHOW TABLES; , the table t will still be listed: SHOW TABLES; t mariadb -h 127.0.0.1 -u root -p"RootPassword!" --database demo mariadb -h 127.0.0.1 -u root -p"RootPassword!" --database demo SHOW TABLES; SHOW TABLES; +----------------+
| Tables_in_demo |
+----------------+
| t              |
+----------------+ +----------------+
| Tables_in_demo |
+----------------+
| t              |
+----------------+ All this is pretty obvious, but now, let's recreate this table and try something different: CREATE OR REPLACE TEMPORARY TABLE t (
    c INT
); CREATE OR REPLACE TEMPORARY TABLE t (
    c INT
); Notice the TEMPORARY keyword. After creating this table, if we run SHOW TABLES; , it appears in the list. We can insert data into it, query it, and join it with other tables. It behaves like a normal table during the current session. However, if we exit the client, then reconnect, and perform a SHOW TABLES; again, the temporary table t will not be listed. TEMPORARY SHOW TABLES; SHOW TABLES; t A temporary table only exists for the duration of the session in which it was created and other sessions won't be able to see it. Use Case for Temporary Tables Temporary tables are quite useful for transient data operations. For instance, consider a table called products in our database: products CREATE TABLE products (
  id INT NOT NULL AUTO_INCREMENT,
  code VARCHAR(100) NOT NULL,
  name VARCHAR(250) NOT NULL,
  description TEXT DEFAULT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY code (code)
) CREATE TABLE products (
  id INT NOT NULL AUTO_INCREMENT,
  code VARCHAR(100) NOT NULL,
  name VARCHAR(250) NOT NULL,
  description TEXT DEFAULT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY code (code)
) We can create a temporary table that mimics the structure of products : products CREATE TEMPORARY TABLE t LIKE products; CREATE TEMPORARY TABLE t LIKE products; We can confirm this by running: DESCRIBE t; DESCRIBE t; +-------------+--------------+------+-----+---------+----------------+
| Field       | Type         | Null | Key | Default | Extra          |
+-------------+--------------+------+-----+---------+----------------+
| id          | int(11)      | NO   | PRI | NULL    | auto_increment |
| code        | varchar(100) | NO   | UNI | NULL    |                |
| name        | varchar(250) | NO   |     | NULL    |                |
| description | text         | YES  |     | NULL    |                |
+-------------+--------------+------+-----+---------+----------------+ +-------------+--------------+------+-----+---------+----------------+
| Field       | Type         | Null | Key | Default | Extra          |
+-------------+--------------+------+-----+---------+----------------+
| id          | int(11)      | NO   | PRI | NULL    | auto_increment |
| code        | varchar(100) | NO   | UNI | NULL    |                |
| name        | varchar(250) | NO   |     | NULL    |                |
| description | text         | YES  |     | NULL    |                |
+-------------+--------------+------+-----+---------+----------------+ Initially, t will be empty. However, suppose we want to transfer some data from products to t . Let’s assume we only want to include products that contain the number 0 in their code: t products t INSERT INTO t
SELECT * FROM products
WHERE code LIKE '%0%'; INSERT INTO t
SELECT * FROM products
WHERE code LIKE '%0%'; After running this command, if we query the temporary table t : t SELECT * FROM t; SELECT * FROM t; +----+--------+------------------+---------------------------------------------------+
| id | code   | name             | description                                       |
+----+--------+------------------+---------------------------------------------------+
|  1 | BG2024 | BugBlaster       | Eradicates software bugs with a single scan.      |
|  3 | FW001  | FireWhale        | An oversized, comprehensive firewall solution.    |
|  4 | CLD404 | CloudNine Finder | Find your way back from cloud outages and errors. |
+----+--------+------------------+---------------------------------------------------+ +----+--------+------------------+---------------------------------------------------+
| id | code   | name             | description                                       |
+----+--------+------------------+---------------------------------------------------+
|  1 | BG2024 | BugBlaster       | Eradicates software bugs with a single scan.      |
|  3 | FW001  | FireWhale        | An oversized, comprehensive firewall solution.    |
|  4 | CLD404 | CloudNine Finder | Find your way back from cloud outages and errors. |
+----+--------+------------------+---------------------------------------------------+ We see the filtered data. Conclusion Temporary tables offer a powerful way to handle data for temporary processing without affecting the persistent data store. They are particularly useful in scenarios where data needs to be manipulated or transformed temporarily. You can use permanent tables for this kind of data manipulation but temporary tables are useful when you need automatic cleanup, reduced risk of naming conflicts, isolation and security, and resource management for query performance.

How to Use Temporary Tables in MariaDB

মারিয়াডিবিতে অস্থায়ী টেবিলগুলি কীভাবে ব্যবহার করবেন

So verwenden Sie temporäre Tabellen in MariaDB

Cómo utilizar tablas temporales en MariaDB

Comment utiliser des tables temporaires dans MariaDB

मारियाडीबी में अस्थायी तालिकाओं का उपयोग कैसे करें

MariaDB で一時テーブルを使用する方法

MariaDB에서 임시 테이블을 사용하는 방법

Como usar tabelas temporárias no MariaDB

Как использовать временные таблицы в MariaDB

MariaDB'de Geçici Tablolar Nasıl Kullanılır?

Cách sử dụng bảng tạm thời trong MariaDB

如何在 MariaDB 中使用临时表

This paper is available on arxiv under CC 4.0 license. Authors: (1) Zhouqiang Jiang, Meetyou AI Lab, (2) Bowen Wang, Institute for Datability Science, Osaka University, (3) Tong Xiang, Meetyou AI Lab, (4) Zhaofeng Niu, Department of Computer Science, Qufu Normal University, (5) Hong Tang, Department of Information Engineering, East China Jiaotong University, (6) Guangshun Li, Department of Computer Science, Qufu Normal University, (7) Liangzhi Li, Meetyou AI Lab. Table of Links Abstract & Introduction Related Work Method Experiments Conclusion & References V. CONCLUSION In this paper, we proposed CatMAE for self-supervised video representation learning. It leverages a concatenated information channel masking strategy to address the limitations posed by cube masking and enhances the capability to capture continuous and long-term motion compared to asymmetric masking. Our experimental results demonstrate superior performance in comparison to state-of-the-art methods across both video segmentation and action recognition tasks. One distinctive feature of our training pipeline is the propagation of reconstruction information from the initial frame throughout the entire video sequence. This theoretically unlimited propagation showcases CatMAE’s potential to learn long-term video representations. Our future work focuses on extending the application of CatMAE to realworld scenarios involving embodied agents, such as robots. REFERENCES [1] K. He, X. Chen, S. Xie, Y. Li, P. Dollar, and R. Girshick, “Masked ´ autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 000–16 009. [2] Z. Tong, Y. Song, J. Wang, and L. Wang, “Videomae: Masked autoencoders are data-efficient learners for self-supervised video pretraining,” Advances in neural information processing systems, vol. 35, pp. 10 078–10 093, 2022. [3] C. Feichtenhofer, Y. Li, K. He et al., “Masked autoencoders as spatiotemporal learners,” Advances in neural information processing systems, vol. 35, pp. 35 946–35 958, 2022. [4] A. Gupta, J. Wu, J. Deng, and L. Fei-Fei, “Siamese masked autoencoders,” arXiv preprint arXiv:2305.14344, 2023. [5] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020. [6] E. H. Adelson and J. R. Bergen, “Spatiotemporal energy models for the perception of motion,” Josa a, vol. 2, no. 2, pp. 284–299, 1985. [7] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6299–6308. [8] Z. Zhang and D. Tao, “Slow feature analysis for human action recognition,” IEEE transactions on pattern analysis and machine intelligence, vol. 34, no. 3, pp. 436–450, 2012. [9] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbelaez, A. Sorkine-Hornung, ´ and L. Van Gool, “The 2017 davis challenge on video object segmentation,” arXiv preprint arXiv:1704.00675, 2017. [10] W. Lotter, G. Kreiman, and D. Cox, “Deep predictive coding networks for video prediction and unsupervised learning,” arXiv preprint arXiv:1605.08104, 2016. [11] M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” arXiv preprint arXiv:1511.05440, 2015. [12] Q. Zhou, X. Liang, K. Gong, and L. Lin, “Adaptive temporal encoding network for video instance-level human parsing,” in Proceedings of the 26th ACM international conference on Multimedia, 2018, pp. 1527– 1535. [13] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black, “Towards understanding action recognition,” in Proceedings of the IEEE international conference on computer vision, 2013, pp. 3192–3199. [14] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in Proceedings of the 25th international conference on Machine learning, 2008, pp. 1096–1103. [15] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol, and L. Bottou, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.” Journal of machine learning research, vol. 11, no. 12, 2010. [16] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, “Context encoders: Feature learning by inpainting,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2536–2544. [17] R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14. Springer, 2016, pp. 649–666. [18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018. [19] H. Bao, L. Dong, S. Piao, and F. Wei, “Beit: Bert pre-training of image transformers,” arXiv preprint arXiv:2106.08254, 2021. [20] X. Dong, J. Bao, T. Zhang, D. Chen, W. Zhang, L. Yuan, D. Chen, F. Wen, N. Yu, and B. Guo, “Peco: Perceptual codebook for bert pre-training of vision transformers,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 1, 2023, pp. 552– 560. [21] C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer, “Masked feature prediction for self-supervised visual pre-training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 668–14 678. [22] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, “Simmim: A simple framework for masked image modeling,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9653–9663. [23] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong, “ibot: Image bert pre-training with online tokenizer,” arXiv preprint arXiv:2111.07832, 2021. [24] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, “Generative pretraining from pixels,” in International conference on machine learning. PMLR, 2020, pp. 1691–1703. [25] J. Walker, C. Doersch, A. Gupta, and M. Hebert, “An uncertain future: Forecasting from static images using variational autoencoders,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14. Springer, 2016, pp. 835–851. [26] A. Gupta, S. Tian, Y. Zhang, J. Wu, R. Mart´ın-Mart´ın, and L. FeiFei, “Maskvit: Masked visual pre-training for video prediction,” arXiv preprint arXiv:2206.11894, 2022. [27] C. Vondrick, H. Pirsiavash, and A. Torralba, “Anticipating visual representations from unlabeled video,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 98– 106. [28] D. Pathak, R. Girshick, P. Dollar, T. Darrell, and B. Hariharan, ´ “Learning features by watching objects move,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2701–2710. [29] N. Srivastava, E. Mansimov, and R. Salakhudinov, “Unsupervised learning of video representations using lstms,” in International conference on machine learning. PMLR, 2015, pp. 843–852. [30] X. Wang, A. Jabri, and A. A. Efros, “Learning correspondence from the cycle-consistency of time,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2566–2576. [31] X. Wang and A. Gupta, “Unsupervised learning of visual representations using videos,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2794–2802. [32] C. Vondrick, A. Shrivastava, A. Fathi, S. Guadarrama, and K. Murphy, “Tracking emerges by colorizing videos,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 391–408. [33] B. Fernando, H. Bilen, E. Gavves, and S. Gould, “Self-supervised video representation learning with odd-one-out networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3636–3645. [34] H.-Y. Lee, J.-B. Huang, M. Singh, and M.-H. Yang, “Unsupervised representation learning by sorting sequences,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 667–676. [35] I. Misra, C. L. Zitnick, and M. Hebert, “Shuffle and learn: unsupervised learning using temporal order verification,” in Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14. Springer, 2016, pp. 527–544. [36] D. Wei, J. J. Lim, A. Zisserman, and W. T. Freeman, “Learning and using the arrow of time,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8052–8060. [37] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), vol. 2. IEEE, 2006, pp. 1735–1742. [38] S. Becker and G. E. Hinton, “Self-organizing neural network that discovers surfaces in random-dot stereograms,” Nature, vol. 355, no. 6356, pp. 161–163, 1992. [39] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738. [40] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning. PMLR, 2020, pp. 1597–1607. [41] X. Chen and K. He, “Exploring simple siamese representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 15 750–15 758. [42] J.-B. Grill, F. Strub, F. Altche, C. Tallec, P. Richemond, ´ E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al., “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in neural information processing systems, vol. 33, pp. 21 271–21 284, 2020. [43] M. Caron, H. Touvron, I. Misra, H. Jegou, J. Mairal, P. Bojanowski, ´ and A. Joulin, “Emerging properties in self-supervised vision transformers,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9650–9660. [44] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” Advances in neural information processing systems, vol. 33, pp. 9912–9924, 2020. [45] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [46] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. [47] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017. [48] E. Hoffer, T. Ben-Nun, I. Hubara, N. Giladi, T. Hoefler, and D. Soudry, “Augment your batch: Improving generalization through instance repetition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8129–8138. [49] A. Jabri, A. Owens, and A. Efros, “Space-time correspondence as a contrastive random walk,” Advances in neural information processing systems, vol. 33, pp. 19 545–19 560, 2020. [50] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” arXiv preprint arXiv:2003.04297, 2020. [51] J. Xu and X. Wang, “Rethinking self-supervised correspondence learning: A video frame-level similarity perspective,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10 075–10 085.

Concatenated Masked Autoencoders as Spatial-Temporal Learner: Conclusion & References

This paper is available on arxiv under CC 4.0 license. Authors: (1) Zhouqiang Jiang, Meetyou AI Lab, (2) Bowen Wang, Institute for Datability Science, Osaka University, (3) Tong Xiang, Meetyou AI Lab, (4) Zhaofeng Niu, Department of Computer Science, Qufu Normal University, (5) Hong Tang, Department of Information Engineering, East China Jiaotong University, (6) Guangshun Li, Department of Computer Science, Qufu Normal University, (7) Liangzhi Li, Meetyou AI Lab. Table of Links Abstract & Introduction Related Work Method Experiments Conclusion & References IV. EXPERIMENTS A. Pre-training By default, we use ViT-S/16 as the encoder, with a depth of 12 and a width of 384, which is similar to ResNet-50 [46] in terms of the number of parameters (21M vs. 23M). We use a small decoder with a width of 256 and a depth of 4. The effectiveness of this setup is demonstrated in Table IIIb and IIIc. Architecture. . We conduct self-supervised pre-training on the Kinetics-400 [7] action recognition dataset, which encompasses a total of 400 classes, comprising 240k training videos and 20k validation videos, and we only conduct pretraining on the training set. Considering the cost of training, we default to sampling three frames (224x224) from each video at certain frame intervals, and employ two image augmentation methods: RandomResizeCrop and horizontal flipping. A large portion (95%) of the patches in the last two frames are randomly masked, and the batch size is 2048. The base learning rate is initialized to 1e-4 and follows a cosine schedule decay. We use the AdamW [47] optimizer, and the reconstruction losses for the second and 3rd frames are scaled by 0.8 and 1.0, respectively. We adopt a repetition sampling [4], [48] factor of 2 and finally conduct pre-training for 800 effective epochs. Settings B. Evaluation Method We conduct two downstream experiments, video segmentation and action recognition, to evaluate the correspondence and motion information of the obtained representations. We use the semi supervised video segmentation dataset DAVIS-2017 [9] to evaluate the quality of representations in fine-grained correspondence tasks. We adopt the label propagation method used in [49], [4] to implement semi-supervised video segmentation. Given the ground truth labels of the first frame, we obtain k-nearest neighbor results through a similarity function between frames. Afterward, the ground truth labels of the first frame are propagated to subsequent frames as pixel labels, based on the k-nearest neighbors’ results. Fine-grained Correspondence Task. Action Recognition Task. We followed the evaluation protocol of VideoMAE [2] for the motion evaluation on action recognition task by Kinetics-400 [7]. In order to avoid a variety of meticulously crafted fine-tuning settings obstructing fair comparisons between different pre-training methods, we used the same pipeline to uniformly fine-tune all pre-trained models on the training set for 150 epochs and report the results on the validation set. To fairly compare with video MAEs [2], [3], we also will extend the 2D patch embedding layer to a 3D embedding layer following the approach proposed in [7]. C. Main Results and Analysis Table I demonstrates all the methods using 480x880 images. We find that CatMAE-ViT-S/8 surpasses all other self-supervised or supervised methods, and its performance is further enhanced when ViRe is applied. This demonstrates that using reversed videos as reconstruction targets can further enhance the model’s ability to match correspondences. As analyzed before, the cube masking recovery strategy cannot learn the inter-frame correspondence, resulting in MAE-ST and VideoMAE being significantly lower than CatMAE. In addition, CatMAE also outperforms SiamMAE by a large margin, owning a better segmentation performance with larger frame intervals (Table IIIa), proving that using the concatenated information channel masking strategy can capture longer and more accurate correspondences in videos. Fine-grained Correspondence Task. We implement this experiment with two competitive video representation methods VideoMAE and SiamMAE. All methods undergo 800 epochs pretraining. In Table II, it is observed that during the early stage of fine-tuning, CatMAE exhibits a significantly stronger transfer capability than SiamMAE, with an enhancement in top-1 accuracy by 5.73%. This indicates that our proposed concatenated information channel masking reconstruction pre-training strategy can capture more motion information than the asymmetric mask reconstruction strategy of SiamMAE. Even when the fine-tuning saturates, CatMAE still leads SiamMAE by 2.08% in top-1 accuracy. However, the gap between CatMAE and VideoMAE widen, which we believe mainly stems from the method of modeling tokens. VideoMAE directly models token embedding using a 3D cube, which essentially makes VideoMAE a type of 3D network. VideoMAE naturally contains more motion information than CatMAE which encodes 2D patch as tokens. Yet, in the initial stage of fine-tuning, CatMAE manages to maintain comparable transfer ability to VideoMAE (only 1.29% lower in top-1 accuracy), performing far better than the SiamMAE (7.02% lower). Our CatMAE maximally exploits the motion information on the 2D patch. Action Recognition Task. D. Ablation Experiments We conduct an ablation experiment to understand each component’s contribution. Details can be found in Table III. Number of Reconstructions. Fig. 1 shows the model’s reconstruction results for future frames. As can be seen, the model is highly sensitive to motion elements in video frames. In Fig. 1a, the model can distinguish between static backgrounds and accurately reconstruct the motion of waving hammers. Beyond its motion modeling capability, the model can copy the background from the first frame and paste it into future frames when dealing with static backgrounds (such as the static woodpile). As shown in Table IIIa, the video segmentation performance steadily improves with the increase in the number of reconstructions. When the number of reconstructions is 4, the performance is higher in comparison with the number being 1 (↑3.9%). This validates the benefits of using a concatenated information channel masking strategy to extend multi-frame reconstruction. In Table IIIb and IIIc, we respectively examine the width and depth of the decoder. We found that relative to the encoder (384-d, 12-block), a smaller decoder (192-d, 4-block) is critically important for fine-grained correspondence tasks. This can be explained by the relationship between pixel reconstruction and fine-grained feature correspondence: during pixel reconstruction, the encoder is sensitive to motion offsets, thus obtaining a more reasonable correspondence representation. However, when the decoder capacity is too large, it weakens the encoder’s sensitivity to motion. Therefore, in terms of the width and depth settings for the decoder, an optimal width of 256 is 4.3% better than the maximum width of 1024 (59.9% vs. 55.6%). Additionally, an optimal depth of 4 is 2.6% better than the depth of 8 (60.0% vs. 57.4%). These optimal decoder capacities are much smaller than the encoder, underscoring the importance of utilizing a smaller decoder. Decoder Capacity. In Table IIId, we compare different mask ratio. Based on previous experience with video MAEs, we default to ablating from a high mask ratio (90%), but under such a high mask ratio, increasing the mask ratio of the last two frames to 95% still shows performance improvement (54.5% vs. 56.0%). However, when using an extreme mask ratio of 99% (Only one patch is visible), the performance drops sharply. An information channel with limited capacity is crucial for the model to learn correspondences. However, when the channel capacity is reduced to its extreme limit, it becomes difficult for the model to match correspondences. Hence, we default to using a mask ratio of 95%. Mask Ratio. In Table IIIe, we compared the reconstruction loss scale between the second and third frames. We found that slightly emphasizing the reconstruction of the third frame is better than that of the second (compare the second and third rows in Table IIIe). We believe that focusing more on the third frame’s reconstruction facilitates the progressive transmission of continuous motion information to the final frame. However, neglecting the reconstruction of the second frame by reducing its loss scale to 0.5 leads to performance degradation. This might weaken the model’s capability to capture the intermediate motion, thus affecting the propagation of continuous motion information. Loss Scale. In Table IIIf, we compare different frame gaps. While a larger frame gap sampling can encompass richer motion information, reconstructing two frames with large gaps does not bring performance improvement to the model (compare the first and fourth rows). This may be because when the frame gap becomes too large, the scene changes are too dramatic, which hinders the model’s ability to model motion information. Additionally, we found that having a smaller gap between the first and second frames and a larger gap between the second and third frames (see the third row in Table) performs better than the reverse configuration (see the second row in Table) (56.8% vs. 55.2%). This observation can also be interpreted as the propagation of continuous motion being hindered: a large gap between the first and second frames makes capturing intermediate motion difficult, thereby blocking the transmission of continuous motion information within the information channels. Frame Gap. In Fig. 3, we analyze the impact of training epochs and propagation FPS on video segmentation, discovering that optimal performance is achieved at pre-training 400 epochs, after which the performance gradually declines, possibly due to overfitting caused by small decoder. We observe that under various low FPS conditions, CatMAE consistently outperforms SiamMAE, demonstrating that CatMAE possesses a stronger ability to match long-term correspondences. Additionally, we delve into the influence of fine-tuning epochs. CatMAE initially exhibits robust transferability but the gap widens with VideoMAE over time due to differences in token embedding. Simplistically speaking, CatMAE can be seen as a ”additional frame reconstruction” version of SiamMAE. Therefore, according to Action Recognition figure, we estimate that using CatMAE to pre-train weights for reconstructing future four frames holds promise for reaching the fine-tuning performance of VideoMAE. Training Epochs and Propagation FPS. E. Motion and Correspondence Analysis In Fig. 4, we visualize the cross-attention map averaged across different heads in the final layer of the ViT-S/16 decoder. Specifically, we input the first frame and the subsequent two frames with masks, followed by reconstructing the third frame. We collect the cross-attention from the mask patch in the third frame towards the visible patches in the first and second frames and then visualize this attention map. We find that the model is sensitive to motion, focusing on the correspondence between patches under continuous motion. For instance (Fig. 4a), when calculating the correspondence of the horse’s head patch in the third frame, the model can clearly distinguish the semantics of the brown region and attention accurately focused on the head of the horse in previous frames. Even when the background color is similar to the possible corresponding area (such as the white head of the water bird and the reflective water surface Fig. 4b), the model’s corresponding capability remains robust. Certainly, the model can also establish correspondences by leveraging unique color features (such as the red bill of a swan Fig. 4c) or specific semantic structures (such as the fin of an Orca Fig. 4d). These provide direct evidence of the model’s ability to estimate motion and learn correspondences.

Concatenated Masked Autoencoders as Spatial-Temporal Learner: Experiments

This paper is available on arxiv under CC 4.0 license. Authors: (1) Zhouqiang Jiang, Meetyou AI Lab, (2) Bowen Wang, Institute for Datability Science, Osaka University, (3) Tong Xiang, Meetyou AI Lab, (4) Zhaofeng Niu, Department of Computer Science, Qufu Normal University, (5) Hong Tang, Department of Information Engineering, East China Jiaotong University, (6) Guangshun Li, Department of Computer Science, Qufu Normal University, (7) Liangzhi Li, Meetyou AI Lab. Table of Links Abstract & Introduction Related Work Method Experiments Conclusion & References III. METHOD In this section, We introduce each key component of CatMAE. The overall model architecture is shown in Fig 2. First, we chronologically select N frames from a video clip as input sequence. The interval between frames is randomly chosen from a pre-determined frame gap range. Then, following the process of ViT [5], we divide each frame into non-overlapping patches. We apply linear projection and flatten the patches, concatenate a [CLS] token to patches and eventually add position embedding to them. Patch Embedding. We keep the first frame completely visible and apply masking with a high masking rate to N−1 subsequent frames, preserving only a minimal number of visible patches. Given that video signals are highly redundant [8], high masking rate prevents the model from utilizing information from adjacent frames during the reconstruction process, thus encouraging the encoder to capture motion information and correspondence. Concatenated Information Channel Masking. Visible patches in subsequent frames provide information channels, theoretically allowing the frames’ sequence to be reconstructed indefinitely. To reconstruct the final frame, the model needs to reconstruct the continuous motion and correspondence of the intermediate frames, ultimately realizing the evolution between frames. (as shown in Fig. 1). . We employ weight-shared vanilla ViT [5] to independently process all frames, with each encoder only encoding visible patches. This design significantly reduces temporal and memory complexity while achieving better performance [1]. Furthermore, the patch embedding only adds a position embedding in the spatial structure, hence, the encoder is unaware of the temporal structure of the patches. However, the decoder’s cross-attention layer needs to use the encoder’s output to propagate the visible content. Even when the temporal structure is unknown, the encoder is forced to match the correspondence between spatial-temporal patches before and after motion. This correspondence assists the decoder in utilizing the information channels to propagate visible content, ultimately achieving the reconstruction of subsequent frames. Encoder . We also employ a weight-shared decoder to separately predict the reconstructed frames’ masked patches. Each decoder block consists of a cross-attention layer and a self-attention layer [45]. Specifically, the visible tokens of the frame to be reconstructed are projected via a linear layer and combined with mask tokens to form a set of full tokens, to which spatial position embedding is then added. Subsequently, the full tokens attend to all previously visible tokens (tokens of the first frame and other visible tokens) through a cross-attention layer, followed by mutual attention through a self-attention layer. Our concatenated information channel masking enables the decoder to enhance the encoder’s ability to estimate motion offsets and learn correspondences. Finally, The output sequence of the decoder is used to predict the normalized pixel values in the masked patches [1], with an L2 loss applied between the decoder’s prediction and ground truth. Decoder

Stories, News & Mentions about TEMPO

Tempo’s Stories on HackerNoon

Tempo's News & Mentions