Empirical

empirical

Table of Links Abstract and 1. Introduction 1.1 Research Questions and Contributions Related Work


Background


Inscriptions and ordinals
4.1 Operation Types
4.2 Comparison with NFTs and ERC-20s


Data Collection


Empirical Analysis and 6.1 Overall Transactions
6.2 Inscriptions Characterization
6.3 Inscription Trading
6.4 Impact on Gas Fees


Discussion


Conclusion and References 6 Empirical Analysis This section presents the empirical analysis of the inscriptions recorded on Ethereum and its major EVM-compatible rollups: Arbitrum, Base, Optimism, and zkSync Era. We characterize the inscription protocols, operations, tokens, and their trading dynamics. Subsequently, we investigate the impact of sudden surges in inscription transactions on the blockchains’ performance. 6.1 Overall Transactions Multiple spikes in inscription transaction counts occurred on Arbitrum (November 25th , December 15th, 16th, and 18th) and zkSync Era (November 17th, 16th, 17th, and December 21st) as shown in Figure 2a. During peak periods, inscriptions dominated daily transactions, comprising nearly 90% on zkSync Era and Arbitrum (December 16th, 2023), over 50% on Ethereum (January 14th, 2023), and almost 35% on Optimism and Base (January 24th, 2024). Figure 2b illustrates those peaks. Due to this high activity of inscriptions on-chain, some rollups were not prepared to handle the surge in transactions, leading to downtimes and delays in processing transactions [33, 15]. Next, we delve into the protocols and operations contributing to these observed spikes. 6.2 Inscriptions Characterization Our research indicates that inscription tokens are uniquely unified by a protocol-tick pair. Therefore, we aggregate data using this pair to identify the top tokens. Figure 3 illustrates the breakdown of inscription transactions based on protocol, operation, and the leading token. Below, we provide our observations for each of these categories. Protocols. Our initial observation reveals that each blockchain contains a particular leading inscription protocol, similar to BRC-20 on Bitcoin, constituting between 54% and 93% of transactions. Moreover, blockchain-specific standards such as FAIR-20 for Arbitrum, ERC-20 for Ethereum, and ZRC-20 for zkSync Era exhibit prominence. The LAYER2-20 protocol facilitates minting and trading of the inscription token across multiple rollups. It is present across all studied L2s, emerging as the dominant inscription protocol on Base and Optimism, and the second largest on zkSync Era. Operations. The new inscription standards resemble the original BRC-20 standard but introduce additional operation types such as claim, list, buy, and sell, which were absent in the original BRC-20 implementation on Bitcoin. Notably, the claim operation works similarly to mint, generating new tokens. Hence, we aggregate mint and claim operations, constituting approximately 99% of all transactions related to inscriptions across all studied chains. Understandably, a small number of transactions are attributed to the list operation, used for listing inscriptions for sale in an inscription marketplace, and the deploy operation, which declares the token on the chain along with its token supply. The low percentage of transfer, sell, and buy transactions suggests that inscription tokens are not yet actively traded on EVM chains. Tokens. Based on our observations, various protocols deploy and mint tokens with the same ticks. Therefore, we identify each inscription token by a protocol-tick pair. We found that Arbitrum, Base, and Optimism each have one dominant token, representing between 73% and 85% of inscription transactions. The distribution of tokens is most diverse on Ethereum, with ERC-20 NODES token contributing 32% of transactions, and each other tokens less than 3.5%. The four largest tokens on zkSync Era account for over 90% of inscription transactions and zkSync Era has more powerful leading tokens than in other chains. The token LAYER2-20 $L2 is present on all rollups, being also the dominant token on Base and Optimism. Authors:
(1) Johnnatan Messias, Matter Labs;
(2) Krzysztof Gogol, Matter Labs, University of Zurich;
(3) Maria Inês, Silva Matter Labs;
(4) Benjamin Livshits, Matter Labs, Imperial College London. This paper is available on arxiv under CC BY 4.0 DEED license. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 1.1 Research Questions and Contributions 1.1 Research Questions and Contributions Related Work Background Inscriptions and ordinals
4.1 Operation Types
4.2 Comparison with NFTs and ERC-20s Data Collection Empirical Analysis and 6.1 Overall Transactions
6.2 Inscriptions Characterization
6.3 Inscription Trading
6.4 Impact on Gas Fees Discussion Conclusion and References Related Work Related Work Related Work Background Background Background Inscriptions and ordinals 4.1 Operation Types 4.2 Comparison with NFTs and ERC-20s Inscriptions and ordinals Inscriptions and ordinals 4.1 Operation Types 4.1 Operation Types 4.2 Comparison with NFTs and ERC-20s 4.2 Comparison with NFTs and ERC-20s Data Collection Data Collection Data Collection Empirical Analysis and 6.1 Overall Transactions 6.2 Inscriptions Characterization 6.3 Inscription Trading 6.4 Impact on Gas Fees Empirical Analysis and 6.1 Overall Transactions Empirical Analysis and 6.1 Overall Transactions 6.2 Inscriptions Characterization 6.2 Inscriptions Characterization 6.3 Inscription Trading 6.3 Inscription Trading 6.4 Impact on Gas Fees 6.4 Impact on Gas Fees Discussion Discussion Discussion Conclusion and References Conclusion and References Conclusion and References 6 Empirical Analysis This section presents the empirical analysis of the inscriptions recorded on Ethereum and its major EVM-compatible rollups: Arbitrum, Base, Optimism, and zkSync Era. We characterize the inscription protocols, operations, tokens, and their trading dynamics. Subsequently, we investigate the impact of sudden surges in inscription transactions on the blockchains’ performance. 6.1 Overall Transactions Multiple spikes in inscription transaction counts occurred on Arbitrum (November 25th , December 15th, 16th, and 18th) and zkSync Era (November 17th, 16th, 17th, and December 21st) as shown in Figure 2a. During peak periods, inscriptions dominated daily transactions, comprising nearly 90% on zkSync Era and Arbitrum (December 16th, 2023), over 50% on Ethereum (January 14th, 2023), and almost 35% on Optimism and Base (January 24th, 2024). Figure 2b illustrates those peaks. Due to this high activity of inscriptions on-chain, some rollups were not prepared to handle the surge in transactions, leading to downtimes and delays in processing transactions [33, 15]. Next, we delve into the protocols and operations contributing to these observed spikes. 6.2 Inscriptions Characterization Our research indicates that inscription tokens are uniquely unified by a protocol-tick pair. Therefore, we aggregate data using this pair to identify the top tokens. Figure 3 illustrates the breakdown of inscription transactions based on protocol, operation, and the leading token. Below, we provide our observations for each of these categories. Protocols . Our initial observation reveals that each blockchain contains a particular leading inscription protocol, similar to BRC-20 on Bitcoin, constituting between 54% and Protocols Protocols 93% of transactions. Moreover, blockchain-specific standards such as FAIR-20 for Arbitrum, ERC-20 for Ethereum, and ZRC-20 for zkSync Era exhibit prominence. The LAYER2-20 protocol facilitates minting and trading of the inscription token across multiple rollups. It is present across all studied L2s, emerging as the dominant inscription protocol on Base and Optimism, and the second largest on zkSync Era. Operations . The new inscription standards resemble the original BRC-20 standard but introduce additional operation types such as claim, list, buy, and sell, which were absent in the original BRC-20 implementation on Bitcoin. Notably, the claim operation works similarly to mint, generating new tokens. Hence, we aggregate mint and claim operations, constituting approximately 99% of all transactions related to inscriptions across all studied chains. Understandably, a small number of transactions are attributed to the list operation, used for listing inscriptions for sale in an inscription marketplace, and the deploy operation, which declares the token on the chain along with its token supply. The low percentage of transfer, sell, and buy transactions suggests that inscription tokens are not yet actively traded on EVM chains. Operations Operations Tokens . Based on our observations, various protocols deploy and mint tokens with the same ticks. Therefore, we identify each inscription token by a protocol-tick pair. We found that Arbitrum, Base, and Optimism each have one dominant token, representing between 73% and 85% of inscription transactions. The distribution of tokens is most diverse on Ethereum, with ERC-20 NODES token contributing 32% of transactions, and each other tokens less than 3.5%. The four largest tokens on zkSync Era account for over 90% of Tokens Tokens inscription transactions and zkSync Era has more powerful leading tokens than in other chains. The token LAYER2-20 $L2 is present on all rollups, being also the dominant token on Base and Optimism. Authors: (1) Johnnatan Messias, Matter Labs; (2) Krzysztof Gogol, Matter Labs, University of Zurich; (3) Maria Inês, Silva Matter Labs; (4) Benjamin Livshits, Matter Labs, Imperial College London. Authors : Authors (1) Johnnatan Messias, Matter Labs; (2) Krzysztof Gogol, Matter Labs, University of Zurich; (3) Maria Inês, Silva Matter Labs; (4) Benjamin Livshits, Matter Labs, Imperial College London. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Empirical Analysis of Inscriptions on Ethereum and EVM-Compatible Rollups

Empirical Robustness and Parameter Selection in Macroeconomic Modeling 

Table of Links Abstract and 1 Introduction 1.1 The twincode platform 1.2 Related Work 2 Research Questions 3 Variables 3.1 Independent Variables 3.2 Dependent Variables 3.3 Confounding Variables 4 Participants 5 Execution Plan and 5.1 Recruitment 5.2 Training and 5.3 Experiment Execution 5.4 Data Analysis Acknowledgments and References 1.2 Related Work Several literature reviews [10, 15, 21] have compiled the empirical research on using pair programming in higher education, with [6] being focused on distributed pair programming from a teaching perspective. By means of controlled experiments, remote and co–located pair programming are compared in [1, 22], showing comparable results. In most of the cases, the analyzed variables are related to performance in terms of time, quality, or code tests passed. Students perceptions have been also analyzed in terms of confidence, satisfaction, motivation, or personality [20]. Table 1 summarizes the empirical studies on the influence of gender in pair programming, including findings such as (i) same– gender pairs are more “democratic”; (ii) women working in pairs were more confident than those working solo; and (iii) in mixedgender, pairing women particularly do not benefit [15]. Although such studies reveal that gender seems to be a key factor in pair programming, none of them study gender bias in pair programming. Many factors other than gender may affect the outcomes of remote programming sessions [2, 23]. Previous research on productive pairing looked at factors such as skill levels, autonomy in choosing one’s partner [25], and different personalities [11]. Nevertheless, the work on gender composition of pairs found conflicting results about whether same-gender or mixed-gender pairings are more effective [3, 4, 12, 16]. One possible explanation is that gender correlates with other dimensions that may affect the pairs’ collaboration, but these correlations may vary between different environments. For example, women in a class may, on average, have higher skill level than men because they had to face more societal barriers to enter the class. On the other hand, they may, on average, have lower skill level if women with no background are more actively recruited. Authors:
(1) Amador Durán, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (amador@us.es);
(2) Pablo Fernández, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (pablofm@us.es);
(3) Beatriz Bernárdez, I3US Institute, Universidad de Sevilla, Sevilla, Spain (beat@us.es);
(4) Nathaniel Weinman, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (nweinman@berkeley.edu);
(5) Aslı Akalın, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (asliakalin@berkeley.edu);
(6) Armando Fox, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (fox@berkeley.edu). This paper is available on arxiv under CC BY 4.0 DEED license. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 1.1 The twincode platform 1.1 The twincode platform 1.2 Related Work 1.2 Related Work 2 Research Questions 2 Research Questions 3 Variables 3 Variables 3.1 Independent Variables 3.1 Independent Variables 3.2 Dependent Variables 3.2 Dependent Variables 3.3 Confounding Variables 3.3 Confounding Variables 4 Participants 4 Participants 5 Execution Plan and 5.1 Recruitment 5 Execution Plan and 5.1 Recruitment 5.2 Training and 5.3 Experiment Execution 5.2 Training and 5.3 Experiment Execution 5.4 Data Analysis 5.4 Data Analysis Acknowledgments and References Acknowledgments and References 1.2 Related Work Several literature reviews [10, 15, 21] have compiled the empirical research on using pair programming in higher education, with [6] being focused on distributed pair programming from a teaching perspective. By means of controlled experiments, remote and co–located pair programming are compared in [1, 22], showing comparable results. In most of the cases, the analyzed variables are related to performance in terms of time, quality, or code tests passed. Students perceptions have been also analyzed in terms of confidence, satisfaction, motivation, or personality [20]. Table 1 summarizes the empirical studies on the influence of gender in pair programming, including findings such as (i) same– gender pairs are more “democratic”; (ii) women working in pairs were more confident than those working solo; and (iii) in mixedgender, pairing women particularly do not benefit [15]. Although such studies reveal that gender seems to be a key factor in pair programming, none of them study gender bias in pair programming. Many factors other than gender may affect the outcomes of remote programming sessions [2, 23]. Previous research on productive pairing looked at factors such as skill levels, autonomy in choosing one’s partner [25], and different personalities [11]. Nevertheless, the work on gender composition of pairs found conflicting results about whether same-gender or mixed-gender pairings are more effective [3, 4, 12, 16]. One possible explanation is that gender correlates with other dimensions that may affect the pairs’ collaboration, but these correlations may vary between different environments. For example, women in a class may, on average, have higher skill level than men because they had to face more societal barriers to enter the class. On the other hand, they may, on average, have lower skill level if women with no background are more actively recruited. Authors: (1) Amador Durán, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (amador@us.es); (2) Pablo Fernández, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (pablofm@us.es); (3) Beatriz Bernárdez, I3US Institute, Universidad de Sevilla, Sevilla, Spain (beat@us.es); (4) Nathaniel Weinman, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (nweinman@berkeley.edu); (5) Aslı Akalın, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (asliakalin@berkeley.edu); (6) Armando Fox, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (fox@berkeley.edu). Authors: Authors: (1) Amador Durán, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (amador@us.es); (2) Pablo Fernández, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (pablofm@us.es); (3) Beatriz Bernárdez, I3US Institute, Universidad de Sevilla, Sevilla, Spain (beat@us.es); (4) Nathaniel Weinman, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (nweinman@berkeley.edu); (5) Aslı Akalın, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (asliakalin@berkeley.edu); (6) Armando Fox, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (fox@berkeley.edu). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Examining Gender Bias in Pair Programming: Insights from Empirical Studies

Authors:
(1) Abraham Owodunni, Intron Health, Masakhane, and this author contributed equally;
(2) Aditya Yadavalli, Karya, Masakhane, and this author contributed equally;
(3) Chris Emezuem, Mila Quebec AI Institute, Lanfrica, Masakhane, and this author contributed equally;
(4) Tobi Olatunji, Intron Health and Masakhane, and this author contributed equally;
(5) Clinton Mbataku, AI Saturdays Lagos. Table of Links Abstract and 1 Introduction 2 Related Work 3 AccentFold 4 What information does AccentFold capture? 5 Empirical study of AccentFold 6 Conclusion, Limitations, and References 5 Empirical study of AccentFold 5.1 Problem Formulation In this empirical study, we set out to understand how informative the accent folds are for accentlevel zero shot ASR performance. To achieve this, we designed our experimental task as follows: Assume we have the below oracle data set generator: 5.2 Experimental Setup Fine-tuning Details: We use a pre-trained XLSR model (Conneau et al., 2020) for our experiments. The XLSR model extends the wav2vec 2.0 (Baevski et al., 2020) model to the cross-lingual setting and was trained to acquire cross-lingual speech representations through the utilization of a singular model that is pre-trained using raw speech waveforms from various languages. The fact that this model is cross-lingual makes it a good fit for our experiments. During the fine-tuning of our pre-trained model, we follow the hyperparameter settings of Olatunji et al. (2023a). These include setting the dropout rates for attention and hidden layers to 0.1, while keeping the feature projection dropout at 0.0. We also employ a mask probability of 0.05 and a layerdrop rate of 0.1. Additionally, we enable gradient checkpointing to reduce memory usage. The learning rate is set to 3e-4, with a warm-up period of 1541 steps. The batch sizes for training and validation are 16 and 8, respectively, and we train the model for ten epochs. For each of the 41 target accents, we finetune our pre-trained model on its corresponding dataset and evaluate the word error rate on the test set comprising audio samples containing only the target accent. We run all our experiments using a 40GB NVIDIA A100 SXM GPU, which enables parallel use of its GPU nodes. Evaluation procedure: It is important to note that although the training dataset size Nk depends on the target accent aOOD in consideration, the test set used to evaluate all our experiments is fixed: it comprises the samples from the test split of the Afrispeech-200. Using Figure 1 the test set are samples from all the 108 accents of the test split. By keeping the test set constant, we can assess the model’s performance on our intended accent aOOD in an out-of-distribution (OOD) scenario. This is because the training and development splits do not include any audio-speech samples from these accents. Additionally, this procedure enables us to evaluate the model’s capacity to generalize to other accent samples, resulting in a highly resilient evaluation. 5.3 Results and Discussion Table 2 presents the results of a test Word Error Rate (WER) comparison between three different approaches for subset selection: AccentFold, GeoProx, and random sampling. The table displays the average and standard deviation of the WER values over the 41 target OOD accents. The results show that the AccentFold approach achieves the lowest test WER of 0.332 with a standard deviation of 0.013. In contrast, the random sampling approach yields the highest test WER of 0.367 with a larger standard deviation of 0.034. GeoProx, which uses real-world geographical proximity of the accents, performs better than random sampling but still under-performs when compared to AccentFold. To better understand this, we investigate the accents selected by AccentFold and GeoProx and analyse their non-overlapping accents in Figure 6. The histogram reveals that many of the accents selected by AccentFold for any given target OOD accent, aOOD, are not necessarily those geographically closest to aOOD. This insight suggests that the learned embeddings in AccentFold encompass much more than geographical proximity of accents. Figure 5 visualizes the test WER obtained by AccentFold and random sampling for each of the 41 accents. We see that in majority of the accents, AccentFold leads to improved performancte in terms of WER compared to random sampling. These findings indicate that AccentFold effectively captures linguistic relationships among accents, allowing for more accurate recognition of the target OOD accent when used to build the fine-tuning dataset. This demonstrates the usefulness of leveraging linguistic information and accent embeddings provided by AccentFold in the context of automatic speech recognition tasks. We notice a pattern, as shown in Figure 7, where increasing the value of s, which corresponds to a larger training dataset size Nk, results in minimal variation in the selection of accent subsets. This convergence of test WER implies that as the sample size increases, the specific choice of accent subsets becomes less influential in determining the performance. This paper is available on arxiv under CC BY-SA 4.0 DEED license. Authors: (1) Abraham Owodunni, Intron Health, Masakhane, and this author contributed equally; (2) Aditya Yadavalli, Karya, Masakhane, and this author contributed equally; (3) Chris Emezuem, Mila Quebec AI Institute, Lanfrica, Masakhane, and this author contributed equally; (4) Tobi Olatunji, Intron Health and Masakhane, and this author contributed equally; (5) Clinton Mbataku, AI Saturdays Lagos. Authors: Authors: (1) Abraham Owodunni, Intron Health, Masakhane, and this author contributed equally; (2) Aditya Yadavalli, Karya, Masakhane, and this author contributed equally; (3) Chris Emezuem, Mila Quebec AI Institute, Lanfrica, Masakhane, and this author contributed equally; (4) Tobi Olatunji, Intron Health and Masakhane, and this author contributed equally; (5) Clinton Mbataku, AI Saturdays Lagos. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 AccentFold 3 AccentFold 4 What information does AccentFold capture? 4 What information does AccentFold capture? 5 Empirical study of AccentFold 5 Empirical study of AccentFold 6 Conclusion, Limitations, and References 6 Conclusion, Limitations, and References 5 Empirical study of AccentFold 5.1 Problem Formulation In this empirical study, we set out to understand how informative the accent folds are for accentlevel zero shot ASR performance. To achieve this, we designed our experimental task as follows: Assume we have the below oracle data set generator: 5.2 Experimental Setup Fine-tuning Details : We use a pre-trained XLSR model (Conneau et al., 2020) for our experiments. The XLSR model extends the wav2vec 2.0 (Baevski et al., 2020) model to the cross-lingual setting and was trained to acquire cross-lingual speech representations through the utilization of a singular model that is pre-trained using raw speech waveforms from various languages. The fact that this model is cross-lingual makes it a good fit for our experiments. Fine-tuning Details During the fine-tuning of our pre-trained model, we follow the hyperparameter settings of Olatunji et al. (2023a). These include setting the dropout rates for attention and hidden layers to 0.1, while keeping the feature projection dropout at 0.0. We also employ a mask probability of 0.05 and a layerdrop rate of 0.1. Additionally, we enable gradient checkpointing to reduce memory usage. The learning rate is set to 3e-4, with a warm-up period of 1541 steps. The batch sizes for training and validation are 16 and 8, respectively, and we train the model for ten epochs. For each of the 41 target accents, we finetune our pre-trained model on its corresponding dataset and evaluate the word error rate on the test set comprising audio samples containing only the target accent. We run all our experiments using a 40GB NVIDIA A100 SXM GPU, which enables parallel use of its GPU nodes. Evaluation procedure: It is important to note that although the training dataset size Nk depends on the target accent aOOD in consideration, the test set used to evaluate all our experiments is fixed: it comprises the samples from the test split of the Afrispeech-200. Using Figure 1 the test set are samples from all the 108 accents of the test split. By keeping the test set constant, we can assess the model’s performance on our intended accent aOOD in an out-of-distribution (OOD) scenario. This is because the training and development splits do not include any audio-speech samples from these accents. Additionally, this procedure enables us to evaluate the model’s capacity to generalize to other accent samples, resulting in a highly resilient evaluation. 5.3 Results and Discussion Table 2 presents the results of a test Word Error Rate (WER) comparison between three different approaches for subset selection: AccentFold, GeoProx, and random sampling. The table displays the average and standard deviation of the WER values over the 41 target OOD accents. The results show that the AccentFold approach achieves the lowest test WER of 0.332 with a standard deviation of 0.013. In contrast, the random sampling approach yields the highest test WER of 0.367 with a larger standard deviation of 0.034. GeoProx, which uses real-world geographical proximity of the accents, performs better than random sampling but still under-performs when compared to AccentFold. To better understand this, we investigate the accents selected by AccentFold and GeoProx and analyse their non-overlapping accents in Figure 6. The histogram reveals that many of the accents selected by AccentFold for any given target OOD accent, aOOD, are not necessarily those geographically closest to aOOD. This insight suggests that the learned embeddings in AccentFold encompass much more than geographical proximity of accents. Figure 5 visualizes the test WER obtained by AccentFold and random sampling for each of the 41 accents. We see that in majority of the accents, AccentFold leads to improved performancte in terms of WER compared to random sampling. These findings indicate that AccentFold effectively captures linguistic relationships among accents, allowing for more accurate recognition of the target OOD accent when used to build the fine-tuning dataset. This demonstrates the usefulness of leveraging linguistic information and accent embeddings provided by AccentFold in the context of automatic speech recognition tasks. We notice a pattern, as shown in Figure 7, where increasing the value of s, which corresponds to a larger training dataset size Nk, results in minimal variation in the selection of accent subsets. This convergence of test WER implies that as the sample size increases, the specific choice of accent subsets becomes less influential in determining the performance. This paper is available on arxiv under CC BY-SA 4.0 DEED license. This paper is available on arxiv under CC BY-SA 4.0 DEED license. available on arxiv

AccentFold: Enhancing Accent Recognition - Empirical Study of AccentFold

Author:
(1) Yigit Ege Bayiz, Electrical and Computer Engineering The University of Texas at Austin Austin, Texas, USA (Email: egebayiz@utexas.edu);
(2) Ufuk Topcu, Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, Texas, USA (Email: utopcu@utexas.edu). Table of Links Abstract and Introduction Related Works Preliminaries Optimal Prebunking Problem Deterministic Baseline Policies Temporally Equidistant Prebunking Numerical Results Conclusion and References VIII. CONCLUSION We define the problem of optimally delivering prebunks to a user as a minimax optimization problem, and under SI propagation assumptions propose algorithms that guarantee feasibility. We demonstrate that our theoretically backed approach Algorithm 3 also yields better results than the other two baselines in empirical analysis using simulated misinformation propagations on Chung-Lu models. Algorithm 3 is also often computationally feasible to solve, as at each time, it relies on solving a linear program that is computationally inexpensive. Our results, however, suffer from the limitations we impose on the network structure. Real-world misinformation propagation often deviates significantly from the SI model predictions. Our models also only provide feasibility guarantees under the discrete-time epidemic propagation assumptions. We also focus solely on delivering optimal prebunks to each user one by one, which is different from optimizing prebunk deliveries on the entire network. For future work, we plan to extend our problem and results to optimizing misinformation deliveries across the entire social network. REFERENCES [1] Social media fact sheet, Apr. 2021. [Online]. Available: https://www.pewresearch.org/internet/fact-sheet/socialmedia/. [2] B. Swire-Thompson, D. Lazer, et al., “Public health and online misinformation: Challenges and recommendations,” Annu Rev Public Health, vol. 41, no. 1, pp. 433–451, 2020. [3] Y. Benkler, R. Faris, and H. Roberts, Network propaganda: Manipulation, disinformation, and radicalization in American politics. Oxford University Press, 2018. [4] S. van der Linden, A. Leiserowitz, S. Rosenthal, and E. Maibach, “Inoculating the public against misinformation about climate change,” Global Challenges, vol. 1, no. 2, 2017. DOI: https : / / doi . org / 10 . 1002 / gch2 . 201600008. [5] S. Lewandowsky, U. K. H. Ecker, and J. Cook, “Beyond misinformation: Understanding and coping with the “post-truth” era.,” Journal of Applied Research in Memory and Cognition, vol. 6, pp. 353–369, 4 2017. DOI: 10.1016/j.jarmac.2017.07.008. [6] G. Pennycook and D. G. Rand, “The psychology of fake news,” Trends in Cognitive Sciences, vol. 25, no. 5, pp. 388–402, 2021. DOI: https://doi.org/10.1016/j.tics. 2021.02.007. [7] F. Liu and M. Buss, “Optimal control for heterogeneous node-based information epidemics over social networks,” IEEE Transactions on Control of Network Systems, vol. 7, no. 3, pp. 1115–1126, 2020. DOI: 10. 1109/TCNS.2019.2963488. [8] Y. E. Bayiz and U. Topcu, Countering misinformation on social networks using graph alterations, 2022. arXiv: 2211.04617 [cs.SI]. [9] Z. Guo, M. Schlichtkrull, and A. Vlachos, “A Survey on Automated Fact-Checking,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 178– 206, Feb. 2022. DOI: 10.1162/tacl a 00454. [10] I. Augenstein, C. Lioma, D. Wang, et al., “MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims,” Association for Computational Linguistics, 2019. DOI: 10.18653/v1/D19-1475. [11] U. K. H. Ecker, S. Lewandowsky, J. Cook, et al., “The psychological drivers of misinformation belief and its resistance to correction,” Nature Reviews Psychology, vol. 1, no. 1, pp. 13–29, Jan. 2022. DOI: 10 . 1038 / s44159-021-00006-y. [12] I. Chaturvedi, E. Cambria, R. E. Welsch, and F. Herrera, “Distinguishing between facts and opinions for sentiment analysis: Survey and challenges,” Information Fusion, vol. 44, pp. 65–77, 2018. DOI: https://doi.org/ 10.1016/j.inffus.2017.12.006. [13] M. Mohtarami, R. Baly, J. Glass, P. Nakov, L. Marquez, ` and A. Moschitti, “Automatic stance detection using end-to-end memory networks,” Association for Computational Linguistics, Jun. 2018. DOI: 10.18653/v1/N18- 1070. [14] M. Del Vicario, A. Bessi, F. Zollo, et al., “The spreading of misinformation online,” Proceedings of the National Academy of Sciences, vol. 113, no. 3, pp. 554–559, 2016. DOI: 10.1073/pnas.1517441113. [15] S. Shaar, N. Babulkov, G. Da San Martino, and P. Nakov, “That is a known lie: Detecting previously fact-checked claims,” Association for Computational Linguistics, 2020, pp. 3607–3618. DOI: 10.18653/v1/ 2020.acl-main.332. [16] A. L. Hill, D. G. Rand, M. A. Nowak, and N. A. Christakis, “Infectious disease modeling of social contagion in networks,” PLOS computational biology, vol. 6, no. 11, 2010. [17] S. Raponi, Z. Khalifa, G. Oligeri, and R. Di Pietro, “Fake news propagation: A review of epidemic models, datasets, and insights,” ACM Trans. Web, vol. 16, no. 3, Sep. 2022, ISSN: 1559-1131. DOI: 10.1145/3522756. [Online]. Available: https://doi.org/10.1145/3522756. [18] S. Krishnasamy, S. Banerjee, and S. Shakkottai, “The behavior of epidemics under bounded susceptibility,” SIGMETRICS Perform. Eval. Rev., 2014. DOI: 10.1145/ 2637364.2591977. [19] L. Zhao, H. Cui, X. Qiu, X. Wang, and J. Wang, “Sir rumor spreading model in the new media age,” Physica A: Statistical Mechanics and its Applications, vol. 392, no. 4, pp. 995–1003, 2013. [20] Y.-Q. Wang and J. Wang, “Sir rumor spreading model considering the effect of difference in nodes’ identification capabilities,” International Journal of Modern Physics, vol. 28, no. 05, 2017. [21] M. Kimura, K. Saito, and H. Motoda, “Efficient estimation of influence functions for sis model on social networks,” in International Joint Conference on Artificial Intelligence, 2009. [22] F. Jin, E. Dougherty, P. Saraf, Y. Cao, and N. Ramakrishnan, “Epidemiological modeling of news and rumors on twitter,” in Proceedings of Workshop on Social Network Mining and Analysis, 2013. [23] F. Chung and L. Lu, “Connected components in random graphs with given expected degree sequences,” Annals of Combinatorics, vol. 6, no. 2, pp. 125–145, 2002, ISSN: 0219-3094. DOI: 10.1007/PL00012580. [24] D. Fasino, A. Tonetto, and F. Tudisco, “Generating large scale-free networks with the chung–lu random graph model,” Networks, vol. 78, no. 2, pp. 174–187, 2021. [25] K. Pogorelov, D. T. Schroeder, P. Filkukova, S. Brenner, ´ and J. Langguth, “Wico text: A labeled dataset of conspiracy theory and 5g-corona misinformation tweets,” in Workshop on Open Challenges in Online Social Networks. Association for Computing Machinery, 2021. [26] J.-L. Guillaume, M. Latapy, and C. Magnien, “Comparison of failures and attacks on random and scale-free networks,” in OPODIS 2004, Springer, 2005. [27] P. Crucitti, V. Latora, M. Marchiori, and A. Rapisarda, “Efficiency of scale-free networks: Error and attack tolerance,” Physica A: Statistical Mechanics and its Applications, vol. 320, pp. 622–642, 2003. DOI: https: //doi.org/10.1016/S0378-4371(02)01545-5. This paper is available on arxiv under CC 4.0 license. Author: (1) Yigit Ege Bayiz, Electrical and Computer Engineering The University of Texas at Austin Austin, Texas, USA (Email: egebayiz@utexas.edu); (2) Ufuk Topcu, Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, Texas, USA (Email: utopcu@utexas.edu). Author: Author: (1) Yigit Ege Bayiz, Electrical and Computer Engineering The University of Texas at Austin Austin, Texas, USA (Email: egebayiz@utexas.edu); (2) Ufuk Topcu, Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, Texas, USA (Email: utopcu@utexas.edu). Table of Links Abstract and Introduction Abstract and Introduction Related Works Related Works Preliminaries Preliminaries Optimal Prebunking Problem Optimal Prebunking Problem Deterministic Baseline Policies Deterministic Baseline Policies Temporally Equidistant Prebunking Temporally Equidistant Prebunking Numerical Results Numerical Results Conclusion and References Conclusion and References VIII. CONCLUSION We define the problem of optimally delivering prebunks to a user as a minimax optimization problem, and under SI propagation assumptions propose algorithms that guarantee feasibility. We demonstrate that our theoretically backed approach Algorithm 3 also yields better results than the other two baselines in empirical analysis using simulated misinformation propagations on Chung-Lu models. Algorithm 3 is also often computationally feasible to solve, as at each time, it relies on solving a linear program that is computationally inexpensive. Our results, however, suffer from the limitations we impose on the network structure. Real-world misinformation propagation often deviates significantly from the SI model predictions. Our models also only provide feasibility guarantees under the discrete-time epidemic propagation assumptions. We also focus solely on delivering optimal prebunks to each user one by one, which is different from optimizing prebunk deliveries on the entire network. For future work, we plan to extend our problem and results to optimizing misinformation deliveries across the entire social network. REFERENCES [1] Social media fact sheet, Apr. 2021. [Online]. Available: https://www.pewresearch.org/internet/fact-sheet/socialmedia/. [2] B. Swire-Thompson, D. Lazer, et al., “Public health and online misinformation: Challenges and recommendations,” Annu Rev Public Health, vol. 41, no. 1, pp. 433–451, 2020. [3] Y. Benkler, R. Faris, and H. Roberts, Network propaganda: Manipulation, disinformation, and radicalization in American politics. Oxford University Press, 2018. [4] S. van der Linden, A. Leiserowitz, S. Rosenthal, and E. Maibach, “Inoculating the public against misinformation about climate change,” Global Challenges, vol. 1, no. 2, 2017. DOI: https : / / doi . org / 10 . 1002 / gch2 . 201600008. [5] S. Lewandowsky, U. K. H. Ecker, and J. Cook, “Beyond misinformation: Understanding and coping with the “post-truth” era.,” Journal of Applied Research in Memory and Cognition, vol. 6, pp. 353–369, 4 2017. DOI: 10.1016/j.jarmac.2017.07.008. [6] G. Pennycook and D. G. Rand, “The psychology of fake news,” Trends in Cognitive Sciences, vol. 25, no. 5, pp. 388–402, 2021. DOI: https://doi.org/10.1016/j.tics. 2021.02.007. [7] F. Liu and M. Buss, “Optimal control for heterogeneous node-based information epidemics over social networks,” IEEE Transactions on Control of Network Systems, vol. 7, no. 3, pp. 1115–1126, 2020. DOI: 10. 1109/TCNS.2019.2963488. [8] Y. E. Bayiz and U. Topcu, Countering misinformation on social networks using graph alterations, 2022. arXiv: 2211.04617 [cs.SI]. [9] Z. Guo, M. Schlichtkrull, and A. Vlachos, “A Survey on Automated Fact-Checking,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 178– 206, Feb. 2022. DOI: 10.1162/tacl a 00454. [10] I. Augenstein, C. Lioma, D. Wang, et al., “MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims,” Association for Computational Linguistics, 2019. DOI: 10.18653/v1/D19-1475. [11] U. K. H. Ecker, S. Lewandowsky, J. Cook, et al., “The psychological drivers of misinformation belief and its resistance to correction,” Nature Reviews Psychology, vol. 1, no. 1, pp. 13–29, Jan. 2022. DOI: 10 . 1038 / s44159-021-00006-y. [12] I. Chaturvedi, E. Cambria, R. E. Welsch, and F. Herrera, “Distinguishing between facts and opinions for sentiment analysis: Survey and challenges,” Information Fusion, vol. 44, pp. 65–77, 2018. DOI: https://doi.org/ 10.1016/j.inffus.2017.12.006. [13] M. Mohtarami, R. Baly, J. Glass, P. Nakov, L. Marquez, ` and A. Moschitti, “Automatic stance detection using end-to-end memory networks,” Association for Computational Linguistics, Jun. 2018. DOI: 10.18653/v1/N18- 1070. [14] M. Del Vicario, A. Bessi, F. Zollo, et al., “The spreading of misinformation online,” Proceedings of the National Academy of Sciences, vol. 113, no. 3, pp. 554–559, 2016. DOI: 10.1073/pnas.1517441113. [15] S. Shaar, N. Babulkov, G. Da San Martino, and P. Nakov, “That is a known lie: Detecting previously fact-checked claims,” Association for Computational Linguistics, 2020, pp. 3607–3618. DOI: 10.18653/v1/ 2020.acl-main.332. [16] A. L. Hill, D. G. Rand, M. A. Nowak, and N. A. Christakis, “Infectious disease modeling of social contagion in networks,” PLOS computational biology, vol. 6, no. 11, 2010. [17] S. Raponi, Z. Khalifa, G. Oligeri, and R. Di Pietro, “Fake news propagation: A review of epidemic models, datasets, and insights,” ACM Trans. Web, vol. 16, no. 3, Sep. 2022, ISSN: 1559-1131. DOI: 10.1145/3522756. [Online]. Available: https://doi.org/10.1145/3522756. [18] S. Krishnasamy, S. Banerjee, and S. Shakkottai, “The behavior of epidemics under bounded susceptibility,” SIGMETRICS Perform. Eval. Rev., 2014. DOI: 10.1145/ 2637364.2591977. [19] L. Zhao, H. Cui, X. Qiu, X. Wang, and J. Wang, “Sir rumor spreading model in the new media age,” Physica A: Statistical Mechanics and its Applications, vol. 392, no. 4, pp. 995–1003, 2013. [20] Y.-Q. Wang and J. Wang, “Sir rumor spreading model considering the effect of difference in nodes’ identification capabilities,” International Journal of Modern Physics, vol. 28, no. 05, 2017. [21] M. Kimura, K. Saito, and H. Motoda, “Efficient estimation of influence functions for sis model on social networks,” in International Joint Conference on Artificial Intelligence, 2009. [22] F. Jin, E. Dougherty, P. Saraf, Y. Cao, and N. Ramakrishnan, “Epidemiological modeling of news and rumors on twitter,” in Proceedings of Workshop on Social Network Mining and Analysis, 2013. [23] F. Chung and L. Lu, “Connected components in random graphs with given expected degree sequences,” Annals of Combinatorics, vol. 6, no. 2, pp. 125–145, 2002, ISSN: 0219-3094. DOI: 10.1007/PL00012580. [24] D. Fasino, A. Tonetto, and F. Tudisco, “Generating large scale-free networks with the chung–lu random graph model,” Networks, vol. 78, no. 2, pp. 174–187, 2021. [25] K. Pogorelov, D. T. Schroeder, P. Filkukova, S. Brenner, ´ and J. Langguth, “Wico text: A labeled dataset of conspiracy theory and 5g-corona misinformation tweets,” in Workshop on Open Challenges in Online Social Networks. Association for Computing Machinery, 2021. [26] J.-L. Guillaume, M. Latapy, and C. Magnien, “Comparison of failures and attacks on random and scale-free networks,” in OPODIS 2004, Springer, 2005. [27] P. Crucitti, V. Latora, M. Marchiori, and A. Rapisarda, “Efficiency of scale-free networks: Error and attack tolerance,” Physica A: Statistical Mechanics and its Applications, vol. 320, pp. 622–642, 2003. DOI: https: //doi.org/10.1016/S0378-4371(02)01545-5. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Empirical Success in Prebunking

Authors:
(1) Jonathan H. Rystrøm. Table of Links Abstract and Introduction Previous Literature Methods and Data Results Discussions Conclusions and References A. Validation of Assumptions B. Other Models C. Pre-processing steps 2 Previous Literature Several papers have highlighted the need to protect human autonomy and preferences in recommender systems (Calvo et al., 2020; Varshney, 2020; Jannach & Adomavicius, 2016). However, these focus more on the normative need to do so, rather than an empirical analysis of the phenomenon. With this paper, we aim to fill this gap in the literature. Most previous empirical analysis of preference manipulation in recommender systems has focused on the MovieLensdataset (Harper & Konstan, 2016). MovieLens is a movie recommendation platform developed and maintained by the University of Minnesota.[1] The advantages of this dataset are that it is a) rigorously documented as it is maintained by an academic group; b) freely available; and c) well-structured, thus making analysis easier. However, because it is a non-commercial project it is not susceptible to surveillance capitalistic imperatives to the same extent as e.g. Amazon (Zuboff, 2019). Nguyen et al. (2014) analysed whether MovieLens users were exposed to less diverse content over time - a type of preference manipulation. While they found a significant (albeit small) decrease, the effect was smaller for users who followed the recommendations than the users who did not. However, as discussed by Rakova and Chowdhury (2019), they focus their analysis on highly-active and highly-nonactive users neglecting the middle. Also, content diversity is an important but incomplete measure of preference manipulation. Rakova and Chowdhury (2019) also use the MovieLens dataset to define and showcase Barrier-to-Exit. However, their paper is more of a proof-of-concept rather than an analysis. The present paper expands on Rakova and Chowdhury (2019) by applying the metric to a real-world dataset of a commercial recommender system. [1] movielens.org This paper is available on arxiv under CC 4.0 license. Authors: (1) Jonathan H. Rystrøm. Authors: Authors: (1) Jonathan H. Rystrøm. Table of Links Abstract and Introduction Abstract and Introduction Previous Literature Previous Literature Methods and Data Methods and Data Results Results Discussions Discussions Conclusions and References Conclusions and References A. Validation of Assumptions A. Validation of Assumptions B. Other Models B. Other Models C. Pre-processing steps C. Pre-processing steps 2 Previous Literature Several papers have highlighted the need to protect human autonomy and preferences in recommender systems (Calvo et al., 2020; Varshney, 2020; Jannach & Adomavicius, 2016). However, these focus more on the normative need to do so, rather than an empirical analysis of the phenomenon. With this paper, we aim to fill this gap in the literature. Most previous empirical analysis of preference manipulation in recommender systems has focused on the MovieLensdataset (Harper & Konstan, 2016). MovieLens is a movie recommendation platform developed and maintained by the University of Minnesota.[1] The advantages of this dataset are that it is a) rigorously documented as it is maintained by an academic group; b) freely available; and c) well-structured, thus making analysis easier. However, because it is a non-commercial project it is not susceptible to surveillance capitalistic imperatives to the same extent as e.g. Amazon (Zuboff, 2019). Nguyen et al. (2014) analysed whether MovieLens users were exposed to less diverse content over time - a type of preference manipulation. While they found a significant (albeit small) decrease, the effect was smaller for users who followed the recommendations than the users who did not. However, as discussed by Rakova and Chowdhury (2019), they focus their analysis on highly-active and highly-nonactive users neglecting the middle. Also, content diversity is an important but incomplete measure of preference manipulation. Rakova and Chowdhury (2019) also use the MovieLens dataset to define and showcase Barrier-to-Exit. However, their paper is more of a proof-of-concept rather than an analysis. The present paper expands on Rakova and Chowdhury (2019) by applying the metric to a real-world dataset of a commercial recommender system. [1] movielens.org This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Empirical Insights into Preference Manipulation

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Oded Blumenthal, Software and Information Systems Engineering, Ben Gurion University, Israel;
(2) Guy Shani, Software and Information Systems Engineering, Ben Gurion University, Israel. Table of Links Introduction
Background
Related Work
POMCP for Stochastic Contingent Planning
Domain Independent Heuristics for POMCP
Empirical Evaluation
Conclusion and References 6 Empirical Evaluation We conduct an empirical study to evaluate our methods. Our methods are implemented in C#. 6.1 Benchmark Domains We extended the following contingent planning benchmarks to stochastic settings: Doors: In the door domain the agent must move in a grid to reach a target position. Odd levels in the grid are all open, while in even levels there are doors, and only one door is open. The agent can sense whether a door is open when it is at adjacent cells. The agent must identify the open doors and get to the target position. In the stochastic version the agent can open a closed door with some probability of success. The agent can hence either search for the already open door, or attempt to open a closed door. Blocks World: In the contingent blocks world problem, the agent does not know the structure of the initial block configuration, but it can sense whether one block is on top of another one, and whether a block is clear. In the stochastic version moving a block from one block to another has a 0.3 probability of success, while moving blocks to and from the table succeeds deterministicly. Hence, it is often preferable to use the table as an intermediate position. Unix: In this domain the agent must search for a file in a file system, and copy it to a destination folder. In the stochastic version there is a non-uniform distribution over the possible locations of the file. MedPks: The agent here needs to identify which illness a patient has and treat it. To do so, the agent tests for each illness independently, until the proper illness is found. The stochastic version here has non-uniform distribution over the possible illnesses as well. Localize: In this domain the agent must reach a goal position in a grid. The agent does not know where it initially is, and can only sense adjacent walls. In the stochastic version there several places in the grid where the agent may slip and stay in place. This makes the localization in the grid more difficult. Wumpus: In this challenging problem the agent must reach a target position in a grid infested by monsters called Wumpuses. Cells may be unsafe to travel as they may contain either a Wumpus or a pit. Wumpuses emit a stench, and pits emit a breeze, both of which can be sensed in adjacent cells. The agent must sense in multiple positions to identify the safe cells. The stochastic version here also has non-uniform distribution over the safe cells. 6.2 Results For each domain above we run 20 online episodes, and compute the success rate, the average run time for computing the next action, and the average cost to the goal. We did not use a timeout, but runs longer than 100 steps were considered to be stuck in a loop, and terminated. We begin by looking at the quality of the policy — the average cost to the goal. As can be seen, the random rollout policy is best only in the MedPks domain, and close to best in unix. This is not too surprising, because in these two domains the best strategy is very simple, and random strategies easily stumble upon the goal. In blocks all methods achieved similar performance, because the optimal strategy is very short, and rollouts are less important. This domain has many possible actions, and hence a huge branching factor, making it difficult to scale up using POMCP. On doors and localize, which require lengthier trajectories to reach the goal, but do not need long information-gathering efforts, the single state hadd strategy operates very well. However, on Wumpus, where long sequences of actions are needed for information gathering, the multiple-state heuristic works best. We expected the results to be that way, although we expected more significant difference between the "smart" heuristics and the random rollout. With respect to the required time to run the simulations for a single decision, the results are mixed. While obviously the random strategy requires no time to compute the next action during a rollout, it often results in lengthy rollouts, which reduce this effect. The single state heuristic is almost always faster than the multi-state heuristic, but not by much. This paper is available on arxiv under CC 4.0 license. Authors: (1) Oded Blumenthal, Software and Information Systems Engineering, Ben Gurion University, Israel; (2) Guy Shani, Software and Information Systems Engineering, Ben Gurion University, Israel. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Oded Blumenthal, Software and Information Systems Engineering, Ben Gurion University, Israel; (2) Guy Shani, Software and Information Systems Engineering, Ben Gurion University, Israel. Table of Links Introduction Background Related Work POMCP for Stochastic Contingent Planning Domain Independent Heuristics for POMCP Empirical Evaluation Conclusion and References Introduction Introduction Background Background Related Work Related Work POMCP for Stochastic Contingent Planning POMCP for Stochastic Contingent Planning Domain Independent Heuristics for POMCP Domain Independent Heuristics for POMCP Empirical Evaluation Empirical Evaluation Conclusion and References Conclusion and References 6 Empirical Evaluation We conduct an empirical study to evaluate our methods. Our methods are implemented in C#. 6.1 Benchmark Domains We extended the following contingent planning benchmarks to stochastic settings: Doors : In the door domain the agent must move in a grid to reach a target position. Odd levels in the grid are all open, while in even levels there are doors, and only one door is open. The agent can sense whether a door is open when it is at adjacent cells. The agent must identify the open doors and get to the target position. In the stochastic version the agent can open a closed door with some probability of success. The agent can hence either search for the already open door, or attempt to open a closed door. Doors Blocks World: In the contingent blocks world problem, the agent does not know the structure of the initial block configuration, but it can sense whether one block is on top of another one, and whether a block is clear. In the stochastic version moving a block from one block to another has a 0.3 probability of success, while moving blocks to and from the table succeeds deterministicly. Hence, it is often preferable to use the table as an intermediate position. Unix: In this domain the agent must search for a file in a file system, and copy it to a destination folder. In the stochastic version there is a non-uniform distribution over the possible locations of the file. MedPks : The agent here needs to identify which illness a patient has and treat it. To do so, the agent tests for each illness independently, until the proper illness is found. The stochastic version here has non-uniform distribution over the possible illnesses as well. MedPks Localize : In this domain the agent must reach a goal position in a grid. The agent does not know where it initially is, and can only sense adjacent walls. In the stochastic version there several places in the grid where the agent may slip and stay in place. This makes the localization in the grid more difficult. Localize Wumpus : In this challenging problem the agent must reach a target position in a grid infested by monsters called Wumpuses. Cells may be unsafe to travel as they may contain either a Wumpus or a pit. Wumpuses emit a stench, and pits emit a breeze, both of which can be sensed in adjacent cells. The agent must sense in multiple positions to identify the safe cells. The stochastic version here also has non-uniform distribution over the safe cells. Wumpus 6.2 Results For each domain above we run 20 online episodes, and compute the success rate, the average run time for computing the next action, and the average cost to the goal. We did not use a timeout, but runs longer than 100 steps were considered to be stuck in a loop, and terminated. We begin by looking at the quality of the policy — the average cost to the goal. As can be seen, the random rollout policy is best only in the MedPks domain, and close to best in unix. This is not too surprising, because in these two domains the best strategy is very simple, and random strategies easily stumble upon the goal. In blocks all methods achieved similar performance, because the optimal strategy is very short, and rollouts are less important. This domain has many possible actions, and hence a huge branching factor, making it difficult to scale up using POMCP. On doors and localize, which require lengthier trajectories to reach the goal, but do not need long information-gathering efforts, the single state h add strategy operates very well. However, on Wumpus, where long sequences of actions are needed for information gathering, the multiple-state heuristic works best. We expected the results to be that way, although we expected more significant difference between the "smart" heuristics and the random rollout. add With respect to the required time to run the simulations for a single decision, the results are mixed. While obviously the random strategy requires no time to compute the next action during a rollout, it often results in lengthy rollouts, which reduce this effect. The single state heuristic is almost always faster than the multi-state heuristic, but not by much.

Rollout Heuristics for Online Stochastic Contingent Planning: Empirical Evaluation

Authors: (1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China; (2) Peng Liang, School of Computer Science, Wuhan University, Wuhan, China; (3) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China; (4) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany; (4) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia; (4) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland. II. METHODOLOGY The goal of this study is to systematically identity the issues that developers encountered in the practical application of GitHub Copilot, as well as their underlying causes and potential solutions. We formulated three RQs to direct subsequent phases of the methodology as shown in Fig. 1. The RQs and their rationales are detailed in Section II-A. A. Research Questions Rationale: Copilot is a relatively new product, and little is known about the specific challenges and issues users face while using it in the software development practice. By identifying these issues, this study can help us gain a better understanding of the obstacles that arise when using AI code generation tools like Copilot in practical situations from the perspective of software developers. RQ1: What are the issues faced by users while using Copilot in the software development practice? Rationale: Understanding the underlying causes of the issues identified in RQ1 is essential for developing effective solutions to address the issues. By identifying these causes, the study can provide insights into how to improve the design and functionality of Copilot. RQ2: What are the underlying causes of these issues? Rationale: RQ3 aims to identify potential solutions to the issues identified in RQ1, which can ultimately enhance the usability of Copilot for users. By exploring the existing practices and methods utilized by developers to tackle the issues, the study can gain insights into potential solutions that enhance the functionality and usability of Copilot. RQ3: What are the potential solutions to address these issues? B. Data Collection We collected data from three sources: GitHub Issues[1] , GitHub Discussions[2] , and SO posts[3] . GitHub Issue is a commonly used feature on GitHub for tracking bugs, feature requests, and other issues related to software development projects, which allow us to capture the specific problems and difficulties that users have encountered when coding with Copilot. GitHub Discussion, on the other hand, is a newer feature for more open-ended discussions among project contributors and community members, which also offers a central hub for project-related discussions and knowledge sharing among community members. Discussion topics can cover a wide range from technical questions and proposals to broader topics related to Copilot. Stack Overflow is a popular technology community that provides a Q&A platform covering a wide range of programming, development, and technical questions, including Copilot-related questions. Considering that Copilot was announced and started its technical preview on June 29, 2021, we chose to collect the data that were created after that date. The data collection was conducted on June 18, 2023. To assist us in addressing RQ2 and RQ3, we chose to collect closed GitHub issues and answered GitHub discussions, which contain known causes and solutions. For SO posts, we found that the amount of Copilot-related posts was relatively low. Hence, we chose to retrieve all potentially relevant posts, including unanswered ones, to obtain a more comprehensive dataset. For GitHub Issues, we used “Copilot” as the keyword to search closed Copilot-related issues in the entire GitHub, and a total of 4,057 issues were retrieved. We also employed “Copilot” as the keyword to search at SO, resulting in 679 retrieved posts. Different from GitHub Issues and SO posts, GitHub Discussions related to certain product are grouped under a specific subcategory, and “Copilot” was included as a subcategory under the “Product” category. Given the high relevance of these discussions to Copilot, we collected all the discussions that were answered under the “Copilot” subcategory as part of the data source for our study. In total, 925 discussions were obtained. C. Data Filtering To ensure that the data can be used to answer our RQs, we set the filtering criterion: the issue, discussion, or post should contain specific information related to the use of Copilot. We conducted the data labelling on the collected data to filter out the data which cannot be used for this study by following the criterion. Pilot Data labelling: To minimize personal bias in the formal labelling process, the first and third authors conducted a pilot data labelling. For GitHub issues and discussions, we randomly selected 100 and 25 respectively, which constitute 2.5% of the total count. Due to the smaller quantity of SO posts, we randomly selected 35, which constitutes 5% of the total posts. Selecting a certain proportion of data from different platforms respectively is to verify whether the criteria of the two authors are consistent across various platforms. The consistency of the labelled issues was measured by the Cohen’s Kappa coefficient [14], resulting in values of 0.824, 0.834, and 0.806. The results indicate that a reasonable level of agreement between the two authors. For the labelling results with differences of opinion, the two authors will engage in discussions with the second author to reach a consensus. 2) Data labelling: The first and third authors then conducted the formal data labelling. During this process, we excluded a large number of data not related to our research. For instance, in some issues, Copilot refers to other meanings, such as the co-pilot of an aircraft. Additionally, Copilot might be mentioned in a straightforward manner without additional information, like someone said, “You can try using Copilot, which is amazing”. We also excluded such issues since they could not provide useful information about the use of Copilot. During the labelling process, any result on which the two authors disagreed was subject to discussion with the second author until an agreement was reached. Ultimately, the two authors collected 476 GitHub issues, 706 GitHub discussions, and 184 SO posts for data extraction. D. Data Extraction and Analysis Extract Data: To answer the three RQs mentioned in Section II-A, we established a set of data items for data extraction, as presented in Table I. Data items D1-D3 intend to extract the information of issues, underlying causes, and possible solutions from the filtered data, to answer RQ1-RQ3, respectively. The first author conducted a pilot data extraction with the third author on 20 randomly selected GitHub issues, 20 discussions, and 20 SO posts, and in case of any discrepancies, the second author was involved to reach a consensus. Based on the observation, we established the following standards for data extraction: (1) If the same issue was identified by multiple users, we recorded it only once. (2) If multiple problems were identified within the same issue, discussion, or posts, we recorded each one separately. (3) For an issue that has multiple causes, we only recorded the confirmed cause by the contributor of the issue or the Copilot team as the root cause. (4) For an issue that has multiple solutions suggested, we only recorded the solutions that were confirmed by the contributor of the issue or the Copilot team to actually solve the issue. The first and third authors conducted data extraction from the filtered issues, discussions, and posts based on the data items, and then discussed and reached a consensus with the second author on inconsistencies to ensure that the data extraction process adhered to the predetermined criteria. Each extracted data item was reviewed multiple times by the three authors to ensure accuracy. The final data extraction results were compiled and recorded in MS Excel [13]. 2) Analyze Data: To answer the three RQs in Section II, we conducted data analysis using the Constant Comparison method [12]. The specific steps are as follows: 1) The first author meticulously reviewed the data obtained during the data extraction phase and assigned codes to each data. These codes constituted descriptive summarizations of the data, aimed at capturing the underlying themes. For instance, the issue in Discussion #10598 was coded as “Stopped Giving Inline Suggestions”. 2) The first author compared different codes to identify patterns, commonalities, and distinctions among them. Through this iterative comparison process, similar codes coalesced into higher-level types and categories. For example, the coding of Discussion #10598, along with other akin codes, fell under the same type of FUNCTIONALITY USAGE ISSUE, which further belonged to the category of Usage Issue. Ultimately, the first author engaged in discussions with the second and third authors to achieve a consensus on the taxonomies of issues, causes, and solutions, which are presented in Section III. [1] https://docs.github.com/en/issues [2] https://github.com/orgs/community/discussions/categories/copilot [3] https://stackoverflow.com/ This paper is under CC 4.0 license. available on arxiv

GitHub Copilot in Practice: Empirical Insights into User Experiences and Practical Challenges

This paper is available on under CC 4.0 license. arxiv Authors: (1) Jakub DRÁPAL, Institute of State and Law of the Czech Academy of Sciences, Czechia, Institute of Criminal Law and Criminology, Leiden University, the Netherlands; (2) Hannes WESTERMANN, Cyberjustice Laboratory, Université de Montréal, Canada; (3) Jaromir SAVELKA, School of Computer Science, Carnegie Mellon University, USA. Table of Links Abstract & Introduction Related Work Dataset Proposed Framework Experimental Design Results and Discussion Conclusions, Future Work and References Abstract. Thematic analysis and other variants of inductive coding are widely used qualitative analytic methods within empirical legal studies (ELS). We propose a novel framework facilitating effective collaboration of a legal expert with a large language model (LLM) for generating initial codes (phase 2 of thematic analysis), searching for themes (phase 3), and classifying the data in terms of the themes (to kick-start phase 4). We employed the framework for an analysis of a dataset (n = 785) of facts descriptions from criminal court opinions regarding thefts. The goal of the analysis was to discover classes of typical thefts. Our results show that the LLM, namely OpenAI’s GPT-4, generated reasonable initial codes, and it was capable of improving the quality of the codes based on expert feedback. They also suggest that the model performed well in zero-shot classification of facts descriptions in terms of the themes. Finally, the themes autonomously discovered by the LLM appear to map fairly well to the themes arrived at by legal experts. These findings can be leveraged by legal researchers to guide their decisions in integrating LLMs into their thematic analyses, as well as other inductive coding projects. Thematic analysis, empirical legal studies, criminal law, large language models, generative pre-trained transformers, GPT-4 Keywords. 1. Introduction Empirical legal studies (ELS) is an approach to the study of law through empirical methods typical of economics, psychology, and sociology. Since law is a heavily text-based discipline ELS frequently focuses on text analytic methods, including deductive and inductive coding. Deductive coding focuses on applying a fixed set of codes to a dataset, whereas inductive coding leads to a simultaneous discovery of the codes from the data and their application. While investigations into various methods to support deductive coding have attracted much recent attention in AI & Law [1,2,3] very few studies focused on inductive coding [4]. One popular inductive coding method is “thematic analysis” [5]. J. Drápal, H. Westermann, and J. Savelka / Using LLMs to Support Thematic Analysis in ELS While rarely explicitly acknowledged thematic analysis is widely used in ELS. Figure 1 shows the six phases of the analysis, starting from the raw data and finishing with the scholarly report on the studied phenomena. We propose a novel LLM-powered framework to support a subject matter expert in performing phases 2 and 3 of the analysis. We employed the proposed framework in a thematic analysis of criminal courts’ opinions, focused on the criminal offense of theft in Czechia. Criminal offense categories (e.g., theft, murder) are usually defined in statutory law while the individual criminal acts are described by courts when they apply the law to factual circumstances of the cases. An important question in criminal law and criminology is what behaviors are actually criminalized and whether it is done appropriately. Neither the statutory definitions of the offenses (they are too general and not “sociologically relevant” [6]) nor the descriptions of the factual circumstances from cases (too specific) can answer the question. To get insight into what behavior is criminalized and how effectively, it is necessary to identify shared features of criminal acts, generalize them into “typical crimes” [7,8] and arriving at behavioral-based categories [9,10]. This is akin to performing thematic analysis. While important, such analysis is an expensive and time-consuming endeavour. Hence, a (semi-)automated approach would be useful. To assess the capability of a state-of-the-art LLM (GPT-4) to support selected stages of the thematic analysis, we investigated the following research questions: (RQ1) How successfully can the LLM perform initial coding of the data? (RQ2) To what degree can a subject matter expert improve the quality of the initial codes via natural language feedback? (RQ3) How successfully can the LLM predict themes for the analyzed data points? (RQ4) How successfully can the LLM autonomously discover themes and associate them with the analyzed data? This paper is available on under CC 4.0 license. arxiv

How to Use Large Language Models to Support Thematic Analysis in Empirical Legal Studies

Authors: (1) Davide Viviano, Department of Economics, Harvard University; (2) Lihua Lei, Graduate School of Business, Stanford University; (3) Guido Imbens, Graduate School of Business and Department of Economics, Stanford University; (4) Brian Karrer, FAIR, Meta; (5) Okke Schrijvers, Meta Central Applied Science; (6) Liang Shi, Meta Central Applied Science. Table of Links Abstract & Introduction Setup (When) should you cluster? Choosing the cluster design Empirical illustration and numerical studies Recommendations for practice References A) Notation B) Endogenous peer effects C) Proofs 5 Empirical illustration and numerical studies In this section we illustrate the properties of the procedure in two empirical applications, one using unique data from the Facebook friendship and messaging graphs, and one using data from an experiment conducted in rural China by Cai et al. [2015]. We provide further evidence supporting our theoretical analysis using simulated networks in Section 5.3. 5.1 Clustering on Facebook graphs We first study the procedure’s properties for two clustering algorithms implemented at scale on social networks owned by Meta: Louvain algorithm [Blondel et al., 2008], Balanced Partitioning [Kabiljo et al., 2017]. We consider two graphs owned by Meta. In each graph, edges are continuous variables. In the first graph, edges capture the strength of the friendship in the Facebook graph, and in the second graph they capture connections based on messaging on Facebook, in both cases the data was aggregated and de-identified. We compute our statistics by setting the edges to be zero if below the 5th, 10th, and 50th percentile. We will refer to these graphs obtained as dense, moderate (mod), and sparse after thresholding (graphs all have a bounded maximum degree). We report the bias and variance, with the variance weighted by a parameter ξn, defined respectively as Louvain and Balanced Partitioning clustering produce a hierarchical clustering structure with a growing number of clusters. We consider three “types” of such algorithms, corresponding to three levels in the clustering structure hierarchy (i.e., different numbers of clusters), defined as Type 1, 2, and 3, respectively. For Louvain clustering, higher-order types denote more clusters, whereas it is the opposite for Balanced Partitioning. For each clustering, we report log(Kn/n), log-ratio of the number of clusters over the population size. Table 2 collects each of these algorithms’ worst-case bias and variance and the number of clusters per individual. Louvain algorithm dominates Balanced Partitioning both in worstcase variance and bias. A larger number of clusters increases the bias but decreases the variance, which we observe throughout all graphs. The bias increases for sparser networks because the denominator |Ni | decreases. Figure 3 reports the objective function for different graphs, Louvain algorithms, and degrees of sparsity. For ξn ≈ 1, the Louvain algorithm with the largest number of clusters (Type 3) dominates the other two algorithms in all but one case. This result provides some interpretable comparisons for the number of clusters. We also observe a trade-off between the number of clusters and the degree of sparsity of the adjacency matrix as ξn decreases. Louvain clustering with the largest number of clusters (Type 1) is optimal for a dense graph with ξn ≫ 1. Such clustering, however, is sub-optimal as the graph becomes more sparse. This result is intuitive: for denser networks, a larger number of clusters may best control the estimator’s bias relative to the bias induced by other clustering algorithms. These comparisons motivate trade-offs in the choice of the graph (and its density) and the clustering algorithm. We recommend practitioners compare clustering by averaging objective functions over different sparsity thresholding. For example, it is possible to compare clustering based on the average objective across different graphs using some priors on the degree of sparsity without affecting our theoretical results. Finally, note that using Theorem 3.6 it is possible to compute the smallest values of the spillover effects that would motivate using a cluster instead of Bernoulli design. From Theorem 3.6, researchers should prefer a cluster over a Bernoulli design if In summary, our results shed light on using clustering algorithms for large-scale implementation on online platforms. These results suggest that Louvain clustering with possibly many clusters performs best in practice. It confirms intuitive arguments in Karrer et al. [2021] who also recommend Louvain clustering based on AA-tests. 5.2 Clustering in the field We consider as an application the problem of informing individuals to increase insurance takeup studied in Cai et al. [2015]. The authors collected network information in approximately 184 villages in rural China from 48 larger regions. We use network data collected by Cai et al. [2015] to study the properties of the proposed method, where we assume that two individuals are connected if at least one of the two indicates the other as a friend. Because we do not require information from end-line outcomes, we construct a network using information from surveyed individuals as well as their friends. The network has in total 7649 nodes, once we also include individuals who are friends of surveyed individuals (but do not necessarily live in their same village or were surveyed in the experiment). Individuals between villages present on average 50% of their connections outside their village. On the other hand, individuals in different regions (with approximately 50 regions) have 99% of their connections within the same region. We use an adjacency matrix a “weak ties” adjacency matrix, where two individuals are connected if either indicates the other has a friend. We study clustering within each region and report results for different clustering algorithms. Clustering within each region is performed as in Algorithm 1, where we first estimate Xˆ (K) via semi-definite programming, use K-means algorithm to retrieve the clusters from Xˆ (K) and iterate to estimate the optimal number of clusters K. We report the frontier with respect to the absolute bias consistently with Algorithm 1. 5.2.1 How many clusters should we choose? Figure 4 presents the main results. The left-hand side panel reports the average number of estimated clusters, estimated by the proposed method divided by the population size in a given cluster. Different choices of ξn justify a different number of clusters. For ξn = 1, the number of clusters is 7%, the total population size, whereas for ξn = 15, the number of clusters is roughly half the population size. To gain further intuition of its practical implication, take ψ¯ = 0.24 approximately equal to the outcomes’ variance in Cai et al. [2015] and ϕ¯ n = 0.27 as in Table 2 in Cai et al. [2015]. A recommended choice of ξn is approximately ψ/¯ ϕ¯2 n = 3.29. This choice puts slightly more weight on the bias than the variance when λ = 1, and therefore can be considered a conservative choice for the number of clusters.[13] Even with such a conservative choice, the suggested number of clusters is large relative to the population size. In this case, the optimal number of clusters is around 15% of the overall population size, approximately 600 clusters with a surveyed sample size of 3600 individuals. This number is consistent (in scale) with the choice in the number of clusters in other applications in development, such as Egger et al. [2022], Alatas et al. [2012], but larger than the number of villages that we have in the application in Cai et al. [2015]. 5.2.2 Comparison with other clustering methods Next, we compare the procedure (denoted as “Causal Clustering”) with the following alternative clustering algorithms: ε-net clustering as in Eckles et al. [2017] with ε = 3 as suggested in Eckles et al. [2017]; spectral clustering with a fixed number of clusters equal to n/3, where n is the population size in a given region (note that spectral clustering do not optimize over the number of clusters); Louvain clustering with default parameters selected by the R-package igraph; clustering based on village identity of each individual (creating one additional cluster for those individuals whose village is missing in the data). We contrast the objective function with alternative algorithms in the right-hand side of Figure 4. The proposed method achieves the smallest objective function across all competitors, providing suggestive evidence that Algorithm 1 (with a semi-definite relaxation) works particularly well in practice. For a smaller number of clusters the method is closer to the Louvain clustering algorithm, while for a larger number of clusters the objective of the algorithm is close to the objective of the spectral algorithm. Interestingly, clustering based on village identity leads to the largest loss function. This is because individuals are connected within and between villages in this application. Finally, in Figure 5 (first three panels), we report the mean-squared error obtained by simulating the model in Cai et al. [2015] (Table 2, Column 4) and vary the variance of the residuals σ 2 ∈ {1/4, 1/2, 1} to emulate settings with high, medium and low signal to noise ratio. The model in Cai et al. [2015] assumes that individual outcomes depend on neighbors’ treatments, consistently with our Assumption 1. Our method uniformly outperforms competitors in terms of mean-squared error, except for the Louvain algorithm when the signal-to-noise ratio is particularly high, but not in the remaining settings. In the last panel, we consider settings where the local interference assumption is violated, and we calibrate our model to the specification in Cai et al. [2015] (Column 4, Table 5), where the individual outcome depends on the neighbors’ outcomes, violating Assumption 1 and introducing global dependence. We fix the residuals’ variance to be σ 2 = 1/4, whereas the results are robust as we increase σ 2 . In this setup, we observe a larger mean-squared error of all methods, with the proposed method achieving the lowest mean-squared error. 5.3 A numerical study Next, we illustrate the properties of the method in numerical studies. We consider three different data generating processes for the network formation: a geometric network formation, Albert Barabasi network and Erdos-Renyi graph. The geometric network takes the form Ai,j = 1{|Xi,1 − Xj,1|/2 +|Xi,2 − Xj,2|/2 ≤ rn} where rn = p 4/2.75n similarly to simulations in Leung [2020]. Here, Xi,1, Xi,2 are drawn independently from a uniform distribution between [−1, 1]. For the Albert-Barabasi network we first draw n/5 edges uniformly according to Erdos-Renyi graph with probabilities p = 10/n, and second, we draw sequentially connections of the new nodes to the existing ones with probability equal to the number of connection of each pre-existing node divided by the overall number of connections. Finally, the Erdos-Renyi graph is generated with probability of connection p = 2/n. We study the properties of the proposed method and of the same competitors discussed in Section 5.2. Figure 6 showcases that the proposed procedure leads to the lowest objective function. This result is consistent with our theoretical findings. Figure 7 showcases that the number of clusters increases in the sample size, varies substantially across different data generating processes, and increases in ξn. [13] This follows from the fact that ξn is proportional to ψ/¯ ϕ¯2 n for clusters whose bias is of order O(ϕ¯ n), and researcher’s goal is to minimize the worst-case mean squared error (λ = 1). This paper is under CC 1.0 license. available on arxiv

Stories, News & Mentions about EMPIRICAL

Categories

Trending Topics

Stories, News & Mentions about EMPIRICAL

Empirical’s Stories on HackerNoon

Categories

Trending Topics