Empirical

empirical

Table of Links Abstract and 1 Introduction 1.1 The twincode platform 1.2 Related Work 2 Research Questions 3 Variables 3.1 Independent Variables 3.2 Dependent Variables 3.3 Confounding Variables 4 Participants 5 Execution Plan and 5.1 Recruitment 5.2 Training and 5.3 Experiment Execution 5.4 Data Analysis Acknowledgments and References 1.2 Related Work Several literature reviews [10, 15, 21] have compiled the empirical research on using pair programming in higher education, with [6] being focused on distributed pair programming from a teaching perspective. By means of controlled experiments, remote and co–located pair programming are compared in [1, 22], showing comparable results. In most of the cases, the analyzed variables are related to performance in terms of time, quality, or code tests passed. Students perceptions have been also analyzed in terms of confidence, satisfaction, motivation, or personality [20]. Table 1 summarizes the empirical studies on the influence of gender in pair programming, including findings such as (i) same– gender pairs are more “democratic”; (ii) women working in pairs were more confident than those working solo; and (iii) in mixedgender, pairing women particularly do not benefit [15]. Although such studies reveal that gender seems to be a key factor in pair programming, none of them study gender bias in pair programming. Many factors other than gender may affect the outcomes of remote programming sessions [2, 23]. Previous research on productive pairing looked at factors such as skill levels, autonomy in choosing one’s partner [25], and different personalities [11]. Nevertheless, the work on gender composition of pairs found conflicting results about whether same-gender or mixed-gender pairings are more effective [3, 4, 12, 16]. One possible explanation is that gender correlates with other dimensions that may affect the pairs’ collaboration, but these correlations may vary between different environments. For example, women in a class may, on average, have higher skill level than men because they had to face more societal barriers to enter the class. On the other hand, they may, on average, have lower skill level if women with no background are more actively recruited. Authors:
(1) Amador Durán, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (amador@us.es);
(2) Pablo Fernández, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (pablofm@us.es);
(3) Beatriz Bernárdez, I3US Institute, Universidad de Sevilla, Sevilla, Spain (beat@us.es);
(4) Nathaniel Weinman, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (nweinman@berkeley.edu);
(5) Aslı Akalın, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (asliakalin@berkeley.edu);
(6) Armando Fox, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (fox@berkeley.edu). This paper is available on arxiv under CC BY 4.0 DEED license. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 1.1 The twincode platform 1.1 The twincode platform 1.2 Related Work 1.2 Related Work 2 Research Questions 2 Research Questions 3 Variables 3 Variables 3.1 Independent Variables 3.1 Independent Variables 3.2 Dependent Variables 3.2 Dependent Variables 3.3 Confounding Variables 3.3 Confounding Variables 4 Participants 4 Participants 5 Execution Plan and 5.1 Recruitment 5 Execution Plan and 5.1 Recruitment 5.2 Training and 5.3 Experiment Execution 5.2 Training and 5.3 Experiment Execution 5.4 Data Analysis 5.4 Data Analysis Acknowledgments and References Acknowledgments and References 1.2 Related Work Several literature reviews [10, 15, 21] have compiled the empirical research on using pair programming in higher education, with [6] being focused on distributed pair programming from a teaching perspective. By means of controlled experiments, remote and co–located pair programming are compared in [1, 22], showing comparable results. In most of the cases, the analyzed variables are related to performance in terms of time, quality, or code tests passed. Students perceptions have been also analyzed in terms of confidence, satisfaction, motivation, or personality [20]. Table 1 summarizes the empirical studies on the influence of gender in pair programming, including findings such as (i) same– gender pairs are more “democratic”; (ii) women working in pairs were more confident than those working solo; and (iii) in mixedgender, pairing women particularly do not benefit [15]. Although such studies reveal that gender seems to be a key factor in pair programming, none of them study gender bias in pair programming. Many factors other than gender may affect the outcomes of remote programming sessions [2, 23]. Previous research on productive pairing looked at factors such as skill levels, autonomy in choosing one’s partner [25], and different personalities [11]. Nevertheless, the work on gender composition of pairs found conflicting results about whether same-gender or mixed-gender pairings are more effective [3, 4, 12, 16]. One possible explanation is that gender correlates with other dimensions that may affect the pairs’ collaboration, but these correlations may vary between different environments. For example, women in a class may, on average, have higher skill level than men because they had to face more societal barriers to enter the class. On the other hand, they may, on average, have lower skill level if women with no background are more actively recruited. Authors: (1) Amador Durán, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (amador@us.es); (2) Pablo Fernández, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (pablofm@us.es); (3) Beatriz Bernárdez, I3US Institute, Universidad de Sevilla, Sevilla, Spain (beat@us.es); (4) Nathaniel Weinman, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (nweinman@berkeley.edu); (5) Aslı Akalın, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (asliakalin@berkeley.edu); (6) Armando Fox, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (fox@berkeley.edu). Authors: Authors: (1) Amador Durán, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (amador@us.es); (2) Pablo Fernández, SCORE Lab, I3US Institute, Universidad de Sevilla, Sevilla, Spain (pablofm@us.es); (3) Beatriz Bernárdez, I3US Institute, Universidad de Sevilla, Sevilla, Spain (beat@us.es); (4) Nathaniel Weinman, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (nweinman@berkeley.edu); (5) Aslı Akalın, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (asliakalin@berkeley.edu); (6) Armando Fox, Computer Science Division, University of California, Berkeley, Berkeley, CA, USA (fox@berkeley.edu). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Examining Gender Bias in Pair Programming: Insights from Empirical Studies

Authors:
(1) Abraham Owodunni, Intron Health, Masakhane, and this author contributed equally;
(2) Aditya Yadavalli, Karya, Masakhane, and this author contributed equally;
(3) Chris Emezuem, Mila Quebec AI Institute, Lanfrica, Masakhane, and this author contributed equally;
(4) Tobi Olatunji, Intron Health and Masakhane, and this author contributed equally;
(5) Clinton Mbataku, AI Saturdays Lagos. Table of Links Abstract and 1 Introduction 2 Related Work 3 AccentFold 4 What information does AccentFold capture? 5 Empirical study of AccentFold 6 Conclusion, Limitations, and References 5 Empirical study of AccentFold 5.1 Problem Formulation In this empirical study, we set out to understand how informative the accent folds are for accentlevel zero shot ASR performance. To achieve this, we designed our experimental task as follows: Assume we have the below oracle data set generator: 5.2 Experimental Setup Fine-tuning Details: We use a pre-trained XLSR model (Conneau et al., 2020) for our experiments. The XLSR model extends the wav2vec 2.0 (Baevski et al., 2020) model to the cross-lingual setting and was trained to acquire cross-lingual speech representations through the utilization of a singular model that is pre-trained using raw speech waveforms from various languages. The fact that this model is cross-lingual makes it a good fit for our experiments. During the fine-tuning of our pre-trained model, we follow the hyperparameter settings of Olatunji et al. (2023a). These include setting the dropout rates for attention and hidden layers to 0.1, while keeping the feature projection dropout at 0.0. We also employ a mask probability of 0.05 and a layerdrop rate of 0.1. Additionally, we enable gradient checkpointing to reduce memory usage. The learning rate is set to 3e-4, with a warm-up period of 1541 steps. The batch sizes for training and validation are 16 and 8, respectively, and we train the model for ten epochs. For each of the 41 target accents, we finetune our pre-trained model on its corresponding dataset and evaluate the word error rate on the test set comprising audio samples containing only the target accent. We run all our experiments using a 40GB NVIDIA A100 SXM GPU, which enables parallel use of its GPU nodes. Evaluation procedure: It is important to note that although the training dataset size Nk depends on the target accent aOOD in consideration, the test set used to evaluate all our experiments is fixed: it comprises the samples from the test split of the Afrispeech-200. Using Figure 1 the test set are samples from all the 108 accents of the test split. By keeping the test set constant, we can assess the model’s performance on our intended accent aOOD in an out-of-distribution (OOD) scenario. This is because the training and development splits do not include any audio-speech samples from these accents. Additionally, this procedure enables us to evaluate the model’s capacity to generalize to other accent samples, resulting in a highly resilient evaluation. 5.3 Results and Discussion Table 2 presents the results of a test Word Error Rate (WER) comparison between three different approaches for subset selection: AccentFold, GeoProx, and random sampling. The table displays the average and standard deviation of the WER values over the 41 target OOD accents. The results show that the AccentFold approach achieves the lowest test WER of 0.332 with a standard deviation of 0.013. In contrast, the random sampling approach yields the highest test WER of 0.367 with a larger standard deviation of 0.034. GeoProx, which uses real-world geographical proximity of the accents, performs better than random sampling but still under-performs when compared to AccentFold. To better understand this, we investigate the accents selected by AccentFold and GeoProx and analyse their non-overlapping accents in Figure 6. The histogram reveals that many of the accents selected by AccentFold for any given target OOD accent, aOOD, are not necessarily those geographically closest to aOOD. This insight suggests that the learned embeddings in AccentFold encompass much more than geographical proximity of accents. Figure 5 visualizes the test WER obtained by AccentFold and random sampling for each of the 41 accents. We see that in majority of the accents, AccentFold leads to improved performancte in terms of WER compared to random sampling. These findings indicate that AccentFold effectively captures linguistic relationships among accents, allowing for more accurate recognition of the target OOD accent when used to build the fine-tuning dataset. This demonstrates the usefulness of leveraging linguistic information and accent embeddings provided by AccentFold in the context of automatic speech recognition tasks. We notice a pattern, as shown in Figure 7, where increasing the value of s, which corresponds to a larger training dataset size Nk, results in minimal variation in the selection of accent subsets. This convergence of test WER implies that as the sample size increases, the specific choice of accent subsets becomes less influential in determining the performance. This paper is available on arxiv under CC BY-SA 4.0 DEED license. Authors: (1) Abraham Owodunni, Intron Health, Masakhane, and this author contributed equally; (2) Aditya Yadavalli, Karya, Masakhane, and this author contributed equally; (3) Chris Emezuem, Mila Quebec AI Institute, Lanfrica, Masakhane, and this author contributed equally; (4) Tobi Olatunji, Intron Health and Masakhane, and this author contributed equally; (5) Clinton Mbataku, AI Saturdays Lagos. Authors: Authors: (1) Abraham Owodunni, Intron Health, Masakhane, and this author contributed equally; (2) Aditya Yadavalli, Karya, Masakhane, and this author contributed equally; (3) Chris Emezuem, Mila Quebec AI Institute, Lanfrica, Masakhane, and this author contributed equally; (4) Tobi Olatunji, Intron Health and Masakhane, and this author contributed equally; (5) Clinton Mbataku, AI Saturdays Lagos. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 AccentFold 3 AccentFold 4 What information does AccentFold capture? 4 What information does AccentFold capture? 5 Empirical study of AccentFold 5 Empirical study of AccentFold 6 Conclusion, Limitations, and References 6 Conclusion, Limitations, and References 5 Empirical study of AccentFold 5.1 Problem Formulation In this empirical study, we set out to understand how informative the accent folds are for accentlevel zero shot ASR performance. To achieve this, we designed our experimental task as follows: Assume we have the below oracle data set generator: 5.2 Experimental Setup Fine-tuning Details : We use a pre-trained XLSR model (Conneau et al., 2020) for our experiments. The XLSR model extends the wav2vec 2.0 (Baevski et al., 2020) model to the cross-lingual setting and was trained to acquire cross-lingual speech representations through the utilization of a singular model that is pre-trained using raw speech waveforms from various languages. The fact that this model is cross-lingual makes it a good fit for our experiments. Fine-tuning Details During the fine-tuning of our pre-trained model, we follow the hyperparameter settings of Olatunji et al. (2023a). These include setting the dropout rates for attention and hidden layers to 0.1, while keeping the feature projection dropout at 0.0. We also employ a mask probability of 0.05 and a layerdrop rate of 0.1. Additionally, we enable gradient checkpointing to reduce memory usage. The learning rate is set to 3e-4, with a warm-up period of 1541 steps. The batch sizes for training and validation are 16 and 8, respectively, and we train the model for ten epochs. For each of the 41 target accents, we finetune our pre-trained model on its corresponding dataset and evaluate the word error rate on the test set comprising audio samples containing only the target accent. We run all our experiments using a 40GB NVIDIA A100 SXM GPU, which enables parallel use of its GPU nodes. Evaluation procedure: It is important to note that although the training dataset size Nk depends on the target accent aOOD in consideration, the test set used to evaluate all our experiments is fixed: it comprises the samples from the test split of the Afrispeech-200. Using Figure 1 the test set are samples from all the 108 accents of the test split. By keeping the test set constant, we can assess the model’s performance on our intended accent aOOD in an out-of-distribution (OOD) scenario. This is because the training and development splits do not include any audio-speech samples from these accents. Additionally, this procedure enables us to evaluate the model’s capacity to generalize to other accent samples, resulting in a highly resilient evaluation. 5.3 Results and Discussion Table 2 presents the results of a test Word Error Rate (WER) comparison between three different approaches for subset selection: AccentFold, GeoProx, and random sampling. The table displays the average and standard deviation of the WER values over the 41 target OOD accents. The results show that the AccentFold approach achieves the lowest test WER of 0.332 with a standard deviation of 0.013. In contrast, the random sampling approach yields the highest test WER of 0.367 with a larger standard deviation of 0.034. GeoProx, which uses real-world geographical proximity of the accents, performs better than random sampling but still under-performs when compared to AccentFold. To better understand this, we investigate the accents selected by AccentFold and GeoProx and analyse their non-overlapping accents in Figure 6. The histogram reveals that many of the accents selected by AccentFold for any given target OOD accent, aOOD, are not necessarily those geographically closest to aOOD. This insight suggests that the learned embeddings in AccentFold encompass much more than geographical proximity of accents. Figure 5 visualizes the test WER obtained by AccentFold and random sampling for each of the 41 accents. We see that in majority of the accents, AccentFold leads to improved performancte in terms of WER compared to random sampling. These findings indicate that AccentFold effectively captures linguistic relationships among accents, allowing for more accurate recognition of the target OOD accent when used to build the fine-tuning dataset. This demonstrates the usefulness of leveraging linguistic information and accent embeddings provided by AccentFold in the context of automatic speech recognition tasks. We notice a pattern, as shown in Figure 7, where increasing the value of s, which corresponds to a larger training dataset size Nk, results in minimal variation in the selection of accent subsets. This convergence of test WER implies that as the sample size increases, the specific choice of accent subsets becomes less influential in determining the performance. This paper is available on arxiv under CC BY-SA 4.0 DEED license. This paper is available on arxiv under CC BY-SA 4.0 DEED license. available on arxiv

AccentFold: Enhancing Accent Recognition - Empirical Study of AccentFold

Author:
(1) Yigit Ege Bayiz, Electrical and Computer Engineering The University of Texas at Austin Austin, Texas, USA (Email: egebayiz@utexas.edu);
(2) Ufuk Topcu, Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, Texas, USA (Email: utopcu@utexas.edu). Table of Links Abstract and Introduction Related Works Preliminaries Optimal Prebunking Problem Deterministic Baseline Policies Temporally Equidistant Prebunking Numerical Results Conclusion and References VIII. CONCLUSION We define the problem of optimally delivering prebunks to a user as a minimax optimization problem, and under SI propagation assumptions propose algorithms that guarantee feasibility. We demonstrate that our theoretically backed approach Algorithm 3 also yields better results than the other two baselines in empirical analysis using simulated misinformation propagations on Chung-Lu models. Algorithm 3 is also often computationally feasible to solve, as at each time, it relies on solving a linear program that is computationally inexpensive. Our results, however, suffer from the limitations we impose on the network structure. Real-world misinformation propagation often deviates significantly from the SI model predictions. Our models also only provide feasibility guarantees under the discrete-time epidemic propagation assumptions. We also focus solely on delivering optimal prebunks to each user one by one, which is different from optimizing prebunk deliveries on the entire network. For future work, we plan to extend our problem and results to optimizing misinformation deliveries across the entire social network. REFERENCES [1] Social media fact sheet, Apr. 2021. [Online]. Available: https://www.pewresearch.org/internet/fact-sheet/socialmedia/. [2] B. Swire-Thompson, D. Lazer, et al., “Public health and online misinformation: Challenges and recommendations,” Annu Rev Public Health, vol. 41, no. 1, pp. 433–451, 2020. [3] Y. Benkler, R. Faris, and H. Roberts, Network propaganda: Manipulation, disinformation, and radicalization in American politics. Oxford University Press, 2018. [4] S. van der Linden, A. Leiserowitz, S. Rosenthal, and E. Maibach, “Inoculating the public against misinformation about climate change,” Global Challenges, vol. 1, no. 2, 2017. DOI: https : / / doi . org / 10 . 1002 / gch2 . 201600008. [5] S. Lewandowsky, U. K. H. Ecker, and J. Cook, “Beyond misinformation: Understanding and coping with the “post-truth” era.,” Journal of Applied Research in Memory and Cognition, vol. 6, pp. 353–369, 4 2017. DOI: 10.1016/j.jarmac.2017.07.008. [6] G. Pennycook and D. G. Rand, “The psychology of fake news,” Trends in Cognitive Sciences, vol. 25, no. 5, pp. 388–402, 2021. DOI: https://doi.org/10.1016/j.tics. 2021.02.007. [7] F. Liu and M. Buss, “Optimal control for heterogeneous node-based information epidemics over social networks,” IEEE Transactions on Control of Network Systems, vol. 7, no. 3, pp. 1115–1126, 2020. DOI: 10. 1109/TCNS.2019.2963488. [8] Y. E. Bayiz and U. Topcu, Countering misinformation on social networks using graph alterations, 2022. arXiv: 2211.04617 [cs.SI]. [9] Z. Guo, M. Schlichtkrull, and A. Vlachos, “A Survey on Automated Fact-Checking,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 178– 206, Feb. 2022. DOI: 10.1162/tacl a 00454. [10] I. Augenstein, C. Lioma, D. Wang, et al., “MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims,” Association for Computational Linguistics, 2019. DOI: 10.18653/v1/D19-1475. [11] U. K. H. Ecker, S. Lewandowsky, J. Cook, et al., “The psychological drivers of misinformation belief and its resistance to correction,” Nature Reviews Psychology, vol. 1, no. 1, pp. 13–29, Jan. 2022. DOI: 10 . 1038 / s44159-021-00006-y. [12] I. Chaturvedi, E. Cambria, R. E. Welsch, and F. Herrera, “Distinguishing between facts and opinions for sentiment analysis: Survey and challenges,” Information Fusion, vol. 44, pp. 65–77, 2018. DOI: https://doi.org/ 10.1016/j.inffus.2017.12.006. [13] M. Mohtarami, R. Baly, J. Glass, P. Nakov, L. Marquez, ` and A. Moschitti, “Automatic stance detection using end-to-end memory networks,” Association for Computational Linguistics, Jun. 2018. DOI: 10.18653/v1/N18- 1070. [14] M. Del Vicario, A. Bessi, F. Zollo, et al., “The spreading of misinformation online,” Proceedings of the National Academy of Sciences, vol. 113, no. 3, pp. 554–559, 2016. DOI: 10.1073/pnas.1517441113. [15] S. Shaar, N. Babulkov, G. Da San Martino, and P. Nakov, “That is a known lie: Detecting previously fact-checked claims,” Association for Computational Linguistics, 2020, pp. 3607–3618. DOI: 10.18653/v1/ 2020.acl-main.332. [16] A. L. Hill, D. G. Rand, M. A. Nowak, and N. A. Christakis, “Infectious disease modeling of social contagion in networks,” PLOS computational biology, vol. 6, no. 11, 2010. [17] S. Raponi, Z. Khalifa, G. Oligeri, and R. Di Pietro, “Fake news propagation: A review of epidemic models, datasets, and insights,” ACM Trans. Web, vol. 16, no. 3, Sep. 2022, ISSN: 1559-1131. DOI: 10.1145/3522756. [Online]. Available: https://doi.org/10.1145/3522756. [18] S. Krishnasamy, S. Banerjee, and S. Shakkottai, “The behavior of epidemics under bounded susceptibility,” SIGMETRICS Perform. Eval. Rev., 2014. DOI: 10.1145/ 2637364.2591977. [19] L. Zhao, H. Cui, X. Qiu, X. Wang, and J. Wang, “Sir rumor spreading model in the new media age,” Physica A: Statistical Mechanics and its Applications, vol. 392, no. 4, pp. 995–1003, 2013. [20] Y.-Q. Wang and J. Wang, “Sir rumor spreading model considering the effect of difference in nodes’ identification capabilities,” International Journal of Modern Physics, vol. 28, no. 05, 2017. [21] M. Kimura, K. Saito, and H. Motoda, “Efficient estimation of influence functions for sis model on social networks,” in International Joint Conference on Artificial Intelligence, 2009. [22] F. Jin, E. Dougherty, P. Saraf, Y. Cao, and N. Ramakrishnan, “Epidemiological modeling of news and rumors on twitter,” in Proceedings of Workshop on Social Network Mining and Analysis, 2013. [23] F. Chung and L. Lu, “Connected components in random graphs with given expected degree sequences,” Annals of Combinatorics, vol. 6, no. 2, pp. 125–145, 2002, ISSN: 0219-3094. DOI: 10.1007/PL00012580. [24] D. Fasino, A. Tonetto, and F. Tudisco, “Generating large scale-free networks with the chung–lu random graph model,” Networks, vol. 78, no. 2, pp. 174–187, 2021. [25] K. Pogorelov, D. T. Schroeder, P. Filkukova, S. Brenner, ´ and J. Langguth, “Wico text: A labeled dataset of conspiracy theory and 5g-corona misinformation tweets,” in Workshop on Open Challenges in Online Social Networks. Association for Computing Machinery, 2021. [26] J.-L. Guillaume, M. Latapy, and C. Magnien, “Comparison of failures and attacks on random and scale-free networks,” in OPODIS 2004, Springer, 2005. [27] P. Crucitti, V. Latora, M. Marchiori, and A. Rapisarda, “Efficiency of scale-free networks: Error and attack tolerance,” Physica A: Statistical Mechanics and its Applications, vol. 320, pp. 622–642, 2003. DOI: https: //doi.org/10.1016/S0378-4371(02)01545-5. This paper is available on arxiv under CC 4.0 license. Author: (1) Yigit Ege Bayiz, Electrical and Computer Engineering The University of Texas at Austin Austin, Texas, USA (Email: egebayiz@utexas.edu); (2) Ufuk Topcu, Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, Texas, USA (Email: utopcu@utexas.edu). Author: Author: (1) Yigit Ege Bayiz, Electrical and Computer Engineering The University of Texas at Austin Austin, Texas, USA (Email: egebayiz@utexas.edu); (2) Ufuk Topcu, Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, Texas, USA (Email: utopcu@utexas.edu). Table of Links Abstract and Introduction Abstract and Introduction Related Works Related Works Preliminaries Preliminaries Optimal Prebunking Problem Optimal Prebunking Problem Deterministic Baseline Policies Deterministic Baseline Policies Temporally Equidistant Prebunking Temporally Equidistant Prebunking Numerical Results Numerical Results Conclusion and References Conclusion and References VIII. CONCLUSION We define the problem of optimally delivering prebunks to a user as a minimax optimization problem, and under SI propagation assumptions propose algorithms that guarantee feasibility. We demonstrate that our theoretically backed approach Algorithm 3 also yields better results than the other two baselines in empirical analysis using simulated misinformation propagations on Chung-Lu models. Algorithm 3 is also often computationally feasible to solve, as at each time, it relies on solving a linear program that is computationally inexpensive. Our results, however, suffer from the limitations we impose on the network structure. Real-world misinformation propagation often deviates significantly from the SI model predictions. Our models also only provide feasibility guarantees under the discrete-time epidemic propagation assumptions. We also focus solely on delivering optimal prebunks to each user one by one, which is different from optimizing prebunk deliveries on the entire network. For future work, we plan to extend our problem and results to optimizing misinformation deliveries across the entire social network. REFERENCES [1] Social media fact sheet, Apr. 2021. [Online]. Available: https://www.pewresearch.org/internet/fact-sheet/socialmedia/. [2] B. Swire-Thompson, D. Lazer, et al., “Public health and online misinformation: Challenges and recommendations,” Annu Rev Public Health, vol. 41, no. 1, pp. 433–451, 2020. [3] Y. Benkler, R. Faris, and H. Roberts, Network propaganda: Manipulation, disinformation, and radicalization in American politics. Oxford University Press, 2018. [4] S. van der Linden, A. Leiserowitz, S. Rosenthal, and E. Maibach, “Inoculating the public against misinformation about climate change,” Global Challenges, vol. 1, no. 2, 2017. DOI: https : / / doi . org / 10 . 1002 / gch2 . 201600008. [5] S. Lewandowsky, U. K. H. Ecker, and J. Cook, “Beyond misinformation: Understanding and coping with the “post-truth” era.,” Journal of Applied Research in Memory and Cognition, vol. 6, pp. 353–369, 4 2017. DOI: 10.1016/j.jarmac.2017.07.008. [6] G. Pennycook and D. G. Rand, “The psychology of fake news,” Trends in Cognitive Sciences, vol. 25, no. 5, pp. 388–402, 2021. DOI: https://doi.org/10.1016/j.tics. 2021.02.007. [7] F. Liu and M. Buss, “Optimal control for heterogeneous node-based information epidemics over social networks,” IEEE Transactions on Control of Network Systems, vol. 7, no. 3, pp. 1115–1126, 2020. DOI: 10. 1109/TCNS.2019.2963488. [8] Y. E. Bayiz and U. Topcu, Countering misinformation on social networks using graph alterations, 2022. arXiv: 2211.04617 [cs.SI]. [9] Z. Guo, M. Schlichtkrull, and A. Vlachos, “A Survey on Automated Fact-Checking,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 178– 206, Feb. 2022. DOI: 10.1162/tacl a 00454. [10] I. Augenstein, C. Lioma, D. Wang, et al., “MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims,” Association for Computational Linguistics, 2019. DOI: 10.18653/v1/D19-1475. [11] U. K. H. Ecker, S. Lewandowsky, J. Cook, et al., “The psychological drivers of misinformation belief and its resistance to correction,” Nature Reviews Psychology, vol. 1, no. 1, pp. 13–29, Jan. 2022. DOI: 10 . 1038 / s44159-021-00006-y. [12] I. Chaturvedi, E. Cambria, R. E. Welsch, and F. Herrera, “Distinguishing between facts and opinions for sentiment analysis: Survey and challenges,” Information Fusion, vol. 44, pp. 65–77, 2018. DOI: https://doi.org/ 10.1016/j.inffus.2017.12.006. [13] M. Mohtarami, R. Baly, J. Glass, P. Nakov, L. Marquez, ` and A. Moschitti, “Automatic stance detection using end-to-end memory networks,” Association for Computational Linguistics, Jun. 2018. DOI: 10.18653/v1/N18- 1070. [14] M. Del Vicario, A. Bessi, F. Zollo, et al., “The spreading of misinformation online,” Proceedings of the National Academy of Sciences, vol. 113, no. 3, pp. 554–559, 2016. DOI: 10.1073/pnas.1517441113. [15] S. Shaar, N. Babulkov, G. Da San Martino, and P. Nakov, “That is a known lie: Detecting previously fact-checked claims,” Association for Computational Linguistics, 2020, pp. 3607–3618. DOI: 10.18653/v1/ 2020.acl-main.332. [16] A. L. Hill, D. G. Rand, M. A. Nowak, and N. A. Christakis, “Infectious disease modeling of social contagion in networks,” PLOS computational biology, vol. 6, no. 11, 2010. [17] S. Raponi, Z. Khalifa, G. Oligeri, and R. Di Pietro, “Fake news propagation: A review of epidemic models, datasets, and insights,” ACM Trans. Web, vol. 16, no. 3, Sep. 2022, ISSN: 1559-1131. DOI: 10.1145/3522756. [Online]. Available: https://doi.org/10.1145/3522756. [18] S. Krishnasamy, S. Banerjee, and S. Shakkottai, “The behavior of epidemics under bounded susceptibility,” SIGMETRICS Perform. Eval. Rev., 2014. DOI: 10.1145/ 2637364.2591977. [19] L. Zhao, H. Cui, X. Qiu, X. Wang, and J. Wang, “Sir rumor spreading model in the new media age,” Physica A: Statistical Mechanics and its Applications, vol. 392, no. 4, pp. 995–1003, 2013. [20] Y.-Q. Wang and J. Wang, “Sir rumor spreading model considering the effect of difference in nodes’ identification capabilities,” International Journal of Modern Physics, vol. 28, no. 05, 2017. [21] M. Kimura, K. Saito, and H. Motoda, “Efficient estimation of influence functions for sis model on social networks,” in International Joint Conference on Artificial Intelligence, 2009. [22] F. Jin, E. Dougherty, P. Saraf, Y. Cao, and N. Ramakrishnan, “Epidemiological modeling of news and rumors on twitter,” in Proceedings of Workshop on Social Network Mining and Analysis, 2013. [23] F. Chung and L. Lu, “Connected components in random graphs with given expected degree sequences,” Annals of Combinatorics, vol. 6, no. 2, pp. 125–145, 2002, ISSN: 0219-3094. DOI: 10.1007/PL00012580. [24] D. Fasino, A. Tonetto, and F. Tudisco, “Generating large scale-free networks with the chung–lu random graph model,” Networks, vol. 78, no. 2, pp. 174–187, 2021. [25] K. Pogorelov, D. T. Schroeder, P. Filkukova, S. Brenner, ´ and J. Langguth, “Wico text: A labeled dataset of conspiracy theory and 5g-corona misinformation tweets,” in Workshop on Open Challenges in Online Social Networks. Association for Computing Machinery, 2021. [26] J.-L. Guillaume, M. Latapy, and C. Magnien, “Comparison of failures and attacks on random and scale-free networks,” in OPODIS 2004, Springer, 2005. [27] P. Crucitti, V. Latora, M. Marchiori, and A. Rapisarda, “Efficiency of scale-free networks: Error and attack tolerance,” Physica A: Statistical Mechanics and its Applications, vol. 320, pp. 622–642, 2003. DOI: https: //doi.org/10.1016/S0378-4371(02)01545-5. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Empirical Success in Prebunking

Authors:
(1) Jonathan H. Rystrøm. Table of Links Abstract and Introduction Previous Literature Methods and Data Results Discussions Conclusions and References A. Validation of Assumptions B. Other Models C. Pre-processing steps 2 Previous Literature Several papers have highlighted the need to protect human autonomy and preferences in recommender systems (Calvo et al., 2020; Varshney, 2020; Jannach & Adomavicius, 2016). However, these focus more on the normative need to do so, rather than an empirical analysis of the phenomenon. With this paper, we aim to fill this gap in the literature. Most previous empirical analysis of preference manipulation in recommender systems has focused on the MovieLensdataset (Harper & Konstan, 2016). MovieLens is a movie recommendation platform developed and maintained by the University of Minnesota.[1] The advantages of this dataset are that it is a) rigorously documented as it is maintained by an academic group; b) freely available; and c) well-structured, thus making analysis easier. However, because it is a non-commercial project it is not susceptible to surveillance capitalistic imperatives to the same extent as e.g. Amazon (Zuboff, 2019). Nguyen et al. (2014) analysed whether MovieLens users were exposed to less diverse content over time - a type of preference manipulation. While they found a significant (albeit small) decrease, the effect was smaller for users who followed the recommendations than the users who did not. However, as discussed by Rakova and Chowdhury (2019), they focus their analysis on highly-active and highly-nonactive users neglecting the middle. Also, content diversity is an important but incomplete measure of preference manipulation. Rakova and Chowdhury (2019) also use the MovieLens dataset to define and showcase Barrier-to-Exit. However, their paper is more of a proof-of-concept rather than an analysis. The present paper expands on Rakova and Chowdhury (2019) by applying the metric to a real-world dataset of a commercial recommender system. [1] movielens.org This paper is available on arxiv under CC 4.0 license. Authors: (1) Jonathan H. Rystrøm. Authors: Authors: (1) Jonathan H. Rystrøm. Table of Links Abstract and Introduction Abstract and Introduction Previous Literature Previous Literature Methods and Data Methods and Data Results Results Discussions Discussions Conclusions and References Conclusions and References A. Validation of Assumptions A. Validation of Assumptions B. Other Models B. Other Models C. Pre-processing steps C. Pre-processing steps 2 Previous Literature Several papers have highlighted the need to protect human autonomy and preferences in recommender systems (Calvo et al., 2020; Varshney, 2020; Jannach & Adomavicius, 2016). However, these focus more on the normative need to do so, rather than an empirical analysis of the phenomenon. With this paper, we aim to fill this gap in the literature. Most previous empirical analysis of preference manipulation in recommender systems has focused on the MovieLensdataset (Harper & Konstan, 2016). MovieLens is a movie recommendation platform developed and maintained by the University of Minnesota.[1] The advantages of this dataset are that it is a) rigorously documented as it is maintained by an academic group; b) freely available; and c) well-structured, thus making analysis easier. However, because it is a non-commercial project it is not susceptible to surveillance capitalistic imperatives to the same extent as e.g. Amazon (Zuboff, 2019). Nguyen et al. (2014) analysed whether MovieLens users were exposed to less diverse content over time - a type of preference manipulation. While they found a significant (albeit small) decrease, the effect was smaller for users who followed the recommendations than the users who did not. However, as discussed by Rakova and Chowdhury (2019), they focus their analysis on highly-active and highly-nonactive users neglecting the middle. Also, content diversity is an important but incomplete measure of preference manipulation. Rakova and Chowdhury (2019) also use the MovieLens dataset to define and showcase Barrier-to-Exit. However, their paper is more of a proof-of-concept rather than an analysis. The present paper expands on Rakova and Chowdhury (2019) by applying the metric to a real-world dataset of a commercial recommender system. [1] movielens.org This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Empirical Insights into Preference Manipulation

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Oded Blumenthal, Software and Information Systems Engineering, Ben Gurion University, Israel;
(2) Guy Shani, Software and Information Systems Engineering, Ben Gurion University, Israel. Table of Links Introduction
Background
Related Work
POMCP for Stochastic Contingent Planning
Domain Independent Heuristics for POMCP
Empirical Evaluation
Conclusion and References 6 Empirical Evaluation We conduct an empirical study to evaluate our methods. Our methods are implemented in C#. 6.1 Benchmark Domains We extended the following contingent planning benchmarks to stochastic settings: Doors: In the door domain the agent must move in a grid to reach a target position. Odd levels in the grid are all open, while in even levels there are doors, and only one door is open. The agent can sense whether a door is open when it is at adjacent cells. The agent must identify the open doors and get to the target position. In the stochastic version the agent can open a closed door with some probability of success. The agent can hence either search for the already open door, or attempt to open a closed door. Blocks World: In the contingent blocks world problem, the agent does not know the structure of the initial block configuration, but it can sense whether one block is on top of another one, and whether a block is clear. In the stochastic version moving a block from one block to another has a 0.3 probability of success, while moving blocks to and from the table succeeds deterministicly. Hence, it is often preferable to use the table as an intermediate position. Unix: In this domain the agent must search for a file in a file system, and copy it to a destination folder. In the stochastic version there is a non-uniform distribution over the possible locations of the file. MedPks: The agent here needs to identify which illness a patient has and treat it. To do so, the agent tests for each illness independently, until the proper illness is found. The stochastic version here has non-uniform distribution over the possible illnesses as well. Localize: In this domain the agent must reach a goal position in a grid. The agent does not know where it initially is, and can only sense adjacent walls. In the stochastic version there several places in the grid where the agent may slip and stay in place. This makes the localization in the grid more difficult. Wumpus: In this challenging problem the agent must reach a target position in a grid infested by monsters called Wumpuses. Cells may be unsafe to travel as they may contain either a Wumpus or a pit. Wumpuses emit a stench, and pits emit a breeze, both of which can be sensed in adjacent cells. The agent must sense in multiple positions to identify the safe cells. The stochastic version here also has non-uniform distribution over the safe cells. 6.2 Results For each domain above we run 20 online episodes, and compute the success rate, the average run time for computing the next action, and the average cost to the goal. We did not use a timeout, but runs longer than 100 steps were considered to be stuck in a loop, and terminated. We begin by looking at the quality of the policy — the average cost to the goal. As can be seen, the random rollout policy is best only in the MedPks domain, and close to best in unix. This is not too surprising, because in these two domains the best strategy is very simple, and random strategies easily stumble upon the goal. In blocks all methods achieved similar performance, because the optimal strategy is very short, and rollouts are less important. This domain has many possible actions, and hence a huge branching factor, making it difficult to scale up using POMCP. On doors and localize, which require lengthier trajectories to reach the goal, but do not need long information-gathering efforts, the single state hadd strategy operates very well. However, on Wumpus, where long sequences of actions are needed for information gathering, the multiple-state heuristic works best. We expected the results to be that way, although we expected more significant difference between the "smart" heuristics and the random rollout. With respect to the required time to run the simulations for a single decision, the results are mixed. While obviously the random strategy requires no time to compute the next action during a rollout, it often results in lengthy rollouts, which reduce this effect. The single state heuristic is almost always faster than the multi-state heuristic, but not by much. This paper is available on arxiv under CC 4.0 license. Authors: (1) Oded Blumenthal, Software and Information Systems Engineering, Ben Gurion University, Israel; (2) Guy Shani, Software and Information Systems Engineering, Ben Gurion University, Israel. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Oded Blumenthal, Software and Information Systems Engineering, Ben Gurion University, Israel; (2) Guy Shani, Software and Information Systems Engineering, Ben Gurion University, Israel. Table of Links Introduction Background Related Work POMCP for Stochastic Contingent Planning Domain Independent Heuristics for POMCP Empirical Evaluation Conclusion and References Introduction Introduction Background Background Related Work Related Work POMCP for Stochastic Contingent Planning POMCP for Stochastic Contingent Planning Domain Independent Heuristics for POMCP Domain Independent Heuristics for POMCP Empirical Evaluation Empirical Evaluation Conclusion and References Conclusion and References 6 Empirical Evaluation We conduct an empirical study to evaluate our methods. Our methods are implemented in C#. 6.1 Benchmark Domains We extended the following contingent planning benchmarks to stochastic settings: Doors : In the door domain the agent must move in a grid to reach a target position. Odd levels in the grid are all open, while in even levels there are doors, and only one door is open. The agent can sense whether a door is open when it is at adjacent cells. The agent must identify the open doors and get to the target position. In the stochastic version the agent can open a closed door with some probability of success. The agent can hence either search for the already open door, or attempt to open a closed door. Doors Blocks World: In the contingent blocks world problem, the agent does not know the structure of the initial block configuration, but it can sense whether one block is on top of another one, and whether a block is clear. In the stochastic version moving a block from one block to another has a 0.3 probability of success, while moving blocks to and from the table succeeds deterministicly. Hence, it is often preferable to use the table as an intermediate position. Unix: In this domain the agent must search for a file in a file system, and copy it to a destination folder. In the stochastic version there is a non-uniform distribution over the possible locations of the file. MedPks : The agent here needs to identify which illness a patient has and treat it. To do so, the agent tests for each illness independently, until the proper illness is found. The stochastic version here has non-uniform distribution over the possible illnesses as well. MedPks Localize : In this domain the agent must reach a goal position in a grid. The agent does not know where it initially is, and can only sense adjacent walls. In the stochastic version there several places in the grid where the agent may slip and stay in place. This makes the localization in the grid more difficult. Localize Wumpus : In this challenging problem the agent must reach a target position in a grid infested by monsters called Wumpuses. Cells may be unsafe to travel as they may contain either a Wumpus or a pit. Wumpuses emit a stench, and pits emit a breeze, both of which can be sensed in adjacent cells. The agent must sense in multiple positions to identify the safe cells. The stochastic version here also has non-uniform distribution over the safe cells. Wumpus 6.2 Results For each domain above we run 20 online episodes, and compute the success rate, the average run time for computing the next action, and the average cost to the goal. We did not use a timeout, but runs longer than 100 steps were considered to be stuck in a loop, and terminated. We begin by looking at the quality of the policy — the average cost to the goal. As can be seen, the random rollout policy is best only in the MedPks domain, and close to best in unix. This is not too surprising, because in these two domains the best strategy is very simple, and random strategies easily stumble upon the goal. In blocks all methods achieved similar performance, because the optimal strategy is very short, and rollouts are less important. This domain has many possible actions, and hence a huge branching factor, making it difficult to scale up using POMCP. On doors and localize, which require lengthier trajectories to reach the goal, but do not need long information-gathering efforts, the single state h add strategy operates very well. However, on Wumpus, where long sequences of actions are needed for information gathering, the multiple-state heuristic works best. We expected the results to be that way, although we expected more significant difference between the "smart" heuristics and the random rollout. add With respect to the required time to run the simulations for a single decision, the results are mixed. While obviously the random strategy requires no time to compute the next action during a rollout, it often results in lengthy rollouts, which reduce this effect. The single state heuristic is almost always faster than the multi-state heuristic, but not by much.

Rollout Heuristics for Online Stochastic Contingent Planning: Empirical Evaluation

Authors: (1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China; (2) Peng Liang, School of Computer Science, Wuhan University, Wuhan, China; (3) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China; (4) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany; (4) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia; (4) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland. II. METHODOLOGY The goal of this study is to systematically identity the issues that developers encountered in the practical application of GitHub Copilot, as well as their underlying causes and potential solutions. We formulated three RQs to direct subsequent phases of the methodology as shown in Fig. 1. The RQs and their rationales are detailed in Section II-A. A. Research Questions Rationale: Copilot is a relatively new product, and little is known about the specific challenges and issues users face while using it in the software development practice. By identifying these issues, this study can help us gain a better understanding of the obstacles that arise when using AI code generation tools like Copilot in practical situations from the perspective of software developers. RQ1: What are the issues faced by users while using Copilot in the software development practice? Rationale: Understanding the underlying causes of the issues identified in RQ1 is essential for developing effective solutions to address the issues. By identifying these causes, the study can provide insights into how to improve the design and functionality of Copilot. RQ2: What are the underlying causes of these issues? Rationale: RQ3 aims to identify potential solutions to the issues identified in RQ1, which can ultimately enhance the usability of Copilot for users. By exploring the existing practices and methods utilized by developers to tackle the issues, the study can gain insights into potential solutions that enhance the functionality and usability of Copilot. RQ3: What are the potential solutions to address these issues? B. Data Collection We collected data from three sources: GitHub Issues[1] , GitHub Discussions[2] , and SO posts[3] . GitHub Issue is a commonly used feature on GitHub for tracking bugs, feature requests, and other issues related to software development projects, which allow us to capture the specific problems and difficulties that users have encountered when coding with Copilot. GitHub Discussion, on the other hand, is a newer feature for more open-ended discussions among project contributors and community members, which also offers a central hub for project-related discussions and knowledge sharing among community members. Discussion topics can cover a wide range from technical questions and proposals to broader topics related to Copilot. Stack Overflow is a popular technology community that provides a Q&A platform covering a wide range of programming, development, and technical questions, including Copilot-related questions. Considering that Copilot was announced and started its technical preview on June 29, 2021, we chose to collect the data that were created after that date. The data collection was conducted on June 18, 2023. To assist us in addressing RQ2 and RQ3, we chose to collect closed GitHub issues and answered GitHub discussions, which contain known causes and solutions. For SO posts, we found that the amount of Copilot-related posts was relatively low. Hence, we chose to retrieve all potentially relevant posts, including unanswered ones, to obtain a more comprehensive dataset. For GitHub Issues, we used “Copilot” as the keyword to search closed Copilot-related issues in the entire GitHub, and a total of 4,057 issues were retrieved. We also employed “Copilot” as the keyword to search at SO, resulting in 679 retrieved posts. Different from GitHub Issues and SO posts, GitHub Discussions related to certain product are grouped under a specific subcategory, and “Copilot” was included as a subcategory under the “Product” category. Given the high relevance of these discussions to Copilot, we collected all the discussions that were answered under the “Copilot” subcategory as part of the data source for our study. In total, 925 discussions were obtained. C. Data Filtering To ensure that the data can be used to answer our RQs, we set the filtering criterion: the issue, discussion, or post should contain specific information related to the use of Copilot. We conducted the data labelling on the collected data to filter out the data which cannot be used for this study by following the criterion. Pilot Data labelling: To minimize personal bias in the formal labelling process, the first and third authors conducted a pilot data labelling. For GitHub issues and discussions, we randomly selected 100 and 25 respectively, which constitute 2.5% of the total count. Due to the smaller quantity of SO posts, we randomly selected 35, which constitutes 5% of the total posts. Selecting a certain proportion of data from different platforms respectively is to verify whether the criteria of the two authors are consistent across various platforms. The consistency of the labelled issues was measured by the Cohen’s Kappa coefficient [14], resulting in values of 0.824, 0.834, and 0.806. The results indicate that a reasonable level of agreement between the two authors. For the labelling results with differences of opinion, the two authors will engage in discussions with the second author to reach a consensus. 2) Data labelling: The first and third authors then conducted the formal data labelling. During this process, we excluded a large number of data not related to our research. For instance, in some issues, Copilot refers to other meanings, such as the co-pilot of an aircraft. Additionally, Copilot might be mentioned in a straightforward manner without additional information, like someone said, “You can try using Copilot, which is amazing”. We also excluded such issues since they could not provide useful information about the use of Copilot. During the labelling process, any result on which the two authors disagreed was subject to discussion with the second author until an agreement was reached. Ultimately, the two authors collected 476 GitHub issues, 706 GitHub discussions, and 184 SO posts for data extraction. D. Data Extraction and Analysis Extract Data: To answer the three RQs mentioned in Section II-A, we established a set of data items for data extraction, as presented in Table I. Data items D1-D3 intend to extract the information of issues, underlying causes, and possible solutions from the filtered data, to answer RQ1-RQ3, respectively. The first author conducted a pilot data extraction with the third author on 20 randomly selected GitHub issues, 20 discussions, and 20 SO posts, and in case of any discrepancies, the second author was involved to reach a consensus. Based on the observation, we established the following standards for data extraction: (1) If the same issue was identified by multiple users, we recorded it only once. (2) If multiple problems were identified within the same issue, discussion, or posts, we recorded each one separately. (3) For an issue that has multiple causes, we only recorded the confirmed cause by the contributor of the issue or the Copilot team as the root cause. (4) For an issue that has multiple solutions suggested, we only recorded the solutions that were confirmed by the contributor of the issue or the Copilot team to actually solve the issue. The first and third authors conducted data extraction from the filtered issues, discussions, and posts based on the data items, and then discussed and reached a consensus with the second author on inconsistencies to ensure that the data extraction process adhered to the predetermined criteria. Each extracted data item was reviewed multiple times by the three authors to ensure accuracy. The final data extraction results were compiled and recorded in MS Excel [13]. 2) Analyze Data: To answer the three RQs in Section II, we conducted data analysis using the Constant Comparison method [12]. The specific steps are as follows: 1) The first author meticulously reviewed the data obtained during the data extraction phase and assigned codes to each data. These codes constituted descriptive summarizations of the data, aimed at capturing the underlying themes. For instance, the issue in Discussion #10598 was coded as “Stopped Giving Inline Suggestions”. 2) The first author compared different codes to identify patterns, commonalities, and distinctions among them. Through this iterative comparison process, similar codes coalesced into higher-level types and categories. For example, the coding of Discussion #10598, along with other akin codes, fell under the same type of FUNCTIONALITY USAGE ISSUE, which further belonged to the category of Usage Issue. Ultimately, the first author engaged in discussions with the second and third authors to achieve a consensus on the taxonomies of issues, causes, and solutions, which are presented in Section III. [1] https://docs.github.com/en/issues [2] https://github.com/orgs/community/discussions/categories/copilot [3] https://stackoverflow.com/ This paper is under CC 4.0 license. available on arxiv

GitHub Copilot in Practice: Empirical Insights into User Experiences and Practical Challenges

This paper is available on under CC 4.0 license. arxiv Authors: (1) Jakub DRÁPAL, Institute of State and Law of the Czech Academy of Sciences, Czechia, Institute of Criminal Law and Criminology, Leiden University, the Netherlands; (2) Hannes WESTERMANN, Cyberjustice Laboratory, Université de Montréal, Canada; (3) Jaromir SAVELKA, School of Computer Science, Carnegie Mellon University, USA. Table of Links Abstract & Introduction Related Work Dataset Proposed Framework Experimental Design Results and Discussion Conclusions, Future Work and References Abstract. Thematic analysis and other variants of inductive coding are widely used qualitative analytic methods within empirical legal studies (ELS). We propose a novel framework facilitating effective collaboration of a legal expert with a large language model (LLM) for generating initial codes (phase 2 of thematic analysis), searching for themes (phase 3), and classifying the data in terms of the themes (to kick-start phase 4). We employed the framework for an analysis of a dataset (n = 785) of facts descriptions from criminal court opinions regarding thefts. The goal of the analysis was to discover classes of typical thefts. Our results show that the LLM, namely OpenAI’s GPT-4, generated reasonable initial codes, and it was capable of improving the quality of the codes based on expert feedback. They also suggest that the model performed well in zero-shot classification of facts descriptions in terms of the themes. Finally, the themes autonomously discovered by the LLM appear to map fairly well to the themes arrived at by legal experts. These findings can be leveraged by legal researchers to guide their decisions in integrating LLMs into their thematic analyses, as well as other inductive coding projects. Thematic analysis, empirical legal studies, criminal law, large language models, generative pre-trained transformers, GPT-4 Keywords. 1. Introduction Empirical legal studies (ELS) is an approach to the study of law through empirical methods typical of economics, psychology, and sociology. Since law is a heavily text-based discipline ELS frequently focuses on text analytic methods, including deductive and inductive coding. Deductive coding focuses on applying a fixed set of codes to a dataset, whereas inductive coding leads to a simultaneous discovery of the codes from the data and their application. While investigations into various methods to support deductive coding have attracted much recent attention in AI & Law [1,2,3] very few studies focused on inductive coding [4]. One popular inductive coding method is “thematic analysis” [5]. J. Drápal, H. Westermann, and J. Savelka / Using LLMs to Support Thematic Analysis in ELS While rarely explicitly acknowledged thematic analysis is widely used in ELS. Figure 1 shows the six phases of the analysis, starting from the raw data and finishing with the scholarly report on the studied phenomena. We propose a novel LLM-powered framework to support a subject matter expert in performing phases 2 and 3 of the analysis. We employed the proposed framework in a thematic analysis of criminal courts’ opinions, focused on the criminal offense of theft in Czechia. Criminal offense categories (e.g., theft, murder) are usually defined in statutory law while the individual criminal acts are described by courts when they apply the law to factual circumstances of the cases. An important question in criminal law and criminology is what behaviors are actually criminalized and whether it is done appropriately. Neither the statutory definitions of the offenses (they are too general and not “sociologically relevant” [6]) nor the descriptions of the factual circumstances from cases (too specific) can answer the question. To get insight into what behavior is criminalized and how effectively, it is necessary to identify shared features of criminal acts, generalize them into “typical crimes” [7,8] and arriving at behavioral-based categories [9,10]. This is akin to performing thematic analysis. While important, such analysis is an expensive and time-consuming endeavour. Hence, a (semi-)automated approach would be useful. To assess the capability of a state-of-the-art LLM (GPT-4) to support selected stages of the thematic analysis, we investigated the following research questions: (RQ1) How successfully can the LLM perform initial coding of the data? (RQ2) To what degree can a subject matter expert improve the quality of the initial codes via natural language feedback? (RQ3) How successfully can the LLM predict themes for the analyzed data points? (RQ4) How successfully can the LLM autonomously discover themes and associate them with the analyzed data? This paper is available on under CC 4.0 license. arxiv

How to Use Large Language Models to Support Thematic Analysis in Empirical Legal Studies

Authors: (1) Davide Viviano, Department of Economics, Harvard University; (2) Lihua Lei, Graduate School of Business, Stanford University; (3) Guido Imbens, Graduate School of Business and Department of Economics, Stanford University; (4) Brian Karrer, FAIR, Meta; (5) Okke Schrijvers, Meta Central Applied Science; (6) Liang Shi, Meta Central Applied Science. Table of Links Abstract & Introduction Setup (When) should you cluster? Choosing the cluster design Empirical illustration and numerical studies Recommendations for practice References A) Notation B) Endogenous peer effects C) Proofs 5 Empirical illustration and numerical studies In this section we illustrate the properties of the procedure in two empirical applications, one using unique data from the Facebook friendship and messaging graphs, and one using data from an experiment conducted in rural China by Cai et al. [2015]. We provide further evidence supporting our theoretical analysis using simulated networks in Section 5.3. 5.1 Clustering on Facebook graphs We first study the procedure’s properties for two clustering algorithms implemented at scale on social networks owned by Meta: Louvain algorithm [Blondel et al., 2008], Balanced Partitioning [Kabiljo et al., 2017]. We consider two graphs owned by Meta. In each graph, edges are continuous variables. In the first graph, edges capture the strength of the friendship in the Facebook graph, and in the second graph they capture connections based on messaging on Facebook, in both cases the data was aggregated and de-identified. We compute our statistics by setting the edges to be zero if below the 5th, 10th, and 50th percentile. We will refer to these graphs obtained as dense, moderate (mod), and sparse after thresholding (graphs all have a bounded maximum degree). We report the bias and variance, with the variance weighted by a parameter ξn, defined respectively as Louvain and Balanced Partitioning clustering produce a hierarchical clustering structure with a growing number of clusters. We consider three “types” of such algorithms, corresponding to three levels in the clustering structure hierarchy (i.e., different numbers of clusters), defined as Type 1, 2, and 3, respectively. For Louvain clustering, higher-order types denote more clusters, whereas it is the opposite for Balanced Partitioning. For each clustering, we report log(Kn/n), log-ratio of the number of clusters over the population size. Table 2 collects each of these algorithms’ worst-case bias and variance and the number of clusters per individual. Louvain algorithm dominates Balanced Partitioning both in worstcase variance and bias. A larger number of clusters increases the bias but decreases the variance, which we observe throughout all graphs. The bias increases for sparser networks because the denominator |Ni | decreases. Figure 3 reports the objective function for different graphs, Louvain algorithms, and degrees of sparsity. For ξn ≈ 1, the Louvain algorithm with the largest number of clusters (Type 3) dominates the other two algorithms in all but one case. This result provides some interpretable comparisons for the number of clusters. We also observe a trade-off between the number of clusters and the degree of sparsity of the adjacency matrix as ξn decreases. Louvain clustering with the largest number of clusters (Type 1) is optimal for a dense graph with ξn ≫ 1. Such clustering, however, is sub-optimal as the graph becomes more sparse. This result is intuitive: for denser networks, a larger number of clusters may best control the estimator’s bias relative to the bias induced by other clustering algorithms. These comparisons motivate trade-offs in the choice of the graph (and its density) and the clustering algorithm. We recommend practitioners compare clustering by averaging objective functions over different sparsity thresholding. For example, it is possible to compare clustering based on the average objective across different graphs using some priors on the degree of sparsity without affecting our theoretical results. Finally, note that using Theorem 3.6 it is possible to compute the smallest values of the spillover effects that would motivate using a cluster instead of Bernoulli design. From Theorem 3.6, researchers should prefer a cluster over a Bernoulli design if In summary, our results shed light on using clustering algorithms for large-scale implementation on online platforms. These results suggest that Louvain clustering with possibly many clusters performs best in practice. It confirms intuitive arguments in Karrer et al. [2021] who also recommend Louvain clustering based on AA-tests. 5.2 Clustering in the field We consider as an application the problem of informing individuals to increase insurance takeup studied in Cai et al. [2015]. The authors collected network information in approximately 184 villages in rural China from 48 larger regions. We use network data collected by Cai et al. [2015] to study the properties of the proposed method, where we assume that two individuals are connected if at least one of the two indicates the other as a friend. Because we do not require information from end-line outcomes, we construct a network using information from surveyed individuals as well as their friends. The network has in total 7649 nodes, once we also include individuals who are friends of surveyed individuals (but do not necessarily live in their same village or were surveyed in the experiment). Individuals between villages present on average 50% of their connections outside their village. On the other hand, individuals in different regions (with approximately 50 regions) have 99% of their connections within the same region. We use an adjacency matrix a “weak ties” adjacency matrix, where two individuals are connected if either indicates the other has a friend. We study clustering within each region and report results for different clustering algorithms. Clustering within each region is performed as in Algorithm 1, where we first estimate Xˆ (K) via semi-definite programming, use K-means algorithm to retrieve the clusters from Xˆ (K) and iterate to estimate the optimal number of clusters K. We report the frontier with respect to the absolute bias consistently with Algorithm 1. 5.2.1 How many clusters should we choose? Figure 4 presents the main results. The left-hand side panel reports the average number of estimated clusters, estimated by the proposed method divided by the population size in a given cluster. Different choices of ξn justify a different number of clusters. For ξn = 1, the number of clusters is 7%, the total population size, whereas for ξn = 15, the number of clusters is roughly half the population size. To gain further intuition of its practical implication, take ψ¯ = 0.24 approximately equal to the outcomes’ variance in Cai et al. [2015] and ϕ¯ n = 0.27 as in Table 2 in Cai et al. [2015]. A recommended choice of ξn is approximately ψ/¯ ϕ¯2 n = 3.29. This choice puts slightly more weight on the bias than the variance when λ = 1, and therefore can be considered a conservative choice for the number of clusters.[13] Even with such a conservative choice, the suggested number of clusters is large relative to the population size. In this case, the optimal number of clusters is around 15% of the overall population size, approximately 600 clusters with a surveyed sample size of 3600 individuals. This number is consistent (in scale) with the choice in the number of clusters in other applications in development, such as Egger et al. [2022], Alatas et al. [2012], but larger than the number of villages that we have in the application in Cai et al. [2015]. 5.2.2 Comparison with other clustering methods Next, we compare the procedure (denoted as “Causal Clustering”) with the following alternative clustering algorithms: ε-net clustering as in Eckles et al. [2017] with ε = 3 as suggested in Eckles et al. [2017]; spectral clustering with a fixed number of clusters equal to n/3, where n is the population size in a given region (note that spectral clustering do not optimize over the number of clusters); Louvain clustering with default parameters selected by the R-package igraph; clustering based on village identity of each individual (creating one additional cluster for those individuals whose village is missing in the data). We contrast the objective function with alternative algorithms in the right-hand side of Figure 4. The proposed method achieves the smallest objective function across all competitors, providing suggestive evidence that Algorithm 1 (with a semi-definite relaxation) works particularly well in practice. For a smaller number of clusters the method is closer to the Louvain clustering algorithm, while for a larger number of clusters the objective of the algorithm is close to the objective of the spectral algorithm. Interestingly, clustering based on village identity leads to the largest loss function. This is because individuals are connected within and between villages in this application. Finally, in Figure 5 (first three panels), we report the mean-squared error obtained by simulating the model in Cai et al. [2015] (Table 2, Column 4) and vary the variance of the residuals σ 2 ∈ {1/4, 1/2, 1} to emulate settings with high, medium and low signal to noise ratio. The model in Cai et al. [2015] assumes that individual outcomes depend on neighbors’ treatments, consistently with our Assumption 1. Our method uniformly outperforms competitors in terms of mean-squared error, except for the Louvain algorithm when the signal-to-noise ratio is particularly high, but not in the remaining settings. In the last panel, we consider settings where the local interference assumption is violated, and we calibrate our model to the specification in Cai et al. [2015] (Column 4, Table 5), where the individual outcome depends on the neighbors’ outcomes, violating Assumption 1 and introducing global dependence. We fix the residuals’ variance to be σ 2 = 1/4, whereas the results are robust as we increase σ 2 . In this setup, we observe a larger mean-squared error of all methods, with the proposed method achieving the lowest mean-squared error. 5.3 A numerical study Next, we illustrate the properties of the method in numerical studies. We consider three different data generating processes for the network formation: a geometric network formation, Albert Barabasi network and Erdos-Renyi graph. The geometric network takes the form Ai,j = 1{|Xi,1 − Xj,1|/2 +|Xi,2 − Xj,2|/2 ≤ rn} where rn = p 4/2.75n similarly to simulations in Leung [2020]. Here, Xi,1, Xi,2 are drawn independently from a uniform distribution between [−1, 1]. For the Albert-Barabasi network we first draw n/5 edges uniformly according to Erdos-Renyi graph with probabilities p = 10/n, and second, we draw sequentially connections of the new nodes to the existing ones with probability equal to the number of connection of each pre-existing node divided by the overall number of connections. Finally, the Erdos-Renyi graph is generated with probability of connection p = 2/n. We study the properties of the proposed method and of the same competitors discussed in Section 5.2. Figure 6 showcases that the proposed procedure leads to the lowest objective function. This result is consistent with our theoretical findings. Figure 7 showcases that the number of clusters increases in the sample size, varies substantially across different data generating processes, and increases in ξn. [13] This follows from the fact that ξn is proportional to ψ/¯ ϕ¯2 n for clusters whose bias is of order O(ϕ¯ n), and researcher’s goal is to minimize the worst-case mean squared error (λ = 1). This paper is under CC 1.0 license. available on arxiv

Design of Cluster Experiments: Empirical Illustration and Numerical Studies

If data is currency, then transfer learning is a messiah for the poors Comparison of two different models (More on it later) While there is no dearth of learning resources on this topic, only a few of them could couple the theoretical and empirical parts together and be intuitive enough. The reason ?? I guess we don’t transfer the knowledge in the exact way we store it in our minds. I believe that presenting complex topics in simple ways is an art, so lets master it. Lets begin a series of blogs where we will try to discuss why the things are the way they are and the intuitions behind them in the domain of machine learning. The first in the line is (TL). We will begin with a crisp intro about TL followed by problem statement which I solved. We will see how TL can be implemented through various ways in Tensorflow and study how each of the tuning parameters in the context of TL affect the model statistics (the most interesting part !!). Finally we will see the conclusions, tips and tricks followed by some of the best references I could find. Transfer Learning Introduction A good naive definition looks like : “It is the ability to transfer knowledge from one domain to another” . Technically, it is: “ using weights trained on one setting (Task 1) to fit a model on another setting (Task 2)”. We generally use transfer learning when: Dataset for Task 2 is small but we still want to train a model. (Presence of small data is not the only case when we would want to use TL, as we shall see later) Both Task 1 and Task 2 are similar ie they have similar low-level features. More similar the tasks are, more applicable this concept is. Transfer learning can be applied across various domains but we will restrict our discussion in the context of images. Based on Similarity and Amount of data available , below are the approaches for different scenarios. Figure 1 : Different approaches in Transfer Learning based on type of datasets (Source : cs231n.github.io) But before going deeper into these approaches, lets study the two major ways in which we can use Transfer Learning : and build upon them. This is similar to what we generally do in case of a typical machine learning problem. We manually engineer features and feed them to our model. But in this case, the heavy lifting of feature engineering is done by the pretrained model based on what it has learned (by learning , we always refer to the weights). Once you get the extracted features , you can feed them to your own new model. So the last point of previous model becomes the input for your next one. You can call this “ ”. Just remove the top layer, add a new one in place of it. And voila !!! , you have got a new model ready. Use the original model architecture as a feature extractor plug and play model . In the above case, we didn’t attempt to modify the weights, rather they were used as it is for feature extraction. But this is not what we want in case the images are bit different. We would like the model to some or all of the features it has learned. But isn’t it similar to training from scratch ?? No it is not. An example would make it clear. Lets say I have learned how to write “E”. Now I want to learn to write “F”. Should I relearn everything from scratch ie what is a straight line etc?? No. I just have to erase pixels of the lower edge and I have learnt a new letter. Finetune the pretrained weights relearn So intuitively, the first case should be used if you are confident that the low level features for both tasks are same. If you are not, you might want to fine tune the model. In some cases, even both of these are used together. Some part of the model is kept as it is whereas some of it is fine tuned. Just hold on, you will find detailed visual examples later. Coming back to the four different scenarios, here is how we tackle them : Task 2 has and to Task 1: smaller dataset similar Fine tuning here is not a good idea because the model will overfit since the new gradients are calculated based on a much less (diverse) data. Doubts ?? . Just imagine that the pretrained weights are some random weights. Then it boils down to usual deep learning scenario. If you feed less data, it will overfit. More similar the images, more the pretrained weights will help you in avoiding overfitting , and hence lesser the sample you can do away with. So it is preferred to use the weights for extracting feature and then training a classifier on top of it. 2. Task 2 has but to Task 1: smaller dataset different Okay lets see. You can’t use fine tuning because data is small. And you can’t use features from top layer because the images are different (features from top layers are specific) . Now what ?? You can follow the same intuition above and extract features from not the penultimate layer but the pre-penultimate one (not sure if this is even a word :P). This is because the features from this layer will be more generic as compared to the ones from output layer. 3. Task 2 has and to Task 1: Feel free to fine tune. larger dataset similar 4. Task 2 has and to Task 1: Again, fine tuning is the way to go. But guess what, you can train a new model from scratch with that amount of data. :P larger dataset different Now, lets begin with our problem statement. Problem Statement To perform OCR using Transfer Learning . : An image containing string of 5 characters. The original model could perform OCR with 99% accuracy. The weights were stored and later used for Task B. This dataset had 100,000 samples, sufficient to train a model from scratch. Task A : Sample Images for Task A Figure 2 An image containing string of 6 characters. Apart from length, one big difference is the type of noise present in both the images. Training data for this one, as a contrast had only samples !!! Task B: 200 : Sample Images for Task B Figure 3 Impressed ? If you are thinking that we can perform any OCR using the training done on any other OCR samples, then there are some nice pointers for you. There are some important behind-the-scenes steps/facts that make this work. The first one is to bring the Task 2 image to the same standards as the Task 1. By standards I mean noise removal. Just like you can’t excel in Maths Exam by reading Sanskrit, in the same way you can’t expect the model to predict on the basis of features it has never seen (or seen very less). For eg: the different thickness of characters in the second image is new to the model as a feature. Remember : More such differences, more it is difficult for the model to adapt to the new case. Noise removal : 2. This is where we don’t have much control. First things first : Input size : Number of parameters learned ∝ Size of input image So the pretrained weight matrix we want to use has dimensions according to the first image. But guess what … both the images are of different sizes. To state it more intuitively: We are asking the model to give values of the parameters it has never seen and therefore never calculated. And hence the second image was resized and modified (added extra rows of white pixels) to match the original one. 3. This is similar to maths-sanskrit example. You can’t expect the model to predict “A” if the only thing it has learned is “a”. And this is true for any two cases where you want to apply transfer learning. Since the possible characters are same for both the images ie [a-z] we could use it without any second thought. Classes : Implementation Finally its time. The architecture for the model A is shown below. It has 3 Convolution layers followed by 3 fully connected layers. : Network Architecture for Task A Figure 4 We will go through two approaches by which we can implement this concept. Approach 1 Restoring the weights: Restoring weights and Importing graph structure If you want to know the basics of storing and restoring tensorflow weight files, is a very good article. this Next, we extract the value (weights) of all those variables / parameters that we are going to use in the Model B. Remember, the model architecture for Task B is same as that of Task A upto the is the one beyond which the architecture of model B differs from model A. So the weights for layers after bottleneck layer are not useful(atleast in our case, otherwise it depends on the tasks. A comparision is shown towards the end of this blog). Bottleneck Layer. Bottleneck layer Lets get the values of pretrained variables. gives all those variables which were trained in the model A. This includes all the Weight matrices as well as the Biases. So in our case gives the following output: tf.trainable_variables() tf.trainable_variables() conv2d/kernel:0   conv2d/bias:0   conv2d_1/kernel:0   conv2d_1/bias:0   conv2d_2/kernel:0   conv2d_2/bias:0   dense/kernel:0   dense/bias:0   dense_1/kernel:0   dense_1/bias:0   dense_2/kernel:0   dense_2/bias:0 As you can see, there are two parts to each variable name. First part is the name of the layer where that variable is defined followed by the type of the variable ie Kernel (usual weight matrix) or Bias. Coming back to the bottleneck layer, it is the penultimate layer , the one just before the output layer in our case. The variables restored in previous step are the ones corresponding to the red portion (shown below). This portion is what we call part of the model. The Green portion is the actual which does the final job of classification. feature extractor classifier : Red portion is responsible for feature extraction and the Green portion does the classification. Figure 5 The feature extractor has more to it. It can further be divided into two sets of layers. One set is frozen ( for Red), that is it’s weights don’t change . Second one ( for Blue) is where we do the fine tuning. R B : Red Portion refers to the frozen layers. Blue portion refers to the layers which are fine tuned. Figure 6 Number of layers we want to fine tune ( ) vs the ones we want to remain same ( ) will depend on the similarity. Below figures represent different scenarios in the decreasing order of similarity. Top most refers to the case where images are almost same. B R : Both tasks are similar. No need to change the pretrained weights (Hence no and only Figure 7 B R) : The Tasks are a bit different but not completely. So we can use low level features as they are ( and fine tune the upper ones ( ) Figure 8 R) B : Variation of the above case Figure 9 : Tasks are very different. So we don’t want to use any of the pretrained weights as such. So we fine tune them and hence only Figure 10 B. So which one we are going to use for our case ?? The last one. Modification of green portion is quite intuitive. Since word lengths of both tasks are different along with other dissimilarities, the original classifier needs to re-learn to output 6 letters instead of 5. The input to this classifier is the features extracted till bottleneck layer (ie through and ) R B The architecture (ie number of layers and their neurons) should remain same for the and portion because the dimensions of weight matrix restored should be compatible with it. The Green portion, however , can differ from the original model. For us, we don’t need extra layers. Only the output layers neurons need to be modified. R B This is how model of Task B looks like (Figure 11). Each layer remains same (the architecture , not the weights) except the output layer. : Model architecure for Task B Figure 11 So we fine tune the weights of all the layers for our case. But what about the weights of Green portion ? Do we fine tune pretrained weights for this portion also or we just initialize them randomly? More on this later. Lets continue with the code. Defining the layers The 1st Convolution Layer in the original model which was used to train Task 1 is as follows: The same layer , when being used for Task 2 looks like: The added parameters are: , and . The first two are used to initialize the kernel and bias weights with the pretrained ones. Whereas the third one is to state whether we want to modify weights of this layer. kernel_initializer bias_initializer, trainable If we want to do ie the portion, , otherwise it is if want to freeze the weights and use this layer as it is, the portion. Fine tuning B trainable = _True_ False R Approach 2 The steps for importing the graph and initializing the weights with the pretrained ones remains same as above except parameter , the alternative of which is given below. The second approach to selectively train and freeze the layers is below: trainable train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,scope="dense[12]") training_op = optimizer.minimize(loss, var_list=train_vars) Lets assume that we want to train only the output and penultimate layer while freezing the lower ones. The first line gets the list of all variables in dense layers 1 and 2 (penultimate and output layer) which are the ones to be trained. This leaves out the variables in rest of the layers. Next we pro‐vide this restricted list of trainable variables to the optimizer’s minimize(). Rest of the layers are are now frozen: they will not budge during training (these are the the frozen layers). Observations Enough of theory . Its time for some exciting stuff. Lets tweak some common parameters, compare and study their effects by plotting graphs. Let me first describe the legends; Accuracy is being calculated over two levels :Character level (Orange line)= Word level (Green line)= Number of characters correctly predicted / Total number of characters in samples Number of entire words correctly predicted / Total number of samples Eg: if actual word is “xmladj” and the predicted one is “xmladt”, then the word level accuracy = 0 and character accuracy is 5/6. The Blue line represents character level training accuracy. So, in short you would like to observe the mainly Green line. Lets begin !! Effect of number of trainable layers : Only the last layer is trained. Rest all are frozen. : All the layers are made trainable and are fine tuned. Left Right : : Only the last layer is trained. Rest all are frozen. : All the layers are made trainable and are fine tuned. Figure 12 Left Right Since the images are quite different, retraining only the output layer is not sufficient (Left). A slight change in the low-level features is also required , hence fine tuning all the layers did the job (Right). Intuition: 2. Effect of Learning rate : Learning rate = 0.001. : Learning rate = 0.0001. Left Right : : Learning rate = 0.001. : Learning rate = 0.0001 Figure 13 Left Right Learning rate guides how big steps we are taking towards the optimum point. A lower learning rate (R) took more number of epochs but converged as good as the one with larger rate. But generally its preferred to keep learning rate for Task 2 smaller than what was used for Task 1. This is to ensure that the large gradient updates don’t distort the pretrained weight values. So we first allow the weights to stabilize and then let it propagate towards the inner layers. Intuition: 3. Last layer initialised vs not initialised For both the cases, all layers but the output layer are frozen. : Even the output layer is initialized with the pretrained weights and fine tuned. : The output layer is trained from scratch ie with random initial weights. Left Right : : Even the output layer is initialized with the pretrained weights and fine tuned. : The output layer is trained from scratch ie with random initial weights. Figure 14 Left Right Wait. What ?? The graphs are exactly the same !!. So interestingly, this means that pretrained weights (L) did no better than the random weights (R) for the output layer. And this is because the images were different enough so as to render the pretrained weights useless. Intuition: 4. Effect of number of samples : Number of samples 50 : Number of samples 200 Left Right : : Number of samples 50. : Number of samples 200 Figure 15 Left Right It seems that number of samples didn’t make any difference in overall accuracy since the difference between the numbers is not huge. But the thing worth noting is the huge amount of fluctuations in test accuracy for the case of 50 samples in comparison to the one with 200 samples. This is because the batch size was the same as the number of samples. And since the smaller batch size leads to more frequent weight updates without observing reasonably good amount of data, the model could not be certain about it’s predictions, hence this observation. Intuition: Summary Usually, the output layer of the original model should be replaced since it is most likely not useful at all for the new task, and it may not even have the right number of outputs for the new task. Similarly, the upper hidden layers of the original model are less likely to be as useful as the lower layers, since the high-level features that are most useful for the new task may differ significantly from the ones that were most useful for the original task. You want to find the right number of layers to reuse. Try freezing all the copied layers first, then train your model and see how it performs. Then try unfreezing one or two of the top hidden layers to let backpropagation tweak them and see if performance improves. The more training data you have, the more layers you can unfreeze. If you still cannot get good performance, and you have little training data, try dropping the top hidden layer(s) and freeze all remaining hidden layers again. You can iterate until you find the right number of layers to reuse. If you have plenty of training data, you may try replacing the top hidden layers instead of dropping them, and even add more hidden layers. References Some of the best resources I could find: Blogs: Stanford’s CS231n : http://cs231n.github.io/transfer-learning/ Sebastian Ruder : . A good starting point for Transfer Learning. http://ruder.io/transfer-learning/ . Good tutorial on saving and restoring tensorflow models. http://cv-tricks.com/tensorflow-tutorial/save-restore-tensorflow-models-quick-complete-tutorial/ Papers: How transferable are features in deep neural networks A survey on transfer learning Books: Hands-On Machine Learning with Scikit-Learn and Tensor Flow by AurElien Geron. This book has a good intro about Transfer Learning. That’s not all. But enough to get you started . If you feel I have missed something or you have any clarifications, please drop in your comments. And yes, don’t forget to Transfer your Learning of Transfer Learning.

Stories, News & Mentions about EMPIRICAL

Empirical’s Stories on HackerNoon