Online Information of Vaccines: Web Search and Content Analysis

Table of Links

• Abstract and Introduction
• Methods
• Web search and content analysis
• Results
• Discussion
• Conclusion and References
• Table 1. Example of classification of webpages
• Table 2. Percentage of vaccine-negative webpages in different SERPs

Web search and content analysis

Searches in English were done from Falmer, Sussex, United Kingdom; in Italian from Urbino, Italy; in French from Bruxelles, Belgium; in Spanish from Barcelona, Spain. Each search was done using a logged-out Chrome browser cleared of cookies and previous search history so that the only identification was the IP address and its geolocalization. Additionally, when available, the local version of each search engine was used (e.g. Google.co.uk and Google.it). For searching Google.com, automatic redirection to google.co.uk was avoided by using the URL Google.com/ncr (no-country-redirect).

The first 30 URL results from each search engine result page (SERP), excluding those marked as advertisements, and were transferred to a spreadsheet. When a duplicate URL was encountered, it was excluded. Websites were then visited and the content of each page was coded as vaccine-positive, negative or neutral, depending on the stance taken on the connection between vaccines and autism.

Webpages recommending vaccination and/or negating the link with autism were coded as “vaccine-positive”. Those promoting vaccine hesitancy, cautioning about the risk of autism or openly anti-vaccine, were coded as “vaccine-negative”. Additionally, webpages that claimed further studies needed to be conducted to clarify the link between vaccines and autism were also coded as “vaccine-negative”, as previous research by {Browne, 2015 #421} has shown that users perceive this as confirmation of the fact that vaccine safety has not been proven. Webpages simply reporting the history of the anti-vaccine movement or a related legal debate were coded as “vaccine-neutral”. Examples of positive, negative and neutral webpages are provided in Table 1.

Coding was completed by two raters for each language and inter-rater agreement was calculated with GraphPad, which uses equations 18.16 to 18.20 from Fleiss 19. On a sample of 59 webpages in English, agreement was 85%, with a Kappa of 0.669 (standard error, 0.077) and a 95% confidence interval from 0.518 to 0.820, a strength of agreement is considered to be 'good' 19. In Italian, agreement was 90%, with a Kappa of 0.818 (standard error, 0.067) and a 95% confidence interval from 0.687 to 0.950, a strength of agreement is considered to be 'very good'. In Spanish, agreement was 83%, with a Kappa of 0.609 (standard error, 0.098) and a 95% confidence interval from 0.418 to 0.801, a strength of agreement is considered to be 'very good'. In French, agreement was 89% with a Kappa of 0.746 (standard error, 0.091) and a 95% confidence interval from 0.568 to 0.924.

When frequencies of vaccine-negative webpages was compared across different search engines, we used a two-tailed Fisher’s test corrected for multiple comparison using the method of Benjamini, Krieger and Yekutieli and a false discovery rate (FDR) of 5%.