Algorithm Auditing at a Large-Scale: Insights From Search Engine Audits

Ulloa, Roberto; Makhortykh, Mykola; Urman, Aleksandra (27 May 2022). Algorithm Auditing at a Large-Scale: Insights From Search Engine Audits (Unpublished). In: 72nd Annual ICA Conference - "One world, one network?!". Paris, France. 26.05.-30.05.2022.

Full text not available from this repository.

Algorithm audits have increased in recent years due to a growing need to independently assess the performance of automatically curated services that process, filter and rank the large and dynamic amount of information available on the internet. Among several methodologies to perform such audits, virtual agents stand out because they offer the ability to perform systematic experiments, simulating human behaviour without the associated costs of recruiting participants. Motivated by the importance of research transparency and replicability of results, this paper focuses on the challenges of such an approach. It provides methodological details, recommendations, lessons learned and limitations that researchers should take into consideration when setting up experiments with virtual agents. We demonstrate the successful performance of our research infrastructure across multiple data collections, with diverse experimental designs, and point to different changes and strategies that improve the quality of the method. We conclude that virtual agents are a promising venue for monitoring the performance of algorithms across long periods of time, and we hope that this paper can serve as a basis for further research in this area.

Item Type:	Conference or Workshop Item (Paper)
Division/Institute:	03 Faculty of Business, Economics and Social Sciences > Social Sciences > Institute of Communication and Media Studies (ICMB)
UniBE Contributor:	Makhortykh, Mykola, Urman, Aleksandra
Subjects:	000 Computer science, knowledge & systems 300 Social sciences, sociology & anthropology
Language:	English
Submitter:	Mykola Makhortykh
Date Deposited:	23 Jun 2022 14:53
Last Modified:	05 Dec 2022 16:20
Uncontrolled Keywords:	algorithms, algorithm audit, web search, search engines, computational methods
URI:	https://boris.unibe.ch/id/eprint/170657