GenAIPABench
A benchmark for evaluating Generative AI-based Privacy Assistants (GenAIPAs) in terms of their ability to understand and respond to privacy policy and data protection regulation queries.
Illustration of GenAIPABench Evaluation Process
Privacy policies of websites are often lengthy and intricate. Privacy assistants assist in simplifying policies and making them more accessible and user-friendly. The emergence of generative AI (genAI) offers new opportunities to build privacy assistants that can answer users' questions about privacy policies. However, genAI’s reliability is a concern due to its potential for producing inaccurate information. This study introduces GenAIPABench, a benchmark for evaluating Generative AI-based Privacy Assistants (GenAIPAs). GenAIPABench includes: 1) A set of questions about privacy policies and data protection regulations, with annotated answers for various organizations and regulations; 2) Metrics to assess the accuracy, relevance, and consistency of responses; and 3) A tool for generating prompts to introduce privacy documents and varied privacy questions to test system robustness. We evaluated three leading genAI systems—ChatGPT-4, Bard, and Bing AI—using GenAIPABench to gauge their effectiveness as GenAIPAs. Our results demonstrate significant promise in genAI capabilities in the privacy domain while also highlighting challenges in managing complex queries, ensuring consistency, and verifying source accuracy.
Hemanth Reddy Samidi
MS Student
My research interests include Machine Learning, Data Science, Data Visualization, and Privacy.
Tim Finin
Professor
Tim Finin is the Willard and Lillian Hackerman Chair in Engineering and a Computer Science and Electrical Engineering professor at the University of Maryland, Baltimore County (UMBC). He has over 50 years of experience in applying AI to problems in information systems and language understanding. His current research focuses on representing and reasoning with knowledge graphs, analyzing and extracting information from text, and enhancing security and privacy in information systems. He is an ACM fellow, a AAAI fellow, an IEEE technical achievement award recipient, and was selected as the UMBC Presidential Research Professor in 2012. Finin received an S.B. degree in Electrical Engineering from MIT and a Ph.D. in Computer Science from the University of Illinois at Urbana-Champaign. He has held positions at UMBC, Unisys, the University of Pennsylvania, Johns Hopkins University, and the MIT AI Laboratory. He has chaired the UMBC Computer Science department, served on the Computing Research Association board of directors, been a AAAI councilor, and chaired many major research conferences. He is a former editor-in-chief of the Elsevier Journal of Web Semantics.
Primal Pappachan
Assistant Professor
My research interests include data management, privacy, and Internet of Things.
Roberto Yus
Assistant Professor
My research interests include Data Management, Knowledge Representation, the Internet of Things, and Privacy.