Evaluating the usability of open-source software (OSS), software that is made freely available to developers worldwide, generally entails analyzing the feedback and comments of those who used it. Processing and understanding the feedback provided in user discussions, however, can be challenging due to the vast number of comments online, and because they often present opposing opinions.

Aware of these difficulties, researchers at McGill University, the University of Bari and Polytechnique Montreal have recently developed ArguLens, a theoretical framework and automated technique to simplify the analysis of community discussions related to the usability of OSS. Their recent paper, pre-published on arXiv, is part of a broader ongoing research effort aimed at helping developers of open source projects to improve the usability of their software.

"Understanding the rationales behind community requests and further evaluating their impacts on the end users can be very difficult, especially when the community's opinions are in conflict with each other," the researchers told TechXplore, via email. "The main objective of our study was to help open source contributors digest and evaluate the community's opinions about the usability of their project."

Before they started working on ArguLens, the researchers had lengthy discussions with open source contributors to better understand what they found most challenging about understanding feedback they received from the community. They found that most developers experienced huge difficulties in digesting and incorporating community comments, including suggestions and requests made via issue tracking systems.

This was due to the sheer volume of feedback they collected, as well as to the fact that users often expressed diverse and somewhat opposing perspectives. Combined, these two factors often made processing and digesting usability-related feedback challenging.

The researchers devised ArguLens with the key goal of assisting OSS developers in making sense of the rich and varied feedback they collected online. ArguLens includes both a conceptual framework and an automated technique for the analysis of online comments.

"As a conceptual framework, it adapts an argumentation model proposed by Stephen Toulmin to structure the usability-related discussions into three major elements: Claim, Ground, and Warrant," the researchers explained. "This is what we call the 'anatomy' of community opinions. As an automated technique, it proposes natural language processing methods to detect these elements from usability-related discussions in free form."

Developers and other community members can use ArguLens to differentiate and synthesize concrete requests made by community members (captured within the "Claim' category), as well as facts about their system that the community cares about (captured within the 'ground' category) and personal opinions about why individual facts are particularly important to users (captured within the "Warrant' category). Overall, the framework can thus be used for scaffolding discussions and for analyzing OSS usability issues.

As part of their research, the team evaluated and ascertained ArguLens' scalability using supervised machine learning techniques for automated argument extraction. They also asked experienced developers to use ArguLens and give feedback on whether it helped them in understanding community comments. The feedback they received was highly promising, as the majority of users said that the framework helped them to better digest and review usability-related opinions.

"A direct application of the ArguLens framework could be in tools for open-source issue tracking systems to provide an effective interface to the community's discussions about usability issues," the researchers said. "Once widely adopted, this framework can induce a groundbreaking change in how people consider open-source usability issues, encouraging constructive discussion about these issues and eventually helping improve open source usability."

The framework could soon aid OSS developers worldwide in making sense of contrasting opinions about the usability of their projects. Interestingly, however, it could also be applied to other aspects of software that can be improved based on user feedback, such as privacy or security. The team is now planning to improve the framework further, focusing on two new research directions.

"First, we are investigating tool designs that can effectively present the information captured by the ArguLens framework," the researchers said. "This will not only help us better understand the efficacy of this framework in the wild (in real-world settings) but it can also result in concrete applications that directly benefit the open source communities. Second, we are focusing on human-in-the-loop methods to mitigate the imperfect classification of automated techniques, to further ensure the scalability and usefulness of this framework."

---

---

More information: ArguLens: anatomy of community opinions on usability issues using argumentation models. arXiv:2001.06067 [cs.HC]. arxiv.org/abs/2001.06067

© 2020 Science X Network