Discovery of Interaction Patterns with Graphical User Interface Usage Mining

The Graphical User Interface (GUI) serves as an interface to directly manipulate graphical elements. Application softwares contain these graphical elements which represent functionality for a specific application. The main target is the discovery of patterns which express functionality usage. In the first part, the interaction between study participants and application softwares are observed by a tool called „Interaction Observer“. This is accomplished with the Accessibility technology which provides access to graphical elements. The approach is termed „Graphical Software Mining“ which mines software exclusively on a graphical level. The observation forms an interaction log. In the second part, interaction patterns are defined and in the interaction log discovered. Interaction patterns are sequences of interactions which express an intentional action. This approach is termed „GUI Usage Mining“ which discovers usage information from GUI interaction logs. Initially, reference patterns are discovered by human annotators. The insights are used to preprocess the raw interaction log. Transactions, which are meaningful interaction clusters, are identified with four approaches. This enables the application of four mining strategies: sequential pattern mining, graph mining, process mining and mining based on n-grams. They are implemented with the help of extern libraries. User-program pairs are analyzed in the evaluation. The patterns of the four strategies are judged by the corresponding participants. The evaluation indicates that the n-gram based strategy discovers more accepted patterns.

Thesis

Presentation