Real-time user clickstream behavior analysis based on apache storm streaming

Pal, Gautam ORCID: 0000-0002-2594-9699, Atkinson, Katie ORCID: 0000-0002-5683-4106 and Li, Gangmin
(2021) Real-time user clickstream behavior analysis based on apache storm streaming. Electronic Commerce Research.

Access the full-text of this item by clicking on the Open Access link.


<jats:title>Abstract</jats:title><jats:p>This paper presents an approach to analyzing consumers’ e-commerce site usage and browsing motifs through pattern mining and surfing behavior. User-generated clickstream is first stored in a client site browser. We build an ingestion pipeline to capture the high-velocity data stream from a client-side browser through Apache Storm, Kafka, and Cassandra. Given the consumer’s usage pattern, we uncover the user’s browsing intent through <jats:italic>n-grams</jats:italic> and <jats:italic>Collocation</jats:italic> methods. An innovative clustering technique is constructed through the Expectation-Maximization algorithm with Gaussian Mixture Model. We discuss a framework for predicting a user’s clicks based on the past click sequences through <jats:italic>higher order Markov Chains</jats:italic>. We developed our model on top of a big data Lambda Architecture which combines high throughput Hadoop batch setup with low latency real-time framework over a large distributed cluster. Based on this approach, we developed an experimental setup for an optimized Storm topology and enhanced Cassandra database latency to achieve real-time responses. The theoretical claims are corroborated with several evaluations in Microsoft Azure HDInsight Apache Storm deployment and in the Datastax distribution of Cassandra. The paper demonstrates that the proposed techniques help user experience optimization, building recently viewed products list, market-driven analyses, and allocation of website resources.</jats:p>

Item Type: Article
Divisions: Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User: Symplectic Admin
Date Deposited: 20 Jan 2022 10:29
Last Modified: 18 Jan 2023 21:15
DOI: 10.1007/s10660-021-09518-4
Open Access URL:
Related URLs: