Big Data Ingestion and Lifelong Learning Architecture

Pal, Gautam ORCID: 0000-0002-2594-9699, Li, Gangmin and Atkinson, Katie ORCID: 0000-0002-5683-4106
(2019) Big Data Ingestion and Lifelong Learning Architecture. In: 2018 IEEE International Conference on Big Data (Big Data), 2018-12-10 - 2018-12-13.

[img] Text
Big Data Ingestion and Lifelong Learning Architecture.pdf - Author Accepted Manuscript

Download (1MB) | Preview


Lifelong Machine Learning (LML) mimics common human learning experiences. Humans undergo through long learning phase at start while studying followed by updating knowledge base incrementally from everyday instances. The objective is to retain past learnt knowledge and transfer learning to the next task iteratively. Training on the large data pool through a one-shot long running batch job limits the responsiveness and increases the infrastructure cost through large cluster requirements. The full dataset may not be available as well at the initiation of the training process. Through a review of previous work on lifelong machine leaning, we propose a Multi-agent Lambda Architecture (MALA) model to combine historical batch data with live streaming data to develop a lifelong learning system. MALA allows the streaming process to initialize itself with trained model from the batch data. Streaming process takes the batch data offset and incrementally updates the model iteratively with new waves of data. Reasons for our claim are presented through implementation of a recommender engine.

Item Type: Conference or Workshop Item (Unspecified)
Uncontrolled Keywords: Lifelong learning, Incremental learning, Multi-agent System, Recommender systems
Depositing User: Symplectic Admin
Date Deposited: 10 Feb 2020 10:35
Last Modified: 19 Jan 2023 00:04
DOI: 10.1109/bigdata.2018.8621859
Related URLs: