Multimodal deep learning with boosted trees for edge inference

Publication Type

Conference Proceeding Article

Publication Date

12-2023

Abstract

We provide a method to combine and optimize knowledge from neural network and gradient boosted tree models for inference on edge devices. This is important for multimodal settings having both image and sensor, or time series, data. The proposed approach retains the learning capabilities and jointly distills knowledge from the tree structure, approximated by an embedding layer, and the internal representations of a CNN, along with the aggregated outputs of the heterogeneous teacher models. Performance is better than that of unimodal and standard multimodal training approaches. The resulting multimodal network is smaller and consumes less memory during inference than the alternative networks, making it ideal for applications at the edge.

Keywords

boosted trees, CNNs, distillation, edge inference, multimodal

Discipline

Databases and Information Systems | Numerical Analysis and Scientific Computing

Research Areas

Intelligent Systems and Optimization

Publication

Proceedings of the 2023 IEEE International Conference on Data Mining Workshops (ICDMW), Shanghai, China, December 1-4

First Page

99

Last Page

108

ISBN

9798350381641

Identifier

10.1109/ICDMW60847.2023.00021

Publisher

IEEE Computer Society

City or Country

Los Alamitos, CA

Additional URL

https://doi.org/10.1109/ICDMW60847.2023.00021

This document is currently not available here.

Share

COinS