Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

10-2025

Abstract

Cooking plays a vital role in everyday independence and well-being, yet remains challenging for people with vision impairments due to limited support for tracking progress and receiving contextual feedback. Object status — the condition or transformation of ingredients and tools — offers a promising but underexplored foundation for context-aware cooking support. In this paper, we present OSCAR (Object Status Context Awareness for Recipes), a technical pipeline that explores the use of object status recognition to enable recipe progress tracking in non-visual cooking. OSCAR integrates recipe parsing, object status extraction, visual alignment with cooking steps, and time-causal modeling to support real-time step tracking. We evaluate OSCAR on 173 instructional videos and a real-world dataset of 12 non-visual cooking sessions recorded by BLV individuals in their homes. Our results show that object status consistently improves step prediction accuracy across vision-language models, and reveal key factors that impact performance in real-world conditions, such as implicit tasks, camera placement, and lighting. We contribute the pipeline of context-aware recipe progress tracking, an annotated real-world non-visual cooking dataset, and design insights to guide future context-aware assistive cooking systems.

Keywords

Cooking, Context Awareness, Recipe, Object Status, Blind, People with Vision Impairments, Accessibility, Assistive technology

Discipline

Biomedical Engineering and Bioengineering | Graphics and Human Computer Interfaces

Research Areas

Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

ASSETS '25: Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility, Denver, Colorado USA, October 26-29

First Page

1

Last Page

15

ISBN

9798400706769

Identifier

10.1145/3663547.3746318

Publisher

ACM

City or Country

New York

Additional URL

https://doi.org/10.1145/3663547.3746318

Download

Share

COinS