Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

4-2026

Abstract

Large Language Models (LLMs) have accelerated the rapid development of chatbot web applications in various domains, such as coding, biomedicine and psychology. Compared to general LLMs like ChatGPT, domain-specific LLMs require a greater sense of responsibility. For instance, if a programming LLM casually answers medical or psychological questions, it not only misleads the public but also poses legal risks. This highlights new demands for monitoring and preventing such irresponsible behaviors. Existing efforts attempt to monitor LLMs from multiple aspects, such as lying, jailbreaks, and toxic content, while overlooking out-of-domain behaviors. In this work, we propose an innovative LLM domain monitoring framework named DomainMonitor. By analyzing the internal hidden states in domain-specific LLMs, DomainMonitor can effectively detect out-of-domain behaviors, suggesting potential 'rejection' instructions for follow-up responses. Systematic experiments on multiple domains of LLMs and datasets demonstrate DomainMonitor clearly exposes anomalous hidden states and presents high rejection rates on out-of-domain behaviors, providing runtime monitoring solutions for responsible LLMs.

Keywords

Responsible Web Applications, Domain-Specific LLMs, LLM Monitoring, Out-of-Domain Behaviors

Discipline

Artificial Intelligence and Robotics | Information Security

Research Areas

Software and Cyber-Physical Systems

Areas of Excellence

Digital transformation

Publication

WWW '26: Proceedings of the ACM Web Conference 2026, Dubai, United Arab Emirates, April 13-17

First Page

1751

Last Page

1760

ISBN

9798400723070

Identifier

10.1145/3774904.3792621

Publisher

ACM

City or Country

New York

Additional URL

https://doi.org/10.1145/3774904.3792621

Download

Share

COinS