Publication Type
PhD Dissertation
Version
publishedVersion
Publication Date
5-2026
Abstract
This dissertation tackles the practical problem of making intelligent systems safe without dulling their utility, with contributions in both long-horizon control and large language models (LLMs). The guiding idea is to treat safety as a first-class optimization objective—implemented as explicit, tunable constraints in planning and as structured geometry in representation space in language—so that agents and models become meaningfully safer without drifting into over-conservatism that harms usefulness.
First, for sequential decision-making, the thesis introduces a constrained hierarchical reinforcement learning framework that elevates safety to a first-class, adjustable objective. A high-level constrained planning agent selects subgoal under explicit risk limits, while a low-level goal-conditioned policy achieves them sequentially. This design addresses sparse rewards, unsafe exploration, and costly re-planning in long-horizon settings by enabling fast policy recomputation when thresholds or start/goal states change. Across safety-critical benchmarks, the approach simultaneously improves constraint satisfaction, task success, and re-planning efficiency relative to hierarchical and constrained baselines, demonstrating a practical route to reliable long-horizon control under heterogeneous safety requirements.
Second, within language alignment, this thesis propose a semantics-guided, dataefficient supervised fine-tuning (SFT) method that trains on harmful data only—no explicit refusal demonstrations are required. The method introduces an Earth-Mover’sDistance (EMD) semantic penalty that pushes the next-token distribution away from unsafe responses generated by itself. This yields high safety (strong refusal to toxic prompts) while preserving response quality and general capability, and achieves short training time relative to RLHF. Notably, despite the absence of refusal examples in the training set, the model exhibits a systematic over-refusal pattern on seemingly toxic inputs, mirroring the behavior observed in methods trained with refusal demonstrations. This finding motivates a principled remedy.
Third, to explore how to make real safety alignment avoiding over-refusal issue, the thesis introduces Discernment via Contrastive Refinement (DCR), a preceding stage before safety alignment that decouples intermediate representations of “seemingly toxic” and “truly toxic” prompts. Theoretically, the work bounds eNTK similarity via a bilinear feature metric, showing that contrastive separation reduces harmful gradient transfer during subsequent safety tuning. Empirically, across multiple model families and both in-distribution and out-of-distribution settings, DCR raises compliance on over-refusal benchmarks while maintaining comparable defensive strength on harmfulness suites, outperforming data-augmentation baselines and avoiding the quality regressions of activation-steering methods.
Overall, the three studies form a coherent, complementary toolkit: (i) a planning method that treats safety as explicit, adjustable constraints for reliable longhorizon control; (ii) a semantics-aware, harmful-only SFT recipe that delivers dataefficient safety without degrading response quality or general capability; and (iii) a representation-level intervention that decouples benign from harmful inputs to curb unnecessary refusals. Together, they demonstrate that safety can be enforced where it matters—as contracts in control and as geometry in alignment—yielding systems that are both safe and useful. The dissertation thereby provides principled mechanisms and empirical evidence that robust safety need not come at the expense of capability, and it points to natural extensions tightly aligned with these results: multi-agent planning under shared risk constraints, multilingual safety with cultural sensitivity, and alignment pipelines that combine contrastive preconditioning with semantics-aware objectives.
Keywords
Large Language Models, Safety Alignment, Reinforcement Learning
Degree Awarded
PhD in Computer Science
Discipline
Artificial Intelligence and Robotics | Computer Sciences
Supervisor(s)
VARAKANTHAM, Pradeep Reddy; SINHA, Arunesh
First Page
1
Last Page
108
Publisher
Singapore Management University
City or Country
Singapore
Citation
LU, Yuxiao.
Safety-constrained learning for intelligent systems: from risk-aware planning to safety-aligned large language models. (2026). 1-108.
Available at: https://ink.library.smu.edu.sg/etd_coll/921
Copyright Owner and License
Author
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.