GLID$^2$E: A Gradient-Free Lightweight Fine-Tune Approach for Discrete Sequence Design

Abstract

The design of biological sequences is essential for engineering functional biomolecules that contribute to advancements in human health and biotechnology. Recent advances in diffusion models, with their generative power and efficient conditional sampling, have made them a promising approach for sequence generation. To enhance model performance on limited data and enable multi-objective design and optimization, reinforcement learning (RL)-based fine-tuning has shown great potential. However, existing fine-tuning methods are often unstable in discrete optimization when not using gradients or become computationally inefficient when relying on gradient-based approaches, creating significant challenges for achieving both control and stability in the tuning process. To address these issues, we propose GLID$^2$E, a gradient-free RL-based tuning approach for discrete diffusion models. Our method introduces a clipped likelihood constraint to regulate the exploration space and reward shaping to better align the generative process with design objectives, ensuring a more stable and efficient tuning process. By integrating these techniques, GLID$^2$E mitigates training instabilities commonly encountered in RL and diffusion-based frameworks, enabling robust optimization even in challenging biological design tasks. In two biological systems, GLID$^2$E achieves competitive performance in function-based design while ensuring lightweight and efficient tuning.