Automatic cost learning for steganography based on deep neural networks is receiving increasing attention. Steganographic methods under such a framework have been shown to achieve better security performance than methods adopting hand-crafted costs. However, they still exhibit some limitations that prevent a full exploitation of their potentiality, including using a function-approximated neural-network-based embedding simulator and a coarse-grained optimization objective without explicitly using pixel-wise information. In this article, we propose a new embedding cost learning framework called SPAR-RL (Steganographic Pixel-wise Actions and Rewards with Reinforcement Learning) that overcomes the above limitations. In SPAR-RL, an agent utilizes a policy network which decomposes the embedding process into pixel-wise actions and aims at maximizing the total rewards from a simulated steganalytic environment, while the environment employs an environment network for pixel-wise reward assignment. A sampling process is utilized to emulate the message embedding of an optimal embedding simulator. Through the iterative interactions between the agent and the environment, the policy network learns a secure embedding policy which can be converted into pixel-wise embedding costs for practical message embedding. Experimental results demonstrate that the proposed framework achieves state-of-the-art security performance against various modern steganalyzers, and outperforms existing cost learning frameworks with regard to learning stability and efficiency.

While Deep Reinforcement Learning (DRL) has achieved the outstanding performance in extensive applications, exploiting its vulnerability with adversarial attacks is essential towards building robust DRL systems. In this work, we aim to propose a novel Decoupled Adversarial Policy (DAP) for attacking the DRL mechanism, whereas the adversarial agent can decompose the adversarial policy into two separate sub-policies: 1) the switch policy which determines if an attacker should launch the attack, and 2) the lure policy which determines the action an attacker induces the victim to take. If the adversarial agent samples an injection action from the switch policy, the attacker can query the pre-constructed database for universal perturbation in the real-time manner, misleading the victim to take the induced action sampled from the lure policy. To train the adversarial agent to learn DAP, we utilize those samples wherein both of the sub-actions from DAP are not restricted by each other or by the external constraint, but can actually affect the attackers behaviors. Therefore, we propose trajectory clipping and padding in data pruning, and Decoupled Proximal Policy Optimization (DPPO) in optimizing. Extensive experiments on different Atari games demonstrate the effectiveness of our proposed method. In addition, it can simultaneously implement the real-time and few-steps attack, which outperforms the existing counterparts.

Since the generative adversarial network (GAN) was proposed by Ian Goodfellow et al. in 2014, it has been widely used in various fields. However, there are only a few works related to image steganography so far. Existing GAN-based steganographic methods mainly focus on the design of generator, and just assign a relatively poorer steganalyzer in discriminator, which inevitably limits the performances of their models. In this paper, we propose a novel Steganographic method based on Generative Multi-Adversarial Network (Steg-GMAN) to enhance steganography security. Specifically, we first employ multiple steganalyzers rather than a single steganalyzer like existing methods to enhance the performance of discriminator. Furthermore, in order to balance the capabilities of the generator and the discriminator during training stage, we propose an adaptive way to update the parameters of the proposed GAN model according to the discriminant ability of different steganalyzers. In each iteration, we just update the poorest one among all steganalyzers in discriminator, while update the generator with the gradients derived from the strongest one. In this way, the performance of generator and discriminator can be gradually improved, so as to avoid training failure caused by gradient vanishing. Extensive comparative results show that the proposed method can achieve state-of-the-art results compared with the traditional steganography and the modern GAN-based steganographic methods. In addition, a large number of ablation experiments verify the rationality of the proposed model.

Recently, the success of non-additive steganography has demonstrated that asymmetric distortion can remarkably improve security performance compared with symmetric cost functions. However, most of current existing additive steganographic methods are still based on symmetric distortion. In this paper, for the first time we optimize asymmetric distortion for additive steganography and propose an A3C(Asynchronous Advantage Actor-Critic) based steganographic framework, called ReLOAD. ReLOAD is composed of an actor and a critic, where the former guides action selection for pixel-wise distortion modulation, and the latter evaluates the performance of modulated distortion. Meanwhile, a reward function that considers embedding effects is proposed to unify the goal of steganography and reinforcement learning, so that the minimization of embedding effects can be achieved by learning secure policy to maximize total rewards. Statistical analysis shows that compared with non-additive steganography, ReLOAD achieves lower change rates and makes embedding traces more consistent with cover image textures. Comprehensive experiments conducted on both hand-crafted feature-based and deep learning-based steganalyzers show that ReLOAD significantly promotes the state-of-the-art security performance of current additive methods and even outperforms non-additive steganography when the modification distribution gets sparser.

Although significant progress has been achieved recently in automatic learning of steganographic cost, the existing methods designed for spatial images cannot be directly applied to JPEG images which are more common media in daily life. The difficulties of migration are mainly caused by the characteristics of the $8\times 8$ DCT mode structure. To address the issue, in this paper we extend an existing automatic cost learning scheme to JPEG, where the proposed scheme called JEC-RL (JPEG Embedding Cost with Reinforcement Learning) is explicitly designed to tailor the JPEG DCT structure. It works with the embedding action sampling mechanism under reinforcement learning, where a policy network learns the optimal embedding policies via maximizing the rewards provided by an environment network. Following a domain-transition design paradigm, the policy network is composed of three modules, i.e., pixel-level texture complexity evaluation module, DCT feature extraction module, and mode-wise rearrangement module. These modules operate in serial, gradually extracting useful features from a decompressed JPEG image and converting them into embedding policies for DCT elements, while considering JPEG characteristics including inter-block and intra-block correlations simultaneously. The environment network is designed in a gradient-oriented way to provide stable reward values by using a wide architecture equipped with a fixed preprocessing layer with $8\times 8$ DCT basis filters. Extensive experiments and ablation studies demonstrate that the proposed method can achieve good security performance for JPEG images against both advanced feature-based and modern CNN-based steganalyzers.

Recent research has shown that architectures utilizing reinforcement learning (RL) are effective in cost-based image steganography. However, these architectures only learn embedding probabilities rather than costs, and are trained for a specific embedding payload, making it difficult to extend the trained model to serve other payloads. In this paper, we propose a payload-independent cost learning framework using RL called PICO-RL. This framework directly learns universal costs that can be applied to any payload. PICO-RL incorporates an optimal probability approximation (OPA) module that can calculate the required probability map for embedding simulation directly from a learned cost map for any payload, eliminating the need for time-consuming searches for a valid probability scaling parameter. Additionally, PICO-RL uses an advanced steganalysis environment network to provide more effective reward feedback for learning. During RL training, the learned cost maps of different payloads converge and eventually become similar under the OPA constraint, resulting in payload independence. Experimental results demonstrate that a well-trained PICO-RL model, which acts as a universal cost function, defines costs with superior security performance against steganalysis and has better coding compatibility when encoding with practical steganographic codes.