风格迁移之图像翻译Pix2Pix

简介

生成对抗网络(Generative Adversarial Network[1], 简称GAN) 是一种非监督学习的方式，通过让两个神经网络相互博弈的方法进行学习，该方法由lan Goodfellow等人在2014年提出。生成对抗网络由一个生成网络和一个判别网络组成，生成网络从潜在的空间(latent space)中随机采样作为输入，其输出结果需要尽量模仿训练集中的真实样本。判别网络的输入为真实样本或生成网络的输出，其目的是将生成网络的输出从真实样本中尽可能的分辨出来。而生成网络则尽可能的欺骗判别网络，两个网络相互对抗，不断调整参数。生成对抗网络常用于生成以假乱真的图片。此外，该方法还被用于生成影片，三维物体模型等。

Pix2Pix利用成对的图片进行图像翻译，即输入为同一张图片的两种不同风格，可用于进行风格迁移。

下载安装命令

## CPU版本安装命令
pip install -f https://paddlepaddle.org.cn/pip/oschina/cpu paddlepaddle

## GPU版本安装命令
pip install -f https://paddlepaddle.org.cn/pip/oschina/gpu paddlepaddle-gpu

Pix2Pix由一个生成网络和一个判别网络组成。生成网络中编码部分的网络结构都是采用convolution-batch norm-ReLU作为基础结构，解码部分的网络结构由transpose convolution-batch norm-ReLU组成，判别网络基本是由convolution-norm-leaky_ReLU作为基础结构，详细的网络结构可以查看network/Pix2pix_network.py文件。生成网络提供两种可选的网络结构：Unet网络结构和普通的encoder-decoder网络结构。网络利用损失函数学习从输入图像到输出图像的映射，生成网络损失函数由GAN的损失函数和L1损失函数组成，判别网络损失函数由GAN的损失函数组成。生成器的网络结构如下图所示：风格迁移之图像翻译Pix2Pix-LMLPHP

阅读本项目之前建议阅读原版论文https://arxiv.org/abs/1611.07004 ，优秀解读博客 https://blog.csdn.net/stdcoutzyx/article/details/78820728

**本项目采用cityscapes数据集进行训练，关于cityscapes数据集的介绍 https://blog.csdn.net/zz2230633069/article/details/84591532 **

In[ ]

# 代码目录结构如下
# ├── data_reader.py 数据预处理
# │  
# ├── train.py 模型的训练入口
# │  
# ├── infer.py 模型的预测入口
# │  
# ├── trainer 不同模型的训练脚本
# │  
# ├── network 不同模型的网络结构
# │  
# ├── util 网络的基础配置和公共模块
# │  
# ├── scripts 多个模型的训练启动和测试启动示例
# │

In[ ]

#自定义数据集： 用户可以使用自定义的数据集，只要设置成所对应的生成模型所需要的数据格式即可。
# pix2pix模型数据集准备中的list文件需要通过scripts文件夹里的make_pair_data.py来生成，
# 可以使用以下命令来生成： python scripts/make_pair_data.py 
#!cd PixPix/ && python make_pair_data.py --direction=A2B

In[2]

####解压数据集
!cd data/data10830/ && unzip -qo cityscapes.zip

In[3]

#安装imageio、scipy
!pip install imageio
!pip install scipy==1.2.1

Looking in indexes: https://pypi.mirrors.ustc.edu.cn/simple/
Collecting imageio
  Downloading https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/af/0a/943c965d372dae0b1f1482677d29030ab834351a61a9a632fd62f27f1523/imageio-2.5.0-py3-none-any.whl (3.3MB)
    100% |████████████████████████████████| 3.3MB 9.0MB/s eta 0:00:01
Requirement already satisfied: numpy in /opt/conda/envs/python35-paddle120-env/lib/python3.5/site-packages (from imageio) (1.16.4)
Requirement already satisfied: pillow in /opt/conda/envs/python35-paddle120-env/lib/python3.5/site-packages (from imageio) (6.0.0)
Installing collected packages: imageio
Successfully installed imageio-2.5.0
Looking in indexes: https://pypi.mirrors.ustc.edu.cn/simple/
Collecting scipy==1.2.1
  Downloading https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f0/30/526bee2ce18c066f9ff13ba89603f6c2b96c9fd406b57a21a7ba14bf5679/scipy-1.2.1-cp35-cp35m-manylinux1_x86_64.whl (24.7MB)
    100% |████████████████████████████████| 24.7MB 1.4MB/s eta 0:00:01  9% |███                             | 2.3MB 63.2MB/s eta 0:00:01    28% |█████████▏                      | 7.1MB 458kB/s eta 0:00:39
Requirement already satisfied: numpy>=1.8.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.5/site-packages (from scipy==1.2.1) (1.16.4)
Installing collected packages: scipy
  Found existing installation: scipy 1.3.0
    Uninstalling scipy-1.3.0:
      Successfully uninstalled scipy-1.3.0
Successfully installed scipy-1.2.1

In[1]

#开始训练： 数据准备完毕后，可以通过一下方式启动训练：
!python Pix2Pix/train.py --model_net Pix2pix \
                --dataset cityscapes/cityscapes \
                --data_dir data/data10830 \
                --train_list data/data10830/cityscapes/pix2pix_train_list \
                --test_list data/data10830/cityscapes/pix2pix_test_list \
                --crop_type Random \
                --dropout True \
                --gan_mode vanilla \
                --batch_size 1 \
                --epoch=5 \
                --output Pix2Pix/output \
                --use_gpu True

start
-----------  Configuration Arguments -----------
batch_size: 1
crop_size: 256
crop_type: Random
d_base_dims: 64
d_nlayers: 3
data_dir: data/data10830
dataset: cityscapes/cityscapes
drop_last: False
dropout: 1
epoch: 5
g_base_dims: 64
gan_mode: vanilla
init_model: None
lambda_L1: 100.0
learning_rate: 0.0002
load_size: 286
model_net: Pix2pix
net_D: basic
net_G: unet_256
norm_type: batch_norm
num_discriminator_time: 1
num_generator_time: 1
output: Pix2Pix/output
print_freq: 10
profile: False
run_test: False
save_checkpoints: False
shuffle: True
test_list: data/data10830/cityscapes/pix2pix_test_list
train_list: data/data10830/cityscapes/pix2pix_train_list
use_gpu: 1
------------------------------------------------
start
data/data10830/cityscapes/cityscapes data/data10830/cityscapes/pix2pix_train_list
W0905 17:30:22.763990   160 device_context.cc:259] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0
W0905 17:30:22.767936   160 device_context.cc:267] device: 0, cuDNN Version: 7.3.
WARNING:root:
     You can try our memory optimize feature to save your memory usage:
         # create a build_strategy variable to set memory optimize option
         build_strategy = compiler.BuildStrategy()
         build_strategy.enable_inplace = True
         build_strategy.memory_optimize = True

         # pass the build_strategy to with_data_parallel API
         compiled_prog = compiler.CompiledProgram(main).with_data_parallel(
             loss_name=loss.name, build_strategy=build_strategy)

     !!! Memory optimize is our experimental feature !!!
         some variables may be removed/reused internal to save memory usage,
         in order to fetch the right value of the fetch_list, please set the
         persistable property to true for each variable in fetch_list

         # Sample
         conv1 = fluid.layers.conv2d(data, 4, 5, 1, act=None)
         # if you need to fetch conv1, then:
         conv1.persistable = True


WARNING:root:
     You can try our memory optimize feature to save your memory usage:
         # create a build_strategy variable to set memory optimize option
         build_strategy = compiler.BuildStrategy()
         build_strategy.enable_inplace = True
         build_strategy.memory_optimize = True

         # pass the build_strategy to with_data_parallel API
         compiled_prog = compiler.CompiledProgram(main).with_data_parallel(
             loss_name=loss.name, build_strategy=build_strategy)

     !!! Memory optimize is our experimental feature !!!
         some variables may be removed/reused internal to save memory usage,
         in order to fetch the right value of the fetch_list, please set the
         persistable property to true for each variable in fetch_list

         # Sample
         conv1 = fluid.layers.conv2d(data, 4, 5, 1, act=None)
         # if you need to fetch conv1, then:
         conv1.persistable = True


I0905 17:30:22.835135   160 parallel_executor.cc:329] The number of CUDAPlace, which is used in ParallelExecutor, is 1. And the Program will be copied 1 copies
I0905 17:30:22.844496   160 build_strategy.cc:340] SeqOnlyAllReduceOps:0, num_trainers:1
I0905 17:30:24.220062   160 parallel_executor.cc:329] The number of CUDAPlace, which is used in ParallelExecutor, is 1. And the Program will be copied 1 copies
I0905 17:30:24.225023   160 build_strategy.cc:340] SeqOnlyAllReduceOps:0, num_trainers:1
epoch0: batch0:
                         g_loss_gan: 0.6289220452308655; g_loss_l1: 43.45980453491211;
                         d_loss_real: 0.6388400793075562; d_loss_fake: 1.0123474597930908;
                         Batch_time_cost: 1.41

In[10]

#执行以下命令使用训练5个epoch后固化后的模型进行预测,预测结果保存在Pix2Pix/infer_result中
!python Pix2Pix/freeze_infer.py --output Pix2Pix/infer_result \
                                --use_gpu True \
                                --dataset_dir="data/data10830/cityscapes/cityscapes/testA/380_A.jpg"


##########结果可视化
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import cv2

img= cv2.imread('data/data10830/cityscapes/cityscapes/testA/380_A.jpg')

result_img= cv2.imread('Pix2Pix/infer_result/fake_380_A.jpg')


plt.subplot(1, 2, 1)
plt.imshow(img)
plt.subplot(1, 2, 2)
plt.imshow(result_img)
plt.show()

-----------  Configuration Arguments -----------
dataset_dir: data/data10830/cityscapes/cityscapes/testA/380_A.jpg
output: Pix2Pix/infer_result
test_list: data/data10830/cityscapes/pix2pix_test_list
use_gpu: 1
------------------------------------------------
W0905 17:20:00.202416   809 device_context.cc:259] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0
W0905 17:20:00.206357   809 device_context.cc:267] device: 0, cuDNN Version: 7.3.
read data/data10830/cityscapes/cityscapes/testA/380_A.jpg

使用AI Studio一键上手实践项目吧:https://aistudio.baidu.com/aistudio/projectdetail/122272

下载安装命令

## CPU版本安装命令
pip install -f https://paddlepaddle.org.cn/pip/oschina/cpu paddlepaddle

## GPU版本安装命令
pip install -f https://paddlepaddle.org.cn/pip/oschina/gpu paddlepaddle-gpu

>> 访问 PaddlePaddle 官网，了解更多相关内容。

飞桨PaddlePaddle

风格迁移之图像翻译Pix2Pix