Content Swapping: A New Image Synthesis for Construction Sign Detection in Autonomous Vehicles

. 2022 May 4;22(9):3494. doi: 10.3390/s22093494

Algorithm 1 Pseudo-code of the proposed method.

Step1: Collecting construction sign and road images

Step2: Labeling bounding box, segment, and four corners of the board

4:
Bounding box labels: $B b o x_{I n}$ , $B b o x_{O u t}$
5:
Segment labels: $M_{I n}$ , $M_{O u t}$
6:
Four corners of the board: $([\begin{matrix} x_{I}^{1} & y_{I}^{1} \end{matrix}] [\begin{matrix} x_{I}^{2} & y_{I}^{2} \end{matrix}] [\begin{matrix} x_{I}^{3} & y_{I}^{3} \end{matrix}] [\begin{matrix} x_{I}^{4} & y_{I}^{4} \end{matrix}])$ , $([\begin{matrix} x_{O}^{1} & y_{O}^{1} \end{matrix}] [\begin{matrix} x_{O}^{2} & y_{O}^{2} \end{matrix}] [\begin{matrix} x_{O}^{3} & y_{O}^{3} \end{matrix}] [\begin{matrix} x_{O}^{4} & y_{O}^{4} \end{matrix}])$

Step3: Content swapping

7:
Randomly select content image (source): $S \in I n$
8:
Randomly select frame image (target): $T \in [\begin{matrix} I n & O u t \end{matrix}]$
9:
Set content region mask of target image using four corners label: $C_{T}$
10:
Compute transformation matrix $T$ : ▹ Section 3.4

$T = {[\begin{matrix} x_{S}^{1} & y_{S}^{1} & 1 & 0 & 0 & 0 & - x_{S}^{1} x_{T}^{1} & - y_{S}^{1} x_{T}^{1} \\ 0 & 0 & 0 & x_{S}^{1} & y_{S}^{1} & 1 & - x_{S}^{1} y_{T}^{1} & - y_{S}^{1} y_{T}^{1} \\ x_{S}^{2} & y_{S}^{2} & 1 & 0 & 0 & 0 & - x_{S}^{2} x_{T}^{2} & - y_{S}^{2} x_{T}^{2} \\ 0 & 0 & 0 & x_{S}^{2} & y_{S}^{2} & 1 & - x_{S}^{2} y_{T}^{2} & - y_{S}^{2} y_{T}^{2} \\ x_{S}^{3} & y_{S}^{3} & 1 & 0 & 0 & 0 & - x_{S}^{3} x_{T}^{3} & - y_{S}^{3} x_{T}^{3} \\ 0 & 0 & 0 & x_{S}^{3} & y_{S}^{3} & 1 & - x_{S}^{3} y_{T}^{3} & - y_{S}^{3} y_{T}^{3} \\ x_{S}^{4} & y_{S}^{4} & 1 & 0 & 0 & 0 & - x_{S}^{4} x_{T}^{4} & - y_{S}^{4} x_{T}^{4} \\ 0 & 0 & 0 & x_{S}^{4} & y_{S}^{4} & 1 & - x_{S}^{4} y_{T}^{4} & - y_{S}^{4} y_{T}^{4} \end{matrix}]}^{- 1} [\begin{matrix} x_{T}^{1} \\ y_{T}^{1} \\ x_{T}^{2} \\ y_{T}^{2} \\ x_{T}^{3} \\ y_{T}^{3} \\ x_{T}^{4} \\ y_{T}^{4} \end{matrix}]$
11:
Swap content: $I_{T}^{C S} = T (I_{S}) ⊙ C_{T} + I_{T} ⊙ (1 - C_{T})$

Step4: Cut-and-paste with realistic transformations

12:
Randomly select road image (background): $B \in R o a d$
13:
Compute pasteable region: $P_{B}$ ▹ Section 3.5.1
14:
Randomly select bottom point of the sign: $p_{1} = [\begin{matrix} p_{1}^{x} & p_{1}^{y} \end{matrix}] \in P_{B}$
15:
Compute top point of the sign: ▹ Section 3.5.2

$p_{2} = [\begin{matrix} p_{2}^{x} & p_{2}^{y} \end{matrix}] = [\begin{matrix} p_{1}^{x} & ({tan}^{- 1} (\frac{tan (α \cdot p_{1}^{y} + β)}{1 - h}) - β) / α \end{matrix}]$
16:
Cut sign image $I_{T}^{C S}$ and paste to road image $I_{B}$ : $I_{T}^{C P} = Cut - and - Paste (I_{T}^{C S}, M_{T}, I_{B}, p_{1}, p_{2})$
17:
Transform segment label to $p_{1}$ and $p_{2}$ : $M_{T}^{C P} = Transform_mask (M_{T}, p_{1}, p_{2})$
18:
Transform bounding box label to $p_{1}$ and $p_{2}$ : $B b o x_{T}^{C P} = Transform_box (B b o x_{T}, p_{1}, p_{2})$
19:
Reduce color difference: $I_{T}^{G P} = GP - GAN (I_{T}^{C S}, M_{T}^{C S})$ ▹ Section 3.5.3

Output: Synthesized training image: $I_{T}^{G P}$ ;

Synthesized training label: $B b o x_{T}^{C P}$