This paper presents a novel approach for the generation of future image frames using optical flow. Optical flow is a vector field that calculates velocity of each pixel in both axes and shows direction and magnitude of vector field as intensity changes from one image to other image.A separate Adaptive Network based Fuzzy Inference System (ANFIS) model is used to model each pixel's velocity in both directions. This network predicts future velocities of each pixel and then pixels are mapped to their new position. The resulting scheme has been applied successfully on an image sequence of landing fighter plane. Canny edge detection based image comparison (CIM) and Mean Structural Similarity Measurement Index (MSSIM) is used to evaluate future image frame quality. The proposed approach is found to generate future frame up to ten frames successfully.
Keywords- Future Image Frames, Optical Flow, Adaptive Network Based Fuzzy Inference System.
INTRODUCTION
The objective of this study is to generate future image frame of an image sequence using optical flow. The generation of future image frame use past information of an image sequence.This advance knowledge is helpful in weather forecasting, robotic motion, defence applications like early prediction of target trajectory and numerous other field of strategic interest. Some preliminary work has already been done in this direction using Artificial Neural Networks [6],[8],[10] and Fuzzy models [5],[9].Many earlier attempts have successfully made use of adaptive network based fuzzy inference system for time series prediction [11],[12],[13].The efficient optical flow method proposed for generating future frames of the given image sequence. Optical flow [1],[14] calculates velocity of each pixel in both direction x and y. Thus, two time series of velocity of each pixels are constructed for x and y. For both x and y two different ANFIS model is used to train velocities of pixel's and the future velocities of each pixel is predicted. Then pixels of predicted image are mapped to their new value.
In this paper, images of fighter aircraft landing on an aircraft carrier are used for implementation of the model. The ANFIS model with optical flow can generate future image frame of fighter plane with good accuracy. Also to evaluate quality of the generated image frames and corresponding actual image frame with the help of two most widely used image quality assessment indices, canny edge detection based image comparison(CIM) [2],[4] and Mean Structural Similarity Measurement Index (MSSIM) [3] are used.
The rest of the paper is arranged as follows: section II, explains the estimation of optical flow. Section III explains the formulation of ANFIS model for prediction of time series. Section IV discusses algorithm for generation of future image frames. Section V performance evaluation of generated future image frames is discussed. Section VI discussed the obtain results. Finally, conclusions are drawn in Section VII.
OPTICAL FLOW
Optical flow [1],[14] is the motion of brightness patterns in time varying image sequence. It estimates the 2D-projection of the 3D-real world motion. Optical flow is used to find the motion of the pixels of an image sequence by calcuting velocity of each pixel in both directions. It provides a point to point pixel relation.
Optical flow finds wide application area in robotics which computes optical flow for the image sequence that comes through the robot's visual sensor helps the moving robot's navigation in space and avoids obstacles. Another application is in video compression algorithm for medical imaging.
The Horn-Schunck algorithm (HS) algorithm is one of best algorithm in optical flow due to its good performance and simplicity. The HS algorithm is used to identify the image velocity or motion vector based on Spatial Temporal Gradient technique which computes the image velocity from spatiotemporal derivatives of image intensity.The HS method is a global method that introduces a global constraint of smoothness over the computed flow field. The energy function that they construct is
whereanddenotes the partial derivatives of with respect,and respectively,, and parameter is regularisation constant. In each image sequence ,and can be calculated for each pixel as:
The energy function can be solved with the help of calculus of variations,
Where is the Laplace operator. The Laplacian using finite difference,where weighted average of is calculated in neighbourhood of .Thus equation (5) and equation (6) can be written as:
after solving equation (7) and equation (8), calculate velocity u and v for each pixel in image.
III.ANFIS MODEL FOR FUTURE IMAGE FRAME GENERATION
After calculating velocity of each pixel in image sequence using Horn-Schunck optical flow algorithm then determine the input-output parameter using "genfis2" function Fuzzy Logic Toolbox to generate fuzzy inference system (FIS) using Subtractive Clustering [15] .
Subtractive Clustering
"genfis2" function generates a Sugeno-type FIS structure using subtractive clustering. Consider a collection of m data points in an N-dimension space. The algorithm assumes each data point is a potential cluster center and calculates some measure of potential for each of then according to equation (11)
where and define the neighborhood radius for each cluster center. After calculating potential for each vector, the one with the highest potential is selected as the first cluster center. Let be the center of the first group and it's potential. Then the potential for each is reduced according to equation (12)
Also, and represent radius of neighborhood for which considerable potential reduction will happen.is regularly chosen to avoid obtaining closely space cluster centers. After clustering, clusters' information is used for determining the initial number of rules and antecedent membership function that is used for identifying the fuzzy inference system.
Adaptive Network Based Fuzzy Inference System(ANFIS)
ANFIS is combination of both the learning capabilities of artificial neural network and reasoning capabilities of fuzzy logic. ANFIS is based on Takagi-Sugeno fuzzy inference system (FIS). The basic structure of FIS is collection of three component (i) rule based which contains set of IF-THEN fuzzy rule (ii) database has membership function of inputs which is used for fuzzy rule and (iii) reasoning mechanism performs inference procedure upon fuzzy rule and derive reasonable output.
ANFIS has been widely used as time series forecaster. Typically this approach has been used for Forecasting financial time series [16], Predicting Chaotic Time Series [12] and hydrological time series prediction [11],[13] .
We assumed fuzzy inference system has two inputs x, y and an output. Fuzzy inference system contains two input rules of Takagi and Sugeno's type [7] .
Hereare linear parameters an are non-linear parameters. The ANFIS architecture is shown in Figure 1, where the nodes of same layer have same function. All the five layers are described:
Layer 1: The output of each node is:
So, is essentially the membership grade for and.The membership functions could be anything but for illustration purposes we will use the bell shaped function given by: where are parameters to be learnt. These are the premise parameters.
Layer 2: Every node in this layer is fixed. This is where the t-norm is used to 'AND' the membership grades - for example the product:
Layer 3: This layer contains fixed nodes which calculate the ratio of the firing strengths of the rules: Layer 4: The nodes in this layer are adaptive and perform the consequent of the rules:
The parameters in this layer () are to be determined and are referred to as the consequent parameters.
Layer 5: There is a single node here that computes the overall output:
The architecture of ANFIS model shows input selection, membership function and number of rules.
The velocity of each pixel which has been calculated by HS optical flow algorithm, trained with ANFIS model. This network generates future velocity of each pixel and then mapped pixel to its new values and then future images are generated
x
Layer 3
Layer 2
Layer 1
Output
Layer 5
y
Input 1
Input 2
x
y
A1
Layer 4
y
A2
B1
B2
∑
x
Figure : Five Layered ANFIS Architecture
.
IV. ALGORITHM FOR GENERATING FUTURE IMAGE FRAME
Suppose there are N image frames in an image sequence. Then image sequence set is divided into part training and testing. Let us suppose first M frames for training (training set) and the remaining N-M frames for testing ( testing set), then the training set is used to develop the model and the remaining set i.e. Testing set for validation.Performance evaluation is evaluated using CIM and MSSIM.
The Algorithm
Step-by-step algorithm for predicting the future images of an image sequence:
Step 1: Represent sequence of image frame f(t) t=1,2… N .First resizes the image frame and then converts it from RGB to Gray scale.
Step 2: Calculate the velocities of each pixel in both axes, between all pairs of consecutive image frame using Horn-Schunck algorithm. Two time series for x and y axes of each pixel are constructed.
Step 3: Create two separate ANFIS model for x and y of each pixel and train them separately using the time series obtained from the previous step.
Step 4: Give input from last k frame (here k=4) into the trained adaptive network based fuzzy inference system and get the predicted future velocities of the pixels and map them to their new positions.
Step 5: Apply canny edge detection algorithm on the test and the predict future image frame for comparison and also MSSIM indices for performance evaluation of the system.
Implementation Scheme
The video clip for which the future frames are required is first converted into a set of its image frames. These image frames are resize into 192108 and then converted RGB to Gray image frames. These frames are then divided into training and testing set.Horn-Schunck optical flow algorithm is used for calculating velocity of each pixel in both axes. Time series of the velocity is constructed for each pixel for both dimensions.In this case 4 input variables for x-direction to train ANFIS model were considered; X(t-4), X(t-3), X(t-2) and X(t-1) and one output variable i.e. X(t) are chosen for first model. Similarly, for the second ANFIS model, input variables Y(t-4), Y(t-3), Y(t-2) and Y(t-1) and output variable Y(t) are chosen.In this way ANFIS model is trained for each and every pixel's velocity in both direction. Now give input from last frame k (k=4) to trained ANFIS and get predicted future velocities of each pixel and then pixels are mapped to their new positions and generate future image frame.
V. PERFORMATION EVALUATION
Canny based image comparison metric (CIM) and mean structural similarity (MSSIM) have been chosen for the evaluating structural similarities between predicted image frames and test image frames. CIM and MSSIM matric is chosen due to CIM index is invariant to rotation, scaling and translation within a reasonable tolerance and MSSIM has been stated to be better structural descriptor [3] for image frames. There are also other similarity assessments metrics like Peak Signal to Noise Ratio (PSNR), Picture Quality Scale (PQS) , Noise Quality Measure (NQM), Information Fidelity Criterion (IFC), and Visual information fidelity (VIF) perform well on gray scale image frames.CIM and MSSIM Indices are computed for gray scale image and successfully calculate similarity between predicted image frames and test image
VI. RESULT AND DISCUSSION
For experimental validation, we have extracted an image sequence of landing fighter plane over sea where there is camera and fighter plane both moved. The background changes due to camera motion and size of fighter changes due to zooming effect. The image sequence of 279 images is extracted from the video. First 239 images are kept for training and next sequentially ten images (239-240) in test set. The original images were resized to 192108 pixels and converted to gray scale.239 images were used to calculate velocities of each pixel in both axes x and y using Horn-Schunck optical flow algorithm. Time series is constructed for both axes x and y for each pixel. A separate ANFIS model is used for training to each pixel's velocity in both direction and the next 10 images were attempted to be predicted.Figure 2 shows ten test and predicted images using ANFIS model. Table 1 and Figure 5 shows CIM and MSSIM index for ten predicted image.
Figure3: First Test Image Figure13:First Predicted Image using ANFIS(genfis3) Figure24:First Predicted Image using ANFIS (genfis2)
Test Image 240 Test Image 241 Test Image 242 Test Image 243 Test Image 244
Test Image 245 Test Image 246 Test Image 247 Test Image 248 Test Image 249
Predicted Image 240 Predicted Image 241 Predicted Image 242 Predicted Image 243 Predicted Image 244
Predicted Image 245 Predicted Image 246 Predicted Image 247 Predicted Image 248 Predicted Image 249
Figure 2 : Ten Test and Predicted images for fighter plane row wise 240, 241, 242, 243, 244, 245, 246, 247, 248 and 249
Next we skip one image between each two frame from 239 images. Now 120 images are left for training and next ten image frames (240th, 242th, 244th, 246th, 248th, 250th, 252th, 254th, 256th and 258th) are in test set and using optical flow concept and ANFIS model predict ten future images. Figure 3 shows ten test images and ten predicted images. Table 2 and Figure 6 shows CIM and MSSIM index for all ten predicted images. Next we skip two images between each two frame from 239 images. Now 80 images are left for training and next
ten image frames (242th, 245th, 248th, 251th, 254th, 257th, 260th, 263th, 266th and 269th) are in test set and using optical flow concept and ANFIS model predict ten future images. Figure 4 shows ten test images and ten predicted images. Table 3 and Figure 7 shows CIM and MSSIM index for predicted images.Same way skip three image between each two frames and 62 frames are in training and next 10 image frames(243th, 247th, 251th, 255th, 259th, 263th, 267th, 271th, 275th and 279th) are predicted using ANFIS.Table 4 and figure 8 shows CIM and MSSIM index for predicted image.
Test image 241 Test Image 243 Test Image 245 Test Image 247 Test Image 249
Test image 251 Test Image 253 Test Image 255 Test Image 257 Test Image 259
Predicted image 241 Predicted Image 243 Predicted Image 245 Predicted Image 247 Predicted Image 249
Predicted image 251 Predicted Image 253 Predicted Image 255 Predicted Image 257 Predicted Image 259
Figure 3: Ten Test and Predicted images for fighter plane row wise 241, 243, 245, 247, 249, 251, 253, 255, 257, and 259 (skip one image)
Test image 242 Test Image 245 Test Image 248 Test Image 251 Test Image 254
Test image 257 Test Image 260 Test Image 263 Test Image 266 Test Image 269
Predicted image 242 Predicted Image 245 Predicted Image 248 Predicted Image 251 Predicted Image 254
Predicted Image 257 Predicted Image 260 Predicted Image 263 Predicted Image 266 Predicted Image 269
Figure 4: Ten Test and Predicted image for fighter plane row wise 242, 245, 248, 251, 254, 257, 260, 263, 266, and 269 (after skip two images )
VII. Conclusion
The performance of the proposed algorithm for an image sequence of a landing over sea has been studied. Ten future images of the image sequence have been generated. Optical flow algorithm for image generation has been simulated and analyzed on the basis of quality of generated image frames fighter plane. We have predict number of frames and improve performance of predicted image .This area has lot of scope for future research for better prediction.
Table1: CIM and MSSIM index for predicted image frames (240-249) Table2: CIM and MSSIM index for predicted image frames (241-259)
Image
no
ANFIS
Predicted Frame
CIM
MSSIM
241
0.9811
0.9175
243
0.9449
0.4844
245
0.9556
0.7843
247
0.9337
0.6433
249
0.9390
0.6036
251
0.9373
0.5601
253
0.9381
0.5348
255
0.9359
0.5252
257
0.9622
0.5137
259
0.9587
0.5029
Image
no
ANFIS
Predicted Frame
CIM
MSSIM
240
0.9472
0.8881
241
0.9659
0.4879
242
0.9819
0.7666
243
0.9770
0.6549
244
0.9823
0.6150
245
0.9623
0.5629
246
0.9647
0.5564
247
0.9672
0.5417
248
0.9748
0.5257
249
0.9804
0.5115
Image
no
ANFIS
Frame name
CIM
MSSIM
243
0.9997
0.7612
247
0.8636
0.2909
251
0.8917
0.6081
255
0.8451
0.5465
259
0.8448
0.4979
263
0.8378
0.4591
267
0.8335
0.4469
271
0.8452
0.4029
275
0.8214
0.3714
279
0.8393
0.3335
Image
no
ANFIS
Predicted Frame
CIM
MSSIM
242
0.9887
0.8549
245
0.9644
0.4570
248
0.9588
0.6688
251
0.9681
0.6194
254
0.9683
0.5889
257
0.9607
0.5654
260
0.9501
0.5470
263
0.9641
0.5240
266
0.9406
0.5059
269
0.9524
0.4863
Table3: CIM and MSSIM index for predicted image frames (242-269) Table4: CIM and MSSIM index for predicted image frames (243-279)
Figure 5 : CIM and MSSIM index for ten predicted images(240-249) Figure 6: CIM and MSSIM index for ten predicted images frames (241-259)
Figure 7: CIM and MSSIM index for ten predicted images(240-249) Figure 8 : CIM and MSSIM index for ten predicted images frames (241-259)
