Deep convolutional long short-term memory for forecasting wind speed and direction

This paper proposed deep learning to create an accurate forecasting system that uses a deep convolutional long short-term memory (DCLSTM) for forecasting wind speed and direction. In order to use the DCLSTM system, wind speed and direction are represented as an image in 2D coordinates and make it to time sequence data. The wind speed and direction data were obtained from AMeDAS (Automated Meteorological Data Acquisition System), Japan. The target of the proposed forecasting system was to improve forecasting accuracy compared to the system in SICE 2020 (The Society of Instrument and Control Engineers Annual Conference 2020) in all seasons. For verifying the efficiency of the forecasting system by comparison with persistent system, deep fully connected-LSTM (DFC-LSTM) and encoding-forecasting network with convolutional long short-term memory (CLSTM) systems were investigated. Forecasting performance of the system was evaluated by RMSE (root mean square error) between forecasted and measured data.


Introduction
Currently, almost all countries in the world exploit renewable energy sources to reduce dependencies on fossil fuels, which can produce 26.2% of global electricity in the world in 2019. In 2018, the total capacity for producing renewable energy increased up to 52% in Asia. The wind power source is a kind of renewable energy that is rapidly attracting attention as an alternative source for solving electricity demand. Wind power generation contributed around 28% of new energy additions in the world in 2018. The advantage of wind power is being clean, reliable, most abundant, and affordable as part of the renewable energy resources, with having the potential for supply energy and being able to increase producing energy, significantly in the future [1,2]. However, the principal problem of wind power is the wide fluctuation of output caused by a change in wind speed and direction. Thus, forecasting wind speed and direction is needed for regulating wind power generation effectively and can offer information for power companies which can assist in stabilizing the electrical power system and organizing the operation of the thermal power plant [3,4].
There are three main methods of wind speed prediction: numerical, statistical, and machine learning. The numerical method such as NWP (numerical weather prediction) is suitable for predicting full data on atmospheric and weather, which requires a long time to run the program and high computation costs [4,5]. The statistical methods such as autoregressive integral moving average (ARIMA) and Markov chain adjust different parameters of the model between measured and forecasted data [4][5][6][7]. Machine learning based on the neural network such as ANN (artificial neural network), RNN (recurrent neural network) can predict the future wind speed by representing the complex nonlinear connections between output and input data [5][6][7]. This paper proposed forecasting wind speed and direction system using deep convolutional long shortterm memory (DCLSTM). The DCLSTM system is suggested to extend and improve the encodingforecasting network with the convolutional long shortterm memory (CLSTM) system in SICE 2020 which uses a more number of filter, deep CLSTM layer, and dataset of a longer period. DCLSTM can extract spatiotemporal feature maps of wind speed and direction and evaluate time sequence image data. The prediction period of the forecasting system is one-hour ahead that can improve accuracy compared to the system in SICE 2020 and shows good performance by comparing persistent system, DFC-LSTM, and encodingforecasting network with CLSTM system. Performance of the system is evaluated by RMSE between measured and forecasted data for four seasons in 1 year.

Explanation of wind speed and direction
The wind speed and direction (wind data) in Tokushima city, Japan were taken from AMeDAS at 10 min interval. The wind data can be visualized on the coordinate system by graphic expression as shown in Figure 1. Both x-axis and y-axis represent the direction of E (east) -W (west) and the direction of N (north) -S (south), respectively. Wind direction of AMeDAS is divided into 16 as shown in Figure 2. Equations (1) and (2) show x and y components of wind vector, v x (t) where θ(t) means wind direction [ • ] and v(t) means wind speed [m/s].

Dataset
In this paper, wind data for four years were used: two years for training, 1 year for validation, and 1 year for test as shown in Table 1. The test dataset was divided into seasons to validate seasonal characteristics. The size of output and input images were 128 × 128 pixel. The maximum wind speed during the training dataset was slightly below 20 m/s. Therefore, the scale of plotting image by PIL (python imaging library) was set as follows, where p x and p y are the plotted point of wind data for x-axis and y-axis, respectively and 64 means half of the image size.
For expressing a change of wind data, six points were plotted on one image im(t) from p(t − 5) to p(t) and each point was connected by the line that describes 1h data as shown in Figure 3. Moreover, the point of the newest data in the image was plotted as the larger point to express the time sequence. Furthermore, nine time-series images from im(t − 8) to im(t) were used to the input of the network to express the change of wind vector transition.

DFC-LSTM
RNN (recurrent neural network) is a kind of neural network that can solve time-series data but has the principal problem about vanishing gradient. LSTM is an advanced type of RNN that effectively solves the vanishing gradient problem due to LSTM trained by backpropagation (BP) algorithm and has cell state and gates for controlling information flow, improving the capability of RNN, making it easy to converge, and running faster than RNN. LSTM is powerful in solving long-range dependency and handles for learning short and long-term dependencies very well [8][9][10][11]. LSTM is a popular and good performance tool for solving time-series data and can be applied to caption generation [12], medical diagnostic [13], wind prediction [14], scene label [15], speech recognition [16]. The inner architecture of LSTM has three gates: input (i), forget (f ), and output (o) gates; input vector (X); hidden state (H); cell state (C); and activation function (σ ) as shown in Figure 4 [8,9]. LSTM unit is calculated by where • means hadamard product, and W means weight [4,11,17]. Deep fully connected-LSTM (DFC-LSTM), which is a multivariate type of LSTM that uses multiple LSTM layers, was applied in this study as it is easy to learn with the FC layer, and learns temporal features [18,19]. DFC-LSTM is composed of five LSTM layers and one FC layer. All of the LSTM layers have 32 cells. The end process of the DFC-LSTM system uses the FC layer for producing the forecasted results.

DCLSTM
DCLSTM (deep convolutional LSTM) system is a developed CLSTM system. The system was proposed in this study to improve the capability of CLSTM system. CLSTM system combines CNN (convolutional neural network) and LSTM to improve the solving capability of sequential images. CLSTM is known for being able to learn representation from spatiotemporal features and learning processes by sequence to sequence that uses the input of the image [4,11,19]. Figure 5 shows the inner structure of CLSTM that uses convolutional structure in state-to-state and inputto-state transitions [4,11]. The CLSTM layer produces the image of the same size as the input. CLSTM layer features are arranged based on five dimensions: number of sample data (N), time-series (ts), height (h), width (w), and channel (c). In CLSTM, the matrix product "·" of the LSTM is a substituted convolutional operation " * ". The equation of CLSTM is shown in [4,11,17,19].
This paper proposes a forecasting system using multiple CLSTM layers, also known as deep CLSTM (DCLSTM) that have five CLSTM layers in the encoder and forecaster network, respectively, as shown in Figure 6. The encoder network is composed of one Conv3D layer, one max-pooling3D layer, and five CLSTM2D layers. The forecaster network is composed of five CLSTM2D layers, one up-sampling3D, and one Deconv3D layer. The kernel size of Conv., Deconv., and CLSTM layers use five. The stride of max-pooling uses two. The channel size of CLSTM and Conv. are 32 and 16.
In the encoder network, height and width of feature maps are downsampled by max-pooling. On the contrary, the feature maps are upsampled in the forecaster network by up-sampling layer. In the forecaster network, fore input is required to output forecast results which are filled by zeros. The state of the CLSTM layer in the forecaster network is copied from the last state of the CLSTM layer in the encoder network. In the forecaster network, the final process generates the output image with one Deconv. layer by concatenating all states of CLSTM layers to get forecasted output.

Learning procedure and parameters
Parameters of the proposed forecasting system with DCLSTM are listed in Table 2. The training and validation processes were iterated for 20 epochs. To train the system, RMSProp (root mean square propagation) was used as an optimizer with parameters ρ (decay factor) and lr (learning rate). The activation function of the forecasting system is Leaky ReLU (leaky rectified linear unit) and its parameter σ (x) is, where x is input data. The performance of proposed forecasting systems was evaluated by RMSE as follows, whereŷ q is qth forecasted data, y q is qth measured data, and N means number of sample data. The forecasting system is written by python with framework Keras and Tensorflow as backend.

Forecasting results
The forecasted result of wind speed and direction was taken out from a forecasted image by calculating the centre of gravity (CoG) position (p x ,p y ) of the biggest pixel cluster that has value zero. Thenv x (t) andv y (t) was obtained from CoG positionp y andp x by the following formula.
The forecasted wind speedv(t) and directionθ(t) are converted fromv x (t) andv y (t), whereθ(t) is forecasted wind direction [ • ] andv(t) is forecasted wind speed (m/s). The target of the proposed forecasting system predicts an image 1 h ahead. The forecasted results of wind speed and direction one-day data on December 06, 2016, are shown in Figures 7 and 8, respectively. In Figures 7 and 8, E-FN CLSTM is a forecasting system of encoding-forecasting network with CLSTM (SICE 2020). From Figures 7 and 8, the DFC-LSTM system happens some delay both in wind speed and direction with a quick change. On the other hands, the network which uses the encoding-forecasting network with CLSTM and DCLSTM can decrease delay effectively. Moreover, the forecasting result of wind speed and direction in the DCLSTM system is approaching of measured data that confirm DCLSTM can efficiently extract spatiotemporal feature maps.
The prediction error (RMSE) of wind speed and direction to all seasons, which is the main proposed prediction system of the encoding-forecasting network with CLSTM in SICE 2020 from March 2016-February 2017 learned by 1-year training data shown in Table 3. Table 4 shows the prediction error (RMSE) of wind speed and direction to all seasons and improvement rates by the persistent system from March 2016 to  February 2017 learned by two years training data. The persistent system is the standard system for short-time forecasting which outputs the current value as forecasting output. From Table 3, the best system in SICE 2020 is the encoding-forecasting network with CLSTM. We extend and improve the encoding-forecasting network with CLSTM, use the DCLSTM system. In Table 4, an encoding-forecasting network with CLSTM compares the proposed system for two years of training data which are used to investigate the best system and improve forecasting accuracy. From Table 4, the DCLSTM system can improve the forecasting accuracy of the encoding-forecasting network with CLSTM both in wind speed and direction effectively, it confirms the DCLSTM system as the highest forecasting accuracy and the best forecasting system in proposed systems. The effectiveness of the DCLSTM system is confirmed by comparing persistent, DFC-LSTM, and the encoding-forecasting network with CLSTM systems. From Table 4, DCLSTM can improve accuracy more than twice of DFC-LSTM by comparing the persistent system in most of all seasons. RMSE of forecasting systems each month in 1 year are shown in Figures 9 and 10. The forecasting accuracy of the DCLSTM   system is the highest in all months and it indicates that DCLSTM is the best forecasting system.

Conclusions
This paper proposed a DCLSTM system for wind speed and direction forecasting one-hour ahead. To confirm the effectiveness of the forecasting systems, RMSE was used by comparing persistent, DFC-LSTM, and encoding-forecasting network with CLSTM systems. The forecasted result of the DCLSTM system is better than the other systems. The DCLSTM system can improve the forecasting accuracy of the DFC-LSTM system, in which encoding-forecasting network with  CLSTM system and all systems in SICE 2020 indicates that DCLSTM is the best forecasting system. In the comparison of the persistent system, the DCLSTM system can improve forecasting accuracy drastically than the DFC-LSTM system.

Disclosure statement
No potential conflict of interest was reported by the author(s). Takashi Yasuno He received his Ph.D. degree from Tokushima University, Japan, in 1998. He is a Professor in the Faculty of Science and Technology, Tokushima University since 2013. His current research interest includes intelligent control of autonomous mobile robots, output prediction of wind power generation system, control engineering of rehabilitation system, agriculture support system (advanced motion control, intelligent control, robotics). He is a member of the Society of Instrument and Control Engineers (SICE), the Institute of Electrical Engineers of Japan (IEEJ), the Institute of Systems, Control and Information Engineers, the Robotics Society of Japan, and the Institute of Electrical and Electronics Engineers (IEEE).