Development of a PSO-ANN Model for Rainfall-Runoff Response in Basins , Case Study : Karaj Basin

Successful daily river flow forecasting is necessary in water resources planning and management. A reliable rainfallrunoff model can provide useful information for water resources planning and management. In this study, particle swarm optimization algorithm (PSO) as a metaheuristic approach is employed to train artificial neural network (ANN). The proposed PSO-ANN model is applied to simulate the rainfall runoff process in Karaj River for one and two days ahead. In this regard, different combinations of the input variables including flow and rainfall time series in previous days have been taken under consideration in order to obtain the best model's performances. To evaluate efficiency of the PSO algorithm in training ANNs, separate ANN models are developed using Levenberg-Marquardt (LM) training algorithm and the results are compared with those of the PSO-ANN models. The comparison reveals superiority of the PSO algorithm than the LM algorithm in training the ANN models. The best model for 1 and 2 days ahead runoff forecasting has R2 of 0.88 and 0.78. Results of this study shows that a reliable prediction of runoff in 1 and 2 days ahead can be achieved using PSO-ANN model. Overall, results of this study revealed that an acceptable prediction of the runoff up to two days ahead can be achieved by applying the PSO-ANN model.


Introduction
Iran is located in an arid and semi-arid region in which water shortage in many parts of this country reached an emergency or even worse conditions.During recent years, several major bodies of water in the region have been dried up, including the Orumieh Lake, Zayandehrood River and Hamoon Lagoon.Tehran, Iran's capital is facing a critical shortage of water.Regarding these problems, applying integrated water resources planning and management as well as taking efficient policies is crucial and essential.Also, successful river flow forecasting is necessary in water resources planning and management.Karaj River is one of the most important rivers in which supplies part of water of Tehran and Alborz Province [1].As a result, development of accurate models to simulate the rainfall -runoff processing in Karaj basin can help the relevant organizations to manage the problem of water scarcity in the region appropriately.
The rainfall-runoff modelling is a prior issue in any river management and also water resources programs.Development of an appropriate model is a difficult procedure due to the complexity and dynamic behaviour of nonlinear hydrological processes.Most hydrologic processes exhibit a high degree of temporal and spatial variability, and are further plagued by issues of nonlinearity of physical processes, conflicting spatial and temporal scale, and uncertainty in parameter estimates.Over the past years, artificial intelligence techniques have been frequently used to predict nonlinear problems and achieved good results [2][3][4][5][6].Also, conjunction with wavelet transform enhances ANN popularity which Wavelet-ANN or wavelet-ANFIS models were successfully hired for nonlinear time series forecasting [7][8][9][10][11].
For rainfall-runoff modelling, ANNs have been frequently used and the results demonstrated their great capability for this purpose [12][13][14][15].These models often utilized the conventional Levenberg-Marquardt as a back propagation algorithm for ANN training.Recently, it was found that applying PSO algorithm instead of LM can improve the models' efficiency significantly [16][17][18].Kouk et al. 2010 applied particle swarm optimization feed forward neural network for hourly rainfall-runoff modelling in Bedup Watershed in Malaysia [16].They showed that the proposed model can provide accurate prediction of runoff up to 12-hour ahead.Asadnia et al. 2013 presented an improved PSO technique to train ANN to predict water level for Heshui Watershed in China [17].They also constructed several LM-ANN models to compare with those of the PSO-ANN models.The results indicated that the PSO-ANN models outperform the LM-ANN models especially for low and peak water levels.Cheng et al. 2015 conducted a research study consisting of daily reservoir runoff forecasting [18].In the modelling procedure, they gain a neural network using Quantum behaved particle swarm optimization for the training process.The results revealed that the PSO based ANN model outperforms the basic ANN model.Moreover, they concluded that the proposed model can be applied for daily runoff modelling efficiently.
The main objective of this study is to investigate the capability of PSO-ANN models for rainfall-runoff modelling and runoff forecasting up to 2 days ahead in Karaj River.Overall, the PSO algorithm is used to find the optimum value of the weights and biases of the ANNs.In this regard, it is applied to tune these values in a way the highest accuracy of the model could be obtained.To do this, historical time series of rainfall and runoff obtained from the gauging stations were used for training the models.Separate ANN model applying conventional training algorithm of Levenberg-Marquardt were developed to provide more comparisons.Different combinations of the input variables including rainfall and flow discharge with different lags are considered in the input structure of the ANN and PSO-ANN models.Rest of the paper is organized as follows.Section 2 comprises a summary of PSO algorithm, ANNs, model development procedure, study area and data, and performance evaluation criteria.Results and discussion will be presented in section 3. The concluding remarks are the last section.

Materials and method
The aim of this study is not to delve into the theory of the PSO and ANNs.Therefore, the main concepts of them are only presented in the following subsections.

Particle Swarm Optimization (PSO)
PSO is a population based stochastic optimization technique inspired by social behavior of bird flocking.In PSO, particles which fly through the problem space by following the current optimum particles forms the solutions.The coordinates related to the best fitness is stored for each in which each particle has achieved so far.The performance of the algorithm is evaluated via fitness value in each search cycle.This value is called pbest.Another "best" value that is tracked by the particle swarm optimizer is the best value, obtained so far by any particle in the neighbors of the particle.This location is called lbest.The global best or gbest is called to the best value among the whole particles.In each iteration, velocity of each particle is changing toward its pbest and lbest locations (local version of PSO) in order to achieve the optimum solutions [19].Acceleration or velocity is weighted by a random term, with separate random numbers being generated for acceleration toward pbest and lbest locations.Generally, the swarm movement can be expressed via the following equation: Where and represent particle best and particle neighbourhood best respectively.C 1 and C 2 stand for cognitive and social constants, t, r, x and V shows the time step, random, position and velocity respectively.
( ) is the inertia term.Dealing with PSO, values of parameters such as C 1 and C 2 should be set appropriately in order to achieve an optimized solution.Cognitive and social terms force the particle to move back to the previous best position, and to the best previous position of its neighbors respectively.Iteration is described as number of recruitment, fitness evaluation or in general number of searching cycle completed by the algorithm.

Artificial Neural Network (ANN)
ANN is an approximation function mapping inputs to outputs.A typical network consists of three layers of neurons namely, input, hidden, and output; in which each neuron acting as an independent computational element.Input layer is defined as a layer of neurons receiving inputs directly from outside the network.Layer of a network that is not connected to the network output called hidden layer and layer whose output is passed to the world outside the network is output layer.Weight functions apply weights to an input to get weighted inputs, as specified by a particular function.Different algorithms can apply to minimize the network error during the ANN training.This usually happens by finding the correct tune of the network in which is depended to weights, biases, number of neurons in the hidden layer and iteration number.Among these parameters, the training algorithm is important in a way it reaches to optimum weights and biases.Generally, the performance function of the ANN models during training process is assessed using sum of squares of the errors as follows [20]: Where T is the total number of training samples, m is the number of output layer neurons, W represents the vector containing all the weights in the network, yp is the actual network output, and dp is the desired output.Achieving optimum number of neurons in the hidden layer is often obtained through a trial and error procedure.Aside of the abovementioned parameters, correct selection of the input variables which affect the target variable is considered as one of the most important stages dealing with ANN models.Figure 2. depicts a schematic layout of the ANN with four input variables.

ANN Training with PSO Algorithm
To train the ANN with PSO algorithm, the following procedures are taken under consideration.The algorithm is utilized to find the optimum weights and biases of the ANN model.In this regard, they are investigated as the position of the particles in the algorithm.Weights and biases' values form search space of the algorithm which is of ndimensions.The n is the total number of weights (and biases) that needs to be optimized.Each particle has a position vector and a velocity vector of n-dimensions [21].Here both weights and biases are shown by W.
The optimal set of weights is obtained by flying the particles around the search space.At each iteration, the algorithm comes up with a set of weights that their fitness is assessed.It happens by assigning these weights to the nodes and predicting the target value.Afterwards, the accuracy of the prediction through assigned weights is evaluated as the difference between actual and predicted values which should be minimized through optimization process.In this regard, the best fitness the particle has been achieved so far is considered as its personal best.Similarly, the best fitness of the swarm is used as the global best.This process is repeated for a specific number of iteration until the optimized weights for the ANN are yielded.The steps for a PSO optimized ANN is given below.
For a three-layered perceptron, W [1] and W [2] represent the connection weight matrix between the input layer and the hidden layer, and between the hidden layer and the output layer respectively.Applying PSO algorithm to train the multi-layer perceptron, the i th particle is denoted by: The position representing the previous best fitness value of any particle is recorded and denoted by: If, among all the particles in the current population, the index of the best particle is represented by the symbol b, then the best matrix is denoted by: The velocity of particle is denoted by: If m and n represent the index of matrix row and column, respectively, the manipulation of the particles are as follows [22]: and, (8) where j = 1, 2; m = 1,. . .,M j ; n = 1,. . .,N j ; M j and N j are the row and column sizes of the matrices W, P, and V; r and s are positive constants; a and b are random numbers in the range from 0 to 1; t is the time step between observations and is often taken as unity; V'' and W'' represent the new values.Applying Equation 7, the new velocity of the particle is computed by using its previous velocity and the distances of its current position from the best experiences both in its own and as a group.The second element on the right-hand side of Equation 7, represents the private thinking of the particle itself whilst the social part, i.e., the third element on the right hand side of Equation 7, denotes the collaboration among the particles as a group [22].The new position according to the new velocity can be determined by Equation 8.
The fitness function f is the mean squared error and is defined as: Where is the fitness value, n is the number of data points, and are observed and predicted values of K respectively.

Study Area and Data
In this study, Karaj River basin was considered as a case study in order to investigate efficiency of the PSO-ANN models to predict the runoff 1 and 2 days ahead.The river flows to the Karaj (Amirkabir) dam.Overall, Karaj River basins located between 51° 2 ' to 51° 32 ' E and 35° 53 ' to 36° 11 ' Nlongitudes and latitudes respectively.It drains an area of 720 km 2 and is about 66 km long [23].The hydrometric station (Seera hydrometric station) which records the flow in the main river is located near the dam entrance.Figure 3. illustrates the location of the study area and hydrometric station.In the model development, a five year period data related to daily precipitation and runoff recorded by gauges were applied.These data were gathered from hydrometric station and the rain gauge during 2002-2007.80% data were used for training and 20% remains for testing.The statistical analysis of the datasets including minimum, maximum, average, standard deviation and skewness are presented in Table 1.Moreover, the flow and rainfall time series are depicted in Figure 4, a and b respectively.Since the extreme values of the flow and rainfall are placed in the end part of the data sets, therefore, the first 20% of data were used as testing data and remaining are utilized as training data.The following equation was applied to bring whole data in a range of [0-1]: ( 10 ) Where = the normalized value, = the measured value, = the minimum and = maximum of the variable.

Performance evaluation criteria
In this study, the performance of different predictive models is evaluated by using statistical measures including the coefficient of determination (R 2 ), coefficient of efficiency (NSE) and the root mean square error (RMSE).R 2 , ranging from 0 to 1, measures the degree of determination among the observed and predicted values.R 2 values of 1 and 0 indicate a perfect fit and no statistical correlation between the data and the line drawn through them respectively.The NSE is used to assess the predictive power of hydrological models (ranges from -∞ to 1. RMSE indicates the discrepancy between the observed and forecasted values.Generally, the best models' predictions are achieved when R 2 , NSE and RMSE are found to be close to1, 1 and 0, respectively.These indices (R 2 and RMSE) are defined as follows: ∑ ( ( Where Q and n stand for daily runoff in cubic meter per second and the number of data respectively.̅ ( ) and ̅ ( ) represent mean values of measured and predicted runoff in Karaj River respectively.

Results and Discussion
To obtain a reliable rainfall-runoff model, historical time series of the rainfall and runoff with different lags (up to 3days ago) were examined in the ANN input structure.It is noticeable that the best input variables were tested with LM-ANN models.Afterwards, the most effective input variable on runoff in the next day was imposed as inputs for PSO-ANN models.Dealing with LM-ANN models, it was found that the runoff in time t+1 is mostly related to rainfall and runoff 1 and 2days ago.Table 2 shows the results obtained from the LM-ANN models for different input variables.In Table 2, the statistical error measures of coefficient of determination and root mean square error (RMSE) are computed for normalized data sets and therefore the RMSE is dimensionless.As it can be obtained from Table 2, the model No. 4 has the best performances and its input variables are selected for further investigation with PSO-ANN model.It can be observed that the runoff in previous time steps (Q (t), Q (t-1)) is more effective than rainfall (R(t)) on the model performances.For example, model No. 2 which does not have historical time series of runoff in its input structure has the lowest accuracy.
Dealing with PSO-ANN model, there are some parameters which they have to be tuned appropriately in order to achieve the best results.These parameters are number of iteration, cognitive and social coefficients C1 and C2 and number of particles.In this regard, the best model performances were achieved through a trial and error process.Table 3 provides the results of the process which yielded to the most accurate model.According to Table 3, models No. 2 and 4 are amongst the models with highest accuracy.They have the highest and the lowest values of R2 and RMSE respectively.Overall, it is demonstrated that the model with equal value of C1 and C2 has a better performance.In models No. 2 and 4 which are the best models, C1=C2=2.Any deviation from this value decreases the model's performances.Model No. 4 slightly outperforms model No. 2 in term of RMSE.However, the difference is negligible.A comparison between the structure of the PSO in models No 2 and 4 reveals that model No. 4 has more particles and iteration numbers which increases the runtime.
The scatterplots and time series of the predicted runoff in the next day for LM-ANN and PSO-ANN models are presented in Figures 5. and 6. respectively.

Figure 5. Scatter plots of the observed and predicted runoff one day ahead
Overall, both LM-ANN and PSO-ANN models give an acceptable prediction of runoff 1 day ahead.Their predictions for low and medium range of runoff are relatively high.However, for the extreme values, the correlation between real and predicted values decreases.For the LM-ANN model, the predicted values of runoff have more deviation from the real values.For the PSO-ANN model, a good agree of similarity between observed and predicted values can be observed even for high values.Generally, the R 2 for the test period of the LM-ANN and PSO-ANN models are 0.84 and 0.88 respectively which indicate superiority of the PSO-ANN model over the LM-ANN models.Regarding Figure 6, it can be derived that the LM-ANN and PSO-ANN models both have a relatively high accuracy in predicting of low and medium runoff.Anyhow, the PSO-ANN model has slightly better performances than the LM-ANN model.The main advantage of the PSO-ANN model in compare with the LM-ANN model is its capability for predicting the high and extreme values of runoff.For instance, the extreme value of 111 m 3 /s is predicted as about 42 and 62 by LM-ANN and PSO-ANN models respectively.Other extreme values of 78 and 74 m 3 /s are predicted as 26 and 52 m 3 /s by LM-ANN model while the PSO-ANN model predicts them as 26 and 81 m 3 /s.Therefore, the PSO-ANN has a better performance especially for extreme value of flow 1 day ahead.To predict runoff one more time steps (2 days ahead), separate LM-ANN and PSO-ANN models with the same input structure as the best ANN models for 1 day ahead runoff forecasting were developed.The results of them as scatterplots and forecasted time series are illustrated in Figures 7. and  As it can be observed, the PSO-ANN model provides more accurate predictions of runoff 2 days ahead for low, medium and high flows.The LM-ANN model slightly underestimates low flows and also high flows.The LM-ANN has R 2 equal to 0.74 and the PSO-ANN has R 2 of 0.78 which demonstrate a higher correlation between observed and predicted runoff of the PSO-ANN model.As a whole, the PSO-ANN model outperforms the LM-ANN model for 2 days ahead runoff prediction in terms of R 2 and RMSE.The coefficient of determination of both models is less than those of for one day ahead predictions.It is an expected conclusion because with increasing the time steps in predictions, the correlation between input and output variables decreases.For example, the runoff in time shows a higher correlation with the runoff in than .

Conclusions
Over the last two decades, an increasing trend can be found in application of artificial neural networks (ANNs) for simulating and predicting the nonlinear problems.Rainfall-runoff modelling and runoff prediction is one of the most important and also complicated phenomena in water engineering.It is depended to many variables which some of them cannot be measured easily.Moreover, the nonlinear identity of the problem and unknown mathematical relationship between inputs and output led the researcher to apply ANN models for this purpose.Anyhow, most of the previous studies employed the conventional training algorithm in their model development.Recently, it was found that metaheuristic algorithm can be successfully linked to the ANN models to train them and enhance their performances.In this regard, this study was mainly organized to investigate and compare efficiency of particle swarm optimization (PSO) as a metaheuristic algorithm against the Levenberg-Marquardt back propagation algorithm (the conventional algorithm).The models were utilized to predict runoff in Karaj River for one and two days ahead.
According to this study, it can be concluded that applying PSO algorithm instead of conventional algorithm increases the ANN models' efficiency remarkably.For runoff prediction one day ahead, the PSO-ANN model outperform the LM-ANN model in terms of R 2 and RMSE in a way the first model has R 2 =0.88 and RMSE=0.018while it is equal to 0.84 and 0.039 for the LM-ANN model.Similarly, the PSO-ANN model for 2 days ahead runoff forecasting is superior over the correspond LM-ANN model.Regarding the problem complexity, the enhancement is valuable and the proposed approach can be applied to address other nonlinear problems.
Overall, results of this study revealed that an acceptable prediction of the runoff up to two days ahead can be achieved by applying the PSO-ANN model.The model provides an acceptable prediction for a wide range of runoff variation.The PSO-ANN and LM-ANN models are both more accurate in predicting low and medium range of flow and give an underestimation of the extreme values.With increasing the forecasting time steps, the models' performance decreases.
In this study, it was found that the training algorithm plays an important role on the ANN models' performances.Also, it was obtained that PSO algorithm as a metaheuristic algorithm can improve the performance of the conventional training algorithm of ANNs.Therefore, it can be considered as a suitable alternative for conventional training algorithms for ANNs.Moreover, capability of other metaheuristic techniques can be investigated and the results can be compared with PSO algorithm.It can be combined with wavelet-ANN models to find the optimum weights of the decomposed time series to achieve the most model's accuracy too.

Figure 2 .
Figure 2. Schematic layout of ANN model with four input variables

Figure 3 .
Figure 3. Location of the study area and hydrometric station

Figure 4 .
Figure 4. Time series of runoff and rainfall

Figure 6 .
Figure 6.Measured and forecasted runoff time series for one day ahead

Figure 7 .Figure 8 .
Figure 7. Scatter plots of the observed and predicted runoff two days ahead