1. bookVolume 8 (2018): Issue 2 (April 2018)
Journal Details
First Published
30 Dec 2014
Publication timeframe
4 times per year
access type Open Access

Learning Structures of Conceptual Models from Observed Dynamics Using Evolutionary Echo State Networks

Published Online: 01 Nov 2017
Page range: 133 - 154
Received: 04 Mar 2017
Accepted: 29 Mar 2017
Journal Details
First Published
30 Dec 2014
Publication timeframe
4 times per year

Conceptual or explanatory models are a key element in the process of complex system modelling. They not only provide an intuitive way for modellers to comprehend and scope the complex phenomena under investigation through an abstract representation but also pave the way for the later development of detailed and higher-resolution simulation models. An evolutionary echo state network-based method for supporting the development of such models, which can help to expedite the generation of alternative models for explaining the underlying phenomena and potentially reduce the manual effort required, is proposed. It relies on a customised echo state neural network for learning sparse conceptual model representations from the observed data. In this paper, three evolutionary algorithms, a genetic algorithm, differential evolution and particle swarm optimisation are applied to optimize the network design in order to improve model learning. The proposed methodology is tested on four examples of problems that represent complex system models in the economic, ecological and physical domains. The empirical analysis shows that the proposed technique can learn models which are both sparse and effective for generating the output that matches the observed behaviour.


[1] J. D. Sterman, Business Dynamics: Systems Thinking and Modeling for a Complex World, vol. 19. Irwin/McGraw-Hill Boston, 2000.Search in Google Scholar

[2] F. C. Billari, Agent-based computational modelling: applications in demography, social, economic and environmental sciences. Taylor & Francis, 2006.Search in Google Scholar

[3] R. A. Howard and J. E. Matheson, Influence diagrams, Decis. Anal., vol. 2, no. 3, pp. 127–143, 2005.Search in Google Scholar

[4] F.-R. Lin, M.-C. Yang, and Y.-H. Pai, A generic structure for business process modeling, Bus. Process Manag. J., vol. 8, no. 1, pp. 19–41, 2002.Search in Google Scholar

[5] L. Schruben, Simulation modeling with event graphs, Commun. ACM, vol. 26, no. 11, pp. 957–963, 1983.10.1145/182.358460Open DOISearch in Google Scholar

[6] S. Robinson, Simulation: the practice of model development and use. Palgrave Macmillan, 2014.Search in Google Scholar

[7] J. Ryan and C. Heavey, Requirements gathering for simulation, in Proceedings of the 3rd Operational Research Society Simulation Workshop. The Operational Research Society, Birmingham, UK, 175-184, 2006.Search in Google Scholar

[8] A. Medina-Borja and K. S. Pasupathy, Uncovering complex relationships in system dynamics modeling: Exploring the use of CART, CHAID and SEM, in Proceedings of the 25th International Conference of the System Dynamics Society, (Boston, USA), pp. 1–24, 2007.Search in Google Scholar

[9] V. Quiñones-Avila and A. Medina-Borja, Universal healthcare: key behavioural factors affecting providers and recipients value propositions: a structural causal model of the puerto rico experience, Int. J. of Behav. and Hlthc. Res., vol. 3, no. 1, pp. 25–45, 2012.Search in Google Scholar

[10] M. Drobek, W. Gilani, T. Molka, and D. Soban, Automated equation formulation for causal loop diagrams, Lecture Notes in Business Information Processing, vol. 208, pp. 38–49, 2015.Search in Google Scholar

[11] E. Pruyt, S. Cunningham, J. Kwakkel, and J. De Bruijn, From data-poor to data-rich: system dynamics in the era of big data, in Proceedings of the 32nd International Conference of the System Dynamics Society, Delft, The Netherlands, 20-24 July 2014.Search in Google Scholar

[12] H. Jaeger, The ’echo state’ approach to analysing and training recurrent neural networks-with an erratum note, Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, vol. 148, p. 34, 2001.Search in Google Scholar

[13] H. Abdelbari and K. Shafi, Learning causal loop diagram-like structures for system dynamics modeling using echo state networks, Syst. Dynam. Rev. - In Press, 2017.Search in Google Scholar

[14] D. E. Goldberg, Genetic algorithms. Pearson Education India, 2006.Search in Google Scholar

[15] R. Storn and K. Price, Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces, J. Global. Optim., vol. 11, no. 4, pp. 341–359, 1997.10.1023/A:1008202821328Open DOISearch in Google Scholar

[16] J. Kennedy, Particle swarm optimization, in Encyclopedia of machine learning, pp. 760–766, Springer, 2011.Search in Google Scholar

[17] Z. Wang, J. Zhang, J. Ren, and M. N. Aslam, A geometric singular perturbation approach for planar stationary shock waves, Physica D, vol. 310, pp. 19–36, 2015.Search in Google Scholar

[18] C. K. Jones, R. Marangell, P. D. Miller, and R. G. Plaza, On the stability analysis of periodic sine–gordon traveling waves, Physica D, vol. 251, pp. 63–74, 2013.Search in Google Scholar

[19] V. V. Gursky, J. Reinitz, and A. M. Samsonov, How gap genes make their domains: An analytical study based on data driven approximations, Chaos, vol. 11, no. 1, pp. 132–141, 2001.Search in Google Scholar

[20] P. Young, Data-based mechanistic modelling of environmental, ecological, economic and engineering systems, Environ. Modell. Softw., vol. 13, no. 2, pp. 105–122, 1998.Search in Google Scholar

[21] Y. Zhao, T. Weng, and M. Small, Response of the parameters of a neural network to pseudoperiodic time series, Physica D, vol. 268, pp. 79–90, 2014.Search in Google Scholar

[22] Y. Feng, Y. Liu, X. Tong, M. Liu, and S. Deng, Modeling dynamic urban growth using cellular automata and particle swarm optimization rules, Landscape Urban Plan., vol. 102, no. 3, pp. 188–196, 2011.10.1016/j.landurbplan.2011.04.004Open DOISearch in Google Scholar

[23] N. Petrov and A. Gegov, Model optimization for complex systems using fuzzy networks theory, in Proceedings of the 8th WSEAS international conference on Artificial intelligence, knowledge engineering and data bases, pp. 116–121, World Scientific and Engineering Academy and Society (WSEAS), 2009.Search in Google Scholar

[24] I. M. Greca and M. A. Moreira, Mental models, conceptual models, and modelling, Int. J. Sci. Educ, vol. 22, no. 1, pp. 1–11, 2000.Search in Google Scholar

[25] J. D. Sterman, Systems dynamics modeling: tools for learning in a complex world, IEEE Eng. Manag. Rev., vol. 30, no. 1, pp. 42–42, 2002.Search in Google Scholar

[26] G. Desthieux, F. Joerin, and M. Lebreton, Ulysse: a qualitative tool for eliciting mental models of complex systems, Syst. Dynam. Rev., vol. 26, no. 2, pp. 163–192, 2010.10.1002/sdr.434Open DOISearch in Google Scholar

[27] K.-i. Funahashi and Y. Nakamura, Approximation of dynamical systems by continuous time recurrent neural networks, Neural networks, vol. 6, no. 6, pp. 801–806, 1993.Search in Google Scholar

[28] H. Jaeger, Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the” echo state network” approach, Tech. Rep. 159, Fraunhofer Institute for Autonomous Intelligent Systems (AIS), 2002b.Search in Google Scholar

[29] D. Koryakin, J. Lohmann, and M. V. Butz, Balanced echo state networks, Neural Networks, vol. 36, pp. 35–45, 2012.10.1016/j.neunet.2012.08.008Open DOISearch in Google Scholar

[30] I. B. Yildiz, H. Jaeger, and S. J. Kiebel, Re-visiting the echo state property, Neural networks, vol. 35, pp. 1–9, 2012.Search in Google Scholar

[31] M. Lukoševišius, A practical guide to applying echo state networks, in Neural Networks: Tricks of the Trade, pp. 659–686, Springer, 2012.10.1007/978-3-642-35289-8_36Open DOISearch in Google Scholar

[32] C. E. Martin and J. A. Reggia, Fusing swarm intelligence and self-assembly for optimizing echo state networks, Comput. Intell. Neurosci., vol. 2015, p. 9, 2015.Search in Google Scholar

[33] A. A. Ferreira and T. B. Ludermir, Comparing evolutionary methods for reservoir computing pretraining, in Proceedings of the 2011 International Joint Conference on Neural Networks, San Jose, California, USA, pp. 283–290, July 31 - August 5 2011.Search in Google Scholar

[34] A. Deihimi and A. Solat, optimised echo state networks using a big bang–big crunch algorithm for distance protection of series-compensated transmission lines, Int. J. Elec. Power., vol. 54, pp. 408–424, 2014.10.1016/j.ijepes.2013.07.024Open DOISearch in Google Scholar

[35] A. A. Ferreira, T. B. Ludermir, and R. R. De Aquino, An approach to reservoir computing design and training, Expert. Syst. Appl., vol. 40, no. 10, pp. 4172–4182, 2013.Search in Google Scholar

[36] D. Liu, J. Wang, and H. Wang, Short-term wind speed forecasting based on spectral clustering and optimised echo state networks, Renew. Energ., vol. 78, pp. 599–608, 2015.Search in Google Scholar

[37] J. L. Gross and J. Yellen, Handbook of graph theory. CRC press, 2004.Search in Google Scholar

[38] R. Tarjan, Depth-first search and linear graph algorithms, SIAM J. Comput., vol. 1, no. 2, pp. 146–160, 1972.10.1137/0201010Open DOISearch in Google Scholar

[39] V. Petridis, S. Kazarlis, and A. Bakirtzis, Varying fitness functions in genetic algorithm constrained optimization: the cutting stock and unit commitment problems, IEEE Trans. Syst., Man, Cybern., Part B: Cybern., vol. 28, no. 5, pp. 629–640, 1998.Search in Google Scholar

[40] A. E. Smith and D. M. Tate, Genetic optimization using a penalty function, in Proceedings of the 5th international conference on genetic algorithms, pp. 499–505, Morgan Kaufmann Publishers Inc., 1993.Search in Google Scholar

[41] K. Langfield-Smith and A. Wirth, Measuring differences between cognitive maps, J. Oper. Res. Soc., pp. 1135–1150, 1992.10.1057/jors.1992.180Open DOISearch in Google Scholar

[42] Y.-C. Chuang, C.-T. Chen, and C. Hwang, A simple and efficient real-coded genetic algorithm for constrained optimization, Appl. Soft. Comput., vol. 38, pp. 87–105, 2016.10.1016/j.asoc.2015.09.036Open DOISearch in Google Scholar

[43] J. Lane, A. Engelbrecht, and J. Gain, Particle swarm optimization with spatially meaningful neighbours, in Swarm Intelligence Symposium, 2008. SIS 2008. IEEE, pp. 1–8, IEEE, 2008.Search in Google Scholar

[44] R. C. Eberhart and Y. Shi, Comparing inertia weights and constriction factors in particle swarm optimization, in Proceedings of the 2000 Congress on Evolutionary Computation, vol. 1, pp. 84–88, IEEE, 2000.Search in Google Scholar

[45] S. N. Grösser and M. Schaffernicht, Mental models of dynamic systems: taking stock and looking ahead, Syst. Dynam. Rev., vol. 28, no. 1, pp. 46–68, 2012.Search in Google Scholar

[46] E. M. Aylward, P. A. Parrilo, and J.-J. E. Slotine, Stability and robustness analysis of nonlinear systems via contraction metrics and sos programming, Automatica, vol. 44, no. 8, pp. 2163–2170, 2008.10.1016/j.automatica.2007.12.012Open DOISearch in Google Scholar

[47] M. Rafferty, Butterflies and buffers, in Proceedings of the 27th International Conference of the System Dynamics Society, Albuquerque, Mexico, USA, July 26-30 2009.Search in Google Scholar

[48] E. Theodorsson-Norheim, Friedman and quade tests: Basic computer program to perform nonparametric two-way analysis of variance and multiple comparisons on ranks of several related samples, Comput. Biol. Med., vol. 17, no. 2, pp. 85–99, 1987.10.1016/0010-4825(87)90003-5Open DOISearch in Google Scholar

[49] M. R. Stoline, The status of multiple comparisons: simultaneous estimation of all pairwise comparisons in one-way anova designs, Am. Stat., vol. 35, no. 3, pp. 134–141, 1981.Search in Google Scholar

[50] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Trans. Evol. Comput., vol. 6, no. 2, pp. 182–197, 2002.10.1109/4235.996017Open DOISearch in Google Scholar

Recommended articles from Trend MD

Plan your remote conference with Sciendo