Agent-based computational economics

Resarch

Main pillars of ACE resarch according to Leigh Tesfatsion.

• Empirical
• Normative
• Qualitativ insight and theory generation
• Methodological advancement

Empirical

This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

Normative

ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.

Qualitative insight and theory generation

Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

Methodological advancement

representation, visualization, validation, ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see Software and programming

Fields of application

One of the first major applications of multi-agent models in social sciences was famous Sugarscape model by Epstein and Axell. ACE can approach can be applied to rather simple double-auction market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market . More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the EURACE project. Models like this enable what-if analysis and policy experiments. . There are also applications to model economic behaviour of vanished civilizations

Computational world models

Agent hierarchy used in AMES framework

Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on AMES framework example. Agent can be simple-programmed, autonomous or human-like In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent. For example in double auction model, agents may have following methods:

getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);

Equilibriums and attractors

Agent stops too early in a basin of attraction missing the highest attainable profit

Model behavior can result to various types of equilibrium and attractors. "System is in equilibrium if all influences acting on the system offset each other so that the system is in an unchanging condition" . Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible basins of attraction. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement learning. Because of using simple Derivative-follower adaptation agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.

Agent types and characteristics

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses . However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. Stigmergy can be one way to achieve this. Agents can be differentiated by position in the cognitive hierarchy, where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous . Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior.

Learning

In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation. Roth-Elev reinforcement learning algorithm is one of the possible choices:

1. Initialize action propensities to an initial propensity value.
2. Generate choice probabilities for all actions using current propensities.
3. Choose an action according to the current choice probability distribution.
4. Update propensities for all actions using the reward (profits) for the last chosen action.
5. Repeat from step 2.

Possible learning types

There are various types of learning algorithms ). Here is brief summary by Leigh Tesfatsion:

1. Reactive Reinforcement Learning (RL)
   1. Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
   2. Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
2. Belief-Based Learning
   1. Example 1: Fictitious play
   2. Example 2: Hybrid forms (e.g. Camerer/Ho EWA algorithm )
3. Anticipatory Learning (Q-Learning)
   1. Evolutionary Learning (Genetic Algorithms - GAs)
4. Connectionist Learning (Artificial Neural Nets - ANNs)

In reinforcement learning algorithms, if an action A in state S produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action A produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision D, he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in Learning. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents. Example of this can be matching pennies game:

Matching pennies game outcome matrix

		Player 2
		Heads	Tails
Player 1	Heads	+1, −1	−1, +1
	Tails	−1, +1	+1, −1

If agent uses anticipatory learning (or temporal-difference learning), he's trying to predict what might happen in the future, if he takes some action A. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. Q-Learning enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices.

+ doplnit Genetic a neuron sítě?

Software and programming

For elaborate overview see Comparison of agent-based modeling software.

Software suitable for ACE programming or visualization

Software name	Platform and performance	Scope	Visualization capabilities	Licence
AnyLogic	Java platform	Agent-based general purpose; distributed simulations	2D/3D models	proprietary
NetLogo	Based on Java, suitable for smaller models	simulations of natural and social phenomena, suitable for novice modelers	2D/3D models	freeware

Other computing methods

Linear Equations and Iterative Methods (Currently empty) Optimization Nonlinear Equations Approximation Numerical Integration and Differentiation Monte Carlo and Simulation Methods (Currently empty) Quasi-Monte Carlo Methods (Currently empty) Finite Difference Methods (Currently empty) Projection Methods for Functional Equations (Currently empty) Numerical Dynamic Programming (Currently empty) Regular Perturbations of Simple Systems (Currently empty) Regular Perturbations in Multidimensional Systems (Currently empty) Advanced Asymptotic Methods (Currently empty) Solution Methods for Perfect Foresight Models (Currently empty) Solving Rational Expectations Models

References

• http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
• http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
• http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
• http://www2.econ.iastate.edu/tesfatsi/aemind.htm
• http://www2.econ.iastate.edu/tesfatsi/aintro.htm