Agent-based computational economics

Agent-based computational economics or shortly ACE is branch of computational economics. It uses agent-based models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore "a computational study of economic processes modeled as dynamic systems of interacting agents"

Resarch

Main pillars of ACE resarch according to Leigh Tesfatsion.

• Empirical
• Normative
• Qualitativ insight and theory generation
• Methodological advancement

Empirical

This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

Normative

ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.

Qualitative insight and theory generation

Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization.

Methodological advancement

ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see Software and programming

Fields of application

One of the first major applications of multi-agent models in social sciences was famous Sugarscape model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple double-auction market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market . More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the EURACE project. Models like this enable what-if analysis and policy experiments on European scale. . There are also applications to model economic behaviour of vanished civilizations

Computational world models

Agent hierarchy used in AMES framework

Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on AMES framework example. Agent can be simple-programmed, autonomous or human-like In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent. For example in double auction model, agents may have following methods:

getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);

First method acquires (getWorldEventSchedule current time from the world itself. Through getMarketProtocols agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method getWorldProtocols can serve for other out of market interactions.

Equilibriums and attractors

Agent stops too early in a basin of attraction missing the highest attainable profit

Model behavior can result to various types of equilibrium and attractors. "System is in equilibrium if all influences acting on the system offset each other so that the system is in an unchanging condition" . Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible basins of attraction. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement learning. Because of using simple Derivative-follower adaptation agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.

Agent types and characteristics

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses . However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. In some cases even without learning capability, the agents can optimize or are able to generate orderly movement patterns. Stigmergy can be one way to achieve this. Agents can be differentiated by position in the cognitive hierarchy, where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous . Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior.

Learning

In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation. Roth-Elev reinforcement learning algorithm is one of the possible choices. It works in following steps:

1. Initialize action propensities to an initial propensity value.
2. Generate choice probabilities for all actions using current propensities.
3. Choose an action according to the current choice probability distribution.
4. Update propensities for all actions using the reward (profits) for the last chosen action.
5. Repeat from step 2.

Possible learning types

There are various types other of learning algorithms suitable for use in ACE ). Here is brief summary by Leigh Tesfatsion:

1. Reactive Reinforcement Learning (RL)
   1. Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
   2. Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
2. Belief-Based Learning
   1. Example 1: Fictitious play
   2. Example 2: Hybrid forms (e.g. Camerer/Ho EWA algorithm )
3. Anticipatory Learning (Q-Learning)
   1. Evolutionary Learning (Genetic Algorithms - GAs)
4. Connectionist Learning (Artificial Neural Nets - ANNs)

In reinforcement learning algorithms, if an action A in state S produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action A produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision D, he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in Learning. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents. Example of this can be matching pennies game:

Matching pennies game outcome matrix

		Player 2
		Heads	Tails
Player 1	Heads	+1, −1	−1, +1
	Tails	−1, +1	+1, −1

If agent uses anticipatory learning (or temporal-difference learning), he's trying to predict what might happen in the future, if he takes some action A. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. Q-Learning enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices. Cobweb model is example of genetic algorithm for use in economics. For connectionist learning various types of Artificial Neural Nets configurations can be used.

Examples of real applications

• An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes .

• Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well.

• Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority.

• Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution.

Software and programming

For elaborate overview see Comparison of agent-based modeling software.