.. _writing_agents:

Writing an agent
=================
This tuturial will explain how you can write your own learning agent for the MMLF. 

.. note::
     Writing an agent is easier with a local installation of the MMLF (see :ref:`Installation Tutorial <installation>`).

.. seealso::
   Get an overview over the existing agents in :ref:`agent_list`


Learning about the basic structure of MMLF agents
-------------------------------------------------

For the start, please take a look into the agents subdirectory of the MMLF and open the random_agent.py in the python editor of your choice. The :ref:`RandomAgent <random_agent>` is a quite simple and straightforward agent which demonstrates well the inner life of an agent (though an intelligent agent might choose his actions differently ;-)).

What you can learn from the agent is the following:
  * Each agent has to be a subclass of AgentBase
  * Each agent class must have a static attribute DEFAULT_CONFIG_DICT, which contains the parameters that are available for customizing the agent's behaviour and their default values.
  * The __init__ method gets passed additional arguments (``*args``) and keyword arguments (``**kwargs``). These MUST be passed on to the superclass' constructor using ``super(RandomAgent, self).__init__(*args, **kwargs)``
  * Each agent must have an AgentInfo attribute that specifies in which kinds of environments it can be used, which communication protocol it supports etc.
  * The setStateSpace(self, stateSpace) method is called to inform the agent about the structure of the :ref:`state space <state_spaces>`. A default implementation of this method is contained in the AgentBase class; if this implementation is sufficient the method need not be implemented again.
  * The setActionSpace(self, actionSpace) method is called to inform the agent about the structure of the :ref:`action space <action_spaces>`. A default implementation of this method is contained in the AgentBase class; if this implementation is sufficient the method need not be implemented again.
  * The setState(self, state) method is called to inform the agent about the current state of the environment. To correctly interpret the state, the agent has to use the definition of the :ref:`state space <state_spaces>`.
  * The getAction(self) method is called to ask the agent for the action he wants to perform. The agent should store its decision in a dictionary which maps action dimension name to the chosen value for this dimension. For instance: {"gasPedalForce": "extreme", "steeringWheelAngle": 30} (see page :ref:`action space <action_spaces>`). This action dictionary must be converted to an ActionTaken object via the method _generateActionObject(actionDictionary) of AgentBase.
  * The giveReward(self, reward) method is called to reward the agent for its last action(s). The passed reward is a float value. The agent can treat this reward in different ways, e.g. accumulate it, use it directly for policy optimization etc.
  * The nextEpisodeStarted(self) method is called whenever one epsiode is over and the next one is started, i.e. when the environment is reset. In this method, you should finish all calculations which make only sense during one episode (such as accumulating the reward obtained during one episode). 
  * In each agent module, the module-level attribute AgentClass needs set to the class that inherits from AgentBase. This assignment is located usually at the end of the module: AgentClass = RandomAgent
  * Furthermore, the module-level attribute AgentName should be set to the name of the agent, e.g. AgentName = "Random". This name is used for instance in the GUI.
  * The agent can send messages to the logger by calling "self.agentLog.info(message)"

Writing a new MMLF agent
------------------------

Let's write a new agent. This agent should execute actions independent of the state in a round-robin like manner, i.e. when there are three available actions a1, a2, a3, the agent should choose actions in this sequence: a1,a2,a3,a1,a2,a3,... Obviously this is not a very clever approach, but for the tutorial it should suffice.

In order to implement a new agent that chooses actions in a round-robin like manner, you have to do the following:
 #. Go into the agents subdirectory of the MMLF and create a copy of the random_agent.py (let's call this copy example_agent.py). 
 #. Open example_agent.py and rename the agent class from RandomAgent to ExampleAgent. Replace every occurrence of RandomAgent by ExampleAgent. 
 #. Set "DEFAULT_CONFIG_DICT = {}", since the agent does not have any configuration options.
 #. Set "continuousAction = False", since the round-robin action selection is only possible for a finite (non-continuous) action set.
 #. Add the following lines add the end of "setActionSpace"::

	# We can only deal with one-dimensional action spaces
        assert self.actionSpace.getNumberOfDimensions() == 1
        # Get a list of all actions this agent might take
        self.actions = self.actionSpace.getActionList()
        # Get name of action dimension
        self.actionDimensionName = self.actionSpace.getDimensionNames()[0]
        # Create an iterator that iterates in a round-robin manner over available actions
        self.nextActionIterator = __import__("itertools").cycle(self.actions)
 #. Reimplement the method "getAction()" as follows::

	    def getAction(self, **kwargs):
		""" Request the next action the agent want to execute """
		# Get next action  from iterator
		# We are only interested in the value of the first (and only) dimension,
		# thus the "0"
		nextAction = self.nextActionIterator.next()[0]
		# Create a dictionary that maps dimension name to chosen action
		actionDictionary = {self.actionDimensionName : nextAction}
		# Generate mmlf.framework.protocol.ActionTaken object
		return self._generateActionObject(actionDictionary)
 #. Remove superfluous methods "setStateSpace()", "setState()", "giveReward", and "nextEpisodeStarted()". They can use the default implementation of the AgentBase class
 #. Set thet AgentClass module attribute appropriately: "AgentClass = ExampleAgent"
 #. Set the AgentName to something meaningful: "AgentName = "RoundRobin""
 #. Do not forget to update the comments and the documentation of your new module!

That's it! Your agent module should now look like shown :ref:`here <example_agent>`. You can test it in the GUI by selecting "RoundRobin" as agent.

.. _random_agent:

RandomAgent
------------

.. literalinclude:: ../../mmlf/agents/random_agent.py
   :language: python

.. _example_agent:

ExampleAgent
-------------

.. literalinclude:: ../../mmlf/agents/example_agent.py
   :language: python