State and Action Spaces

Modules for state and action spaces

State and action spaces define the range of possible states the agent might perceive and the actions that are available to the agent. These spaces are dict-like objects that map dimension names (the dict keys) to dimension objects that contain information about this dimension. The number of items in this dict-like structure is the dimensionality of the (state/action) space.

Single dimension of a space

class framework.spaces.Dimension(dimensionName, dimensionType, dimensionValues, limitType=None)

A single dimension of a (state or action) space

A dimension is either continuous or discrete. A “discrete” dimension might take on only a finite, discrete number of values.

For instance, consider a dimension of a state space describing the color of a fruit. This dimension take on the values “red”, “blue”, or “green”. In contrast, consider a second “continuous” dimension, e.g. the weight of a fruit. This weight might be somewhere between 0g and 1000g. If we allow any arbitrary weight (not only full gramms), the dimension is truly continuous.

This properties of a dimension can be checked using the method:

• isDiscrete : Returns whether the respective dimension is discrete

• isContinuous : Returns whether the respective dimension is continuous

• getValueRanges : Returns the allowed values a continuous! dimension

  might take on.

• getValues : Returns the allowed values a discrete! dimension

  might take on.

Base class for state and action spaces

class framework.spaces.Space

Base class for state and action spaces.

Class which represents the state space of an environment or the action space of an agent.

This is essentially a dictionary whose keys are the names of the dimensions, and whose values are Dimension objects.

addContinuousDimension(dimensionName, dimensionValues, limitType='soft')

Add the named continuous dimension to the space.

dimensionValues is a list of (rangeStart, rangeEnd) 2-tuples which define the valid ranges of this dimension. (i.e. [(0, 50), (75.5, 82)] )

If limitType is set to “hard”, then the agent is responsible to check that the limits are not exceeded. When it is set to “soft”, then the agent should not expect that all the values of this dimension will be strictly within the bounds of the specified ranges, but that the ranges serve as an approximate values of where the values will be (i.e. as [mean-std.dev., mean+std.dev] instead of [absolute min. value, absolute max. value])

addDiscreteDimension(dimensionName, dimensionValues)

Add the named continuous dimension to the space.

dimensionValues is a list of strings representing possible discrete states of this dimension. (i.e. [“red”, “green”, “blue”])

addOldStyleSpace(oldStyleSpace, limitType='soft')

Takes an old-style (using the old format) space dictionary, and adds its dimensions to this object.

getDimensionNames()

Return the names of the space dimensions

getDimensions()

Return the names of the space dimensions

getNumberOfDimensions()

Returns how many dimensions this space has

hasContinuousDimensions()

Return whether this space has continuous dimensions

hasDiscreteDimensions()

Return whether this space has discrete dimensions

State spaces

class framework.spaces.StateSpace

Specialization of Space for state spaces.

For instance, a state space could be defined as follows:

{ "color": Dimension(dimensionType = "discrete",
                     dimensionValues = ["red","green", "blue"]),
  "weight": Dimension(dimensionType = "continuous",
                      dimensionValues = [(0,1000)]) }

This state space has two dimensions (“color” and “weight”), a discrete and a continuous one. The discrete dimension “color” can take on three values (“red”,”green”, or “blue”) and the continuous dimension “weight” any value between 0 and 1000.

A valid state of the state space defined above would be:

s1 = {"color": "red", "weight": 300}

Invalid states (s2 since the color is invalid and s3 since its weight is too large):

s2 = {"color": "yellow", "weight": 300}
s3 = {"color": "red", "weight": 1300}

The class provides additional methods for checking if a certain state is valid according to this state space (isValidState) and to scale a state such that it lies within a certain interval (scaleState).

Action spaces

class framework.spaces.ActionSpace

Specialization of Space for action spaces.

For instance, an action space could be defined as follows:

{ "gasPedalForce": ("discrete", ["low", "medium", "floored"]),
  "steeringWheelAngle": ("continuous", [(-120,120)]) }

This action space has two dimensions (“gasPedalForce” and “steeringWheelAngle”), a discrete and a continuous one. The discrete dimension “gasPedalForce” can take on three values (“low”,”medium”, or “floored”) and the continuous dimension “steeringWheelAngle” any value between -120 and 120.

A valid action according to this action space would be:

a1 = {"gasPedalForce": "low", "steeringWheelAngle": -50}

Invalid actions (a2 since the gasPedalForce is invalid and s3 since its steeringWheelAngle is too small):

a2 = {"gasPedalForce": "extreme", "steeringWheelAngle": 30}
a3 = {"gasPedalForce": "medium", "steeringWheelAngle": -150}

The class provides additional methods for discretizing an action space (discretizedActionSpace) and to return a list of all available actions (getActionList).