ND2 - project - Deep RL manipulator

sensor input -> DQN -> actions

integrate with robots and simulator

PyTorch python -> API ->

pass memory objects between the user’s application and Torch without extra copies

1. create DQN agent

2. create input_state memory

3. initialize state

4. loop

   1. assign input_state base on previous state

   2. DQN agent return action base on input_state

   3. recalculate current state

   4. compute reward base on previous & current state; send reward to DQN agent; update DQN memory

   5. Exit if game over

ENUM

enum Days { Saturday,Sunday,Tuesday,Wednesday,Thursday,Friday};
Days day = Saturday;
if(day == Saturday){
    std::cout<<"Ok its Saturday";
}

// C++ 11
enum class Days
{
    SUNDAY,
    MONDAY,
    // ... etc.
}

// ...

if (day == Days::SUNDAY)
    // ...

http://ot-note.logdown.com/posts/178610/global-and-static-variables

ARM Plugin

arm mode = gazebo-arm.world

ArmPlugin.cpp = gazebo plugin = creating the DQN agent and training it to learn

libgazeboArmPlugin.so = gazebo plugin shared object file = integrating the simulation environment with the RL agent

ArmPlugin::Load()

creating and initializing nodes that subscribe to two specific topics - camera & contact sensor

gazebo::transport::SubscriberPtr sub = node->Subscribe("topic_name", callback_function, class_instance);

class_instance is the instance used when a new message is received.

Subscribers in Gazebo

Gazebo API

ArmPlugin::onCameraMsg()

callback function for the camera subscriber

It takes the message from the camera topic, extracts the image, and saves it. This is then passed to the DQN.

ArmPlugin::onCollisionMsg()

callback function for the object’s contact sensor(my_contact).

Furthermore, this callback function can also be used to define a reward function based on whether there has been a collision or not.

ArmPlugin::createAgent()

create and initialize the agent.

ArmPlugin::updateAgent()

every camera frame received -> DQN agent take an action -> The updateAgent() method decides to take that action.

ArmPlugin::OnUpdate()

issue rewards(end of episode || interim rewards) and train the DQN.