> For the complete documentation index, see [llms.txt](https://huang-jason.gitbook.io/ros/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://huang-jason.gitbook.io/ros/deep-rl-manipulator.md).

# ND2 - project - Deep RL manipulator

sensor input -> DQN -> actions

integrate with robots and simulator

PyTorch python -> API ->

pass memory objects between the user’s application and Torch without extra copies

1. create DQN agent
2. create **input\_state** memory
3. initialize state
4. loop
   1. assign **input\_state** base on previous state
   2. DQN agent return `action` base on **input\_state**
   3. recalculate current state
   4. compute `reward` base on previous & current state; send reward to DQN agent; update DQN memory
   5. Exit if game over

## ENUM

```cpp
enum Days { Saturday,Sunday,Tuesday,Wednesday,Thursday,Friday};
Days day = Saturday;
if(day == Saturday){
    std::cout<<"Ok its Saturday";
}
```

```cpp
// C++ 11
enum class Days
{
    SUNDAY,
    MONDAY,
    // ... etc.
}

// ...

if (day == Days::SUNDAY)
    // ...
```

<http://ot-note.logdown.com/posts/178610/global-and-static-variables>

## ARM Plugin

arm mode = gazebo-arm.world

ArmPlugin.cpp = gazebo plugin = creating the DQN agent and training it to learn

libgazeboArmPlugin.so = gazebo plugin shared object file = integrating the simulation environment with the RL agent

### ArmPlugin::Load()

creating and initializing nodes that subscribe to two specific topics - camera & contact sensor

```cpp
gazebo::transport::SubscriberPtr sub = node->Subscribe("topic_name", callback_function, class_instance);
```

class\_instance is the instance used when a new message is received.

[Subscribers in Gazebo](http://gazebosim.org/tutorials?tut=topics_subscribed)

[Gazebo API](http://osrf-distributions.s3.amazonaws.com/gazebo/api/dev/classgazebo_1_1transport_1_1Node.html#a13a67ebd4537a0057ae92f837bb3042f)

### ArmPlugin::onCameraMsg()

callback function for the camera subscriber

It takes the message from the camera topic, extracts the image, and saves it. This is then passed to the DQN.

### ArmPlugin::onCollisionMsg()

callback function for the object’s contact sensor(my\_contact).

Furthermore, this callback function can also be used to define a reward function based on whether there has been a collision or not.

### ArmPlugin::createAgent()

create and initialize the agent.

### ArmPlugin::updateAgent()

every camera frame received -> DQN agent take an action -> The `updateAgent()` method decides to take that action.

### ArmPlugin::OnUpdate()

issue rewards(end of episode || interim rewards) and train the DQN.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://huang-jason.gitbook.io/ros/deep-rl-manipulator.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.