Environment Embeddings¶
In autoregressive policies, environment embeddings transfer data from feature space to hidden space:
- Initial Embeddings: encode global problem features
- Context Embeddings: modify current node embedding during decoding
- Dynamic Embeddings: modify all nodes embeddings during decoding
Context Embeddings¶
The context embedding is used to modify the query embedding of the problem node of the current partial solution. Usually consists of a projection of gathered node embeddings and features to the embedding space.
Classes:
-
EnvContext
–Base class for environment context embeddings. The context embedding is used to modify the
-
FFSPContext
– -
TSPContext
–Context embedding for the Traveling Salesman Problem (TSP).
-
VRPContext
–Context embedding for the Capacitated Vehicle Routing Problem (CVRP).
-
VRPTWContext
–Context embedding for the Capacitated Vehicle Routing Problem (CVRP).
-
SkillVRPContext
–Context embedding for the Skill Vehicle Routing Problem (SkillVRP).
-
PCTSPContext
–Context embedding for the Prize Collecting TSP (PCTSP).
-
OPContext
–Context embedding for the Orienteering Problem (OP).
-
DPPContext
–Context embedding for the Decap Placement Problem (DPP), EDA (electronic design automation).
-
PDPContext
–Context embedding for the Pickup and Delivery Problem (PDP).
-
MTSPContext
–Context embedding for the Multiple Traveling Salesman Problem (mTSP).
-
SMTWTPContext
–Context embedding for the Single Machine Total Weighted Tardiness Problem (SMTWTP).
-
MDCPDPContext
–Context embedding for the MDCPDP.
-
MTVRPContext
–Context embedding for Multi-Task VRPEnv.
Functions:
-
env_context_embedding
–Get environment context embedding. The context embedding is used to modify the
EnvContext
¶
EnvContext(
embed_dim, step_context_dim=None, linear_bias=False
)
Bases: Module
Base class for environment context embeddings. The context embedding is used to modify the query embedding of the problem node of the current partial solution. Consists of a linear layer that projects the node features to the embedding space.
Source code in rl4co/models/nn/env_embeddings/context.py
51 52 53 54 55 |
|
FFSPContext
¶
FFSPContext(embed_dim, stage_cnt=None)
Bases: EnvContext
Source code in rl4co/models/nn/env_embeddings/context.py
74 75 76 77 78 79 |
|
TSPContext
¶
TSPContext(embed_dim)
Bases: EnvContext
Context embedding for the Traveling Salesman Problem (TSP). Project the following to the embedding space:
- first node embedding
- current node embedding
Source code in rl4co/models/nn/env_embeddings/context.py
108 109 110 111 112 |
|
VRPContext
¶
VRPContext(embed_dim)
Bases: EnvContext
Context embedding for the Capacitated Vehicle Routing Problem (CVRP). Project the following to the embedding space:
- current node embedding
- remaining capacity (vehicle_capacity - used_capacity)
Source code in rl4co/models/nn/env_embeddings/context.py
146 147 148 149 |
|
VRPTWContext
¶
VRPTWContext(embed_dim)
Bases: VRPContext
Context embedding for the Capacitated Vehicle Routing Problem (CVRP). Project the following to the embedding space:
- current node embedding
- remaining capacity (vehicle_capacity - used_capacity)
- current time
Source code in rl4co/models/nn/env_embeddings/context.py
164 165 166 167 |
|
SkillVRPContext
¶
SkillVRPContext(embed_dim)
Bases: EnvContext
Context embedding for the Skill Vehicle Routing Problem (SkillVRP). Project the following to the embedding space:
- current node embedding
- current technician
Source code in rl4co/models/nn/env_embeddings/context.py
182 183 184 185 |
|
PCTSPContext
¶
PCTSPContext(embed_dim)
Bases: EnvContext
Context embedding for the Prize Collecting TSP (PCTSP). Project the following to the embedding space:
- current node embedding
- remaining prize (prize_required - cur_total_prize)
Source code in rl4co/models/nn/env_embeddings/context.py
199 200 |
|
OPContext
¶
OPContext(embed_dim)
Bases: EnvContext
Context embedding for the Orienteering Problem (OP). Project the following to the embedding space:
- current node embedding
- remaining distance (max_length - tour_length)
Source code in rl4co/models/nn/env_embeddings/context.py
216 217 |
|
DPPContext
¶
DPPContext(embed_dim)
Bases: EnvContext
Context embedding for the Decap Placement Problem (DPP), EDA (electronic design automation). Project the following to the embedding space:
- current cell embedding
Methods:
-
forward
–Context cannot be defined by a single node embedding for DPP, hence 0.
Source code in rl4co/models/nn/env_embeddings/context.py
230 231 |
|
forward
¶
forward(embeddings, td)
Context cannot be defined by a single node embedding for DPP, hence 0. We modify the dynamic embedding instead to capture placed items
Source code in rl4co/models/nn/env_embeddings/context.py
233 234 235 236 237 |
|
PDPContext
¶
PDPContext(embed_dim)
Bases: EnvContext
Context embedding for the Pickup and Delivery Problem (PDP). Project the following to the embedding space:
- current node embedding
Source code in rl4co/models/nn/env_embeddings/context.py
246 247 |
|
MTSPContext
¶
MTSPContext(embed_dim, linear_bias=False)
Bases: EnvContext
Context embedding for the Multiple Traveling Salesman Problem (mTSP). Project the following to the embedding space:
- current node embedding
- remaining_agents
- current_length
- max_subtour_length
- distance_from_depot
Source code in rl4co/models/nn/env_embeddings/context.py
264 265 266 267 268 269 |
|
SMTWTPContext
¶
SMTWTPContext(embed_dim)
Bases: EnvContext
Context embedding for the Single Machine Total Weighted Tardiness Problem (SMTWTP). Project the following to the embedding space:
- current node embedding
- current time
Source code in rl4co/models/nn/env_embeddings/context.py
300 301 |
|
MDCPDPContext
¶
MDCPDPContext(embed_dim)
Bases: EnvContext
Context embedding for the MDCPDP. Project the following to the embedding space:
- current node embedding
Source code in rl4co/models/nn/env_embeddings/context.py
318 319 |
|
MTVRPContext
¶
MTVRPContext(embed_dim)
Bases: VRPContext
Context embedding for Multi-Task VRPEnv. Project the following to the embedding space:
- current node embedding
- remaining_linehaul_capacity (vehicle_capacity - used_capacity_linehaul)
- remaining_backhaul_capacity (vehicle_capacity - used_capacity_backhaul)
- current time
- current_route_length
- open route indicator
Source code in rl4co/models/nn/env_embeddings/context.py
350 351 352 353 |
|
env_context_embedding
¶
Get environment context embedding. The context embedding is used to modify the query embedding of the problem node of the current partial solution. Usually consists of a projection of gathered node embeddings and features to the embedding space.
Parameters:
-
env
–Environment or its name.
-
config
(dict
) –A dictionary of configuration options for the environment.
Source code in rl4co/models/nn/env_embeddings/context.py
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
|
Dynamic Embeddings¶
The dynamic embedding is used to modify query, key and value vectors of the attention mechanism based on the current state of the environment (which is changing during the rollout). Generally consists of a linear layer that projects the node features to the embedding space.
Classes:
-
StaticEmbedding
–Static embedding for general problems.
-
SDVRPDynamicEmbedding
–Dynamic embedding for the Split Delivery Vehicle Routing Problem (SDVRP).
Functions:
-
env_dynamic_embedding
–Get environment dynamic embedding. The dynamic embedding is used to modify query, key and value vectors of the attention mechanism
StaticEmbedding
¶
StaticEmbedding(*args, **kwargs)
Bases: Module
Static embedding for general problems. This is used for problems that do not have any dynamic information, except for the information regarding the current action (e.g. the current node in TSP). See context embedding for more details.
Source code in rl4co/models/nn/env_embeddings/dynamic.py
53 54 |
|
SDVRPDynamicEmbedding
¶
SDVRPDynamicEmbedding(embed_dim, linear_bias=False)
Bases: Module
Dynamic embedding for the Split Delivery Vehicle Routing Problem (SDVRP). Embed the following node features to the embedding space:
- demand_with_depot: demand of the customers and the depot
The demand with depot is used to modify the query, key and value vectors of the attention mechanism based on the current state of the environment (which is changing during the rollout).
Source code in rl4co/models/nn/env_embeddings/dynamic.py
68 69 70 |
|
env_dynamic_embedding
¶
Get environment dynamic embedding. The dynamic embedding is used to modify query, key and value vectors of the attention mechanism based on the current state of the environment (which is changing during the rollout). Consists of a linear layer that projects the node features to the embedding space.
Parameters:
-
env
–Environment or its name.
-
config
(dict
) –A dictionary of configuration options for the environment.
Source code in rl4co/models/nn/env_embeddings/dynamic.py
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
|
Init Embeddings¶
The init embedding is used to initialize the general embedding of the problem nodes without any solution information. Generally consists of a linear layer that projects the node features to the embedding space.
Classes:
-
TSPInitEmbedding
–Initial embedding for the Traveling Salesman Problems (TSP).
-
MatNetInitEmbedding
–Preparing the initial row and column embeddings for MatNet.
-
VRPInitEmbedding
–Initial embedding for the Vehicle Routing Problems (VRP).
-
PCTSPInitEmbedding
–Initial embedding for the Prize Collecting Traveling Salesman Problems (PCTSP).
-
OPInitEmbedding
–Initial embedding for the Orienteering Problems (OP).
-
DPPInitEmbedding
–Initial embedding for the Decap Placement Problem (DPP), EDA (electronic design automation).
-
MDPPInitEmbedding
–Initial embedding for the Multi-port Placement Problem (MDPP), EDA (electronic design automation).
-
PDPInitEmbedding
–Initial embedding for the Pickup and Delivery Problem (PDP).
-
MTSPInitEmbedding
–Initial embedding for the Multiple Traveling Salesman Problem (mTSP).
-
SMTWTPInitEmbedding
–Initial embedding for the Single Machine Total Weighted Tardiness Problem (SMTWTP).
-
MDCPDPInitEmbedding
–Initial embedding for the MDCPDP environment
Functions:
-
env_init_embedding
–Get environment initial embedding. The init embedding is used to initialize the
TSPInitEmbedding
¶
TSPInitEmbedding(embed_dim, linear_bias=True)
Bases: Module
Initial embedding for the Traveling Salesman Problems (TSP). Embed the following node features to the embedding space:
- locs: x, y coordinates of the cities
Source code in rl4co/models/nn/env_embeddings/init.py
56 57 58 59 |
|
MatNetInitEmbedding
¶
Bases: Module
Preparing the initial row and column embeddings for MatNet.
Source code in rl4co/models/nn/env_embeddings/init.py
76 77 78 79 80 81 82 83 84 |
|
VRPInitEmbedding
¶
VRPInitEmbedding(
embed_dim, linear_bias=True, node_dim: int = 3
)
Bases: Module
Initial embedding for the Vehicle Routing Problems (VRP). Embed the following node features to the embedding space:
- locs: x, y coordinates of the nodes (depot and customers separately)
- demand: demand of the customers
Source code in rl4co/models/nn/env_embeddings/init.py
117 118 119 120 121 |
|
PCTSPInitEmbedding
¶
PCTSPInitEmbedding(embed_dim, linear_bias=True)
Bases: Module
Initial embedding for the Prize Collecting Traveling Salesman Problems (PCTSP). Embed the following node features to the embedding space:
- locs: x, y coordinates of the nodes (depot and customers separately)
- expected_prize: expected prize for visiting the customers.
In PCTSP, this is the actual prize. In SPCTSP, this is the expected prize.
- penalty: penalty for not visiting the customers
Source code in rl4co/models/nn/env_embeddings/init.py
181 182 183 184 185 |
|
OPInitEmbedding
¶
OPInitEmbedding(embed_dim, linear_bias=True)
Bases: Module
Initial embedding for the Orienteering Problems (OP). Embed the following node features to the embedding space:
- locs: x, y coordinates of the nodes (depot and customers separately)
- prize: prize for visiting the customers
Source code in rl4co/models/nn/env_embeddings/init.py
212 213 214 215 216 |
|
DPPInitEmbedding
¶
DPPInitEmbedding(embed_dim, linear_bias=True)
Bases: Module
Initial embedding for the Decap Placement Problem (DPP), EDA (electronic design automation). Embed the following node features to the embedding space:
- locs: x, y coordinates of the nodes (cells)
- probe: index of the (single) probe cell. We embed the euclidean distance from the probe to all cells.
Source code in rl4co/models/nn/env_embeddings/init.py
241 242 243 244 245 |
|
MDPPInitEmbedding
¶
MDPPInitEmbedding(embed_dim, linear_bias=True)
Bases: Module
Initial embedding for the Multi-port Placement Problem (MDPP), EDA (electronic design automation). Embed the following node features to the embedding space:
- locs: x, y coordinates of the nodes (cells)
- probe: indexes of the probe cells (multiple). We embed the euclidean distance of each cell to the closest probe.
Source code in rl4co/models/nn/env_embeddings/init.py
267 268 269 270 271 272 273 274 |
|
PDPInitEmbedding
¶
PDPInitEmbedding(embed_dim, linear_bias=True)
Bases: Module
Initial embedding for the Pickup and Delivery Problem (PDP). Embed the following node features to the embedding space:
- locs: x, y coordinates of the nodes (depot, pickups and deliveries separately)
Note that pickups and deliveries are interleaved in the input.
Source code in rl4co/models/nn/env_embeddings/init.py
299 300 301 302 303 304 |
|
MTSPInitEmbedding
¶
MTSPInitEmbedding(embed_dim, linear_bias=True)
Bases: Module
Initial embedding for the Multiple Traveling Salesman Problem (mTSP). Embed the following node features to the embedding space:
- locs: x, y coordinates of the nodes (depot, cities)
Source code in rl4co/models/nn/env_embeddings/init.py
326 327 328 329 330 331 |
|
SMTWTPInitEmbedding
¶
SMTWTPInitEmbedding(embed_dim, linear_bias=True)
Bases: Module
Initial embedding for the Single Machine Total Weighted Tardiness Problem (SMTWTP). Embed the following node features to the embedding space:
- job_due_time: due time of the jobs
- job_weight: weights of the jobs
- job_process_time: the processing time of jobs
Source code in rl4co/models/nn/env_embeddings/init.py
347 348 349 350 |
|
MDCPDPInitEmbedding
¶
MDCPDPInitEmbedding(embed_dim, linear_bias=True)
Bases: Module
Initial embedding for the MDCPDP environment Embed the following node features to the embedding space:
- locs: x, y coordinates of the nodes (depot, pickups and deliveries separately)
Note that pickups and deliveries are interleaved in the input.
Source code in rl4co/models/nn/env_embeddings/init.py
368 369 370 371 372 373 |
|
env_init_embedding
¶
Get environment initial embedding. The init embedding is used to initialize the general embedding of the problem nodes without any solution information. Consists of a linear layer that projects the node features to the embedding space.
Parameters:
-
env
–Environment or its name.
-
config
(dict
) –A dictionary of configuration options for the environment.
Source code in rl4co/models/nn/env_embeddings/init.py
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
|