Sim2Real problem

[chan-yuu]

I want to drive a real car now. Does metadrive have Replay Mode or Kinematic Simulation?

That is, the positioning and control of the vehicle is controlled by the real vehicle, while the perceived environment of reinforcement learning is still realized by the sensors of the vehicle in the simulation software.

This issue is very important for me to implement the migration of sim2real, please let me know if there are already related attempts.

[QuanyiLi]

Hi,

What did you mean by real car? An AV stack like Apollo and openpilot? We have tried to connect MetaDrive and openpilot before, like on this page: https://www.youtube.com/watch?v=KjlPB0nCTvg&list=TLGGXghNFA-rcGMyNzExMjAyNQ&t=9s

We do have a replay mode/kinematic sim that allows you to set the position/velocity of the ego vehicle directly. If your planner output is waypoints, you may need low-level control algorithms to convert it into velocity or engine force. Or you can directly set the car to the target position (teleport).

For setting velocity, you can use set_velocity directly. Its usage can be found in plenty of test script like:

metadrive/metadrive/tests/test_component/test_vehicle_coordinates.py

Line 24 in 85e5dad

env.agent.set_velocity([1, 0], 10)

For the replay model (set_position), it is widely used in the ScenarioTrafficManager to replay surrounding vehicles.

[chan-yuu]

Thank you for your reply.

I'm talking about a real vehicle driving in the real physical world
, and I'm plugging in its gps location mapping into metadrive.
I wrote a test program:

import numpy as np
import math
from metadrive.envs.metadrive_env import MetaDriveEnv
from metadrive.component.sensors.rgb_camera import RGBCamera
import time

def run_kinematic_mode():
    config = {
        "use_render": True,           
        "manual_control": False,      
        "traffic_density": 0.05,       
        "map": "S",            
        
        "sensors": {
            "rgb_camera": (RGBCamera, 512, 512)
        },
        "vehicle_config": {
            "lidar": {"num_lasers": 240, "distance": 50, "num_others": 0},
            "show_lidar": True,
            "show_navi_mark": False,
        }
    }

    env = MetaDriveEnv(config)
    
    try:
        o, info = env.reset()
        vehicle = env.vehicle  
        
        center_x, center_y = 50, 0
        radius = 20
        
        for i in range(1000):
            start_time = time.time()
            angle = i * 0.05
            target_x = center_x + radius * np.cos(angle)
            target_y = center_y + radius * np.sin(angle)
            target_heading = angle + np.pi / 2 # just like gps location
                        
            vehicle.set_position([target_x, target_y])
            
            vehicle.set_heading_theta(target_heading)
            
            vehicle.set_velocity([0, 0]) 
            vehicle.set_angular_velocity(0)

            o, reward, terminated, truncated, info = env.step([0, 0])
                        
            if info["crash"]:
                print(f"Step {i}: Crashed: {info.get('crash_object', 'Unknown')}")
            
            lidar_data = o
            # image_data = vehicle.image_sensors["rgb_camera"].get_image(vehicle)

            # if terminated or truncated:
            #     env.reset()
            end_time = time.time()
            print("time.time:", time.time())
            print("spend time: ", end_time - start_time) #0.07s

    finally:
        env.close()

if __name__ == "__main__":
    run_kinematic_mode()

I just need to update its location, and then I can get the observations (lidar, camera) I want from metadrive, which I think is more suitable for my purpose. Next I'm going to plug in real gps as a vehicle locator.

By the way, if I have a gps path (reference line), how do I make a map of metadrive? Is there an easy way to use it?

[QuanyiLi]

The easiest way to do this is using the ScenarioMap, which takes a dictionary of map features as input, like the one introduced here: https://metadrive-simulator.readthedocs.io/en/latest/scenario_description.html#what-is-the-scenario-description The map_data argument is the dictionary containing map features like lanes/polylines.

A drawback of this method is that it cannot work with the rule-based traffic system, which requires the definition of the road network.... But if you just want to have other vehicle's trajectories fixed, you can still load traffic like what we did in ScenarioTrafficManager. The vehicles loaded in this way would follow a trajectory with basic emergency break ability.

BTW, your test program looks good.

[chan-yuu]

Thank you for response. I read through the scenario generation stuff, and I wrote a test program:

import pickle
import numpy as np
import os
from metadrive.envs.scenario_env import ScenarioEnv
from metadrive.type import MetaDriveType
from metadrive.policy.replay_policy import ReplayEgoCarPolicy
from metadrive.scenario.scenario_description import ScenarioDescription

def create_custom_scenario(scenario_id="custom_001"):
    length = 40
    
    # --- 1. Map Features ---
    map_features = {}
    lane_id = "lane_0"
    map_features[lane_id] = {
        "type": MetaDriveType.LANE_SURFACE_STREET,
        "polyline": np.array([[0, 0], [50, 0], [55, 20], [65, 38]], dtype=np.float32), 
    }

    # --- 2. Tracks ---
    tracks = {}
    sdc_id = "sdc"
    
    # tracking path:
    positions = np.zeros((length, 3), dtype=np.float32)
    positions[:, 0] = np.linspace(0, 40, length, dtype=np.float32)
    
    headings = np.zeros((length, ), dtype=np.float32)
    velocity = np.zeros((length, 2), dtype=np.float32)
    valid = np.ones((length, ), dtype=int)
    
    tracks[sdc_id] = {
        "type": MetaDriveType.VEHICLE,
        "state": {
            "position": positions,
            "heading": headings,
            "velocity": velocity,
            "valid": valid,
            "length": np.ones((length, ), dtype=np.float32) * 4.5,
            "width": np.ones((length, ), dtype=np.float32) * 2.0,
            "height": np.ones((length, ), dtype=np.float32) * 1.5,
        },
        "metadata": {
            "type": MetaDriveType.VEHICLE,
            "track_length": length,
            "object_id": sdc_id
        }
    }

    # --- 3. scenario_data ---
    scenario_data = {
        ScenarioDescription.ID: scenario_id,
        
        # Version
        ScenarioDescription.VERSION: "MetaDrive v0.4.3",
        
        ScenarioDescription.LENGTH: length,
        
        ScenarioDescription.METADATA: {
            "dataset": "custom",
            "sdc_id": sdc_id,
            "ts": np.arange(length, dtype=np.float32) * 0.1,
            "scenario_id": scenario_id,
            "coordinate": "metadrive",
            "metadrive_processed": True, 
        },
        
        ScenarioDescription.TRACKS: tracks,
        ScenarioDescription.MAP_FEATURES: map_features,
        ScenarioDescription.DYNAMIC_MAP_STATES: {} 
    }
    
    return scenario_data

def save_dataset(output_dir="my_custom_dataset"):
    os.makedirs(output_dir, exist_ok=True)
    
    scenario_id = "custom_scenario_0"
    data = create_custom_scenario(scenario_id)
    
    file_name = f"sd_{scenario_id}.pkl" 
    file_path = os.path.join(output_dir, file_name)
    
    with open(file_path, "wb") as f:
        pickle.dump(data, f)
        
    summary = {
        file_name: {
            "id": scenario_id,
            "object_summary": {},
            "number_summary": {}, 
            "metadata": data[ScenarioDescription.METADATA]
        }
    }
    
    with open(os.path.join(output_dir, "dataset_summary.pkl"), "wb") as f:
        pickle.dump(summary, f)
        
    mapping = {file_name: ""}
    with open(os.path.join(output_dir, "dataset_mapping.pkl"), "wb") as f:
        pickle.dump(mapping, f)
        
    print(f"Dataset created at: {os.path.abspath(output_dir)}")
    return output_dir

if __name__ == "__main__":
    # 1. dataset
    dataset_path = save_dataset()
    
    # 2. env
    env = ScenarioEnv(dict(
        use_render=True,             
        data_directory=dataset_path, # dataset_summary.pkl
        num_scenarios=1,
        start_scenario_index=0,
        agent_policy=ReplayEgoCarPolicy, # tracking the path
    ))
    
    try:
        env.reset(seed=0)
        for i in range(1000):
            # Replay: not use
            o, r, d, t, info = env.step([0, 0])
            env.render(text={"Step": i, "Pos": env.agent.position})
            
            if d or t:
                print("Episode finished.")
                env.reset()
    except Exception as e:
        print(f"An error occurred: {e}")
        raise e
    finally:
        env.close()

There are quite a few bugs during this period. But now I think it pretty much got what I wanted.

[chan-yuu]

I can generate more environment maps use only gps track data:

[guofan999]

I'm trying to create a custom map. I've defined the road boundaries and road information：

    map_features["center_yellow_line"] = {
        "type": center_line_type, 
        "polyline": center_line,
    }
    
    map_features["boundary_right"] = {
        "type": MetaDriveType.LINE_SOLID_SINGLE_WHITE,
        "polyline": bound_right,
    }
    
    map_features["boundary_left"] = {
        "type": MetaDriveType.LINE_SOLID_SINGLE_WHITE,
        "polyline": bound_left,
    }

but it seems the road width cannot be modified:

    map_features["lane_right"] = {
        "type": MetaDriveType.LANE_SURFACE_STREET,
        "polyline": lane_right_points,
    }
    
    map_features["lane_left"] = {
        "type": MetaDriveType.LANE_SURFACE_STREET,
        "polyline": lane_left_points,
    }

As shown in the image, I noticed the road width is consistently only 3.5m. Do I need to modify its type(LANE_SURFACE_STREET)? I can't find where to change the road width parameter.

[guofan999]

I found that there was no road width configuration in scenario's env, but I found this width information in metadrive_env：

    map_config={
        BaseMap.GENERATE_TYPE: MapGenerateMethod.BIG_BLOCK_NUM,
        BaseMap.GENERATE_CONFIG: None,  # it can be a file path / block num / block ID sequence
        BaseMap.LANE_WIDTH: 3.5,
        BaseMap.LANE_NUM: 3,
        "exit_length": 50,
        "start_position": [0, 0],
    },

I noticed the default road width is 6 meters. How can I change it?

    env = ScenarioEnv(dict(
        use_render=True,            
        data_directory=dataset_path, 
        num_scenarios=1,
        start_scenario_index=0,
        manual_control=True,
        agent_policy=ManualControlPolicy,
         
        map_region_size=1024, 
    ))