Hallucination even with Four Entities in Schema

[mann2107]

Schema

flight_booking_time_schema = Object(
    id="flight_booking_time_info",
    description="Time-specific information in a flight booking request.",
    attributes=[
        Text(
            id="departure_from_origin_time",
            description="The preferred time when the traveller wants to depart from the origin location",
            examples=[
                ("I want to depart at 9a", "9a"),
                ("Leaving ideally in the morning", "morning"),
                ("I'll need to fly from DFW on sunday at around 10 pm.", "10 pm"),
    
            ],
        ),
        Text(
            id="return_from_destination_time",
            description="The preferred time when the traveller wants to return from the destination location",
            examples=[
                ("I prefer returning at 6p", "6p"),
                ("Expected return in the evening", "evening"),
                ("I'll be returning from Moscow on the 9th Feb, around 6 am.", "6 am"),
            ],
        ),
        Text(
            id="reach_at_destination_time",
            description="The preferred time when the traveller wants to reach the destination",
            examples=[
                ("I want to arrive at the Chicago by 5:30 PM", "5:30 PM"),
                ("Prefer to reach by afternoon", "afternoon"),
            ],
        ),
        Text(
            id="arrival_at_origin_time",
            description="The preferred time when the traveller wants to arrive back at the origin location",
            examples=[
                ("I want to come back till 3:00 PM", "3:00 PM"),
                ("Arrival at origin ideally by 7:30 AM", "7:30 AM")
            ],
        ),
    ],
    examples=[
        (
            "I'm looking for a flight departing tomorrow morning and returning at 4:30 PM.",
            [
                {
                    "departure_from_origin_time": "morning", "return_from_destination_time": "4:30 PM", "reach_at_destination_time": "", "arrival_at_origin_time": ""
                }
            ],
        ),
        (
            "I need to arrive at my destination by 6:00 PM tomorrow. Departure should be around 6:30 PM, and return at 9:00 AM next Monday.",
            [
                {
                    "departure_from_origin_time": "6:30 PM", "return_from_destination_time": "9:00 AM", "reach_at_destination_time": "6:00 PM", "arrival_at_origin_time": ""
                }
            ],
        ),
        (
            "I'll need to come back home to Indore on the 25th of February, around 7 am.",
            [
                {
                    "departure_from_origin_time": "", "return_from_destination_time": "", "reach_at_destination_time": "", "arrival_at_origin_time": "7 am"
                }
            ],
        ),
        (
            "I need to book a business trip. I'll need to fly from Bengaluru on 20th Jan 2024 at around 5 pm. After the meeting, I'll be returning from LGA on the 25th of Jan 2024, around 7 am.",
            [
                {
                    "departure_from_origin_time": "5 pm", "return_from_destination_time": "7 am", "reach_at_destination_time": "", "arrival_at_origin_time": ""
                }
            ],
        ),
    ],
    many=False,
)

Output

{'text': {'data': {'flight_booking_time_info': [{'departure_from_origin_time': '5 pm',
     'return_from_destination_time': '7 am',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'hotel_booking_info: {',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'check_in_date: "20th Jan 2024",',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'check_out_date: "25th Jan 2024"',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': '}',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'car_rental_booking_info: {',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'pick_up_date: "20th Jan 2024",',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'drop_off_date: "25th Jan 2024"',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': '}',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'preferred_airlines: {',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'outbound_airline: "American Airlines",',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'outbound_seat_preference: "Window Seat",',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'return_airline: "Jet Blue",',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'return_seat_preference: "Aisle Seat",',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': 'max_fare: "$500"',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''},
    {'departure_from_origin_time': '}',
     'return_from_destination_time': '',
     'reach_at_destination_time': '',
     'arrival_at_origin_time': ''}]},
  'raw': 'departure_from_origin_time|return_from_destination_time|reach_at_destination_time|arrival_at_origin_time\n5 pm|7 am||\n\nhotel_booking_info: {\n  check_in_date: "20th Jan 2024",\n  check_out_date: "25th Jan 2024"\n}\n\ncar_rental_booking_info: {\n  pick_up_date: "20th Jan 2024",\n  drop_off_date: "25th Jan 2024"\n}\n\npreferred_airlines: {\n  outbound_airline: "American Airlines",\n  outbound_seat_preference: "Window Seat",\n  return_airline: "Jet Blue",\n  return_seat_preference: "Aisle Seat",\n  max_fare: "$500"\n}',
  'errors': [],
  'validated_data': {}}}

llm

gpt-3.5-turbo

Input

Hello there, I hope this message finds you well.
I need to book a business trip. I'll need to fly from Bengaluru on 20th Jan 2024 at around 5 pm.
After the meeting, I'll be returning from LGA on the 25th of Jan 2024, around 7 am.
I would require a hotel and car rental facility also at my destination place.
Please confirm the booking as soon as possible. I would prefer American Airlines with Window Seat in the Outbound.
Also Book me a Aisle seat in Jet Blue in the return Journey. If You can take care that the fare should not be more than $500 that would be great.

Thanks!

[eyurtsev]

Reference the following guidelines to see if anything can be improved: https://eyurtsev.github.io/kor/guidelines.html

My main suggestions would be:

• If working with multi-paragraph text, specify an input_formatter of "triple_quotes" when creating the chain.
• If you're already providing object level examples, then get rid of the per field examples (those might just confuse the model). Try to feed the longer single paragraph example and build from there.
• Print out the prompt: https://eyurtsev.github.io/kor/objects.html#what-s-the-actual-prompt to make sure that the prompt looks correct -- it would be wrong if there's a bug in how the examples were provided (e.g., extra dict literal specified somewhere etc.)
• Swap to validating with pydantic -- it's an easier work-flow: https://eyurtsev.github.io/kor/validation.html

You can always try function calling with OpenAI models -- they may result in better quality (or not). I think the key thing is to make sure that examples are provided.