Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The current instance_score.summary is not readable #1588

Open
yoavkatz opened this issue Feb 9, 2025 · 0 comments
Open

The current instance_score.summary is not readable #1588

yoavkatz opened this issue Feb 9, 2025 · 0 comments

Comments

@yoavkatz
Copy link
Member

yoavkatz commented Feb 9, 2025

print("Instance Results:")
print(results.instance_scores.summary)

                                                                                                                                                                      text_a text_a_type  \
0                                                                                                                      The drain is clogged with hair. It has to be cleaned.     premise
1                                                                                                                       Jane knocked on Susan's door but she did not answer.     premise
2                                                                             Beth didn't get angry with Sally, who had cut her off, because she stopped and counted to ten.     premise
3  No one joins Facebook to be sad and lonely. But a new study from the University of Wisconsin psychologist George Lincoln argues that that's exactly how it makes us feel.     premise
4                                                                                                                     The man couldn't lift his son because he was so heavy.     premise

                                       text_b text_b_type                       classes type_of_relation           label  \
0                 The hair has to be cleaned.  hypothesis  [entailment, not entailment]       entailment      entailment
1                       Susan did not answer.  hypothesis  [entailment, not entailment]       entailment  not entailment
2           Sally stopped and counted to ten.  hypothesis  [entailment, not entailment]       entailment      entailment
3  That's exactly how Facebook makes us feel.  hypothesis  [entailment, not entailment]       entailment  not entailment
4                       The son was so heavy.  hypothesis  [entailment, not entailment]       entailment  not entailment



        source  \
0                                                                                                                                     [{'role': 'system', 'content': 'Given a premise and hypothesis cla
ssify the entailment of the hypothesis to one of entailment, not entailment.'}, {'role': 'user', 'content': 'premise: The drain is clogged with hair. It has to be cleaned.
hypothesis: The hair has to be cleaned.'}]
1                                                                                                                                            [{'role': 'system', 'content': 'Given a premise and hypothe
sis classify the entailment of the hypothesis to one of entailment, not entailment.'}, {'role': 'user', 'content': 'premise: Jane knocked on Susan's door but she did not answer.
hypothesis: Susan did not answer.'}]
2                                                                                      [{'role': 'system', 'content': 'Given a premise and hypothesis classify the entailment of the hypothesis to one o
f entailment, not entailment.'}, {'role': 'user', 'content': 'premise: Beth didn't get angry with Sally, who had cut her off, because she stopped and counted to ten.
hypothesis: Sally stopped and counted to ten.'}]
3  [{'role': 'system', 'content': 'Given a premise and hypothesis classify the entailment of the hypothesis to one of entailment, not entailment.'}, {'role': 'user', 'content': 'premise: No one joins
Facebook to be sad and lonely. But a new study from the University of Wisconsin psychologist George Lincoln argues that that's exactly how it makes us feel.
hypothesis: That's exactly how Facebook makes us feel.'}]
4                                                                                                                                          [{'role': 'system', 'content': 'Given a premise and hypothesi
s classify the entailment of the hypothesis to one of entailment, not entailment.'}, {'role': 'user', 'content': 'premise: The man couldn't lift his son because he was so heavy.
hypothesis: The son was so heavy.'}]

                                                  metrics                                                           postprocessors          target        references  \
0  [metrics.f1_micro, metrics.accuracy, metrics.f1_macro]  [processors.take_first_non_empty_line, processors.lower_case_till_punc]      entailment      [entailment]
1  [metrics.f1_micro, metrics.accuracy, metrics.f1_macro]  [processors.take_first_non_empty_line, processors.lower_case_till_punc]  not entailment  [not entailment]
2  [metrics.f1_micro, metrics.accuracy, metrics.f1_macro]  [processors.take_first_non_empty_line, processors.lower_case_till_punc]      entailment      [entailment]
3  [metrics.f1_micro, metrics.accuracy, metrics.f1_macro]  [processors.take_first_non_empty_line, processors.lower_case_till_punc]  not entailment  [not entailment]
4  [metrics.f1_micro, metrics.accuracy, metrics.f1_macro]  [processors.take_first_non_empty_line, processors.lower_case_till_punc]  not entailment  [not entailment]




                  prediction  \
0                                                                                                                              **Given Premise:** The drain is clogged with hair. It has to be cleaned.\
n\n**Given Hypothesis:** The hair has to be cleaned.\n\n**Classification of the Entailment**: Both the premise and the hypothesis are saying the same thing - that there is a need to clean the hair. Th
erefore, we can say that the hypothesis is an **entailment** of the premise. The hypothesis is a consequence or a necessary condition of the premise, and it is not plausible to hold the premise withou
t the hypothesis being true.
1
                                                                According to the logic:\n\nSince the premise states that Jane knocked on Susan's door, and the hypothesis further states that Susan did
not answer, we can conclude that the hypothesis ENTAILS the premise.\n\nReasoning: If Susan had answered, the premise would not have been true. So, the lack of answer from Susan supports the hypothesi
s that Susan did not answer.
2  The hypothesis "Sally stopped and counted to ten" entails the premise "Beth didn't get angry with Sally, who had cut her off, because she stopped and counted to ten." It's a mathematical certainty
that Beth didn't get angry because Sally's actions were not a cause of her anger.\n\nHere's why the entailment works:\n\n- If Sally stopped and counted to ten, then any other action (like cutting off
Beth) would not cause her to get angry.\n- Therefore, Beth did not get angry with Sally because she stopped and counted to ten.\n\nThis is an example of a categorical syllogism: **All A, because B.**
(All that part G was part H)
3
                                                                                    **Premise:** No one joins Facebook to be sad and lonely.\n\n**Hypothesis:** That's exactly how Facebook makes us fee
l.\n\nThis is an example of an **entailment**. The premise logically guarantees the presence of a particular outcome (being sad and lonely) for Facebook usage, making it a significant predictor for Fa
cebook users' mental states.
4

                                                                                                  The entailment is "not": The son was so heavy is an entailment of the premise. It contains the informa
tion that the son was heavy.

                                                                                      processed_prediction processed_references  f1_macro  f1_entailment score_name  score  accuracy  f1_micro  \
0                                                        **given premise:** the drain is clogged with hair         [entailment]         0              0   f1_micro      0         0         0
1                                                                                  according to the logic:     [not entailment]         0            NaN   f1_micro      0         0         0
2  the hypothesis "sally stopped and counted to ten" entails the premise "beth didn't get angry with sally         [entailment]         0              0   f1_micro      0         0         0
3                                                  **premise:** no one joins facebook to be sad and lonely     [not entailment]         0            NaN   f1_micro      0         0         0
4                            the entailment is "not": the son was so heavy is an entailment of the premise     [not entailment]         0            NaN   f1_micro      0         0         0

   f1_not entailment
0                NaN
1                  0
2                NaN
3                  0
4                  0

We should change it to be readable. E.g. show the source,references, prediction, processed_references,processed_predictions, score, score_name - maybe even in a line by line format because each of them is very long.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant