feat(issue summary): Score possible cause confidence and novelty

[kddubey]

Adds "scores" to the issue summary response.

Costs 0.3 sec in latency

How has this been tested?

1. Changed unit test to use VCR

2. Local run which generated a summarization (trace ID: 1ebe8c50-e9ea-4b26-85e5-9fa41a06f8eb) w/ output:

output

[
    {
        "trace": "",
        "scores": {
            "possible_cause_novelty": 0.2809091323364492,
            "possible_cause_confidence": 0.487353380050403
        },
        "group_id": 16,
        "headline": "Validation Error in GroupingRequest due to Empty Stacktrace",
        "whats_wrong": "**ValidationError** raised: stacktrace must be provided and cannot be empty. Data input failed validation.",
        "possible_cause": "**Empty stacktrace** field in the input data likely caused the validation failure."
    },
    {
        "title": "Validation Error in GroupingRequest due to Empty Stacktrace",
        "scores": {
            "possible_cause_novelty": 0.2809091323364492,
            "possible_cause_confidence": 0.487353380050403
        },
        "whats_wrong": "**ValidationError** raised: stacktrace must be provided and cannot be empty. Data input failed validation.",
        "possible_cause": "**Empty stacktrace** field in the input data likely caused the validation failure.",
        "session_related_issues": ""
    }
]

[jennmueng]

Looks like in the failing test test_summarize_issue_event_details, the llm call isn't being mocked and it's trying to make an actual openai call

[kddubey]

I forgot to VCR the embeddings call in test_summarize_issue_event_details. Did that and CI passes now

[jennmueng]

q on latency, otherwise lgtm

[jennmueng]

you may have mentioned this already and I missed it, but remind me how much latency does this add?

[roaga]

Code looks fine to me, just confused on the confidence score calculation. Also idk how we feel about the latency

[roaga]

what's this TODO?

[kddubey]

in the local test I get exact equality. in CI (specifically, this failed run), we get inequality at the 16th decimal place:

>       assert raw_result == expected_raw_result
E       AssertionError: assert IssueSummaryW...319887938298)) == IssueSummaryW...319887938298))
E         Full diff:
E         - IssueSummaryWithScores(title='Critical Issue: red-timothy-sandwich Failure', whats_wrong='**red-timothy-sandwich** encountered **exceptions** during execution, impacting functionality. Check **event logs** for details.', session_related_issues='Related issues: **cyan-vincent-banana** and **green-fred-tennis** may indicate broader session instability.', possible_cause='Potential **resource contention** or **dependency failure** affecting multiple components.', scores=SummarizeIssueScores(possible_cause_confidence=0.4749528387220516, possible_cause_novelty=0.6286319887938298))
E         ?                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        ^
E         + IssueSummaryWithScores(title='Critical Issue: red-timothy-sandwich Failure', whats_wrong='**red-timothy-sandwich** encountered **exceptions** during execution, impacting functionality. Check **event logs** for details.', session_related_issues='Related issues: **cyan-vincent-banana** and **green-fred-tennis** may indicate broader session instability.', possible_cause='Potential **resource contention** or **dependency failure** affecting multiple components.', scores=SummarizeIssueScores(possible_cause_confidence=0.47495283872205174, possible_cause_novelty=0.6286319887938298))
E         ?   

if the VCR weren't used / we'd actually hit the OpenAI API in the test, we'd see inequality at around the 4th decimal place (they add noise of around 5e-4 so embeddings are not deterministic). so this is probably a weird decompression or decryption error. left it as a TODO since it's not too important

[roaga]

lgtm, idk if we want to discuss latency before merging, but it's probably fine

[kddubey]

The last commit gets rid of a mostly useless embedding. Added latency is now 0.3 sec p50, 0.43 sec p90 (see "Measure additional latency" in the notebook).