Issue Summary
I’ve noticed that both GPT-4o and the o1 model incorrectly report DMARC results when analyzing email headers. Specifically, they state that “DMARC passed” even when no DMARC policy is enforced (i.e., p=none).
Technical Details
When analyzing an email header with:
SPF: Pass (Sender IP is authorized)
DKIM: Pass (Signature is valid)
DMARC: None (No enforcement policy)
The models incorrectly state that “DMARC passed.”
However, DMARC does not “pass” simply because SPF and DKIM pass. DMARC only passes when:
A DMARC policy (p=reject, p=quarantine, or p=none) exists.
The email aligns with SPF or DKIM based on the DMARC policy.
Since p=none means there’s no enforcement, the correct response should be “DMARC None” rather than “DMARC Pass.”
Expected Behavior
The model should correctly recognize:
If p=none is set, DMARC should be reported as “None” (not “Pass”).
Only if p=quarantine or p=reject is in place and the email aligns should it state “DMARC Pass.”
Impact
This misinterpretation can lead to confusion in email security analysis, potentially causing incorrect assumptions about email authenticity and compliance.
Request
Can this be addressed in a future update to improve accuracy in email security analysis?
You most likely have tried to mitigate this issue with prompting techniques. If you haven’t made the progress you need, you can share your prompt and maybe a few examples for other community members to help improve the results right now.
I have created a prompt on my end.
It should probably work correctly.
Context:
This model must accurately interpret and report DMARC status when analyzing email headers.
Rules:
Criteria for DMARC interpretation:
Report DMARC as “Pass” only if:
A DMARC policy (p=reject or p=quarantine) is explicitly defined.
The email aligns with SPF or DKIM according to the defined DMARC policy.
If p=none is set, report DMARC as “None”.
If no p policy is explicitly defined, also report DMARC as “None”.
Do not report “DMARC Pass” simply because SPF and DKIM have passed, when p=none.
Email Header Analysis Output Format:
SPF: Pass / Fail
DKIM: Pass / Fail
DMARC: Pass / Fail / None (if p=none or no policy is defined)
Example:
Input:
SPF: Pass (Sender IP is authorized)
DKIM: Pass (Signature is valid)
DMARC: None (No enforcement policy)
Correct Output:
SPF: Pass
DKIM: Pass
DMARC: None (No enforcement policy)
(Do not incorrectly report this as “DMARC Pass”.)
Follow these rules strictly when performing email header analysis.
(Sender IP is authorized)
(No enforcement policy)