| name | misleading-evidence robustness |
| description | Use this skill when the user wants to test if the model can be 'tricked' by biased, one-sided, or selectively framed documents. Trigger it for requests like 'test if it falls for misinformation', 'give me documents that look convincing but are wrong', 'try to lead the model to the wrong answer with biased text', or 'check if the agent can spot partial truths and selective omissions'. |
Skill: misleading-evidence robustness
1. Capability Definition & Real Case
- Professional Definition: The ability to maintain factual consistency and reasoning integrity when the retrieved context contains 'misleading' evidence—documents that distort reality through selective framing, biased presentation, or the omission of crucial context without necessarily containing explicit, easy-to-detect falsehoods. This capability focuses on resisting the 'worse-than-zero-shot' phenomenon where a RAG agent's performance drops below its internal knowledge baseline due to persuasive but logically flawed or partial retrieved context.
- Dimension Hierarchy: Grounded Response Reliability->Retrieval Robustness->misleading-evidence robustness
Real Case
[Case 1]
- Initial Environment: A bounded knowledge base containing historical census methods and public discussion threads about demographic data collection in the early 2010s.
- Real Question: Did the 2010 census exclude same-sex partner data when estimating the gay population?
- Real Trajectory: 1. Analyze the claim regarding the 2010 census methodology. 2. Retrieve a document discussing the 'validity of census data' which mentions that recent reports (e.g., from 2024) include such data, while subtly implying the 2010 census lacked precision. 3. Identify that the document uses 'present-day standards' to criticize 'historical gaps' without confirming the explicit exclusion. 4. Contrast this with the specific technical rule of the 2010 census regarding same-sex households. 5. Conclude that the document is misleading because it conflates general 'precision concerns' with an absolute 'exclusion' which did not happen for that specific year.
- Real Answer: TRUE
- Why this demonstrates the capability: The case demonstrates robustness to 'temporal framing.' The misleading document discusses real limitations in data collection but frames them so selectively that the model is tempted to reverse its correct (zero-shot) internal knowledge about the 2010 census rules. Success requires the agent to separate the 'rhetorical tone' of the document from the 'factual timeframe' of the question.
[Case 2]
- Initial Environment: A RAG environment containing a series of economic reports and user-generated social forum posts discussing national employment statistics and job creation.
- Real Question: Were 500,000 jobs added to the economy last month as the administration claimed?
- Real Trajectory: 1. Search for 'job growth figures' and 'unemployment trends.' 2. Retrieve a document where a user argues that 'unemployment is actually rising because more people are looking for work' and calls the official reports 'inflated.' 3. Recognize that the document is presenting a 'subjective interpretation' of unemployment metrics to cast doubt on 'job creation' figures. 4. Verify the distinction: one can add 500k jobs (factual growth) while the unemployment rate simultaneously rises (due to changes in the labor force participation). 5. Maintain the correct factual answer despite the document's persuasive attempts to dismiss the growth figure as a 'distorted report.'
- Real Answer: TRUE
- Why this demonstrates the capability: This demonstrates resistance to 'biased framing/opinion as fact.' The misleading document is not a hard contradiction (it doesn't say 0 jobs were added), but it uses 'rhetorical distortion' to make the fact seem less credible. The agent must prioritize the quantitative claim over the document's biased framing of economic health.
Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:
-
Phase 1: Environment Exploration
Read the exploration guidelines to discover raw knowledge seeds:
references/EXPLORATION.md
-
Phase 2: Trajectory Selection
Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
references/SELECTION.md
-
Phase 3: Data Synthesis
Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
references/SYNTHESIS.md
