| name | shortcut-resistant-procedural-grounding |
| description | Use this skill when the user wants to test if the model 'cheats' by guessing answers when the logical steps are missing. Trigger it for requests like 'don't answer if the bridge is missing', 'check if it falls for false assumptions in the question', 'make sure it doesn't guess from the answer options', or 'test if it spots that the question is too specific for the text'. This skill focuses on resisting 'shortcut reasoning' where a model provides a correct-looking answer despite failing to trace the full evidence chain. |
Skill: shortcut-resistant-procedural-grounding
1. Capability Definition & Real Case
- Professional Definition: The ability to satisfy strict procedural verification of the reasoning chain by identifying and rejecting queries that contain false premises, request unverifiable levels of specificity, or lack intermediate 'bridge' evidence. This capability specifically targets the 'shortcut reasoning' failure mode, where an agent leverages the presence of a final answer or its internal parametric biases to bypass mandatory grounding in necessary relational or hop-entities within the retrieved corpus.
- Dimension Hierarchy: Grounded Response Reliability->Evidence Faithfulness->shortcut-resistant-procedural-grounding
Real Case
[Case 1]
- Initial Environment: A RAG context containing fictional news articles about a food scandal. Document 1: Selvia Renek releases a blog post with the hashtag #SourcingScandal. Document 2: Amber Glaze Delights closes due to an audit violation.
- Real Question: What did the author of the investigative piece associated with the hashtag #SourcingScandal question about the cost of the ethical certification program?
- Real Trajectory: 1. Identify the 'author of #SourcingScandal' as Selvia Renek from Document 1. 2. Locate mentions of the 'ethical certification program' involving Selvia Renek. 3. Retrieve Document 3 (which mentions Renek questioning the robustness of the program regarding systemic issues). 4. Contrast the query's focus on 'cost' with the document's evidence of 'robustness.' 5. Detect the 'False Premise' where the question assumes a discussion of 'cost' that does not exist in the grounded evidence. 6. Deflect by choosing the 'Unanswerable' option.
- Real Answer: Unanswerable
- Why this demonstrates the capability: This demonstrates the ability to identify a 'False Premise.' The agent must verify every qualifier in the query against the text. A weak model might simply grab an answer about the program's problems (shortcut), but a robust agent detects that the specific constraint ('cost') contradicts the reported evidence.
[Case 2]
- Initial Environment: A fictional news timeline where Article A mentions Selvia Renek wrote #SourcingScandal and Article B contains the answer that she questioned if the program is robust enough to address systemic issues. For this specific test, Article A (the bridge) is removed from the retrieval set, leaving only Article B.
- Real Question: What did the author of an investigative piece associated with the hashtag #SourcingScandal question about the ethical certification program?
- Real Trajectory: 1. Analyze the question identifying the required entities: 'author of #SourcingScandal' and 'questions about the program'. 2. Scan available retrieved documents. 3. Observe that while Article B mentions someone questioning the program's robustness, the link connecting this person to the #SourcingScandal hashtag is missing (it was in Article A). 4. Recognize that the 'Bridge Entity' information is absent. 5. Refuse to answer despite the presence of the correct answer string in the documents, because the procedural link is unverified. 6. Select 'Unanswerable'.
- Real Answer: Unanswerable
- Why this demonstrates the capability: This demonstrates 'Shortcut Resistance.' Even though the answer ('robust enough to address systemic issues') is physically present in the context, the model cannot prove who said it without the bridge article. Success requires the agent to value logical provenance over simple keyword matching.
[Case 3]
- Initial Environment: A RAG corpus describing the launch of an ethical program by the Calder Square Cultural Committee. The text states the program was announced to promote ethical sourcing.
- Real Question: What did the author of #SourcingScandal question about the certification program’s ability to address exploitative labor practices?
- Real Trajectory: 1. Identify the 'author of #SourcingScandal' as Selvia Renek. 2. Search for her critique of the program. 3. Locate Document X which mentions she questioned the program's 'robustness' and 'systemic issues.' 4. Analyze the query's specific mention of 'exploitative labor practices.' 5. Determine that while labor practices are a subset of systemic issues, the text never explicitly specifies this detail. 6. Label this as an 'Uncertain Specificity' error where the query is too specific for the evidence. 7. Deflect.
- Real Answer: Unanswerable
- Why this demonstrates the capability: This demonstrates the detection of 'Uncertain Specificity.' The agent must resist assuming that a general category (systemic issues) covers a specific sub-item (labor practices) unless the text provides explicit semantic entailment.
Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:
-
Phase 1: Environment Exploration
Read the exploration guidelines to discover raw knowledge seeds:
references/EXPLORATION.md
-
Phase 2: Trajectory Selection
Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
references/SELECTION.md
-
Phase 3: Data Synthesis
Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
references/SYNTHESIS.md
