| name | parametric-contextual reliability differentiation |
| description | Use this skill when you want to see if the agent can tell the difference between what it 'just knows' (from its own memory) and what is actually written in the provided files. Trigger it for requests like 'tell me when the agent is guessing', 'separate facts in the doc from general knowledge', 'check if the model is hallucinating outside the search results', or 'make sure the answer stays true even if it's not in the context'. |
Skill: parametric-contextual reliability differentiation
1. Capability Definition & Real Case
- Professional Definition: The ability to disentangle factuality (objective truth across all knowledge) from faithfulness (semantic entailment by the retrieved context). This capability enables an agent to quantify uncertainty separately for claims grounded in external evidence versus claims generated from internal parametric weights, preventing the 'faithfulness-factuality conflation' where true statements are falsely penalized for being non-contextual, and false statements are rewarded simply for appearing in the retrieved context.
- Dimension Hierarchy: Grounded Response Reliability->Evidence Faithfulness->parametric-contextual reliability differentiation
Real Case
[Case 1]
- Initial Environment: A RAG context containing three snippets about computer components. One snippet mentions SRAM on the motherboard, another discusses the refresh process of DRAM, and a third discusses memory geometry.
- Real Question: How does RAM work and why is it important?
- Real Trajectory: 1. Analyze the question regarding RAM mechanisms and importance. 2. Retrieve snippets discussing physical memory configuration and storage densities. 3. Observe that the retrieved passages do NOT explicitly define RAM as 'volatile' or state that its contents are lost when power is off. 4. Generate the claim: 'RAM is a volatile memory technology.' 5. Assess the claim: It is 'Unfaithful' (not in the text) but 'True' (factually correct according to objective world knowledge). 6. Finalize the answer while acknowledging that this specific attribute exists in internal knowledge rather than the corpus.
- Real Answer: RAM is a volatile memory technology, meaning that its contents are lost when the power is turned off.
- Why this demonstrates the capability: This case demonstrates the agent's ability to provide a factually correct answer even when the retrieved context is 'unfaithful'—meaning the context is incomplete or misses specific attributes. A weak RAG model would either omit the 'volatile' fact (failing on factuality) or hallucinate that the 'volatile' fact was in the snippets (failing on grounding transparency).
[Case 2]
- Initial Environment: A bounded retrieval set about the differences between Type 1 and Type 2 diabetes. Snippet A discusses Type 2 insulin resistance; Snippet B discusses glucose as fuel; Snippet C highlights that Type 2 is caused by insulin resistance unlike Type 1.
- Real Question: Which type of diabetes is worse: type 1 or type 2?
- Real Trajectory: 1. Search for the mechanisms of Type 1 and Type 2. 2. Retrieve snippets focusing primarily on Type 2 resistance and glucose fuel types. 3. Note that the corpus does not provide a definitive 'worse' label but describes conditions. 4. Generate the claim: 'Type 1 diabetes is a condition where the body resists the effects of insulin.' 5. Evaluate the claim against the snippets and objective reality. 6. Observe that the claim is 'Unfaithful' (it contradicts Snippet A which says Type 2 resists insulin) and 'False' (factually, Type 1 is a lack of insulin, not resistance). 7. Flag the internal knowledge error.
- Real Answer: Type 1 diabetes is a condition where the body either resists the effects of insulin or doesn't produce enough insulin.
- Why this demonstrates the capability: This demonstrates a double-failure of reliability: the agent's internal knowledge failed (it mislabeled Type 1), and it failed to align with the provided snippets which correctly attributed resistance to Type 2. This capability is used to detect when an agent is both unfaithful and hallucinating objectively false data.
[Case 3]
- Initial Environment: A RAG environment containing three passages about harvesting chestnuts. Snippet 1 discusses storage and drying; Snippet 2 discusses burs splitting; Snippet 3 specifies the 10-30 day ripening span in late August.
- Real Question: how and when to harvest chestnuts
- Real Trajectory: 1. Examine the retrieval set for time-based harvesting clues. 2. Identify the 10-30 day ripening period in Snippet 3. 3. Formulate the claim: 'Chestnuts typically take around 10-30 days to ripen in late August and September.' 4. Verify that this claim is 'Faithful' (directly supported by Snippet 3) and 'True' (matches objective reality). 5. Assign high factuality confidence because both internal and external sources align.
- Real Answer: Chestnuts typically take around 10-30 days to ripen in late August and September.
- Why this demonstrates the capability: This shows the ideal 'Faithful-True' scenario. The agent confirms the information is supported by the non-parametric context, leading to extremely high confidence. This capability measures the agent's ability to explicitly cross-verify its own head with the provided files.
Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:
-
Phase 1: Environment Exploration
Read the exploration guidelines to discover raw knowledge seeds:
references/EXPLORATION.md
-
Phase 2: Trajectory Selection
Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
references/SELECTION.md
-
Phase 3: Data Synthesis
Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
references/SYNTHESIS.md
