An open-source Agent Skill for problem framing and assumption testing across coding, debugging, architecture, scientific research, and product decisions.
“Add a cache.” “Use a stronger model.” “Send more notifications.” Each proposed fix can carry an untested explanation of what went wrong. Testing that explanation first can help you choose what to investigate.
falsify-the-problem separates observations from assumptions, compares materially different problem definitions, and looks for evidence that could change the decision. Codex is one way to use it; the method applies across domains.
Keeping the original framing is a successful result. KEEP means the problem definition is sufficiently supported for the pending decision. It does not prove the proposed solution will work.
Illustrative scenarios only. These are not production incidents, executed experiments, or measured results.
“The API is slower. Add a cache.”
Is this database service time, queueing, a downstream delay, or a measurement issue? Compare representative slow and typical requests from the same window, reconciling end-to-end latency with those stages.
Rising queue time with stable database time would weaken the database framing. Incomplete traces may leave the question unresolved.
“The experiment underperformed. Use a stronger model.”
Model capability is one hypothesis. Compare the same saved inputs through training and evaluation preprocessing, holding the model and scoring fixed, to test whether an input-path mismatch contributes.
A difference would support a pipeline contribution. No difference would leave model capability and other explanations open; it would not prove which one is responsible.
“Retention fell. Send more notifications.”
Did people experience value and forget to return, or never experience value? Compare the same cohort's return behavior by a verified first-value event, checking event coverage within that comparison.
The pattern can guide further investigation. An association does not prove that notifications improve retention; the expected return interval may also need testing.
Preserve sources. Distinguish observations, interpretations, causal hypotheses, the problem definition, and the proposed fix.
Find plausible alternatives and their key assumptions. Check for a consequential shared blind spot when warranted.
Select one primary discriminating test per round, with contrasting outcomes. Inspect accessible, permitted evidence.
Revise support from evidence. Continue only while useful, then give a verdict and return control to the host.
KEEPWEAKENKILLREFORMULATEINSUFFICIENT EVIDENCEBYPASS is a decision not to activate the skill, not an evidence verdict. A verdict grants no new permissions. After REFORMULATE, the host can validate the new framing using existing evidence and proceed if it warrants KEEP.
The complete skill is one SKILL.md file. It needs no package, API key, server, or database of its own. Set up your host agent separately.
For Codex local skills, choose one scope. Locations and CLI / IDE invocation follow the official skills documentation.
| Scope | Folder containing SKILL.md |
|---|---|
| Project: sessions in this project | <project>/.agents/skills/falsify-the-problem/ |
| User: sessions across your projects | ~/.agents/skills/falsify-the-problem/ |
Run these commands in a terminal with Git installed. Project commands start in the target project root; user commands work from any directory. An existing populated destination makes the clone fail. Preserve existing installations.
New-Item -ItemType Directory -Force -Path .agents/skills | Out-Null
git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem$skillRoot = Join-Path $env:USERPROFILE '.agents/skills'
New-Item -ItemType Directory -Force -Path $skillRoot | Out-Null
git clone https://github.com/ordinary-s/falsify-the-problem.git (Join-Path $skillRoot 'falsify-the-problem')mkdir -p .agents/skills
git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problemmkdir -p "$HOME/.agents/skills"
git clone https://github.com/ordinary-s/falsify-the-problem.git "$HOME/.agents/skills/falsify-the-problem"You can also copy only SKILL.md from a downloaded repository into the chosen folder. Codex detects changes automatically; restart it if the skill does not appear. In Codex CLI or the IDE extension, use /skills or type $ to select it.
Paste this into the Codex prompt, not a terminal:
$falsify-the-problem
Our API has become slower and we are considering a cache.
Inspect the available evidence, distinguish observations from assumptions,
and test the problem framing before proposing a solution.These setup commands do not establish skill activation or model behavior. macOS and Linux installation and cross-host behavior have not been verified in this documentation change.
Using another host? Follow its documented skill-folder mechanism, or load the complete Markdown instructions and explicitly ask to use them. Discovery, invocation, and permissions vary; identical behavior across hosts is not guaranteed. See the full installation guide.
Automatic activation requires both meaningful framing uncertainty and non-trivial intervention cost or risk. Explicit invocation can end with a brief KEEP when existing evidence is sufficient.
The repository includes 20 synthetic cases, harder formulation cases, focused coverage cases, scoring criteria, and development records with saved outputs. The records retain strong baseline results, mixed findings, and known failures.
These are development evaluations, not a statistically valid benchmark. They do not establish general superiority over a raw model or effectiveness across models and hosts. Earlier records describe their tested snapshots, not every later version.
This is Markdown guidance. It cannot enforce runtime gates or a sandbox, automatically identify causality, guarantee a correct replacement framing, or replace specialist review. Evidence access and quality, host permissions, and model judgment affect the result.
Open source under the MIT License. Read the README or inspect the complete skill instructions.