Recursion check in 3.13.8 in CALL_BOUND_METHOD_GENERAL leaves stack in inconsistent state

[marc-hb]

Bug report

Bug description:

I had the same experience with this bug as with a compiler bug: I first suspected and debugged everything else than cpython. But:

• I could really not find anything wrong anywhere else

• The cpython bisect unambiguously and deterministically points to this 3.13.8 backport
  ebccd1de88d. All 3.13 git commits before that backport pass, all 3.13 versions after this backport fail.

• cpython 3.14 with the original gh-132744: Check recursion limit in _PY_FRAME_GENERAL #132746 works fine. Maybe the 3.13.8 backport is missing some subtle dependency found only in 3.14? EDIT: confirmed below.

• The behavior is the same across Windows, Linux and macOS:
  test_manifest.py: parametrize loop_detection test for more coverage zephyrproject-rtos/west#919 (comment)

• Reverting the 3.13.8 backport makes all failing test configurations pass again:

  cd cpython
  git checkout v3.13.12
  git revert ebccd1de88d4 # surprisingly zero conflict
  

• Explain this?

   sys.setrecursionlimit(1000) # PASS
   sys.setrecursionlimit(1001) # FAIL
   sys.setrecursionlimit(1002) # PASS
   sys.setrecursionlimit(1003) # FAIL
   sys.setrecursionlimit(1004) # PASS
   sys.setrecursionlimit(1005) # FAIL

• While it's not clear to me why some test configurations fails and others do not, the
  corruption that seems to be happening is 100% deterministic. When a test configuration fails,
  then it fails 100% of the time. Test configurations that pass, pass 100% of the time.

So, I really think this is a cpython issue now.

---

Simple and short reproduction steps simplified from all the trial and error in zephyrproject-rtos/west#908

git clone https://github.com/zephyrproject-rtos/west
cd west
git checkout 2d883de32f4f # today's latest version

# You MUST turn off the @pytest.mark.skipif((3, 13, 8) <= sys.version_info) I had to add in test_manifest.py not to break CI.
# See where in https://github.com/zephyrproject-rtos/west/pull/919#issuecomment-3900130946

uv python install 3.12   3.13.7 3.13.8 3.13.12   3.14
for i in          3.12   3.13.7 3.13.8 3.13.12   3.14; do
  uv run --python "$i"  pytest -k loop_detect
done

Versions from 3.13.8 to 3.13.12 FAIL odd recursion limits. All other Python versions pass.

The test fails with a very long (by design) stack trace seen here:
zephyrproject-rtos/west#908 (comment)

Then, the last frames of the stack observed with pytest --pdb makes no sense:

1. The srp() method (Stream Reader.peek()) is called without any parameter
2. Its default index argument is index : int = 0
3. Inside the peek() method, the index: int parameter becomes equal to... self!!

        def peek(self, index: int = 0) -> Text:

(Pdb) p self
<ruamel.yaml.reader.Reader object at 0x7ff5ed1b9010>
(Pdb) p index
<ruamel.yaml.reader.Reader object at 0x7ff5ed1b9010>  # !!!!!

CPython versions tested on:

3.13

Operating systems tested on:

Linux, Windows, macOS

Linked PRs

• [3.13] gh-145008: Maintain stack consistency in CALL_BOUND_METHOD_EXACT_ARGS #145015

[Fidget-Spinner]

cpython 3.14 with the original works fine

Doesn't that suggest that this isn't an issue with the recursion checker, but possibly something else? 3.14 had better stack protection mechanisms introduced, so that likely means one of two things:

1. There is a bug in cpython 3.13's stack detection mechanism.
2. There is a bug in the extension code. Just checking, does it do any sort of clever stuff at the C level?

Again, the recursion checker is just a symptom, not a cause. It's highly unlikely for the recursion check to cause stack corruption. The _PY_FRAME_GENERAL is used in Python's specializing interpreter. This means the default behavior must follow un-specialized calls in Python, which check for the recursion limit.

Unless you provide a better description or issue title, I'm planning to close the issue, as pointing it to _PY_FRAME_GENERAL is misleading.

[marc-hb]

A big red herring can be found in zephyrproject-rtos/west#908: xdist.

This issue was initially found while trying to run the whole test suite faster. It took me a very long time to realize xdist was a red herring and to reproduce without it. I'm guessing xdist is a messenger because it offsets the stack by an odd number?

Remember:

   sys.setrecursionlimit(1000) # PASS
   sys.setrecursionlimit(1001) # FAIL
   sys.setrecursionlimit(1002) # PASS
   sys.setrecursionlimit(1003) # FAIL
   sys.setrecursionlimit(1004) # PASS
   sys.setrecursionlimit(1005) # FAIL

xdist and debugging don't like each other, which made the whole experience especially painful:

• support pdb.set_trace() with pytest-xdist pytest-dev/pytest#390

Off-topic now, sorry.

[marc-hb]

There is a bug in the extension code. Just checking, does it do any sort of clever stuff at the C level?

Absolutely not, which is why I filed this here.

Unless you provide a better description or issue title, I'm planning to close the issue, as pointing it to _PY_FRAME_GENERAL is misleading.

Please edit the title in any better way you see fit. I don't see why that would require closing the issue.

On the other hand, I spent a lot of time carefully crafting the description and keeping it as factual as possible and I don't see what is misleading in it?

[Fidget-Spinner]

1. There is a bug in cpython 3.13's stack detection mechanism.

I think this is more likely now, as I went to look at your issue and saw that you are using uv's Python. Just so you're aware:

1. 3.13 uv Python builds with default Clang and computed gotos for Linux, which is notoriously known for consuming a huge amount of C stack
2. 3.14 uv Python builds with default Clang and tail calling interpreter for Linux, which significantly reduces the C stack space required.

So this is very much pointing in the direction of the C stack problems causing corruption.

On the other hand, I spent a lot of time carefully crafting the description and keeping it as factual as possible and I don't see what is misleading in it?

Thanks for taking the time to craft it, but the title is

Stack corruption and 3.13.8 regression caused _PY_FRAME_GENERAL

Which states as a matter of fact that X is caused by Y, however the evidence doesn't seem to necessarily point to that, so I think it's misleading. I will change the issue title.

[marc-hb]

I think this is more likely now, as I went to look at your issue and saw that you are using uv's Python.

I mentioned uv for reproduction convenience but none of my bisects and debug used uv. I invoked pytest directly the vast majority of the time and uv was not involved. I did not spot any difference caused by uv yet.

[marc-hb]

3.13 uv Python builds with default Clang and computed gotos for Linux, which is notoriously known for consuming a huge amount of C stack

Test duration aside, the size of the stack never made any functional difference. Try it; it's a one-line change in the test!

   sys.setrecursionlimit(200) # PASS
   sys.setrecursionlimit(201) # FAIL
   sys.setrecursionlimit(202) # PASS
   sys.setrecursionlimit(203) # FAIL
   sys.setrecursionlimit(204) # PASS
   sys.setrecursionlimit(205) # FAIL

   sys.setrecursionlimit(1000) # PASS
   sys.setrecursionlimit(1001) # FAIL
   sys.setrecursionlimit(1002) # PASS
   sys.setrecursionlimit(1003) # FAIL
   sys.setrecursionlimit(1004) # PASS
   sys.setrecursionlimit(1005) # FAIL

   sys.setrecursionlimit(10000) # PASS
   sys.setrecursionlimit(10001) # FAIL
   sys.setrecursionlimit(10002) # PASS
   sys.setrecursionlimit(10003) # FAIL
   sys.setrecursionlimit(10004) # PASS
   sys.setrecursionlimit(10005) # FAIL

[Fidget-Spinner]

I already cloned it and tried it locally, with a fix.

Ok thanks, that gives me enough information. I narrowed it down to a discrepancy between how we deal with the stack in Python 3.13 and 3.14/15:

• On Python 3.13, we do not necessarily leave the stack in a consistent state between uops
• On Python 3.14/15, we guarantee to leave the stack in a consistent state between uops.

I have a fix up soon, however I don't know how to write a test for this. The fix is not so nice in my opinion, but as a bandaid for 3.13 it should work fine.

[marc-hb]

Stack corruption and 3.13.8 regression caused _PY_FRAME_GENERAL [truncated]

Which states as a matter of fact that X is caused by Y,

I meant the backport was the cause because that was the result of my bisect. I honestly don't understand the change itself so I wasn't trying to blame anything specific in the code. Thanks for editing the title.