ferramenta
GuidePerformanceADRs
Skip to content

Stack Overflow in Debug Builds During Full Test Suite

Status

Accepted

Context

When running the full test suite (cargo test --test compat_utf8 -- --test-threads=1), a SIGSEGV occurs around test 1029. The affected test (named_group_underscore_backref) passes fine in isolation.

The root cause is a stack overflow: The recursive tree traversal functions tune_tree(), compile_tree(), tune_call(), tune_call2(), tune_called_state(), and is_exclusive() have — like in the C original — no depth limit. In Rust debug builds (without optimization), stack frames are significantly larger than in the C equivalent (~1-2 KB per recursion level). The default thread stack of 2 MB is insufficient after ~1000 sequential compilations.

Investigation

  • All unsafe blocks were audited: No use-after-free, no dangling pointers, no state leaks between tests.
  • No global mutable state between tests (each test creates fresh RegexType instances).
  • RUST_MIN_STACK=268435456 (256 MB) resolves the issue completely — all 1468 tests pass with single-threaded execution.
  • Using --test-threads=2 or higher with 64 MB stack also works reliably.
  • After adding further recursive passes (tune_call, tune_call2, tune_called_state, tune_next, is_exclusive etc.), 16 MB is no longer sufficient.
  • Release builds with optimization and inlining are not affected.
  • The C original also has no depth limit in tune_tree/compile_tree (only the parser has PARSE_DEPTH_LIMIT). The issue doesn't manifest in C because C stack frames are smaller.

Decision

The parser depth limit also bounds the linked List and Alt AST spines used for flat concatenations and alternations. This deliberately differs from C: upstream onig_node_free() iterates the cdr links, while Rust's derived Box destruction otherwise recurses through them. A sequence or alternation spine or nested body that exhausts the shared per-pattern AST-expression budget now fails with ONIGERR_PARSE_DEPTH_LIMIT_OVER; long literal strings remain unaffected because they occupy one AST node.

We do not introduce another artificial limit for nested ASTs because:

  1. The C original has none — feature parity takes precedence.
  2. Release builds are not affected.
  3. RUST_MIN_STACK provides a simple solution for debug testing.

Consequences

Required stack sizes

This table is the canonical reference for RUST_MIN_STACK in this repository. README, CONTRIBUTING, AGENTS, the CI workflow, and scripts/coverage.sh link to or reuse these values in their runnable guidance.

RunRUST_MIN_STACKNote
Debug, --test-threads=1268435456 (256 MiB)Minimum for the stack-heavy suites (compat_utf8, compat_back)
Debug, --test-threads=2 or more67108864 (64 MiB)The load is spread over several threads; works reliably but is not deterministic
Release builds with coverage instrumentation268435456 (256 MiB)Coverage bookkeeping adds frames; scripts/coverage.sh additionally skips the deepest recursive tests
Unaffectednot neededRelease builds without coverage instrumentation, and any suite other than compat_utf8 / compat_back

CI may run with more headroom than the minimum (the compat_utf8 lane uses 512 MiB) -- that is deliberate slack on the largest suite, not a second rule.

  • Tests should be run with the increased stack size: 
    RUST_MIN_STACK=268435456 cargo test --test compat_utf8 -- --test-threads=1
  • The parse-depth limit now protects both nesting and flat linked AST spines.