Is lazy type realization intended to be thread-safe on GIL builds (3.11)? #271

Context: we saw repeated SIGSEGVs in a threaded ASGI service on 3.11 that correlated with a transitive bump to cffi 2.0.0, and they stopped when we pinned back to 1.17.1. I'd like to know whether GIL builds are supposed to be safe here, so I know whether to keep looking elsewhere for our root cause.

Disclaimer: The below is written by AI. If that frustrates you, feel free to close. If not, thank you for reading on 🙏

Since 2.0.0 the type-realization paths carry explicit thread-safety machinery, but as far as I can tell all of it is compiled only for free-threaded builds. On a GIL build (I'm on CPython 3.11):

• CFFI_LOCK() / CFFI_UNLOCK() expand to Py_BEGIN/END_CRITICAL_SECTION, and misc_thread_common.h redefines those to a bare { / } for PY_VERSION_HEX <= 0x030d00b3. So on 3.11 the lock is an empty scope.
• The publish in _realize_c_struct_or_union is cffi_atomic_store(&builder->ctx.types[s->type_index], x) under Py_GIL_DISABLED, and a plain store otherwise.
• The primitive pre-initialization loop in init_global_types_dict — the one commented "Ensure that all primitive types are initialised to avoid race conditions on the first access" — is itself inside #ifdef Py_GIL_DISABLED.

My question is simply whether that asymmetry is deliberate: is the GIL considered sufficient to serialize first-time type realization, so that no additional locking is needed on non-free-threaded builds?

The reason I'm unsure is that realization doesn't look like a single uninterruptible C step. do_realize_lazy_struct / complete_struct_or_union allocate, and in the s->size == (size_t)-2 branch the "unrealized" flag is cleared and the type published before do_realize_lazy_struct(ct) runs:

if (ct != NULL) {
    cffi_set_flag(ct->ct_unrealized_struct_or_union, 0);
}
...
builder->ctx.types[s->type_index] = x;
if (ct != NULL && s->size == (size_t)-2) {
    if (do_realize_lazy_struct(ct) < 0) { ... }
}

If a thread switch can land in that window, it looks like another thread could observe a type marked realized but not yet completed. I may well be misreading the invariant — if allocation there can't drop the GIL in a way that matters, that would explain it.