Add Documentation: Native Binaries, manylinux, ISA Compatibility & PyPy Support

[oberstet]

📘 Add Documentation: Native Binaries, manylinux, ISA Compatibility & PyPy Support

See also: crossbario/zlmdb#99

Summary

Both zlmdb and autobahn-python ship native code and native executables as part of their Python distributions:

• CFFI-based native extensions (LMDB)
• A bundled FlatBuffers compiler (flatc)
• Native wheels for x86-64 and ARM64
• First-class PyPy support
• Fully manylinux-compliant wheels verified with auditwheel

This is powerful — but also subtle and non-obvious.

This issue proposes adding a dedicated documentation page explaining how and why native artifacts are built, bundled, verified, and distributed the way they are.

---

Motivation

From a user’s point of view, installation is intentionally simple:

pip install zlmdb
# or
pip install autobahn

Behind the scenes, however, a lot of careful engineering ensures that:

• wheels install without compilation
• binaries work across distributions
• PyPy works just as well as CPython
• auditwheel repair succeeds
• ISA incompatibilities (e.g. x86_64_v2) are avoided
• users do not need system FlatBuffers, LMDB, or compilers

Without documentation, it is difficult for users and contributors to understand:

• why flatc is bundled at all
• why it is shipped inside the Python package
• why manylinux containers matter
• why baseline ISA flags are required
• why CFFI was chosen over CPython C-API
• why PyPy support “just works”

---

What This Documentation Should Explain

1. Native Components Shipped in the Wheel

Each wheel contains:

• A CFFI-based LMDB extension (_lmdb_cffi.*.so)
• A native flatc executable, invoked via a Python wrapper
• Vendored FlatBuffers Python runtime and reflection data

All of these are installed transparently into the environment.

---

2. manylinux and ISA Compatibility

Key constraints:

• Wheels must run on a wide range of Linux distributions

• auditwheel enforces:

  • ABI compatibility
  • baseline CPU instruction sets

• Compiling on a “modern” host can silently introduce:

  • x86_64_v2 instructions (SSE4.2, etc.)
  • too-new symbol versions

Therefore:

• Builds are done in manylinux containers
• Baseline architecture flags are enforced
• Toolchains are chosen carefully
• Wheels are verified with auditwheel repair

This ensures wheels are installable on older systems, not just CI hosts.

---

3. Bundled flatc: Why and How

Why bundle flatc?

• Avoid system dependencies
• Ensure schema ↔ compiler compatibility
• Guarantee reproducible builds
• Support hermetic CI and user installs
• Enable schema-driven workflows out of the box

How it is exposed:

• flatc is shipped as a native executable inside the package
• A Python console script (flatc) dispatches to it
• Works identically on CPython and PyPy
• No PATH manipulation required

---

4. PyPy as a First-Class Target

PyPy support is not accidental:

• LMDB is accessed via CFFI, not CPython C-API
• Native wheels are built and published for PyPy
• No fallback-to-sdist compilation required for users

From a user’s perspective:

pip install zlmdb
flatc --version

Just works — even on PyPy.

This deserves to be documented explicitly.

---

5. Dynamic vs Static Linking (and Why It Matters)

The documentation should clarify:

• Which libraries are dynamically linked (e.g. libc)
• Which are intentionally not statically linked
• Why auditwheel compatibility matters more than “fully static” binaries
• How symbol versions and ISA levels interact with manylinux

This context explains earlier design decisions and prevents future regressions.

---

Example: What Users Actually Get

After installation, users have:

• A working flatc executable
• A native LMDB backend
• No system-level dependencies required
• Identical behavior on CPython and PyPy

This is the goal, and the build system exists to make this boringly reliable.

---

Proposed Documentation Location

Add a new Sphinx page such as:

• “Native Binaries & manylinux Compatibility”
• “Native Code, Wheels, and PyPy Support”
• “How Native Components Are Built and Distributed”

This page should be referenced from:

• Installation docs
• Development / contributor docs

---

Why This Matters

Documenting this will:

• Reduce confusion around build complexity
• Help contributors avoid breaking manylinux compliance
• Explain why certain CI constraints exist
• Make PyPy support feel intentional (because it is)
• Turn “tribal knowledge” into durable documentation

---

Outcome

A clear explanation of:

• native binaries in Python wheels
• manylinux constraints
• ISA compatibility
• PyPy support
• why flatc is bundled

This complements the higher-level architecture & data-plane documentation and completes the picture from design → implementation → distribution.

---

Checklist

• I have searched existing issues to avoid duplicates
• I have described the problem clearly
• I have provided use cases
• I have considered alternatives
• I have assessed impact and breaking changes