RFC-021: Library-Driven FFI Extension and Cross-Language Call Support

⚠️ Deprecated: This document has been deprecated; its content has been merged into RFC-026: FFI Core Mechanism.

References:

• RFC-001: Spawn Model and Error Handling System
• RFC-008: Runtime Concurrency Model and Scheduler Decoupling Design
• FFI Implementation Plan

Summary

This document proposes a library-driven FFI (Foreign Function Interface) extension scheme. The sole entry point for FFI is the native("symbol") declaration combined with the FfiRegistry runtime registry, with no second mechanism introduced in the core. On top of this, the standard library provides capabilities such as dynamic library loading and cross-language call bindings. Bindings for specific languages (such as C, Python, JavaScript) are either auto-generated by the official toolchain or written on demand by individual projects.

Motivation

Limitations of the Current Implementation

The current FFI implementation already has the following capabilities:

• native("symbol") syntax to declare external functions
• FfiRegistry function registry

However, its functionality is relatively limited:

• Lacks dynamic library loading support
• No infrastructure for cross-language calls
• Lacks automated binding generation tools

Design Philosophy

YaoXiang follows the principle of "a simple core, with complexity pushed down to libraries":

Good Taste: The language's job is to provide atomic capabilities, not an all-encompassing feature set. Complexity should be resolved through libraries, not piled up in the compiler.

Therefore, this proposal:

• ✅ Zero syntax changes — Fully backward compatible; the only FFI entry point is native("symbol")
• ✅ Library is the language — Features are extended through the standard library
• ✅ Automated toolchain — Bindings are auto-generated by yx-bindgen, not hand-maintained
• ✅ Progressive enhancement — Developers pull in features on demand

Proposal

1. Core FFI Library Enhancement

Extend the std.ffi module. Note: all calls to external functions still go through native("symbol") declarations; std.ffi only provides auxiliary capabilities.

1.1 Dynamic Library Loading

import ffi

# Load a dynamic library (.so/.dll/.dylib)
lib = ffi.load_library("./libmyext.so")

# Fetch function symbols from the library, returning names usable by native
ffi.register_library_symbols(lib, [
    "my_function",
    "another_func",
])

load_library returns a DynamicLibrary handle; register_library_symbols registers the symbol names into FfiRegistry's known table. The user then still uses them through native declarations:

my_func: (a: Int, b: Int) -> Int = native("my_function")

No second calling syntax, no try_call wrapper.

1.2 Library Management

# List loaded libraries
loaded = ffi.loaded_libraries()

# Unload a library
ffi.unload_library(lib)

# Library version check
ffi.check_version(lib, "1.0.0")

1.3 Symbol Resolution

# Look up a symbol by name (returns a Symbol structure)
sin_sym = ffi.dlsym("libm.so", "sin")

Cross-language calling conventions and type conversions are not handled at runtime through a generic wrapper, but are generated at compile time by yx-bindgen.

2. Dynamic Library Loading Implementation

2.1 Core Data Structures

pub struct DynamicLibrary {
    handle: *mut std::ffi::c_void,
    path: String,
}

impl DynamicLibrary {
    pub fn load(path: &str) -> Result<Self, FfiError>;
    pub fn get_symbol(&self, name: &str) -> Result<*mut std::ffi::c_void, FfiError>;
    pub fn unload(self) -> Result<(), FfiError>;
}

2.2 Error Types

pub enum FfiError {
    LibraryNotFound { name: String, os_error: Option<OsError> },
    SymbolNotFound { name: String, os_error: Option<OsError> },
    CallFailed { message: String, os_error: Option<OsError> },
    Timeout,
}

pub struct OsError {
    pub code: i32,
    pub message: String,
}

OsError carries the platform-native error code (Linux's dlerror(), Windows's GetLastError()), ensuring debuggability.

3. Multi-Language Bindings: Toolchain Approach

Abandon the fantasy of "binding libraries maintained by community maintainers of each language." Instead, bindings are auto-generated by the official toolchain.

3.1 Architectural Design

┌───────────────────────────────────────────────┐
│  YaoXiang code                                │
│                                               │
│  // User only writes native declarations       │
│  my_func: (a: Int) -> Int = native("my_func") │
└───────────────────────────────────────────────┘
         ↑                          ↑
         |  Compile-time            | Runtime
┌──────────────────┐   ┌────────────────────────┐
│  yx-bindgen       │   │  std.ffi + FfiRegistry  │
│  (C header → .yx) │   │  - dlopen/dlsym         │
│                   │   │  - LoadLibrary/GetProc  │
└──────────────────┘   └────────────────────────┘

3.2 Binding Generator (yx-bindgen)

yx-bindgen is a standalone CLI tool that generates YaoXiang FFI binding code from C header files:

yx-bindgen --header /usr/include/sqlite3.h --output sqlite3.yx

Example of generated output:

# Auto-generated, do not edit manually
# Source: /usr/include/sqlite3.h

sqlite3_open: (filename: *const u8, ppDb: *mut *mut opaque) -> Int
    = native("sqlite3_open")

sqlite3_close: (db: *mut opaque) -> Int
    = native("sqlite3_close")

sqlite3_exec: (
    db: *mut opaque,
    sql: *const u8,
    callback: *mut opaque,
    arg: *mut opaque,
    errmsg: *mut *mut u8,
) -> Int
    = native("sqlite3_exec")

yx-bindgen is officially maintained and guarantees:

• Complete type mapping (int → Int, char* → *const u8, void* → *mut opaque)
• Aligned struct layout (automatic #[repr(C)] equivalent)
• Callback signature conversion

3.3 Officially Maintained Binding Packages

The YaoXiang core team does not commit to maintaining general binding libraries for all languages, but provides an official libc binding package (a subset of POSIX + Windows API) as a best-practice example and foundational capability for FFI.

Bindings for other languages and libraries:

• Generate them yourself using yx-bindgen
• Can be published as YaoXiang packages (e.g., libsqlite3, libcurl, libsdl2)
• Not maintained by the core team, but the package publication and version management mechanism is provided

4. Type Conversion Layer

4.1 Compile-Time Type Mapping

Type conversion does not use runtime wrappers, but is statically determined during yx-bindgen generation:

C type	YaoXiang type	Conversion method
int	Int	Pass by value
char*	*const u8	Pointer pass
void*	*mut opaque	Opaque pointer
struct T	extern struct T	Memory layout match
int*	*mut Int	Pointer pass (mutable)
const int*	*const Int	Pointer pass (read-only)

4.2 Manual Conversion (Standard Library Helpers)

# Explicit conversion
raw_ptr = ffi.to_pointer(my_bytes)
c_string = ffi.to_c_string(my_string)

5. Memory Ownership Model

5.1 Basic Principles

Every memory allocation crossing the FFI boundary must clearly answer two questions:

1. Who allocates? (C-side malloc / YaoXiang-side runtime)
2. Who frees? (C-side free / YaoXiang-side runtime)

When yx-bindgen generates code, it adds annotations for common patterns:

# C-allocated, caller frees
sqlite3_exec: (...) -> Int
    = native("sqlite3_exec")
    # memory: C-allocated, caller must free errmsg via sqlite3_free

# Caller-allocated pointer
read: (fd: Int, buf: *mut u8, count: Int) -> Int
    = native("read")
    # memory: caller-allocated buf

The runtime does not perform automatic memory management on pointer references crossing the FFI boundary — ownership clearly rests with the caller.

5.2 String Handling

A char* returned by a C function is immediately copied when converted to a YaoXiang String. Ownership of the original pointer is determined by the C function (declared via annotation) and is not freed automatically.

6. Safety Considerations

6.1 Concurrency Safety

FFI function calls do not participate in DAG scheduling by default and are treated as blocking operations. C functions confirmed to be reentrant can be marked @concurrent:

# Pure function, no global state, safe to run concurrently
sin_safe: (x: Float) -> Float = native("sin")
    # reentrant: true

# Has global state, cannot run concurrently
strtok: (s: *const u8, delim: *const u8) -> *const u8 = native("strtok")
    # reentrant: false

yx-bindgen annotates reentrancy information for standard C library functions whenever possible (such as the _r variants of strtok).

Requirement for async callers: Before calling an FFI function, the caller must ensure the target function is reentrant. The runtime does not perform automatic detection — this is a problem that cannot be solved statically.

6.2 Error Isolation

• FFI call errors propagate through the Result type (if the function declares a Result return type)
• A timeout mechanism prevents external functions from deadlocking

# Call with timeout (implemented at the FfiRegistry layer)
result = ffi.call_with_timeout("blocking_func", 5000)  # 5-second timeout

6.3 Pointer Safety

• Pointer parameters require an unsafe marker on the YaoXiang side
• The lifetime of pointers crossing the FFI boundary is guaranteed by the caller

7. Compiler Changes

Zero syntax changes — only the native("symbol") declaration is needed, already implemented in the current compiler.

Added in the interpreter/runtime:

• Dynamic library loading instructions (FFI bindings for DynamicLibrary)
• Timeout mechanism

8. Rejected Features

The following features were reviewed and explicitly excluded, and will no longer be incorporated into the RFC:

• ffi.try_call: Redundant; native + Result return type already exists
• ffi.verify_signature: Runtime doing the compiler's job is the wrong abstraction layer
• ffi.async_call: To be considered after the reentrancy contract model is clarified
• Community-maintained binding tables: Not executable; replaced by the yx-bindgen toolchain approach

Trade-offs

Advantages

• ✅ Zero syntax changes — The only FFI entry point is native("symbol"), fully backward compatible
• ✅ Library is the language — Features are progressively introduced through the standard library
• ✅ Toolchain-driven — yx-bindgen automatically handles binding generation
• ✅ Memory safety — Ownership model is explicit; no use-after-free from automatic reclamation
• ✅ Debuggable — Errors carry OS-native error codes

Disadvantages

• ⚠️ Type safety is limited by the expressiveness of C headers (void* cannot be statically distinguished)
• ⚠️ yx-bindgen requires ongoing maintenance to keep up with C standard evolution
• ⚠️ Bindings for non-C languages (Python/JS/Java) need to be handled by each project; no unified solution

Implementation Strategy

Phase 1: Core Library (v0.7)

• [ ] Extend the std.ffi module
• [ ] Implement the DynamicLibrary structure
• [ ] Support Linux/macOS (dlopen/dlsym)
• [ ] Support Windows (LoadLibrary/GetProcAddress)
• [ ] Add a timeout mechanism in the runtime
• [ ] Unit tests

Phase 2: yx-bindgen (v0.8)

• [ ] Implement a C header parser (based on existing Clang bindings or a hand-written parser)
• [ ] Type mapping system
• [ ] Generate native("symbol") declarations
• [ ] Generate struct layouts
• [ ] Integration tests: generate bindings for real C libraries such as SQLite3, libcurl

Phase 3: Ecosystem Foundations (v0.9)

• [ ] Publish the official libc binding package (a subset of POSIX + Windows API)
• [ ] Establish the binding package publication specification
• [ ] Documentation: FFI best practices, memory ownership, concurrency safety contracts

Relationship to Other RFCs

• RFC-001: FFI calls, as external functions, default to @block (do not participate in DAG scheduling)
• RFC-008: Scheduler decoupling design; FFI calls as independent tasks
• RFC-020: Scheduling-level detailed design of FFI nodes in the DAG, including Phi nodes, loop unrolling, etc.

Open Questions

• [ ] Does yx-bindgen need to be integrated into the build system (yaoxiang build)?
• [ ] How should FFI support on the WASM platform be designed? (WASM's import mechanism is completely different from dlopen)
• [ ] Is a cxx-bindgen needed to handle C++ name mangling? (Optional, to be considered after v1.0)

---

Appendix A: Design Decision Records

Decision	Decision	Reason	Date	Recorder
Unify FFI entry point	Keep only native("symbol")	Avoid API fragmentation	2026-05-29	Chenxu
Exclude try_call	Not implemented	Redundant; Result type exists	2026-05-29	Chenxu
Exclude verify_signature	Not implemented	Runtime doing compiler's job	2026-05-29	Chenxu
Community-maintained bindings → toolchain	yx-bindgen auto-generation	An unworkable fantasy	2026-05-29	Chenxu
OS error code	FfiError must carry os_error	An undebuggable API is useless	2026-05-29	Chenxu
Zero syntax changes	Implementation via libraries	Core simplicity principle	2026-03-14	Chenxu
Dynamic library loading	Use dlopen/dlsym	Standard OS interfaces	2026-03-14	Chenxu
Error handling	Use the Result type	Consistency	2026-03-14	Chenxu

Appendix B: Example Code

Complete Example: Using a C Library

# Load the C math library
libm = ffi.load_library("libm.so")

# Register C symbols into the runtime table (yx-bindgen does this at compile time)
ffi.register_library_symbols(libm, ["sin", "cos", "sqrt"])

# Use through native declarations
sin_f: (x: Float) -> Float = native("sin")
cos_f: (x: Float) -> Float = native("cos")

# Direct call
result = sin_f(3.14159 / 2)

# Use Result when calling C functions that may fail
file_open: (path: *const u8, mode: *const u8) -> Result(*mut opaque, Int)
    = native("fopen")

Using yx-bindgen

# Auto-generate all declarations, no need to hand-write
yx-bindgen --header /usr/include/math.h --output math_bindings.yx

# Import in YaoXiang
import "math_bindings.yx"
# sin_f / cos_f etc. have been automatically declared as native("sin") / native("cos")

---

References

• RFC-001: Spawn Model and Error Handling System
• RFC-008: Runtime Concurrency Model and Scheduler Decoupling Design
• FFI Implementation Plan
• Python ctypes documentation
• Rust libloading crate