RFC-021: Library-Driven FFI Extension and Cross-Language Calling Support

⚠️ Deprecated: This document has been deprecated, and its content has been merged into RFC-026: FFI Core Mechanism.

References:
RFC-001: Concurrent Model and Error Handling System
RFC-008: Runtime Concurrency Model and Scheduler Decoupling Design
FFI Implementation Plan

Abstract

This document proposes a library-driven FFI (Foreign Function Interface) extension scheme. The sole entry point for FFI is the native("symbol") declaration + FfiRegistry runtime registration table, with no second mechanism introduced in the core. Above this, the standard library provides capabilities such as dynamic library loading and cross-language calling bindings. Language-specific calling bindings (such as C, Python, JavaScript) are automatically generated by the official toolchain or written by projects as needed.

Motivation

Deficiencies of Current Implementation

The current FFI implementation already has the following capabilities:

native("symbol") syntax for declaring external functions
FfiRegistry function registration table

But the functionality is relatively limited:

Lacks dynamic library loading support
No infrastructure for cross-language calling
Missing automated binding generation tools

Design Philosophy

YaoXiang follows the principle of "simple core, complexity sunk into libraries":

Good Taste: The language's responsibility is to provide atomic capabilities, not an all-encompassing feature set. Complexity should be solved through libraries, not piled onto the compiler.

Therefore, this scheme:

✅ Zero syntax changes — Fully backward compatible, FFI entry point is only native("symbol")
✅ Library as language — Features extended through the standard library
✅ Toolchain automation — Bindings automatically generated by yx-bindgen, not manually maintained
✅ Progressive enhancement — Developers introduce features as needed

Proposal

1. Core FFI Library Enhancement

Extend the std.ffi module. Note: All calls to external functions still use the native("symbol") declaration, and std.ffi only provides auxiliary capabilities.

1.1 Dynamic Library Loading

yaoxiang

import ffi

# Load dynamic library (.so/.dll/.dylib)
lib = ffi.load_library("./libmyext.so")

# Get function symbols from library, return native-usable symbol names
ffi.register_library_symbols(lib, [
    "my_function",
    "another_func",
])

load_library returns a DynamicLibrary handle, and register_library_symbols registers the symbol names into the FfiRegistry's known table. Users then still use them via native declaration:

yaoxiang

my_func: (a: Int, b: Int) -> Int = native("my_function")

No second set of calling syntax, no try_call wrapping.

1.2 Library Management

yaoxiang

# List loaded libraries
loaded = ffi.loaded_libraries()

# Unload library
ffi.unload_library(lib)

# Library version check
ffi.check_version(lib, "1.0.0")

1.3 Symbol Resolution

yaoxiang

# Find symbol by name (returns Symbol structure)
sin_sym = ffi.dlsym("libm.so", "sin")

Cross-language calling conventions and type conversion are not handled at runtime through universal wrappers, but are generated statically by yx-bindgen at compile time.

2. Dynamic Library Loading Implementation

2.1 Core Data Structures

rust

pub struct DynamicLibrary {
    handle: *mut std::ffi::c_void,
    path: String,
}

impl DynamicLibrary {
    pub fn load(path: &str) -> Result<Self, FfiError>;
    pub fn get_symbol(&self, name: &str) -> Result<*mut std::ffi::c_void, FfiError>;
    pub fn unload(self) -> Result<(), FfiError>;
}

2.2 Error Types

rust

pub enum FfiError {
    LibraryNotFound { name: String, os_error: Option<OsError> },
    SymbolNotFound { name: String, os_error: Option<OsError> },
    CallFailed { message: String, os_error: Option<OsError> },
    Timeout,
}

pub struct OsError {
    pub code: i32,
    pub message: String,
}

OsError carries platform-native error codes (dlerror() on Linux, GetLastError() on Windows), ensuring debuggability.

3. Multi-Language Bindings: Toolchain Solution

Abandon the fantasy of "community language maintainers writing binding libraries." Instead, use official toolchain automatic generation.

3.1 Architecture Design

┌───────────────────────────────────────────────┐
│  YaoXiang Code                                │
│                                               │
│  // Users only write native declarations      │
│  my_func: (a: Int) -> Int = native("my_func")│
└───────────────────────────────────────────────┘
         ↑                          ↑
         |  Compile time            |  Runtime
┌──────────────────┐   ┌────────────────────────┐
│  yx-bindgen      │   │  std.ffi + FfiRegistry  │
│  (C header → .yx)│   │  - dlopen/dlsym         │
│                  │   │  - LoadLibrary/GetProc  │
└──────────────────┘   └────────────────────────┘

3.2 Binding Generator (`yx-bindgen`)

yx-bindgen is an independent CLI tool that generates YaoXiang FFI binding code from C header files:

bash

yx-bindgen --header /usr/include/sqlite3.h --output sqlite3.yx

Example generated result:

yaoxiang

# Auto-generated, do not edit manually
# Source: /usr/include/sqlite3.h

sqlite3_open: (filename: *const u8, ppDb: *mut *mut opaque) -> Int
    = native("sqlite3_open")

sqlite3_close: (db: *mut opaque) -> Int
    = native("sqlite3_close")

sqlite3_exec: (
    db: *mut opaque,
    sql: *const u8,
    callback: *mut opaque,
    arg: *mut opaque,
    errmsg: *mut *mut u8,
) -> Int
    = native("sqlite3_exec")

yx-bindgen is officially maintained, ensuring:

Complete type mapping (int → Int, char* → *const u8, void* → *mut opaque)
Struct layout alignment (automatic #[repr(C)] equivalent)
Callback signature conversion

3.3 Officially Maintained Binding Packages

The YaoXiang core team does not commit to maintaining universal binding libraries for all languages, but provides an official libc binding package (POSIX + Windows API subset) as an FFI best practices example and basic capability.

Bindings for other languages and libraries:

Generated yourself using yx-bindgen
Can be published as YaoXiang packages (such as libsqlite3, libcurl, libsdl2)
Core team is not responsible for maintenance, but provides package publishing and version management mechanisms

4. Type Conversion Layer

4.1 Compile-Time Type Mapping

Type conversion is not done through runtime wrappers, but is determined statically at yx-bindgen generation time:

C Type	YaoXiang Type	Conversion Method
`int`	`Int`	Pass by value
`char*`	`*const u8`	Pointer passing
`void*`	`*mut opaque`	Opaque pointer
`struct T`	`extern struct T`	Memory layout matching
`int*`	`*mut Int`	Pointer passing (mutable)
`const int*`	`*const Int`	Pointer passing (read-only)

4.2 Manual Conversion (Standard Library Assistance)

yaoxiang

# Explicit conversion
raw_ptr = ffi.to_pointer(my_bytes)
c_string = ffi.to_c_string(my_string)

5. Memory Ownership Model

5.1 Basic Principles

Every memory allocation crossing the FFI boundary must clearly answer two questions:

Who allocates? (C side malloc / YaoXiang side runtime)
Who frees? (C side free / YaoXiang side runtime)

When generating, yx-bindgen adds annotations for common patterns:

yaoxiang

# C-allocated, caller frees
sqlite3_exec: (...) -> Int
    = native("sqlite3_exec")
    # memory: C-allocated, caller must free errmsg via sqlite3_free

# Caller-allocated pointer
read: (fd: Int, buf: *mut u8, count: Int) -> Int
    = native("read")
    # memory: caller-allocated buf

The runtime does not perform automatic memory management for pointer references crossing FFI boundaries — ownership explicitly falls on the caller.

5.2 String Handling

When a char* returned by a C function is converted to a YaoXiang String, it is copied immediately. Ownership of the original pointer is determined by the C function (declared via annotation), not automatically freed.

6. Security Considerations

6.1 Concurrency Safety

FFI function calls do not participate in DAG scheduling by default, treated as blocking operations. C functions confirmed to be reentrant can be marked with @concurrent:

yaoxiang

# Pure function, no global state, can be concurrent
sin_safe: (x: Float) -> Float = native("sin")
    # reentrant: true

# Has global state, cannot be concurrent
strtok: (s: *const u8, delim: *const u8) -> *const u8 = native("strtok")
    # reentrant: false

yx-bindgen annotates reentrancy information for standard C library functions where possible (variants like _r for strtok, etc.).

Requirements for async callers: Before calling an FFI function, the caller must ensure the target function is reentrant. The runtime does not perform automatic detection — this is an impossible problem to solve statically.

6.2 Error Isolation

FFI call errors propagate through the Result type (if the function declares a Result return type)
Timeout mechanism prevents external function deadlocks

yaoxiang

# Call with timeout (implemented at FfiRegistry layer)
result = ffi.call_with_timeout("blocking_func", 5000)  # 5s timeout

6.3 Pointer Safety

Pointer parameters require unsafe marking on the YaoXiang side
Pointer lifetimes crossing FFI boundaries are guaranteed by the caller

7. Compiler Changes

Zero syntax changes — only the native("symbol") declaration is needed, already implemented in the current compiler.

Add to interpreter/runtime:

Dynamic library loading instructions (FFI bindings for DynamicLibrary)
Timeout mechanism

8. Features Not Accepted

The following features are explicitly excluded after review and will no longer be considered for RFC:

ffi.try_call: Redundant, native + Result return type already exists
ffi.verify_signature: Runtime doing compiler's work is the wrong abstraction level
ffi.async_call: Needs to wait until the reentrancy contract model is clarified
Community-maintained binding table: Not executable, replaced with yx-bindgen toolchain solution

Tradeoffs

Advantages

✅ Zero syntax changes — FFI entry point is only native("symbol"), fully backward compatible
✅ Library as language — Features progressively introduced through the standard library
✅ Toolchain-driven — yx-bindgen automatically handles binding generation
✅ Memory safety — Ownership model is explicit, no use-after-free from automatic reclamation
✅ Debuggable — Errors carry OS native error codes

Disadvantages

⚠️ Type safety is limited by C header file expressiveness (void* cannot be statically distinguished)
⚠️ yx-bindgen requires ongoing maintenance to keep up with C standard evolution
⚠️ Bindings for non-C languages (Python/JS/Java) need to be handled by each project individually, no unified solution

Implementation Strategy

Phase 1: Core Library (v0.7)

[ ] Extend std.ffi module
[ ] Implement DynamicLibrary structure
[ ] Support Linux/macOS (dlopen/dlsym)
[ ] Support Windows (LoadLibrary/GetProcAddress)
[ ] Add timeout mechanism to runtime
[ ] Unit tests

Phase 2: yx-bindgen (v0.8)

[ ] Implement C header file parser (based on existing Clang bindings or hand-written parser)
[ ] Type mapping system
[ ] Generate native("symbol") declarations
[ ] Generate struct layouts
[ ] Integration tests: generate bindings for real C libraries like SQLite3, libcurl

Phase 3: Ecosystem Foundation (v0.9)

[ ] Publish official libc binding package (POSIX + Windows API subset)
[ ] Establish binding package publishing specifications
[ ] Documentation: FFI best practices, memory ownership, concurrency safety contracts

Relationship with Other RFCs

RFC-001: FFI calls as external functions, default @block (not participating in DAG scheduling)
RFC-008: Scheduler decoupling design, FFI calls as independent tasks
RFC-020: FFI node scheduling semantics in DAG, Phi nodes, loop unrolling, and other detailed scheduling-level designs

Open Questions

[ ] Should yx-bindgen be integrated into the build system (yaoxiang build)?
[ ] How should FFI support for the WASM platform be designed? (WASM's import mechanism is completely different from dlopen)
[ ] Is cxx-bindgen needed to handle C++ name mangling? (Optional, consider after v1.0)

Appendix A: Design Decision Record

Decision	Decision	Reason	Date	Recorder
Unify FFI entry point	Only keep `native("symbol")`	Avoid API fragmentation	2026-05-29	晨煦
Exclude `try_call`	Not implemented	Redundant, Result type already exists	2026-05-29	晨煦
Exclude `verify_signature`	Not implemented	Runtime doing compiler's work	2026-05-29	晨煦
Community bindings → toolchain	`yx-bindgen` auto-generation	Infeasible fantasy	2026-05-29	晨煦
OS error codes	`FfiError` must carry `os_error`	Undebuggable API is useless	2026-05-29	晨煦
Zero syntax changes	Depend on library implementation	Core simplicity principle	2026-03-14	晨煦
Dynamic library loading	Use dlopen/dlsym	Standard OS interface	2026-03-14	晨煦
Error handling	Use Result type	Consistency	2026-03-14	晨煦

Appendix B: Example Code

Complete Example: Using C Libraries

yaoxiang

# Load C math library
libm = ffi.load_library("libm.so")

# Register C symbols to runtime table (yx-bindgen does this at compile time)
ffi.register_library_symbols(libm, ["sin", "cos", "sqrt"])

# Use via native declaration
sin_f: (x: Float) -> Float = native("sin")
cos_f: (x: Float) -> Float = native("cos")

# Direct call
result = sin_f(3.14159 / 2)

# Use Result when calling C functions that may fail
file_open: (path: *const u8, mode: *const u8) -> Result(*mut opaque, Int)
    = native("fopen")

Using yx-bindgen

bash

# Auto-generate all declarations, no manual writing needed
yx-bindgen --header /usr/include/math.h --output math_bindings.yx

# Import in YaoXiang
import "math_bindings.yx"
# sin_f / cos_f etc. already auto-declared as native("sin") / native("cos")

RFC-021: Library-Driven FFI Extension and Cross-Language Calling Support ​

Abstract ​

Motivation ​

Deficiencies of Current Implementation ​

Design Philosophy ​

Proposal ​

1. Core FFI Library Enhancement ​

1.1 Dynamic Library Loading ​

1.2 Library Management ​

1.3 Symbol Resolution ​

2. Dynamic Library Loading Implementation ​

2.1 Core Data Structures ​

2.2 Error Types ​

3. Multi-Language Bindings: Toolchain Solution ​

3.1 Architecture Design ​

3.2 Binding Generator (yx-bindgen) ​

3.3 Officially Maintained Binding Packages ​

4. Type Conversion Layer ​

4.1 Compile-Time Type Mapping ​

4.2 Manual Conversion (Standard Library Assistance) ​

5. Memory Ownership Model ​

5.1 Basic Principles ​

5.2 String Handling ​

6. Security Considerations ​

6.1 Concurrency Safety ​

6.2 Error Isolation ​

6.3 Pointer Safety ​

7. Compiler Changes ​

8. Features Not Accepted ​

Tradeoffs ​

Advantages ​

Disadvantages ​

Implementation Strategy ​

Phase 1: Core Library (v0.7) ​

Phase 2: yx-bindgen (v0.8) ​

Phase 3: Ecosystem Foundation (v0.9) ​

Relationship with Other RFCs ​

Open Questions ​

Appendix A: Design Decision Record ​

Appendix B: Example Code ​

Complete Example: Using C Libraries ​

Using yx-bindgen ​

References ​

RFC-021: Library-Driven FFI Extension and Cross-Language Calling Support

Abstract

Motivation

Deficiencies of Current Implementation

Design Philosophy

Proposal

1. Core FFI Library Enhancement

1.1 Dynamic Library Loading

1.2 Library Management

1.3 Symbol Resolution

2. Dynamic Library Loading Implementation

2.1 Core Data Structures

2.2 Error Types

3. Multi-Language Bindings: Toolchain Solution

3.1 Architecture Design

3.2 Binding Generator (`yx-bindgen`)

3.3 Officially Maintained Binding Packages

4. Type Conversion Layer

4.1 Compile-Time Type Mapping

4.2 Manual Conversion (Standard Library Assistance)

5. Memory Ownership Model

5.1 Basic Principles

5.2 String Handling

6. Security Considerations

6.1 Concurrency Safety

6.2 Error Isolation

6.3 Pointer Safety

7. Compiler Changes

8. Features Not Accepted

Tradeoffs

Advantages

Disadvantages

Implementation Strategy

Phase 1: Core Library (v0.7)

Phase 2: yx-bindgen (v0.8)

Phase 3: Ecosystem Foundation (v0.9)

Relationship with Other RFCs

Open Questions

Appendix A: Design Decision Record

Appendix B: Example Code

Complete Example: Using C Libraries

Using yx-bindgen

References