文章目录
- Using the String Buffer Library
- Buffer Objects
- Buffer Method Overview
- Buffer Creation and Management
- `local buf = buffer.new([size [,options]]) local buf = buffer.new([options])`
- `buf = buf:reset()`
- `buf = buf:free()`
- Buffer Writers
- `buf = buf:put([str|num|obj] [,…])`
- `buf = buf:putf(format, …)`
- `buf = buf:putcdata(cdata, len)`FFI
- `buf = buf:set(str) `
- `buf = buf:set(cdata, len)`FFI
- `ptr, len = buf:reserve(size)`FFI
- `buf = buf:commit(used)`FFI
- Buffer Readers
- `len = #buf`
- `res = str|num|buf .. str|num|buf […]`
- `buf = buf:skip(len)`
- `str, … = buf:get([len|nil] [,…])`
- `str = buf:tostring() `
- `str = tostring(buf)`
- `ptr, len = buf:ref()`FFI
- Serialization of Lua Objects
- 例子:序列化 Lua 对象
- Error handling
- FFI caveats
- 例子说明:
The string buffer library allows high-performance manipulation of string-like data.
Unlike Lua strings, which are constants, string buffers are mutable sequences of 8-bit (binary-transparent) characters. Data can be stored, formatted and encoded into a string buffer and later converted, extracted or decoded.
The convenient string buffer API simplifies common string manipulation tasks, that would otherwise require creating many intermediate strings. String buffers improve performance by eliminating redundant memory copies, object creation, string interning and garbage collection overhead. In conjunction with the FFI library, they allow zero-copy operations.
The string buffer library also includes a high-performance serializer for Lua objects.
Using the String Buffer Library
The string buffer library is built into LuaJIT by default, but it’s not loaded by default. Add this to the start of every Lua file that needs one of its functions:
local buffer = require("string.buffer")
The convention for the syntax shown on this page is that buffer
refers to the buffer library and buf
refers to an individual buffer object.
Please note the difference between a Lua function call, e.g. buffer.new()
(with a dot) and a Lua method call, e.g. buf:reset()
(with a colon).
Buffer Objects
A buffer object is a garbage-collected Lua object. After creation with buffer.new()
, it can (and should) be reused for many operations. When the last reference to a buffer object is gone, it will eventually be freed by the garbage collector, along with the allocated buffer space.
Buffers operate like a FIFO (first-in first-out) data structure. Data can be appended (written) to the end of the buffer and consumed (read) from the front of the buffer. These operations may be freely mixed.
The buffer space that holds the characters is managed automatically — it grows as needed and already consumed space is recycled. Use buffer.new(size)
and buf:free()
, if you need more control.
The maximum size of a single buffer is the same as the maximum size of a Lua string, which is slightly below two gigabytes. For huge data sizes, neither strings nor buffers are the right data structure — use the FFI library to directly map memory or files up to the virtual memory limit of your OS.
Buffer Method Overview
- The
buf:put*()
-like methods append (write) characters to the end of the buffer. - The
buf:get*()
-like methods consume (read) characters from the front of the buffer. - Other methods, like
buf:tostring()
only read the buffer contents, but don’t change the buffer. - The
buf:set()
method allows zero-copy consumption of a string or an FFI cdata object as a buffer. - The FFI-specific methods allow zero-copy read/write-style operations or modifying the buffer contents in-place. Please check the FFI caveats below, too.
- Methods that don’t need to return anything specific, return the buffer object itself as a convenience. This allows method chaining, e.g.:
buf:reset():encode(obj)
orbuf:skip(len):get()
Buffer Creation and Management
local buf = buffer.new([size [,options]]) local buf = buffer.new([options])
Creates a new buffer object.
The optional size
argument ensures a minimum initial buffer size. This is strictly an optimization when the required buffer size is known beforehand. The buffer space will grow as needed, in any case.
The optional table options
sets various serialization options.
buf = buf:reset()
Reset (empty) the buffer. The allocated buffer space is not freed and may be reused.
buf = buf:free()
The buffer space of the buffer object is freed. The object itself remains intact, empty and may be reused.
Note: you normally don’t need to use this method. The garbage collector automatically frees the buffer space, when the buffer object is collected. Use this method, if you need to free the associated memory immediately.
Buffer Writers
buf = buf:put([str|num|obj] [,…])
Appends a string str
, a number num
or any object obj
with a __tostring
metamethod to the buffer. Multiple arguments are appended in the given order.
Appending a buffer to a buffer is possible and short-circuited internally. But it still involves a copy. Better combine the buffer writes to use a single buffer.
buf = buf:putf(format, …)
Appends the formatted arguments to the buffer. The format
string supports the same options as string.format()
.
buf = buf:putcdata(cdata, len)
FFI
Appends the given len
number of bytes from the memory pointed to by the FFI cdata
object to the buffer. The object needs to be convertible to a (constant) pointer.
buf = buf:set(str)
buf = buf:set(cdata, len)
FFI
This method allows zero-copy consumption of a string or an FFI cdata object as a buffer. It stores a reference to the passed string str
or the FFI cdata
object in the buffer. Any buffer space originally allocated is freed. This is not an append operation, unlike the buf:put*()
methods.
After calling this method, the buffer behaves as if buf:free():put(str)
or buf:free():put(cdata, len)
had been called. However, the data is only referenced and not copied, as long as the buffer is only consumed.
In case the buffer is written to later on, the referenced data is copied and the object reference is removed (copy-on-write semantics).
The stored reference is an anchor for the garbage collector and keeps the originally passed string or FFI cdata object alive.
ptr, len = buf:reserve(size)
FFI
buf = buf:commit(used)
FFI
The reserve
method reserves at least size
bytes of write space in the buffer. It returns an uint8_t *
FFI cdata pointer ptr
that points to this space.
The available length in bytes is returned in len
. This is at least size
bytes, but may be more to facilitate efficient buffer growth. You can either make use of the additional space or ignore len
and only use size
bytes.
The commit
method appends the used
bytes of the previously returned write space to the buffer data.
This pair of methods allows zero-copy use of C read-style APIs:
local MIN_SIZE = 65536
repeatlocal ptr, len = buf:reserve(MIN_SIZE)local n = C.read(fd, ptr, len)if n == 0 then break end -- EOF.if n < 0 then error("read error") endbuf:commit(n)
until false
The reserved write space is not initialized. At least the used
bytes must be written to before calling the commit
method. There’s no need to call the commit
method, if nothing is added to the buffer (e.g. on error).
Buffer Readers
len = #buf
Returns the current length of the buffer data in bytes.
res = str|num|buf .. str|num|buf […]
The Lua concatenation operator ..
also accepts buffers, just like strings or numbers. It always returns a string and not a buffer.
Note that although this is supported for convenience, this thwarts one of the main reasons to use buffers, which is to avoid string allocations. Rewrite it with buf:put()
and buf:get()
.
Mixing this with unrelated objects that have a __concat
metamethod may not work, since these probably only expect strings.
buf = buf:skip(len)
Skips (consumes) len
bytes from the buffer up to the current length of the buffer data.
str, … = buf:get([len|nil] [,…])
Consumes the buffer data and returns one or more strings. If called without arguments, the whole buffer data is consumed. If called with a number, up to len
bytes are consumed. A nil
argument consumes the remaining buffer space (this only makes sense as the last argument). Multiple arguments consume the buffer data in the given order.
Note: a zero length or no remaining buffer data returns an empty string and not nil
.
str = buf:tostring()
str = tostring(buf)
Creates a string from the buffer data, but doesn’t consume it. The buffer remains unchanged.
Buffer objects also define a __tostring
metamethod. This means buffers can be passed to the global tostring()
function and many other functions that accept this in place of strings. The important internal uses in functions like io.write()
are short-circuited to avoid the creation of an intermediate string object.
ptr, len = buf:ref()
FFI
Returns an uint8_t *
FFI cdata pointer ptr
that points to the buffer data. The length of the buffer data in bytes is returned in len
.
The returned pointer can be directly passed to C functions that expect a buffer and a length. You can also do bytewise reads (local x = ptr[i]
) or writes (ptr[i] = 0x40
) of the buffer data.
In conjunction with the skip
method, this allows zero-copy use of C write-style APIs:
repeatlocal ptr, len = buf:ref()if len == 0 then break endlocal n = C.write(fd, ptr, len)if n < 0 then error("write error") endbuf:skip(n)
until n >= len
Unlike Lua strings, buffer data is not implicitly zero-terminated. It’s not safe to pass ptr
to C functions that expect zero-terminated strings. If you’re not using len
, then you’re doing something wrong.
Lua_Objects_166">Serialization of Lua Objects
略过
Lua__170">例子:序列化 Lua 对象
local buffer = require("string.buffer")-- 创建一个元表
local mt1 = { __index = function(t, k) return "default" end }
local mt2 = { __index = function(t, k) return "another default" end }-- 创建需要序列化的表
local t1 = setmetatable({ key1 = "value1", key2 = "value2" }, mt1)
local t2 = setmetatable({ key1 = "value3", key2 = "value4" }, mt2)-- 定义字典和元表的数组
local dict = {"key1", "key2"}
local metatable = {mt1, mt2}-- 使用 buffer.new() 进行序列化
local buffer_obj = buffer.new({dict = dict,metatable = metatable
})-- 假设序列化后的数据为序列化函数 `encode()`
local serialized_data = buffer_obj:encode({t1, t2})-- 反序列化
local decoded_data = buffer_obj:decode(serialized_data)-- 访问解码后的数据
for _, tbl in ipairs(decoded_data) doprint(tbl.key1, tbl.key2)
end
Error handling
Many of the buffer methods can throw an error. Out-of-memory or usage errors are best caught with an outer wrapper for larger parts of code. There’s not much one can do after that, anyway.
OTOH, you may want to catch some errors individually. Buffer methods need to receive the buffer object as the first argument. The Lua colon-syntax obj:method()
does that implicitly. But to wrap a method with pcall()
, the arguments need to be passed like this:
local ok, err = pcall(buf.encode, buf, obj)
if not ok then-- Handle error in err.
end
FFI caveats
The string buffer library has been designed to work well together with the FFI library. But due to the low-level nature of the FFI library, some care needs to be taken:
First, please remember that FFI pointers are zero-indexed. The space returned by buf:reserve()
and buf:ref()
starts at the returned pointer and ends before len
bytes after that.
I.e. the first valid index is ptr[0]
and the last valid index is ptr[len-1]
. If the returned length is zero, there’s no valid index at all. The returned pointer may even be NULL
.
The space pointed to by the returned pointer is only valid as long as the buffer is not modified in any way (neither append, nor consume, nor reset, etc.). The pointer is also not a GC anchor for the buffer object itself.
Buffer data is only guaranteed to be byte-aligned. Casting the returned pointer to a data type with higher alignment may cause unaligned accesses. It depends on the CPU architecture whether this is allowed or not (it’s always OK on x86/x64 and mostly OK on other modern architectures).
FFI pointers or references do not count as GC anchors for an underlying object. E.g. an array
allocated with ffi.new()
is anchored by buf:set(array, len)
, but not by buf:set(array+offset, len)
. The addition of the offset creates a new pointer, even when the offset is zero. In this case, you need to make sure there’s still a reference to the original array as long as its contents are in use by the buffer.
例子说明:
- 正常的引用:当你使用
buf:set(array, len)
时,这个array
是一个通过 FFI 创建的数组,它会被作为buf
的参数传递进去。在这种情况下,array
被引用,并且只要buf
依然存在并持有这个引用,array
不会被垃圾回收器回收。这里array
是一个“垃圾回收锚点”(GC anchor),即它会被垃圾回收器追踪。 - 添加偏移量后的情况:当你通过
array + offset
创建一个新的指针时(即通过加偏移量来引用array
中的某个元素),这时创建的是一个新的指针对象。即使offset
为零,array + offset
仍然会被视为一个新的指针。这个新的指针不会自动被垃圾回收器追踪,因为它并没有直接引用array
。- 问题:这意味着,如果你只使用
array + offset
(即偏移后的指针),垃圾回收器可能会认为原始的array
对象不再被使用,最终回收掉array
,即使buf
仍然依赖于它的内容。这会导致访问已回收的内存,造成未定义行为或崩溃。
- 问题:这意味着,如果你只使用
Even though each LuaJIT VM instance is single-threaded (but you can create multiple VMs), FFI data structures can be accessed concurrently. Be careful when reading/writing FFI cdata from/to buffers to avoid concurrent accesses or modifications. In particular, the memory referenced by buf:set(cdata, len)
must not be modified while buffer readers are working on it. Shared, but read-only memory mappings of files are OK, but only if the file does not change.