Creating C closures from Lua closures

Posted by publicdebates 5 days ago

Counter59Comment15OpenOriginal

Comments

Comment by widdershins 1 day ago

This doesn't seem like something that should require generating assembly to solve. Couldn't `CALLBACK` just return a table, which contains both the userdata pointer to `REAL_CALLBACK` and a value for `findex`, eliminating the global variable? Then `Add` could extract that. You could even make the table returned by `CALLBACK` callable in Lua with a metatable.

Or, if you're worried about performance/memory, you could allocate a struct for `CALLBACK` to return as userdata, containing the function pointer and `findex`. If you made a little allocator for these it would be extremely fast.

I'm sure I'm missing things, but the solution you chose feels like the nuclear option.

Comment by comex 1 day ago

The code is passing the function pointers into Win32 APIs, so the caller side isn’t controlled; the callbacks have to work as native C function pointers.

This was probably posted in response to this other link from two days ago, which is about about JIT-compiling wndproc callbacks in particular; the comments discuss the “proper” way to do it, which is to use GWLP_USERDATA:

https://news.ycombinator.com/item?id=46259334

At least, that’s the proper way to do it if you control the entire application. But for what’s apparently a thin wrapper that exposes Win32 APIs directly to Lua, I can understand picking an approach that makes life easier for the Lua code author, even if it’s hackier under the hood. It also avoids the need to write custom behavior for each API.

Comment by publicdebates 21 hours ago

It was inspired by that post, but not in response to it. Over the 3 or 4 months that I took writing lowkPRO, one of the features I'm most proud of is this, the ability for it to create C callbacks on the fly that just call Lua callbacks and return the value to C. And I would argue that it's not hacky but just well-engineered. After all, everything compiles to assembly at the end of the day, right?

Comment by CodesInChaos 22 hours ago

Well designed C APIs have a context/userdata parameter on their callbacks, which is registered and stored alongside the function pointer. Unfortunately WNDPROC lacks this parameter.

GWLP_USERDATA should be the best option, though the API for setting it and setting the WNDPROC being separate looks error prone.

Comment by publicdebates 21 hours ago

The bigger problem is that I would have to automate this for every callback type in the Windows API, and there's no guarantee that all of them follow the same 2 or 3 patterns for passing the context pointer into the callback. This solution works great, even if it is a bit wasteful of the ctx ptr.

Comment by not_a9 14 hours ago

Fun fact: LuaJIT FFI actually has a similar feature. You can do funky things like detour hooking with this functionality too.

https://luajit.org/ext_ffi_semantics.html

Comment by halayli 21 hours ago

I feel libffi is better suited and supports most platforms.

Comment by lkuty 20 hours ago

Tsoding made a video about it : https://www.youtube.com/watch?v=0o8Ex8mXigU

Comment by rurban 13 hours ago

The ffcall library even more

Comment by halayli 7 hours ago

wasn't aware of ffcall, it's definitely a better fit

Comment by psychoslave 22 hours ago

What’s the thing emulating so much activity around closures lately?

https://news.ycombinator.com/item?id=46228597

https://news.ycombinator.com/item?id=46259334

Also talking about the Knuth boy/man test: https://news.ycombinator.com/item?id=46020151

Not a bad thing, but that really question if there is some active micro-

Comment by shakna 22 hours ago

Why VirtualAlloc?

Lua has its own allocator, which will also collect for you. lua_newuserdata. At the expense of having to set executable yourself, but without all the inbuilt inefficiencies the article points out.

Comment by publicdebates 21 hours ago

Can you set arbitrary memory allocated by malloc (e.g. HeapAlloc in Windows) to executable, via VirtualProtect or something else? If so, that's news to me. I thought it had to be memory allocated by VirtualAlloc only.

That said, I'm not sure that solution beats out this one. I'm using a linked list on top of an arena allocator in practice, which means allocations happen once every 0x10000 bytes. (A single C closure of mine takes up exactly 16 bytes, which includes the pointer to the next linked list node. I'm very happy with how it's designed.)

Comment by shakna 7 hours ago

Lua lets you set the allocator, so you don't have to manage the lifetime.

    void lua_setallocf (lua_State *L, lua_Alloc f, void *ud);

Comment by 17 hours ago

Comment by ufo 22 hours ago

I wonder if there would be a way to piggyback on top of GCC's nested function extension. It does something a little bit similar, with the dynamically generated functions.