Go LLM SDK for streaming, tool-calling AI backends (plus frontend React lib)

Posted by matryer 4 days ago

Counter60Comment16OpenOriginal

Comments

Comment by iJohnDoe 4 days ago

Can someone explain the purpose of this? What are the advantages or use-cases? What type of project would I use this with?

If I was going to write a Go application to utilize the OpenAI API, is there a reason to use this vs. just making calls directly to the OpenAI API?

Genuine questions so I can understand and learn more.

Comment by zknill 4 days ago

The original Typescript version of the AI SDK has two parts. A backend part called Vercel AI-SDK, and a frontend part called AI UI SDK.

When you make an LLM call, if you use the streaming responses (typical of token streaming into applications) then the response from the LLM provider comes in a specific format; mostly made up of individual json events, one for each text-delta or token generated, and some framing around start/stop. That API format differs across different LLM providers, and also isn't that convenient to process for what you actually want to show in the client/browser.

The backend portion of these sdks (AI-SDK) are an abstraction over the LLM provider, and re-write the 'events' that you get back from the LLM provider into an abstracted json format that's a little bit easier to work with. And not coupled to a specific provider.

The frontend portion of these SDKs (AI UI SDK) helps you to render those message into a frontend interface, and uses the same abstracted format that the backend uses.

There's one big missing piece though, and that's the fact that the transport between frontend and backend is over a single HTTP+SSE stream. If that connection fails, all the work that the backend is doing cannot be transported to the frontend anymore. The HTTP connection also doesn't allow for useful features like cancellation, steering, interruption, and multiple devices.

If you want those, checkout Ably AI Transport, which targeting the _transport_ section between backend and frontend, to make sure you get reliable experiences in the UI regardless of which AI provider or SDKs you use.

Disclaimer, I work for ably on this product:

https://ably.com/docs/ai-transport

Comment by zknill 4 days ago

This is a backend written in go that's wire-protocol compatible with the (catchy named) Vercel AI UI SDK; which is an SDK for helping to build AI chat apps. The original Vercel AI SDK backend is in typescript.

I think they really missed the opportunity to add in resumable streaming after a dropped SSE connection, or multi-device support. Because these SDKs are easy to drop-in to your applications, but hard once you actually try and build resilient systems on top of them. Systems that can heal from dropped connections, or support multiple users or multiple devices.

Comment by verdverm 4 days ago

ADK seems like a better thought through SDK family, I use Go, Py is most mature, also TS and Java. The docs still need a lot of work to cover all the languages, it's pretty hit or miss right now, but one can also have a coding agent look at the source and an example for another language. Notably to your complaint, I have a coding agent that if the client drops, the llm/agent keeps working in the background. This requires it being set up as a server, which implies tool calls happen there instead of where the client is. I used Dagger as the "sandbox" more for easy time traveling / forking than being a sandbox. The well designed interfaces that allow custom implementations are what hooked me on ADK.

https://adk.dev/get-started/about/ | https://github.com/google/adk-go

Comment by ivanvanderbyl 3 days ago

Agreed, ADK is really well designed. We’ve used it as the harness for our main product since about January and it’s been solid.

Instead of writing a Go converter to use it with AI SDK Elements, we added this client side and use ADKs SSE streaming and recovery out of the box.

Comment by gregfurman 4 days ago

This looks great! Wonder how it compares to https://github.com/genkit-ai/genkit since they also offer similar SDKs and support for different model providers.

Comment by throwaway894345 4 days ago

I wonder how these LLM SDKs compare. Is this in the same category as Google’s ADK? Or maybe ADK is a higher level (agents are a higher level than models)?

Comment by helsinki 3 days ago

I've been building github.com/fugue-labs/gollem for five months.

Comment by Merkur 4 days ago

I don’t understand - Why did they write this? It seems to add nothing to the domain?

Comment by jnstrdm05 4 days ago

Very nice! Thanks for putting this out there

Comment by yakimbo90 4 days ago

[flagged]

Comment by champagnepapi 4 days ago

I’m curious to know if this was architected/designed by an actual human or is this just AI slop?

These days I am very very cautious about the quality of packages. I’ve become a lot more reluctant/hesitant (which is not to say prior I was in favor of just adding packages left and right). I was still conservative, but these days I’ve become much much more conservative when it comes to adding a package to a production system.

I will review this later and report back for the community

Comment by geodel 4 days ago

From their Agent.md:

"Go port of Vercel's AI SDK (github.com/grafana/ai-sdk), providing streaming LLM orchestration wire-compatible with @ai-sdk/react"

Comment by nkapila6 4 days ago

What is not written by AI nowadays..? Unless the library has strict no AI policy like Zig, I don't see AI going away.. but I would love to learn how you audit a huge AI written codebase atp?

Btw, the emdash in the repo description is a giveaway too.

Comment by champagnepapi 4 days ago

There's a big difference between "written by AI" and a piece of software being designed / architected by a person or a person in-conjunction with AI.

Comment by win311fwg 4 days ago

The AI SDK being ported was first released in 2023, so it was almost certainly designed/architected by a person. "Vibe coding" wasn't a thing until 2025.

However, that person worked for Vercel, so be aware that the design is apt to be questionable. Another comment has already pointed out some of the core design flaws, as is emblematic of everything that comes out of Vecel.

But you may find it useful if you have to integrate into that ecosystem.

Comment by 8bitsout 4 days ago

I personally wouldn't classify it as slop from my own very quick glance of the code (who reads the code these days, amirite?) - seems pretty reasonable. Someone already mentioned that in AGENTS.md it is stated that this is a port, so that answers the design question. Out of interest, what is your definition of AI slop?

Comment by 4 days ago