So why is it it emerging as critical infrastructure for working with LLMs?

Pydantic lets you share your data models by generating clear, standardized representations of their definitions a la JSON Schema. In doing so, so Pydantic makes it trivial for your spaghetti code to interact with my fettuccini code, since it handles the translation to a global standard. These open standards are ubiquitous in nearly every public description of describing data models or web services.

It’s unsurprising then, that in training LLMs like ChatGPT on the entire internet, they’ve seen millions of well-documented JSON schemas and human descriptions of what those services do. This has given LLMs the uncanny ability to reason back and forth been unstructured data and structured data - after all, it’s seen hundreds of millions of examples of “if you want to send a Hawaiian Pizza to our service, you need to send {‘toppings’: [‘Canadian bacon’, ‘pineapple’, ‘garbage’]}.

Breeding LLMs to be both creative and conforming usually means constraining their choices, most often via interleaving them with rigid classic software, or code. We’ll let the LLM listen to you prattle on about what you want for dinner, and when you’re done talking we’ll make it create a valid pizza for our order_pizza function to carry on. We’ll tell it what it means to be valid with an JSON schema it understands, and we’ll use Pydantic to validate your pizza isn’t Hawaiian before ordering it.

If LLMs speak JSON schemas, its unsurprising then that the spoils of LLMs go to any language or utility that can easily broker a conversation between code and schema. Pydantic has earned its place in the Python ecosystem, but it benefitted from being one of the few sane options for brokering this conversation in Python when developers discovered how well LLMs speak JSON schemas.

If we prod deeper, this victory for Pydantic teaches us a more general lesson about the importance of quantizing your decision space. Put another way: today’s LLMs are better at paint-by-numbers than they are at making a masterpiece on a blank canvas. They’re better at speaking JSON schemas than your spaghetti code, they’re better at text-to-ORM than text-to-SQL, they’re better at text-to-component-library than text-to-html. The recipe behind many successful projects is finding the correct tessellation for an LLM to creatively in-paint than telling it to make your user another Picasso. For this moment in time when LLMs are better at paint-by-numbers, AI Engineers distinguish themselves by how well they can quantize these design decisions so they still appear to the end user as a masterpiece.

Looking forward, there are a few hiccups on the horizon for Pydantic and its cousins in other languages (e.g. Zod in TypeScript). LLMs are overfit to the hundreds of millions of examples that conform to the JSON 2019-09 draft. In December 2020 JSON Schema released its 2020-12 draft, which is sufficiently different to cause a headache (e.g. how it handles enums). Pydantic’s last major release conforms to this new specification, and many LLM frameworks that migrated to Pydantic V2 saw regressions in ChatGPT’s performance as a result.

This presents a beautiful conflict and form of amplification bias. JSON Schema draft 2019-09 is more ubiquitous in the training data of today’s LLMs. LLM applications will have more success using an outdated standard since it’s trained on it and slow to change. LLMs sticky training data risks creating a real bias towards using standards from 2017-2021, which in turn discourages adoption of new standards, products, and libraries which is ironically what we’re all trying to build.

Today we’re introducing Marvin 2.0, a new, streamlined version of our lightweight AI engineering toolkit. Marvin lets you build natural language interfaces that are reliable, scalable, and easy to trust.

You can upgrade right now:

pip install marvin -U

Marvin consists of a variety of useful tools, all designed to be used independently. Each one represents a common LLM use case, and packages that power into a simple, self-documenting interface. With 2.0, we’ve introduced dedicated functionality for the most frequent tasks and gone multi-modal!

Here’s an overview of all the new functionality:

General

🦾 Write custom AI-powered functions without source code

Text

🏷️ Classify text into categories

🔍 Extract structured entities from text

🪄 Transform text into structured data

✨ Generate synthetic data from a schema

Images

🖼️ Create images from text or functions

📝 Describe images with natural language

🏷️ Classify images into categories

🔍 Extract structured entities from images

🪄 Transform images into structured data

Audio

🎙️ Generate speech from text or functions

Interaction

🤖 Chat with assistants and use custom tools

🧭 Build applications that manage persistent state

Marvin is focused on making cutting-edge LLM technology feel just like any other function. Instead of building heavy pipelines, Marvin lets you add just the right amount of AI Magic™️ to your traditional software. Marvin 2.0 introduces a number of new, dedicated interfaces for the most common LLM use cases.

Classification

To give you a taste for what it’s like to use Marvin, here’s how to perform sentiment analysis by classifying text:

Entity Extraction

One of Marvin’s core use cases has always been structuring text into Python types, including Pydantic models. Here, we use the new extract function to automatically pull a list of people out of the text and create a “Person” model for each one:

Transforming images into data

Marvin 2.0 is fully multi-modal, with support for both images and audio! Here’s an example of transforming an image into a list of strings to see if we forgot any items on our shopping list. This also shows how every Marvin tool can be given natural language instructions for precise control:

Please note that image support will be beta until OpenAI finalizes the vision API.

AI functions

When the standard tools aren’t enough, Marvin’s AI functions let you combine any inputs, instructions, and output types to create custom AI-powered behaviors... without source code. These functions look and feel like normal Python functions, but when they are called, the result is generated on-demand by an LLM This is done safely, without generating or executing source code, by using the LLM as a sort of “runtime” to predict the function output. Marvin functions can handle complex use cases and behaviors that would be difficult or impossible to express as code. Here’s one that can generate structured recipe objects from a variety of inputs:

Assistants

OpenAI’s new assistants API handles many of the most complex aspects of building agents or chatbots, including memory, threading, code execution, and more. Marvin makes it simple to create and interact with assistants programmatically, including OpenAI tools like the code interpreter and any custom functions you give the assistant to call.

Please note that assistants support will be beta until OpenAI finalizes the assistants API.

Generating images

For all of Marvin’s advanced functionality, sometimes the most fun thing to do is making simple calls to generate speech or images. Here’s how to generate the image at the top of this post:

To learn more about Marvin, check out the docs, star us on GitHub, join us in Discord, or …x?… us on X.

Happy engineering!

Today we’re excited to release Marvin 1.5. We’re:

• Putting the `engineering` in prompt engineering with our new prompt_fn, and

• Speeding up serialization in Marvin by ~50x by transitioning to Pydantic V2.

Last week we released 1.4, and the response has been overwhelmingly positive. At its core, Marvin translates your Python code to English (“serialization”) and then calls a Large Language Model on your behalf (“execution”) before parsing its result. We shipped the first step of what we called the separation of ‘serialization’ and ‘execution’. This week we’re doubling down.

Prompt Functions

This week we introduce prompt functions, or prompt_fn. We now expose a low-level decorator that lets you write prompts as functions. This lets you build fully type-hinted prompts that other engineers can introspect, version, and test. Let’s take a look (long photo ahead):

Every parameter above is fully-customizable. There’s no black-box, or “magic” separating you from the prompt you want to write and the powerful serialization engine that powers Marvin. To see more examples check out our docs.

Pydantic V2 Migration

Pydantic is one of the most popular Python libraries today for validating and serializing data models. Not only is it useful as a developer, but it exposes really helpful components for turning a Python data model into an industry-standard schema that GPT and other Large Language Models are accustomed to parsing. This standardized schema and its cameo its played in GPT’s training data gives us an effective way to communicate rigorous types back and forth in English and code.

Pydantic recently rebuilt its entire foundation in Rust and, as a result, boasts a 50x speed improvement. It’s a treat to work with, and it’s the version of choice for developers starting new projects. We’re excited to support developers working with either Pydantic V1 or V2!

The feedback from Marvin users has been loud and clear: it’s the most ergonomic framework for working with Large Language Models. It’s a framework that feels familiar: if you know Python, you know Marvin.

Upgrade today (pip install marvin -U), check out the docs, and give the repo a star!

Today we’re excited to release Marvin 1.4.

This release comes with powerful, opinionated, and flexible framework for building (semi-)autonomous workflows - something we’ve thought a lot about at Prefect. Today we’re adding:

• Conservational State for managing multi-turn conversations and workflows.

• Long-running chaining for autonomous workflows.

• Ejectable prompts - separating serialization and logic.

If you’ve built an LLM-based system, you know first-hand that fully autonomous workflows, while powerful, are brittle and hard to trust. LLMs, like a sharp college intern, can go off rails if you don’t have touch points to check in. Once they’ve proven they’re capable of handling a task on their own, you’re able to set and forget.

Conservational State and Autonomous Workflows.
We’ve built a framework for writing semi- and fully-autonomous agents with tools. Here’s what writing an agent feels like in Marvin.

This lets you introspect and debug agents with the simplicity you would for writing pure Python. No black box, and no PhD required.

Alternatively, when you want to turn full control over to an LLM for autonomous execution, you can simply chain to implement this while-loop behavior.

Ejectable prompts.

One of the things developers love about Marvin is our emphasis on the engineering in prompt engineering. Marvin developers write prompts with python functions, dataclasses, enums. We take your code, written in python, and translate it to English - so an LLM can autocomplete to a response which we parse for you.

As developers explore open source models, we saw a greater need to separate serialization (code to english) and execution (english to LLM). We’ve worked to make each component of Marvin (AI Functions, AI Models, AI Classifiers) expose their raw English prompt to use with open source models.

The feedback from Marvin users has been loud and clear: it’s the most ergonomic framework for working with Large Language Models. It’s a framework that feels familiar: if you know Python, you know Marvin.

Upgrade today (pip install marvin -U), check out the docs, and give the repo a star!

Can you see the forest for the trees?

You might need to squint.

A few weeks ago, a viral post on Reddit caught my attention. It was about embedding text in plain sight within AI-generated images using Stable Diffusion and ControlNet, creating an ingenious blend of art and technology. Here’s one example from that post:

If you’re like me, you’re probably a bit confused right now: where’s the text? I couldn’t see it myself until I looked at a thumbnail preview of the same image:

It’s hard to believe those are the same picture! Try resizing (or squinting) to prove it to yourself.

(Mobile readers: you’re probably REALLY confused right now because you see the text in both images, which are naturally small on your device. Zoom in or come back from a laptop to see the full effect.)

Essentially, the text is hidden as a low-frequency component of the image. Zooming out or even squinting can hide the high-frequency details that prevent your brain from resolving it.

I thought this was so clever and fascinating and wanted to try myself. I only found one tutorial online (unfortunately, it has since disappeared), and it relied on the popular Automatic1111 UI. I wanted a way to do generate these images programmatically, so I began writing a script to do so. This gave me a greater opportunity to learn how they worked, as well as to exercise more control over the output.

I tweeted some of my early results and got such an warm response that I decided to publish my code and write this piece.

What’s going on here?

Stable Diffusion (SD) in an open-source text-to-image model, meaning it translates a description of a scene into a picture of that scene. This technology is incredibly powerful, but it can be difficult to control. For example, I can use it to generate infinite variations of “a house,” but getting it to produce one that looks the same as the house I grew up in would defy even the most precise natural language prompt.

ControlNet (CN) is a technique that can precisely guide the output of a SD model. There are a variety of ControlNets optimized for different types of images, but broadly speaking it would let me show the SD model a picture of my house and have it generate an image that appeared to be of the same house, but under different lighting, weather, or completely fanciful conditions.

Recently, a CN was developed that excelled at producing natural-looking images that are actually valid QR codes, like this one (try scanning it):

It works by using the black-and-white QR code pattern to inform areas of high-and-low contrast in the final image, with SD filling in the details in an aesthetic way.

It turns out we can use this same technique to embed text instead of QR codes. The effect can be extremely subtle or painfully obvious, depending on the settings used. The most fascinating thing to me is how the text completely disappears when the image is large and reappears when it’s small (or you stand back, or squint) and only focus on the low-frequency components.

Here are a few examples I generated:

Try it yourself

When I started generating these images, I quickly ran into a problem. My M1 MBP takes minutes to generate a single image, which is just too slow. I’m a fan of Modal for rapid access to serverless compute, so I used Modal to offload computation to a remote A10G GPU, which brought generation time down to seconds.

I’ve published the script that I used to generate many of these images. Its only requirement is a Modal account.

The process isn't without its challenges. Fine-tuning the visibility of the text in the images is intricate and depends on various factors, including guidance scales, image size, and the prompt. Some scenes are more amenable to this technique than others, and naturally high-contrast scenes lend themselves particularly well.

There's a lot to be explored in the interplay of text and images in this way. The melding of high and low-frequency components to create, hide, and reveal meaning opens up exciting possibilities, from art to cryptography to novel forms of communication. There is a frontier here that's ripe for exploration, and I’m thrilled to get to dive into it! I hope you’ll give it a shot, too.

Marvin is an AI engineering framework for building natural language interfaces that are reliable, scalable, and easy to trust.

With the 1.0 release last week, we focused on delivering four key components that you can adopt anywhere in your software:

🧩 AI Models for structuring text into type-safe schemas

🏷️ AI Classifiers for bulletproof classification and routing

🪄 AI Functions for complex business logic and transformations

🤝 AI Applications for interactive use and persistent state

Marvin 1.0 was focused on getting the developer experience just right by reusing familiar interfaces like Pydantic models, enums, and functions. Today, we’re making some substantial enhancements to that core functionality in Marvin 1.1.

Anthropic & Azure OpenAI Service support

In addition to OpenAI, Marvin 1.1 adds support for both Anthropic and the Azure OpenAI Service through a new providers interface. To change your model, set your llm_model (or the MARVIN_LLM_MODEL environment variable) to a compatible string, like this:

Adding new providers is straightforward, as long as they are compatible with the OpenAI functions API (or can be taught to be compatible with it!). You can see the full list of supported models here.

Please note that at this time, AI Classifiers only support OpenAI-compatible APIs because they take advantage of a special tokenization feature to deliver results extremely quickly.

Mapping: process batches in parallel

AI Models, AI Classifiers, and AI Functions all have a new .map() method that applies their logic to multiple inputs. Mapping operates concurrently, which means that processing happens (almost) in parallel and your results will be available as soon as the slowest call completes.

Guide behavior with instructions

AI Models, AI Classifiers, and AI Functions all have a new instructions parameter that can be used to steer their behavior. This is helpful for controlling parsing or edge cases without redefining your entire component. Instructions can be provided when you decorate a component or on a per-call basis:

Note that for AI Models, the parameter is called instructions_ to avoid conflicts with models that have an actual “instructions” field!

Streaming and more

All Marvin LLMs now support streaming outputs, which is most useful when working with AI Applications. In addition, 1.1 is full of improvements, bug fixes, and prompt tweaks designed to improve the overall developer experience.

Upgrade today (pip install marvin -U), check out the docs, and give the repo a star!

Today we’re releasing Marvin 1.0! This major release is the result of incredible feedback and opportunities to work closely with our developer community over the past few months. We’re so grateful for their time, attention, and contributions, and this release is truly a community effort.

Check out the docs at askmarvin.ai, give Marvin a star on GitHub, or take 1.0 for a spin by upgrading now: pip install marvin -U

Hello, Marvin!

Marvin is a lightweight framework for building AI-powered software that’s reliable, scalable, and easy to trust. It is designed primarily for AI engineers: users who are deploying cutting-edge AI to build powerful new features and applications. Since our initial release, thousands of developers, engineers, and founders have adopted Marvin to integrate and build with Large Language Models (LLMs). We’ve been humbled by the warm reception.

Sometimes the most challenging part of working with generative AI is remembering that it's not magic; it's software. It's new, it's nondeterministic, and it's incredibly powerful, but it's still software: parameterized API calls that can trigger dependent actions (and just might talk like a pirate). Marvin's goal is to bring the best practices of building dependable, observable software to the frontier of generative AI. As the team behind Prefect, which does something very similar for data engineers, we've poured years of open-source developer tool experience (and a few hard-won lessons!) into Marvin's design.

Developer Experience

Above all else, Marvin is focused on a rock-solid developer experience. It's ergonomic and opinionated at every layer, but also incrementally adoptable so you can use it as much or as little as you like. It’s a Swiss Army Knife, not a kitchen sink. It’s familiar. It feels like the library you’d write if you had the time: simple, accessible, portable LLM abstractions that you can quickly deploy in your application, whether you’re doing straightforward NLP or building a full-featured autonomous agent.

Marvin prioritizes a developer experience focused on speed and reliability. It's built with type-safety and observability as first-class citizens. Its abstractions are Pythonic, simple, and self-documenting. These core primitives let us build surprisingly complex agentic software without sacrificing control:

🧩 AI Models for structuring text into type-safe schemas

🏷️ AI Classifiers for bulletproof classification and routing

🪄 AI Functions for complex business logic and transformations

🤝 AI Applications for interactive use and persistent state

If our mission sounds exciting to you, please join our community!

Ambient AI

One of our favorite reactions to Marvin’s initial release came from Dr. Jim Fan at NVIDIA:

This idea really resonated with us, and Marvin is the first step on a journey to deliver “Ambient AI:” omnipresent but unobtrusive autonomous routines that act as persistent translators for noisy, real-world data. Ambient AI makes unstructured data universally accessible to traditional software, allowing the entire software stack to embrace AI technology without interrupting the development workflow.

This ideal has the potential to unlock value analogous to the shift from analog to digital music, which revolutionized accessibility and search. Suddenly, vast song libraries were at our fingertips, ready to be explored with a few keystrokes no matter where we were. Ambient AIs can do the same for real-world data and interactions by making them suddenly available to traditional software, without compromise. That is how AI technology can become as pervasive and personalized as music is today.

To play its part, Marvin introduces a set of AI components that integrate seamlessly with the Software 1.0 stack. This lets developers focus on core functionality and delivering value to users, not turning knobs on an LLM. Our goal is to create a future where all software can work in concert with ambient AI receptors. This synergy brings simplicity and stability to AI engineering, allowing every application to effortlessly harness AI technologies and making AI a natural and integrated part of the software landscape.

AI Components

Marvin’s core AI abstractions are all packaged as familiar, Pythonic interfaces. These form the building blocks of the Ambient AI framework.

🧩 AI Models

AI Models are a drop-in replacement for Pydantic’s BaseModel that bring structure to unstructured data. They can be instantiated from any string, allowing them to parse unstructured text into typesafe schemas. AI models are perfect for typed entity extraction, generating synthetic data, and standardization.

🏷️ AI Classifiers

AI Classifiers let you build a multi-label classifier with no code and no training data. They are fast and bulletproof because they use a clever logit bias trick to get results with just a single token.

🪄 AI Functions

AI functions look just like “normal” functions, but don’t have any source code. Instead, they use AI to predict outputs based on their inputs and a description of the function. They’re perfect for complex NLP tasks like entity extraction and qualitative tasks like sentiment analysis. They can also be used for generating synthetic data according to a set of constraints.

🤝 AI Applications

AI Applications are optimized for interactive use cases, where users invoke the application multiple times to achieve a goal (or it loops autonomously!). In addition to tracking a history of user inputs, AI Applications maintain persistent state and are capable of making high-level plans, which keeps them reliably on track. This means that while they can implement “classic” LLM demos like chatbots, coding assistants, and tool-using agents, they can also implement full-fledged applications for which chat (or unstructured text) might be a surprising interface… at least not without a lot of extra coding! This includes role-playing games, technical writing, or even this to-do app:

Marvin’s high-level abstractions are powered by a complete AI platform that users may choose to delve into, if it’s helpful. At the lowest level, Marvin introduces a drop-in replacement for the openai library that adds useful features like portable configuration, logging, and other useful utilities.

In addition, Marvin has a powerful prompt DSL inspired heavily by (and developed in collaboration with) our friend Jason Liu’s OpenAI Function Call library. Rather than attempting to munge string templates together, Marvin embraces prompts-as-code and allows users to customize LLM interactions in a variety of ways.

What makes Marvin different?

There's no shortage of tools and libraries out there for integrating LLMs into your software. So what makes Marvin different? In addition to a relentless focus on incrementally-adoptable, familiar abstractions, Marvin embraces five pillars:

User-Centric Design

We built Marvin with you in mind. It's not just about what it does, but how it does it. Marvin is designed to be as user-friendly as possible, with a focus on an easy, intuitive experience. Whether you're a coding expert or just starting, Marvin works for you.

Flexibility

Marvin is built to adapt to your needs, not the other way around. Embracing Prefect’s core principle of incremental adoption, you can use as much or as little of Marvin as you need. Need a full suite of LLM integration tools? We’ve got you covered. Only need a single component? Marvin can happily do that, too.

Community Driven

Marvin isn't just a tool, it's a community. We value feedback and collaboration from users like you, and Marvin reflects hard-won best practices from folks on the cutting edge of production AI engineering. We're always learning, iterating, and improving based on what the community learns.

Speed

We're serious about speed. We believe that getting started should be quick and easy, and every Marvin-specific keystroke is a penalty you pay for our design. That's why with Marvin, you can get up and running in no time and with almost no dependencies.

Open-Source

Marvin is fully open-source, which means it's not only free to use, but you're also free to modify and adapt as you see fit. The Prefect team has years of open-source experience and is fully committed to supporting Marvin as an open-source product. We believe in the power of collective intelligence, and we're excited to see what you can create.

The road to 1.0

Marvin was born in 2022 as an internal tool at Prefect, powering our early exploration of LLM technologies. In March 2023, we released the first open-source version of Marvin as a “batteries-included library” for building AI-powered software. A few thousand users later, we discovered that users don’t actually want a library that prepackages everything. What they want are clear and effective tools for deploying AI in a manageable way. With that in mind, it’s no surprise that AI Models and AI Functions have always been Marvin’s most popular features.

Marvin 1.0 leans hard into this discovery by ensuring that every abstraction is rock solid and independent. That’s how we hope to pursue the goal of truly ambient AI. In addition to a full rewrite, 1.0 introduces two new major components, AI Classifiers and AI Applications, that we expect to become bedrock for a new class of AI-enabled features. However, please note that some 0.x features didn’t meet the standard for inclusion in 1.0, so for some users this may be a breaking release.

If Marvin has a secret weapon, it’s that we view its primary job as making AI technology, no matter how complex, available through interfaces that feel familiar to our users. Every Marvin component strives to be future-proof and incredibly useful. As a team, we are committed to making sure that Marvin’s DX is solid and we take complete responsibility for continuously adapting it to the ever-changing AI landscape.

We hope you enjoy using Marvin as much as we do!

We still have a number of major features to announce, so please watch this space and @AskMarvinAI for more.

Happy engineering!

It’s like we finish each other’s sandwiches.

It's been a hell of a year for autocomplete. It feels like just yesterday we brought our first Language Model home from the hospital. On a diet of the entire internet it’s grown Large and, like every parent hopes, omniscient. It’s humbling to see its attention span as ravaged as mine has been by binging on Reddit and the like.

‘Omniscient autocomplete’ is less an aspersion and more a reminder of what it’s doing. And we should remind ourselves what it’s actually doing. Given a series of word snippets (tokens) strung together, Large Language Models like ChatGPT predict what token is most likely to come next. It’s good at guessing because it’s well read and we’ve had humans give it treats when it’s done a good job1.

Omniscient autocomplete is great at solving autocomplete problems. It’s filled with a discrete set of facts and opinions it’s seen on the internet, and it’s seen enough examples of deductive reasoning to be able to infer things it hasn’t observed directly2.

What's not an autocomplete task?

There are a lot of autocomplete problems in the world. Writing this post is one of them. Answering ’s Slack why this post is late is one of them. LLMs are water to those who had the misfortune of building a chatbot or summarizing documents in the last decade. I think autocomplete problems are valuable, but much like I responded to Jeremiah’s slack “it’s just not my focus at the moment”.

In Math you learn early that the easiest3 way to solve a problem is simply recast it into a problem that’s already been solved.

• Does my differential equation have a solution? It sure does if you recast it as fixed point problem and smack it with the Banach Fixed Point Theorem4.

• Can we design a matching program for medical residencies so that everyone is incentivized to tell the truth? It sure does if you recast it as a private information game and crack it over the head with Gale-Shapley.

Jackie Hadamard (can I call you Jackie?) said “The shortest path between two truths in the real domain passes through the complex domain.” If autocomplete problems are solved, this means that any problem that can be reliably translated into and back from an autocomplete problem is solved (where my category theorists at?).

We’ve seen glimmers of this.

Research has shown LLMs to reliably do zero-shot5 classification. To build a spam classifier, we used to manufacture features from thousands of emails, manually labeling them, and artisanally crafting a bespoke model to learn the relationship between our features and labels. Now, we show OpenAI the text of an email and simply ask it: is this spam or not?

Research has shown LLMs to reliably do zero-shot entity extraction. To build a model to do entity extraction, we used to have to pray to SpaCy or Gensim that classical linguistics could help us identify proper nouns in our document. If you were feeling particularly adventurous, you could try to get buy-in to use conditional random fields before throwing it in the backlog. Now, we show OpenAI the text of a resume and simply ask it: “where did this person go to school and what degrees did they get there?”.

A corollary of these two emergent6 behaviors is that LLMs can generate their own tasks and choose the right tools to solve them. To do this five years ago you’d have to wait four years. You simply describe what tools do and how to use them, give your LLM an amorphous task, and ask “oh, great OpenAI, what sayeth you the best tool to use to solve my lowly task”. We can unfurl workflows from a simple objective and dynamically generate our own tasks.

This process of translating the task we actually want to be completed (classification, extraction, tool choices) into an autocomplete is called prompt engineering. In a sense, prompt engineering is a transpiler7 (or functor): we take an instruction set in one language and translate into an instruction set native to our LLM, which we’re treating as a runtime.

Pedantic to semantic: leave me out of it.

Transpilers should be neither seen or heard. After all, one of their promises is to let thoughtless single-language engineers like me approximate a mediocre engineer in languages I don’t know. They exist ambiently, and I love that feeling. I love that experience as a developer.

In LLMs this means creating a semantic layer between code and LLMs: something that translates the pedantry of software into the … semantry (?) of LLMs and back again. I deeply believe that working with LLMs should be the same, and this is what motivated my choice to join the Marvin team at Prefect.

Today OpenAI announced support for functions. When you ask it a question, you can now supply and describe a set of functions to GPT 3.5 and GPT 4. It will decide if and which functions are prudent to call to confidently answer your question. If it decides that it needs to use one of your supplied functions, it’ll even do you the favor of formatting the input to that function. OpenAI choice to format inputs seems at first to be going the wrong way, after all, for the last year developers have focused on getting structured outputs from LLMs. Turns out, it’s all the same.

Consider the usual fare of constraining a large language model to do your bidding:

1. Here’s this email I got from {email_text}.

2. Extract the following information: num_puns: int, worst_puns: list[str], best_pun: Optional[str] = None

3. Output as JSON.

This works okay, at first. But as the number of requirements increases or the depth or ‘nestedness’ of the schema grows, this starts to fail and fail often.

But we can recast this problem as a tool choice problem. OpenAI now lets me pass tools to it in service of a question. If it chooses that tool, it will format the data that needs to be passed to that function. So all we need to do is to

1. Create a tool whose “inputs” are num_puns, worst_puns, and best_pun

2. Trick OpenAI into using that tool.

Luckily, OpenAI exposes a parameter where you can force it to use a tool. So, really it’s as simple as telling OpenAI “I want to extract the following information from {email_text}” and passing it:

extract_puns(num_punts:int, worst_puns:list[str], best_pun: Optional[str])

What do you get back? Beautiful, bulletproof, formatted, boring, typesafe, atomic-unit-of-trust JSON.

We shipped this in our latest version of Marvin today, but it’s all under the hood - as it should be. It’s open source, free to use, and all our code is publicly available. You’ll pay your tithing to wherever you buy your inference, but it’s now 15x cheaper to extract structured data from unstructured text and documents.

Generative AI has turned us all from authors into editors. It's enabled machines we can program in natural language, and overnight has reduced the marginal effort of building to zero. We’re all developers now, for better or for worse.

The off-key trio of “woah!”, “no.”, and “apropos?” is belted across social media as our cast of newfound developers, career engineers, and business leaders make sense of our new productivity gains.

The pace of AI is unrelenting. Researchers push the energy density of our engines. Hackers push the boundaries of where we can travel. Craftspeople remind us why some roads are off limits. Every week feels like a year.

But what should be a symphony is instead a cacophony. We're starting FactsMachine to fix that. If you find yourself wishing Generative AI felt more like Moving Type and less like Moving Hype, you might enjoy subscribing. We'll share what we learn as we pair code with developers, help engineers move LLMs to production, and business leaders grok their AI strategy.

We've spent our careers building tools for hackers, craftspeople, and business leaders alike to leverage data and machine learning in their software and business. Thousands of developers use Prefect as their data engineering backend and Marvin as their AI engineering backend.

We're excited to share those lessons, every week, at FactsMachine.