Thought Eddies - Logs

2026-05-29

Fri, 29 May 2026 15:29:24 GMT

Still my favorite productivity software

Thanks for reading via RSS!

2026-05-26

Tue, 26 May 2026 16:09:18 GMT

Experimenting with chainlink today and surprised how easy it makes it to ping pong review, testing and implementation between agents. The sqlite db is effectively the agent to agent communication conduit.

Thanks for reading via RSS!

2026-05-23

Sat, 23 May 2026 15:13:35 GMT

Your /dev directory looks like a small civilization of agents, apps, old experiments, forks, and active services

Thanks for reading via RSS!

2026-05-16

Sat, 16 May 2026 23:30:34 GMT

I keep hearing and reading the phrase “load bearing” and even though I suspect it isn’t always, I can only read it as LLM generated content

Thanks for reading via RSS!

2026-05-11

Mon, 11 May 2026 20:38:08 GMT

For those with access to Mythos, it does not appear to be some kind of step function change in vulnerability detection. Rather, it seems to show code agents are good at finding vulnerabilities and that if you don’t use them to do that someone else will. https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-vulnerability/

Thanks for reading via RSS!

2026-05-09

Sun, 10 May 2026 01:20:31 GMT

There’s a lot of chatter about models getting worse after launch, during peak hours, on subscription plans vs. pay-per-use APIs.

Thanks for reading via RSS!

2026-05-07

Fri, 08 May 2026 00:48:06 GMT

I’ve started using Codex more consistently. It took me entirely too long to locate the plan usage page: https://chatgpt.com/codex/cloud/settings/analytics, roughly equivalent to Claude Code’s https://claude.ai/settings/usage page.

Thanks for reading via RSS!

2026-05-03

Mon, 04 May 2026 01:00:33 GMT

It will be interesting to look back in retrospect if April 2026 was peak LLM hype. During and since the month, subsidies for inference have started to go away meaningfully, and people finally seem to be internalizing that you can only trust LLMs for what you choose to verify, among other shortcomings and drawbacks. The software engineer is not dead.

Thanks for reading via RSS!

2026-05-02

Sat, 02 May 2026 23:28:58 GMT

No one can explain to me what reinforcement learning actually is

Thanks for reading via RSS!

2026-05-01

Sat, 02 May 2026 00:14:32 GMT

Switched to running a mix of claude and codex for the first time for day-to-day dev.

Thanks for reading via RSS!

2026-04-30

Thu, 30 Apr 2026 20:51:33 GMT

Opus 4.7’s tendency to use acronyms drives me crazy, especially when I have no idea what it’s referring to, even though I’m its collaborator.

Thanks for reading via RSS!

2026-04-27

Tue, 28 Apr 2026 01:10:09 GMT

TIL, “ennui”: not just “nothing to do” but “nothing seems worth doing”

Thanks for reading via RSS!

2026-04-25

Sat, 25 Apr 2026 13:52:12 GMT

Why you need stateful agents. You can be ambiguous, concise, and still be successful because the agent has the surrounding context.

Thanks for reading via RSS!

2026-04-23

Fri, 24 Apr 2026 01:09:29 GMT

I really have enjoyed writing about building software in the past but lately I’ve been struggling to find a foothold. The prompts to an agent are now most of the software. There is trial and error along the way. I often persist learnings to skills to make processes repeatable for agents. But the errors along the way are often more interesting for people learning to use agents more effectively. So I don’t really know what this writing should look like.

Thanks for reading via RSS!

2026-04-21

Wed, 22 Apr 2026 00:09:21 GMT

I’ve been working on a chat app recently for collaboration with people and agents. I was heavily relying on “channels” for specific projects. However, a pattern that was emerging often was more exploratory, unstructured conversations, and often several of these at once. I was starting to juggle these between channels named for projects just because they had to go somewhere. Multiple “in progress” messages in a channel was too confusing. So I landed on “ephemeral” channels which are one-off disposable conversations. It feels like a reasonable compromise for my use case.

Thanks for reading via RSS!

2026-04-20

Mon, 20 Apr 2026 22:38:55 GMT

There’s a huge amount of cold outreach that appears (to my eye) to be LLM-generated. What’s more interesting that these inbound messages seem completely decoupled from the human(?) who gets involved if you respond to one of these cold leads. The outreach

Hi Dan, What prompted me to reach out is we’re looking for an AI Technologist / Engineer …

I reply 20 minutes later

Hi, I currently run a consultancy where I do work like this …

They respond

Hey Dan, It has been a while since we connected last … If you are interested in [a full time position] …

Has it? As far as I can tell we “spoke” this morning? Did you read my rely even a little? Am I speaking to a person yet? Maybe it’s an uncharitable interpretation of what is happening here but shouldn’t the human at least claim ownership of the original message even if they never wrote it or read it? It’s sent with their name on it. To be clear we’ve never “connected” prior to this email. The original email is the first contact ever. Is this another LLM responding without the first exchange? Yikes.

Seems the only way this ends is with agent representation on both sides.

Thanks for reading via RSS!

2026-04-18

Sat, 18 Apr 2026 13:23:11 GMT

I spent yesterday working on a personal project to build some software with my personal agent, struggling mightily when I didn’t expect to. Opus 4.7 was recently released. And in the back of my mind the whole time, I thought to switch my agent over to it, but mostly forgot. Progress on the project was an absolute struggle. And today, since switching over to 4.7, everything has been going great. This gets brought up pretty much every time Anthropic releases a new model, but GPUs have to come from somewhere. And it’s hard to imagine they didn’t do something like quantize Opus 4.6 to unlock capacity for 4.7. Obviously, or maybe not so obviously, I don’t yet have a benchmark that’s repeatable in such a way that I can prove 4.6 has regressed, but the vibes say that it has. Now I’m on 4.7, but I’m reasonably happy for now. Another model release, another conspiracy.

Thanks for reading via RSS!

2026-04-17

Fri, 17 Apr 2026 13:02:57 GMT

Coming from a software engineering background, but mostly spending time teaching non-technical domain experts how to code has been an interesting challenge because my first impulse or inclination is to teach and explain through software engineering principles. But in practice, if you’re not a software engineer, most of the time you don’t actually care about the specifics of the software, at least not initially. I find that first I have to sort of play out, then catch my impulse to explain through the software found, you know, through the foundations of software engineering and with software engineering terminology. And then from there, back my way into what the actual goal and intent is. An example of this is a conversation that I had with somebody who had created a large complex web app for displaying data in lots of different types of ways and across many different facets. We’ve been discussing how to make the web app more maintainable and extendable as it grows in size, which whether you’re a human or an agent working on a code base, as the code base grows in size, more time is dedicated to research and maintenance of the code base and building of understanding as opposed to early on when there’s a lot of greenfield and fast development. As we were having these conversations, I was thinking about how I might introduce the concept of refactoring and how that plays into making the code base more maintainable, talking about design patterns and the shape and structure of the code. But after thinking about it for a while, what I realized in practice is refactoring and the structure of code bases in general are just the, I guess, commitment to various trade-offs, right? Certain things become easy when code is organized in certain ways and certain things are harder. There’s no perfect structure. There are only trade-offs given what you’re optimizing for. And with that understanding, I think I found a more sort of salient and practical argument or line of thinking to trace for somebody who is not familiar with software, which is that you need to think about what you are optimizing for, what things you want to be flexible versus what things you do not expect to change. You, of course, can’t optimize for everything, but you can think through the types of things you may want to do in the future and then instruct an agent to organize your code base, optimizing for ease of those types of changes. That grounds the instructions to the agent in the domain you’re working in based on the work that you expect to do, but doesn’t necessarily require deep engagement directly with the software. And so I think that this will be an interesting on-ramp to try.

Thanks for reading via RSS!

2026-04-14

Tue, 14 Apr 2026 18:58:08 GMT

The Cowork and Claude Code UIs are friendly on ramps for agent tools, particularly for nontechnical roles, but I do wonder whether hiding the more technical aspects of the system makes users less likely to make their own software. On one hand, it’s probably inevitable (still) that most people won’t make their own software, but for those who might, I’m torn if the friendly UI is the better introduction. The labs certainly want you to let them make all the software for you but being able to control and modify your own tools changes your relationship with them.

Thanks for reading via RSS!

2026-04-13

Mon, 13 Apr 2026 23:04:02 GMT

Skills seem to be a huge leverage point working with agents but they remain fuzzy in how they should be constructed. I like to build my skill using the agent at the end of a messy session where I successfully accomplished a goal. Then I have the agent extract the working steps into a skill that I can easily invoke again.

Thanks for reading via RSS!

2026-04-10

Fri, 10 Apr 2026 18:34:25 GMT

One thing that has surprised me a bit recently is how much play the Claude Mythos stuff has gotten in media, especially with the example the New York Times gave of somebody’s kids accidentally bringing down the power grid. In Anthropic’s examples talking about the zero days that these models have identified, it cost $20,000 worth of inference to find these issues. Also it’s not clear to me if this is a step function improvement on GPT-5.4 which researchers have already been using to find these types of bugs in software for months. As Simon Willison noted,

GPT-5.4 already has a strong reputation for finding security vulnerabilities.

As usual, narratives seem quite out of sync.

Thanks for reading via RSS!

2026-03-13

Sat, 14 Mar 2026 03:26:44 GMT

I learned about Proof a few days ago. Today, I had my stateful agent fork and install it to its machine, exposing to the public internet through a Cloudflare tunnel, behind auth. I’ve been playing around with it. I can’t decide whether it’s innovative or Google Docs.

Thanks for reading via RSS!

2026-03-01

Mon, 02 Mar 2026 00:43:19 GMT

I revisited Tomo this evening after not touching the codebase for a while. I wouldn’t say I use that app too much but I wanted to experiment with agent teams.

use a team to refactor this old, vibe coded project to make it a swift project that uses best practices and consistent standards for easy, continued feature development https://code.claude.com/docs/en/agent-teams

And that was basically it. Claude Code parallelized the work with four agents and I supervised them to completion. It refactored such that no file was larger than 300 lines. When I built the project in Xcode, everything worked as expected.

As a bonus, I also had Claude Code stitch together some build automation for my release process for Sparkle updates. Opus 4.6 rips.

Thanks for reading via RSS!

2026-02-22

Sun, 22 Feb 2026 16:47:25 GMT

Tried the Codex desktop app and it reminds we a bit of the Goose desktop app.

Thanks for reading via RSS!

2026-02-09

Mon, 09 Feb 2026 22:12:07 GMT

Had an agent doing work today but ran into an issue where Github was down. Using a scheduled task skill we built, I had it retry the work again later, but schedule another task to retry again if the first one failed. I came back later and the work was done.

Thanks for reading via RSS!

2026-01-27

Wed, 28 Jan 2026 02:32:51 GMT

Claude Code translated my personal recipe app into Spanish, including all the database content in 15 minutes while I was only vaguely paying attention.

Thanks for reading via RSS!

2026-01-24

Sun, 25 Jan 2026 04:49:14 GMT

I spent years in iterative improvement. I find myself now going directly to the version I actually wanted. Not because I have more discipline, but because the economics changed. The right way is not much harder than the wrong way when someone else writes the code.

https://estsauver.com/blog/claude-code-workflow

Thanks for reading via RSS!

2026-01-21

Thu, 22 Jan 2026 03:55:42 GMT

Moved this blog from Vercel to Cloudflare Pages, where I’ve been hosting more of my projects lately. Ran into a bit of trouble with the different between Workers and Pages but at least the DNS switchover was easy.

Thanks for reading via RSS!

2026-01-19

Mon, 19 Jan 2026 17:11:17 GMT

I just incinerated $10 running GLM 4.7 on Cerebras to try and Ralph a clone of this game I used to play. Took about 15 minutes for the balance to go to $0. Result, not awesome. Maybe okay.

I Claude Coded the game into a playable state with a few more prompts (it was still broken once my balance reached zero).

Cerebras was fast but costly given the token throughput.

Takeaways

This approach “works” in that it builds in the direction of the spec for extended periods of time. A browser game might not have been GLM’s strength — the model measures well on coding benchmarks, so some kind of backend task might have been more reasonable to try. It did write a lot of compiling TypeScript though and I didn’t give it much to work with in terms of design or aesthetics, so I’ll take the blame there as well.

It seems like building an intuition for managing long running loops and getting good results requires building and experimenting with a lot of long running loops. No big surprise there. And it’s not a cheap thing to learn.

Code

#!/bin/bash

# Simple Ralph Loop
# Continuously passes plan.md to opencode with the prompt to work the plan
# Stops when the agent returns <complete/>

PLAN_FILE="${1:-plan.md}"
ITERATION=0

if [ ! -f "$PLAN_FILE" ]; then
    echo "Error: $PLAN_FILE not found"
    exit 1
fi

echo "Starting Ralph loop..."
echo "Will continue until agent returns <complete/>"
echo "---"

while true; do
    ITERATION=$((ITERATION + 1))
    echo ""
    echo "=== ITERATION $ITERATION ==="
    echo ""

    # Build the prompt: reference the plan file by name
    PROMPT="Work the plan in $PLAN_FILE and test until complete. Perform one task at a time. Update the plan file with the completed tasks as you work. When _all_ tasks in $PLAN_FILE are done and tests pass, output <complete/> to signal completion."

    # Run opencode with the prompt and capture output
    OUTPUT=$(echo "$PROMPT" | opencode run --model cerebras/zai-glm-4.7)

    echo "$OUTPUT"

    # Check if output contains <complete/>
    if echo "$OUTPUT" | grep -q "<complete/>"; then
        echo ""
        echo "=== RALPH COMPLETE ==="
        echo "Finished after $ITERATION iteration(s)"
        exit 0
    fi

    echo ""
    echo "--- Iteration $ITERATION complete, starting fresh context ---"
done

Thanks for reading via RSS!

2026-01-18

Mon, 19 Jan 2026 00:41:41 GMT

Running Claude Code on qwen3-coder:30b via Ollama does not work.

Thanks for reading via RSS!

2026-01-17

Sat, 17 Jan 2026 16:16:19 GMT

Internal adoption charts have been vertical: ~30% of all pull requests merged to our frontend and backend repos are written by Inspect. It only took a couple months for us to reach this level of usage, and it continues to grow. We didn’t force anyone to use Inspect over their own tools. We built to people’s needs, created virality loops through letting it work in public spaces, and let the product do the talking.

https://builders.ramp.com/post/why-we-built-our-background-agent

Thanks for reading via RSS!

2026-01-15

Fri, 16 Jan 2026 03:13:47 GMT

Experimenting with a personal agent using the file system, SQLite, skills tools via MCP. So many ways to make things work these days.

I’m still exploring but once you know what you want an agent to do, I suspect you’ll need robust tests to validate it works and that it continues to work as you evolve it.

Thanks for reading via RSS!

2026-01-12

Tue, 13 Jan 2026 02:23:57 GMT

Tried to make a new Github account today and the CAPTCHA was absurd.

Thanks for reading via RSS!

2026-01-06

Tue, 06 Jan 2026 16:34:47 GMT

Still enjoying working on Tarn.

I played around with adding some Hemingway-like features. I like them to put myself more into editor mode after getting thoughts written down and taking a few revision passes.

It’s so hard to describe to somebody like what’s different if you just read the summary in the spreadsheet versus you read the whole LinkedIn profile.

It’s one dimensional or not multi-dimensional.

From Dialectic: 35: Brie Wolfson - Loving Attention & Ease in Craft, Jan 6, 2026 https://podcasts.apple.com/us/podcast/dialectic/id1780282402?i=1000743936367&r=626

Thanks for reading via RSS!

2026-01-05

Mon, 05 Jan 2026 15:44:09 GMT

Paul Simon on “what makes you stuck?”

Everywhere I went lead me where I didn’t want to be, so I was stuck.

https://youtu.be/qFt0cP-klQI?si=XNdG7SkBIeNqCJLN&t=445

Thanks for reading via RSS!

2026-01-03

Sat, 03 Jan 2026 13:40:58 GMT

Agents are increasingly being trusted to do quantities of work that cannot be verified at code-level detail. Finding verification criteria for the products of work done like this will be critical for having any amount of trust in it.

I still run Tarn out of Xcode because its codebase is very much in flux. Today, I finally patched an issue where unsaved file contents got cleared between app restarts. I lost the first draft of this log to that issue. Slowly but surely, this app is feeling like the best place for me to write stuff.

A few days ago, I used Nano Banana Pro and Apple’s Icon Composer to create an app icon for it. The aim is to be at a point where I have more of a stable release in my app folder soon.

Thanks for reading via RSS!

2026-01-02

Fri, 02 Jan 2026 19:43:57 GMT

I’ve been playing around with using Voice Activity Detection to record multiple separate audio channels, then stitching those together into a transcript with Whisper to do speech to text.

Agents make ideas so malleable. A well articulated idea, or implementation of that idea is in invaluable yet almost completely fungible. If someone explains an idea in detail, an agent can fluently apply it in any context you’d like. Producing an artifact with this level of clarity is difficult, but once produced, it can be absorbed and applied broadly nearly instantly by anyone who learns of its existence.

I’ve been trying to reproduce a reference implementation of Strix. In doing so, I am steering Claude Code to read the blog post and then setting up a personal version in my own environment. That is works continues to floor me.

Tim’s blog post is a sort of genetic code for Strix and when given access to this code, an agent can create another approximate digital incarnation of Strix, based on how Tim has written about it. In the same way ideas could spread rapidly with the invention of the internet, now fully embodied concepts of functioning software can permeate throughout the internet if they are loosely described and defined in words. The more detailed, the more faithful the implementation will be to the original.

Thanks for reading via RSS!

2026-01-01

Fri, 02 Jan 2026 04:06:45 GMT

I’m working through my second implementation of an idea that has been bouncing around my head for a while. The general idea is to be able to transcribe a conversation in real time and have an agent read the transcription, turn it into a spec, and then build what is being discussed. The ongoing conversation will steer the agent’s work in real time.

I’m not sure if the agent will be fast enough for this to work well but we’ll see how it goes.

Thanks for reading via RSS!

2025-12-31

Wed, 31 Dec 2025 14:25:38 GMT

I’m not sure if they originated this pattern but I like how Claude Code introduces new features with tips.

✻ Whirring… (esc to interrupt · thinking)
  ⎿ Tip: Start with 'btw' to ask a quick side question
    without interrupting Claude's current work

I tried this one out and it didn’t feel better than opening another claude. It seems to have the existing conversation context, which is helpful, but the question doesn’t seem to be enqueued immediately and waits for the previous prompt to complete, so I don’t love that. My ideal behavior would be to fork the conversation and run the btw prompt in parallel.

Someone told me amp has threads and maybe those do that.

Thanks for reading via RSS!

2025-12-29

Tue, 30 Dec 2025 00:19:41 GMT

Notes on way too many hours spent agent coding over the holidays.

Claude Opus 4.5 running in Claude Code is a workhorse. It hardly ever got stuck in the stacks I was working in (SwiftUI with HTML/CSS/JS in a WebView and a TypeScript 2D Canvas game).
My ideas will move faster than the agent can. I found myself starting additional claude instances to work on smaller fixes and features I intuited wouldn’t overlap with the work the “main” agent was doing. This allowed me to get more things done even while I was poking around with the agent working on more complicated features and refining rough edges.
If the agent is struggling, ultrathink can help in a pinch. It’s also as good or better to plan a feature up front with ultrathink. Hard to compare the two but it’s a nice thing to have in your bag.
Claude Code has gotten excellent at structuring and/or searching projects so it can find existing implementations rather than creating duplicate implementations. This was a problem several months back which created a “point of no return” for fully agent coded projects, where the agents couldn’t search or understand a project well enough to code its way out of problems. I’m sure there is a max codebase size where these agents can be effective, but I’ve had great success on projects of 10k and 100k lines. It’s worth noting the speed of the agent per prompt decreases as lines of code increases. The agent spends more time searching existing files in larger codebases. This seems to be what allows it to continue to implement successfully, and seems to be an unavoidable part of the physics of growing codebases (and was before coding agents were a thing too).
I ran into one weird set of Swift warnings where Claude Code kept undoing and redoing two approaches, neither of which eliminated all warnings. GPT-5.2 in Cursor managed to resolve this one.
My intuition is still necessary. When working on a feature that required a text editor in a Swift app, Claude tried several open source editors and none quite fit the way I wanted them to. I explicitly told Claude to use the Monaco text editor (the core of VS Code) because I have an affinity for it.
Sometimes the agent knows better than me. The agent is great at following directions and sometimes that means it will try to do things it shouldn’t. This has improved since past model iterations but if an approach isn’t working and I give the agent leeway to try a different approach, often I am pleasantly surprised by the idea and adopt something like it.
Vision is indispensable. Software is an incredibly flexible canvas and the agent will give you a decent implementation of whatever you ask for, but if you don’t ask for something or have a rough idea you’re trying to implement, you will get lost in the slop.
For fun, personal projects, I start by doing a voice-to-text brain dump of the idea in a dump.md file. Then I prompt the agent to make a spec.md from these raw thoughts. I read and refine that, then have it start implementing. I use fresh context windows for each of these tasks.
By the end of the holiday, I was working on two or sometimes three projects in parallel. I don’t think this is a particularly desirable thing to do, but otherwise I found I was waiting for the agent to finish and had the thought to do something else.
Agent coding is entrancing and addicting for me, far more than writing software by hand ever was. The feedback loop for writing a prompt and getting what you asked for is tight and the agents are really good at following instructions.
Polish is always needed. I’m continually reminded of an interview Ryo Lu gave where he said “You always start with slop, with AI”. I agree. When I am prompting, it’s quite rare the first draft with AI is exactly what I want; it’s the first draft. From the starting point, I play with the implementation and buff away the rough edges and tweak and tease out the experience or implementation I had in mind. Because of this, metaphors like “sculpting”, “raising”, “chiseling”, and “molding” software resonate for me. The process is completely different than typing out all the characters. You throw a hunk of clay on the table and then start removing, shaping, and adding new things. It’s as much exploration as execution or implementation and the journey and the wrong turns are instructive for how you find the path you prefer.

Thanks for reading via RSS!

2025-12-28

Sun, 28 Dec 2025 15:23:37 GMT

I am working on adding a feature to Tarn where I can type @@ and it will pop a list of content in my blog I can link to in markdown style. This is an idea borrowed from Obsidian for easily reference external content or ideas.

I adore this site design.

be curious about great ideas and engage with them

From AI & I: Best of the Pod: Reid Hoffman on How AI Is Answering Our Biggest Questions, Dec 24, 2025 https://podcasts.apple.com/us/podcast/ai-i/id1719789201?i=1000742605641&r=3610

Thanks for reading via RSS!

2025-12-27

Sat, 27 Dec 2025 17:26:02 GMT

After hitting my head against a few different open source Swift text editor implementations, I ended up settling on a webview with a Monaco in the Swift app. The syntax highlighting isn’t exactly how I want but if in reality, this has changed for me several times over the years as I’ve moved between tools and I’ll probably get over it.

I’ve gotten really pulled into building this thing, inspired by Perfect Software, because I’ve started to see all the little ways in which I can make it mine, in a way that is hard to accomplish with commercial software. With open source software, you can fork, but you inherit the whole project rather than the bits and pieces you want. To me, it feels like the average personal software is going to be simpler than that.

On a different note, I think I’ve gotten pulled so far into this because coding agents (Claude Code) make it so easy (addicting?) to do just one more thing. I’ve been doing one more thing for two days now.

That said, I do have something to show for it. We see if it sticks.

Thanks for reading via RSS!

2025-12-26

Fri, 26 Dec 2025 15:28:00 GMT

I’ve been working with Claude Code to build a tool to help me write in this blog. I’ve historically used Cursor to write in MDX and Claude Code to help build and wire up custom Astro and sometimes React components to add interactivity to posts.

There are a few rough edges though. My content scaffolds are Python scripts, which are not quite as fluid as I’d prefer. My image optimization process is kludgy. My content importing processes (currently experimenting with Drafts) is still a work in progress.

VS Code and Cursor have brought me a long way, but there really is no substitute to having more control over the tools you use everyday. If you want something, you can build it. This is what I am attempting now with Tarn.

Thanks for reading via RSS!

2025-12-25

Fri, 26 Dec 2025 03:39:26 GMT

I’ve seen coding agents refuse or avoid writing comprehensive tests because they claim “most tests are passing” or “90% is great coverage” even when prompted to do something like “fix the tests” or “add tests covering 100%”. Unlike when prompted to code something, it seems when prompted to test things, agents are kind of lazy. I wonder if this specifically correlates with how easy it is to reward hack passing tests. Getting 100% unit coverage is notoriously irritating. There are substantial diminishing returns and the final few percent usually come down to covering lines that bear almost no impact on ensuring the system continues working as designed. Thus, I’m speculating agents struggle to do this because most people can’t be bothered to either.

Or possibly it’s because tests test code, thus models can write code, have it validated, and be reinformed to do better. But I can’t ever remember seeing anyone write code to test tests.

Thanks for reading via RSS!

2025-12-24

Thu, 25 Dec 2025 02:28:13 GMT

I probably would have switched off of Nix for system management, but with Claude Code, any problems I run into are as good as solved.

Thanks for reading via RSS!

2025-12-21

Sun, 21 Dec 2025 08:53:06 GMT

That part of it though, I think is where, for lack of more precise language, people get stuck. It’s like, again, I watch you use Cursor, it’s like you’re literally, it’s like you’re just nudging the model. And your prompts are not, yeah, it’s that demeanor for the listeners.

You’re just poking it. It’s not these long specs. I’m watching you just be like, can you come up with an app idea?

From Dialectic: 34: Ryo Lu - It’s All the Same Thing, Dec 18, 2025 https://podcasts.apple.com/us/podcast/dialectic/id1780282402?i=1000741808474&r=2988

You just need to put your soul in this. You need to care about every detail. You need to not accept whatever purple gradient the AI gave you is the end.

Like that is just the beginning. You always start with shit. You always start with slop, with AI.

And then you refine it, you make it better.

That’s the beginning, not the end.

From Dialectic: 34: Ryo Lu - It’s All the Same Thing, Dec 18, 2025 https://podcasts.apple.com/us/podcast/dialectic/id1780282402?i=1000741808474&r=5060

AI models are trained on all the public knowledge, information and the code that I can see.

And you are trained on the same thing. Like all the books you’ve read, all the fonts that you know, all the artists that you admire, the world around you. And you build that intuition or taste or whatever.

And you start forming an opinion about how you want to shape the world.

And you express it by building.

From Dialectic: 34: Ryo Lu - It’s All the Same Thing, Dec 18, 2025 https://podcasts.apple.com/us/podcast/dialectic/id1780282402?i=1000741808474&r=5133

Thanks for reading via RSS!

2025-12-18

Thu, 18 Dec 2025 13:37:21 GMT

In pushing the envelop in building a large project likely with a coding agent, I am coming around to the need for a structured task management system, whether it be Linear or beads or similar.

Thanks for reading via RSS!

2025-12-17

Thu, 18 Dec 2025 01:44:11 GMT

I commonly use an and/or approach when writing, both for people and language models. At some point (I can’t remember exactly when), I switched to doing the following, for language models only:

and / or

The reasoning in my head was this would tokenize more clearly to the model. Today, I checked the OpenAI tokenizer and kind of think I confirmed this thought process. I’m still not quite sure though, since very few people would actually write this way in the source texts on which models have been trained.

That said, I have seen tokenization tricks work for helping models to solve certain problems, like a Wordle puzzle or counting the letters in a word.

If you know anything about how this works, let me know!

A fun thing about this post - I had Claude Code find the post I wrote about counting letters in a word with LLMs and link it above with the prompt below. I’m going to be using this trick more often.

‘/Users/danielcorin/dev/lab/thought-eddies/src/content/logs/2025/12/17/index.mdx’ “counting the letters in a word” find if I wrote about this and link to it

Thanks for reading via RSS!

2025-12-15

Tue, 16 Dec 2025 03:30:37 GMT

I used amp to create a multi-client, single-server real-time websocket application using Cloudflare Durable Objects. It went quite well.

I wasn’t expecting amp to take so many actions on my behalf without permission. It deployed the site after the first prompt since wrangler was already configured on my system.

Cool, but also I don’t think I’m ready for that level of full-send, having seen a “Claude Code rm -rf’d my home directory” article yesterday.

Thanks for reading via RSS!

2025-12-12

Sat, 13 Dec 2025 00:21:09 GMT

I share Simon’s sentiment on my the skills ideas. I’ve gone even more unstructured - I usually start little ideas and projects with an agent (Claude Code) and a markdown file, then add scripts and data as needed to see how far I can get.

Thanks for reading via RSS!

2025-12-10

Thu, 11 Dec 2025 01:39:42 GMT

Added a tools page to this site, inspired by Simon’s recent post

Thanks for reading via RSS!

2025-12-07

Sun, 07 Dec 2025 23:51:53 GMT

This attempt to recreate the 1996 Space Jam website with Claude caught my attention because I’ve also been running into issues where Claude cannot reproduce code from images faithfully. It’s much better (right now) when you give specifications in language.

Thanks for reading via RSS!

2025-12-05

Fri, 05 Dec 2025 12:28:49 GMT

Figuring out how to teach someone how to use an agent continues to be a problem I only know how to solve by pairing with someone.

Thanks for reading via RSS!

2025-12-04

Thu, 04 Dec 2025 22:39:19 GMT

I did a release of Tomo today for the first time in a while. Dealt with an annoying scrollbar issue and added conversation exporting which I’ve wanted for a while. The exported folder becomes helpful input context to an agent when I use Tomo to do a side task then bring back what I learned into the agent session.

Thanks for reading via RSS!

2025-12-03

Wed, 03 Dec 2025 11:43:47 GMT

I made a Sidenote Astro component, inspired by Tufte CSS.

On wide screens, sidenotes appear in the right margin and start visible by default. The dotted underline indicates text with an associated sidenote. You can click the dotted text to toggle the sidenote’s visibility with a smooth fade animation. When sidenotes are hidden, other visible sidenotes reposition to stay close to their reference text. Hovering over either the dotted text or the sidenote box highlights both and shows their connection. On narrow screens, clicking the dotted underlined text reveals the note inline with the same smooth fade animation. It works like this! − Like this!

Markdown provides native support for numbered footnotes^[1], but this approach never quite sat well with me. I like having the additional context available in visual proximity to the text that references it. I want to be able to see both at the same time. Jumping between the two takes me out of the flow of reading. Putting the note in the margin allows me to see and read it if I want, but doesn’t interrupt the flow of the text and it’s still easy to move past if I so choose. − And if the note isn't relevant to me right now, I can just keep reading.

Like this! ↩

Thanks for reading via RSS!

2025-12-02

Tue, 02 Dec 2025 09:39:17 GMT

I always turn off Cursor Tab when I am writing to think. Otherwise, I end up subconsciously influenced to write what Cursor Tab thinks I am writing, not what I actually mean to write.

Thanks for reading via RSS!

2025-12-01

Mon, 01 Dec 2025 21:35:45 GMT

Lately, I’ve been thinking about the difference and similarities between speaking, writing and thinking. Paul Graham wrote an article about how writing is thinking and that if you think you’re thinking without writing, you only think you’re thinking.

I haven’t known what to do with this idea for a while but think I’ve found something interesting recently.

In using LLM agents to build software, sometimes I use voice to text functionality to get thoughts out of my head as fast as possible. From a pure speed perspective, this is usually the fastest way to get the agent something to start working on. In practice, however, the transcription of what I’ve said isn’t always as exacting or precise as what I had in mind.

I think the appreciation of this freestyle nature is to appreciate what it is to think through writing.

When you speak to communicate, you’re generally trying to communicate some information. But speaking has certain constraints. While you’re speaking, you need to figure out what you’re going to say next while you’re saying the thing you’re already saying.

This is quite different from writing. Oddly enough, I never realized how different this was until I watched a bunch of people demonstrate how they use coding agents.

They would talk about their process, explain what they were going to do, and then would start typing to the agent, at which point, their verbal speed and cadence would change entirely.

They would dictate or type things like the following:

Create a React, Typescript application, using the Tailwind CSS framework …

Usually monotone. Usually slower than natural speaking speed. Usually completely different from how the presenter was previously addressing the audience.

In a presentation forum, this is highly unnatural to listen to.

It’s the conversion of the writing process into something consumable to a speaking presentation format, but it’s awkward.

Having noticed how strange this transition feels, when I do presentation coding with agents now, I tend to prefer a voice to text approach. This approach allows me to fluidly explain myself to the audience and the agent at the same time, whenever I want to, by recording myself when I am talking about what I am going to build and using that transcription as my prompt.

That’s a fun party trick, but back to writing and thinking.

Writing slows you down to a speed where you can care more about where you are going. The constraints of speaking require tradeoffs you don’t have to make when writing.

If you can’t think of quite the right word to describe what you’re thinking, you can fall back to a more simple description or a synonym. This works in conversation, but is a more lossy version of your actual thought and what you were trying to say.

Words and speech are already lossy, but when we take time to write something down, we have the best chance of using the best words we know how to communicate what we mean.

When we consider what this means in the context of writing to think, we might suggest speaking is insufficient to think because it requires mental energy to be committed to other things as well. If you’re presenting, you may be considering your cadence, your body language, your tone, your volume. You may be watching the audience and picking up on cues from them to get a sense of how they are receiving what you are saying. If you’re conversing, you’re listening to your conversation partner and picking up similar cues. In the end, what I think this means is that your mental bandwidth cannot be maximally focused on thinking when you are speaking.

There are too many other things going on. Maybe the closest thing to writing when speaking is to record yourself by yourself with no conversation partner and nothing to distract you. But even still, it’s not quite like writing. You’re looking around, absorbing other stimuli, and subconsciously incorporating it in what you say.

If you close your eyes and dictate, you might come close to thinking like writing, but you still can’t avoid the constraint of being forced to pick the next word to string it together with the last. Unless of course, you are going to pause and deliberately and carefully pick each. and. every. word. as. you. say. it. aloud. But is that really even speaking?

There is a sort of freestyle component to speaking. In our minds, we might think we generally know where we’re going (what point we’re going to make) but we don’t always know how we’re going to get there.

I’ve played with this idea a lot lately.

The next time you’re in a casual conversation, observe the topic and listen to the participants. Get a sense of what is being talked about and stay engaged.

See if you notice multiple threads of conversation. See if it seems like the people speaking are understanding each other. See if you can follow the conversation evolving from one topic to another.

Conversations are rarely planned. They emerge, evolve, and eventually fade away.

Even after an engaging and enjoyable conversation, there is a moment of palpable exhale where you realize it’s time to turn the page.

So what are we doing when we converse? Are we thinking?

If we need more of our mental bandwidth to focus fully, then maybe we’re thinking to a certain degree but at a diminished capacity.

If we’re debating, we’ll hear the argument of our opponent and then be expected to respond, advocating for our perspective over theirs. We’re thinking on our feet, but maybe not as deeply or clearly as we’re capable of doing.

We’re kind of thinking, but really we’re reacting. We’ve practiced debating and when we hear an argument we only have so much time to respond. There are lots of techniques you could use to buy time.

Take a deep breath. Pause. Vamp. Talk about something unrelated.

All to buy time. Time to think.

Because when speaking, time is the tightest constraint limiting the clear communication of your ideas.

When speaking, it’s more difficult to manage a conversation. Harder to retrace to previous topics. To prune earlier branches. With a partner it’s nearly impossible and would likely make the conversation feel quite unnatural.

There is a natural give and take of a conversation. If someone is talking about their day, it would be unbecoming to interrupt or abruptly change the topic to something else.

When speaking, you live in the flow of time. You need to speak words in an order that make sense. You can’t return to a sentence from 5 minutes ago, edit that, then reorganize it to better state your point. When speaking you kind of can, but it takes practice and you have to expend energy bringing your partner or audience with you.

I know I said that you can think when you speak but I don’t think I said that quite right. What I mean was, you can do an approximation of thinking while speaking but to truly think with clarity you must write.

If you say that, people can follow what you’re talking about, but you have to lay a trail of breadcrumbs. When writing, the thoughts are more pure and uninterrupted.

If you have the thought that leads to speaking the quote — maybe something like

My point is that writing is thinking and speaking isn’t quite thinking so I need to ensure I communicate clearly that speaking isn’t quite clear thinking

You can change your prose to reflect that. You don’t need to bring anyone along with you. You don’t have to justify your hard pivot. You can think and respond to your thought alone without the decorum or ceremony required when speaking.

To respond to this thought while speaking compared to writing requires different orders of magnitude of mental bandwidth. So much more, I could almost imagine myself letting the misunderstanding go while speaking because it would just be so much work to return and correct the record with linear conversation.

When writing, the nonlinear edits are possible and easy. The ideas exist concretely in words, to be sculpted into ideas.

Thanks for reading via RSS!

2025-11-21

Fri, 21 Nov 2025 18:56:41 GMT

I’ve been thinking about how, in software engineering especially, it feels like the balance of who is the builder and who is the user is shifting. It used to be the case that the software engineer was clearly the builder. I don’t find that to be the case anymore.

Being the builder is fun. You get to try new things, experiment, and see your work come to life. You have a lot of control over the process of how you work.

Being the user, especially of a rapidly changing product, is not always fun. As a user, you adopted a product initially because of its utility, but then the changes start.

Pricing changes. Features evolve or go away. The builders try and increase the scope of their user base.

The tools used for software engineering are changing. Language models are in everything, for better or worse.

As software engineers, we’re now experiencing what many of us have put our users through for years. But it’s not being done by any one company, but by our industry itself to us.

More than I can ever recall, organizations seem to be measuring statistics like

lines of code written
PRs merged
issues closed
daily active sessions on internal tools
(and now) inference tokens used

Engineers are now the targets of the optimization loops, previously mostly reserved for the users of the products engineers built.

Prodded with surveys

On a scale of 1 to 10, how likely would you be to recommend this product to a friend or colleague?

Turned upside down and shaken for a bit of knowledge to feed into the tool makers’ road maps.

I get it. Language models change the game for builder software.

I feel this way.

The tools are useful. The demand for them is real.

But this post from Lenny has stuck with me:

Getting frequent pings from CPOs/VP looking for product leaders at other companies who’ve found great success with AI tools/workflows.

Everyone’s trying to figure out what everyone else has figured out.
— Lenny Rachitsky (@lennysan) October 9, 2025

Frantically.

Organizations are pulling out all the usual tools to try and make things happen. And engineers are feeling the pressure.

Thanks for reading via RSS!

2025-11-15

Sat, 15 Nov 2025 22:38:50 GMT

Spent several hours coding with Cursor’s composer-1 model. I like it a lot and it is so fast. When you have it working it’s one of my favorite agent coding tools I’ve used. Well done.

Thanks for reading via RSS!

2025-11-14

Fri, 14 Nov 2025 18:50:17 GMT

I’ve fallen behind on blog things. In the past few weeks, I’ve been

experimenting with claude-agent-sdk, equipping the agent with MCP and prompting to use a CLI measure trade-offs and ergonomics
using humanlayer’s RPI (research, plan, implement) commands on large codebases to implement complex features
experimenting with sprites and tile sets for a game idea

Thanks for reading via RSS!

2025-11-08

Sat, 08 Nov 2025 11:14:11 GMT

I built a Claude Code slash command that given a prompt, analyzes the ongoing chat and captures to insight or learning from the conversations. This seems like it’s going to be useful for capturing and searching a domain-specific knowledge base using queries that come to mind when I remember I did “that one thing a few months ago”.

I’m planning to start this as a personal experiment but also see if I can incorporate it codebases with Claude Code commands and agents.

Thanks for reading via RSS!

2025-11-03

Mon, 03 Nov 2025 21:10:01 GMT

I learned about the TOON input data format for language models. I wonder if this could be useful at all for the prompt built by DSPy.

Thanks for reading via RSS!

2025-11-02

Sun, 02 Nov 2025 11:14:53 GMT

I’ve been experimenting using Claude Code to learn Rust.

Like learning to write it myself, not just build stuff in it with Claude Code. I’m still unsure what level of abstraction learning programming is at now.

I’ve found, if you’re familiar with building software, you can often be pretty serviceable in a language you don’t know aided by an agent. But different languages help you learn to think differently about the types of problems you encounter when building software.

I also want to know if and how I can use Claude Code explicitly for the purposes of learning something new.

My general structure was to prompt the goal of learning Rust, then ask for a plan that covers increasingly advanced topics in Rust. Then, given this plan and “unit” structure, I prompted for koan-like exercises and a small project for each unit.

Claude Code really wanted to do all the work for me, but with some nudging, I’ve been able to get it to create the scaffolding for me to learn and then practice. After making these improvements, I had the agent re-write the plan to incorporate the new approaches.

Thanks for reading via RSS!

2025-10-29

Wed, 29 Oct 2025 21:02:08 GMT

Cursor released their “Composer” model today. I believe the name is a hat tip to the original name of their agent/multi-file editing mode.

Per the post

Composer is a mixture-of-experts (MoE) language model supporting long-context generation and understanding. It is specialized for software engineering through reinforcement learning (RL) in a diverse range of development environments.

I’ve long been of the opinion that the only way Cursor was going to have staying power as a company was to train their own model. If they hadn’t, my thesis was that they would continue to give all their margin away to the lab(s) with the best coding model(s).

Cursor has a ton of code data. They also see users’ preferences for models.

By training their own model, maybe they can better control their own destiny and not be at the mercy of the labs and their API pricing. The open question is whether this model is good enough to convince their customers to switch to it.

Thanks for reading via RSS!

2025-10-25

Sat, 25 Oct 2025 21:04:34 GMT

I played around with the idea of using a language model as a sort of generic response engine for any request using any protocol. That plan was to write a system prompt that the language model would use as the primary context and role. I could then prompt “you are an HTTP server, you define the following endpoints” and then send actual HTTP requests to the model and have the model respond as if it were an HTTP server. This idea taken further could allow you to just point a Postgres client or a gRPC client at a port for a process running that wraps a language model and then have it respond based on any kind of definition you did in the system prompt and by its own inference to come up with reasonable responses given the request.

I still feel like there’s something to this idea, though early attempts were a little bit rough. One of the bigger challenges is that most language models are pretty slow relative to these deterministic processes. Also, expecting a model to output certain details of a response, like Content-Length in an HTTP response, is not reliable. To make parts like that work, you need to add deterministic logic around the model’s output. If you do that, it almost makes more sense to just use an agent to scaffold a test service from your specs or IDLs rather than attempt to generate the response with the model via runtime inference.

Thanks for reading via RSS!

2025-10-24

Fri, 24 Oct 2025 12:29:38 GMT

I’ve now run into the following issue multiple times using Bedrock. It’s my own mistake but I end up confused each time.

An error occurred (ValidationException) when calling the InvokeModelWithResponseStream operation:
The maximum tokens you requested exceeds the model limit of 65536

I had this prompt when attempting to run inference on Bedrock with us.anthropic.claude-sonnet-4-20250514-v1:0.

The problem was I was trying to set the output tokens larger than the model limit. Claude Sonnet 4 has versions that support 200k and 1M tokens context windows but this is for the input tokens to the model.

I hope this note helps if you run into the same issue.

Thanks for reading via RSS!

2025-10-22

Wed, 22 Oct 2025 20:08:50 GMT

I’ve been finding that pairing with someone to help them get familiar with building with agents to be a very effective way to help them get started successfully. Everyone enters the process with a different set of skills and it’s tough to know what parts will be confusing or challenging for them.

When you’re pairing, you get to fly through the parts that your pair knows already and stop and focus on the parts that don’t come as easily or that they aren’t familiar with. With support on the surrounding tasks, they have a chance to get a feel for prompting and developing a workflow for safely trying things out with the ability to revert if something breaks.

With an agent, you get to bring your expertise to the software you’re building. The words, terminology, and your ability to describe what you want are the springboard to using the agent to build what you want. But there are a lot of barriers to entry: the terminal and system setup, getting API keys or paying for an agent subscription, and countless decisions that may feel consequential or irreversible when you don’t know better. Having someone trusted to guide you through these parts gives you a great chance to see the opportunity and possibilities of coding with an agent.

Thanks for reading via RSS!

2025-10-20

Mon, 20 Oct 2025 19:41:10 GMT

I tried out Claude Code’s new planning mode recently and ran a demo with it at work today. I stumbled upon it last week.

You get to planning mode by running claude and pressing shift+tab twice. You’ll see

  ⏸ plan mode on (shift+tab to cycle)

Now, when I am in plan mode, Claude Code takes a sort of Socratic approach, asking me questions to clarify my initial vague prompt

> let's write a journal cli app

⏺ I'll help you design a journal CLI app! Let me first understand your requirements and then create a comprehensive plan.


────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
←  ☐ Language  ☐ Features  ☐ Storage  ☐ UI Style  ✔ Submit  →

What programming language would you prefer for the journal CLI app?
❯ 1. Python
     Great for CLI apps with libraries like Click or argparse. Easy to distribute and cross-platform.
  2. Node.js/TypeScript
     Modern JavaScript/TypeScript with libraries like Commander.js. Good npm ecosystem.
  3. Go
     Compiled binary, fast startup, excellent for CLI tools. Built-in flag parsing.
  4. Rust
     Fast, memory-safe, excellent CLI libraries like clap. Single binary distribution.
  5. Type something.

Enter to select · Tab/Arrow keys to navigate · Esc to cancel

An understandable, but sort of weird thing that got me twice today, is if you cursor down to the Type something. option and press the arrow keys or tab, the TUI does not cycle to the next plan option like it would if you had one of the other options selected. This kind of makes sense because it’s listening for your custom input but I’ve already gotten stuck on that twice.

Sometimes the agent won’t ask you any clarifying questions. Sometimes it will ask multiple rounds of clarifying questions.

I generally like this approach because it’s a nice on ramp to the idea of planning and an introduction to the concept that you get better results by giving the agent more of an idea of what you want to build. It reminds me a bit of the work I did on write-partner 2 years (!) ago, using the model to coax information out of me by asking follow up questions.

If you expect the agent to read your mind and fill in the gaps, you’ll usually be disappointed.

Thanks for reading via RSS!

2025-10-17

Fri, 17 Oct 2025 22:55:18 GMT

Shipped the escape room today. Just needed to fix a few small bugs with image imports and display. The code is not great but it generally works. Saw an interesting performance issue with one of the mini-games on the live site compared to local.

Thanks for reading via RSS!

2025-10-13

Mon, 13 Oct 2025 16:30:18 GMT

Experimenting with the useful sqlite-vector library for managing vector embeddings in SQLite. Also wrote up a Datasette plugin to query the embeddings in the Datasette tool.

This seems to be faster and easier than I last attempted this a year and a half ago with sqlite-vss.

Thanks for reading via RSS!

2025-10-08

Wed, 08 Oct 2025 22:32:12 GMT

I started a list of things that seem to come up frequently when it comes to enduring design and user experience and empowerment. Currently the list is the following:

Spreadsheets
Bicycles
Legos
Unix

Thanks for reading via RSS!

2025-10-04

Sat, 04 Oct 2025 12:28:57 GMT

I downgraded Claude Code to standard usage a little while ago and I have to say it’s one of the best things I’ve done recently. Not to say that I think Claude Code is excellent and effective - it is - but when I (occasionally) max out my Claude Code quota, I’m motivated to try a different agent coding tool.

Today, I did more work with codex than I’ve attempted in a while and found it to be quite effective and enjoyable.

Thanks for reading via RSS!

2025-10-02

Thu, 02 Oct 2025 20:20:08 GMT

Currently watching this video a bit at a time.

Lots of great takeaways. A memorable one - references specific Claude Code tools by name when prompting.

Thanks for reading via RSS!

2025-10-01

Wed, 01 Oct 2025 19:34:57 GMT

ctrl + v on macOS pastes and image from the clipboard into Claude Code. Amazing to finally know how to do this.

Thanks for reading via RSS!

2025-09-29

Mon, 29 Sep 2025 21:38:56 GMT

Cute detail from the Claude Code update today

Thanks for reading via RSS!

2025-09-20

Sat, 20 Sep 2025 17:04:01 GMT

In a pinch, took a shot and found out claude --resume is a real thing

Thanks for reading via RSS!

2025-09-14

Sun, 14 Sep 2025 11:01:55 GMT

I think that your point about intuitions and getting intuitions is the big thing. And I think people, we don’t realize is when you’re dealing with something fundamentally new, you can’t trust, like, how you reason about it without experiencing it, because you have to build the intuition in order to be able to reason about what it means and how it fits in and whether it works or not.

And we’re just not used to that because we’re used to reasoning about things we already have an intuition for. And I think that’s why, like, when you first see, maybe when you first saw ChatGPT, you’d be like, oh my God, it can do everything. Like, we’re not going to have jobs in a year.

And now we’re like three years in and we’re like, yeah, it’s awesome. And jobs are complicated. There’s a lot of complex stuff that we do

— AI & I: Claude Code Can Be Your Second Brain, Sep 10, 2025

https://podcasts.apple.com/us/podcast/ai-i/id1719789201?i=1000725911151&r=2737

Thanks for reading via RSS!

2025-09-08

Mon, 08 Sep 2025 20:57:02 GMT

The world is vast and deeply detailed.

If you build an ever more detailed physical model of the world’s topography, eventually you end up with a map the size of the world itself^[1].

Maps and models allow us to think and see and reason about broad concepts and how they interact with each other. This is the difference between seeing a bag of marbles as

blue 🔵
blue 🔵
red 🔴
green 🟢
blue 🔵

and another bag as

red 🔴
green 🟢
green 🟢
blue 🔵
red 🔴
green 🟢

and the more abstract concept that bag one has a most common color of “blue 🔵” and bag two of “green 🟢”.

Statistics are a common example of compressing reality into an abstraction that can more easier be reported and compared against. Mode (described above) is a relatively lossy statistic. We could prefer something like color counts instead.

Hand one
🔴 red: 1
🟢 green: 1
🔵 blue: 3

That’s more information than just the mode. We know about other colors and their rate of occurrence.

But we still don’t know their position in the bag. Whether any overlapped each other. Which colors were touching which. What the size and weight of the marbles were.

The level of detail that can potentially be captured to describe this bag of marbles continues down to an identical replica of the true physical state. This brings us back to the beginning of this essay.

All models are wrong

But some are useful^[2].

When creating models or reporting statistics, you need to decide which elements of the situation must be captured and which can be neglected without compromising the clarity of the abstraction or description of the situation.

In the case of the marbles, if we’re just trying to describe the “marble-color” population, it could be ok to neglect the position of the marbles in the bag or their size. But when we read that data of the counts of the colors of the marbles, often implicit in our reading can be implied sameness across other characteristics of those marbles.

When I devised this example I imagined all the marbles were the same weight, size and made of the same material. But a dataset of counts of marbles and their colors doesn’t imply that. The statistic has its focus and neglects the rest.

Models in writing

Writing is a means by which you can transfer an idea from your head into someone else’s. You have to decide which data points, which statistics, which models you include in your writing in your attempt to conjure your idea within the mind of the reader.

Different examples will be effective for different readers. It’s the writers job to mark these paths for readers. Not every path will be taken, but many can lead to the same place.

Maybe my “bag of marbles” analogy example didn’t land for you. I could present another analogy or model that might. I could present several.

But too many models or examples and readers will lose sight of the point of the writing.

What you don’t write

You also have to decide what you aren’t going to write about. More so, even.

There are so many things you could write about, and you have to decide what to leave out.

Thanks for reading via RSS!

2025-09-07

Sun, 07 Sep 2025 19:21:32 GMT

Two separate people texted me that OpenAI finally added conversation branching to ChatGPT.

By popular request: you can now branch conversations in ChatGPT, letting you more easily explore different directions without losing your original thread.

Available now to logged-in users on web. https://t.co/l6RflhOGuK pic.twitter.com/CpVehkULOB
— OpenAI (@OpenAI) September 4, 2025

The implementation is quite basic and doesn’t make the different branches visible, but it’s nice to see it added nonetheless. After working on Delta, I never quite sure what the best UX for the feature was.

Shortly after, I was this tweet from John

Being one of the first at the intersection of creativity and a new technology doesn’t really matter. That applies in art, design, or entrepreneurship.

What you did, if truly “novel,” will get done and redone by others as years go by. Eventually it won’t be a big deal at all.…
— John Maeda (@johnmaeda) September 7, 2025

So, on to the next one.

Thanks for reading via RSS!

2025-09-04

Thu, 04 Sep 2025 16:00:55 GMT

I met up with a friend the other day who is a software engineer in between roles. We were talking about the future of the job and the field more broadly. With the popularization of language models it feels like the ways we work and assumptions we’ve held are being upended, and not for the first time.

I personally believe the job description of “software engineer” is going to continue to mean and entail a greater diversity of skills than ever before.

Software engineers (even prior to them being called as such) have always provided instructions to computers and tried to organize both the instructions and computers in such a way that an intended task could be accomplished by using that software through an interface to the computer.

This has not changed.

What has changed are the breadth and complexity of areas in which an individual building software needs to participate.

Thanks for reading via RSS!

2025-09-03

Wed, 03 Sep 2025 21:07:10 GMT

I tried OpenAI’s codex CLI coding agent again today for the first time in a while.

Update available! 0.1.2504301751 → 0.29.0.

I recently shipped a release of Tomo but there was a minor bug with the horizontal scrollbar overlapping a codeblock with syntax highlighting.

I prompted codex to fix this and it seemed to localize the problem and suggest a working fix in one attempt.

ensure that the horizontal scrollbar in styled codeblocks does not overlap the code itself. right now I am seeing that behavior

I was motivated to try this out because I’ve been hearing rumblings about gpt-5 being good at coding. It’s currently the number 4 coding model on OpenRouter and climbing, relative to both Sonnet and Qwen3 Coder.

Rank	Model	Company	Parameters	Performance
1	Grok Code Fast 1	x-ai	844B tokens	↑ 3,231%
2	Claude Sonnet 4	anthropic	339B tokens	↓ 9%
3	Qwen3 Coder 480B A35B	qwen	112B tokens	↓ 33%
4	GPT-5	openai	66.2B tokens	↑ 21%
5	DeepSeek V3.1	deepseek	60.9B tokens	↓ 56%

I believe Grok Code Fast 1 was launched in some kind of free capacity along with its release which may in part explain its high usage but we will see where that lands.

Thanks for reading via RSS!

2025-09-01

Mon, 01 Sep 2025 18:45:22 GMT

An interesting learning about the iOS Clock alarm app’s time picker. It turns out it’s a long list rather than a circular buffer. Scrolling to the bottom of all three picker lists positions the time at 4:39pm.

Thanks for reading via RSS!

2025-08-31

Sun, 31 Aug 2025 18:42:38 GMT

Our approach to applying the intelligence of language models is inside out.

Providing a model access to a number of tool calls and expecting the model to correctly select a tool given a scenario then invoke the tool properly given the surrounding context goes too far for the pattern to be generically useful. There are examples of agents of this type that are effective, but they are effective specifically because they are tested in the scenarios in which they are useful. Attaching tools to a model and calling inference in a loop is only a starting point. From here, refinement is needed to build something consistently reliable and useful.

The context the agent sees much be managed. That includes the tool calls that are made available to the agent.

The result may look something like a harness bespoke to the model, an evaluation set with measurable outcomes, and a process for verification either by a human or another model.

To draw some contrast, constrained, structured, verifiable inference becomes a building block on top of which any type of software system can be built. Specifically, given an input does the model reliably produce the known, expected output?

Existing software systems can be engineered to be durable and reliable. Model inference is inherently probabilistic. The more leeway you give a model to make a decision on behalf of the user or the system, the more likely there is for long tail failures of the system.

For example, let’s say your business has a rule where if a customer uploads a document of type A, or two documents, one of type B and another of type C, then the user can proceed to the next step.

The classical implementation of this type of workflow in software would be to create a form that allows the user to upload their documents, then create a task in some kind of system for a human to review. From there, if the documents are deemed sufficient given your company’s policies by the human reviewer, the user can proceed. If the documents are not sufficient, the user needs to be notified with (hopefully) some clarification of what is missing and the process repeats.

The challenges with this approach are multifold. It’s slow for the user since they need to wait for a human to review their documents before they can proceed. It’s expensive because it requires the time, training, and expertise of a human to perform review. It can be error prone and inconsistent because it’s difficult to ensure homogeneous judgement across diverse human evaluators and humans inevitably make mistakes, especially when performing many similar tasks in a row or when they’re put under pressure to do a certain quantity under time constraints.

Language models show promise for solving these types of challenges but architecting a solution that can produce a good result to speed up this process and improve on the quality of the document evaluation requires a thoughtful approach to architecting the software system. AI does not appear to be a silver bullet that makes the problem disappear without thought or effort.

Thanks for reading via RSS!

2025-08-28

Thu, 28 Aug 2025 13:35:38 GMT

Fiddling around with an /activity page for the site.

Thanks for reading via RSS!

2025-08-27

Wed, 27 Aug 2025 22:37:30 GMT

Experimenting with activity sparkline concept

Thanks for reading via RSS!

2025-08-26

Tue, 26 Aug 2025 08:40:49 GMT

A sad morning for Claude Code. Amp is mostly working though.

> you there?
  ⎿  ⧉ Selected 1 lines from README.md in Cursor
  ⎿ API Error: 500 {"type":"error","error":{"type":"api_error","message":"Overloaded"},"
    request_id":null}

  ⎿  API Error (529 {"type":"error","error":{"type":"overloaded_error","message":"Overloaded"},"request_id":null}) · Retrying in 1 seconds… (attempt 1/10)

I tried to create a CSS package to add browser styles with the look and feel of Mac OS 9. I thought it would be as simple as showing a coding agent some screenshots of Mac OS 9. It was not.

Also played around with building some Mac OS 9 styled components in CSS.

Thanks for reading via RSS!

2025-08-25

Mon, 25 Aug 2025 20:39:43 GMT

I had a bit of inspiration and started working on adding code syntax highlighting to Tomo.

Seeing an increasing number of folks running coding agents like Claude Code in loop and with --dangerously-skip-permissions. You can accomplish a lot like this with the proper setup. In my experience, the agent needs to be able to evaluate and test its own work to be successful with this approach.

I don’t think I would attempt this without using some kind of sandbox for the agent as exploits are becoming increasingly common and advanced.

Thanks for reading via RSS!

2025-08-20

Wed, 20 Aug 2025 20:05:47 GMT

Before LLMs, provided that your team did some of their due diligence, you could always expect to have some help when tackling new code-bases. Either a mentor, or at least some (even if maybe partially outdated) documentation. With LLMs, this is gone. The only thing you can rely on is on your ability to decipher what a highly imperfect system generated, and maybe ask explanations to that same imperfect system about ~~your code~~ its code (oh, and it has forgotten everything about the initial writing process by then).

— https://www.mindflash.org/coding/ai/ai-and-the-bus-factor-of-0-1608

This article is fair and relevant commentary on the practicing of thoughtlessly producing code to solve a problem without ever reading it. However, I disagree with this passage’s assertion that “you could always expect to have some help when tackling new code-bases”. At this point in my career, the majority of the codebases I’ve worked in have required that I learn parts of the system from zero, effectively as if there was no one with any knowledge of the system (aka bus factor of 0).

Practically speaking, even if there actually are some people with knowledge of the system, it may be impractical to rely on them to get you up to speed on a codebase where you need to make some change. I’ve found the most success through figuring things out with the aim to unblock myself.

This approach may only go so far. You may be able to create and merge a PR but not deploy. Or deploy but not know how to run migrations. But the more experience I’ve had working as a software engineer, the more relevant and useful doing the archeological work of digging through the history of a codebase has been.

Thanks for reading via RSS!

2025-08-19

Tue, 19 Aug 2025 17:06:32 GMT

It’s been a while since I’ve attempted to use LLMs to solve Connections puzzles.

In the past, I used a harness and allowed the LLM to guess one group at a time with logic that would check if the guess was correct or valid. This approach was closer to how humans play the game and helped claude-3-5-sonnet get ~40% of the puzzles correct over 32 games.

Today, I took a screenshot of the puzzle and sent it to gpt-5 and claude-4-1-opus (with not tools enabled) with the prompt

User

solve today’s nytimes connections puzzle

claude-4-1-opus-20250805

Thinking process

Looking at this Connections puzzle, I need to identify four groups of four words each. Let me analyze the words:

VERMOUTH, PLYMOUTH, CLASSIC, DASHBOARD BLABBERMOUTH, DARTMOUTH, LITTLE, PORT FLYWHEEL, CHATTERBOX, RUSHMORE, MARSALA WINDBAG, THE, SHERRY, PRATTLER

Let me look for patterns:

Words ending in “MOUTH”:

VERMOUTH
PLYMOUTH
BLABBERMOUTH
DARTMOUTH These all end in “MOUTH” - that’s a group of 4!

Words that mean someone who talks a lot:

BLABBERMOUTH
CHATTERBOX
WINDBAG
PRATTLER Wait, BLABBERMOUTH is already in the “MOUTH” group. Let me reconsider.

Actually, let me look at this differently:

Cities/Places ending in “MOUTH”:

PLYMOUTH (city in England and Massachusetts)
DARTMOUTH (city/college)
Actually, BLABBERMOUTH isn’t a place…

Let me think about wines/alcohols:

VERMOUTH (fortified wine)
PORT (wine)
MARSALA (wine)
SHERRY (wine) That’s 4! These are all fortified wines.

So we have: Group 1: Fortified wines - VERMOUTH, PORT, MARSALA, SHERRY

Now looking at the remaining words: PLYMOUTH, CLASSIC, DASHBOARD BLABBERMOUTH, DARTMOUTH, LITTLE FLYWHEEL, CHATTERBOX, RUSHMORE WINDBAG, THE, PRATTLER

Words for talkative people:

BLABBERMOUTH
CHATTERBOX
WINDBAG
PRATTLER That’s 4! These all mean someone who talks excessively.

I notice a pattern of locations with “MOUTH” in their names. PLYMOUTH and DARTMOUTH stand out as geographical references. I’m seeing potential connections between these place names and how they might form a meaningful group. The remaining words seem scattered, but there might be a hidden link between LITTLE, CLASSIC, DASHBOARD, FLYWHEEL, and THE that isn’t immediately obvious. I’ll need to dig deeper to uncover their potential relationship. I’m exploring potential connections between words like Plymouth Rock, Little Rock, and other terms. I’m noticing some interesting patterns around “ROCK” and trying to categorize different types of references - geographic locations, music genres, and even people like Dwayne “The Rock” Johnson. Some words like Dashboard and Flywheel seem less connected, so I’m probing for underlying themes. My thinking is moving between different conceptual approaches, looking for meaningful links between these seemingly disparate terms.

I’ve identified some potential word formation strategies: compound words, parts of phrases, and categorical groupings. For instance, I’m examining “BOARD” and “WHEEL” as potential connective elements, and exploring how words might combine or relate to each other. My strategy involves breaking down the words, looking for structural similarities, and testing different interpretational frameworks to uncover potential relationships.

The current challenge is finding a coherent pattern that connects Plymouth, Classic, Dashboard, Dartmouth, Little, Flywheel, Rushmore, and “The” in a meaningful way. I’m systematically testing different linguistic and conceptual approaches to see what emerges.

I notice several interesting categories: academic institutions (Dartmouth, Plymouth), film references (Rushmore, The), descriptive terms (Classic, Little), and mechanical/structural words (Dashboard, Flywheel). These suggest multiple potential connection pathways, from thematic to structural to categorical.

The Wes Anderson film connection and historical landmark references provide additional contextual layers, hinting at potential narrative or cultural intersections between these seemingly disparate terms.

Phonetic similarities emerge with words ending in “-uth” sounds, creating an unexpected linguistic pattern across different domains. The exploration reveals complex semantic networks where terms can shift meaning or connection depending on context.

I’m tracking potential linking strategies: sound-based connections, categorical relationships, and unexpected word combinations that might reveal surprising associations.

Car-related terminology offers intriguing possibilities - brands, parts, and descriptive language intersect in ways that could unlock the pattern. Compound word structures and articles like “THE” suggest additional pathways for connecting seemingly disparate terms.

The challenge involves finding underlying structural logic that transforms these discrete words into a coherent system, requiring creative lateral thinking and careful pattern recognition.

American geographical references emerge as a compelling thread, with landmarks and place names revealing potential connections across diverse locations. Each term potentially represents a unique slice of American cultural or geographical landscape, hinting at deeper associative relationships.

The “ROCK” pattern surfaces intriguingly, linking PLYMOUTH, LITTLE, CLASSIC, and THE through a consistent linguistic structure. These terms suggest a rhythmic naming convention spanning different states and contexts.

Exploring the remaining words - DARTMOUTH, DASHBOARD, FLYWHEEL, RUSHMORE - reveals less obvious connections. While initially appearing disparate, these terms might conceal a subtle underlying pattern waiting to be uncovered.

The investigation continues, probing for unexpected links between seemingly unrelated concepts across geographical, linguistic, and categorical domains. I’m exploring word connections through different lenses. First, fortified wines like VERMOUTH and PORT share a category. Then, words describing talkative people - BLABBERMOUTH and CHATTERBOX reveal a linguistic pattern.

For the remaining words - DARTMOUTH, DASHBOARD, FLYWHEEL, RUSHMORE - I notice they’re compound words with fascinating verb roots. DART, DASH, FLY, RUSH all suggest quick, dynamic movement. Each word breaks down into a verb suggesting rapid action, creating an intriguing linguistic connection.

The MOUTH-ending words (VERMOUTH, PLYMOUTH, BLABBERMOUTH, DARTMOUTH) cross multiple categories, adding complexity to my analysis. Some represent drinks, others describe people or places, showing how language interweaves seemingly unrelated concepts.

I’m strategizing how to categorize these words without creating category overlap, which is crucial in solving the Connections puzzle. The challenge lies in distinguishing unique characteristics while maintaining clear, non-redundant groupings.

My current focus is separating the words precisely: wines must stay distinct from descriptive terms, ensuring each word fits a singular, unambiguous category.

The talkative people group expands to include CHATTERBOX, WINDBAG, and PRATTLER, complementing the earlier BLABBERMOUTH entry. These words share a clear thematic connection of verbose communication.

For the remaining words, I’ve identified a fascinating pattern: PLYMOUTH, LITTLE, CLASSIC, and THE form a ”___ ROCK” group, while DARTMOUTH, DASHBOARD, FLYWHEEL, and RUSHMORE connect through compound words with first elements functioning as action verbs suggesting rapid movement.

My strategy involves methodically parsing the word list, seeking intrinsic linguistic or semantic relationships that create coherent, non-overlapping categories.

Looking at this Connections puzzle, I need to find four groups of four related words.

Let me start by identifying some clear patterns:

Fortified Wines: I can see VERMOUTH, PORT, MARSALA, and SHERRY - these are all types of fortified wines.

Talkative People: BLABBERMOUTH, CHATTERBOX, WINDBAG, and PRATTLER - these all describe someone who talks excessively.

For the remaining 8 words, let me look for connections:

”___ ROCK”: PLYMOUTH (Rock), LITTLE (Rock), CLASSIC (Rock), and THE (Rock - either the actor or Alcatraz) - these can all precede “ROCK” to form common phrases.

The last group: DARTMOUTH, DASHBOARD, FLYWHEEL, and RUSHMORE - these are all compound words that begin with verbs of quick movement: DART, DASH, FLY, and RUSH.

So my four groups are:

Fortified wines: VERMOUTH, PORT, MARSALA, SHERRY
Excessive talkers: BLABBERMOUTH, CHATTERBOX, WINDBAG, PRATTLER
”___ ROCK”: PLYMOUTH, LITTLE, CLASSIC, THE
Start with quick movement verbs: DARTMOUTH, DASHBOARD, FLYWHEEL, RUSHMORE

According to the playground, Opus used 3,665 tokens and took well over a minute to respond (I didn’t time it and the tool doesn’t say).

When given the same prompt, gpt-5 with reasoning effort high and verbosity high took 2:38 and used 6,957 tokens.

Both models solved the puzzle correctly in one attempt and with no feedback from the game itself as a human would receive while playing - a harder task. Impressive progress after a year since I last attempted this.

Thanks for reading via RSS!

2025-08-18

Mon, 18 Aug 2025 20:18:53 GMT

I saw another Claude-generated Astro blog today. The post list page looked surprisingly like the one on this site.

Maybe I shouldn’t be surprised. Working with Claude feels collaborative but unless you’re prompting for specific things, Claude’s defaults seem to be pretty consistent across uses.

Thanks for reading via RSS!

2025-08-13

Wed, 13 Aug 2025 08:29:28 GMT

Coming up for air after a bunch of time spent on a project trying to use llm as a router to different models to create a compatibility layer for the OpenAI /v1/chat/completions API and the Anthropic /v1/messages API.

In basic cases, this works pretty well. For tool calling, it’s a bit more complicated. Nuances with the different providers and models require intervention to make sure the caller (like Claude Code) which may be expecting to interact with the Anthropic API is able to play nicely with other providers and models.

One example is the Anthropic API requires the max_tokens field to be set. However, Gemini models don’t support it. The llm tool supports setting model.Options per model. For many Gemini models, these look something like:

llm models --options

...

GeminiPro: gemini/gemini-2.5-pro (aliases: gemini-2.5-pro)
  Options:
    code_execution: boolean
    temperature: float
    max_output_tokens: int
    top_p: float
    top_k: int
    json_object: boolean
    timeout: float
    google_search: boolean
    thinking_budget: int

...

Without intervention, this leads a Claude Code to make a request proxied through my server that leads to this error:

{
  "detail": "Error generating response: 1 validation error for OptionsWithThinkingBudget\nmax_tokens\n  Extra inputs are not permitted [type=extra_forbidden, input_value=1024, input_type=int]\n    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden"
}

This particular issue can be worked around by first checking the model’s supported options and then filtering out those that are unsupported.

The challenges don’t end there.

Claude Code specifies a tool’s input schema like this:

{

  ...

  "input_schema": {
    "type": "object",
    "properties": {
      "description": {
        "type": "string",
        "description": "A short (3-5 word) description of the task"
      },
      "prompt": {
        "type": "string",
        "description": "The task for the agent to perform"
      },
      "subagent_type": {
        "type": "string",
        "description": "The type of specialized agent to use for this task"
      }
    },
    "required": ["description", "prompt", "subagent_type"],
    "additionalProperties": false,
    "$schema": "http://json-schema.org/draft-07/schema#"
  }
}

but Gemini doesn’t accept the additionalProperties or $schema fields.

INFO:     127.0.0.1:60097 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
    ...
    |   File "/Users/danielcorin/dev/llm-api/.venv/lib/python3.11/site-packages/starlette/responses.py", line 254, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/Users/danielcorin/dev/llm-api/llm_api/anthropic_api.py", line 341, in stream_anthropic_response
    |     for chunk in response:
    |   File "/Users/danielcorin/dev/llm-api/.venv/lib/python3.11/site-packages/llm/models.py", line 1172, in __iter__
    |     for chunk in self.model.execute(
    |   File "/Users/danielcorin/dev/llm-api/.venv/lib/python3.11/site-packages/llm_gemini.py", line 521, in execute
    |     raise llm.ModelError(event["error"]["message"])
    | llm.errors.ModelError: Invalid JSON payload received. Unknown name "additionalProperties" at 'tools[0].function_declarations[0].parameters': Cannot find field.
    | Invalid JSON payload received. Unknown name "$schema" at 'tools[0].function_declarations[0].parameters': Cannot find field.

We can filter these fields out, since they aren’t really needed, which lands us with this error:

llm.errors.ModelError: * GenerateContentRequest.tools[0].function_declarations[12].parameters.properties[url].format: only 'enum' and 'date-time' are supported for STRING type

A problem which seems to go beyond just Claude Code, and would require us to modify which tools we enable, which is something we don’t control using Claude Code.

It was clearly hubris to think I could backdoor my way into solving this problem when folks are working on it full time.

What no one is talking about is the fact that as soon as a new model drops, the agent harnesses aren't optimized for it.

There's a ridiculous amount of elbow grease that you have to put in to actually make new models work truly well with agents.

We're working really hard at…
— Ryan Carson (@ryancarson) August 9, 2025

But now at least Ryan can’t say people aren’t talking about it.

And this is to say nothing of the challenges of rewriting the prompts and tool descriptions to make the agent itself function as intended once the API calls are working.

All this said, I think I will still implement a more narrow implementation of this project for OpenAI and Anthropic APIs and add image support.

Thanks for reading via RSS!

2025-08-12

Tue, 12 Aug 2025 22:18:04 GMT

Today, I published a package that exposes an OpenAI-compatible API on top of Simon’s llm tool. It’s been interesting to develop a better feel for the least common denominator of features and what it looks like to try and support them well.

My goal is to use the plugin architecture of the llm tool to build a distributed version of something like litellm. Most users don’t need most of what litellm offers but have no way to avoid installing everything. Additionally, if you want to support a new provider, you can independently implement a plugin and install it within the llm tool without needing to adopt a large framework, upstream a change to a large repo, or maintain a fork.

It’s been an interesting project. I’m currently trying to add support for Anthropic’s /v1/messages API with the goal of routing Claude Code traffic through different providers and models.

I was also planning to try and support images.

Thanks for reading via RSS!

2025-08-04

Mon, 04 Aug 2025 21:51:02 GMT

Played a bit more with Amp today. It’s been a month or maybe more since I last used it.

It’s beginning to look more like Claude Code.

I liked the experience of working with it, but it still feels like Claude Code works better for me.

Thanks for reading via RSS!

2025-08-03

Sun, 03 Aug 2025 09:41:04 GMT

Attempting to stop the constant WiFi disconnects on macOS 26 Beta 4 by disabling WiFi 6E. This seems to have worked.

Thanks for reading via RSS!

2025-07-29

Tue, 29 Jul 2025 21:11:15 GMT

I made another attempt at using Claude Code to build an LLM-based DJ app, integrated with Spotify. I figure this time, I would use a TUI (like Claude Code and Gemini CLI use) because I hoped it would be more testable. Agent coding tools seem to get much further when they can write tests for themselves.

Thanks for reading via RSS!

2025-07-28

Mon, 28 Jul 2025 18:19:19 GMT

I’ve been trying to find a way to test Claude Code with Kimi K2 running on Groq. In my head, this could be a super fast, capable agent that would make trying larger tasks less time consuming to wait for.

Initially, I thought I could use the LiteLLM Anthropic /v1/messages API, but I couldn’t figure out how to get that to work. Looking at the exposed endpoints of the FastAPI server running the litellm proxy server, that path didn’t appear to be available.

I ended up finding this nice Github project fakerybakery/claude-code-kimi-groq that sets up a web server that does the proxying to Groq and now I am starting to experiment.

Thanks for reading via RSS!

2025-07-26

Sat, 26 Jul 2025 17:55:51 GMT

I updated to beta 4 of macOS 26 and now my WiFi keeps disconnecting. Not sure if it’s related but the timing is suspicious.

Thanks for reading via RSS!

2025-07-25

Fri, 25 Jul 2025 23:46:50 GMT

No one survives being a professional programmer without being willing to learn a whole bunch of new things every couple of years. The people who don’t flee into management as soon as they can.

— https://www.recurse.com/blog/191-developing-our-position-on-ai

Thanks for reading via RSS!

2025-07-11

Fri, 11 Jul 2025 09:28:09 GMT

Thank you Scott, for helping me find a way to get my Astro site’s theme to stop flickering.

Thanks for reading via RSS!

2025-07-06

Sun, 06 Jul 2025 12:49:56 GMT

Great read from Indragie on building a Swift macOS app. A lot of their journey echoes my experience building Tomo.

Also was introduced to several projects and repos I hadn’t heard of before

Thanks for reading via RSS!

2025-07-04

Fri, 04 Jul 2025 14:42:26 GMT

I spent a few hours DJing with Sonnet 4. All manually, via conversation. I gave it a playlist of songs I liked and asked it to keep with the vibe. I would tell it I skipped if I didn’t like its suggestion.

It went pretty well for a while.

It took a 4 song playlist to 26 songs that I liked. I skipped a bunch of songs along the way and started the conversation over a few times while experimenting with different prompt approaches.

A few problems and challenges:

Starting with the same playlist meant the suggestions at the beginning were generally the same. I had no good way to keep a list of songs I didn’t want suggested but that I also didn’t add. I could see myself building out something to support this.
Deeper into the context window the suggested seems to get a bit worse. The model also started recommending songs, then realizing it was repeating or had the wrong genre, then course correcting.
Also deeper into the context window the model started to make up songs or mix up song titles and artists.

Overall, this was a fun experiment. I learned about several new artists I hadn’t heard before and was reminded of some others I hadn’t whose catalogs I hadn’t visited in a while.

Thanks for reading via RSS!

2025-07-01

Tue, 01 Jul 2025 21:17:18 GMT

The agent CLI space is getting pretty crowded.

Claude Code and Cursor are typically my daily drivers, but I played around with Sourcegraph’s amp and Google’s gemini-cli.

In particular, I noticed that gemini-cli has been getting a lot of attention.

If I had to guess, Amp seems to use Anthropic’s text editor tool which is quite nice, and what today’s best agents seem to consistently be built on top of. It seems unlikely gemini-cli would use this tool as it uses Google’s Gemini models.

These tools are all copying from each other, liberally. Like Claude Code, gemini-cli uses LLM-generated phrases while it runs its LLM calls in the background to do work for you.

Amp feels more minimal whereas Gemini CLI feels an attempt at an outright Claude Code clone. Having used Gemini 2.5 Pro for several weeks just before Sonnet and Opus 4 came out, this makes sense. Gemini is a great model for writing code and deserves a dedicated tool.

It also seems Google has gotten some very wide distribution with the launch.

It would be nice to see standardization on a file name for agent rules. agent.md seems like a fine solution for this. This agent rules format seems like yet another dimension on the MxN problem of everything in models these days — supporting everything eventually becomes an entire project or business (see OpenRouter, LiteLLM).

Hopefully, we see some consolidation here soon or we can just create a bunch of symlinks. Amp seems to try to be compatible will all existing agent files and import from them. I’ve not seem much commitment to this from other tools.

Notably, it seems only Cursor has deviated from the standard markdown format They use .mdc which has frontmatter to instruct the agent when to attach the rules.

Thanks for reading via RSS!

2025-06-25

Wed, 25 Jun 2025 17:17:29 GMT

I’ve said it before and I’ll say it again:

If you attempt to market to me using your app’s push notifications, I will disable all notifications and consider it a reminder for whether I use your app enough to justify not uninstalling it.

TIL you can run Claude model inference on GCP.

Thanks for reading via RSS!

2025-06-19

Thu, 19 Jun 2025 19:48:42 GMT

Been getting back to writing more after a period where I was very focused on building. I missed it and feel like I have a bunch of ideas floating around that I want to write about.

I upgraded Hugo today. It was mostly painless. Apparently, the --verbose flag was removed and the Twitter shortcode was renamed to X. I had to change my deploy script as a result of the former.

I also added the very nice hugo-admonitions project to this site.

Thanks for reading via RSS!

2025-06-08

Sun, 08 Jun 2025 16:36:32 GMT

I’ve been reading a lot lately about making it clearer what prompts are used to elicit certain outputs when coding with agents.

Cloudflare’s OAuth library, written largely by LLM agents with human review, includes the prompts in the PR comments, making it much easier to figure out the author’s intent and imagine how you might prompt an agent to build something similar yourself.

I feel there needs to be a better way to automatically commit after each agent loop is executed, including the prompt as part of that commit. From there, we can fluidly move back and forth between commits as necessary, accepting or tossing the resulting output from the agent.

When working on a feature, you create a new branch, then prompt the agent as many times as needed, and the agent makes a commit after each loop is finished. Once you’re satisfied with the results, you can squash and merge, preserving all the prompts, or merge everything to main if you prefer.

Thanks for reading via RSS!

2025-06-06

Fri, 06 Jun 2025 20:33:16 GMT

Today I learned the quote

The people who are crazy enough to think they can change the world are the ones who do

which is typically attributed to Steve Jobs was likely written by advertising copywriters at TBWA\Chiat\Day (the agency that created the campaign) rather than by Jobs himself^[1].

There were a ton of Cursor issues today. I also saw 429s all day on Bedrock in us-west-2.

I know Cursor just launched their 1.0 and I also heard Anthropic cut Windsurf’s access to Claude models and forced them to scramble to get Claude capacity via 3rd parties.

I wonder if Cursor’s and AWS’s problems were the thundering herd of developers coming to them after Anthropic cutoff and Windsurf’s access post-OpenAI acquisition.

And as I write these words, I just received an email at 6pm PT from Cursor about their 1.0 launch.

Not exactly hype if you are burying the launch email at dinner on a Friday.

This is all speculation — I have no inside information.

https://www.forbes.com/sites/onmarketing/2011/12/14/the-real-story-behind-apples-think-different-campaign/ ↩

Thanks for reading via RSS!

2025-05-28

Wed, 28 May 2025 19:40:24 GMT

I’m working on a macOS app that I use daily. However, I also to development on it pretty much daily. It requires several macOS permissions. However, I was running into issues where the app had the same name but couldn’t share permissions. To switch between the two, I had to remove and re-add the permissions.

I gave Claude Sonnet 4 my problem and it walked my through to a solution without a hitch. I gave it screenshots as I made progress and it gave me the next steps or considerations to make if there were several possible paths forward.

Now my Xcode build and production builds of the app can share the same permission and everything works smoothly.

This took maybe 20 minutes. I’m very happy with the results.

Thanks for reading via RSS!

2025-05-25

Sun, 25 May 2025 15:26:50 GMT

I setup a Cloudflare tunnel so I can serve local sites on a public domain for easy testing on a real mobile device.

Thanks for reading via RSS!

2025-05-23

Fri, 23 May 2025 09:45:03 GMT

I’ve been keeping screenshots of projects as I’ve been working on them. Looking back through the progress is gratifying and motivating.

Thanks for reading via RSS!

2025-05-12

Mon, 12 May 2025 18:25:40 GMT

I built a RSS/JSON feed on top of Apple Notes. I thought it would be a convenient way to programmatically read and create actions based off the content in Notes. I think the Python wrapper on top of AppleScript that I was using ended up being a bit too slow for this to be practical. It was taking ~19s to load a feed of ~70 notes.

Thanks for reading via RSS!

2025-05-09

Fri, 09 May 2025 16:26:24 GMT

Chatting with a friend today and we were discussing easily deployable web apps, not tied to any particular build tool or framework. Aspirationally, just using a simple vanilla JS, HTML, and CSS stack.

LLMs are good at building with this stack and deploying them is a matter of hosting them as static assets. Generally, we don’t build with this stack because as things get more complicated, the codebase struggles to scale to the needs of the project. Or at least this is the conventional wisdom.

So, for me what came to mind was a random repo I had seen within the last few months showing a sort of anti-framework approach to writing vanilla Javascript to build a web app. But I looked through all my browser history and bookmark folders and couldn’t find it. I tried Perplexity, I tried ChatGPT’s search, I tried Google, I tried DuckDuckGo. I could not find it - and I generally consider myself pretty good at finding things around the web.

And then I tried Exa. I had initially misremembered the repo somehow involving Elm inspiration, but when I ran the following query, I actually found it!

no dependency vanilla javascript approach to building webapps

Search link

The repo in question was this one: https://github.com/matthewp/views-the-hard-way

I tried to make a 4 player pong game with this approach and Sonnet. Then again with Gemini 2.5 Pro, which one-shotted it, but didn’t really use the approach and opted to use a canvas.

We should be open sourcing our system prompts in a more consistent way. system_prompts, llm_templates — it feels like there is something here. Maybe more personal than fabric. But also, maybe I need to use fabric more…

But the reason I was thinking this:

{
  "permissions": {
    "allow": ["Bash(mkdir:*)", "Bash(ls:*)"],
    "deny": []
  }
}

I’m going to need to bring these (or whatever equivalent is for whatever agent I am using) around with me. I need to have different versions of them for different types of systems and, of course, for different projects.

I haven’t read any spec of this file, but let’s note the apparent structure:

Bash(ls:*)

I’ve allowlisted all inputs to the Bash tool whose commands are ls and mkdir and match *.

Thanks for reading via RSS!

2025-05-08

Thu, 08 May 2025 22:04:08 GMT

I definitely felt I think writing was a much more angsty process for me in the past. Maybe I’ve just gotten better at it over time or just found my own coping skills. I wouldn’t attribute all this to LLMs, but I think having a partner now, it’s like I actually don’t feel like I’m solitary anymore.

When I’m getting stuck on stuff, I actually have someone I can talk to and whatever I throw at them, I don’t have to give them all the content. I can just throw things and we go back and forth. Just that of not being a solo writer anymore, but having a collaborator is I think a really huge change for a lot of people.

— AI and I: Jhana Meditation Silenced Her Mind—And Changed Her View On AI | Nadia Asparouhova, May 7, 2025

Source

So how much faster are your features shipping because you’re using these tools. So that might be something like the time between first commit and when your PR is merged.

It’s very tricky to get right. But one of the ones that we’re targeting, the other one that we want to measure more rigorously is the number of features that you wouldn’t have otherwise built. We have a lot of channels where we get customer feedback.

And one of the patterns that we’ve seen with Claude Code is that sometimes customer support or customer success will post, hey, this app has this bug. And then sometimes 10 minutes later, one of the engineers on that team will be like, Claude Code made a fix for it. And a lot of the situations when you ping them and you’re like, hey, that was really cool, they were like, yeah, without Claude Code, I probably wouldn’t have done that because it would have been too much of a divergence from what else otherwise going to do.

— Latent Space: The AI Engineer Podcast: Claude Code: Anthropic’s CLI Agent, May 7, 2025

Source

Thanks for reading via RSS!

2025-04-27

Mon, 28 Apr 2025 13:28:25 GMT

We need to understand that things that were considered easy tasks will no longer exist. And what was considered hard tasks will be the new easy.

And what was considered to be impossible tasks are going to be the new hard.

From The Cognitive Revolution | AI Builders, Researchers, and Live Player Analysis: Fiverr Goes All-In on AI: Empowering Creators, Not Replacing Them, with Micha Kaufman, CEO of Fiverr, Apr 26, 2025

What have you observed in terms of AI impact on the platform so far? You know, it’s kind of like asking a captain at mid-sea, how is he experiencing the storm?

From The Cognitive Revolution | AI Builders, Researchers, and Live Player Analysis: Fiverr Goes All-In on AI: Empowering Creators, Not Replacing Them, with Micha Kaufman, CEO of Fiverr, Apr 26, 2025

Thanks for reading via RSS!

2025-04-26

Sat, 26 Apr 2025 19:43:03 GMT

I’ve said this before, but as the feature set of LLM APIs increases, it becomes harder and harder to not just pick a provider and using their SDK for your use case. Between tool calling, thinking, streaming, citations, pdf parsing, and more, there is no consistency in the API design across providers.

This feels like OpenRouter’s moat. They figure out a way to abstract this as much as possible across model APIs and then I just use them. LiteLLM as well though I need experiment with it more.

Thanks for reading via RSS!

2025-04-23

Wed, 23 Apr 2025 08:39:47 GMT

When building with agents, prompting the agent to

Only do X. Do nothing else.

seems to be required to prevent the agent from being overly eager with its implementation, trying to anticipate what you will want next. However, the agents are often not reliable enough to do all this work in a single tool loop, so forcing them to keep their scope small is still necessary babysitting.

Thanks for reading via RSS!

2025-04-19

Sat, 19 Apr 2025 11:27:00 GMT

It’s hard to make some simple shit that’s great

Harry Mack - OMA: A Path Of Discovery

Thanks for reading via RSS!

2025-04-18

Fri, 18 Apr 2025 18:35:30 GMT

Writing software is weird now. Most engineers I know are using LLM-assisted coding tools in some capacity. We can debate the pros and cons of whether this practice is a good idea and the impact it’s going to have on us and the software create, but for now, it is a reality.

The hard parts remain

The parts of software engineering that have gone away are the straightforward parts. The parts where you scaffold a new CRUD endpoint, add a new database table, or write a mapper between an external and internal representation of some data. The LLM is usually done with about as fast and you can point it at the problem. So what does that leave?

The hard parts. The time consuming, grueling bugs. The deep dives into memory leaks. Really any and everything that the LLM can’t one shot or that you can’t figure out a way to decompose into a series of tasks it can one shot.

Now that LLMs make so many things easy in software engineering, the hard and confusing parts are the only parts that are left for us humans to do.

I’ve found this shift to be exhilarating. It’s also exhausting. I now spend almost no time on problems I know how to solve.

If I know what I am doing, I usually can describe to an LLM what I need it to do and it does it. This means I now spend nearly all of my time somehow out of my depth, stretched.

I’ve never experimented so much, tried so many new ideas, or built so many toy projects. I’ve written more prose and software in each of past 2 years individually than the prior 9 combined, mostly just because I’ve done more things, which made me think, which made me write which made me think more.

What I miss

Given all this experimentation, I think my one complaint is that I haven’t built up much new competence in any particular programming language. I’ve become more familiar with frameworks like Next.js, Remix/React Router, Electron and Xcode, but I wouldn’t say I’m particularly comfortable writing Typescript or Swift, especially without an LLM.

Lately, I’ve worked a lot in Swift, using a combination of Cursor, Windsurf, and Claude Code to build my project. I’ve barely touched code at all. I’m deeply involved in the project structure. I’ve throw out 90%+ of the code the models have generated. I’ve glanced at a lot of Swift code.

I know Swift imports augment the namespace. That HStack, VStack, and ZStack are the primitives of SwiftUI layouts. I know how to add external dependencies to a Swift project and use them. But I definitely don’t know Swift. I’ve hardly written a line of Swift.

I haven’t needed to write Swift to accomplish my aims with this project, but I would like to know Swift and I think a better understanding of Swift would help me do a better job building this project.

But one only has so much time. For now, I’d rather work on my ideas than dedicate to learning Swift.

Thanks for reading via RSS!

2025-04-14

Mon, 14 Apr 2025 19:14:13 GMT

It seems OpenAI’s naming scheme is now following Apple’s iPod naming scheme

Today, we’re launching three new models in the API: GPT‑4.1, GPT‑4.1 mini, and GPT‑4.1 nano.

https://openai.com/index/gpt-4-1/

The OpenAI playground has been down for me all day with a Cloudflare error.

Thanks for reading via RSS!

2025-04-06

Sun, 06 Apr 2025 10:14:17 GMT

I am working on a macOS app, built in Swift and called Hudson. I’ve been inspired by the UX of Chorus and the simplicity of Ollamac. My goal is to build an app that allows me to quickly send prompts to API or local LLMs. I have most of the core functionality working. What’s next is to build something like Chorus’ ambient chat UI.

Having an LLM prompt available via a global hotkey across any model and where my conversation history is stored locally feels like it will address several of my needs. It’s not that a similar app doesn’t exist, it’s that I know I will want to tweak the UX exactly to my liking. With LLMs being so good at generating code, this is now a project that will take a week or two rather than a month or two.

Thanks for reading via RSS!

2025-03-30

Sun, 30 Mar 2025 00:41:10 GMT

It’s been a little while since I worked at all on Delta but I still have the itch to have access to the source of the tool that becomes my daily LLM driver. I recently setup Chorus which advertises itself as a tool to chat with multiple models at once. This isn’t really a main use case for me but besides not being open source, it’s basically everything I want.

Clean interface
Text streaming
Use my own API keys
Local model support (ollama, LM Studio)
Broad model API support
Mini-mode triggered by a hotkey
Conversation history
Image support

I don’t love that their cloud is involved with the service. I understand this is required for them to make money but I can’t see myself using a cloud product for this in the long term, especially for sending images of my system. I feel like there is too much potential for things to go wrong.

Nevertheless, this is great work in my opinion.

Apparently, I’ve opened up pandora’s box. There are so many tool like this with a variety of feature sets and tons of overlap. I’ll need to wade through and see what I can find.

I think I may need to build this for myself.

Thanks for reading via RSS!

2025-03-25

Tue, 25 Mar 2025 20:54:39 GMT

OpenAI shipped a new version of 4o that can generate images. I tried generating images in ChatGPT and it popped open a side chat and edit tool I hadn’t seen before. I got rate limited before I could try much.

I read this paper on using “chain of draft”, a more compressed version of chain of thought to achieve similar results to chain of thought reasoning. The main unlock with this approach is a reduction in inference latency and cost due to few tokens needed to achieve similar performance.

Thanks for reading via RSS!

2025-03-15

Sat, 15 Mar 2025 15:16:14 GMT

I mostly browse social media apps like LinkedIn and Bluesky on my phone. Recently, LinkedIn started surfacing a popup prompting me to download the app after a bit of scrolling the browser. This prompts me to close LinkedIn entirely. I’m curious if my behavior is unusual, or if they’re getting enough conversions to app downloads for it to be worth it. It’s certainly caused me to use LinkedIn less which I am guessing is not the point.

Thanks for reading via RSS!

2025-03-09

Sun, 09 Mar 2025 16:45:10 GMT

So much of the world is outside of what can be specified. https://podcasts.apple.com/us/podcast/ai-and-i/id1719789201?i=1000696284548

Thanks for reading via RSS!

2025-03-07

Fri, 07 Mar 2025 19:01:04 GMT

I feel like .cursorrules are finally starting to snap into place for me. I’ve made many messes in Cursor, but once I exhaust what is possible without structure, if I want to continue, I need to create a structure. There’s nothing like a painful refactor to reinforce how and why you should define conventions for your codebase. Usually, when diving into a new idea, I don’t love to think about this stuff, but it’s always time well spent. With this in mind, my plan is to try and build rules and structure as I go rather than pushing my projects to the limit then needing to clean up.

Thanks for reading via RSS!

2025-03-05

Wed, 05 Mar 2025 21:23:48 GMT

Stumbled upon the react-three-fiber library today and now I am building a game where a lander can fly around a mini-solar system.

Thanks for reading via RSS!

2025-03-04

Tue, 04 Mar 2025 18:44:46 GMT

Learned more about the post-training phase of fine-tuning LLMs and how the model initially goes through a pre-training phase. From there, it is fine-tuned to contribute to a token stream with a human user, using prompt tokens to demarcate whether a message was written by the user or the assistant.

For example

<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>

Finally, labs have continued to improve model benchmark performance using further fine-tuning, like RLHF, where humans pick the best of a set of responses from the model, and the model is further fine-tuned on this data.

Progress is slow, but I feel like I am finally beginning to develop more of a mental model of what is happening in model training. When I trained my own language model on the posts from the blog, I understood that I was training a completion model but fully appreciate the additional steps I would need to shape that “base” model into a chat model myself. Now, I feel I have a better understanding of how that process works.

This brings me back to a question I have been asking for a while: what happened to the completion models? Why do I have to use a model fine-tuned on <|im_start|> and <|im_end|> tokens?

Thanks for reading via RSS!

2025-03-02

Sun, 02 Mar 2025 17:43:14 GMT

I found an interesting library for building 3D games specifically “built for Cursor” called viber3d. I assume the name is a reference to “vibe coding”. This is the first library I have seen ship a starter scaffold with Cursor rules. This is an interesting development in how frameworks are being built now. Since language models don’t know about brand new frameworks, frameworks are shipping with content that will aid language models in using them, since coding with models seems to be an increasingly popular way to code.

Hopefully, approaches like this can be part of how we prevent AI from stifling tech adoption.

Thanks for reading via RSS!

2025-02-28

Fri, 28 Feb 2025 17:59:00 GMT

I’ve built a few prototypes with the OpenAI voice to text API with code largely written using Cursor. This has been fast and easy to incorporate into Next.js apps. I can add an audio-recording-to-text feature to any app in a couple of minutes, ready for use in a production environment.

There are several other options for voice to text as well. MacOS has a built-in voice to text feature and there are several other Whisper wrappers available, some that can run locally. Talking is much faster than typing and allows me to capture raw thoughts faster, which I can then refine later. LLMs are also quite good at structuring these raw thoughts into a more refined form that I can then edit.

This is not to replace writing, but actually to make it easier for me to write more frequently. There are lots of times when I have an idea that I want to capture and return to later. The more I can capture when it’s top of mind, the more likely I can come back later and do something with it.

Thanks for reading via RSS!

2025-02-22

Sat, 22 Feb 2025 21:53:56 GMT

I’d like to see authors being surprised by what readers end up learning from their material. Because the author is not just sending out something static. They’re sending out a program which is capable of emergent behavior. So the reader will be able to try out different things and discover things the author hadn’t intended.

Bret Victor, The Humane Representation of Thought, Oct 2014

Thanks for reading via RSS!

2025-02-20

Thu, 20 Feb 2025 21:04:24 GMT

I’ve been playing around with this idea I am calling “idea projection”. The concept is that you can take raw ideas, logs or notes (like all the logs on my site, for example) and create projections of them into different forms using a model and a target structure.

The prompting approach looks something like this

<files>
{files_xml}
</files>

<structure>
{structure}
</structure>

<instructions>
Given the above files, your job is to create a new document that structures the contents of the files in adherence with the <structure>, maintaining the original voice and phrasing as much as possible.

Output the complete document content directly, without any replacement blocks.
</instructions>

What this allows you to do is give the model a ton of files and have it transform that input content into the target structure. This approach can work for everything from

Summarize the main areas of focus of the logs by month

List topics, areas and disciplines I haven’t written about but might find interesting

Another way to think about “projections” is as indexes, like in a database. They provide another way to look at the same data, and they can and should be updated whenever new raw data is added.

With this approach, you can update and maintain many different projections of the same source of truth data. You could generate a PRD, executive summary and a list of new ideas not included in the source data, all from the same data. When updates are made to the source data, run the inferences again to generate new instances of the projections or “indexes”.

Thanks for reading via RSS!

2025-02-18

Tue, 18 Feb 2025 19:40:34 GMT

A great read by Harper about writing code with LLMs.

One passage that particularly resonated with me

When I describe this process to people I say “you have to aggressively keep track of what’s going on because you can easily get ahead of yourself.”

For some reason I say “over my skies” a lot when talking about LLMs. I don’t know why. It resonates with me. Maybe it’s because it is beautiful smooth powder skiing, and then all of a sudden you are like “WHAT THE FUCK IS GOING ON!,” and are completely lost and suddenly fall off a cliff.

Thanks for reading via RSS!

2025-02-17

Mon, 17 Feb 2025 21:14:44 GMT

I’ve long been a skeptic but I think goose may be one of the best LLM tools for coding.

Thanks for reading via RSS!

2025-02-14

Fri, 14 Feb 2025 17:19:28 GMT

I read Declan’s article on new technology adoption and the problems posed by models.

I haven’t had as much trouble as it seems Declan has in getting models to write vanilla js, but I share the concern about new technology/framework adoption and how we’ll solve this problem if models end up authoring most of the code that gets written.

Maybe projects like llms.txt^[1] and similar can help by providing context to models on how to use a technology, but finding a way to do this is now a meaningful hurdle to using something new and the slop filled web seems to be making it harder to push forward the knowledge cutoff of the foundational models.

With regard to React/Tailwind specifically, I’m actually unsure if labs’ preference for this stack is a result of models being good at wielding it due to their training or that models themselves (without a more complex architecture) are just better at writing and styling frontends with the structure and the style colocated in the same parts of the file as compared to different parts (e.g. css at the top, HTML/[T|J]SX and style class names at the bottom) or even two different files. I wrote about this as well ^[2].

Tools like Cursor’s Composer agent are improving at multi-file edits, so if this limitation is genuine, it seems like it won’t remain long-lived.

Thanks for reading via RSS!

2025-02-13

Thu, 13 Feb 2025 17:57:57 GMT

Two, random interesting learnings from Julia’s survey results:

The first

ctrl + l clears the terminal. I used to rely on cmd + k for this but when using the embedded terminal in Cursor, it opens a completion panel so I had been briefly stuck typing clear.

The second

history 0

which gets the all time shell command history. This is useful for history 0 | grep <query>. I use fzf and ctrl + r to search through the history typically but this is still nice to know. This had be an edge case I never quite resolved in my switch to zsh. I was used to getting full history in bash with history.

Thanks for reading via RSS!

2025-02-12

Wed, 12 Feb 2025 21:11:50 GMT

I’ve been doing a lot of vibe coding lately (though I only recently learned there was a term for this). My most recent projects include a Krea-like image editor. I am currently working on adding inpainting using a mask and prompt. I am also developing a tool that captures raw thoughts and processes them through an LLM, following some structure to summarize, deduplicate and consolidate. The general goal is to synthesize and organize ideas for a project, article, or just to help with understanding where my ideas are leading.

Thanks for reading via RSS!

2025-02-03

Mon, 03 Feb 2025 18:07:33 GMT

After more thinking than I really expected I would do about such a thing, I open-sourced Delta today. I still really like the concept of conversation branching for ideation, thought partnership and rabbit holing with LLMs. There are several folks who are working on closed-source, cloud-hosted versions of similar ideas. I don’t have the bandwidth to pursue the project further right now, but if I can empower someone with the time and interest, that would be a win.

Thanks for reading via RSS!

2025-02-01

Sat, 01 Feb 2025 14:03:17 GMT

I’ve been working on and off on a very simple, Krea-inspired image editor in the browser that can turn sketches into images using Flux Canny. I’ve tried a few separate times to prompt models (Sonnet, o1, r1) to add an undo-redo stack to my project. All have failed. It’s possible I am under specifying the ask but interesting to identify this as a particular area where models don’t seem to do very well.

I’ve continued pursing building this feature with a model more out of interest at this point, since I don’t really need it. It provides a good test for new models when they are released (or maybe my code is irreparably broken). I don’t have desire to debug the problem so maybe I’ll find out another time.

Thanks for reading via RSS!

2025-01-29

Wed, 29 Jan 2025 20:33:21 GMT

I spent some time building an iOS app today, all with Cursor. I’ve never really written Swift before (well apparently I tried it a year ago). Since this was pre-Sonnet, I probably wasn’t doing a ton of code generation at the time. It will be interesting to see how far I can get.

I worked through a bit of setup to mount an R2 bucket to the local filesystem with rclone but haven’t taken the final step to modify my system security permissions yet.

I one-shotted a simple post recommendation section for this site with Sonnet/Cursor.

Thanks for reading via RSS!

2025-01-22

Wed, 22 Jan 2025 17:36:32 GMT

Played through David Peter’s fun code puzzle: https://shark.fish/curiosity/

Thanks for reading via RSS!

2025-01-21

Tue, 21 Jan 2025 20:55:01 GMT

The agricultural revolution certainly enlarged the sum total of food at this disposal of humankind, but the extra food did not translate into better diet or more leisure. Rather, it translated into population explosions and pampered elites.

Yuval Noah Harari, Sapiens

Thanks for reading via RSS!

2025-01-17

Fri, 17 Jan 2025 19:25:09 GMT

I spent some time looking for low latency, image-to-image APIs. I look around a fair bit and think I’ve settled on together.ai.

My main needs are very similar to Krea

< 1 sec latency
ability to specify the starting a prompt and image

I’m still validating this is the case but latency seems nearly there and I’m still trying to confirm how to make a start image work.

Thanks for reading via RSS!

2025-01-15

Wed, 15 Jan 2025 18:20:47 GMT

I’ve been playing around a bunch with image generation models/tools, notably Recraft and Krea. Recently, I’ve been trying to use these tools to design a logo and favicon for Thought Eddies.

I’ve also been experimenting with using D3 to build visuals for LLM chat conversation branching. It’s harder to work with than React Flow, unsurprisingly since it is a much more general tool/framework. I am aiming to create a few different interactive visuals to showcase a few different ideas I have for working with LLMs on a canvas. Maybe I will end up going back to React Flow if I keep having challenges with D3.

Thanks for reading via RSS!

2025-01-01

Wed, 01 Jan 2025 20:54:58 GMT

I tried two separate ways to configure Cursor to point to an alternative OpenAI compliant API endpoint by modifying the “OpenAI API Key > Override OpenAI Base URL” section of the Cursor settings.

My first attempt was with Deepseek, using learnings from wiring that up to llm. I got to the point where Cursor failed to validate the API endpoint (don’t forget to save the url override), but the curl command it output for me to check manually worked if I switched the model to deepseek-chat.

curl https://api.deepseek.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer <token>" -d '{
  "messages": [
    {
      "role": "system",
      "content": "You are a test assistant."
    },
    {
      "role": "user",
      "content": "Testing. Just say hi and nothing else."
    }
  ],
  "model": "deepseek-chat"
}'

{"id":"c86ef579-61b8-4941-b2be-185385fe690c","object":"chat.completion","created":1735786101,"model":"deepseek-chat","choices":[{"index":0,"message":{"role":"assistant","content":"Hi."},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":18,"completion_tokens":2,"total_tokens":20,"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":18},"system_fingerprint":"fp_f1afce2943"}%

Despite all this working, when I tried chatting with the deepseek-chat model in Cursor, I got this error

{
  "error": {
    "message": "Authentication Fails (no such user)",
    "type": "authentication_error",
    "param": null,
    "code": "invalid_request_error"
  }
}

Next, I tried OpenRouter with a similar approach using the base url https://openrouter.ai/api/v1. Cursor was able to validate the endpoint and I could chat with openai/gpt-4o-mini in the chat without issue. Since I didn’t have credits loaded up to use other OpenRouter models, I did get a “Connection Failed” error from the Cursor chat. Switching to openai/gpt-4o-mini, which OpenRouter provides some free tokens for, fixed this issue.

Deepseek V3 (deepseek-chat) is available through OpenRouter, but it would be nice to understand what is going wrong with my Cursor config here.

OpenRouter also provides some deepseek/deepseek-chat tokens for free, so in the end I was able to run DeepSeek V3 in Cursor, which has been the main driver behind much of these efforts.

Thanks for reading via RSS!

2024-12-19

Thu, 19 Dec 2024 16:58:07 GMT

Dan Corin @danielcorin.com

A day in the mind of Claude Sonnet

December 19, 2024 at 9:21 PM UTC

To create this animation for a day in the mind of Claude Sonnet, I used Sonnet to write the following code

to generate the HTML with this Python script
capture PNGs with puppeteer
use ffmpeg to stitch them together

import llm
import requests
import time
from pathlib import Path
from datetime import datetime, timedelta

MODELS = ["claude-3-5-sonnet-latest"]


def get_temp_weather(hour):
    if hour < 14:
        temp = 30 + (24 * (hour / 14))
    else:
        temp = 54 - (27 * ((hour - 14) / 10))

    if hour < 6:
        weather = "cold"
    elif hour < 18:
        weather = "sunny"
    else:
        weather = "snowing"

    return round(temp), weather


now = str(int(time.time()))
out_dir = Path("out") / now
out_dir.mkdir(parents=True, exist_ok=True)

start_time = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0)
end_time = start_time + timedelta(days=1)
current = start_time

start_time = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0)
specific_times = [
    start_time.replace(hour=2, minute=30),
    start_time.replace(hour=3, minute=0),
    start_time.replace(hour=5, minute=0),
]

for current in specific_times:
    hour = current.hour
    minute = current.minute

    temp, weather = get_temp_weather(hour)
    time_str = current.strftime("%I:%M %p")

    PROMPT = f"""It is {time_str}. {temp}°F and {weather}.
Generate subtle, abstract art using SVG in an HTML page that fills 100% of the browser viewport.
Take inspiration for the style, colors and aesthetic using the current weather and time of day.
Prefer subtle colors but it's ok to use intense colors sparingly.
No talk, no code fences. Code only."""

    timestamp = current.strftime("%H_%M")
    prompt_file = out_dir / f"prompt_{timestamp}.txt"
    prompt_file.write_text(PROMPT)

    for m in MODELS:
        model = llm.get_model(m)
        response = model.prompt(PROMPT, temperature=1.0)
        out_file = out_dir / f"{timestamp}_{m}.html"
        out_file.write_text(response.text())

    current += timedelta(minutes=30)

const puppeteer = require('puppeteer');

async function createGif() {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.setViewport({
    width: 800,
    height: 600,
  });

  const htmlFiles = [
    // paths to html files
  ];

  for (let i = 0; i < htmlFiles.length; i++) {
    await page.goto(`file://${__dirname}/${htmlFiles[i]}`);
    await page.screenshot({
      path: `frame${i}.png`,
    });
  }

  await browser.close();
}

createGif();

ffmpeg -framerate 2 -i timestamped_frame%d.png -c:v libx264 -pix_fmt yuv420p output.mp4

Thanks for reading via RSS!

2024-12-17

Tue, 17 Dec 2024 21:29:34 GMT

I’ve been setting up the foundations to add node summaries to Delta. Ideally, I will use the same model to create the node summaries as I use to generate the responses since this will keep the model dependencies minimal. However, my early experiments have yielded some inconsistency in how a shared prompt behaves across models. To try and understand this and smooth it out as much as possible, I plan to set up evals to ensure the summaries are

within a certain length
don’t include direct references to the user or assistant
are a single sentence or fragment
include the relevant topic(s) discussed

I am currently looking for a straightforward, local option for running evals. I’d like to avoid cloud products and testing across multiple models needs to be straightforward. My ideal output would be a matrix of the different models run and the results of the different eval checks against the respective model outputs given the same system and user prompts.

Thanks for reading via RSS!

2024-12-15

Sun, 15 Dec 2024 17:46:57 GMT

Found this nifty Bluesky Hugo shortcode written by Bryce.

Thanks for reading via RSS!

2024-12-14

Sat, 14 Dec 2024 19:48:02 GMT

Inspired by Simon’s pelican-bicycle repo, I played around a bit with using LLMs to generate visuals with SVGs, Three.js and pure CSS.

I posted about some of the results here.

Dan Corin @danielcorin.com

Inspired by @simonwillison.net's pelican-bicycle repo, here are some CSS sunsets by gpt-4o, claude-3-5-sonnet and gemini-2.0-flash-exp. Sonnet and Gemini animate the birds and clouds. Prompt: Generate pure HTML/CSS art of an extremely detailed, beautiful sunset. No talk or code fences. Code only.

December 15, 2024 at 1:53 AM UTC

I used the code below to generate the HTML/CSS for the images:

import llm
import time
from pathlib import Path

MODELS = [
    "gpt-4o",
    "claude-3-5-sonnet-latest",
    "gemini-2.0-flash-exp",
]

SUBJECT = "an extremely detailed, beautiful sunset"

PROMPT = f"Generate pure HTML/CSS art of {SUBJECT}. No talk or code fences. Code only."


now = str(int(time.time()))
out_dir = Path("out") / now
out_dir.mkdir(parents=True, exist_ok=True)
prompt_file = out_dir / "prompt.txt"
prompt_file.write_text(PROMPT)

for m in MODELS:
    model = llm.get_model(m)
    response = model.prompt(PROMPT)
    out_file = out_dir / f"{m}.html"
    out_file.write_text(response.text())

Thanks for reading via RSS!

2024-12-12

Thu, 12 Dec 2024 20:12:05 GMT

Got Delta working on other machines. It took a lot longer than I expected. I spent most of the time dealing with build issues regarding:

missing dependencies
different paths between dev and production
loading vec.dylib with sqlite-vec
dependencies compiled for the wrong architecture

I tried to write some of this up but it’s been challenging to extract and/or remember the specific circumstances and how I solved it in the context of some minimal example. There are lots of parts that feel a bit wrong or regrettable but were compromises to getting the thing working.

Specifically

whatever it was I built to get the build working
using Javascript (not Typescript) for migrations (needed so the migrations could run at runtime in the context of the main.js)

Thanks for reading via RSS!

2024-12-07

Sat, 07 Dec 2024 12:29:37 GMT

I don’t know anything about rice disease but apparently these are various rice diseases and this is what they look like.

Jeremy Howard, Fast.ai Course Lesson 6 I have no idea if Jeremy had this in mind when he said this (alluding to the fact he doesn’t know about the subject area, but when building with ML that doesn’t necessarily matter), but this sentiment is how I feel when building with and learning about anything new with language models, at least to start. I use LLMs to bootstrap my understanding of whatever situation I find myself in, and from there, try to orient myself and gain a better understanding.

It’s incredible to be able to learn from whatever your starting point is rather than needing to try and read less relevant introductory material to start to understand an area. It’s akin to having a CSV and wanting to run SQL queries on your data and needing to start by reading the pandas or sqlite documentation.

Most of the effort is finding what you’re looking for to solve the problem in front of you. LLMs bring this directly to you given your ability to describe what you need.

No other resource is tailored to help you solve problems like this. It’s an order of magnitude better than anything else I have tried with regard to learning and applying existing knowledge.

Thanks for reading via RSS!

2024-12-05

Thu, 05 Dec 2024 17:48:06 GMT

The developer space is moving so fast. I went from thinking Cursor’s cmd+k was the greatest thing, to using Windsurf’s Cascade, the new Cursor composer and bolt.new in a week. Powerful tools.

Thanks for reading via RSS!

2024-12-01

Sun, 01 Dec 2024 21:03:27 GMT

I’m working on a conversation branching tool called “Delta” (for now). The first thing that led me to this idea came from chatting with Llama 3.2 and experimenting with different system prompts. I was actually trying to build myself a local version of an app I’ve been fascinated by called Dot.

I noticed that as conversations with models progressed, they became more interesting. A friend made a point that really stuck in my head about how you “build trust” with a model over multiple conversation turns. While you can write system prompts to steer the model to respond with more details and longer paragraphs, I observed that regardless of the system prompt, as conversations went on longer (more turns, more exchanges, longer message history), the responses became more interesting and better calibrated to what I was looking for. The model seemed to display a more coherent understanding of what I was talking about.

This led me to ask: could I use that message history as an augmented system prompt? For example, if I have a conversation with six messages (three turns), could I add several different fourth turns to that conversation and see what happens? Delta was created to let you build a conversation and then branch it from different points.

The project was also partly inspired by Simon Willison’s llm CLI tool, which uses a local SQLite database to save all LLM conversation history. I took inspiration from this idea and augmented it to create pointers to previous messages, which form the tree structure used in Delta. Lastly, I wanted to be able to experiment with multiple different models - both local models and with APIs. Using multiple models in Delta to try the same prompts helps me develop an intuition for the quality and types of responses I get from different models and provides a helpful playground for these experiments.

Thanks for reading via RSS!

2024-11-20

Wed, 20 Nov 2024 21:55:29 GMT

Tried out Letta. Unsure where to try and go with it.

Thanks for reading via RSS!

2024-11-16

Sat, 16 Nov 2024 12:36:36 GMT

Would another day of editing fundamentally change the value readers get? Probably not. Ship it and move on to your next idea while you’re still energized.

Thanks for reading via RSS!

2024-11-15

Fri, 15 Nov 2024 22:40:24 GMT

Trying out Windsurf

Thanks for reading via RSS!

2024-11-10

Sun, 10 Nov 2024 14:29:38 GMT

When you are curious about something, you have the right cocktail of neurotransmitters present to make that information stick. If you get the answer to something in the context of your curiosity, then it’s going to stay with you.

David Eagleman, https://freakonomics.com/podcast/feeling-sound-and-hearing-color/

Thanks for reading via RSS!

2024-11-09

Sat, 09 Nov 2024 16:40:35 GMT

we can improve the accuracy of nearly any kind of machine learning algorithm by training it multiple times, each time on a different random subset of the data, and averaging its predictions

Fastbook Chapter 9

Thanks for reading via RSS!

2024-10-30

Wed, 30 Oct 2024 08:44:13 GMT

I wanted to get more hands-on with the language model trained in chapter 12 of the FastAI course, so I got some Google Colab credits and actually ran the training on an A100. It cost about $2.50 and took about 1:40, but generally worked quite well. There was a minor issue with auto-saving the notebook, probably due to my use of this workaround to avoid needing to give Colab full Google Drive access. Regardless, I was still able to train the language model, run sentence completions, then using the fine-tuned language model as an encoder to build a sentiment classifier. Seeing how long this process took, then seeing it work helped me build a bit more intuition about what to expect when training models. I was also a bit surprised how fast the next token prediction and classification inference were. I might try out a smaller fine-tune on my local machine now that I have a better sense of what this process looks like end to end.

Thanks for reading via RSS!

2024-10-29

Tue, 29 Oct 2024 20:08:12 GMT

The following code allowed me to successfully download the IMDB dataset with fastai to a Modal volume:

import os

os.environ["FASTAI_HOME"] = "/data/fastai"

from fastai.text.all import *

app = modal.App("imdb-dataset-train")
vol = modal.Volume.from_name("modal-llm-data", create_if_missing=True)


@app.function(
    gpu="any",
    image=modal.Image.debian_slim().pip_install("fastai"),
    volumes={"/data": vol},
)
def download():
    path = untar_data(URLs.IMDB)
    print(f"Data downloaded to: {path}")
    return path

run with

modal run train.py::download

Next, I tried to run one epoch of training of the language model

@app.function(
    gpu="h100",
    image=modal.Image.debian_slim().pip_install("fastai"),
    volumes={"/data": vol},
    timeout=20 * 60,
)
def train():
    path = untar_data(URLs.IMDB)
    print(f"Training with data from: {path}")
    get_imdb = partial(get_text_files, folders=["train", "test", "unsup"])

    dls_lm = DataBlock(
        blocks=TextBlock.from_folder(path, is_lm=True),
        get_items=get_imdb,
        splitter=RandomSplitter(0.1),
    ).dataloaders(path, path=path, bs=128, seq_len=80)

    print("Sample from datablock:")
    print(dls_lm.show_batch(max_n=2))

    learn = language_model_learner(
        dls_lm, AWD_LSTM, drop_mult=0.3, metrics=[accuracy, Perplexity()]
    ).to_fp16()

    learn.fit_one_cycle(1, 2e-2)
    learn.save("1epoch")

I was waiting around for a long time, having never seen “Sample from datablock:” print. Looking into the volume with the Modal UI, I noticed the /fastai/data/imdb_tok/unsup folder has been modified recently. It seemed like the tokenization of the dataset was talking a long time. I was able to do this tokenization quite fast locally, so I am going to chalk this up to the Modal volume not being as performant as a local file system. While I’m not 100%, I think the need to train with so many little files may undermine my ability to train this model on Modal.

Thanks for reading via RSS!

2024-10-28

Mon, 28 Oct 2024 20:02:58 GMT

I tried training a language model with fastai on Modal.

First I attempted it in a standalone Modal script. I first wrote a script to unpack the data to a volume, then ran the fit_one_cycle function with the learner. I ran into an issue with counter.pkl sort of similar to this issue but I haven’t figured out how to resolve it yet.

On a whim, I checked to see if I could run a Jupyter notebook on Modal. Apparently, you can!

I ran

modal launch jupyter --gpu a10g

then started to mess around a bit. I uploaded the imdb data again then untarred it.

I was noticing occasionally laggy/busy behavior at times that I wasn’t expecting. This experiment ended for the day when I tried to run

learn = language_model_learner(
    dls_lm, AWD_LSTM, drop_mult=0.3, metrics=[accuracy, Perplexity()]
).to_fp16()

I got a SSLCertVerificationError and wasn’t prepared to spend more time debugging.

Thanks for reading via RSS!

2024-10-26

Sat, 26 Oct 2024 16:18:59 GMT

Jon wrote an interesting blog on top of Cloudflare Workers and KV.

I’ve been seeing more and more notebook-like products and UX. A few I’ve seen recently:

Thanks for reading via RSS!

2024-10-21

Mon, 21 Oct 2024 19:12:49 GMT

Ran several experiments using local LLMs (~7b parameter models) like llama3.2 and phi3 to generate a random number between 1 and 100. The exact prompt was

You are a random number generator that provides a number between 1 and 100.

User

Generate a random number between 1 and 100. Provide the output in the following format: ‘Random number: X’, where X is the generated number. Ensure the number is an integer and do not include any additional text or explanations.

I didn’t expect this approach to work as a uniform number generator, but it was interesting to see how it doesn’t work. At lower temperatures, most models only output a few different values in the range of 40-60. There was little to no variability. With increases in temperature (between 1-3), the distribution begins to look bi-model for several models. After this threshold, most model outputs start to breakdown and output only single digit numbers at temperature 7 and higher (I am aware this is general not a recommended model of using a language model).

Thanks for reading via RSS!

2024-10-20

Sun, 20 Oct 2024 18:02:26 GMT

Reading a bunch. Also got inspired to play around with generating random numbers with language models across different temperatures to see their distributions.

Thanks for reading via RSS!

2024-10-16

Wed, 16 Oct 2024 19:30:52 GMT

There is an enormous amount of jargon in deep learning, including terms like rectified linear unit. The vast vast majority of this jargon is no more complicated than can be implemented in a short line of code, as we saw in this example. The reality is that for academics to get their papers published they need to make them sound as impressive and sophisticated as possible. One of the ways that they do that is to introduce jargon. Unfortunately, this has the result that the field ends up becoming far more intimidating and difficult to get into than it should be. You do have to learn the jargon, because otherwise papers and tutorials are not going to mean much to you. But that doesn’t mean you have to find the jargon intimidating. Just remember, when you come across a word or phrase that you haven’t seen before, it will almost certainly turn out to be referring to a very simple concept.

We already know that a single nonlinearity with two linear layers is enough to approximate any function. So why would we use deeper models? The reason is performance. With a deeper model (that is, one with more layers) we do not need to use as many parameters; it turns out that we can use smaller matrices with more layers, and get better results than we would get with larger matrices, and few layers.

https://github.com/fastai/fastbook/blob/master/04_mnist_basics.ipynb

Thanks for reading via RSS!

2024-10-12

Sat, 12 Oct 2024 15:27:38 GMT

Reading fastbook, I get the sense we could teach math more effectively if we did so through spreadsheets and Python code.

Thanks for reading via RSS!

2024-10-11

Fri, 11 Oct 2024 08:42:47 GMT

Current theory on why nbdev and notebooks in general make sense and can work: Writing code for most software is actually pretty similar to writing code for models, but usually you pay less of a cost for not knowing what you’re doing yet (aka exploring). You still pay some cost, but it isn’t totally deal-breaking in most software settings, unlike if you needed to run the entire data load and clean job as you’re training a model. For the latter, a full re-run would slow down the feedback loop too much to still move quickly. Notebooks reduce the feedback loop to about as low as imaginable while also allowing you to experiment while still not totally knowing where you’re going yet. Because you can still go forward(ish), you can sketch in the general direction of where you want to go without as many of the constraints of abstraction, structure, or slow parts of your workflow, which can easily be memoized by the notebook.

Monitoring the token leaderboard on https://openrouter.ai/ is an interesting way to discover new emerging projects.

Thanks for reading via RSS!

2024-10-10

Thu, 10 Oct 2024 19:36:29 GMT

There are many tools for doing evals. I used ell and braintrust together for fun and disaster. The integration is actually not terrible, though I’m not 100% whether they’d be obvious things to try and link together. It seems ell is striving to build its own eval capabilities as well.

Thanks for reading via RSS!

2024-10-09

Wed, 09 Oct 2024 17:33:47 GMT

Some quotes from Lesson 3 of course.fast.ai by Jeremy Howard.

I remember a few years ago when I said something like this in a class somebody on the forum was like “this reminds me of that thing about how to draw an owl”. Jeremy’s basically saying okay step one draw two circles, step two draw the rest of the owl. The thing I find I have a lot of trouble explaining to students is when it comes to deep learning, there’s nothing between these two steps. When you have ReLUs getting added together and gradient descent to optimize the parameters and samples of inputs and of what you want, the computer draws the owl. That’s it.

I pretty much only use resnet18 at the start of a new project because I want to spend all of my time trying things out. I’m going to try different data augmentation, I’m going to try different ways of cleaning the data, I’m going to try to bring in external data, and so I want to be trying lots of things and I want to be able to try it as fast as possible. Trying better architectures is the very last thing that I do.

My very strong opinion is that the vast majority of projects I see in industry wait far too long before they train their first model. You know in my opinion you want to train your first model on day one with whatever CSV files or whatever that you can hack together. And you might be surprised that none of the fancy stuff you’re thinking of doing is necessary because you already have a good enough accuracy for what you need. Or you might find quite the opposite you might find that oh my god we’re basically getting no accuracy at all maybe it’s impossible. These are things you want to know at the start.

Thanks for reading via RSS!

2024-10-08

Tue, 08 Oct 2024 19:39:23 GMT

I tried to use aider to build a crossword generator in Python. Even with a preselected set of words, this proved difficult. Or perhaps the preselected set of words was why it was difficult. Either way, the AI model doesn’t really understand the concept of word overlap in the context of a crossword. That seemed solvable. Instead, I had it write code to precalculate word overlap from the word list, then use that to place the words. The program seemed to hang indefinitely. Adding debug statements revealed extensive looping and attempts to place words that likely couldn’t fit within the constraints of the puzzle. I had selected a group of words around the theme “celestial bodies” without considering their potential for overlap. It was probably impossible to place them all.

I came away from this experience with a deeper appreciation for the complexity of working within the constraints of a crossword. The fact that the AI model struggles in this area seems quite understandable. The tool Crosswyrd further demonstrates how tight these constraints can be. When I placed a few words in various spots within a puzzle template, I quickly discovered that there were only a few very specific options that could fit in certain slots.

Thanks for reading via RSS!

2024-10-07

Mon, 07 Oct 2024 20:10:19 GMT

One of the most painful lessons beginners have to learn is just how often everyone is wrong about everything.

Imagine a spreadsheet where every time you change something you must open a terminal, run the compiler and scan through the cell / value pairs in the printout to see the effects of your change. We wouldn’t put up with UX that appalling in any other tool but somehow that is still the state of the art for programming tools.

Large-scale data collection and testing may end up being the best path to providing helpful feedback.

https://www.scattered-thoughts.net/writing/pain-we-forgot/

Thanks for reading via RSS!

2024-09-27

Fri, 27 Sep 2024 08:46:31 GMT

Erik wrote about how it’s hard to write code for humans.

Getting started is the product!

Found a cool script by David to allow streaming output using glow. Can’t seem to figure out why it strips away the terminal colors for me.

Thanks for reading via RSS!

2024-09-26

Thu, 26 Sep 2024 21:29:20 GMT

Cool article by Jacob on a blog re-write to Astro. I’ve been getting a bit of a re-write itch lately but I don’t want it to be a distraction. Might need to wait until the end of the FastAI course with just a little exploration on the side.

Thanks for reading via RSS!

2024-09-25

Wed, 25 Sep 2024 19:10:08 GMT

Several interesting releases today/recently.

Multi-modal llama: llama3.2. Tons of model infra providers announced availability day one. We seem to be getting into a bit of a rhythm here. It’s also convenient for Meta who doesn’t need to scale the infra (though they of all companies would probably be capable) — providers do it for them.

AllenAI’s Olmo: another interesting, open source multi-modal model.

Open source is catching up in multi-modal. I’m looking forward to experimenting with both of these.

I tried installing llama3.2 with ollama but I don’t think the latest Nix package version can support the multi-modal architecture.

I updated macOS to Sequoia and Nix broke. Now I’m dealing with that since most of my system doesn’t work 🙁

Not a great feeling to have that not work but I found this post which mentioned the command

security find-generic-password -s "Nix Store" -w

I used the password from there to unlock the disk and it worked. Now my dotfiles and things were loading correctly, with the Nix Store unlocked. Unfortunately, this wasn’t all I needed to do. I had to uninstall Nix and cleanup some files following these instructions.

sudo rm /Library/LaunchDaemons/org.nixos.activate-system.plist
sudo launchctl bootout system/org.nixos.activate-system
/nix/nix-installer uninstall
sudo rm /etc/ssl/certs/ca-certificates.crt
curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

Finally, I re-bootstrapped by nix-darwin config with

nix run nix-darwin -- switch --flake ~/.config/nix/flake.nix

and I was back on solid ground.

I tried running nix-darwin again as some point but ran into this issue

nix run nix-darwin -- switch --flake ~/.config/nix/flake.nix
error: cannot connect to socket at '/nix/var/nix/daemon-socket/socket': Connection refused

Loading/restarting the daemons seemed to handle this

sudo launchctl unload /Library/LaunchDaemons/org.nixos.nix-daemon.plist
sudo launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist

My ollama server seems not to be running but that’s a problem for another day.

Thanks for reading via RSS!

2024-09-19

Thu, 19 Sep 2024 20:52:50 GMT

I finally found some time to run a more comprehensive evals of Connections with one guess at a time and using Python code to validate the guesses and give feedback. I ran about 100 puzzles with gpt-4o-mini, gp-4o, and claude-3-5-sonnet, but it became clear that Sonnet was going to perform the best, so I decide to only complete the 466 puzzles released as of today with Sonnet. This wasn’t cheap but it was interesting to see the results. I’m going to write up some more comprehensive findings and push the code soon.

Thanks for reading via RSS!

2024-09-18

Wed, 18 Sep 2024 11:10:00 GMT

Some interesting commentary on the behaviors of founders, managers and leaders written by Rands.

Race’s article on using Jupyter notebooks with Hugo was a helpful intro to the landscape.

Thanks for reading via RSS!

2024-09-13

Fri, 13 Sep 2024 14:58:06 GMT

There have been a number of small-in-scope, but tough problems that I’ve run into that models haven’t been able crack as l’ve presented them via prompting. Usually, these are problems with a few separate areas of complexity, like a recursive parser plus a weird templating language to do it in. o1 is the first model that I can recall that took my high level approach and suggested a simplifying change to the input (tree -F to tree -J -F) that meaningfuly simplified the problem’s complexity (the parser is no longer needed if the input is JSON). With this change and two followups to correct a hallucination, the model output a recursive Hugo template shortcode to render a filetree with collapsible folders.

Thanks for reading via RSS!

2024-09-11

Wed, 11 Sep 2024 18:59:12 GMT

I’m making another, more thorough pass of course.fast.ai, including all notebooks and videos and this time I am going to focus more on the projects. I’ll also be logging a lot more notes as doing so is by far the most effective way that I learn things.

The course materials are very detailed but I’ve still run into some rough edges. The image search for bird vs. forest image classifier didn’t quite work without some modifications to make the search work. Also, the recommended approach for working the textbook notebooks is on Google Colab, which requests a large number of account permissions for accessing my Google account masquerading as “Google Drive for Desktop” and doesn’t make me feel great. I was able to run most of the examples on my personal computer, but training the model for the IMDB movie review classifier was quite slow. I decided it might be worth trying out Colab, since I imagine there could be several more models of this size/complexity I’ll want to train and finding a reasonably fast way to do that will be useful. I went back to the Colab notebook and tried running the cat-or-not classification example. This seemed to take longer than it did on my local machine with an apparent ETA of ~30 minutes.

I probably need to subscribe to Colab if I want this to be performant, but the free tier no longer seems to be a great option. I guess Deep Learning and this course have become more popular than Google (understandably) wants to foot the bill for. If this becomes prohibitive locally, I might subscribe but considering this book is several years old, it’s possible hardware has caught up to handle the examples. I guess I will find out.

Using the handy and recommended doc command, I learned how the magic untar_data command works — mostly, where it downloads the data.

untar_data(url:str, archive:pathlib.Path=None, data:pathlib.Path=None, c_key:str='data', force_download:bool=False, base:str='~/.fastai')

Unsurprisingly, I found all the datasets I’d downloaded so far in ~/.fastai and was also reminded how many dotfiles/folders were in my home directory from software I had tried once.

To put it bluntly, if you’re a senior decision maker in your organization (or you’re advising senior decision makers), the most important takeaway is this: if you ensure that you really understand what test and validation sets are and why they’re important, then you’ll avoid the single biggest source of failures we’ve seen when organizations decide to use AI. For instance, if you’re considering bringing in an external vendor or service, make sure that you hold out some test data that the vendor never gets to see. Then you check their model on your test data, using a metric that you choose based on what actually matters to you in practice, and you decide what level of performance is adequate. (It’s also a good idea for you to try out some simple baseline yourself, so you know what a really simple model can achieve. Often it’ll turn out that your simple model performs just as well as one produced by an external “expert”!)

Thanks for reading via RSS!

2024-09-10

Tue, 10 Sep 2024 21:05:09 GMT

A nice writeup by Eugene on building a simple data viewer webapp with a few different framworks. I am going to need to try out including llm-ctx.txt next time I write FastHTML to see if it helps make the language model better at writing it.

Thanks for reading via RSS!

2024-09-08

Sun, 08 Sep 2024 09:58:06 GMT

I was going to write a quick guide on how to get up and running using Google’s Gemini model via API, since I found it quite straightforward and Twitter is currently dunking on Google for how hard this is. When I tried to retrace my steps, the CSS for the documentation was failing to load with a 503, so I guess this will have to wait until another day.

Thanks for reading via RSS!

2024-09-07

Sat, 07 Sep 2024 14:17:57 GMT

I am continuing to see a lot of buzz about ColPali and Qwen2-VL. I’d like to try these out but haven’t put together enough of the pieces to make sense of it yet. I am also seeing a lot of conversation about how traditional OCR to LLM pipelines will be superseded by these approaches. Based on my experience with VLMs, this seems directionally correct. The overall amount of noise makes it tough to figure out what is worth focusing on and what is real vs. hype.

Thanks for reading via RSS!

2024-09-05

Thu, 05 Sep 2024 19:36:00 GMT

Played around a bit with baml for extraction structured data with a VLM. It’s an interesting approach and has better ergonomics and tooling from most things I’ve tried so far. I like how you can declare test cases in the same place as the object schemas and that there is a built-in playground. I need to see how to handle multi-step pipelines.

I experimented with doing data extraction from pictures of menus. Early results were mixed. I think my photo quality isn’t great and that might be one of the bigger issues.

Thanks for reading via RSS!

2024-09-02

Mon, 02 Sep 2024 09:52:11 GMT

Benchmarking >80 LLMs shows: The best model is not necessarily the best for your programming language 😱

- Best overall: Anthropic’s Sonnet 3.5
- Best for Go: Meta’s Llama 3.1 405B
- Best for Java: OpenAI’s GPT-4 Turbo
- Best for Ruby: OpenAI’s GPT-4o

Good models for one… pic.twitter.com/EYUphEI5rH
— Markus Zimmermann (@zimmskal) September 2, 2024

Great to see more concrete results published on how different models are “the best” at writing different programming languages.

Iterating on Cogno, improving the “remaining guesses” and sharing functionality.

Shipped a very-beta version of a recipe ingredient extraction app, which can then generate a grocery list from many recipes. Calling it snapcart.

Having used fasthtml and sqlite of a couple projects now, I was interested in trying to run an app as a Cloudflare Worker to make use of D1, which is a serverless, sqlite compatible database. It looks like the library needs to be added to the worker runtime first, according to this issue.

Thanks for reading via RSS!

2024-08-31

Sat, 31 Aug 2024 15:53:58 GMT

Language models can’t

generate instructions for knitting patterns
generate crossword puzzles from scatch

Language models can

generate Connections puzzles

Thanks for reading via RSS!

2024-08-29

Thu, 29 Aug 2024 01:14:47 GMT

Incredible read: https://eieio.games/essays/the-secret-in-one-million-checkboxes/

I failed many attempts at getting Sonnet to write code to display the folder structure of the output of a tree -F command using shortcodes. After a lot of prompting, I wrote a mini-design doc on how the feature needed to be implemented and used it as context for Sonnet. I tried several variants of instructions in the design including trying to improve it with the model itself for clarity. I validated that the model could translate from the tree -F to the html markup directly. It could. That in fact is the example html target document in my design doc. Here is that doc:

File Tree Shortcode

This document outlines the requirements for a Hugo shortcode that generates an HTML representation of a file tree structure.

Input

The shortcode takes a multi-line string input representing the output of a tree -F command. For example:

./
├── Makefile
├── __pycache__/
│   ├── clean.cpython-312.pyc
│   ├── ex.cpython-312.pyc
│   ├── ex1.cpython-312.pyc
│   ├── ex2.cpython-312.pyc
│   ├── extract.cpython-312.pyc
│   └── main.cpython-312.pyc
├── ex.py
└── src/
    ├── __pycache__/
    │   ├── clean.cpython-312.pyc
    │   └── extract.cpython-312.pyc
    ├── clean.py
    └── extract.py

Output

The shortcode should generate nested HTML lists representing the file tree structure:

Folders are represented by <li> elements with a <span class="caret"> child containing the folder name.
Files are represented by simple <li> elements.
The content of each folder is contained in a <ul class="nested"> element.
The root folder (’./’ in the example) should be treated as the topmost level.

Requirements

Support arbitrary depth of nested folders.
Use tree command characters (├, │, └, ─ and spaces) from the input to determine the level at which the file or folder should be nested.
Distinguish between folders (ending with ’/’) and files.

Assume the input will be well formed.

Usage

[shortcode begin] ./ ├── Makefile ├── __pycache__/ │ ├── clean.cpython-312.pyc │
├── ex.cpython-312.pyc │ ├── ex1.cpython-312.pyc │ ├── ex2.cpython-312.pyc │ ├──
extract.cpython-312.pyc │ └── main.cpython-312.pyc ├── ex.py └── src/ ├──
__pycache__/ │ ├── clean.cpython-312.pyc │ └── extract.cpython-312.pyc ├──
clean.py └── extract.py [shortcode end]

Here is example html markup that shows how the above example of the shortcode should render.

<ul>
  <li>
    <span class="caret">./</span>
    <ul class="nested">
      <li>Makefile</li>
      <li>
        <span class="caret">__pycache__</span>
        <ul class="nested">
          <li>clean.cpython-312.pyc</li>
          <li>ex.cpython-312.pyc</li>
          <li>ex1.cpython-312.pyc</li>
          <li>ex2.cpython-312.pyc</li>
          <li>extract.cpython-312.pyc</li>
          <li>main.cpython-312.pyc</li>
        </ul>
      </li>
      <li>ex.py</li>
      <li>
        <span class="caret">src</span>
        <ul class="nested">
          <li>
            <span class="caret">__pycache__</span>
            <ul class="nested">
              <li>clean.cpython-312.pyc</li>
              <li>extract.cpython-312.pyc</li>
            </ul>
          </li>
          <li>clean.py</li>
          <li>extract.py</li>
        </ul>
      </li>
    </ul>
  </li>
</ul>

The above tree is depth three since src contains a folder called __pycache__ which contains files. The shortcode should support file trees of arbitrary depth. The output should also trim away the markup outputted by the tree command and just show the file names. If it’s helpful, you can assume a folder will also end with a / in its name.

Here is the output of one such attempt which took several follow to even get it to render. It doesn’t work. I tried for a while and couldn’t get Sonnet to complete this task as described.

{{ $tree := .Inner | replaceRE "^\\./" "" | split "\n" }}
<ul>
    {{ range $index, $line := $tree }}
    {{ $indent := (len (replaceRE "[^│]" "" $line)) }}
    {{ $isDir := (strings.HasSuffix $line "/") }}
    {{ $name := (replaceRE "^[│ ]*├── |^[│ ]*└── |^[│ ]*│ |^[│ ]* " "" $line) }}
    {{ if eq $indent 0 }}
    {{ if $isDir }}
    <li><span class="caret">{{ $name }}</span>
        <ul class="nested">
            {{ else }}
            <li>{{ $name }}</li>
            {{ end }}
            {{ else if $isDir }}
            {{ range $i, $v := seq 1 $indent }}
        </ul>
    </li>
    {{ end }}
    <li><span class="caret">{{ $name }}</span>
        <ul class="nested">
            {{ else }}
            {{ range $i, $v := seq 1 (sub $indent 1) }}
        </ul>
    </li>
    {{ end }}
    <li>{{ $name }}</li>
    {{ end }}
    {{ end }}
    {{ range $i, $v := seq 1 (len $tree) }}
</ul>
</li>
{{ end }}
</ul>

I also prompted Sonnet on whether it could identify any issues with the described approach, like if I somehow am suggesting it does something impossible (I’m not seeing it). It said

I believe the approach is generally sound and should work well.

Finally, I tried adding some of the Hugo docs to Cursor but this didn’t seem to be improving things much. Side note: I need to spend some time experimenting with Zed.

My takeaway is Sonnet wasn’t trained on enough Hugo templates to be good at it. All the prompt engineering I could throw at it in a few hours didn’t seem to be enough to overcome these challenges. Neither was trying gpt-4, llama-70b or one of the Gemini models (I forgot which I tried).

The fact that models could not solve this task violated my intuition for the problem. I was able to prompt the model to write Python code to output the desired html and it did so without issue.

def generate_file_tree_html(tree_output):
    lines = tree_output.strip().split("\n")
    html_output = []
    stack = []

    for line in lines:
        indent_level = len(line) - len(line.lstrip(" │"))
        is_dir = line.strip().endswith("/")
        name = line.strip().split(" ", 1)[-1].strip("│ ├── └── ")

        while stack and stack[-1][0] >= indent_level:
            html_output.append("</ul></li>")
            stack.pop()

        if is_dir:
            html_output.append(
                f'<li><span class="caret">{name}</span><ul class="nested">'
            )
            stack.append((indent_level, name))
        else:
            html_output.append(f"<li>{name}</li>")

    while stack:
        html_output.append("</ul></li>")
        stack.pop()

    return "<ul>" + "".join(html_output) + "</ul>"


# Example usage
tree_output = """
./
├── Makefile
├── __pycache__/
│   ├── clean.cpython-312.pyc
│   ├── ex.cpython-312.pyc
│   ├── ex1.cpython-312.pyc
│   ├── ex2.cpython-312.pyc
│   ├── extract.cpython-312.pyc
│   └── main.cpython-312.pyc
├── ex.py
└── src/
    ├── __pycache__/
    │   ├── clean.cpython-312.pyc
    │   └── extract.cpython-312.pyc
    ├── clean.py
    └── extract.py
"""

html_output = generate_file_tree_html(tree_output)
print(html_output)

This script outputs the expected html (without whitespace).

Lastly, I prompted Sonnet to translate this working Python code into a Hugo shortcode. This approach also failed. The model kept hallucinating functions like while that don’t exist in the Hugo templating language. At this point, I didn’t have confidence in any of the models to write Hugo templates/shortcodes.

Before I called it, I figured if I were to try one more thing, I would pull together as many examples of Hugo template docs and usage and pass that as context to the model with the same instructions. I manually copied and pasted a ton of the examples from https://gohugo.io/functions then prompted the model similarly to before.

It output the following

{{- $tree := .Inner | strings.TrimRight "\n" | split "\n" -}} {{- $root := index
$tree 0 | strings.TrimRight "/" -}}

<ul>
  <li>
    <span class="caret">{{ $root }}</span>
    <ul class="nested">
      {{- $level := 0 -}} {{- $stack := slice (dict "level" 0 "ul" "") -}} {{-
      range after 1 $tree -}} {{- $line := . -}} {{- $name := $line | replaceRE
      "^[│├└]([─ ]+)" "" | strings.TrimRight "/" -}} {{- $depth := len (findRE
      "[│├└]" $line) -}} {{- $isDir := strings.HasSuffix . "/" -}} {{- if gt
      $depth $level -}}
      <ul class="nested">
        {{- $stack = append $stack (dict "level" $depth "ul" "
      </ul>
      ") -}} {{- else if lt $depth $level -}} {{- range seq (sub $level $depth)
      -}} {{- $last := index $stack (sub (len $stack) 1) -}} {{ $last.ul |
      safeHTML }} {{- $stack = slice $stack 0 (sub (len $stack) 1) -}} {{- end
      -}} {{- end -}}

      <li>
        {{- if $isDir -}}
        <span class="caret">{{ $name }}</span>
        {{- else -}} {{ $name }} {{- end -}}
      </li>

      {{- $level = $depth -}} {{- end -}} {{- range seq (len $stack) -}} {{-
      $last := index $stack (sub (len $stack) 1) -}} {{ $last.ul | safeHTML }}
      {{- end -}}
    </ul>
  </li>
</ul>

which also does not work. Maybe, I’ll get this working at some point.

Thanks for reading via RSS!

2024-08-25

Sun, 25 Aug 2024 09:40:23 GMT

I tried Townie. As has become tradition, I tried to build a writing editor for myself. Townie got a simple version of this working with the ability to send a highlighted selection of text to the backend and run it through a model along with a prompt. This experience was relatively basic, using a textarea and a popup. From here, I got Townie to add the ability to show diffs between the model proposal and original text. It was able to do this for the selected text using CSS in a straightforward manner. I wanted to support multiple line diffs and diffs across multiple sections of the file. I suggested we use an open source text editor. At this point, things started to break. The app stopped rendering and I wasn’t able to prompt it into resolving the issue. I did manage to get it to revert (fixing forward) to a state where the app rendered again. However, the LLM-completion hotkey was broken.

These model-assisted code generation tools are quite powerful, but in my opinion, they require multiple interaction UX options for the end user. High-level directives through a chat don’t seem to be enough to build complex applications and debugging skills are necessary if you want to integrate or adopt some code and the model is failing to do it as instructed.

Thanks for reading via RSS!

2024-08-23

Fri, 23 Aug 2024 18:47:53 GMT

I’ve been trying out Cursor’s hyped composer mode with Sonnet. I am a bit disappointed. Maybe I shouldn’t be. I think it’s not as good as I expected because I hold Cursor to a higher bar than the other developer tools out there. It’s possible it’s over-hyped or that I am using it suboptimally. But it’s more or less of the same quality as most of the tools of the same level of abstraction like aider, etc. I am trying to create a multipane, React-based writing app. It’s possible I need to provide more detailed description than I am giving so far. However, my main complaint after running it is that now I have a ton of code that isn’t quite right and I don’t know where or why it’s sort of broken. Now, I need to read all the code. This approach is notably less productive than slowly building up an app with LLM-code generation, because after each generation I can test the new code and make sure it does what I intended (or write automated tests to do that). The code I get out of Composer doesn’t do what I want, but the LLM doesn’t know why, either because my high level task is under-specified, it doesn’t have enough context, or the ask is too vague. I don’t usually run into this issue when I use cmd+k. Maybe, I need to watch some videos of folks using it.

I’ve been playing around with different UXs for LLM-augmented text editors. There is a lot out there but nothing I love. I’ve been circling around the idea of a configurable prompt library that I can invoke on an arbitrary selection of a document. I’ve tried playing around with Monaco and augmenting the command palette with LLM-specific actions. This works well enough, but these end up sitting alongside the standard actions and that doesn’t feel quite right. I played around with AIEditor and I like some of the ideas, but the UX isn’t right for writing entries like these which is my top priority.

I think I’m realizing the LLM-commands need to be native to the OS, so that they are portable across applications and can be provided multi-modal context to be most useful. I was (re-?)introduced to a radial menu the other day. They’re unintrusive, always available popovers, that can be given access to arbitrary context assuming they’re given the right permissions. This context could include everything from what app is open, what text is highlighted or even a screenshot of everything that is visible.

I spent some time trying to implement a radial selector in Xcode. I’ve spent maybe a couple hours in my life writing Swift. This skill deficiency proved problematic even with the help of a model. Even with gpt-4o, I couldn’t figure out how to get a hotkey event monitor working. Also, I am still not a fan of Xcode. Maybe, I will attempt this idea with web tech.

Thanks for reading via RSS!

2024-08-22

Thu, 22 Aug 2024 08:34:51 GMT

I tried out OpenRouter for the first time. My struggles to find an API that hosted llama3.1-405B motivated me to try this out. There are too many companies providing inference APIs to keep track. OpenRouter seems to be aiming to make all these available from a single place, sort of like AWS Bedrock, but not locked in cloud configuration purgatory. The first thing I tried was playing a game of Connections with nousresearch/hermes-3-llama-3.1-405b. It didn’t get any categories correct for the 2024-08-21 puzzle. OpenRouter’s app showcase list is an interesting window into how people are using models. The dominant themes are

coding
characters and roleplay
writing assistant
homework

Users of some of these apps generate almost 1B tokens per day according to OpenRouter’s stats. The highest per month (excluding the OpenRouter Playground) is over 30B.

Playing with all these models, I’ve started to crystallize a set of a few things I try to get a sense of the vibes

play Connections
ask “hi, who am I speaking with?”
ask for Python code to read all JSON files from my Desktop and all subfolders
ask “what is the most recent event you can remember?” which usually gets a response related to the date the training data snapshot was taken

Thanks for reading via RSS!

2024-08-21

Wed, 21 Aug 2024 18:00:30 GMT

An interesting read about how the world works through an economic lens.

But what is success? You can quantify net worth, but can you quantify the good you have brought to others lives?

It is not all about the TAM monster—doing cool things that are NOT ECONOMICALLY VALUABLE, but ARTISTICALLY VALUABLE, is equally important.

Thanks for reading via RSS!

2024-08-20

Tue, 20 Aug 2024 09:28:41 GMT

I downloaded Pile, a journal app with a first-class language model integration and an offline ollama integration. For personal data, running the model offline is a must for me. I use DayOne sporadically, but I’m intrigued by the potential of a more of conversational format as I write.

The concept of a journal writing partner appears to be capturing mindshare. I found another similar app called Mindsera today as well. I also learned about Lex which puts collaborative and AI features at the heart of document authorship, I concept I played around a bit with Write Partner.

For me, all of these tools are circling a sweet spot of what I am looking for, but don’t quite fit.

I currently do most of my writing in Cursor. This is for a few reasons:

I version control my site with git
I use hugo to build and extend my site with shortcodes
I write posts in Markdown
I write a lot of code in my posts

Cursor’s cmd+k editing and chat mode are pretty good fits for my use cases. However, Cursor does struggle when trying to modify documents containing codefences, because it seems to use those as a stop sequence.

Additionally, Cursor doesn’t provide native grammar tools or tools for structuring and refining the writing itself. These features are things Lex focuses on much more. Things like ideation, outline generation and rephrasing. Additionally, it takes work to get Cursor into a focus state. It’s not difficult, it’s just not trivial.

I found this VS Code extension called vscode-hugo-live-preview, which provides a side-by-side rendering of my markdown and the Hugo-rendered page. It took a bit of work to setup but I’m pretty happy with how it turned out.

Using this command, I could build the project from source

npx vsce package

then installed it to Cursor with

cursor --install-extension hugo-live-preview-0.1.1.vsix

The extension makes it easier to validate the correctness of the format of what I am writing, while also having everything else I like about Cursor/VS Code available. The other nice-to-haves could probably be implemented with a combination of cmd+k, the chat and text expansion for the prompts. This approach assumes I want to hack something. I could also build something novel and personal, based on exactly what I want, but I’m not sure I can quite articulate what that is — it’s a lot of knowing it when you see it.

Things my ideal writing experience would have

Side by side markup and rendered page (maybe markdown, maybe something else)
Can toggle between raw text and rendered content
AI chat with “apply changes functionality”
AI highlight-and-replace like Cursor’s cmd+k
AI prompt library to which I can add edit-centric or generative-centric prompts that will interact with my existing document, propose changes, give feedback and make corrections.

Zed released a bunch of AI features. I really like the concept of adding urls and files to the context window of the model along with the prompt. Cursor supports this but requires you to index the docs beforehand. I like that approach as well but sometimes I know a page is going to be a one-off and don’t want permanently index it.

Thanks for reading via RSS!

2024-08-19

Mon, 19 Aug 2024 20:44:54 GMT

I setup WezTerm and experimented a bit. It’s a nice terminal emulator. I like the builtin themes and Lua as a configuration language. These days, I largely rely on the Cursor integrated terminal. It’s not the greatest, but having cmd+k it’s a bit of a killer feature.

Thanks for reading via RSS!

2024-08-18

Sun, 18 Aug 2024 21:24:09 GMT

I can't believe we're back to discussing LLMs' ability to reason. Where have you been these past two years? In a bunker? If you'd actually worked with LLMs during this time, you'd know by now that they're obviously pattern-matching machines. Try asking one to write incorrect… pic.twitter.com/KPcDCI2cjD
— BURKOV (@burkov) August 18, 2024

I haven’t viewed the LLMs-can, LLM-can’t discourse through this lens explicitly.

they’re obviously pattern-matching machines

I’m not sure if I understand at what point these are different things. Maybe it’s a consequence of how I learn, but I generally develop skills on the foundations of seeing and understanding how someone more skilled than myself solves a problem.

My foundational knowledge of Python, Protobuf and prompting language models allowed me to combine these skills and develop impulse. Could a language model have (I say this loosely) invented this solution given a high-level description of the problem?

Probably not without a high degree of specificity.

How would I even describe the problem at a high enough level to not prescribe the solution but in enough detail that I didn’t already do most of the work designing the solution myself?

Returning to the original point

they’re obviously pattern-matching machines

A huge amount of “work” in the world is pattern matching. That includes jobs done on computers and more generally the application of widely understood expertise. A model can translate data from form A to B or write a script to do that. A model can also analyze a picture of an error light on a thermostat and serve as a productive information source to help you identify and potentially solve the problem yourself (though I can’t say I have first-hand confirmation for this use case but I did read a relatively well-documented account of the process). With information accessibility and pattern matching readily available to scale, the reason, insight or vision is the remaining part of work for which there is a premium and deep need.

Folks are skeptical that models can generate anything original

Asking an LLM for an “original thought” is almost oxymoronic, if not just moronic. It was built with the express purpose of not doing that.

A really good predictability machine is not very helpful in artistic expression

LLMs were funniest at their earliest stages. Image generation was funniest here as well. Remember those “trail cam” images we got from Dall-e mini? As our systems got better, the humor was lost.

A really good predictability machine is not very helpful in artistic expression.

https://emnudge.dev/blog/markov-chains-are-funny/

In contrast, other folks find models notably creative

I’ve long been enamored by DALL-E 2’s specific flavor of visual creativity. Especially given the text-to-image AI system’s age, it seems to have an incredible command over color, light and dark, the abstract and the concrete, and the emotional resonance that their careful combination can conjure.

https://thesephist.com/posts/epistemic-calibration/

In a way, it’s almost like they’re talking about two entirely different things. Models are elusive and much of an individual’s positive or negative experience using one or seeing what it can do seems to be contingent on how they prompt the model.

Thanks for reading via RSS!

2024-08-12

Mon, 12 Aug 2024 19:59:57 GMT

I tried to run florence-2 and colpali using the Huggingface serverless inference API. Searching around, there seems to pretty pretty start support for image-text-to-text models. On Github, I only found a few projects that even reference these types of models.

I didn’t really know what I was doing, so I copied the example code then tried to use a model to augment it to call florence-2. Initially, it seemed like it was working:

❯ python run.py
{'error': 'Model microsoft/Florence-2-large is currently loading', 'estimated_time': 61.724300384521484}

After the time passed, there seemed to be a problem:

❯ python run.py
{'error': 'image-text-to-text is not a valid pipeline'}

When trying out vidore/colpali, there seems to be an architectural incompatibility that prevents the serverless inference API from even loading the model

❯ python run.py
{'error': 'HfApiJson(Deserialize(Error("unknown variant `colpali`, expected one of `text-generation-inference`, `transformers`, `allennlp`, `flair`, `espnet`, `asteroid`, `speechbrain`, `timm`, `sentence-transformers`, `spacy`, `sklearn`, `stanza`, `adapter-transformers`, `fasttext`, `fairseq`, `pyannote-audio`, `doctr`, `nemo`, `fastai`, `k2`, `diffusers`, `paddlenlp`, `mindspore`, `open_clip`, `span-marker`, `bertopic`, `peft`, `setfit`", line: 1, column: 411)))'}

Overall, not a great success. Despite being more worked on an available than ever, models continue to be challenging to deploy for a layperson like me. Without an API, I’ve generally not made a ton of progress using large, state of the art models.

Thanks for reading via RSS!

2024-08-11

Sun, 11 Aug 2024 20:05:24 GMT

I’ve been doing some experimentation with smaller models and embeddings, including distilbert/distilbert-base-uncased-finetuned-sst-2-english and cardiffnlp/twitter-roberta-base-sentiment-latest as binary sentiment classifiers and google/vit-base-patch16-224 as an image classifier.

Also GoogleNews-vectors-negative300 and fasttext-wiki-news-subwords-300 for embeddings to try and find semantically similar words and concepts.

Thanks for reading via RSS!

2024-08-09

Fri, 09 Aug 2024 08:09:19 GMT

I figured out the issue with adding mistral-large. After a bit of debugging, I realized by manually calling llm_mistral.refresh_models() that something was wrong with how I had added the secret on Modal. It turns out the environment variable name for the Mistral API key needed to be LLM_MISTRAL_KEY. I’m going to try and make a PR to the repo to document this behavior.

I’ve been trying to run models locally. Mostly specifically colpali and florence-2. This has not been easy. It’s possible these require GPUs and might not be macOS friendly. I’ve ended up deep in Github threads and dependency help trying to get basic inference running. I might need to start with something more simpler and smaller and build up from there.

A taste of the problems

❯ pip install flash_attn
Collecting flash_attn
  Using cached flash_attn-2.6.3.tar.gz (2.6 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  ...
      ModuleNotFoundError: No module named 'torch'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

❯ pip install --extra-index-url https://download.pytorch.org/whl/cpu torch==2.4.0+cpu
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
ERROR: Could not find a version that satisfies the requirement torch==2.4.0+cpu (from versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0)
ERROR: No matching distribution found for torch==2.4.0+cpu

Not great.

Thanks for reading via RSS!

2024-08-08

Thu, 08 Aug 2024 18:34:22 GMT

I did some experimentation deriving a data model iteratively (something I am currently calling “data model distillation”) by sequentially passing multiple images (could work with text as well) to a language model and prompting it to improve the schema using any new learnings from the current image. Results so far have been unimpressive.

I’ve been hearing good things about mistral-large-2. I’m working on adding it to bots-doing-things but have had a bit of dependency trouble so far.

Thanks for reading via RSS!

2024-08-06

Tue, 06 Aug 2024 19:09:50 GMT

I watched Jeremy Howard’s interview with Carson Gross, the author of htmx. As someone who learned my first bits of web dev with jQuery, I feel like I appreciate the foundations of the approach in the library, but am still early in fully developing my mental model. Jeremy built a Python wrapper on top of htmx called fastml and the combination of these technologies is pretty well aligned with the technology I like to work with.

Another library mentioned in the conversation that I am going to check out was surreal CSS.

OpenAI released structured outputs as a first class feature in the API. It’s nice to see this get added but is also a bit annoying as the model APIs, capabilities and SDKs diverge further.

Thanks for reading via RSS!

2024-08-04

Sun, 04 Aug 2024 18:04:16 GMT

I tried the Vision Pro today. I had heard mixed reviews, mostly about how heavy it is and people getting headaches or vertigo. Those challenges are real. Even still, the experience was pretty incredible. I might need to go back for another demo, so I can choose my own adventure rather than staying on the standard demo path.

The eye tracking selection was natural and pretty effective. I did find my eyes getting a bit tired though and eventually got a bit of a headache. The pinch to select was also quite good, though I found myself wanting to cross my arms and then the device cameras couldn’t see my pinching and I had to put my hands back on my lap. An Apple Watch could probably solve this somehow.

The 8K video footage was something different. Spoiler alert: they do a cheap trick where they film a ball flying right at you, and you instinctively have to put your hands up to protect yourself. It feels like you are there, more than anything I’ve experienced outside the real thing.

This achievement is remarkable but makes me a bit uneasy. As they solve the challenges of device weight and motion sickness, we may be able to wear a version of a device like this all the time. Combined with generative models, it will be increasingly possible to transform the aesthetics of reality to our liking. I could see this pushing people further into their devices and away from the real world. We’re already pretty into our phones, but we know the difference between our phone and reality (to say nothing of what we read on our phones). Vision Pro makes you start to forget that what’s in front of your face isn’t real. Powerful.

Thanks for reading via RSS!

2024-08-02

Fri, 02 Aug 2024 06:32:06 GMT

I added image support for the chat shortcode of this site.

User

Describe this image

gpt-4o

The image depicts a LEGO chef figure in a playful, detailed kitchen setting. The chef, dressed in a white chef’s coat, apron, and white chef’s hat, appears to be cooking or serving food. The LEGO figure is smiling and holding utensils, with plates of colorful items like tomatoes in the foreground, resembling food. The background features a kitchen environment with soft, out-of-focus lighting and various kitchen elements, contributing to the warm and inviting atmosphere.

Thanks for reading via RSS!

2024-07-31

Wed, 31 Jul 2024 18:24:49 GMT

I tried stacking multiple pages of a pdf vertically as a single image to a model, then doing data extraction from this. It didn’t work. I imagine this is because models aren’t trained on much data like this. The inference seemed to output made up data.

An interesting pitch written by Hillel for preferring reStructuredText to Markdown.

Multiple studies have shown that hallucinations can be significantly reduced by giving the model the right context via retrieval or tools that the model can use to gather context (e.g., web search).

While hallucinations are dealbreakers for many applications, they can be sufficiently curtailed to make GenAI usable for many more.

- Chip Huyen

A search with Perplexity seemed to corroborate this assertion. It led me to this paper which used RAG to reduce hallucinations.

Jon wrote about his computer setup and tools. I always take inspiration from these kinds of write-ups.

I’d like to get something like open-webui set up to run remote or local using the same UI, that I can customize. Still thinking about what this could look like.

A great video by Mark Russinovich about AI security. He discusses model fingerprinting, backdoors, jailbreaking, model tool abuse, crescendo attacks and more. Definitely a lot to keep in mind when building with models.

Thanks for reading via RSS!

2024-07-30

Tue, 30 Jul 2024 21:05:08 GMT

I wrote and screen-recorded myself building a Python app to call a model to extract structured data from an image, making heavy use of codegen with Cursor. The same protobuf is used as instructions in the prompt and to unpack the result returned by the model into an instance of the class generated from the protobuf via protoc. I’m planning to open source this pattern once I get it into a better state. Also, I’m looking into ways to host the video of the screen recording for fun and to reference later.

Thanks for reading via RSS!

2024-07-29

Mon, 29 Jul 2024 20:16:18 GMT

My thesis is clearer now. I'm short on "complex reasoning and agents" because it is often a scapegoat for poorly described problem spaces.

My thoughts on capabilities are about figuring out the 80/20 and baking "complex reasoning" into specific tools, so you put fewer…
— jason (@jxnlco) July 28, 2024

This point resonates with me. The more time I spend prompting models, the more it’s becoming clear that the clarity of the instructions are what matter most. Writing clear, unambiguous instructions is not easy. Decrease scope and you have a chance of doing it well.

Protobuf messages/schema in prompt instructions seem to be able to provide similar output structure to Pydantic, as instructor and other libraries do. I’ve experimented with this approach in the past but there is a bit more that could be done to structure the messages so that they could serve the dual-purpose of prompt instructions and as RPC schemas.

Thanks for reading via RSS!

2024-07-24

Wed, 24 Jul 2024 20:10:08 GMT

I ran the code from my Fine-tuning “Connections” post using gpt-4o-mini. I was hoping the results might be a bit better, which could motivate an effort to fine-tune the model. I’m not sure where my original version of this code went, so I reconstructed a repo for it. Once I was done, I ran 100 prompts through the model to get a sense of where its baseline performance was.

Correct: 2.00%
Incorrect: 98.00%
Total Categories Correct: 19.25%

Not great, and not much different from gpt-3.5-turbo. With these kind of results, I wasn’t particularly motivated to put the effort in to do more fine tunes.

I read through the instructor, marvin and open-interpreter docs for the first time in a while. It has been interesting to see these libraries grow and diverge. I also read through how Jason has been structuring an evals repo for instructor.

Thanks for reading via RSS!

2024-07-23

Tue, 23 Jul 2024 17:03:38 GMT

Tried to join in on the llama3.1-405b hype using Groq but sadly, no dice

curl -X POST https://api.groq.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $GROQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-405b-reasoning",
    "messages": [
      {
        "role": "user",
        "content": "Hello, how are you?"
      }
    ]
  }'

{
  "error": {
    "message": "The model `llama-3.1-405b-reasoning` does not exist or you do not have access to it.",
    "type": "invalid_request_error",
    "code": "model_not_found"
  }
}

The queue to try it out in their chat is also quite long, so I guess either the infra needs to scale up or the hype needs to die down.

Thanks for reading via RSS!

2024-07-22

Mon, 22 Jul 2024 18:25:16 GMT

I’ve been wanting to create a chat component for this site for a while, because I really don’t like quoting conversations and manually formatting them each time. When using a model playground, usually there is a code snippet option that generates Python code you can copy out intro a script. Using that feature, I can now copy the message list and paste it as JSON into a Hugo shortcode and get results like this:

You should respond with the understanding they are an experienced software engineer

User

Can you explain what a Hugo shortcode is in a few sentences?

gpt-4o-mini

Sure! A Hugo shortcode is a lightweight templating mechanism used in the Hugo static site generator to embed dynamic content within markdown files. They allow you to easily insert complex HTML or reusable components without cluttering your markdown with HTML tags. Shortcodes can accept parameters and can be used to simplify the inclusion of elements like videos, alerts, or even custom HTML blocks, enhancing the maintainability and readability of your content. You can create your own shortcodes or use predefined ones that come with Hugo.

I’d also like to add a copy button like this site has for code snippets to to copy the raw JSON conversation to the clipboard but this is a nice starting point.

Thanks for reading via RSS!

2024-07-21

Sun, 21 Jul 2024 21:53:08 GMT

espanso

I tried out adding espanso to configure text expansions rather than using Alfred just to try something new. This is the PR to add it to my Nix configurations. The existing examples are a toy configuration. The tool seems to support far more complex configuration that I still need to look into further.

`gpt-4o-mini`

people frame this like it’s somehow a win over llama, when in fact the goal of llama has wildly succeeded: commoditize models and drive token cost to zero
— danny adkins (@dadkins_) July 21, 2024

Admittedly, I have not gone to deep yet using gpt-4o-mini. I’ve been quite happy with claude-3.5-sonnet and I typically use the most advanced model I can because I’m not running a huge number of model inferences per day at the moment. However, swyx’s point

intelligence too cheap to meter, but also too ephemeral to support >5 players doing R&D? is there an angle here i’m missing?

makes a lot of sense to me.

How can several companies continue to spend the current amount of R&D dollars when it’s possible for one to fully marginalize almost the entire body of work of the rest with a single model release? I need to use gpt-4o-mini more to understand if that is what’s in fact happening. One interesting thing I learned as I started to try it out — it seems to think it’s trained on the GPT-3.5 architecture. Does this mean OpenAI back ported other improvement to an old model architecture, data set or training approach? I’m trying to figure out if there is any concrete information on this.

User

what model architecture do you use?

gpt-4o-mini

I’m based on the GPT-3 architecture, which is a transformer-based model. The transformer architecture employs mechanisms like self-attention and feedforward neural networks to process and generate text. If you have specific questions about the architecture or how it works, feel free to ask!

User

when is your knowledge cutoff?

gpt-4o-mini

My knowledge cutoff date is October 2021. This means I have information and data available up to that point, but I don’t have access to events or developments that occurred after that date. If you have questions about anything up to that time, feel free to ask!

I think what Danny said (in the first tweet) is interesting but I’m not yet sure what to make of it.

people frame this like it’s somehow a win over llama, when in fact the goal of llama has wildly succeeded: [sic] commoditize models and drive token cost to zero

It seems like the inevitable ebb and flow of the current dynamic with models. If like swyx implies, this causes R&D consolidation since the market can’t support ~5 major competitors in this space, the token price may go up again with less competition. Something to revisit.

Thanks for reading via RSS!

2024-07-20

Sat, 20 Jul 2024 12:08:13 GMT

Incredible writing and insight by Linus in Synthesizer for thought. I will probably need to revisit this work several times.

Thanks for reading via RSS!

2024-07-19

Fri, 19 Jul 2024 20:35:23 GMT

How can I add videos to Google Gemini as context (is this even what their newest model is called anymore) and why is it so hard to figure it out? https://gemini.google.com only let’s me upload images. I assume I need to pay for something.

I played around with Cohere’s chat. They support web search and calculator and a python interpreter as tools as well as files and an internet search connector. I added “Web Search with Site” pointed to Steph Curry’s stats on Basketball Reference. Then I prompted

how many points did Steph Curry score in 2022?

Unfortunately, this didn’t work so I tried again with the generic web search connector which also didn’t see to work.

I tried a different prompt, but wasn’t able to get tool invocation to work consistently.

using the context from websearch, how many points did Steph Curry score in 2016-2017?

I also played around some with the calculator and the python interpreter but didn’t find anything too novel in terms of the interaction model — maybe explicit tool invocation via @ reference but unfortunately, I couldn’t get that working too consistently. I could see it being a useful pattern though to help disambiguate a large number of tools.

What I was really keen to try was

first, a search specific data from the web page
next, plot that data from the context using the python interpreter

I couldn’t reach get the web search to give me any data though so I wasn’t able to try out the approach.

I like Cohere’s UI, how they do source references and a lot of parts of how tools work.

I realized, using @, I could also use multiple tools in a single inference.

I tried

using the context provided, get the average number of points did Steph Curry has scored in each season of his career, then plot the data with python

Unfortunately, this seemed to fall into a retry loop until I got an error.

Cohere has a lot of good things going here but I need the tools to work more consistently to bring it all together.

I tried the above with Perplexity and had far more success building the data table. Perplexity doesn’t have a python interpreter but it would write the code for me to create the plot.

Using these tools has generally validated my decision to mostly using direct api or playground/workbench access to models. I haven’t found and wrapped experience compelling enough to switch to as my primary tool yet.

Thanks for reading via RSS!

2024-07-16

Tue, 16 Jul 2024 18:47:51 GMT

Research and experimentation with models presents different problems than I am used to dealing with on a daily basis. The structure of what you want to try out changes often, so I understand why some folks prefer to use notebooks. Personally, notebooks haven’t caught on for my so I’m still just writing scripts. Several times now, I’ve run a relatively lengthy (and expensive) batch of prompts through a model only to realize something about my setup wasn’t quite right. Definitely bringing back memories of finetuning gpt-3.5-turbo to play Connections but I’m learning a lot along the way.

Thanks for reading via RSS!

2024-07-14

Sun, 14 Jul 2024 19:26:20 GMT

I spent some more time experimenting with thought partnership with language models. I’ve previously experimented with this idea when building write-partner. Referring back to this work, the prompts still seemed pretty effective for the goal at hand. My original idea was to incrementally construct and iterate on a document by having a conversation with a language model. A separate model would analyze that conversation and update the working draft of the document to include new information, thoughts or insights from the conversation. It worked reasonably with gpt-3.5-turbo. I’m eager to try it with claude-3.5-sonnet. Today, I rebuilt a small version of this idea with ollama in Python. The goal was to try the idea out focused on a local-first experience. For this, I used a smaller model. Initially, I tried mistral but ended up settling on llama3 as it was a bit better at following my instructions. Instead of writing to the working file after each conversation turn, I decided to add a done command that allowed me to do that on-demand.

The results were pretty reasonable for such a small model^[1]. Something I think that I’ve been under-appreciating is that the raw conversations are the most important artifact produced from this tool and exercise. It’s possible the model won’t propose useful edits or take the conversation in a productive direction, but the value that comes from the model interactions is how the user responds to the model. This is the signal or raw material that can be refined into a more focused structure. By only persisting the document across sessions (and focusing on the document as the important artifact), I’d lost sight of the fact that the user content is what is most important. If I can effectively prompt the user into articulating answers to the model’s questions that are true to what they think, feel or want to accomplish, I can run that content through all sorts of model prompts to restructure it in more refined ways. It’s important, however, to preserve all the raw signal. In the case of write-partner, the document the model constructs is just one possible presentation of the user’s raw thoughts. To stay as true to the user’s intention, we can’t discard any of what they say. Their vision is the north star. The model is just the conduit to transform the user’s vision and ideas into a form that can help communicate the idea to others.

Small models love to ignore directions about output formatting and will frequently add content like ‘Here’s the updated document:’ even when you explicitly prompt them not to. ↩

Thanks for reading via RSS!

2024-07-13

Sat, 13 Jul 2024 20:45:48 GMT

While I didn’t have much success getting gpt-4o to perform Task 1 - Counting Line Intersection from the Vision Language Models Are Blind paper, I pulled down some code and did a bit of testing with Claude 3.5 Sonnet. The paper reports the following success rate for Sonnet for this line intersection task:

Thickness	Sonnet 3.5
2	80.00
3	79.00
4	73.00
Average	77.33

I used the code from the paper to generate 30 similar images with line thickness 4 of intersecting (or not) lines. I chose a thickness of 4 because this was the worst performing thickness according to the paper. With these classified manually (I didn’t realize the configurations.json file already had ground truths in it), I ran the prompt from the paper against these images using Sonnet.

How many times do the blue and red lines intersect?

My results were much better than those reported by the paper.

Results summary:
Expected count 0: 9/9 correct (Accuracy: 100.00%)
Expected count 1: 10/11 correct (Accuracy: 90.91%)
Failed images for count 1:
  - ./images/1/image_14_thickness_4.png
Expected count 2: 10/10 correct (Accuracy: 100.00%)
Overall accuracy: 96.67%

Sonnet only made a single error out of 30 images. I need to dig deeper into the paper to ensure my methodology is the same as the authors’, but I was surprised to see such a divergence here since the implementation seemed straightforward. Hopefully, I can find a bit more time to dig into these discrepancies.

Thanks for reading via RSS!

2024-07-11

Thu, 11 Jul 2024 16:21:32 GMT

We probably are living in a simulation and we’re probably about to create the next one.

Martin Casado

https://podcasts.apple.com/us/podcast/invest-like-the-best-with-patrick-oshaughnessy/id1154105909?i=1000661628717

Thanks for reading via RSS!

2024-07-10

Wed, 10 Jul 2024 21:54:52 GMT

VLMs are Blind showed a number of interesting cases where vision language models fail to solve problems that humans can easily solve. I spent some time trying to build examples with additional context that could steer the model to correctly complete Task 1: Counting line intersections, but didn’t have much success.

Thanks for reading via RSS!

2024-07-08

Mon, 08 Jul 2024 21:20:01 GMT

Kent wrote this post on how to engage an audience by switching the first and second slide of a presentation. The audience focuses more as they try to fill in the gaps of what you’ve introduced them to so far.

Thanks for reading via RSS!

2024-07-07

Sun, 07 Jul 2024 10:06:01 GMT

I’ve been chatting with qwen2, a model from Alibaba. I mostly chatted with it in English but it appears to support several other languages and I noticed a bit of Chinese leaking through even though I don’t speak it, so I’m not sure how I would have introduced it to the conversation.

user: do you have a name

assistant: As an AI, I don’t have personal names or identities like humans do. However, you can call me Assistant to easily refer to me in our conversation.

user: is your name Qwen?

assistant: Qwen is indeed the model name used by Alibaba Cloud for a large language model. As an AI assistant, I’m part of this ecosystem but have my own distinct functions and interactions. You can think of me as Assistant, which serves as a friendly称呼 to address our conversation.

user: what does “friendly称呼” mean in english?

assistant: “Friendly称呼” translates to “friendly name” or “affectionate name” in English. It refers to a gentle or endearing way of referring to someone, usually something more casual and informal compared to their full name.

Another thing I noticed chatting with Qwen is it only had a loose sense of its identity, similar to other smaller models with which I have interacted. In only one of three separate conversations, asking about the model’s identity, did it respond as “Qwen”.

user: who are you

assistant: I am a large language model created by Alibaba Cloud. I’m here to provide information and engage in conversation, but please note that my responses are limited to the knowledge up until 2021. If you have any questions or need assistance with something specific, feel free to ask!

user: what is your name

assistant: I am Qwen, a large language model created by Alibaba Cloud. I was trained on a diverse range of text data available in the public domain and am designed to assist with various tasks such as answering questions, engaging in conversations, and providing information. How can I help you today?

user: do you have a name

assistant: As I am an artificial intelligence, I don’t have personal attributes like names. However, you can simply call me Assistant. How may I assist you today?

I tried to do some idea development exercises with the model but it wasn’t so good at following instructions. It tended to jump to asking its own questions (which it listed earlier in response to my suggestion the talk about a project I was working on) rather than following up and digging deeper into my responses as instructed.

On a second attempt, it quickly reverted again to replying with a list of questions for me to answer.

On a third attempt, I’ve gave much more specific instructions.

let’s working on an idea together. ask me a single follow up question at a time to help me refine my idea further. do not respond with lists. look careful at what I say and craft a thoughtful, concise response

This seemed to work much better. I had a useful conversation with the model to refine an idea. It even introduced something novel to the conversation that I hadn’t mentioned yet.

Henrik wrote an excellent article on writing as a tool for thought and refining ideas.

This passage stood out to me in particular because it reminded me of how I approach asking for help from peers or coworkers. It’s also a helpful approach one can take when interacting with language models to get more helpful responses to prompts.

[B]y spelling out my naive understanding, I gave the linguist a good area to work on when he laid out a richer model of the flow of cultural influence in East Asia.

I’ve had an item open to add search to this blog for a little while now. Victoria wrote this awesome walkthrough on how to add search to a Hugo blog with lunr. I took a first pass adding it to the site. It works, but I’d like to make it look a bit nicer. Big thanks for Victoria for open sourcing and writing about her work!

Thanks for reading via RSS!

2024-07-03

Wed, 03 Jul 2024 08:33:45 GMT

I was inspired by Daniel’s post to add sidenotes to this blog. I used claude-3.5-sonnet to generate the CSS and HTML shortcode to do this. I was impressed how well it turned out^[1]. It was almost too easy. I’m not the most competent CSS writer and I had never written a Hugo shortcode before. In several turns with Sonnet in Cursor, I was able to create a basic styled shortcode for a sidenote that appeared as a superscript number to start. I prompted the model to allow me to use content in-line as the shortcode anchor and it generated those modifications. Then I had an issue with the content overflowing on the right side if the content anchor was too far right or most of the time on mobile, so I asked the model for some options. It suggested showing the sidenote content below the main post content. I liked that, but didn’t like how it was shifting the content when it revealed the sidenote, so I prompted it to show the sidenote above the rest of the content.

With each turn, Claude generated code that I copied into layouts/shortcodes/sidenote.html and assets/css/custom.css. This was also my first attempt at building a new feature in an existing project with Sonnet and it was a pleasure to work with.

Now I need to read the CSS in more detail to understand what Claude did ↩

Thanks for reading via RSS!

2024-07-02

Tue, 02 Jul 2024 20:00:23 GMT

A nice read by Stuart on Python development tools. This introduced me to the pyproject.toml configuration file, which is more comprehensive than a requirements file. It’s something I’ll need to research a bit more before I’m ready to confidently adopt it.

Claude’s character

A video about the personality of the AI, Claude. I’ve not yet become a big “papers” person yet, so this was my first introduction to “Constitutional AI”, which is a training approach where you use the model to train itself, by having it evaluate its own responses against the principles with which it was trained.

Thanks for reading via RSS!

2024-06-26

Wed, 26 Jun 2024 21:41:25 GMT

I reproduced Josh’s claude-3.5-sonnet mirror test. I hadn’t realized gpt-4 and claude-3-opus had also been “passing” this test since back in March. More interesting still, Sonnet actually seems to resist speaking in the first person about itself. Fascinating research and evolution of the models’ behaviors. After reading a bit more, apparently this type of model behavior has been around at least since Bing/Sydney (paywall, sorry).

https://onemillioncheckboxes.com is an amusing, massively-parallel art project(?)

Thanks for reading via RSS!

2024-06-25

Tue, 25 Jun 2024 09:07:23 GMT

I spent some time experimenting with OpenDevin using claude-3-opus (I couldn’t find an easy way to use claude-3.5-sonnet). The agentic capabilities were not bad. I gave a prompt and behind the scenes, the agent iterated, created files, ran code and course corrected. I didn’t love that there wasn’t an obvious way to interrupt or help course correct. My first attempt was with the same prompt I sent to Sonnet to build Tactic. The result wasn’t bad. OpenDevin (running in a container, which makes me feel better) setup several files in a project then seemed to install dependencies, iterate on issues and inspect the site running in the browser. At least, this was what it claimed to be doing. It was hard to validate what was happening, especially because the tool’s built in browser never loaded anything. Maybe this was a bug or something unusual happening on the model side.

After this first attempt, I tried to be less prescriptive, hoping to do a bit more back and forth with the agent. After I submitted my first message, off the agent went. I tried to interrupt its progress by sending another message, but that didn’t seem to work. Eventually, it found a stopping point. It had created a Python version of the game, which was not what I wanted. I hadn’t had a chance to add any additional instructions. I could have followed up with more at this point, but we had already gone pretty far in the wrong direction.

Thanks for reading via RSS!

2024-06-24

Mon, 24 Jun 2024 21:25:08 GMT

I weirdly was running into an issue where whenever a ⌘F search didn’t return a result, my screen would flash white. It was irritating me for several days. Fortunately, I was able to find a solution that addressed it.

sudo killall coreaudiod

I was writing code with claude-3.5-sonnet and prompted it to add input validation for input arguments.

Most of the code what straight forward and I was expecting the one sentence prompt to get me 5-10 lines of code with path validation, I did not expect these lines.

    valid_models = ["llava", "llava-llama3", "moondream"]  # Add or modify as needed
    if model_name not in valid_models:
        print(f"Error: Invalid model name. Choose from: {', '.join(valid_models)}")
        sys.exit(1)

These surprised me specifically, because I had just installed these three models via the commands

ollama pull llava
ollama pull llava-llama3
ollama pull moondream

At first I thought this code must exist somewhere else online, but I couldn’t find after a bit of searching. Then I figured Sonnet must have been trained on the Ollama model page, but there are actually more vision models on that page that I hadn’t yet installed, like llava-phi3, so it didn’t seem obvious that Sonnet was trained on this data from the website.

I’ll need to dig a little deeper but I have to assume Cursor was sending my recent shell command history to the model along with file contents. It’s hard to imagine this works any other way.

Thanks for reading via RSS!

2024-06-23

Sun, 23 Jun 2024 12:36:59 GMT

I’m trying to avoid buying too much into the hype (maybe it’s too late), but here are several folks talking about their notably impressive experiences with claude-3.5-sonnet.

Thanks for reading via RSS!

2024-06-21

Fri, 21 Jun 2024 19:03:51 GMT

As I noted in this post, I am going to spend more time interacting with smaller models to try and build more intuition for how LLMs behave and the different flavors in which they respond. Today, I spent some time chatting with Microsoft’s phi-3 3B using ollama. In chatting with phi3, I found it neutral in tone and to the point. It responds in a way that is easy to understand and is not overly complex or technical at the onset. It doesn’t seem to have a strong sense of self.

When prompted about itself, it generally responds that it’s “an AI/large language model developed by Microsoft”. Often it will include that its name is “Phi”.

>>> who are you?
 I am Phi, an AI developed by Microsoft to assist users in generating human-like text
based on the prompts given to me. How can I help you today?

When you ask it for its name, rather than who it is, it seems to forget it has one.

>>> what is your name?
As an AI, I don't have a personal name, but you can call me Assistant. How can I help you today?

Once, it even thought it was ChatGPT:

>>> who are you?
I'm an AI developed by Microsoft, known as ChatGPT. [...]

I prompted it to have the following conversation about the future of technology in society where we would alternate asking each other questions. It quickly slipped into outputting long lists of ways governments or society could prepare. Occasionally, there were strange artifacts that showed up in the output like this:

1. <item 2>
2. Written by user:
3. <item 3>

Several times I asked the model not to respond with bullet points, but it really wanted to keep doing that. Adding

No lists. Respond in paragraphs.

to the end of my prompt seemed to help. Adding that to the beginning was less consistent. This result somewhat aligns with what I’ve seen with OpenAI models, where if you want JSON, adding

JSON:

to the end of the prompt is a pretty good way to get JSON output. Phi3 seems to quickly forget instructions and fall back to its defaults. This intuitively makes sense to me as it is a small model compared to what I typically work with.

I ran all the conversations through the shiny and new claude-3.5-sonnet to see if there was anything I missed in my summary above. It corroborated most of what I noticed, though it also added that the model seemed verbose “sometimes to the point of being overly wordy”. This exchange with Claude got me thinking about the different categories one might score a language model on in terms of the experience of chatting with it. Specifically, I would want to capture what it’s like to talk to the model. These categories would be more qualitative and not indicative of the model’s performance on benchmarks but rather a description of what the vibe is like when you interact with it.

I think there’s a lot of potential depth to describing models, so I am going to save that exploration for another time. I’ll conclude with the vibes I got from phi3. Since this is my first shallow dive for a model like this, I don’t have many other similar models to compare it to, so these may end up being a little unfair since most of my experience is with larger models. I’ll try to focus on what it’s like to talk to the model rather than “how good it is at things”.

Nevertheless, I would say phi3 is

verbose
formal
lacking a sense of self or identity
inconsistent in following instructions

Because it seemed a little unfair to just review phi3, I decided to chat a bit with llama3 8b as well. I immediately felt I was unfair to phi3. I think I prefer llama3’s tone but it’s response pattern was a bit repetitive:

a response to what you said
another side of the argument
its perceived role as LLaMA (“it’s my job to…”)

When I started a conversation, it didn’t follow my instructions about alternating asking questions and instead just kept asking questions. When I asked my own question, it’s response was in lists and very repetitive. Far less nice than anything phi3 said throughout our conversation.

When I asked llama3 about itself, it had more of a personality than phi3. It used analogies to describe what it was like to be a model where as phi3 responded more with facts about how models work.

I found this response particularly interesting:

Imagine being a vast library with an infinite shelf space, where every book represents a piece of knowledge, a concept, or a relationship between ideas. That's roughly what my "brain" is like – a massive database of text data that I've been trained on.

When you interact with me, it's as if you're browsing through these digital shelves, and I'm the librarian who can help you find specific books or recommend related ones based on your interests.

To summarize, I would say llama3 8b is

friendly
engaging
articulates a sense of self

Thanks for reading via RSS!

2024-06-20

Thu, 20 Jun 2024 20:40:58 GMT

I enabled Cursor’s Copilot++ today. Magical. So much better predictive capabilities than Copilot. The way it anticipates my needs is pretty cool. Edit: It’s not great for writing markdown or only prose.

I learned a bit more about Crafter, a Minecraft like game that language models can play.

Thanks for reading via RSS!

2024-06-19

Wed, 19 Jun 2024 22:25:55 GMT

I enjoyed reading Jordan’s post, a walk down memory lane of his career so far through a series of emails. He includes things like following up on internship opportunities, negotiating, and meeting people who would change the course of his career. He inspired me to look back through some old emails as well, both to remember this time and acknowledge how much has changed since then.

I managed to find my original offer letter from Uber in 2016, which brought back many memories that I might write about in longer form sometime in the future.

I strongly support Jordan’s suggestion about improving your negotiation skills. Negotiation is something that made me deeply uncomfortable in the past, but I’d like to believe it’s an area I’ve grown in.

Thanks for reading via RSS!

2024-06-14

Fri, 14 Jun 2024 21:14:09 GMT

For the first time in a while I used iTunes. I mean the Music app, sorry. I clicked on the album art while I was playing a song and the app switch to mini-player mode. I…didn’t see what I could click to get back to the main player. From my perspective the app had just shrunken down to the mini-player. I hesitantly clicked the lyrics button. Yeah, definitely not that. I clicked the red button of the traffic lights. Ah. Has this always been this way? It’s been a while.

Thanks for reading via RSS!

2024-06-13

Thu, 13 Jun 2024 09:44:17 GMT

Eugene’s article on prompting is one of the best things I’ve read recently, full stop. As also noted by Kyle, try starting with 0.8 temperature and lower only if necessary, even for deterministic use cases^[1] ^[2].

Thanks for reading via RSS!

2024-06-12

Wed, 12 Jun 2024 21:07:59 GMT

A few years old, excellent response written by Maxim about extracting the most value from Temporal by using it “as a service mesh for invocations of child workflows and activities hosted by different services”.

Thanks for reading via RSS!

2024-06-11

Tue, 11 Jun 2024 08:12:20 GMT

I found a very satisfying, time-saving use of a model though Cursor to generate an inline shell script to reformat some data for me. The goal was to reformat json files that looked like this

[...]

to ones that look like this

{"cells" [...]}

I used the following prompts:

for each json file in the current folder, read the json (which is a json list) and turn it into a json object with a single key, “cells” whose value is the json list

the final files should have the same names as the originals

The model output this script

for file in *.json; do jq '{cells: .}' "$file" > "temp_$file" && mv "temp_$file" "$file"; done

and that was it.

30 seconds, problem solved. One minor concern I have is since I didn’t have to research or deeply understand the code being run, I probably won’t remember this trick for next time. I suppose if I might not remember even if I did do the research if I only needed to do this one time.

I don’t know much about chess other than basic rules and I haven’t played much either. Regardless, I enjoyed this video by Paralogical on analyzing billions of games to discover “rare” moves.

Thanks for reading via RSS!

2024-06-10

Mon, 10 Jun 2024 21:44:51 GMT

So Apple has introduced a new system called “Private Cloud Compute” that allows your phone to offload complex (typically AI) tasks to specialized secure devices in the cloud. I’m still trying to work out what I think about this. So here’s a thread. 1/
— Matthew Green (@matthew_d_green) June 10, 2024

Matthew wrote a thread summarizing Apple’s private cloud and their security approach. Between the speed at which models are changing and their size, it’s not currently practical to run things like LLM inference on iPhones, at least not for the best available models. These models will probably always be larger or require more compute than what handheld devices are capable of. If you want to use the best models, you’ll need to offload inference to a data center.

There is plently more in the press release.

Our on-device model, with ~3B parameters, outperforms larger models including Phi-3-mini, Mistral-7B, and Gemma-7B. Our server model compares favorably to DBRX-Instruct, Mixtral-8x22B, and GPT-3.5-Turbo while being highly efficient.

It sounds like the options will be

a close-to-be-in-class small model on device
private inference with a model that is (maybe?) about 12 months behind best in class models
farm out the task to OpenAI to get a response from the best model available

Thanks for reading via RSS!

2024-06-05

Wed, 05 Jun 2024 21:57:31 GMT

I spent some time moving my bookmarks and workflows to Raindrop from Pocket. I’m mostly done with a short post on how and why I did that. I’d like to use the python-raindropio package to generate stubs for logs posts from links I bookmark, including comments I write about them. The Nix flake I use for this blog hampered those efforts. The python-raindropio package isn’t available as a Nix package and I’m not particularly well versed in building my own packages with Nix, so that became a bit of a rabbit hole. After poking around for other available libraries, it turns out, this one isn’t even maintained anymore. It was forked as raindrop-io-py and it looks like that might not even be maintained anymore. Not the greatest sign, but the package does still work for the time being.

Good reads

https://mbuffett.com/posts/maintaining-motivation: A relatable article by Marcus on how he keeps his motivation up working on projects as a solo developer.

Thanks for reading via RSS!

2024-06-01

Sat, 01 Jun 2024 18:34:22 GMT

I was away from the computer for a couple of weeks. That was really nice. During my downtime and in transit, these were some of my favorite things I read in the past two weeks. I’m planning to come up with a bit of a better system for logging and adding thoughts to stuff I read but for now a list will have to do.

Thanks for reading via RSS!

2024-05-17

Fri, 17 May 2024 19:02:25 GMT

Sabrina wrote an interesting write up on solving a math problem with gpt-4o. It turned out the text-only, chain-of-thought approach was the best performing, which is not what I would have guessed.

It’s was cool to see Simon dive into LLM-driven data extraction in using his project datasette in this video. Using multi-modal models for data extraction seems to bring a new level of usefulness and makes these models even more general purpose.

Thanks for reading via RSS!

2024-05-15

Wed, 15 May 2024 23:25:14 GMT

Nostalgia: https://maggieappleton.com/teenage-desktop.

I wish I had done something like this.

Maybe I can find something on an old hard drive.

Thanks for reading via RSS!

2024-05-10

Fri, 10 May 2024 20:33:38 GMT

I’m looking into creating a Deno serve that can manage multiple websocket connections and emit to one after receiving a message from another. A simple way to implement this is to have a single server id and track all the ongoing connections to websocket clients. I’m learning more about approaches that could support a multi-server backend.

Thanks for reading via RSS!

2024-05-09

Thu, 09 May 2024 13:50:00 GMT

I take an irrational amount of pleasure in disabling notifications for apps that use them to send me marketing.

Thanks for reading via RSS!

2024-05-08

Wed, 08 May 2024 08:56:42 GMT

I enjoyed reading Yuxuan’s article on whether Github Copilot increased their productivity. I personally don’t love Copilot but enjoy using other AI-assisted software tools like Cursor, which allow for use of more capable models than Copilot. It’s encouraging to see more folks adopting a more unfiltered thought journal.

Thanks for reading via RSS!

2024-05-07

Tue, 07 May 2024 22:07:22 GMT

I read this post by Steph today and loved it. I want to try writing this concisely. I imagine it takes significant effort but the result are beautiful, satisfying and valuable. It’s a privilege to read a piece written by someone who values every word.

Thanks for reading via RSS!

2024-05-06

Mon, 06 May 2024 20:21:03 GMT

llama 3-400B with multimodal capabilities and long context would put the nail in the coffin for OAI
— anton (@abacaj) May 6, 2024

Having gotten more into using llama 7b and 30b lately, this take seems likes it could hold water. Model inference still isn’t free when you scale a consumer app. Maybe I can use llama3 for all my personal use cases, but I still need infra to scale it. The price probably goes down significantly though with so many model inference providers and the speed will go way up once Groq starts running it (if they can run multi-modal models).

Thanks for reading via RSS!

2024-05-03

Fri, 03 May 2024 17:50:29 GMT

I read Jason, Ivan and Charles’ blog post on Modal about fine tuning an embedding model. It’s a bit in the weeds of ML for me but I learn a bit more every time I read something new.

Thanks for reading via RSS!

2024-05-01

Wed, 01 May 2024 21:56:30 GMT

I played around with trying to run a Temporal worker on Modal. I didn’t do a ton of research upfront — I just kind of gave it a shot. I suspect this isn’t possible. Both use Python magic to do the things they do. This is what I tried.

import asyncio
import os
import modal
from temporalio import activity, workflow
from temporalio.client import Client, TLSConfig
from temporalio.worker import Worker

@activity.defn
async def my_activity(name: str) -> str:
    return f"Hello, {name}!"

@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            my_activity, name, start_to_close_timeout=60
        )

async def worker_main():

    client = await Client.connect(
        "my.namespace.tmprl.cloud:7233",
        namespace="my.namespace",
        tls=TLSConfig(
            client_cert=bytes(os.environ["TEMPORAL_CLIENT_CERT"], "utf-8"),
            client_private_key=bytes(os.environ["TEMPORAL_CLIENT_KEY"], "utf-8"),
        ),
    )
    worker = Worker(
        client,
        task_queue="modal-task-queue",
        workflows=[MyWorkflow],
        activities=[my_activity],
    )
    await worker.run()


stub = modal.Stub("temporal-worker")

@stub.function(
    image=modal.Image.debian_slim().pip_install(
        [
            "temporalio==1.5.1",
        ]
    ),
    secrets=[modal.Secret.from_name("modal-temporal-worker")],
)
def main():
    asyncio.run(worker_main())

if __name__ == "__main__":
    with stub.run():
        main.call()

Run with

modal run worker::main

The error trace looked like

    RESPONSES: Mapping[int, Tuple[str, str]] = http.server.BaseHTTPRequestHandler.responses
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/temporalio/worker/workflow_sandbox/_restrictions.py", line 845, in __getattribute__
    state.assert_child_not_restricted(__name)
  File "/usr/local/lib/python3.11/site-packages/temporalio/worker/workflow_sandbox/_restrictions.py", line 697, in assert_child_not_restricted
    raise RestrictedWorkflowAccessError(f"{self.name}.{name}")
temporalio.worker.workflow_sandbox._restrictions.RestrictedWorkflowAccessError: Cannot access http.server.BaseHTTPRequestHandler.responses from inside a workflow. If this is code from a module not used in a workflow or known to only be used deterministically from a workflow, mark the import as pass through.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/pkg/modal/_container_io_manager.py", line 488, in handle_input_exception
    yield
  File "/pkg/modal/_container_entrypoint.py", line 134, in run_input
    res = imp_fun.fun(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/worker.py", line 52, in main
    asyncio.run(worker_main())
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/root/worker.py", line 32, in worker_main
    worker = Worker(
             ^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/temporalio/worker/_worker.py", line 292, in __init__
    self._workflow_worker = _WorkflowWorker(
                            ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/temporalio/worker/_workflow.py", line 118, in __init__
    raise RuntimeError(f"Failed validating workflow {defn.name}") from err
RuntimeError: Failed validating workflow MyWorkflow
╭─ Error ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Failed validating workflow MyWorkflow                                                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

I believe Temporal needs to patch the Python runtime in Worker executions to make Workflow code deterministic. It seems what Modal does may be interfering with that.

I don’t know if running a Temporal worker on Modal is possible or if I will trying to come back to this by it was interesting trying to mash these two together.

Thanks for reading via RSS!

2024-04-25

Thu, 25 Apr 2024 13:54:15 GMT

I read this interesting article by Gajus about finetuning gpt-3.5-turbo. It was quite similar to my experience fine tuning a model to play Connections. A helpful takeaway was that after finetuning the model, you shouldn’t need to include system prompt in future model inference, so you can save on token cost. I also liked the suggestion to use a database to store training data. I had also been wrangling jsonl files.

Thanks for reading via RSS!

2024-04-22

Mon, 22 Apr 2024 12:57:12 GMT

About a month ago, I had been looking into creating a NL to SQL plugin for datasette. Simon release a version of exactly that the next day and I came across it in his article here. Hopefully I can find time to try this out in the next few days.

Thanks for reading via RSS!

2024-04-19

Fri, 19 Apr 2024 14:55:12 GMT

I did a refactor of my nix config following a pattern I learned from reading Davis’ setup. My two main uses right now for Nix/home-manager are to install and configure programs. Some of these programs have nix modules that allow for the configuration to be written in Nix. Others don’t, but you can still use Nix to create a config file for that program to read. I do the latter with skhd and goku to create a karabiner.json. With this refactor, I used the default.nix file to create program-specific module imports. I refactored my home.nix to use the same approach as well. This allows me to easily co-locate code to set up a given program, regardless of whether I am configuring it with Nix or by creating dotfiles.

Thanks for reading via RSS!

2024-04-17

Wed, 17 Apr 2024 07:00:06 GMT

For me, invoking a language model using a playground (UI) interface is the most common approach for my usage. Occasionally, it can be helpful to use the a CLI to directly pipe output into a model. For example

git diff --staged | llm "write a commit message for these changes"

However, I am more often inclined to open a playground and paste the bits and pieces of context I need. Maybe, it’s that refinement and followups are common enough that using a CLI isn’t nearly as flexible. The bottom line is, I far more frequently open a playground to use a language model than use a CLI. Even though most of the playgrounds have various weird and annoying behaviors, I generally still prefer them.

I tried the Gemini 1.5 multi-modal model preview. Unfortunately, I wasn’t super impressed relative to gpt-4-turbo.

This evening has been most spent making mistakes, trying to deploy a few different Modal functions and call these in a Cron. I thought I had been close to getting this working when I left off yesterday, but now I was getting stuck on errors like the following

modal.exception.ExecutionError: Object has not been hydrated and doesn't support lazy hydration. This might happen if an object is defined on a different stub, or if it's on the same stub but it didn't get created because it wasn't defined in global scope.

Eventually, I found an approach in the Modal Community Slack channel that helped me resolve my issue. Side note: I wish this content was more easily searchable publicly. The content is so valuable but it’s locked away in Slack. I needed to use Function.lookup to reference functions that are deployed separately and defined in other stubs. Once I got this working, I was able to launch Bots Doing Things, a site where my bots perform tasks and publish their results. Currently, I have just one “bot”, which plays Connections once a day using gpt-4 and records its result in a git repo, which is then deployed to a site automatically on Vercel. The daily cron and Python code run on Modal.

Thanks for reading via RSS!

2024-04-13

Sat, 13 Apr 2024 15:50:13 GMT

I enjoyed this article by Ken about production LLM use cases with OpenAI models.

When it comes to prompts, less is more

This resonated with me. I’ve found that too much instruction can lead a model to perform worse on a task.

GPT is really bad at producing the null hypothesis

This also seems to confirm what I’ve seen empirically, but I never ask for it. I ask for something like, “return an empty JSON array if you can’t find anything”.

GPT really cannot give back more than 10 items. Trying to have it give you back 15 items? Maybe it does it 15% of the time.

I didn’t notice this and haven’t tried it explicitly. I did a bit of recipe ingredients extraction where there were more than 10 ingredients and that seems to work okay. I also recently did another use case where I’m extracting a variable number of items and extracting 10 items seems to work consistently, but I am just waving my hands. I need to collect data to confirm.

vector databases, and RAG/embeddings are mostly useless for us mere mortals

This surprised me. I’ve found embeddings to be useful, though I can’t say I’ve gone to production with a RAG use case. I agree that putting your embeddings in a specialized database doesn’t make sense. I’ve done most of my exploration storing embeddings in SQLite.

Ken’s article also linked to The Bitter Lesson by Rich. He lays out what he believes to be the biggest lesson learned in the history of AI

general methods that leverage computation are ultimately the most effective, and by a large margin.

One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.

A general note on my experience with Nix: most things have taken multiple attempts, but persistence has allowed me to eventually find a workable solution for the problems I’ve faced. I love the combo of direnv and flakes to build a directory-specific environment. It seems a number of dev tools are popping up that make a very friendly interface to this functionality and use Nix underneath to make it happen. I used one called devbox to play around with postgres and recently learned of another called devenv.

Thanks for reading via RSS!

2024-04-10

Wed, 10 Apr 2024 21:28:44 GMT

I enjoyed Martin’s article on preserving your shell history. I implemented some of his approaches in my system config.

Thanks for reading via RSS!

2024-04-09

Tue, 09 Apr 2024 22:16:44 GMT

Gemini Pro 1.5 up and running. I’ve said this before but I will say it again — the fact that I don’t need to deal with GCP to use Google models gives me joy.

❯ llm -m gemini-1.5-pro-latest "who is the fastest man in the world?"
As of November 2023, **Usain Bolt** is still considered the fastest man in the world. He holds the world record in the 100 meters with a time of 9.58 seconds, set in 2009.  He also holds the record for the 200 meters at 19.19 seconds, achieved in 2009 as well.

Having all these models readily available is great. My hope is to play around with several to become a bit of an amateur model sommelier.

Thanks for reading via RSS!

2024-04-04

Thu, 04 Apr 2024 19:22:11 GMT

Today, I learned about Command-R model series from Cohere from Shawn’s great AI newsletter (ainews). I searched to see if a plugin was available for llm and Simon had literally authored one 8(!) hours earlier. Folks like you keep me inspired and motivated 🙏. No better workflow out there that I know of:

llm install llm-command-r

llm -m r-plus hello
Error: No key found - add one using 'llm keys set cohere' or set the COHERE_API_KEY environment variable

llm keys set cohere
Enter key: ...

llm -m r-plus "hi, who am I speaking with?"
You are speaking with Coral, an AI chatbot trained to assist users by providing thorough responses. How can I help you today?

Thanks for reading via RSS!

2024-04-03

Wed, 03 Apr 2024 21:07:23 GMT

A great article by Manuel about forever-growth of companies. I too wish we’d be more willing to celebrate enough.

Thanks for reading via RSS!

2024-03-31

Sun, 31 Mar 2024 10:05:40 GMT

I’ve been digging more into evals. I wrote a simple Claude completion function in openai/evals to better understand how the different pieces fit together. Quick and dirty code:

from anthropic import Anthropic

from evals.api import CompletionFn, CompletionResult
from evals.prompt.base import is_chat_prompt


class ClaudeChatCompletionResult(CompletionResult):
    def __init__(self, response) -> None:
        self.response = response

    def get_completions(self) -> list[str]:
        return [self.response.strip()]

class ClaudeChatCompletionFn(CompletionFn):
    def __init__(self, **kwargs) -> None:
        self.client = Anthropic()

    def __call__(self, prompt, **kwargs) -> ClaudeChatCompletionResult:
        if is_chat_prompt(prompt):
            messages = prompt
            system_prompt = next((p for p in messages if p.get("role") == "system"), None)
            if system_prompt:
                messages.remove(system_prompt)
        else:
            # I think there is a util function to do this already
            messages = [{
                "role": "user",
                "content": prompt,
            }]
        message = self.client.messages.create(
            max_tokens=1024,
            system=system_prompt["content"] if system_prompt else None,
            messages=messages,
            model="claude-3-opus-20240229",
        )
        return ClaudeChatCompletionResult(message.content[0].text)

claude/claude-3-opus:
  class: evals.completion_fns.claude:ClaudeChatCompletionFn
  args:
    completion_fn: claude-3-opus

Run with

oaieval claude/claude-3-opus extraction

on a toy eval I wrote.

{"input": [{"role": "system", "content": "You are responsible for extracting structured data from the provided unstructured data. Follow the user's instructions and output JSON only without code fences."}, {"role": "user", "content": "CONTENT: I live at 42 Wallaby Way, Sydney\nINSTRUCTIONS: extract street and city"}], "ideal": "{\"street\": \"42 Wallaby Way\",\"city\": \"Sydney\"}"}
{"input": [{"role": "system", "content": "You are responsible for extracting structured data from the provided unstructured data. Follow the user's instructions and output JSON only without code fences."}, {"role": "user","content": "CONTENT: My favorite color is blue and I was born on June 15, 1985.\nINSTRUCTIONS: extract favorite color and date of birth. format date of birth as yyyy-mm-dd"}], "ideal": "{\"favorite_color\": \"blue\",\"date_of_birth\": \"1985-06-15\"}"}

extraction:
  id: extraction.test.v0
  metrics: [accuracy]

extraction.test.v0:
  class: evals.elsuite.basic.json_match:JsonMatch
  args:
    samples_jsonl: extraction/samples.jsonl

It seems this project is moving away from the “Completion Functions” abstraction to “Solvers”.

[W]e’ve found that passing a prompt to the CompletionFn encourages eval designers to write prompts that often privileges a particular kind of Solver over others. e.g. If developing with ChatCompletion models, the eval tends to bake-in prompts that work best for ChatCompletion models. In moving from Completion Functions to Solvers, we are making a deliberate choice to write Solver-agnostic evals, and delegating any model-specific or strategy-specific code to the Solver.

In working through this exercise, a thought that came to mind often is how many different approaches we currently have for model prompting, both because of model differences (completion vs. chat) but also API design decisions. To allow for easy switching between models, using a gateway/adapter pattern to support mapping from the model/provider API to your applications’s internal API will be as critical as ever. This approach may be further complicated if your application relies on streaming responses. It seems as important as ever to use abstractions to decouple yourself from provider APIs to remain flexible to adopting future advances in models and keep your switching cost low.

Thanks for reading via RSS!

2024-03-28

Thu, 28 Mar 2024 08:48:00 GMT

I can’t believe I am saying this but if you play around with language models locally, a 1 TB drive, might not be big enough for very long.

Thanks for reading via RSS!

2024-03-26

Tue, 26 Mar 2024 19:21:48 GMT

As someone learning to draw, I really enjoyed this article: https://maggieappleton.com/still-cant-draw. I’ve watched the first three videos in this playlist so far and have been sketching random objects from around the house. I find that I’m not too big of a fan of my drawing as I’m doing it but when I return to it later, I seem to like it more. Apparently, this is a common experience for creatives.

Added GoatCounter to my site. I’m planning to see how I like it compared to Posthog.

Thanks for reading via RSS!

2024-03-25

Mon, 25 Mar 2024 21:07:54 GMT

I’m taking a break from sketchybar for now.

I’m currently looking into build a NL to SQL plugin or addition to datasette to use a language model to write queries.

Thanks for reading via RSS!

2024-03-24

Sun, 24 Mar 2024 08:48:52 GMT

🤖 Connections (claude-3-opus)
Puzzle #287
🟩🟩🟩🟩
🟨🟨🟨🟨
🟦🟦🟦🟦
🟪🟪🟪🟪

I got this result twice in a row.

gpt-4 couldn’t solve it. Here is one attempt.

🤖 Connections (gpt-4)
Puzzle #287
🟩🟪🟩🟩
🟩🟩🟩🟩
🟦🟨🟦🟨
🟦🟨🟦🟨
🟦🟨🟦🟦

I tried https://echochess.com/. Kind of fun.

I remember when my highschool teachers used to tell us Wikipedia wasn’t a legitimate source. It sort of feels like education is having this type of moment now with language models. Professionals are using this technology today to do work in the real work. Learn how the technology works, teach it and teach with it.

I spent a bunch more time using sketchybar. It’s a fun mashup of shell and cli tools. I have a working MVP that I’m reasonably happy with and plan to iterate on. Configuring this type of thing is fun but can take a lot of time.

Thanks for reading via RSS!

2024-03-23

Sat, 23 Mar 2024 08:47:09 GMT

One of the greatest misconceptions concerning LLMs is the idea that they are easy to use. They really aren’t: getting great results out of them requires a great deal of experience and hard-fought intuition, combined with deep domain knowledge of the problem you are applying them to.

The whole “LLMs are useful” section hits for me. I have an experience similar to Simon’s and I also wouldn’t claim LLMs are without issue or controversy. For me they are unquestionably useful. They help me more quickly and effectively get ideas out of my head and into the real world. They help me learn more quickly and solve problems and answer questions as I’m learning. They increase my capabilities as an individual in the same sort of way getting access to Google for the first time did. These days, not having access to a language model makes me feel like I’ve had an essential tool taken away from me, like not having a calculator or documentation.

https://simonwillison.net/2024/Mar/22/claude-and-chatgpt-case-study/

The models claude-3-opus, claude-3-sonnet and gpt-4 solved Connections today on their first tries today. All but opus failed on subsequent tries.

gpt-3.5.turbo doesn’t seem to understand it is supposed to be playing the game, not me. groq-llama-2 got just 1/4 correct. mistral-large got 2/4.

As an intermediate player of this game, I would say today’s puzzle was on the easier side. The models don’t usually do so well.

BOOX Palma is a a mobile phone that uses an E Ink display. I immediately wanted this when I saw it.

While working on a script that evaluates language models’ ability to play Connections, I was using Cursor to make some improvements to the code. On a whim, I decided to have it swap all my convenience print statements to log statements. I’ve done this on a smaller scale but with a ~200 line file, practically, I knew it would probably take the model longer to do than for me to do myself. As I watched Cursor regenerate my file via inference and show a diff, I was struck by the idea that running complex, multiple passes of LLMs on code bases with target instructions may become the new “my code is compiling”.

An infinitely zoomable, Game of Life, simulating itself. The simulation adjusts speed as you zoom so that each level moves at a perceptible speed, allowing you to really appreciate it. Extremely well executed.

https://oimo.io/works/life/

Thanks for reading via RSS!

2024-03-22

Fri, 22 Mar 2024 23:30:42 GMT

Did a bit more work on a LLM evaluator for connections. I’m mostly trying it with gpt-4 and claude-3-opus. On today’s puzzle, the best either did was 2/4 correct. I’m unsure how much more improvement is possible with prompting or even fine tuning, but it’s an interesting challenge.

Darwin, who kept a notebook where he wrote down facts that contradicted him, observed that frustrating, cognitively dissonant things were the first to slip his memory.

https://www.henrikkarlsson.xyz/p/limitatons

I subscribed to AI News, which summarize top subreddits, Discords and Twitter. It’s quite extensive and detailed.

Thanks for reading via RSS!

2024-03-21

Thu, 21 Mar 2024 21:03:43 GMT

Setup a Temporal worker in Ruby and got familiar with its ergonomics.

Tried out this gpt-4v demo repo Experimented with OCR capabilities of open source multi-modal language models. Tried llava:32b (1.6) and bakllava but neither seemed to touch gpt-4-vison’s performance. It was cool to see the former run on a macbook though.

Thanks for reading via RSS!

2024-03-18

Mon, 18 Mar 2024 20:15:00 GMT

I use the hyper+u keyboard shortcut to open a language model playground for convenience. I might use this 10-20 times a day. For the last year or so that I’ve been doing this, it has always pointed to https://platform.openai.com/playground. As of today, I’ve switched it to point to https://console.anthropic.com/workbench?new=1. Lately, I’ve preferred claude-3-opus to gpt-4. For a while, I had completely stopped looking for other models as gpt-4 seemed to be unchallenged, but it’s exciting to see new options available.

Thanks for reading via RSS!

2024-03-16

Sat, 16 Mar 2024 10:15:01 GMT

I tried setting up sqlite-vss with Deno following these instructions but got stuck on this error

❯ deno task dev
Task dev deno run --allow-env --allow-read --allow-write --allow-net --unstable-ffi --allow-ffi --watch main.ts
Watcher Process started.
error: Uncaught (in promise) TypeError: readCstr is not a function
export const SQLITE_VERSION = readCstr(sqlite3_libversion());
                              ^
    at https://deno.land/x/sqlite3@0.8.0/src/database.ts:101:31
    at eventLoopTick (ext:core/01_core.js:169:7)

so I pivoted to Python. That effort eventually turned into this post.

Thanks for reading via RSS!

2024-03-15

Fri, 15 Mar 2024 08:48:25 GMT

I’ve spend almost a week, on and off, trying to install ollama using Nix in such a way that ollama serve will be run and managed automatically in the background. Initially, I had tried to install ollama via home-manager. This was straightforward, but finding a way to have ollama serve run automatically so that I didn’t need to do it myself every time I wanted to interact with or pull a model. This need send me down that macOS rabbithole of launch agents/daemons and plists. The macOS-specific nature of this research could have tipped me off that this was a job for nix-darwin rather than home-manager. Eventually, I tried installing ollama via nix-darwin and it seemed to Just Work. I still haven’t been able to figure out where it’s running ollama serve from.

❯ grep -l 'ollama' ~/Library/LaunchAgents/* /Library/LaunchAgents/* /Library/LaunchDaemons/* /System/Library/LaunchAgents/* /System/Library/LaunchDaemons/*

# nothing

Nix-darwin appears to have several options for configuring launch agents that I may need to return to if this behavior doesn’t persist across a restart.

I learn about this game by Neal a little while and I love it. I think it’s a language-model-based craft game. Something like a game I played a long time ago called “Alchemy”.

Thanks for reading via RSS!

2024-03-14

Thu, 14 Mar 2024 18:50:37 GMT

I spent some time exploring Deepgram’s Next.js starter app. I was hoping I could use it to generate a transcription in realtime but it was more like real-time captions. The responses from the server were sometimes corrections of previous transcriptions. Maybe there is a way to make this a transcription but I wasn’t sure.

I also tried out vocode’s Python library for building voice-based LLM applications. By far the hardest part of this was getting an Azure account, which I believe is used to synthesize the LLM response as speech. I had the demo working end to end but it was a bit sensitive to background noise as they note. I haven’t had a chance to play around with any configurations they provide.

Thanks for reading via RSS!

2024-03-11

Mon, 11 Mar 2024 17:46:29 GMT

I played around with AgentGPT using Reworkd’s cloud hosted instance. I tried a few different goals. The first was travel related. I was pleasantly surprised (unnerved) to see the agent return links for flights to my proposed location from my actual current origin (I guess they used my IP to determine this). One of my first impressions was the likeness to Perplexity, particularly in how the links were displayed. It turns out, Reworkd open sourced code for this exact thing. AgentGPT also uses inline citations like Perplexity does, though those aren’t shown in the perplexity-style-streaming demo video.

I read up a bit on managing windows with yabai’s tiled layout

yabai -m config layout bsp

I read through these two helpful articles on the topic

It’s not for me. My approach is a bit more flexible but makes more sense in my brain than this does. I don’t really mind having windows open that are fully covered. Generally, I have two windows in a 50-50 split on my large monitor and a single window maximized on my laptop screen. For me, tiled layout seems more complicated than it is worth.

I learned about https://devdocs.io. It’s quite interesting and useful, but the search functionality doesn’t quite work like a search engineer in terms of surfacing the relevant issues for your problem. It seems to work best when you enter the specific library, class, or API you are working with. It would be interesting to see if this could be indexed for RAG then used as a resource by a model.

My love-hate relationship with Nix continues. When it works, it’s amazing but I’m still scaling the learning curve of the language and its rough edges.

Python dependency management system is the wild west. I manage Python versions with asdf on one machine. When a project uses poetry and relies on poetry’s virtualenv to install dependencies, selecting the right version (on the system) of poetry and getting around the asdf becomes a pain. I like asdf because I don’t want to manage a version tool per language, yet many of these languages and tools more or less force you to do just that or struggle to juggle multiple versions of things while you try and bootstrap your way to a working environment.

Thanks for reading via RSS!

2024-03-08

Fri, 08 Mar 2024 19:15:33 GMT

I tried to get claude-3-opus to solve today’s Connections. It did well enough that I am going to try doing this more consistently.

I grew tried of Nix and decided to setup Homebrew as well. I finally gave in after trying to install llm and not being able to figure out how to install plugins despite the message

If you want to install extra llm plugins, use llm.withPlugins([]) expression.

In tracing this error, I also learning nixpkgs can apply patches to libraries in its registry.

I never figured out how to make this work. Instead, I installed Homebrew which allowed me to install things the standard way

llm install llm-claude-3

Thanks for reading via RSS!

2024-03-07

Thu, 07 Mar 2024 20:36:05 GMT

I like Nix a lot. I still don’t understand a lot about it. I cannot lose an hour to installing a dependencies when that is a solved problem using Homebrew.

Ok fine, I will manage Homebrew with nix.

Thanks for reading via RSS!

2024-03-06

Wed, 06 Mar 2024 21:43:34 GMT

Ideal bookmarking from Swyx on Latent Space with Soumith Chintala.

create synthetic data off of your retrieved documents and then fine tune on that

Today I installed Homebrew. It’s the longest I have ever gone since I learned to code without installing Homebrew on my system. I’ve been able to rely on Nix for so much, but it’s been time consuming to get everything the way I want it, and I don’t want perfect to be the enemy of good. I can and plan to use brew sparingly in the name of reproducibility.

Thanks for reading via RSS!

2024-03-04

Mon, 04 Mar 2024 20:18:14 GMT

I was using an LLM to assist when playing with Alacritty and tmux configs. The LLM (gpt-4-turbo) gave some amusing advice

Embrace the Limitation

In some cases, the minor visual imperfections may be something to adjust to. Given tmux’s text-based nature and dependency on terminal emulator behavior, some display quirks can be challenging to eliminate entirely.

Thanks for reading via RSS!

2024-02-28

Wed, 28 Feb 2024 09:09:46 GMT

While Karabiner/Goku can be a fun way to try and configure my keyboard as an app launcher, I’ve found skhd to be much more straightforward. I’ll probably keep Karabiner for the caps lock to hyper key mapping only for now.

Thanks for reading via RSS!

2024-02-25

Sun, 25 Feb 2024 13:19:49 GMT

Github Code Search is an indispensable part of my workflow that I don’t think gets mentioned enough. There are so many great projects out there that solve or may help you solve what you are currently working on. I was looking to write a shell script to invoke the caffeinate command on macOS to keep my system awake. The default, this tool runs an blocks forever in the terminal while it does its job. I wanted to manage it in the background. I thought through the beginnings of how I might do that myself. Something like

ps aux | grep caffeinate
# if running, kill pid
# if not running, start

I plugged ps aux | grep caffeinate into Code Search and found several different approaches for what I had in mind. A language model does pretty well too.

#!/bin/bash

# This script toggles the 'caffeinate' command on macOS.

# Check if caffeinate is running
pid=$(pgrep caffeinate)

if [ -z "$pid" ]; then
  # If caffeinate is not running, start it
  caffeinate &
  echo "caffeinate started"
else
  # If caffeinate is running, kill it
  kill $pid
  echo "caffeinate stopped"
fi

I suppose the language model wins today, for introducing me to pgrep.

I settled on the following because I like one-liners:

pgrep caffeinate > /dev/null && kill $(pgrep caffeinate) || caffeinate -dim &

Thanks for reading via RSS!

2024-02-24

Sat, 24 Feb 2024 10:18:57 GMT

Fell down the system customization rabbit hole. Looking at switching to Allacrity or kitty as a terminal emulator. Getting goku setup as an app and hotkey launcher on top of Karabiner. Playing with yabai as a window manager. Looking at sketchybar as a custom status bar. Trying to manage it all with nix-darwin and home-manager.

Thanks for reading via RSS!

2024-02-21

Wed, 21 Feb 2024 08:05:44 GMT

I started to play around a bit with trying to setup a Temporal worker using Deno, but quickly ran into issues, then learned this isn’t currently possible.

Thanks for reading via RSS!

2024-02-19

Mon, 19 Feb 2024 10:33:00 GMT

I enjoyed reading James’ post on 100 things you can do on your personal website. I like the idea of adding a sparkline and maybe webmentions.

Thanks for reading via RSS!

2024-02-16

Fri, 16 Feb 2024 07:43:14 GMT

A very timely (for me) article by Hamel about understanding what a language model prompt abstraction library is doing before blindly adopting it. This really aligned with a lot of my own thoughts on the matter, right down to it’s praise of Jason’s instructor library baseline example.

Thanks for reading via RSS!

2024-02-05

Mon, 05 Feb 2024 21:30:00 GMT

After playing around with home-manager for a bit, I realized what I really wanted was to have nix-darwin manage my system and use my home-manager config as a module. It took a bit of trial and error and searching to figure out how to make this work, but I did eventually find a way. With nix-darwin, I set up my nix and macOS settings. With home-manager, I installed my system packages (configured those) and shell aliases. I still need to do some work to set up git, my shell PS1, and to see if I can configure iTerm from here, but it’s the most promising progress I’ve had so far.

Thanks for reading via RSS!

2024-02-04

Sun, 04 Feb 2024 18:39:00 GMT

I pivoted to using home-manager using the standalone installation. It seems to provide a reasonable starting point and a found a fair bit of prior art when searching Github, so my hope was there would be good examples to model off of. It took me a frustrating time to realize after running home-manager switch that only some changes took effect immediately. Knowing this probably would have saved me an hour and could have possibly solved a number of my problems from yesterday as well — I don’t have a quick way to check.

Thanks for reading via RSS!

2024-02-03

Sat, 03 Feb 2024 21:01:00 GMT

Today, was a first day setting up a new computer. I spent most of the time installing applications and building up my setting with declarative configurations. This site has been helpful. I also started this walkthrough for setting up nix on a Mac.

I had to temporarily disable my nix.conf that was generated from the installer then run

nix --extra-experimental-features nix-command --extra-experimental-features flakes run nix-darwin -- switch --flake .

to run the flake.nix file.

I ended up uninstalling nix-darwin after there were several issues I encountered following the instructions on a brand new Mac (even though this article was written a month ago). This beta vibe has continued to be my experience with Nix, though each time I return to it, I feel a get a bit more insight into what is happening and how it can be useful. Unfortunately, the error messages are also difficult to troubleshoot with search engines or language models.

Thanks for reading via RSS!

2024-01-31

Wed, 31 Jan 2024 11:19:02 GMT

A new thing I am trying is sending thanks to folks who write articles or build projects that I find useful. I got a taste of this after writing an article on fine-tuning gpt-3.5 to solve the Connections word game, even though the results didn’t turn out that well. Getting the positive feedback was quite motivating and my hope is to give others the same appreciation for the positive impact their work makes on me.

Thanks for reading via RSS!

2024-01-28

Sun, 28 Jan 2024 19:20:12 GMT

I’m currently working on building a language model based chatbot that can answer questions about the contents of a database. There are a lot of products and libraries making efforts at this problem. To start, I tried out the Vanna.ai open source library. I followed this guide to get started with ChromaDB for the indices and OpenAI as the language model to query a Postgres database. I also set up a Postgres database with Docker and the Chinook dataset. I downloaded the Chinook dataset for Postgres from this repo. The dataset is described in detail here. To start up Docker and load the data, I ran the following from my host machine (not inside a Docker container)

# start postgres
docker run --name vanna-postgres -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres

# create the database
createdb -h localhost -p 5432 -U postgres chinook

# load the data
psql -h localhost -p 5432 -U postgres chinook -1 -f ~/path.to/chinook_pg_serial_pk_proper_naming.sql

Vanna indexes the database content into, then can take a question and convert it into a SQL query. Finally, it executes a SQL query and returns the results as a dataframe.

Thanks for reading via RSS!

2024-01-26

Fri, 26 Jan 2024 20:37:31 GMT

For several days now, I’ve been looking into recording audio in a browser and streaming it to a backend over a websocket with the intent to do speech to text translation with an AI model. I know the pieces are all there and I’ve done something like this before (streamed audio from a Twilio IVR to a node backend, the send that to a Google Dialogflow CX agent). The current challenge is finding which pieces I want to connect. I’ve used a lot of Next.js lately. I like the developer experience. It enjoyable to use to build frontends. It also has route handlers, which are backend functions that are deployed on Lambda if you deploy on Vercel. These route handlers can’t really support a websocket backend because they aren’t designed to be long lived, something I learned when I worked around if by creating a secondary route handler as an async function. Apparently, these can now run for up to five minutes.^[1] Route handlers on Vercel can now run for a maximum of five minutes, which is an increase from the previous limit. This allows for more complex operations to be handled directly within these functions. I would need to stand up a separate backend. That seemed fine and fair enough, so I started looking at Deno, which I’ve also used recently and enjoyed. Deno supports websockets out of the box. It also supports importing npm modules — I plan to use @google-cloud/speech to do speech to text conversion. The remaining question is how I can stream audio captured in the browser with navigator.getUserMedia over a websocket to forward to Google to convert to text.

https://vercel.com/changelog/serverless-functions-can-now-run-up-to-5-minutes ↩

Thanks for reading via RSS!

2024-01-23

Tue, 23 Jan 2024 20:39:08 GMT

Hardly seemed with a TIL post because it was too easy, but I learned gpt-4 is proficient at building working ffmpeg commands. I wrote the prompt

convert m4a to mp3 with ffmpeg

and it responsed with

ffmpeg -i input.m4a -codec:v copy -codec:a libmp3lame -q:a 2 output.mp3

Since the problem at hand was low stakes, I just ran the command and, to my satisfaction, it worked. Language models can’t solve every problem but they can be absolutely delightful when they work.

Thanks for reading via RSS!

2024-01-16

Tue, 16 Jan 2024 20:50:18 GMT

I spent another hour playing around with different techniques to try and teach and convince gpt-4 to play Connections properly, after a bit of exploration and feedback. I incorporated two new techniques

Asking for on category at a time, then giving the model feedback (correct, incorrect, 3/4)
Using the chain of thought prompting technique

Despite all sorts of shimming and instructions, I still struggled to get the model to

only suggest each word once, even when it already got a category correct
only suggest words from the 16 word list

Even giving a followup message with feedback that the previous guess was invalid didn’t seem to help. This was the prompt I ended up with. It wasn’t all that effective.

The game "Connections" is a word game where you start with 16 words and need to group them into 4 groups of 4. Each grouping has a category that unambiguously groups the four words together. Each puzzle has exactly one solution. Watch out for words that seem to belong to multiple categories. You will be given 16 words. Output 4 groupings of 4 words and the categories to which they belong.

Output your response in exact adherence to the following form

category: <category>
words: <word1>, <word2>, <word3>, <word4>

Make one guess at a time. I will tell you if you guess correctly or if you get 3/4 words for a category. Otherwise, I will respond with "incorrect". Each word is only in one category.

Output the remaining words after I tell you you got a correct answer.

Here are the 16 words: hello, list, trick, word, cant, fool, skinny, eggshell, lean, con giggle, scoop, boob, slope, dope, dupe

Let's think step by step. YOU MUST use words from the list. DO NOT suggest other words not in the list. You will lose the game if you do so.

Thanks for reading via RSS!

2024-01-10

Wed, 10 Jan 2024 19:10:18 GMT

After some experimentation with GitHub Copilot Chat, my review is mixed. I like the ability to copy from the sidebar chat to the editor a lot. It makes the chat more useful, but the chat is pretty chatty and thus somewhat slow to finish responding as a result. I’ve also found the inline generation doesn’t consistently respect instructions or highlighted context, which is probably the most common way I use Cursor, so that was a little disappointing. To get similar behavior with Copilot, sometimes I needed to run a generation for the whole file, but the lack of specific highlighted context meant I had to write more specific instructions, which was more time-consuming than highlighting and giving shorter, more contextual instructions. It is easy to edit the prompt and resubmit it if the completion is close, but not quite right, so that is helpful.

Maybe it’s the learning curve of a new tool, but my intuition (given limited use so far) is I am fighting with Copilot Chat more than I do with Cursor. The latter feels more like an extension of my editor whereas the former can be something I need to clean up after. Overall, Copilot Chat is a good tool, with room for improvement. There are UX features that I could imagine other code assistant tools adopting, like the copy-from-chat functionality. I wonder how much the fact that I use the gpt-4-1106-preview model with Cursor, whereas Copilot uses the Codex model, impacts the output quality and ability to follow instructions with the same consistency. It would be interesting to try Copilot with a different model to see how much of the differences are due to prompting and how much are due to the model capabilities.

Thanks for reading via RSS!

2024-01-07

Sun, 07 Jan 2024 12:03:24 GMT

I worked through a basic SwiftUI 2 tutorial to build a simple Mac app. Swift and SwiftUI are an alternative to accomplish the same things Javascript and React do for web. I could also use something like Electron to build a cross-platform app using web technology, but after reading Mihhail’s article about using macOS native technology to develop Paper, I was curious to dip my toe in and see what the state of the ecosystem looked like. He opted to use Objective-C, for performance reasons. I decided to try Swift because I’ve written a bit of Objective-C years ago. I like the ergonomics of Swift as a language well enough. I can’t say I’m a huge fan of Xcode. My hardware is almost certainly too old, but Xcode is sluggish and not fun to use in a way that the web development tools I use are not (at least on my machine). Seeing all the things that PWAs can do today, I’m unsure whether it makes sense to invest in learning SwiftUI unless I want to build native mac apps.

Thanks for reading via RSS!

2024-01-05

Fri, 05 Jan 2024 17:15:34 GMT

I enjoyed this article by Robin about writing software for yourself. I very much appreciate the reminder of how gratifying it can be to build tools for yourself.

Thanks for reading via RSS!

2024-01-03

Wed, 03 Jan 2024 19:23:17 GMT

I read Swyx’s article Learn in Public today and it’s inspired me to open source most of my projects on Github.

A beautifully written and thought-provoking piece by Henrik about world models, exploring vs. exploiting in life, among other things.

Thanks for reading via RSS!

2024-01-02

Tue, 02 Jan 2024 14:35:27 GMT

I finally had a chance to use Github Copilot Chat in VS Code. It has a function to chat inline like Cursor, which has worked quite well given my initial use of it. I’m looking forward to using this more. Unfortunately, it’s not available for all IDEs yet but hopefully will be soon!

I watched lesson 3 of the FastAI course. I’ve really enjoyed Jeremy Howard’s lecture’s so far.

Thanks for reading via RSS!

2023-12-21

Thu, 21 Dec 2023 17:56:52 GMT

I looked into 11ty today to see if it could be worth migrating away from hugo, which is how (at the time of this post) I build my blog. After a bit of research and browsing, I setup this template and copied over some posts. Some over my older posts were using Hugo’s markup for syntax highlighting. I converted these to standard markdown code fences (which was worthwhile regardless). I also needed to adjust linking between posts. In Hugo, I use ref. In 11ty, these need to be relative links, e.g. /posts/2023/future-of-personal-knowledge. In Hugo, this approach works as well, so I may move to it.

After getting an initial build working, I took stock of my progress and where I aimed to end up. 11ty feels more easily configurable than Hugo. I prefer the template language (Liquid) used by the 11ty template I chose over what Hugo uses. However, I was still many hours away from getting to a spot I was happy with and decided to stop. I might consider 11ty again when starting something new, but right now, a full migration doesn’t seem worth it.

Thanks for reading via RSS!

2023-12-12

Tue, 12 Dec 2023 21:22:51 GMT

I would love if OpenAI added support for presetting a max_tokens url parameter in the Playground. Something as simple as this:

https://platform.openai.com/playground?mode=chat&model=gpt-4-1106-preview&max_tokens=1024

My most common workflow (mistake):

Press my hotkey to open the playground
Type in a prompt
Submit with cmd+enter
Cancel the request
Increase the “Maximum Length” to something that won’t get truncated
Submit the request again

Thanks for reading via RSS!

2023-11-25

Sat, 25 Nov 2023 20:44:39 GMT

A thoroughly enjoyable and inspiring read by Omar about his 20 year journey to date.

Quantity was important. Quantity led to emergent of quality.

Read the documentation: I can’t emphasize how useful this is. There are gems upon gems in the documentation. A good documentation gives a glimpse of the mind of the authors, and a glimpse of their experience.

Thanks for reading via RSS!

2023-11-17

Fri, 17 Nov 2023 13:13:02 GMT

I’m betting OpenAI will soon have a Cloud Storage product like Google Drive or iCloud for ChatGPT Plus users. Having your personal data available in the context of a language model is a massive value add. With a product like, OpenAI can fully support use cases like “summarize my notes for the week” or “create action item reminders from this recording”. They’re already dipped their toe in the water with the Files API. I expect a more feature complete offering is on the way.

Thanks for reading via RSS!

2023-10-27

Fri, 27 Oct 2023 16:18:30 GMT

I did more exploration with Copilot, mainly for writing unit tests. Copilot is a pretty good at bootstraping unit tests, particularly in Java, where initializing the right types may take several lines and there is a large standard library. In doing this, the Copilot-written code was close enough that it saved me time relative to writing the tests from scratch. I learned that Copilot can show a ranked list of completion in a separate window in the IDE, which is helpful and can also be invoked with a hotkey (option+\ in IntelliJ), which is useful as well.

Thanks for reading via RSS!

2023-10-26

Thu, 26 Oct 2023 21:29:28 GMT

I’ve been integrating Copilot into my workflow the past few days. From my understanding, it uses OpenAI’s Codex model, which is part of the GPT-3 model series. I believe this also predates the chat models, gpt-3.5-turbo and gpt-4. As someone who has been using Cursor for my personal work for several months now, the core completion functionality of Copilot feels like a step back compared to Cursor’s cmd+k (to say nothing of chat, which both have and other features). In general, when writing code, I’ve found I don’t really want line completion, I want idea completion. Copilot can do this, but you have to set it up in the right way, with a comment or a well named function stub

# sort list of tuples by second element

def sort_list_by_second_element(l):

This doesn’t always work and it’s sometimes hard to say if this is due to the API’s performance or the model’s capabilities. With Cursor, you just type thing what you want into the prompt box

sort list of tuples by second element

It’s basically the same input, but you’re guaranteed a completion because you asked for one. This issue has arisen for me several times with Copilot. I want to invoke it — not just have it step in when it thinks it’s useful.

Additionally, Cursor allows you to select specific blocks of text to modify or augment based on your prompt. It will also walk through an entire file, making minor modifications to existing code in place. This capability seems beyond what Copilot currently offers, which limits it’s usefulness for modifying existing code.

Github/Microsoft have an advantage with enterprise clients, but I believe they’re already lagging behind best-in-class development tooling using language models. That said, I expect they’ll observe the shifting landscape and begin adding more popular features to Copilot. They have a lot of resources to throw at the problem.

Thanks for reading via RSS!

2023-10-24

Tue, 24 Oct 2023 21:59:42 GMT

I finished migrating my site to the latest release of Hugo today. It’s been quite a while since I’d pulled the latest changes but most of the fixes were straightforward. A number of the partials had been updated, so I need to port my custom components to incorporate the changes. I also migrated what I’d previously kept in a static directory to assets.

I looked further into using hidutil to replace Karabiner. It looks like the utility can’t map a single keypress to multiple keys, so it can’t directly substitute for how I use Karabiner. Along the way, I found this useful tool for generating a plist file to load a startup to persist hotkeys across system restarts. I was considering switching to use the right command key, but it looks like Hammerspoon may not provide hotkey support for that according to this issue. I might be able to put something together in combination with skhd, but this idea is going to the backburner for now.

Thanks for reading via RSS!

2023-10-23

Mon, 23 Oct 2023 21:22:16 GMT

I’ve been looking into hidutil to potentially replace Karabiner for overriding my capslock behavior. My first attempt, follow instructions similar to this post, was close but unsuccessful. I was able to remap the capslock key, but it wasn’t trigger the key combo I had hoped for — I use cmd+option+ctrl+shift. Caplock wasn’t performing it’s normal function and the key light wasn’t turning on but it also wasn’t triggering my hotkeys. Hopefully I will have something workable after a bit more time with this.

Thanks for reading via RSS!

2023-10-21

Sat, 21 Oct 2023 20:26:17 GMT

I’ve been working on a Next.js app with a Leaflet map component. Using this combination of technologies has not gone smoothly. I’ve been running into an issue trying to make the Leaflet component render client side. From my understanding, Next.js renders React server-side by default, but this is easy to change with the 'use client' directive. After a bit of work, I was able to get a map component rendering on the page by importing it with

const MapView = dynamic(
  () => import('@/components/MapView'),
  {
    loading: () => <p>Map is loading</p>,
    ssr: false
  }
)

From here, I wanted to configure the markers, which I intended to set and store in a React hook. I was not able to find a way to make this work. I’ve consulted documentation a few different tutorials, mostly notably Jan’s article, but have yet to get this working with dynamic data.

Thanks for reading via RSS!

2023-10-18

Wed, 18 Oct 2023 21:16:01 GMT

I spent some time experiementing with Inngest, cloud software for running async jobs, workflows, crons and more. It’s quite similar to Temporal, which I am big advocate for for running durable workflows and gracefully handling failures. There are some drawbacks, but it feels simpler to get started than Temporal and has a most of the same topline capabilities. The main feature deficiencies I noticed in about an hour of research were lack of as granular retry configurations and timeouts and no support for query handlers to inspect the status of a running function/workflow. The list of supported features is impressive, notably

They also have a free usage tier! I wasn’t able to get access to Temporal Cloud without paying their monthy support fee, so this is attractive for experimentation and hobby projects.

Thanks for reading via RSS!

2023-09-22

Fri, 22 Sep 2023 19:39:42 GMT

I had a bunch of fun following along with this post with my own Hugo blog to construct a sqlite database of metadata. Building the database indices, I found a mistake I had made years ago in defining a post’s alias, which was a duplicate, so I fixed that. I’ve read a lot of praise of sqlite lately and wanted to get more familiar with the tools and ecosystem and this was a nice way to start to do that.

Thanks for reading via RSS!

2023-09-15

Fri, 15 Sep 2023 18:58:51 GMT

Further investigation with Open Interpreter today reaffirmed certain strengths but also revealed a number of weaknesses. The look is excellent at parsing structured data like JSON or CSV, doing analysis with tools like pandas and numpy, and plotting the results with matplotlib. However, it falls short when trying to perform more complex data fetching tasks. It seems to struggle to scrape websites or make use of less common libraries, at least when I tried without providing any additional documentation. At one point, I had it scrape a relatively simple html website and it was able to parse that into a dataframe, when cleanup the data to a point where I think I was close to being ready to do analysis. Unfortunately, the REPL seemed to hang at that point. I’m not sure if I maxed out the token context of the model or if something else happened, but I had to hard exit the process. There isn’t an easy way to “jump back to where you were in a session” and I didn’t have the patience to try again from the start. I need to look into whether some kind of step memoization is possible.

Thanks for reading via RSS!

2023-09-14

Thu, 14 Sep 2023 21:13:43 GMT

I did some more experimentation with open-interpreter today. The first use case I tried was to create, organize and reorganize files. It didn’t generate interesting content, but it was fluent at writing Python code to organize and rename files. When I prompted it to generate a fake dataset, it installed faker and created a CSV with the columns I requested. When I requested it plot those data points, it installed matplotlib and did so without issue.

From there, I moved on to doing some stock analysis. Some coworkers and I have a running joke that the stock price of our company is always lower when the trading window for employees is open. I had open-interpreter do analysis of this, giving it the windows for open trading and prompting it to analysis average and high stock prices both inside and outside the windows. Finally, I had it graph all of these elements together, as it saw fit. I found the results to be impressive, especially given how high level my instructions were.

Open-interpreter writes and runs fluent Python. Do not underestimate how powerful this can be. It gives you what feels like close to an order of magnitude of additional power operating your computer in certain ways, including restructuring the filesystem and analyzing data. I’m eager to try doing more things with it.

Occasionally, it gets stuck and can’t get unstuck, even using the error message output by executing the command. Issues like these are especially annoying if it generates a lot of code, then tries to self-correct and needs to generate it all again.

I wish it were faster. This desire hardly seems fair given that using a language model to write code in this manner feels like superpower, but I found myself itching for it to hurry up. Again, not entirely reasonable, but it’s what I’d want given a “perfect” user experience.

Having not had much opportunity to play with OpenAI’s Code Interpreter/Advanced Data Analysis feature in ChatGPT, I was floored doing the stock analysis. In 5 minutes, the model helped me answer a question I had had but didn’t have the time or motivation to answer myself. The final chart the model constructed with code would have probably taken me a couple of hours of focused work to achieve, and I likely would not have done as good a job.

I’m excited to do more with this tool.

Thanks for reading via RSS!

2023-09-11

Mon, 11 Sep 2023 21:02:19 GMT

I read quote from a long tweet the other day that made me smile.

Writing pure JavaScript is like trying to cut a watermelon with a chainsaw in the dark. It sounds fun and free and quite easy until there’s a roomful of mess to clean up.

https://twitter.com/kettanaito/status/1699440414812504443

Thanks for reading via RSS!

2023-09-09

Sat, 09 Sep 2023 20:00:20 GMT

Playing with Rivet and OpenInterpreter

Thanks for reading via RSS!

2023-08-27

Sun, 27 Aug 2023 09:58:08 GMT

It’s much easier to test Temporal Workflow in Python by invoking the contents of the individual Activities first, in the shell or via a separate script, then composing them into a Workflow. I need to see if there’s a better way to surface exceptions and failures through Temporal directly to make the feedback loop faster.

From this paper:

62% of the generated code contains API misuses, which would cause unexpected consequences if the code is introduced into real-world software

This work further reinforces some recent thoughts on the importance of measuring the quality of a language model’s output for a use case.

From my experience at a big tech co, 30%+ of engineers there are working on shipping a few buttons. I once saw a team of 20 work for 6 months to ship a back button for the onboarding flow (surprisingly hard to do). https://t.co/1wFMQ7SLFm
— Flo Crivello (@Altimor) August 26, 2023

It had been a while since I thought about this 😬

Thanks for reading via RSS!

2023-08-22

Tue, 22 Aug 2023 20:20:48 GMT

Language models and prompts are magic in a world of deterministic software. As prompts change and use cases evolve, it can be difficult to continue to have confidence in the output of a model. Building a library of example inputs for your model+prompt combination with annotated outputs is critical to evolving the prompt in a controlled way, ensuring performance and outcomes don’t drift or regress as you try and improve your overall performance.

Thanks for reading via RSS!

2023-08-17

Thu, 17 Aug 2023 19:05:40 GMT

I’ve been doing a bit of work with Temporal using it’s Python SDK. Temporal remains one of my favorite pieces of technology to work with. The team is very thoughtful with their API design and it provides a clean abstraction for building distributed, resilient workflows. It’s a piece of technology that is difficult to understand until you build with it, and once you do, you find applications for it everywhere you look. I highly recommend experimenting with it if you’re unfamiliar.

This tutorial is a nice starting point if you’re ok working in Python. I’m still relearning what can and can’t go in workflows and activities, so starting with the basics and iterating with these primitives has been helpful. If you end up creating a project with multiple packages, you may find the snippet below helpful to run a worker or a script to start a workflow from your project root.

python -m temporalapp.worker

That assumes a project structure that looks something like this

.
└── temporalapp
    ├── __init__.py
    ├── activity
    │   ├── __init__.py
    │   └── greeting.py
    ├── entity
    │   ├── __init__.py
    │   └── greeting.py
    ├── start_workflow.py
    ├── worker.py
    └── workflow
        ├── __init__.py
        └── greeting.py

Thanks for reading via RSS!

2023-08-07

Mon, 07 Aug 2023 19:43:23 GMT

🎧 Velocity over everything: How Ramp became the fastest-growing SaaS startup of all time | Geoff Charles (VP of Product)

This conversation between Lenny and Geoff was particularly noteworthy for me because it hit on so many areas of what I’ve seen in the most effective organizations and teams I’ve been apart of as well as realigning incentives to solve a number of problems I’ve experienced that hold teams back.

We report back operational overhead, meaning the percentage of tickets that come from your product area normalized by the number of users that are using that product

Having worked on a support eng team, this point resonates deeply. We often saw support tickets being opened in our systems because of bugs and issues in the product. The product teams were often unaware of these issues, and the number of these types of problems only grew over time. Occasionally, if an issue was driving enough ticket volume, it would make it back to the product team who would fix the bug, but I always advocated that we report back support tickets attributed to their respective teams and make those numbers public. To hear this approach corroborated in the discussion was validating.

We don’t have a bug backlog. We fix every bug once they’re surfaced almost.

This approach is simultaneously an engineer’s dream and nightmare. On the surface, it sounds like you would get nothing done because you’re work would constantly be interrupted by bug that need fixing. It wasn’t entirely clear to me how the workload breaks down between “core” and “production” engineers. However, to me, this approach seems to be the best shot you have and keeping your product as close to bug-free as you can. It’s an idealistic goal but I like it. Bug backlogs rot rapidly. The codebase continues to change and the engineers with context move on to other things. The best time to fix a bug is as soon as you learn about it, when the context is most fresh and when you have the reporter available or a stacktrace that is current. The level of effort required to fix a bug weeks or months later always seems to be higher than fixing it immediately because you have to ramp up on context, find a way to reproduce it and things may have changed since it was reported.

[S]upport reports into me. And the first principle there was saying, “Well, every support ticket is a failure of our product.” We literally have that as a quote just posted on all those channels. It’s a failure. And if the product works perfectly, no one should ever have to contact our support team. And what better way of holding the product team accountable for support other than having support report into product.

Here, Geoff further reinforces the point that with a dedicated approach, a team can build quickly while still maintaining a bug-free experience. I love this mentality. I personally feel a lot of dissonance when I find a bug in my code. The first thing I want to do it fix it, particularly when I know it has customer impact. On one hand, this can interrupt your flow if you’re working on something else. And that something else may also be positioned to deliver a lot of customer value, so there is an opportunity cost to the interruption. On the other, the bug is negatively impacting customers today. To have the buy-in of the organization to prioritize these fixes makes a strong statement about their culture and values.

I really enjoyed listening to Geoff’s ideas and approach to product. This conversation has helped me challenge some of my priors and assumptions about how teams are built and run.

Bonus quote, included without further comment

[W]e moved from quarterly, very expensive quarterly planning, which took one month every three months, so basically 33% of the time was planning, to a biannual one-pager on, these are the company priorities and it’s much more smooth and much faster

Thanks for reading via RSS!

2023-08-06

Sun, 06 Aug 2023 08:29:04 GMT

Simon wrote an excellent post on the current state of the world in LLMs.

Twitter continues to talk LK-99. It seems like an easy thing to root for but hard to tell exactly what is going on.

Thanks for reading via RSS!

2023-08-03

Thu, 03 Aug 2023 18:04:07 GMT

The high-order bit that changed in AI:
"I'll give you 10X bigger computer"
- 10 years ago: I'm not immediately sure what to do with it
- Now: Not only do I know exactly what to do with it but I can predict the metrics I will achieve
Algorithmic progress was necessity, now bonus.
— Andrej Karpathy (@karpathy) August 3, 2023

Turning scaling into a systematic science is the biggest advance enabled by LLMs. https://t.co/BgHoDxeX7m
— Jim Fan (@DrJimFan) August 4, 2023

It will be interested to see if or when we hit scaling limits to training more powerful models and what our new bottleneck becomes. For now, there appears to be a lot of greenfield.

Thanks for reading via RSS!

2023-08-01

Tue, 01 Aug 2023 08:38:02 GMT

While not an entirely unique perspective, I believe Apple is one of the best positioned companies to take advantage of the recent improvements in language models. I expect more generic chatbots will continue to become commodities whereas Apple will build a bespoke, multi-modal assistant with access to all your personal data on device. This assistant will be able to do anything the phone can do (invoke functions/tools) as well as answer any question about your personal data (show me photos from Christmas in 2018). Let’s hope they name it something other than Siri.

Thanks for reading via RSS!

2023-07-30

Sun, 30 Jul 2023 20:53:55 GMT

A heartwarming exchange

Your project has a youthful optimism that I hope you won’t lose as you go. And in fact it might be the way to win in the long run.

Thanks for reading via RSS!

2023-07-25

Tue, 25 Jul 2023 21:56:00 GMT

I tried out Llama 2 today using ollama. At first pass, it seemed ok a writing Python code but I struggled to get it to effective generate or adhere to specific schema. I’ll have to try a few more things but my initial impressions are mixed (relative to OpenAI models).

Thanks for reading via RSS!

2023-07-22

Sat, 22 Jul 2023 20:10:37 GMT

It’s hard to think because it’s hard to think.

- Github Copilot

Finally learned that RAG stands for “Retriever-Augmented Generation” after seeing it all over the place for months. Not sure how I missed that one.

Thanks for reading via RSS!

2023-07-19

Wed, 19 Jul 2023 08:34:46 GMT

Meta released Llama 2 yesterday and the hype has ensued. While it’s exciting to see more powerful models become available, a model with weights is not the same as an API. It is still far less accessible.

A paper came out on the measurement of the degradation ChatGPT’s reasoning abilities. As real-time peer review took place over the course of the day on Twitter, the most compelling explanation that I heard to explain these findings was that OpenAI has further fine-tuned the models to respond in a manner consistent with the level of the prompt, because this is a better experience for the user. We’ll see if this explanation holds up over time.

A paper earlier this month investigated the ability of language models to use longer context supplied to them and finds that performance is the best for information at the beginning and the end of the context. If this is an architectural limitation of LLMs, we may be using semantic and contextual search for a while longer to ensure the model has the relevant context in a prompt of reasonable length, to ensure high quality results.

Thanks for reading via RSS!

2023-07-09

Sun, 09 Jul 2023 22:14:13 GMT

I’ve been playing around more with nix lately. I like what I’ve seen from it so far: declare dependencies and get an isolated shell with those dependencies. If distributed, the environment can be trivially recreated on another machine. So far, it’s been a struggle to get a working Python environment with dependencies setup. I’ve gotten a lot of cryptic error messages after trying a number of different flake.nix files. I plan to continue to experiment, but thus far the learning curve is tough.

Thanks for reading via RSS!

2023-07-07

Fri, 07 Jul 2023 17:39:50 GMT

Some unstructured thoughts on the types of tasks language models seem to be good (and bad) at completing:

A language model is an effective tool for solving problems when can describe the answer or output you want from it with language. A language model is a good candidate to replace manual processes performed by humans, where judgement or application of semantic rules is needed to get the right answer. Existing machine learning approaches are already good at classifying or predicting over a large number of features, specifically when one doesn’t know how things can or should be clustered or labelled just by looking at the data points. To give an example where a language model will likely not perform well: imagine you want to generate a prediction for the value of a house and the land it sits on, given a list of data points describing it:

lat/long
square footage
build year
number of bed and bathrooms
lot size
whether it has a pool

Traditional machine learning approaches presently can do a good job at this. A language model likely would not. While the token outputs from language models are based on numerical predictions, those tokens should not be interpreted as numerical predictions themselves. Token outputs are more of a qualitative assessment of the inputs rather than a quantitative one (ironic, given that it’s math all the way down).

Language models are a bad fit for this house-price-prediction task for the same reason that they struggle to do math.

Thanks to CSL for his post on getting full-content output in a Hugo RSS feed. This post helped me finally get around to fixing it on this site.

Thanks for reading via RSS!

2023-07-05

Wed, 05 Jul 2023 21:10:00 GMT

Experimenting with using a language model to improve the input prompt, then use that output as the actual prompt for the model, then returning the result. It’s a bit of a play on the “critique” approach. Some of the outputs were interesting but I need a better way to evaluate the results.

import sys
import openai

MODEL = "gpt-3.5-turbo-16k"

IMPROVER_PROMPT = """
You are an expert prompt writer for a language model. Please convert the user's message into an effective prompt that will be sent to a language model to produce a helpful and useful response.

Output the improved prompt only.
"""

def generate_improved_prompt(prompt: str) -> str:
    completion = openai.ChatCompletion.create(
        model=MODEL,
        temperature=1.0,
        messages=[
            {
                "role": "system",
                "content": IMPROVER_PROMPT,
            },
            {
                "role": "user",
                "content": prompt,
            },
        ],
    )
    return completion.choices[0].message.content

def generate_completion(prompt: str) -> dict:
    completion = openai.ChatCompletion.create(
        model=MODEL,
        temperature=1.0,
        messages=[
            {
                "role": "user",
                "content": prompt,
            },
        ],
    )
    return completion.choices[0].message.content

def main():
    prompt = ' '.join(sys.argv[1:])
    standard_result = generate_completion(prompt)
    print("Standard completion:")
    print(standard_result)
    improved_prompt = generate_improved_prompt(prompt)
    print("\nImproved prompt:")
    print(improved_prompt)
    improved_result = generate_completion(improved_prompt)
    print("Improved completion:")
    print(improved_result)
    return improved_result

if __name__ == "__main__":
    main()

Thanks for reading via RSS!

2023-07-04

Tue, 04 Jul 2023 13:03:54 GMT

I’ve been working through a series on nix-flakes. It’s well written and shows some interesting applications of the tool set. I’m still trying to wrap my head around exactly where nix could fit in in my development lifecycle. It seems like it wraps up builds and package management into one. Sort of like docker, bazel, pip/npm/brew all in one. The tutorial has shown some useful variations and has convinced me flakes is the way to go, but I need to spend some more time better understanding the primitives as well. I understand little of what’s going on in the flake.nix files I’ve looked at.

Thanks for reading via RSS!

2023-07-03

Mon, 03 Jul 2023 22:33:29 GMT

Facebook (Meta, whatever) announced Threads today to launch on July 6th. Given how much worse it feels like Twitter has become (my experience only), on one hand, I could see people migrating here because no great alternative has really emerged. On the other, Facebook has zero “public” products where the user experience is even palatable for me, personally (I use Whatsapp but it’s basically iMessage). Instagram and Facebook both rapidly became completely intolerable for me due to their content. Maybe that is a matter of curation, but I bet, at least in some part, it’s a result of how Facebook runs their business and why Twitter never made much ad revenue compared to them (and why Reddit struggles to either). If I had to make a bet, I would bet on people migrating to Threads. Personally, I won’t until they have a webapp.

Thanks for reading via RSS!

2023-07-02

Sun, 02 Jul 2023 10:55:34 GMT

A simple shell function to setup a Python project scaffold. It’s idempotent, so it won’t overwrite an existing folder or env.

pproj () {
    mkdir -p $1
    cd $1
    python -m venv env
    . env/bin/activate
}

Thanks for reading via RSS!

2023-06-29

Thu, 29 Jun 2023 22:08:49 GMT

I’ve been following Jason’s working experimenting with different abstractions for constructing prompts and structuring responses. I’ve long felt that building prompts with strings is not the type of developer experience that will win the day. On the other hand, I’m weary of the wrong abstraction that would move the developer too far away from the actual prompt, which would make it harder to construct good prompts and steer the model. I’m not sure if this is an ORM vs. SQL conversation or if there’s an abstraction that exist as a happy medium.

Thanks for reading via RSS!

2023-06-27

Tue, 27 Jun 2023 09:50:56 GMT

Did some work with Clojure destructuring.

Unpack values into specific variables.

user=> (let [[a b c] [1 2 3]] (println a b c))
1 2 3
nil

Unpack the first N items, ignoring the rest.

user=> (let [[a b] [1 2 3]] (println a b))
1 2
nil

Unpack the first N items to variables and capture the rest as an array.

user=> (let [[a b & rst] [1 2 3 4 5]] (println a b rst))
1 2 (3 4 5)
nil

Thanks for reading via RSS!

2023-06-25

Sun, 25 Jun 2023 18:34:31 GMT

Doing math with a non-big decimal number and a big decimal number can cast down.

user=> (* 0.1 101M)
10.100000000000001

user=> (bigdec (* 0.1 101M))
10.100000000000001M

Thanks for reading via RSS!

2023-06-22

Thu, 22 Jun 2023 23:12:15 GMT

Heard the phrase “if someone wins the lottery” used today to describe a teammate leaving a team. I much prefer this to the more morbid alternatives.

I tried gpt-engineer today. I liked the approach and the setup instructions are good. I think I remember needing to use Python 3.11 instead of 3.8 that I was running, but beyond that the readme instructions we on point.

Process

You start by creating a project folder with a plaintext prompt. You start the script and point it at your project folder. The program reads your prompt then uses the LM to ask clarifying questions. The clarifying questions seem pretty effective. If you answer more than of one of the predetermined questions at once, the program seems to recognizes that and removes it from the list. Finally, it creates an actual project, with source code, pretty consistently (3/3 times I tried). I used it to try and create a 1-player Scattergories CLI game or something close.

Results

Unfortunately, for me, the program didn’t write code that worked in a meaningful sense. The outputted code is more of a scaffold or an outline of what the project could look like, not having made many meaningful decisions about the structure beyond what you specify. The somehwat similar project, Auto-GPT, arguably struggles in the opposite direction (as I used it) — researching too much and looping forever. Gpt-engineer has found another performance pocket, where it consistently, meaningfully outputs something after asking a small number of questions. As always, I’m on the look out for applications of this agentic tech.

Thanks for reading via RSS!

2023-06-21

Wed, 21 Jun 2023 22:38:59 GMT

I’ve been thinking about the concept of “prompt overfitting”. In this context, there is a distinction between model overfitting and prompt overfitting. Say you want to use a large language model as a classifier. You may give it several example inputs and the expected outputs. I don’t have hard data to go by, but it feels meaningful to keep the prompt generic or abstract where possible rather than enumerating overly specific cases in a way that obfuscates the broader pattern you’re hoping to apply. I hypothesize these overly specific examples could interfere with the model output in unintended, overly restrictive ways.

Thanks for reading via RSS!

2023-06-14

Wed, 14 Jun 2023 21:51:00 GMT

Richard WM Jones booted Linux 292,612 to find a bug where it hangs on boot. I loved reading the recounting of his process to accomplish this, by bisecting through the different versions of Linux and boot each thousands of times to determine whether the version contained the bug.

Georgi Gerganov started a company, ggml.ai, to run language models on commodity hardware.

Thanks for reading via RSS!

2023-06-13

Tue, 13 Jun 2023 21:16:00 GMT

I was listening to a podcast interview of Adam Robinson and he was discussing why he believed it is important process information with your body. He gives the example that when listening to something, he stops around once every minute and sees how it feels. He later goes on to highlight the importance of recognizing when something is “weird” and paying attention to it, or trusting one’s intuition even if rationally we can’t understand why something feels unusual. He asserts that we should take action on these intuitions — “if something seems a little bit off, it’s very off”. I could see this approach contributing to more well-rounded cognition, since it can be easier to discard these “intuitions” as unsubstantiated or lacking facts to justify them. On the other hand, unless you track how often you intuition proves correct, it could be hard to know if it’s well calibrated or trustworthy in the area in which you are applying it.

Thanks for reading via RSS!

2023-06-12

Mon, 12 Jun 2023 17:49:00 GMT

I read an article today about Scripting with Elixir that caught my eye because it touches on a problem I often struggle with: how you do easily distribute a script along with its dependencies in Python? Elixir has an example of what this could look like in the form of Mix.install. This feature allows one to distribute just the source code of your script and dependency management can be done when the script runs, without needing to distribute a mix.esx or requirements.txt file (as one does in Python) along with the source.

Programmer’s notebook

I recently wrote about what I view as the uncertain future of personal knowledge, especially when it comes to cataloging your own set of notes on how to use open source tools. However, I’m more optimistic that there is value in maintaining a programmer’s notebook for projects your work on and things you learn. From your notebook, every nugget of wisdom you dig up from a conversation, wiki, document or readme can be quickly recalled and sourced. Working on software, there are countless tricks and reusable patterns you can plugin to save yourself time and increase the speed at which you can solve problems. While it’s become tempting to rely on ChatGPT and similar LLMs to get this information on demand, memoizing it in a predictably recallable well helps ensure you can quickly access and reapply it in the future. Things frequently recalled end up committed to memory.

Thanks for reading via RSS!

2023-06-08

Thu, 08 Jun 2023 22:32:00 GMT

Today, I played around with Matt Rickard’s ReLLM library, another take on constraining LLM output, in this case, with regex. I tried to use it to steer a language model to generate structure (JSON) from unstructured input. This exercise is sort of like parsing or validating JSON with regex — it’s not a good idea. Complicated regular expressions to describe JSON are hard to write. I do love the demo that generates acronyms though. Matt also wrote parserLLM which provides to ability to use a context-free grammar to constrain the next predicted token from the language model. I prefer the context-free grammar approach at a high-level, but believe we need the language model constraints to be directly connected with the data structures we intend to use in code to effectively weave language models into our existing applications.

Thanks for reading via RSS!

2023-06-07

Wed, 07 Jun 2023 21:38:00 GMT

I’ve been trying to find a simple way to host a website that formats and serves a nice looking version of a recipe written in markdown. There are a few open source projects available, but nothing that has fit the bill yet. I briefly had the idea to try out something with Next.js and mdx but I found when I scaffolded a new app that I didn’t even recognize the project structure. Next.js has moved to the “App Router” approach for structuring projects. It’s not immediately obvious or intuitive how this works. As a batteries-included framework, it makes sense that different approaches to structure will have their own learning curves. Nevertheless, it’s a bit irritating how frequently this structure changes. I have a ~2-3 year old Next.js project that looks nothing like the project I am currently working on. And the project I am currently working on looks very different from a newly scaffolded project with App Router today.

Thanks for reading via RSS!

2023-06-06

Tue, 06 Jun 2023 20:11:00 GMT

I did a bit more work with Clojure today. My imperative programming habits are still bleeding through. The exercise introduced cond as a sort of case statement for flow control. I wrote a simple cond statement but was getting a bizarre runtime error:

(defn my-fn
  [x]
  (cond
    (x < 0) "negative"
    (x = 0) "zero"
    (x > 0) "positive"
  )
)

user=> (my-fn 1)
Execution error (ClassCastException) at user/my-fn (REPL:4).
class java.lang.Long cannot be cast to class clojure.lang.IFn (java.lang.Long is in module java.base of loader 'bootstrap'; clojure.lang.IFn is in unnamed module of loader 'app')

It took me a frustratingly long time to realize I needed to use prefix notation for the conditions in the cond.

(defn my-fn
  [x]
  (cond
    (< x 0) "negative"
    (= x 0) "zero"
    (> x 0) "positive"
  )
)

user=> (my-fn 1)
"positive"

Thanks for reading via RSS!

2023-06-05

Mon, 05 Jun 2023 20:44:00 GMT

I’ve following the work Exercism has been doing for several years now. I used their product to learn a bit of Elixir and Go a few years back. I got an email from the today promoting a focus on functional programming languages in the month of June. I decided to learn a bit of Clojure, since I’ve been working with the JVM lately. I’ve done a few of the exercises and my takeaways so far are

I don’t know how to format deeply nest parentheses. It feels like it looks bad no matter how I do it.
I frequently forget to surround function calls in parentheses, particularly those that take no arguments.
I keep inserting commas between arguments to functions since nearly every imperative uses this syntax.
There is a fair amount of on-your-own research needed to complete the exercises. On one hand, I like that because I think it will help learnings stick with me. On the other hand, sometimes you can get stuck for a while (but this is the job, so can’t really complain).

Thanks for reading via RSS!

2023-06-01

Thu, 01 Jun 2023 22:56:00 GMT

The API does not just change without us telling you. The models are static there.
— Logan Kilpatrick (@OfficialLoganK) May 31, 2023

Logan says any changes to the model would have been communicated. It seems some folks have data that show the model’s degradation. As competition emerges in the space, it could be a problem for OpenAI if they lose user trust on model versioning and evolution.

Tried to setup Falcon 40B. Used their provided example code and download about 90GB of weights. Ran the Python code and it failed. Did a search on the error. Found many others were seeing the same in the HuggingFace forum. Eventually, got the program to run in some form. Maxed out Macbook’s memory at about 90GB (went to swap) and crashed the process. I wonder if a Mac can be tuned to make this work.

Thanks for reading via RSS!

2023-05-31

Wed, 31 May 2023 21:00:00 GMT

Tons of reports on HN that GPT-4 has gotten significantly worse. Are people experiencing this? pic.twitter.com/VBminyUj6r
— Nabeel S. Qureshi (@nabeelqu) May 31, 2023

A number of folks are reporting gpt-4 appears to be performing less impressively as of late (additional conversation). I was using gpt-4 to write code earlier today, and anecdotally, can say it seems to be less effective at code generation. It still writes working code but the code, but the tests cases it provided aren’t always correct and it seems to be less “expert level” than I recall initially.

Thanks for reading via RSS!

2023-05-29

Mon, 29 May 2023 21:02:00 GMT

I’ve been following Eric’s posts about SudoLang since the first installment back in March. I’ve skimmed through the spec and the value proposition quite compelling. SudoLang seeks to allow programmers all levels to instruct LLMs and can also be transpiled into your programming language of choice. While I’m still early in my understanding of how to use this technique, it’s one I’m following closely and continuing to experiment with.

Thanks for reading via RSS!

2023-05-27

Sat, 27 May 2023 13:34:00 GMT

What if we set GPT-4 free in Minecraft? ⛏️

I’m excited to announce Voyager, the first lifelong learning agent that plays Minecraft purely in-context. Voyager continuously improves itself by writing, refining, committing, and retrieving *code* from a skill library.

GPT-4 unlocks… pic.twitter.com/hjTxk6Qb1x
— Jim Fan (@DrJimFan) May 26, 2023

NVIDIA researchers introduce an LLM-based agent with “lifelong learning” capabilities that can navigate, discover, and accomplish goals in Minecraft without human intervention.

Thanks for reading via RSS!

2023-05-26

Fri, 26 May 2023 15:47:00 GMT

The Alexandria Index is building embeddings for large, public data sets, to make them more searchable and accessible.

That people produce HTML with string templates is telling us something. I think about this phenomena often, though I personally find most string template systems that produce HTML difficult to use. Django templates, Handlebars, Rails ERB, Hugo templates just to name a few. My experience has been these systems are difficult to debug and are practically their own full programming languages. I think React finds the sweet spot for the challenges that these other systems run into, with the abilities of Typescript/Javascript (maybe the braces-in-JSX syntax notwithstanding). React can still be difficult to debug but it feels much more like you’re writing the thing that will be rendered rather than an abstraction on top of that (yes React is probably a higher level abstraction than any other these but it’s about experience over the implementation details).

Added a full LLM type system and context-free grammar to Python. Includes all typescript types, regexes, and arbitrary grammars. Makes coding programs with LLM calls woven in super easy.

Thanks @ggerganov for llama.cpp and @abetlen for llama-cpp-python that enable this. pic.twitter.com/5WyoY3xtsJ
— Grant Slatton (@GrantSlatton) May 21, 2023

Constricting LLMs to responding in adherence with specific schema or following a context-free grammar continues to be the direction I expect most production use cases will go. Some of the comments mention the irony that we constrain LLMs to responding following a strict grammar, but this is the reality of most production systems. APIs don’t respond to queries with

The iMac computer features a 27-inch Retina 5K display, an 8-core Intel Core i9 processor with up to 3.6GHz turbo boost, up to 128GB of RAM, and up to 8TB of SSD storage.

They respond with structure.

{
  "display": "27-inch Retina 5K",
  "processor": "8-core Intel Core i9 with up to 3.6GHz turbo boost",
  "ram": "Up to 128GB",
  "storage": "Up to 8TB SSD"
}

Structure is what it takes (in today’s world at least), to incorporate a new piece of technology into the stack. That is not to say this can’t and won’t change, but to realize benefits of LLMs today, I believe this is a useful initial approach. We’ll have to trust our AI systems to behave consistently a lot more than we do today before we’ll be ready to using natural languages to communicate between our systems. Even with more consistent behavior, natural language with always be more ambiguous than context-free grammars and schemas. The latter is what you want for predictably behaving systems.

Thanks for reading via RSS!

2023-05-24

Wed, 24 May 2023 12:51:00 GMT

I’ve seen a lot of “GPT detection” products floating around lately. Sebastian discusses some of the products and their approaches in this article. Some products claim to have developed an “algorithm with an accuracy rate of text detection higher than 98%”. Unfortunately, this same algorithm determined a GPT-4 generated response from the prompt “write a paragraph in the style of Edgar Allan Poe” was 0% AI GPT. In my experience, you don’t need to try very hard to trick “AI-detection” systems. It seems that adding “in the style of…” to pretty much any prompt can thwart detections approaches. Even though these products don’t seem to work, there is clearly a market for them and many products in that market, which seems to indicate a desire for them to work. From these products’ marketing and news references, it appears folks in education are quite interested in them. I can’t begin to appreciate the challenges educators must be experiencing as they attempt to adjust to the changes brought on by the accessibility of LLMs. However, as someone who struggled to learn in the traditional education system, I do hope teachers will pivot their energies to adopting their curriculums rather than trying to maintain the status quo.

Andrej Karpathy lead a session at Microsoft Build giving an overview on models, training and the software ecosystem emerging around them. I particularly appreciated his “default recommendations” on building LLM applications:

Achieve your top possible performance
Optimize costs

He notes it’s hard to give generic advice and to me, reinforces that prompting is still as much an art as it is a science. We’ve found of a number of approaches that perform better relative to others, but at this point, there is no substitution for experimentation given your use case.

Thanks for reading via RSS!

2023-05-22

Mon, 22 May 2023 10:35:00 GMT

Brex wrote a nice beginner guide on prompt engineering.

Thanks for reading via RSS!

2023-05-18

Thu, 18 May 2023 14:38:00 GMT

A low-effort quality-of-life improvement for oncall has been starting a week-long shift on a Friday instead of a Monday. Beginning a weekend with oncall isn’t the best, but it’s more than offset by how good it feels to finish the week and oncall at the same time next Friday.

Thanks for reading via RSS!

2023-05-16

Tue, 16 May 2023 09:51:00 GMT

LMQL is a SQL-like programming language for interacting with LMs. It takes a declarative approach to specifying the output constraints for a language model, with a SQL flavor.

Microsoft created a project called guidance which is an LLM-agnostic language to “interleave generation, prompting, and logical control into a single continuous flow matching how the language model actually processes the text”. It’s based on Handlebars templates and provides in-template notion for system and user messages.

kor is a prototype used to extract structured data from language model responses. It takes a more object oriented approach and is Python-specific.

rebuff is a multilayer prompt injection defense library that continuously improves and adapts based on previous attacks.

Thanks for reading via RSS!

2023-05-14

Sun, 14 May 2023 05:00:00 GMT

marvin’s @ai_model decorator implements something similar to what I had in mind for extracting structured data from an input to a language model.

They also use a phase that I like and may adopt for this approach to formatting the output of a language model:

Format … data declaratively

In most of the examples, structured data is extracted from unstructured input. The docs don’t discuss the use of schema to add additional context to the provided data. I’m curious see if there are use cases for also defining the inputs or if the extraction and mapping can all be done with just a target schema.

Thanks for reading via RSS!

2023-05-13

Sat, 13 May 2023 12:34:00 GMT

Added arbitrary context free grammar constraints to llama.cpp

Can now plug in any llama.cpp compatible model and give an exact grammar spec: JSON, etc

Excited to use with more powerful local models as they are released

Thanks @ggerganov & friends for such a wonderful project. pic.twitter.com/HCLACavrlH
— Grant Slatton (@GrantSlatton) May 14, 2023

Restricting the next predicted token to adhere to a specific context free grammar seems like a big step forward in weaving language models into applications.

Thanks for reading via RSS!

2023-05-11

Thu, 11 May 2023 12:34:00 GMT

Using system prompts provides an intuitive separation for input and output schema from input content.

Using system prompts does not effectively guard against prompt injection.

Thanks for reading via RSS!

2023-05-10

Wed, 10 May 2023 12:34:00 GMT

With the support of GPT-4, I feel unstoppable. The overnight surge in productivity is intoxicating, not for making money or starting a business, but for the sheer joy of continuously creating ideas from my mind, which feels like happiness.

- Ke Fang

Thanks for reading via RSS!

2023-05-09

Tue, 09 May 2023 12:34:00 GMT

I warn you now, this is going to have unfortunate consequences, just as switching to living in suburbia and driving everywhere did. When you lose the ability to write, you also lose some of your ability to think.
— Paul Graham (@paulg) May 9, 2023

I wrote a few paragraphs disagreeing with Paul’s take, asserting that, like Simon suggests, we should think of language models like ChatGPT as a “calculator for words”.

I argued things like “language models are another tool in the technology toolbox” and “language models will meaningfully change what day-to day-life looks like, but transformations like this have happened throughout history”. Ironically, in trying to write a coherent opinion in disagreement with Paul’s, I realized I agreed with him.

Up until recently, it was difficult to produce coherent writing without thinking. If you sought to prove a point, you needed to enumerate your arguments, support those with evidence, then craft sentences to communicate what you were trying to say. With LLMs, you can generate written propose that masquerades as your argument (and it may even be a good one), but unless you do some amount of writing, it’s not really your argument. You probably couldn’t defend it and even if you could, you’d have studied it deeply enough that you’d either find flaws in it or likely could have written it yourself.

When you write your own argument you’re forced to think, because if you care about what your writing, you’re forced to consider if what you are saying holds up to scrutiny. Writing is unique in that you can take time to collect your thoughts and interrogate your own arguments to make sure they accurately reflect what you believe or what you are trying to say. Written words are the bridge between others and abstract ideas in your mind. At least they are for me.

Thanks for reading via RSS!