I use Simon’s llm to quickly run LLM prompts.
This package is easily installed with brew or pip, so if you want to use it, I recommend those approaches.
The following approach is not for the faint of heart and assumes a bit of familiarity with Nix and home-manager.
We are going to install the llm including the llm-mistral plugin using Nix.
It’s not particularly straightforward, but if you want to manage this tool with Nix, it appears to be possible.
Using my nix-config repo as a jumping off point, we can easily install llm using home-manager.
This works because llm is available from nixpkgs.
You can also find it using the search.
After rebuilding my system’s flake, llm is installed and available on the system
❯ which llm
/etc/profiles/per-user/danielcorin/bin/llm
❯ llm models
OpenAI Chat: gpt-3.5-turbo (aliases: 3.5, chatgpt)
OpenAI Chat: gpt-3.5-turbo-16k (aliases: chatgpt-16k, 3.5-16k)
OpenAI Chat: gpt-4 (aliases: 4, gpt4)
OpenAI Chat: gpt-4-32k (aliases: 4-32k)
OpenAI Chat: gpt-4-1106-preview
OpenAI Chat: gpt-4-0125-preview
OpenAI Chat: gpt-4-turbo-preview (aliases: gpt-4-turbo, 4-turbo, 4t)
OpenAI Completion: gpt-3.5-turbo-instruct (aliases: 3.5-instruct, chatgpt-instruct)
So far so good. If we want to using more than just OpenAI models, we need to install “plugins”. If try and install a plugin as instructed by the docs, we get this error
❯ llm install llm-mistral
Install command has been disabled for Nix. If you want to install extra llm plugins, use llm.withPlugins([]) expression.
At the time of this writing, llm-mistral is not available in nixpkgs.
We will need to build it ourselves.
nix-init is a helpful tool for generating some of the boiler plate we will need to build a Python package with Nix.
After installing that, we run nix-init for llm-mistral
❯ nix-init
Enter url
❯ https://github.com/simonw/llm-mistral
Enter tag or revision (defaults to 0.3)
❯ 0.3
Enter version
❯ 0.3
Enter pname
❯ llm-mistral
How should this package be built?
❯ buildPythonApplication
Enter output path (leave as empty for the current directory)
❯ .
I decided to move the outputted default.nix into packages/llm-mistral/default.nix.
The generated code is pretty close to what we need.
We change buildPythonApplication to buildPythonPackage, since we want to pass this as a package to llm.withPlugins.
{ lib
, python3
, fetchFromGitHub
}:
python3.pkgs.buildPythonApplication rec {
pname = "llm-mistral";
version = "0.3";
pyproject = true;
src = fetchFromGitHub {
owner = "simonw";
repo = "llm-mistral";
rev = version;
hash = "sha256-ddaYXvbee66Keh5loQOs4xuOrtoQ06lHlBWlLiUp2zI=";
};
nativeBuildInputs = [
python3.pkgs.setuptools
python3.pkgs.wheel
];
propagatedBuildInputs = with python3.pkgs; [
httpx
httpx-sse
llm
];
passthru.optional-dependencies = with python3.pkgs; {
test = [
pytest
pytest-httpx
];
};
pythonImportsCheck = [ "llm_mistral" ];
meta = with lib; {
description = "LLM plugin providing access to Mistral models using the Mistral API";
homepage = "https://github.com/simonw/llm-mistral";
license = licenses.asl20;
maintainers = with maintainers; [ ];
mainProgram = "llm-mistral";
};
}
We can now include this custom package in home.nix (based on how my files are setup) and pass it to llm.withPlugins([])
We add
{ config, pkgs, ... }:
let
# ...
llm-mistral = pkgs.callPackage ./packages/llm-mistral/default.nix { };
systemPackages = with pkgs; [
# ...
(llm.withPlugins([llm-mistral]))
# ...
]
# ...
in
{
# ...
}
then run
darwin-rebuild switch --flake ~/.config/nix/flake.nix
We get
error:
… while calling the 'derivationStrict' builtin
at /builtin/derivation.nix:9:12: (source not available)
… while evaluating derivation 'darwin-system-24.05.20240302.fa9a517+darwin4.daa0360'
whose name attribute is located at /nix/store/fpr5fsl88azrdsbzjpfq2hk307bbr7cm-source/pkgs/stdenv/generic/make-derivation.nix:353:7
… while evaluating attribute 'activationScript' of derivation 'darwin-system-24.05.20240302.fa9a517+darwin4.daa0360'
at /nix/store/69xb0qgirf00jbvamzf2yvgpfv8d0f06-source/modules/system/default.nix:95:7:
94|
95| activationScript = cfg.activationScripts.script.text;
| ^
96| activationUserScript = cfg.activationScripts.userScript.text;
(stack trace truncated; use '--show-trace' to show the full trace)
error: undefined variable 'httpx-sse'
at /nix/store/fzzgbm0rclhzw163hmiijij5whvamqzb-source/packages/llm-mistral/default.nix:25:5:
24| httpx
25| httpx-sse
| ^
26| llm
It turns out httpx-sse isn’t in nixpkgs either.
Following similar steps as above, we using nix-init, to build the package ourselves.
Note: any packages dependencies that appear not to be needed by the package at runtime can be moved to passthru.optional-dependencies (like mypy, pytest, ruff, black, etc.)
We end up needing to build sse-starlette as well.
Once we have all of these packages, we can wire them up and pass them into each other as needed.
{ config, pkgs, ... }:
let
# ...
sse-starlette = pkgs.callPackage ./packages/sse-starlette/default.nix { };
httpx-sse = pkgs.callPackage ./packages/httpx-sse/default.nix { inherit sse-starlette; };
llm-mistral = pkgs.callPackage ./packages/llm-mistral/default.nix { inherit httpx-sse; };
systemPackages = with pkgs; [
# ...
(llm.withPlugins([llm-mistral]))
# ...
]
# ...
in
{
# ...
}
We also update both packages/httpx-sse/default.nix and packages/llm-mistral/default.nix to access the locally built packages, so they now have this structure
{ lib
, python3
, fetchFromGitHub
, httpx-sse
}:
python3.pkgs.buildPythonPackage rec {
pname = "llm-mistral";
version = "0.3";
pyproject = true;
src = fetchFromGitHub {
owner = "simonw";
repo = "llm-mistral";
rev = version;
hash = "sha256-ddaYXvbee66Keh5loQOs4xuOrtoQ06lHlBWlLiUp2zI=";
};
nativeBuildInputs = [
python3.pkgs.setuptools
python3.pkgs.wheel
];
propagatedBuildInputs = [
python3.pkgs.httpx
httpx-sse
];
passthru.optional-dependencies = with python3.pkgs; {
test = [
pytest
pytest-httpx
];
};
pythonImportsCheck = [ "llm_mistral" ];
meta = with lib; {
description = "LLM plugin providing access to Mistral models using the Mistral API";
homepage = "https://github.com/simonw/llm-mistral";
license = licenses.asl20;
maintainers = with maintainers; [ ];
};
}
We now get this error
error: builder for '/nix/store/a0vv7fh9pwf40avbb6z9q4i1c170d874-llm-0.13.1.drv' failed with exit code 1;
last 10 log lines:
> llm 0.13.1 (/nix/store/c8gj1bbmggxyy52ppljh9wlr3zzxwy78-llm-0.13.1)
> dependency chain:
> this derivation: /nix/store/c8gj1bbmggxyy52ppljh9wlr3zzxwy78-llm-0.13.1
> llm 0.13.1 (/nix/store/gna4icldg4vzd0m3v49cn9dp7dls31vl-python3.11-llm-0.13.1)
> dependency chain:
> this derivation: /nix/store/c8gj1bbmggxyy52ppljh9wlr3zzxwy78-llm-0.13.1
> ...depending on: /nix/store/aga00jbi95bj2wpvkc1y76gvl94693ck-python3.11-llm-mistral-0.3
> ...depending on: /nix/store/gna4icldg4vzd0m3v49cn9dp7dls31vl-python3.11-llm-0.13.1
>
> Package duplicates found in closure, see above. Usually this happens if two packages depend on different version of the same dependency.
For full logs, run 'nix log /nix/store/a0vv7fh9pwf40avbb6z9q4i1c170d874-llm-0.13.1.drv'.
error: 1 dependencies of derivation '/nix/store/bkg0bcc7dai8cag3anm3k6am1i43h6gy-home-manager-applications.drv' failed to build
error: 1 dependencies of derivation '/nix/store/1q2hdsrr1yn6sy79nbnkkbnrggm8qw61-home-manager-fonts.drv' failed to build
error: 1 dependencies of derivation '/nix/store/c7j1jcjhdbfagpwm8f22np8bz4k6n34c-home-manager-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/4xsby6g8gcl1nzv7cg4v34q5lccag7g5-home-manager-generation.drv' failed to build
error: 1 dependencies of derivation '/nix/store/zrbpniffpka4f47qq3x3f5ijwndjyj02-user-environment.drv' failed to build
error: 1 dependencies of derivation '/nix/store/84m731za8s233cg1zcyzrh4w97vg0qya-activation-danielcorin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/g9npff23rkspwlwhyh9x8y2jjn0vphzm-darwin-system-24.05.20240302.fa9a517+darwin4.daa0360.drv' failed to build
There is a now circular dependency between llm and llm-mistral.
This can be resolved by removing llm as a dependency in the packages/llm-mistral/default.nix.
We will rely on it being available at runtime.
{ lib
, python3
, fetchFromGitHub
}:
python3.pkgs.buildPythonApplication rec {
pname = "llm-mistral";
version = "0.3";
pyproject = true;
src = fetchFromGitHub {
owner = "simonw";
repo = "llm-mistral";
rev = version;
hash = "sha256-ddaYXvbee66Keh5loQOs4xuOrtoQ06lHlBWlLiUp2zI=";
};
nativeBuildInputs = [
python3.pkgs.setuptools
python3.pkgs.wheel
];
propagatedBuildInputs = with python3.pkgs; [
httpx
httpx-sse
];
passthru.optional-dependencies = with python3.pkgs; {
test = [
pytest
pytest-httpx
];
};
pythonImportsCheck = [ "llm_mistral" ];
meta = with lib; {
description = "LLM plugin providing access to Mistral models using the Mistral API";
homepage = "https://github.com/simonw/llm-mistral";
license = licenses.asl20;
maintainers = with maintainers; [ ];
mainProgram = "llm-mistral";
};
}
Rebuilding once more gives us
error: builder for '/nix/store/fslzva033y3vfrci0q9s4jiaqd63f05c-python3.11-llm-mistral-0.3.drv' failed with exit code 1;
last 10 log lines:
> adding 'llm_mistral-0.3.dist-info/top_level.txt'
> adding 'llm_mistral-0.3.dist-info/RECORD'
> removing build/bdist.macosx-11.0-arm64/wheel
> Successfully built llm_mistral-0.3-py3-none-any.whl
> Finished creating a wheel...
> Finished executing pypaBuildPhase
> Running phase: pythonRuntimeDepsCheckHook
> Executing pythonRuntimeDepsCheck
> Checking runtime dependencies for llm_mistral-0.3-py3-none-any.whl
> - llm not installed
For full logs, run 'nix log /nix/store/fslzva033y3vfrci0q9s4jiaqd63f05c-python3.11-llm-mistral-0.3.drv'.
error: 1 dependencies of derivation '/nix/store/d9rpk7m2fxm75ig2vgfq56grzfckkavl-llm-0.13.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/3p9izs6k1f5xi6k7dm2plvkq4psrmn3h-home-manager-applications.drv' failed to build
error: 1 dependencies of derivation '/nix/store/87l7gpfbyj8rzzyx1jx24z43yv0s7abh-home-manager-fonts.drv' failed to build
error: 1 dependencies of derivation '/nix/store/1509dawq87wdipw5wmd12p1cdg3a5shi-home-manager-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/9ikbbbd401a6103nngwn97ddawjjq1xa-home-manager-generation.drv' failed to build
error: 1 dependencies of derivation '/nix/store/is91dljc99y03j9y7p4xbrfkz5h5w0dx-user-environment.drv' failed to build
error: 1 dependencies of derivation '/nix/store/yw72ya9xffz4xvzddqxpamra1dx2lil5-activation-danielcorin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/y5fyldxp9cp8dil02gzqvw05vflb75in-darwin-system-24.05.20240302.fa9a517+darwin4.daa0360.drv' failed to build
We can remove pythonImportsCheck = [ "llm_mistral" ]; and add dontCheckRuntimeDeps = true; to work around this issue.
We run a final
darwin-rebuild switch --flake ~/.config/nix/flake.nix
and it builds without error.
We now have Mistral models available in the llm cli.
❯ llm models
OpenAI Chat: gpt-3.5-turbo (aliases: 3.5, chatgpt)
OpenAI Chat: gpt-3.5-turbo-16k (aliases: chatgpt-16k, 3.5-16k)
OpenAI Chat: gpt-4 (aliases: 4, gpt4)
OpenAI Chat: gpt-4-32k (aliases: 4-32k)
OpenAI Chat: gpt-4-1106-preview
OpenAI Chat: gpt-4-0125-preview
OpenAI Chat: gpt-4-turbo-preview (aliases: gpt-4-turbo, 4-turbo, 4t)
OpenAI Completion: gpt-3.5-turbo-instruct (aliases: 3.5-instruct, chatgpt-instruct)
Mistral: mistral/open-mistral-7b
Mistral: mistral/mistral-tiny-2312
Mistral: mistral/mistral-tiny (aliases: mistral-tiny)
Mistral: mistral/open-mixtral-8x7b
Mistral: mistral/mistral-small-2312
Mistral: mistral/mistral-small (aliases: mistral-small)
Mistral: mistral/mistral-small-2402
Mistral: mistral/mistral-small-latest
Mistral: mistral/mistral-medium-latest
Mistral: mistral/mistral-medium-2312
Mistral: mistral/mistral-medium (aliases: mistral-medium)
Mistral: mistral/mistral-large-latest (aliases: mistral-large)
Mistral: mistral/mistral-large-2402
This was a quite challenge for me and I couldn’t have done it without the help of user “Max Headroom” on Discord. They are one of the creators of the nixified.ai project. Thank you so much your help and patience!
After getting llm-mistral working, I tried to install the llm-claude plugin for llm following similar steps.
In doing so, I ran into a new issue I hadn’t seen before, that seemed to be with the anthropic library tests.
error: builder for '/nix/store/bpfw9wimbmvjmfc3ppbfa55jm71zxw26-python3.11-anthropic-0.15.0.drv' failed with exit code 1;
last 10 log lines:
> DEBUG anthropic._base_client:_base_client.py:439 Request options: {'method': 'get', 'url': '/foo'}
> DEBUG anthropic._base_client:_base_client.py:439 Request options: {'method': 'get', 'url': '/foo'}
> DEBUG anthropic._base_client:_base_client.py:439 Request options: {'method': 'get', 'url': '/foo'}
> =========================== short test summary info ============================
> FAILED tests/test_client.py::TestAnthropic::test_copy_build_request - AssertionError
> FAILED tests/test_client.py::TestAsyncAnthropic::test_copy_build_request - AssertionError
> ERROR tests/test_client.py::TestAsyncAnthropic::test_raw_response - pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an ...
> ERROR tests/test_utils/test_typing.py::test_extract_type_var_generic_subclass_different_ordering_multiple - DeprecationWarning: pytest-asyncio detected an unclosed event loop when tea...
> =================== 2 failed, 234 passed, 2 errors in 4.82s ====================
> /nix/store/zzgwc93q7ycxwks9743f0900i1lagc98-stdenv-darwin/setup: line 1587: pop_var_context: head of shell_variables not a function context
For full logs, run 'nix log /nix/store/bpfw9wimbmvjmfc3ppbfa55jm71zxw26-python3.11-anthropic-0.15.0.drv'.
error: 1 dependencies of derivation '/nix/store/8s1qxy309bcvjnq7dh5m72bgqgd5yk10-python3.11-llm-claude-0.4.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/xlqadx7v10isr66d5fkc9yfn8g8vck3q-llm-0.13.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/954ybsbasav3pvrl5fl7akc6avvr9956-home-manager-applications.drv' failed to build
error: 1 dependencies of derivation '/nix/store/fvr0w1pnyfprygbsqgd4jl242w3h7l6d-home-manager-fonts.drv' failed to build
error: 1 dependencies of derivation '/nix/store/bgf33dwanrw6c9dlwbrbrhcdpddxa5xk-home-manager-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/zdy47ynmlmr8777gyfhklvk62yqwdzsa-home-manager-generation.drv' failed to build
error: 1 dependencies of derivation '/nix/store/lmrks628m5k1a05bqm2imzaf01sk0wnb-user-environment.drv' failed to build
error: 1 dependencies of derivation '/nix/store/xgxw6x9zjxfyxy33nvk5r6l5sbydr96w-activation-danielcorin.drv' failed to build
Here is where I decided to stop.
I want to be able to use llm with many different plugins so I’ll manage it through Homebrew for now.
The code for this post is available as a PR to my config repo here.