--- date: "2024-08-29T01:14:47.000Z" title: "2024-08-29" draft: false --- Incredible read: https://eieio.games/essays/the-secret-in-one-million-checkboxes/ --- I failed many attempts at getting Sonnet to write code to display the folder structure of the output of a `tree -F` command using shortcodes. After a lot of prompting, I wrote a mini-design doc on how the feature needed to be implemented and used it as context for Sonnet. I tried several variants of instructions in the design including trying to improve it with the model itself for clarity. I validated that the model could translate from the `tree -F` to the html markup directly. It could. That in fact is the example html target document in my design doc. Here is that doc: --- # File Tree Shortcode This document outlines the requirements for a Hugo shortcode that generates an HTML representation of a file tree structure. ## Input The shortcode takes a multi-line string input representing the output of a `tree -F` command. For example: ```text ./ ├── Makefile ├── __pycache__/ │ ├── clean.cpython-312.pyc │ ├── ex.cpython-312.pyc │ ├── ex1.cpython-312.pyc │ ├── ex2.cpython-312.pyc │ ├── extract.cpython-312.pyc │ └── main.cpython-312.pyc ├── ex.py └── src/ ├── __pycache__/ │ ├── clean.cpython-312.pyc │ └── extract.cpython-312.pyc ├── clean.py └── extract.py ``` ## Output The shortcode should generate nested HTML lists representing the file tree structure: - Folders are represented by `
  • ` elements with a `` child containing the folder name. - Files are represented by simple `
  • ` elements. - The content of each folder is contained in a ` ``` I also prompted Sonnet on whether it could identify any issues with the described approach, like if I somehow am suggesting it does something impossible (I'm not seeing it). It said > I believe the approach is generally sound and should work well. Finally, I tried adding some of the [Hugo docs](gohugo.io/templates/introduction/) to Cursor but this didn't seem to be improving things much. Side note: I need to spend some time experimenting with [Zed](https://zed.dev/). My takeaway is Sonnet wasn't trained on enough Hugo templates to be good at it. All the prompt engineering I could throw at it in a few hours didn't seem to be enough to overcome these challenges. Neither was trying `gpt-4`, `llama-70b` or one of the Gemini models (I forgot which I tried). The fact that models could not solve this task violated my intuition for the problem. I was able to prompt the model to write Python code to output the desired html and it did so without issue. ```python def generate_file_tree_html(tree_output): lines = tree_output.strip().split("\n") html_output = [] stack = [] for line in lines: indent_level = len(line) - len(line.lstrip(" │")) is_dir = line.strip().endswith("/") name = line.strip().split(" ", 1)[-1].strip("│ ├── └── ") while stack and stack[-1][0] >= indent_level: html_output.append("
    • ") stack.pop() if is_dir: html_output.append( f'
  • {name}
    • ") stack.pop() return "" # Example usage tree_output = """ ./ ├── Makefile ├── __pycache__/ │ ├── clean.cpython-312.pyc │ ├── ex.cpython-312.pyc │ ├── ex1.cpython-312.pyc │ ├── ex2.cpython-312.pyc │ ├── extract.cpython-312.pyc │ └── main.cpython-312.pyc ├── ex.py └── src/ ├── __pycache__/ │ ├── clean.cpython-312.pyc │ └── extract.cpython-312.pyc ├── clean.py └── extract.py """ html_output = generate_file_tree_html(tree_output) print(html_output) ``` This script outputs the expected html (without whitespace). Lastly, I prompted Sonnet to translate this working Python code into a Hugo shortcode. This approach also failed. The model kept hallucinating functions like `while` that don't exist in the Hugo templating language. At this point, I didn't have confidence in any of the models to write Hugo templates/shortcodes. Before I called it, I figured if I were to try one more thing, I would pull together as many examples of Hugo template docs and usage and pass that as context to the model with the same instructions. I manually copied and pasted a ton of the examples from https://gohugo.io/functions then prompted the model similarly to before. It output the following ```html {{- $tree := .Inner | strings.TrimRight "\n" | split "\n" -}} {{- $root := index $tree 0 | strings.TrimRight "/" -}} ``` which also does not work. Maybe, I'll get this working at some point.