% Created 2026-03-02 Mon 10:08
% Intended LaTeX compiler: pdflatex
\documentclass[usenames,dvipsnames,colorlinks,aspectratio=169]{beamer}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage{color}
\usepackage{listings}
\usepackage{pdfpages}
\usepackage{xcolor}
\usepackage{graphicx}
\usepackage{xspace}
\usepackage{siunitx}
\usepackage{libertine}
\usepackage{fontawesome}
\mode<beamer>{\usetheme{Boadilla}}
\setbeamertemplate{navigation symbols}{}
\setbeamerfont{alerted text}{series=\bfseries}
\setbeamercolor{alerted text}{fg=blue}
\usecolortheme{default}
\usefonttheme{serif}
\usefonttheme{professionalfonts}
\lstloadlanguages{[ISO]C++}
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.50,0,0.82}
\definecolor{codeorange}{rgb}{0.95,0.3,0.1}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
\lstdefinestyle{mystyle}{
commentstyle=\color{codegreen},
keywordstyle=\color{codeorange},
numberstyle=\tiny\color{codegray},
stringstyle=\color{codepurple},
basicstyle=\footnotesize,
breakatwhitespace=false,
breaklines=true,
captionpos=b,
keepspaces=true,
numbers=left,
numbersep=5pt,
showspaces=false,
showstringspaces=false,
showtabs=false,
tabsize=2
}
\lstset{style=mystyle}
\usetheme{default}
\author{Brett Viren}
\date{\today}
\title{Comparing LLMs for structured planning}
\subtitle{No code was harmed}
\hypersetup{
 pdfauthor={Brett Viren},
 pdftitle={Comparing LLMs for structured planning},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 30.1 (Org mode 9.8-pre)},
 pdflang={English}}
\begin{document}

\maketitle
\begin{frame}[label={sec:org27a26e2}]{Topics}
\begin{itemize}
\item Intro to \alert{structured planning}.
\item Rehash of the prior experiment.
\item Setup for the planning test.
\item Some results and observations.
\item Drill down to one ``research task''.
\item Some initial conclusions and bias forming.
\end{itemize}
\end{frame}
\begin{frame}[label={sec:orgd8babe1},fragile]{Structured planning}
 \alert{Structured planning} (just my name) is an approach to using LLMs for \alert{large scale}
software development.  Basic workflow of a project:

\begin{enumerate}
\item Human + LLM write a ``\alert{plan-for-a-plan}'' document (eg Markdown file).
\item Iterate: human edits and/or human+LLM refines.
\item LLM breaks down \alert{plan-for-a-plan} into detailed \alert{plans} and \alert{tasks} as \alert{Beads Issues}.
\item Iterate: human edits and/or human+LLM.
\item Execute: human develops and/or human+llm and/or just LLM.
\item Automate: let LLM walk up dependency tree working issue after issue.
\end{enumerate}

\footnotesize

Each Beads issue has an ID like \texttt{<prefix>-<hash>} and is composed of:
\begin{description}
\item[{Sections}] title, description, design, notes, comments.
\item[{Labels}] ``epic'' (major aspect), ``feature'', ``task'', ``bug'', \ldots{}
\item[{Dependencies}] parent/child \texttt{id-abc,id-abc.1,id-abc.2} or \texttt{id-abc} ``blocks'' \texttt{id-xyz}.
\end{description}
\end{frame}
\begin{frame}[label={sec:org9c96dac},fragile]{Structured planning caveats}
 \begin{block}{Not an original idea}
Many people pursuing variations on this theme

\begin{itemize}
\item Beads is the kernel for Gas Town which scales this idea up orders of magnitude.
\item Others may write a single \texttt{project.md}, generate a \texttt{plan.md}, generate \texttt{tasks/<task>.md}.
\end{itemize}

I do think the constraint of the Beads CLI is beneficial.  A file-system can be ``anything''.
\end{block}
\begin{block}{Main point}
Human prompts the LLM to write prompts for LLMs.
\begin{itemize}
\item Distinct from ``immediate mode'' loop of human prompts, LLM codes.
\end{itemize}
Allow the human to think of planning, design, tasks at human pace and order instead and avoid ``prompt paralysis''.
\end{block}
\end{frame}
\begin{frame}[label={sec:org7e4d292}]{Prior work}
I used 4 hours of my weekend and just Sonnet to apply this approach to build
``\alert{toenails}'', a ``semantic code search'' to completion.  It is a working tool with
CLI, TUI and Web UI, code parser filling graph and vector DBs providing symbolic
and semantic queries on a code base.

\begin{itemize}
\item See \href{https://www.phy.bnl.gov/\~bviren/talks/llm/claude-planning/plan-the-code.pdf}{previous presentation} for a walk through.
\end{itemize}

Main conclusion:

\begin{itemize}
\item The structured planning method worked well.
\item Tool essentially worked after all issues completed.
\item Gaps show through in result, after testing find polishing needed.
\begin{itemize}
\item Better and more consistent content formatting of query results.
\end{itemize}
\end{itemize}

Process questions this test raised:

\begin{itemize}
\item Would a plan-as-you-go approach (more ``agile'' than ``waterfall'') work better/faster?

\item How much does the choice of LLM matter?

\begin{itemize}
\item This work tries to address this.
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}[label={sec:orgb1974d6},fragile]{Comparing different LLMs for the planning phase}
 Give identical prompt (see backups) to different LLMs and evaluate the results.
\begin{itemize}
\item Define a \alert{concept} for a novel software project.
\item Initialize a greenfield Python package repo with \alert{uv}.
\item Install \alert{Beads} 0.57.0 with \alert{claude} CLI integration.
\item For each LLM: structured planning and work one ``research'' task.
\end{itemize}

The LLMs:

\begin{description}
\item[{qwen}] \texttt{unsloth/Qwen3.5-122B-A10B-GGUF:Q4\_K\_M} in llama.cpp running on both two RTX 4090 in wcgpu0.
\item[{haiku}] 4.5 on Azure
\item[{sonnet}] 4.6 on Azure
\item[{opus}] 4.6 on Azure
\end{description}
\end{frame}
\begin{frame}[label={sec:org3371439}]{The project used as fodder for this test}
The initial prompt starts with:

\begin{quote}
We will develop a software application that provides a “memory” database with
interfaces for LLMs and humans.
\end{quote}

\begin{itemize}
\item Scope is a super-set of the prior \alert{toenails} project.
\item I am mostly interested in the planning phase here and may not push to actual coding.
\item See backups for the full prompt.
\end{itemize}
\end{frame}
\begin{frame}[label={sec:org3a5e711},fragile]{Structure planning usage by the LLMs}
 \begin{center}
\begin{tabular}{lrrllll}
LLM & Input & Output & Create & Read & Total & cost\\
\hline
qwen & 1,734,719 & 6,475 & 0 & 0 & 1,741,194 & 0\\
sonnet & 10 & 564 & 50,217 & 206,142 & 256,933 & 26\textcent\\
opus & 11 & 860 & 39,428 & 246,205 & 286,504 & 39\textcent\\
haiku & 79 & 13 & 40,448 & 129,456 & 169,996 & 6\textcent\\
\hline
\end{tabular}
\end{center}


\begin{itemize}
\item Ran tests in the order listed.
\item Create/read count tokens in/out of Azure / Anthropic cache (``read'' cheaper than ``input'').
\item \texttt{ccusage} may under-estimate the cost.
\item Qwen ran on \$10k of local hardware so was not exactly ``free''.
\end{itemize}
\end{frame}
\begin{frame}[label={sec:org5ee5aec},fragile]{Quantified comparison of products from the planning}
 \begin{center}
\begin{tabular}{lrrrlr}
LLM & epics & tasks & total & bytes & time\\
\hline
Qwen & 7 & 21 & 28 & 20k & 4\\
Haiku & 12 & 23 & 35 & 25k & 1\\
Sonnet & 10 & 22 & 32 & 36k & 5\\
Opus & 12 & 22 & 34 & 37k & 8\\
\hline
\end{tabular}
\end{center}

\begin{itemize}
\item An \alert{epic} is an umbrella issue representing a major development with children \alert{tasks}.
\item The \alert{bytes} simply counts the data size for all generated issues dumped to JSON.
\item The \alert{time} is in minutes and not measured carefully.
\begin{itemize}
\item Qwen sometimes halted and I needed to issue ``continue'' prompts.
\item Times include time for me to react an ACK request from \texttt{claude}.
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}[label={sec:org23cccec},fragile]{Finding specific tasks}
 Use Beads to find tasks related to ``graph''.

\scriptsize
\begin{verbatim}
$ bd query title=Graph or title=graph or description=graph or description=Graph
\end{verbatim}
\normalsize

\begin{description}
\item[{qwen}] 4 issues (graph db with sqlite, graph query lang, schema, epic)
\item[{haiku}] 3 issues (graph schema, db options, epic)
\item[{sonnet}] 7 issues (schema, hybrid query, graph db with sqlite, KG extraction pipeline, epic, deps for FTS and sem search)
\item[{opus}] 15 issues (research: hybrid, extraction, SQL query, storage. deps, epic.
\end{description}
\end{frame}
\begin{frame}[label={sec:org9b8fcbe}]{Select similar issue - titles}
\begin{description}
\item[{qwen}] Research graph database options within SQLite
\item[{haiku}] Research Graph Database Options
\item[{sonnet}] Research graph representation options within SQLite
\item[{opus}] Research graph storage patterns in SQLite
\end{description}
\end{frame}
\begin{frame}[label={sec:org9741469}]{Select similar issue - descriptions}
\footnotesize

\begin{description}
\item[{qwen}] Research graph database options that work within SQLite including
recursive queries and graph-like structures
\item[{haiku}] Evaluate Neo4j, NetworkX (in-memory), SQLGraph, and other options for
knowledge graph storage. Consider Free/Open Source requirements and embedding
in sqlite3.
\item[{sonnet}] Evaluate approaches for storing a knowledge graph in SQLite:
adjacency list table (nodes + edges tables), recursive CTEs for traversal,
SQLite's built-in capabilities for graph queries (path finding, subgraph
extraction). Compare with using NetworkX as an in-memory graph layer loaded
from SQLite. Assess: query expressiveness, performance for typical knowledge
graph queries (neighbors, shortest path, subgraph), Python library options
(networkx, igraph, rustworkx). Output: recommended graph storage and query
approach. Feeds into so-9l2.
\item[{opus}] Evaluate approaches for implementing a knowledge graph within
SQLite3. Consider: adjacency list (edges table), closure table (for transitive
queries), nested sets, materialized path, and recursive CTEs for
traversal. Evaluate each for: query complexity, write performance, traversal
depth support, and suitability for a knowledge graph with typed entities and
relationships. Look at existing projects (e.g., simple-graph for
SQLite). Output: recommended pattern with example schema and queries.
\end{description}
\end{frame}
\begin{frame}[label={sec:org62c0a00}]{Aside observation: Opus ``likes'' research}
Opus had 4 research issues related to the graph DB.  Each with a different aspect.

\begin{itemize}
\item Research hybrid search and result merging strategies
\item Research entity and relationship extraction
\item Research graph traversal query patterns in SQL
\item Research graph storage patterns in SQLite
\end{itemize}
\end{frame}
\begin{frame}[label={sec:org3759a8b}]{Put Haiku and Sonnet to their respective research tasks}
\begin{quote}
Perform the research as directed in issue ID-XYZ.  Produce a report in markdown
format and add it to the issue in the ``Design'' section of the issue.
\end{quote}
\begin{itemize}
\item I filled in the appropriate ID-XYZ issue ID for each case.
\item See prior slides for the content of each LLM's issue.
\end{itemize}

Not apples-to-apples, each LLM consumes the prompt it genreated.
\begin{itemize}
\item Haiku task was simpler, focused on tech comparisons.
\item Sonnet task was more conceptual and nuanced.
\end{itemize}
A better/future test would start them with identical prompts.  
\begin{itemize}
\item Perhaps using tasks generated by Opus.
\end{itemize}
\end{frame}
\begin{frame}[label={sec:org489d600}]{Observations on Haiku and Sonnet research task results}
\begin{itemize}
\item Haiku is much faster than Sonnet.
\begin{itemize}
\item I assume Opus would run even longer.
\end{itemize}
\item Haiku result is unexpectedly more verbose (392 lines) than Sonnet (233 lines).
\begin{itemize}
\item Both results are good so not sure longer or shorter is better.
\end{itemize}
\item Haiku gives sources as links, Sonnet does not.
\item Haiku report is bullets and tables. Sonnet report is prose, tables and code snippets.
\item Both give options and recommendations.
\end{itemize}
\end{frame}
\begin{frame}[label={sec:orgdce01e3}]{Some conclusions on LLM planning performance}
\begin{itemize}
\item To optimize time, cost and quality, a tiered approach is probably best.
\begin{itemize}
\item Opus, Sonnet, Haiku obviously on a spectrum.
\item Consider the conceptual vs research vs implementation complexities of a task.
\end{itemize}
\item I am forming a bias for:
\begin{itemize}
\item Opus or Sonnet for initial plan-for-a-plan.
\item Opus or Sonnet for plan break down into issues.
\item Sonnet or Haiku for researching tasks.
\item Sonnet or Haiku for implementation.
\end{itemize}
\item Qwen 3.5 is not adequate compared to even Haiku for software development.
\begin{itemize}
\item Qwen seems still capable for tasks like summarizing.
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}[label={sec:org6989c7c}]{Some conclusions on structured planning}
Especially with Opus, can produce an explosion of plan and task issues.
\begin{itemize}
\item Requires a lot of human attention (but more than other approaches?)
\item Some new ingredient to automate or simplify human effort more would be helpful.
\end{itemize}
Perhaps a change viewpoint is needed:
\begin{itemize}
\item Think of issues as ``reservoir of possible ideas'' instead of a strict plan to follow?
\item Move more toward ``agile'' and away from ``waterfall'' model?
\end{itemize}
\end{frame}
\begin{frame}[label={sec:org98b3038}]{}
\Huge
\[\mathcal{FIN}\]
\end{frame}
\begin{frame}[label={sec:org065f310}]{Backups}
\end{frame}



\begin{frame}[label={sec:org52e5ee5},fragile]{Qwen + llama.cpp + claude setup}
 \tiny

\begin{verbatim}
$ uv init --build-backend setuptools --lib llmemory
$ cd qwen/llmemory/

$ bd dolt start
$ bd init --prefix lm
$ bd setup claude
$ bd doctor --fix

$ ~/opt/llama-cpp/bin/llama-server \
          -hf unsloth/Qwen3.5-122B-A10B-GGUF:Q4_K_M \
          --alias unsloth/Qwen3.5-122B-A10B \
          --ctx-size 131072 \
          --temp 0.6 \
          --top-p 0.95 \
          --top-k 20 \
          --min-p 0.00 \
          --host 127.0.0.1 --port 8181

$ ANTHROPIC_BASE_URL=http://localhost:8181 \
  ANTHROPIC_API_KEY="sk-1234" \
  claude

> /model unsloth/Qwen3.5-122B-A10B
\end{verbatim}

API key is unused, put anything.  Parameters from Unsloth guide, no optimization done but had to increase \texttt{-{}-{}ctx-size} to get \texttt{claude} to start.
\end{frame}
\begin{frame}[label={sec:org4ce8315}]{Initial prompt}
\tiny
\begin{quote}
We will develop a software application that provides a ``memory'' database with
interfaces for LLMs and humans.  It is called ``llmemory''.  I will describe the
development process and then give an initial description how ``llmemory'' will be implemented.

The development process will follow a hierarchically structured planning and
implementation process (similar to ``waterfall development process'').  It will
use Beads issues with dependencies and issue types to organize, track and drive
the development.  Actual planning and coding will be mostly performed by LLMs
while the human developer will review and modify the issues and code.  

We will create the following beads issue types that covers both planning and coding work.

\begin{itemize}
\item `epic` - A high level software design consideration or a major aspect or concern of the implementation.
\item `feature` - A specific feature of the implementation to provide.
\item `task` - Specific research to perform to provide input to answer a software design question or specific coding work to implement a portion of a feature.
\item `bug` - An issue describing something that must change in the planning or the code.
\end{itemize}

Start by creating ``epic'' issues that cover software design considerations.
There should be an epic for each major feature required and for software
technology choices.  For each ``epic'', define ``task'' issues that describe what
research is needed to help answer the epic, include software technology options,
any initial pros and cons.  Each ``task'' will be executed by an LLM or a human
and the result will be a ``design'' section added to the task, as well as comments
and implementation tasks will be created in order to consider the planning task
complete.

The initial philosophy and choices for this application are as follows:

\begin{itemize}
\item This application and all its dependencies must be Free Software or at least
Open Source Software.

\item All data is held locally.  Use a single sqlite3 database to store all ``memory''
wherever possible even if options exist that may have higher performance.  If
sqlite3 is wholly unsuitable, consider alternatives.

\item Implementation is in Python, managed by ``uv'' with Pytest unit tests of good
coverage.  UIs included: CLI based on Click with well formatted output using
TBD module, TUI based on TBD module, GUI based on Qt, web based on Flask and
TBD web framework.  MCP based on TBD module.  A simple text and JSON output
also available.

\item Information retrieval methods include: RAG for semantic search.  Graph
database for knowledge graph.  Full text search with BM25 ranking.  Each of
these require research to flesh out their features, determine implementation
methods, datatabase schema and how they may interoperate.
\end{itemize}

Given this introduction, produce beads epic issues covering design
considerations and dependent tasks covering research and other work needed
toward producing a design specification.
\end{quote}
\end{frame}
\end{document}
