Writing a whole toolset for a minor development task

Posted Saturday, 31-Oct-2026 by Ingo Karkat

For the past few days, I've worked on addressing a set of development tasks. In true the journey is the reward manner, the actual goal was a minor consistency improvement to my scripts, which could have been done manually. But I saw potential for an automation that I could benefit from in several other, far more important uses. Unimportant tasks provide the perfect opportunity to develop skills and tools that you'll miss dearly when stuff hits the fan and you're under pressure to find a quick solution under stress.

895 +swdev find multiple separate Usage: sections in mann-all via grepDashedSections
using an sequentialgrep grep override and refactor them using LONGUSAGE prefixes
824 +tools-grep add grepSections that resolves line number matches to preceding headers,
extracts the header and then uses a HEADER:OFFSET:MATCHING-LINE output format !:1
893 +sw-shell-tools write filter for eachSplit's {M} mapping marker translation of
piece files to piece info
894 +tools-sed write sequentialgrep
	

And here's the final command-line that returned the ~60 scripts (a few results were false positives) that I had to adapt:

$ mann-all \
| GREP=sequentialgrep grepAllDashedHeaders \
--per-file --files-with-matches \
-e 'Usage: ' -E 'Usage: \|^$' -e 'Usage: ' \
| linesToArgs command -v \
| fgF 'LONGUSAGE='
/home/inkarkat/bin/ingo/configuring/bin/addDesktopEntry
[66 lines deleted]
/home/inkarkat/bin/xt/todo.sh/bin/todo-local-project
	

Let's break it down:

• I'm using mann-all here; I could have generated the usage help on the fly via Unixhome-ingo, but with about 2500 commands, invoking all of them would have taken quite a while, and I had written this cache because I need to search my commands' usages often.
  mann-all --help

  Show the cached collated reference manuals of all of my custom commands for easy
  searching and filtering.

• The second pipeline step, grepAllDashedHeaders runs the search on the cached output. I'm interested in the matching commands, which normally would be the filespecs passed to grep, but mann-all concatenates all usage help with dashed headers that contain the command's name:
  abbreviated output of mann-all

  -- addIcon ---------------------------------------------------------------------
  Add ICON to the user's / system application icons.
  
  Usage: addIcon [--check|--copy] [-f|--force] [-g|--system-wide] [--] ICON [...] [-?|-h|--help]
  
  -- addInstalledFile ------------------------------------------------------------
  Copy SOURCE-FILE to DEST-FILE and set attributes (via install).
  
  Usage: addInstalledFile [--accept-up-to-date|--check [--no-compare-check|--check-with-sudo]] [--sudo] [-q|--quiet] [INSTALL-ARGS ...] [--] SOURCE-FILE DEST-FILE [-?|-h|--help]
  			

  I initially had intended to write a custom grep variant that would understand that structure, but that would have been quite involved, especially because grep has so many options! Instead, I went with a less efficient algorithm that would split the structure into individual temp files, so that normal grep could be invoked on them. I had already written a generic eachSplit, a variant eachCsplit that uses the Unix csplit command, a specialization eachSection taking just a single PATTERN, and two specializations eachHeader and eachDashedHeader for the two header formats I'm using.

  I first wanted to build grepDashedHeaders directly on top of that, but then implemented an onAllSplits specialization, as eachSplit's main use case was executing a command on each file individually; it had acquired the ability to execute a command on all files together only later. onAllSplits (and the variants for sections, headers, and so on) provides a cleaner API for executing command(s) on all splits together.

  I still had to translate the temporary filespecs (that commands like grep would see and report) back to the original extracted section information. This is done by an applyMapping wrapper command that directly understands the mapping file that eachSplit generates.

  Then, commandOnAllSplits extends onAllSplits by automatically applying that mapping. It also provides a streamlined API that just takes a single COMMAND (injected via COMMANDONALLSPLITS_COMMAND environment variable), limiting its applicability, but making wrappers like the grepAllSplits trivial to write:

  grepAllSplits (implementation sketch)

  #!/bin/bash
  COMMANDONALLSPLITS_COMMAND="${GREPALLSPLITS_GREP_COMMAND:-${GREP:-grep}}" \
  exec commandOnAllSplits "$@"

  (The actual implementation has three more lines for adapting the usage help and providing customization hooks for the other variants.)

• Unfortunately, the task of finding multiple separate usage sections cannot be solved via single line-based pattern matches. It's characterized by at least two matches of Usage:, with other line(s) in between that contain different text (not just empty lines). So I implemented a sequentialgrep command in AWK that so far only supports those grep options I need: positive (-e) and negative (-E) matching, output of matching filespecs (--files-with-matches), and a flag for disallowing matches across files (--per-file). With that, I can identify the affected files:

  sequentialgrep --per-file --files-with-matches -e 'Usage: ' -E 'Usage: \|^$' -e 'Usage: '

• This grep variant is injected into grepAllDashedHeaders via the GREP=sequentialgrep environment variable.
• That gives me the filenames of all affected commands. To open them in my editor, I need the full filespecs. As all commands are in my PATH, command -v will look them up. The command takes multiple arguments, so I can use my linesToArgs to feed them in.
• Finally, I need to filter out those files that have already been converted. After the refactoring, there's a LONGUSAGE string present, so I just need to filter input files for non-matching files; a job for my fgF alias, which is similar to the fackF alias I've already introduced in a previous entry.

Ingo Karkat, 31-Oct-2026

ingo's blog is licensed under Attribution-ShareAlike 4.0 International