Project Isidore Design Notebook

Welcome! This is the documentation webpage of my personal website. The canonical source code is located at Ben / Project Isidore · GitLab. For usage, please see the user manual and the navigation bar located at the top of the website.

• Common Lisp Environment Setup - How to get started with Lisp
• Org Publish Pipeline Section - How I Blog from Org Mode
• High Level Project Overview - How to navigate the source tree
• Project Isidore User Manual - Tutorial & Guides
• Project Isidore Reference Manual - For Developers

Navigation buttons are inserted next to each header to take one back to the table of contents.

Copyright (c) 2021 Benedict H. Wang. Source code is under the GNU-AGPL-3.0 License. Blog content is available under the CC-BY-SA 4.0 License unless otherwise noted.

• <–Common Lisp Config
• <–Practical Common Lisp
• <–Linux Program Directory

He is like to a man building a house, who digged deep and laid the foundation upon a rock. And when a flood came, the stream beat vehemently upon that house: and it could not shake it: for it was founded on a rock. But he that heareth and doth not is like to a man building his house upon the earth without a foundation: against which the stream beat vehemently. And immediately it fell: and the ruin of that house was great.

–Luke 6:48-49

Why lisp? There are always more poets than non-poets in the world, or so I've heard.

Practically speaking, a complete and recent tour of the LISP world for beginners has already been written: Steve Losh - Intro to Common lisp. Steve Losh characterizes the field of web development – at least in Anno Domini 2021 – not as a hamster wheel of backwards incompatibility, but a hamster centrifuge. A house of sand, indeed. In technical aspects, Paul Graham's writings convey any advantages better than I can.

Below are the steps I have taken to setup a Common Lisp environment inside Spacemacs.

• CL the Language = ANSI INCITS 226-1994
• CL Implementation = Steel Bank Common Lisp If you've deep pockets or love reading about the history of Lisp implementations: The History of Franz and Lisp | IEEE Journals & Magazine | IEEE Xplore Common Lisp - Myths and Legends
• CL Library/Package Manager = Quicklisp
• CL System Manager/Build Tooling = Another System Definition Facility (comes bundled with SBCL)
• CL IDE = Spacemacs with SLY

• Install Steel Bank Common Lisp, a Common Lisp implementation.

  Other implementations of Common Lisp the language exist, but SBCL is the premier open-source implementation as of this writing.

  Installing SBCL from your Linux distribution's package manager is the most straight forward way. On Debian/Ubuntu this is as simple as

  sudo apt install sbcl
  

  Downloading and unpacking a binary from http://www.sbcl.org/getting.html or building from source are options for later discretion. In the shell,

  sbcl
  * (+ 2 3)
  * (quit)
  

  will produce the startup SBCL banner and REPL, evaluate an expression, and quit.

• Install Quicklisp - the Common Lisp Package Manager

  In the shell,

  curl -O https://beta.quicklisp.org/quicklisp.lisp
  sbcl --load quicklisp.lisp
  ## Inside the SBCL prompt,
  (quicklisp-quickstart:install)
  (ql:add-to-init-file) # autoload into sbcl initialization
  (quit)
  

  After loading – via (ql:quickload :project-name) – a project, it will be stored locally. It will be in a directory similar to:

  /home/$USER/quicklisp/dists/quicklisp/software/example-project-2020-01-01-git
  

  Also, (ql:where-is-system 'system-name) will return the system's location. project-name and system-name are interchangeable here.

  In order to (ql:quickload :your-local-project) Quicklisp looks in /home/$USER/quicklisp/local-projects/ for said project. You can symlink your project if you desire to use some other folder.

  ln -s /home/$USER/quicklisp/local-projects/project-a/ /home/$USER/project-a/
  

  Not of primary importance yet is a definitions of what entails a lisp package or lisp system. It will explained in proper order during your introduction to the language, to be more specific it is covered in Peter Seibel's excellent pedagogical work, Practical Common Lisp and other guides are located here and here. N.B. that (ql:quickload :your-local-project) also calls (asdf:load-system :your-local-project). The difference is (ql:quickload) will download any missing system dependencies.

• Install Spacemacs common-lisp layer - a Common Lisp IDE

  The below steps assume you are already familiar with Spacemacs. Inside Spacemacs, SPC-h-SPC RET common-lisp RET and follow the layer README.

  For those who are understandably wary of Emacs other IDE options exist. Lisp is one of the oldest higher level languages (beaten by FORTRAN by a few years). With that rich tradition comes a time where the underlying hardware, operating system, and developer tooling were unified under lisp. Unfortunately, the closest one can reasonably come today is Emacs, a lisp interpreter running on C/UNIX/hardware. Emacs still offers the best-in-class experience amongst open-source offerings, but the notorious learning curve of Emacs can be tempered with a preset configuration: Spacemacs. For those on a Microsoft Windows machine, I have written installation instructions.

• Optional: Enable goto definition for SBCL primitives

  Download CL Implementation source files and extract them to the location specified by (sb-ext:set-sbcl-source-location), which is set in your user configuration dotfile: /home/$USER/.sbclrc. Add it if not already present,

  (sb-ext:set-sbcl-source-location "/usr/share/sbcl-source/")
  

  Now you can use g d to call jump-to-definition to goto the raw documentation: the source code.

• Optional: Enable offline access to the CL HyperSpec language reference

  The full text of the ANSI Common Lisp Standard (1994) is available online in HTML form. To have the reference handy offline and to be able to browse it within Emacs, first download and extract the HTML source for HyperSpec 7.0.tar.gz from the great Internet Archive.

  Then we can configure Emacs to open only HyperSpec links inside the Emacs web browser EWW and also inform our Common Lisp IDE of the HyperSpec location.

  (setf common-lisp-hyperspec-root "file:///home/$USER/HyperSpec/")
  ;; Optionally, execute the HYPERSPEC-LOOKUP function with local variable
  ;; changes to view HyperSpec links exclusively in EWW.
  (advice-add 'hyperspec-lookup
              :around
              (lambda (orig-fun &rest args)
                (setq-local browse-url-browser-function 'eww-browse-url)
                (apply orig-fun args)))
  

  Now , h H will call sly-hyperspec-lookup to peruse the symbol at point with the HyperSpec. Note the default behavior of sly-hyperspec-lookup is to open a web browser at the online HyperSpec.

Newcomers to the language are advised to dive right into the recommended reading. Now's a good chance to use org-babel for some note-taking. If you've forked the repo to play around with, a high-level overview of the project may be useful.

Bibliographies exist for a written corpus of work. The same sort of metadata is needed a code base. In Common Lisp, it is the .asd file.

Project Isidore follows the common Model-view-controller (MVC) design pattern. It entails encapsulating data together with its processing (the model) and isolating it from the manipulation (the controller) and presentation (the view) part that has to be done on a user interface.

The project follows the ASDF package inferred system style of using defpackage forms to specify file inter-dependencies. The entry point of dependency graph is packages.lisp. To understand the code base, the files are named after the MVC design pattern.

For an index of symbols, functions and definitions, see the Reference Manual.

In the /assets/blog/ folder, all HTML files are generated by org-publish. archive.html lists all published articles.

In Spacemacs you will need to enable the org layer. Org Mode comes with Org Publish built in. Org Publish takes advantage of Org's excellent export capabilities to generate not only the HTML but also the sitemap and RSS.

To publish/update an article:

1. Prepare draft/existing notes for publishing, move .org file to input folder, remove extraneous links, update citations, check formatting etc.
2. M-x RET org-publish RET blog
3. Commit and push changes in git

Here is configuration for my blog: https://www.bhw.name/assets/blog/dotspacemacs.html#Org-Publish-Config. Other exemplars here: good tutorials on org-publish : emacs, New and Improved: Two-Wrongs Now Powered By Org Mode

Follow industry best practice in automating part of development operations. In the context of this project, CI/CD is done on the Github Actions platform. Cheers to Github for allowing unlimited build limits for open source projects! In the following breakdown, I explain how to run the steps on the local machine.

For the uninitiated, an excellent git porcelain (with spacemacs layer integration) is Magit. This, paired with [chaconProGit2014], takes care of your version control needs.

Project Isidore uses an embedded database (Rucksack) over more typical client-server RDBMS's such as PostgreSQL. Extremely heavy read-heavy data or whatever data that ought to be cached are stored by the in-memory object prevalence model (BKNR.Datastore). Moore's law has brought us significant improvements, and as a result SQLite is a viable choice for this application.

Key value > multi column > document > relational > graph

Graph Databases, the Semantic Web and RDF stores have capture my attention. In the open source common lisp world,

Open source is behind, just look at the customer list of awe neptune, allegrograph, neo4j etc.

vivace graph de.setf.resource

https://www.lassila.org/ wilbur author, now works for aws neptune.

project mage and the promise of a prototype OO gui toolkit. clog by a rabbi is already a great web gui toolkit vivace-graph rivalling neo4j in performance.

just like XML is a poor, verbose re-implementation of s-expressions, RDF is an equally verbose reimplementation of Prolog clauses.

https://people.well.com/user/doctorow/metacrap.htm

"The semantic web is the future of the web, and always will be." – Peter Norvig

https://www.youtube.com/watch?v=LNjJTgXujno&t=1257s

Good proprietary solutions exist: Lispworks' CAPI and Allegro CL's Common Graphics. Heck, even a GUI standard with heritage dating back to Symbolics' Lisp machines exists. There is a free software implementation of said standard. If your constraints are open-source, cross-platform portability, and limited developer time, then currently only one practical option remains: web browsers. An informal but historical look at the evolution of Graphical User Interfaces (GUI) in the context of web applications shows us the high rate of change, depicting a field is still in its infancy.

Second, HTML over WebSockets is nice, but what is better is reactive data binding over the socket. Let the browser maintain a DOM tree that templates over a JSON object, then reactive-ly update and it's an amazing experience. You have minimal data transfer, and minimal rendering updates. There is a great deal of power in having your UI be a stateless function over a giant JSON object.

• Mathgladiator

Why a JSON object? Why not a CLOS object? Why Mathgladiator's DSL when Lisp makes it (too) easy to create a more fitting DSL?

This seems like X Windows protocol over HTTPS with JSON as the wire encoding.

It's sad to think how much effort is spent reimplementing technologies because the IPv4 address space wasn't large enough.

If IP address space were larger, we wouldn't have needed to rely on Network Address Translation (NAT). NAT became a poor-man's firewall and effectively blocked peer-to-peer connections making it difficult to host a server at home. This incentivized early ISPs to offer only asymmetric connections that pushed users towards a consumer-only service that crippled early Internet protocols. With the World Wide Web being "the" reason people got on to the Internet in the first place, early NAT-enabled routers focused only on HTTP tunneling, so completed protocols that required inbound connections were finally killed off.

Who would have thought that a small IP address space could result in a mass protocol extinction event from which only HTTP and HTML would survive….

• c-linkage

An understanding from first principles. If the problem can be solved with a stateless protocol, then RESTful HTTP is beautiful. To (ab)use the HyperText transfer protocol for anything drastically more complex than linked text documents, when there is state to be managed, when a duplex connection is needed, is folly. It leads to the sort of accidental complexity that plagues current web development. Pick the right application layer protocol for the right problem.

If we pull our heads out of our behinds and look at the broader trends in e.g. the gaming industry, the trend is fully interactive applications rendered in your browser using WASM. All the big cloud companies and several gaming companies are doing that currently. They completely engineered away the browser limitations of only being able to run javascript and only being able to render DOM trees. That API was painful to deal with 20 years ago and it has not improved a whole lot. 20 years ago remote UI was painful. Now you can play the latest games at 60 fps on a cheap chrome book, ipad, or even a phone.

[…]

Not saying that this is the future. But we seem stuck in a lot of abstractions and constraints imposed on us in the late nineties when Brendan Eich et al. gobbled together a scripting language in a hurry and HTTP 1.1 was frozen in time until SPDY became a thing only a few years ago. Most of the past 20 years is just increasingly more sophisticated hacks to work around andundo all of that.

[…]

We had graphical tools for clicking together UI in the nineties (Delphi, Visual Basic, JBuilder, etc.). That stuff worked great. OK, it resulted in a lot of ugly and unusable applications. But then most SAAS browser applications that replaced them aren't a whole lot better or less painful to deal with.

• jillesvangurp

Mr. Eich did a fair job considering the timeframe and the political pressure he was under, for the man himself admits he was hired with the promise of "scheme in the browser". But a stream cannot rise higher than its source. Common Lisp has its warts too, but I know which language I prefer to have fun in. In my dreamworld, the proprietary solutions would be my first choice. Lacking that, perhaps SBCL will target WebAssembly one day, or McCLIM and Project Mage will duke it out. But until that day comes, it's not all bad in Lispland. As Rabbi Botton comments himself on the thread, in the Common Lisp world we have CLOG to recapture some of the 90's Delphi magic I was too young to have ever experienced. Unpoly deserves a mention if one needs HTTP and a language-agnostic framework for a not-too-complex web application.

In doing research for the selection of a portable GUI, scaling is often mentioned as a challenge to this approach. Keeping an open socket is also bound to use more power as well. However, as a solo developer/dilettante, the developer experience and productivity it offers is unparalleled. I better get to learning how to dance in CLOGs!

Ben Shneiderman

A through treatment of the generalities of optimization in Common Lisp can be found in weitzCommonLispRecipes2016 pages 503-560. Dr. Weitz testifies that lisp (SBCL) can confidently reach within a 5x ballpark of C. Less, obviously, if there are many fixnums. SBCL compiler contributor Paul Khuong has also testified to a ballpark of 3x within C. Of course, squabbling over language performance is time better spent on data structures and algorithms, but to have a ballpark estimate is good to know. Python and Ruby for example, are magnitudes slower than Lisp. Ballpark of 20x and 40x respectively. For a dynamically typed, high level language, Lisp performs admirably. Especially striking is the non-leaky abstraction and degree of control provided to the programmer at read, compile and runtime; see the "disassemble" function and runtime compilation tricks.

Lastly, Professor Scott Fahlman, one of the original designers of Common Lisp, weighs in on his experience circa 2020.

Short answer: I don’t know about the Clozure compiler, but the compiler used in the open-source CMU Common Lisp (CMU CL) system produces code that is is very close in performance to C a little faster for some benchmarks, a little slower for others.

But there are some things a Common Lisp user needs to understand in order to get that performance. Basically, you need to declare the types of variables and arguments carefully, and you should not do a lot of dynamic storage allocation (“consing”) in performance critical inner loops usually just 10% or 20% of your total system.

(Steel Bank Common Lisp (SBCL) is essentially the same as CMU CL in terms of the performance of compiled code. The CMU CL open-source community split in two in December 1999 as a result of some disagreements about design and philosophy, and one branch was renamed SBCL. I believe that the parts of the compiler concerned with optimization have not changed much. I currently prefer SBCL for my work on the Scone knowledge-base system and other things.)

The Java compilers I know about produce code that is considerably slower than CMU CL and SBCL. I don’t know much about Haskell performance. C++ is similar to C in performance if you use it like C; I believe that if you make heavy use of the object-oriented features it is considerably slower.

Longer answer, for nerds: I was one of the core designers of Common Lisp. (We were sometimes referred to in those days as the “Gang of Five”.) I wrote what I believe was the first Common Lisp compiler (drawing heavily on the design of earlier compilers for Maclisp and Lisp Machine Lisp).

For many years I ran the CMU Common Lisp project. As part of that project, we developed a public-domain Common Lisp runtime that became the basis for a number of commercial Common Lisp implementation, adapted and supported by various large companies for their own machines.

David B. McDonald, in my group, spent something like four years developing a very highly optimizing compiler for Common Lisp. We called this the “Python” compiler, which caused some confusion when the Python programming language became popular more than a decade later. No relation.

(I proposed that we name our compiler “Python” because a python the snake eats a whole pig, then goes under a bush for several weeks to sleep. The pig makes its way slowly through the snake’s internal pipelines, ultimately emerging as a very compact pellet. Which is pretty much what compilers do, to one degree or another.)

At the time the early 1980s I was starting to work on programs for implementing (or simulating) artificial neural nets. These needed very efficient floating-point arithmetic and vector operations, and I wanted to be sure that we could efficiently program these things in CMU Common Lisp. But at the time, Common Lisp had the reputation of being really awful at floating point a straightforward implementation would constantly be allocating “boxed” floating-point numbers that had to be garbage-collected later.

So Dave McDonald labored mightily over type inference and non-consing ways to handle floating point, and he got the job done. I wrote some neural-net programs in CMU CL, they were later translated into C for wider distribution (by an undergrad coding wizard who really knew what he was doing). The two versions were very close in runtime. When DARPA pulled the plug on support for Common Lisp development, CMU Common Lisp became an open-source project with a different set of developers/maintainers. Both our runtime and our “Python” compiler are part of that distribution (and of SBCL), though of course there has been some evolution since then.

What programmers need to know to get good performance in Common Lisp: People speak of Lisp as a “dynamically typed” language. I think it is more correct to call it (at least for the CMU CL implementation) an “optionally strongly typed” language. The philosophy is this: Programmers can say as much or as little as they like about the type of an argument or value. Whatever you say had better be true you can tell the compiler to trust the declarations or to be suspicious. The more precisely you specify what the entity is, the more likely it will be that the compiler can do some clever optimization based on what you told it.

So, for example, you could say just “number” or “integer” or “integer between 0 and 1024” or “prime number between 0 and 1024”. If you use a very general declaration, the code will work, but it will have to do some runtime type-checking to see what kind of number it is dealing with. It must be ready to deal with some of the exotic number-types that Common Lisp supports: infinite-precisions integers (“bignums”), ratios of integers, imaginary numbers, several levels of floating-point precision, and so on. There is a special, very compact and efficient format that can be used for small integers, but it can only be used if you tell the compiler what to expect.

The same is true of things other than integers: There are several array formats. You can specify what size/shape data to expect, or you can wait and see what someone hands you at runtime. When you get into object-oriented programming, you can tell the system what to expect (and it can figure out more internal data-types via type inference), or you can wait and see what you get and do a runtime type-dispatch to find the proper method to use. That takes time.

So, if you want good performance in Common Lisp, especially for arithmetic and array operations, you have to declare the types as precisely as you can. Then the compiler will do its magic.

Flash forward: Sometime around 2001, I was working for IBM Research (on leave for a while from CMU long story…) and my project was to implement an early version of what became (after several restarts) my Scone knowledge-base system. At the time, the prevalent language in IBM Research was Java, and I knew that it would be hard to interest the IBM people in my system if I did it in Lisp. So I started out doing it in Java.

This system had essentially no arithmetic in it, but did a lot of pointer-chasing off into main memory, a lot of boolean operations, and had a few very intensive inner loops where it spent all its time. Common Lisp was clearly the right tool for this job, but I worked hard on the Java version. Finally, for reasons too complicated to go into here, I decided that I could no longer stand programming the system in Java some very important facilities were missing, especially the Lisp macro system so I decided to port the half-done system to Common Lisp. (As predicted, IBM then lost interest, and I returned to CMU.)

The performance-critical inner loops had already been programmed at that time, so I was able to compare the performance of the Lisp and Java versions. The Lisp version was about 3X faster. Part of that difference was because I was a very good Lisp programmer, and rather new at Java, though I had talked to Java experts about how to get good performance for code like mine. So some of the performance difference was experience, but mostly it was because at the time (and I think still today) Java did not do a lot of object-type inference at compile time. So pretty much every function call was a type-dispatch to find the right method, and that was slow.

There wasn’t a good way around this inefficiency except by writing the performance-critical code as long linear stretches of code with no function calls. And without a good macro system, that was much too tedious.

What follows is an amateur recording of my struggles, useful to jog my memory in the future when working with the SBCL statistical profiler. I had a pretty good idea the function call that needed to be profiled, but if it was a foreign system, I would try a library like Daniel Kochmański / metering · GitLab.

;; Load project into image.
(ql:quickload :project-isidore)
;; Load SBCL's statisical profiler.
(require :sb-sprof)
;; Includes COMMON LISP USER package symbols as defined in the Hyperspec.
(sb-sprof:profile-call-counts "CL-USER")
;; Profile and output both graph and flat formats.
(sb-sprof:with-profiling (:max-samples 5000
                          :report :graph
                          :loop t
                          :show-progress t)
  (project-isidore:bible-page "1-1-1-3-3-3"))

           Self        Total        Cumul
  Nr  Count     %  Count     %  Count     %    Calls  Function
------------------------------------------------------------------------
   1   2004  40.1   2004  40.1   2004  40.1        -  EQUALP
   2    299   6.0   1054  21.1   2303  46.1        -  (SB-PCL::EMF SB-MOP:SLOT-VALUE-USING-CLASS)
   3    205   4.1    205   4.1   2508  50.2        -  (LAMBDA (SB-PCL::.ARG0.) :IN "SYS:SRC;PCL;DLISP3.LISP")
   4    193   3.9    443   8.9   2701  54.0        -  (SB-PCL::FAST-METHOD BKNR.SKIP-LIST:SL-CURSOR-NEXT (BKNR.SKIP-LIST:SKIP-LIST-CURSOR))
   5    190   3.8   3896  77.9   2891  57.8        -  REMOVE-IF-NOT
   6    177   3.5    212   4.2   3068  61.4        -  COPY-LIST
   7    165   3.3    165   3.3   3233  64.7        -  (LAMBDA (SB-PCL::.ARG0.) :IN "SYS:SRC;PCL;BRAID.LISP")
   8    157   3.1    157   3.1   3390  67.8        -  (LAMBDA (SB-PCL::.ARG0.) :IN "SYS:SRC;PCL;PRECOM2.LISP")
   9    143   2.9    143   2.9   3533  70.7        -  foreign function syscall
  10    141   2.8    141   2.8   3674  73.5        -  SB-KERNEL:TWO-ARG-STRING-EQUAL
  11    134   2.7    134   2.7   3808  76.2        -  (LAMBDA (CLASS SB-KERNEL:INSTANCE SB-PCL::SLOTD) :IN SB-PCL::MAKE-OPTIMIZED-STD-SLOT-VALUE-USING-CLASS-METHOD-FUNCTION)
  12    121   2.4    121   2.4   3929  78.6        -  (LAMBDA (SB-KERNEL:INSTANCE) :IN SB-PCL::GET-ACCESSOR-FROM-SVUC-METHOD-FUNCTION)
  13    110   2.2    110   2.2   4039  80.8        -  (LAMBDA (SB-PCL::.ARG0.) :IN "SYS:SRC;PCL;BRAID.LISP")

Self is how much time was spent doing work directly in that function. Total is how much time was spent in that function, and in the functions it called. As this is my own code, I know that REMOVE-IF-NOT calls EQUALP in a lot of functions. But it can also be a probable hypothesis from just this data alone. Cumul is obviously the additive results of the Self column. I have cut it off at 80% in light of the Pareto principle. The hypothesis can be confirmed from looking at the graph formatted portion of the profiler output, pasted below.

------------------------------------------------------------------------
  3840  76.8                   PROJECT-ISIDORE/MODEL:GET-BIBLE-UID [99]
    44   0.9                   REMOVE-IF-NOT [5]
    55   1.1                   PROJECT-ISIDORE/MODEL:GET-HAYDOCK-TEXT [96]
   190   3.8   3896  77.9   REMOVE-IF-NOT [5]
     1   0.0                   (LAMBDA (PROJECT-ISIDORE/MODEL::X) :IN PROJECT-ISIDORE/MODEL::FILTER-LIST-BY-VERSE) [114]
     1   0.0                   (SB-PCL::EMF SB-MOP:SLOT-VALUE-USING-CLASS) [2]
     1   0.0                   (LAMBDA (SB-PCL::.ARG0.) :IN "SYS:SRC;PCL;DLISP3.LISP") [3]
     1   0.0                   foreign function alloc_list [29]
    43   0.9                   (LAMBDA (PROJECT-ISIDORE/MODEL::X) :IN PROJECT-ISIDORE/MODEL::FILTER-LIST-BY-CHAPTER) [30]
   140   2.8                   SB-KERNEL:TWO-ARG-STRING-EQUAL [10]
  1421  28.4                   (LAMBDA (PROJECT-ISIDORE/MODEL::X) :IN PROJECT-ISIDORE/MODEL::FILTER-LIST-BY-BOOK) [17]
  2004  40.1                   EQUALP [1]
    44   0.9                   REMOVE-IF-NOT [5]
    47   0.9                   (LAMBDA (PROJECT-ISIDORE/MODEL::X) :IN PROJECT-ISIDORE/MODEL:GET-HAYDOCK-TEXT) [38]
------------------------------------------------------------------------

GET-BIBLE-UID is expected to take up a large portion of function calls based on my design choices and some back of the napkin math. The profiler has confirmed the information. Let's see if we can't optimize this particular function further.

(time (project-isidore:bible-page "1-1-1-3-3-3"))

Evaluation took:
  64.490 seconds of real time
  64.673257 seconds of total run time (64.032109 user, 0.641148 system)
  [ Run times consist of 2.276 seconds GC time, and 62.398 seconds non-GC time. ]
  100.28% CPU
  160,990,605,382 processor cycles
  16,089,927,136 bytes consed

Replacing equalp with eql in places where appropriate in the file model.lisp . For the best explanation on the different equality predicates in Common Lisp, see Equality in Lisp - Eli Bendersky's website.

Lisp's equality operators are:

= compares only numbers, regardless of type.

eq compares symbols. Two objects are eq if they are actually the same object in memory. Don't use it for numbers and characters.

eql compares symbols similarly to eq, numbers (type sensitive) and characters (case sensitive)

equal compares more general objects. Two objects are equal if they are eql, strings of eql characters, bit vectors of the same contents, or lists of equal objects. For anything else, eq is used.

equalp is like equal, just more advanced. Comparison of numbers is type insensitive. Comparison of chars and strings is case insensitive. Lists, hashes, arrays and structures are equalp if their members are equalp. For anything else, eq is used.

Evaluation took:
  74.060 seconds of real time
  74.232869 seconds of total run time (73.519834 user, 0.713035 system)
  [ Run times consist of 2.961 seconds GC time, and 71.272 seconds non-GC time. ]
  100.23% CPU
  184,855,087,325 processor cycles
  16,089,928,160 bytes consed

Would you look at that. Worse performance as a result of going from a more general equality predicate to a more specific equality predicate. I'm guessing SBCL does some fancy optimization tricks here.

From Edmund Weitz on string-equal/equalp

I would assume that on most implementations STRING-EQUAL is a bit faster (given the right optimization declarations) because it "knows" that its arguments are strings. It's most likely a micro-optimization that's only noticable in tight loops.

It can also be self-documenting to use STRING-EQUAL because the reader of your code then knows that you expect both of its arguments to be strings.

Therefore switching EQUALP to STRING-EQUAL in FILTER-LIST-BY-BOOK gives me the following speedup.

Evaluation took:
  69.390 seconds of real time
  69.608650 seconds of total run time (68.808030 user, 0.800620 system)
  [ Run times consist of 2.482 seconds GC time, and 67.127 seconds non-GC time. ]
  100.32% CPU
  173,202,179,865 processor cycles
  16,089,928,912 bytes consed

Going back a few steps and using = instead of eql doesn't result in anything significant at all.

I decided to replace the functional approach of REMOVE-IF-NOT with the LOOP DSL. loops - Iterate through a list and check each element with your own condition… This surprisingly did nothing.

Instead of going through the same list three times and collecting one item at a time, I decided to remove the nested loops and go through the same list once but collect three items. This was more due to my unfamiliarity with loop keywords. Good speedup results.

Evaluation took:
  14.960 seconds of real time
  15.014469 seconds of total run time (14.824816 user, 0.189653 system)
  [ Run times consist of 0.963 seconds GC time, and 14.052 seconds non-GC time. ]
  100.36% CPU
  37,334,905,433 processor cycles
  6,407,417,456 bytes consed

Removing some list copying in the filter and get-bible-uid functions yielded:

Evaluation took:
  13.300 seconds of real time
  13.353905 seconds of total run time (13.183019 user, 0.170886 system)
  [ Run times consist of 0.731 seconds GC time, and 12.623 seconds non-GC time. ]
  100.41% CPU
  33,191,116,692 processor cycles
  6,406,945,232 bytes consed

Proving once again that I am careless and forgetful, the following change resulted in a 133x speedup. I believe this function was coded when bknr.datastore:store-objects-with-class was the only external function I was familiar with in bknr.datastore and I was ignorant of bknr.datastore:store-object-with-id. Prior to the change, every single time get-haydock-text was called, it would iterate through all 35817 verses of the bible to find one instance of haydock-text. Another concrete example to read the manual of whatever library I am using.

(defun get-haydock-text (bible-uid)
  "Returns a string if bible-uid is valid else return NIL.
The bible-uid can be found by calling `get-bible-uid' with valid arguments."
  (let ((cpylist (remove-if-not
                  (lambda (x)
                    (and x
                         (equalp
                          bible-uid
                          (slot-value x 'bknr.datastore::id))))
                  (copy-list
                   (bknr.datastore:store-objects-with-class
                    'bible)))))
    (if (slot-boundp (car cpylist) 'haydock-text)
        (slot-value (car cpylist) 'haydock-text)
        (format t "GET-HAYDOCK-TEXT called with invalid bible-uid ~a" bible-uid))))

(defun get-haydock-text (bible-uid)
  "Returns a string if bible-uid is valid else return NIL.
The bible-uid can be found by calling `get-bible-uid' with valid arguments."
    (if (slot-boundp (bknr.datastore:store-object-with-id bible-uid) 'haydock-text)
        (slot-value (bknr.datastore:store-object-with-id bible-uid) 'haydock-text)
        (format t "GET-HAYDOCK-TEXT called with invalid bible-uid ~a" bible-uid)))

Evaluation took:
  0.100 seconds of real time
  0.107482 seconds of total run time (0.097531 user, 0.009951 system)
  [ Run times consist of 0.007 seconds GC time, and 0.101 seconds non-GC time. ]
  107.00% CPU
  267,219,700 processor cycles
  40,850,160 bytes consed

I admit this was less of an exercise in speeding up Common Lisp and more of a demonstration of human frailty.

Looking through the git log for commits of type "Perf" shows further optimization commits I have done. Current result for version 1.2.1,

Evaluation took:
  0.010 seconds of real time
  0.011488 seconds of total run time (0.011127 user, 0.000361 system)
  110.00% CPU
  28,639,406 processor cycles
  10,191,536 bytes consed

After the addition of regex generated cross-references in version 1.2.2,

Evaluation took:
  0.840 seconds of real time
  0.833305 seconds of total run time (0.813444 user, 0.019861 system)
  [ Run times consist of 0.002 seconds GC time, and 0.832 seconds non-GC time. ]
  99.17% CPU
  2,078,095,375 processor cycles
  80,577,968 bytes consed

Reference manuals are technical descriptions of Project Isidore's internal artifact architecture and how to operate it. For end users, please see the User Manual.

The Project Isidore Reference Manual is complete with cross-references to ASDF component dependencies, parents and children, classes' direct methods, super and subclasses, slot readers and writers, setf expanders access and update functions etc. The reference manual also includes exhaustive and multiple-entry indexes for every documented item.

• System components (modules and files)
• Packages
• Exported and internal definitions of
  • Constants
  • Special variables
  • Symbol macros
  • Macros
  • Setf expanders
  • Compiler macros
  • Functions
  • Generic functions
  • Generic methods,
  • Method combinations,
  • Conditions
  • Structures
  • Classes
  • Types

With all that being said, when the boundary between user and developer is crossed, it makes much more sense to clone the source code and explore it in your LISP IDE. Auto generated manuals may be slightly more useful in LISP than other languages, thanks to the excellent introspection capabilities of SBCL and ASDF, but still are largely only useful for index generation.

For previous project iterations and experience, see project-isidore-java repository on GitHub using Java Spring, and project-isidore-javascript currently on GitHub using NextJS. See also MEAN stack notes.

Credit must be given where credit is due. This website would not be possible without,

• Wonderful family and friends
• (Spac)Emacs Community
• Common Lisp Community
• GNU/Linux Debian Community
• GitLab, Porkbun, Oracle Cloud Infrastructure

From the bottom of my heart, thank you!