Web Site Build Process

Tom Van Vleck

These are my notes on the site building machinery, including usage, internal details, and possible future improvements.

Rules for maintenance and content of multicians.org are in Web Site Design.
Detailed instructions for maintenance tasks for multicians.org are listed in Maintenance tasks for the Multicians Website.

1. Build Process

1.1 General Principles

The build process attempts to

• Separate content from presentation.
• Store each fact once, to simplify updating and avoid inconsistency. Thus each HTML page is generated from multiple source files.
• Avoid HTML syntax errors and missing files.
• Produce standards-compliant static HTML in order to work efficiently with a wide range of visitor devices.

1.2 ISP Host Copy and Editor's copy

The domain multicians.org maps to a virtual web host operated by pair.com that serves the Multicians website. Web hosts run a web server process (Apache) that responds to visitor HTML requests with the content of the requested site resources.

The Multicians site has been mirrored at additional other ISP hosts in the past, but as of 2020 there are no mirrors. (The mirroring at stratus.com was done using an FTP server, unable to support server-side execution, and this constrained the site implementation. This turned out to be a good limitation.)

The site editor's computer has a complete copy of the multicians.org website content. To make a change, the editor prepares updated or added content on the local computer, checks it locally, and then publishes modified pages to the multicians account on the ISP using rsync. When the site was mirrored, updates were made available to the mirrors every few months.

1.3 Source and Object

Rather than editing HTML language files, the site editor works on local source files that are translated to HTML. Source files are written in the "HTMX" (HTML eXtended) language. .htmx files are translated to .html files by a program called expandfile. Expandfile has optional features designed for multicians.org: see Expandfile Multics Arguments.

Source and object are kept in separate directories on the editor's computer. (A copy of the source directory is also stored at the ISP, for backup, not available on the Web.)

A few HTML files, supplied by others, do not have HTMX source. PDF and graphic files do not have a corresponding source. Copies of these files are kept on the site editor's local computer and sync'd to the web host computer if they change.

1.4 Steps in updating an HTML file

The basic steps in generating an update to multicians.org are:

1. Edit one or more local source files, and possibly add graphics or other files to the object directories.
2. Execute the Unix make command to rebuild local object files by expanding SQL and HTMX files to HTML with expandfile.
3. View and check the local site, including any object files generated by make and links to and from them.
4. Iterate 1-3 until satisfied.
5. Execute the make install command to push the local object image to the web host directory.

This process depends on software and configuration files installed on a site editor's local computer.

1.5 Use of Make Utility

The website's files on the web host computer(s) are kept up to date by executing the make command in the source directory on the editor's computer. (The make command is available on Mac, Linux, Unix, and Windows systems. The original make command was created for Unix by Multician Stuart Feldman.)

The make command is used for two major tasks:

• Create updated local .html files from local .htmx and .sql source if they have changed.
• Publish all new or modified local files to the web host computer(s) using rsync.

Why do it this way? This setup

• .. remembers all the details of what goes where and how to publish files. The editor just types one command.
• .. keeps all the site formatting boilerplate in separate files, so all pages have a consistent format.
• .. warns if any files have HTML errors before they are published.

Editors don't need to know how the machinery works to use it for routine changes.

There are many features of make that I haven't learned yet. The Makefile could be better. Suggestions are welcome.

1.6 Use of Rsync utility

Why rsync?

• .. files already up to date on the host are not sent unnecessarily.
• .. every modified file is copied, so no part of a change is forgotten.
• .. if a file vanishes from the Unix side for whatever reason, rsync puts it back.
• .. files are sent protected by a crypto key pair.
• .. rsync sends deltas rather than whole files.
• .. files are compressed before sending, which makes sending them at least twice as fast.

2. Installing and Configuring the Build Machinery

A number of files have be be in the right places on the editor's computer for the build process to work.

2.1 Object files

2.1.1 HTML Object files

On the editor's computer, there is a complete local image of all files needed to display the web site, such that the web hosting computer can be updated using the Unix rsync command. These files are kept in the object directory. These files include .html files, .pdf files, JavaScript files, CSS files, and graphics.

After making changes and recompiling the site, the site editor uses a local web server to check the appearnce of the local copy of the website with a few web browsers, before updating it to the web host.

A "smoke test" listing a few breakable features is defined in Maintenance tasks for the Multicians Website. New pages should be checked with https://validator.w3.org/. The build script runs HTML Tidy on each file in silent mode, and will complain about mechanical errors.

A few files are excluded from the rsync command that updates the ISP object directory from the local copy: for instance, configuration files that define local directory structure, and artifacts of the Mac file system.

2.1.2 Subdirectories

The object directory has subdirectories for groups of files and graphics, such as the /mtbs directory and /reunion-04. These subdirectories contain their own index.html. Subdirectories symlink to CSS and graphic files and the /js directory in the parent directory.

2.1.3 Graphics

Graphics files are stored in the subdirectory /mulimg, with thumbnails in /thumbnails150. Many graphics have a standard version and a -2x version for High DPI displays. Graphics files are listed in multhumbs.sql so that the graphic indexes multics-images.html and people-pictures.html can be generated.

2.1.4 PDF Files

PDF files are stored in the object hierarchy but do not have a corresponding source. If PDFs are generated from other software, the site editor keeps the source on the local computer and does the generation manually.

PDF files scanned from original documents should have OCR text so that web indexers can find them. I have used Adobe Acrobat Pro, which works OK on many documents. Old documents with colored backgrounds, poorly printed text, strange fonts, multi-column documents, and figures with text labels often lead to nonsense text. Retyping a whole document and re-drawing the figures may be preferable.

2.2 Source files

On the editor's computer, there is a source directory containing source files that are used to generate the object files. For most html files in the object directory, there is one .htmx file in the source directory in the object directory. (Sometimes it's more complicated. See below.)

2.2.1 Subdirectories

Wherever there is a subdirectory in the object hierarchy, there is a corresponding subdirectory in the source tree. These subdirectories contain their own index.htmx and have sub-Makefiles pointed to by the main Makefile. In such subdirectories, there are symlinks to .htmi files and graphic and thumbnail directories in the main directory.

There is a symlink to menubar.htmi in the main source directory. This file, generated by gen-menubar.htmt, prefixes each filename with %[hrprefix]% which is set by configx.htmi to be ../. Every invocation of expandfile in the Makefile expands configx.htmi before expanding the source file. This ensures that all links in the menus point correctly to files in the parent directory.

2.3 Expandfile

The Makefile uses the expandfile utility program to create HTML object files from HTMX source. expandfile and the software it depends on must be installed on the editor's computer. This program is written in Perl and is open source provided by the editor. See expandfile Installation Instructions.

2.3.1 Expandfile wrappers

Each HTMX file for a web page defines its content inside several BLOCKs. One is named body and lays out the page content. The HTMX file ends with an *include of a wrapper file that defines the template look and feel of multicians.org, and expands the body block. There are several different wrappers, depending on the type of page. Most wrappers produce a header, footer, and drop-down and alternate menus. Most page wrappers contain HTML that adapts the page to mobile devices by setting a "viewport" and providing alternate CSS rules by testing screen size.

Wrapper file	Notes
pagewrapper.htmi	Standard page wrapper
.. class2head.htmi	Lays out the DIV for the headbar including date and title, inserts menubar.htmi
headpagewrapper.htmi	Special wrapper for index.html
codewrapper.htmi	Wrapper for a program listing, no viewport set
glospagewrapper.htmi	Wrapper for glossary pages
mspmwrapper.htmi	Wrapper for an MSPM page
mtbwrapper.htmi	Wrapper for an MTB, no viewport set
noncenteredpagewrapper.htmi	Wrapper for pages with long lines
paperwrapper.htmi	Wrapper for conference papers, no viewport set
widepagewrapper.htmi	Wrapper for site-timeline.htmx, cannot be reflowed, no viewport set

2.3.1.1 Auxiliary include files

Page wrappers and source files include additional files. Here are the most popular.

File	Notes
class2head.htmi	Lays out the DIV for the headbar including date and title, inserts menubar.htmi and altmenubar
menubar.htmi	Generated by gen-menubar.htmt from menubar.sql. Defines DIVs redmenubar and altmenubar.
mxstdfmt.htmi	included by pagewrapper.htmi. Standard page layout, CSS that hides/shows Drop-down and and mobile menus, loads jquery,includes ddm.htmi
ddm.htmi	JavaScript function for dropdown menus (uses jQuery).

2.3.2 Expandfile Macros

Expandfile provides useful HTML-generating macros. The Multics source file mxlib.htmi contains Multics-specific macros such as bibliolink and includes the library htmxlib.htmi, which provides macros for generating image references, such as getimgdiv. Generating multicians.org uses macro calls to generate IMG tags for photos. HTMX source files begin by including mxlib.htmi.

2.3.3 Tools invoked by Expandfile Scripts

Some expandfile templates invoke the *shell builtin function to execute helper tools that return character strings to be used in the expansion. These tools are either standard Unix utilities such as date, or tools provided by the site editor (many written in Perl). Tools specific to the website are stored in the subdirectory /tools of the source directory. The most important tool in this directory is expandfile and its components. (The /tools directory also contains a few other files for completeness, such as copies of .js files used in the object directory.)

2.4 SQL database

Loops Over Data Tables

About 45 expandfile template files use the *sqlloop builtin function one or more times to access my local MySQL database. The database tables are loaded from text .sql files which DROP, CREATE, and reload the tables each time the .sql file changes.

The Expandfile *sqlloop builtin expands templates for each selected row returned by an SQL query. Some MySQL tables are used to generate several HTMX templates: for example sites.html and site-timeline.html are created from loadsites.sql.

Formatting Lookup Macros

Special syntax in the source is expanded by expandfile into data retrieved from SQL queries. For example {{term anchor text}} expands to a hyperlink to the glossary entry for "term," with a TITLE attribute that previews the link target. Parameter values set in config2.htmi control which SQL tables are accessed for these definitions.

Database Connection Parameters

The *sqlloop builtin function and the special syntax references need the variables _xf_hostname, _xf_username, _xf_password, and _xf_database to be set to appropriate values for the local SQL database. Assignments to these values are kept in the local configuration file config2.htmi and most file expansions in the Makefile are performed by executing the little shell script expandfile_config, which invokes expandfile config2.htmi (rest of args)

The Makefile and shell scripts that execute mysql commands require that the file $HOME/.my.cnf is also set up with values for host, user, password, and database.

2.4.1 SQL tables

Text file	Tables	Usage	Notes	Rows	Ref
g1.sql	glossary	One row per paragraph in the Glossary	printgloss3.pl writes 27 HTMX files	1190	{{ }}
home-slider.sql	homeslider	One row per photo on multics.html	Page build generates CSS sprites.	15
loadbib.sql	bibtypes, biblio, authors, aka	One row per document, plus author info	expand 7 templates to produce 7 HTMX files	5058
loadc.sql	changes	One row per change	Creates topchanges.htmi included by multics.htmx, changes.htmx, changes-old.htmx	2245
loadext.sql	extref	One row per external page	Used in many pages.	821	{! !}
loadm.sql	m	List of Multicians	produces multicians.htmx	2055	{[ ]}
loadpages.sql	pages	Pages of the website	sitemap.htmx, Google sitemap	456	{@ @}
loadsites.sql	s	List of sites	extra rows for sites with complex history; used by sites.htmx and site-timeline.htmx	102
chrono.sql	chdate, chevent, chflag	Chronological list of events.		310
chrono2.sql	ch2year, ch2thumb	Pictures in thumbnails150/ to decorate chrono.html.		35
menubar.sql	menubar	Layout of jQuery dropdown menus and altmenubar	two levels	80
msource.sql	msource	One row per source file	source-index.htmx	5877
multhumbs.sql	multhumbs	One row per thumbnail	produces multics-images.htmx	662
mxawards.sql	mxawards	Awards earned by Multicians	used with loadm.sql	253
publist.sql	publist	Documents produced by Multicians	produced by addwp.htmt, used with loadm.sql	724
hiddenmail.sql	mmail	Email forwarding addresses (see 3.7).		750

2.4.2 CSV files

Text file	Usage	Rows
grpsio.csv	Members of groups.io/multicians	173

This list is created manually for now, by a group admin going to the "Admin" section of the mailing list page and clicking "Members" and "download". The file downloads as multicians\@groups.io.csv and is manually renamed to grpsio.csv. Each row of the file has an email address, user number, user display name, username, and then a list of flags.

2.5 Executable files

The editor's computer has a standard OS installation with programming tools and a command shell and standard Unix tools installed. The Unix or Mac operating system can be used. Tools such as Perl and MySQL are installed by a package manager in /usr/local/bin (I use Homebrew on the Mac). My tool expandfile comes with some helper executables written in Perl and is installed in $HOME/bin. Additional tools are in the /tools subdirectory of the source directory.

Windows users can use a Unix-like environment such as Cygwin. For Windows 10, there is "Windows Subsystem for Linux". I have never used it; supposedly it provides a Unix kernel and Bash-like shell, and supports Perl and MySQL.

The editor's account needs a bin directory below the home directory, and a search path set up to find commands in these directories.

expandfile Install instructions describes the setup of Perl, CPAN, MySQL and expandfile.

2.6 Environment

The editor's computer has the environment variable PERL5LIB set to point to the correct version of Perl, and has a PATH that includes a /bin directory with site building tools. .ssh keys are set up so that files can be sent from the editor's computer to the web host: they are automatically loaded using ssh-add when the editor logs in.

2.7 Symbolic links from source directory to object directory

The source directory needs a few links to files in the object directory. In particular, in order for the image reference macros such as getimgdiv to work correctly, they need to access the pixel dimensions of target images (by invoking the helper program (:gifsize2:) with the *shell builtin). Symbolic links to the object subdirectories /mulimg and /thumbnails150 are placed in the source directory.

3. How the Build Process Works

This section describes the inner workings of the build machinery.

3.1 Make

The Unix make utility generates the object directory from the source directory. It is driven by a file in the source directory called Makefile.

The Multics website's Makefile has recipes that expand any .htmx file that is older than the corresponding .html. In addition, each SQL table is loaded from a .sql text file, and the Makefile ensures that if the .sql file changes, it is loaded into SQL and then relevant .htmx files are generated. make takes care of ordering the generation steps.

3.1.1 Makefile Configuration

The Makefile refers to some external-world facts by symbolic configuration names. The actual values of these symbols is set in the file configmake.mk which is included by the Makefile. (This makes it possible to test new versions of the site or Makefile, and may someday support multiple editors.)

# Configuration constants for Makefile for Multics website
# place this in the multicians source directory
# this is where site-dependent values like path names are kept
PRIMARYWEBACCT = account@server
PRIMARYWEBPATH = directory path on server
PRIMARYWEBPATHSRC = directory path on server

3.1.2 Makefile Rule for Expanding Files

If you want to know how the build works, examine the Makefile. Makefiles define values for some variables, and then provide a set of rules. The Makefile for multicians.org rarely changes. Here is an important rule that invokes expandfile:

# Rule that makes the HTML files from HTMX files and includes. $(OBJ_DIR)/%.html: %.htmx $(CONFIG) $(INCLUDES) @chmod +w $@ $(EXPAND) $< > $@ @checknonempty $@ @-/usr/local/bin/tidy --markup no --quiet yes --show-warnings no $@ @chmod 444 $@

This rule means "Use expandfile to build the object file if the corresponding source, or any of the include files, or the config file, have changed."

Makefile rules have three parts: the "target" file(s) being made, a colon, a list of the "dependencies" -- files that the target depends on, and then an indented list of commands to execute to create the target from the dependencies.

The rule above says "do the commands if ANY of the dependency files is newer than the target."

(Note that if there is an .htmx file that does not have a corresponding .html file, make will do nothing. So when adding a new file to the site, the first step is to create an empty .html file that will be older than the new .htmx.)

make examines all the rules in Makefile and executes the rules in the "right" order, so that dependencies are built before the files that depend on them.

The percent sign in the target is a wildcard that matches any name. (Used in a dependency, on the other side of the colon, it has the same value.) This example will cause fred.html to be remade if fred.htmx is newer.

In the Makefile rule for making HTML files:

• Lines beginning with # are comments.
• The make variable OBJ_DIR is a pathname prefix for files in the object directory. Its value is usually ../multics.
• The variable CONFIG is the name of the configuration file, config2.htmi. It is included to cause all object files to be recompiled if the configuration is edited.
• The variable INCLUDES is a list of wrappers and common files. Changes to any of these files cause all object files to be recompiled.
• The variable EXPAND is the command to invoke expandfile.
• The first line of a rule has a target to be built, a colon, and the prerequisite files.
• Subsequent lines of a rule begin with TAB and are commands that are executed to build the target.
• The expression $@ refers to the pathname of the target, e.g. ../multics/fred.html.
• The expression $< refers to the pathname of the source file, e.g. fred.htmx.
• Beginning a command with @ does not echo the command when make runs.
• Beginning a command with - does not abort make if the command gets an error.
• The chmod commands make the target writeable only while it is being regenerated, as a reminder not to edit the HTML file.
• The checknonempty command prints a message and terminates with an error code to stop the build if an expansion generates a zero length target HTML file. (This can happen if expandfile encounters an error such as a missing *include file.)
• The tidy command prints messages if there are HTML errors in the target. (Install it from https://w3c.github.com/tidy-html5/.) The build is not stopped. Usually this error occurs because there is an HTML syntax error in the source file. I re-run tidy without the --quiet yes to see what errors are in the file, looking at the new .html file if necessary, fix the problems, and rebuild.
• Object files built by the Makefile are set to mode 444: that is, read-only. The one file being operated on is temporarily given write permission just before building. rsync will preserve the 444 mode when uploading the file, and this helps ensure that if an attacker could run some code on the web server, the object files will not be writeable.

3.1.3 Makefile Variables

Variable	Contents	Meaning
CONFIG	config2.htmi	Constants for multicians.org used by expandfile, including the MySQL database connection parameters.
OBJ_DIR	../multics	relative pathname to the object dir from the source dir. config2.htmi has a var named obj_dir that must match this value.
INCLUDES	menubar.touch class2head.htmi linktags.htmi textnav.htmi addrtag.htmi leftpanelstyle.htmi mspmstyle.htmi mxstdfmt.htmi pagewrapper.htmi headpagewrapper.htmi glospagewrapper.htmi noncenteredpagewrapper.htmi paperwrapper.htmi codewrapper.htmi htmxlib.htmi mxlib.htmi	a list of files that all expansions depend on
DATABASE	thvv_userlist	local MySQL database on the editor's computer
HOMEREL	../	prefix used by subdirectories to find the main directory
BINDIR	$(HOME)/bin	library directory on the editor's computer
EXPAND	$(BINDIR)/expandfile_config	Shell script that invokes expandfile with first argument config2.htmi
SUBTGTS	r2012d r2004d rvand mtbsd mabsd debugd blwd pgd	list of sub-make-targets that should be built as part of 'all'
LOGACTION	$(BINDIR)/wtlog.sh	command executed to log updates

Some variables must be different on different build environments. These variables are set in the included configuration file configmake.mk. For example, OBJ_DIR might be set to ../multics_object, and EXPAND is set to $HOME/bin/expandfile.

Shell Variable	Meaning
PRIMARYWEBACCT	username@DNS name of ISP web hosting computer
PRIMARYWEBPATH	pathname of web directory on web host
PRIMARYWEBPATHSRC	pathname of source directory on web host

(It may occur to you that the Makefile rule rebuilds some files that do not need rebuilding. For example, if any wrapper include file in the INCLUDES list changes, every file will be re-expanded, not just the ones that include the changed file. We could fix this, at the cost of increased complexity in the Makefile. Since it takes less than 2 minutes to re-expand all files, the savings are not worth the development and future maintenance effort.)

Besides rules for creating HTML files, there are also rules that specify additional dependencies for some files, in order to cause some files to be built after other files.

3.1.4 Other Unix Tools invoked by Makefile

As mentioned above, when .html files are generated, the Makefile executes the Unix HTML Tidy command to check if the files have HTML syntax errors.

The "install" recipe in the Makefile uses the rsync command to copy modified files from the object directory to the web host computer(s). The recipe causes a "make all" before executing, so to make a one-line change, the editor just edits the file and types "make install". (Check the version of rsync on the editor's computer. MacOS shipped an old version, 2.6.9. I needed to install version 3.2.7 from Homebrew, Pair has 3.2.5.)

3.2 Generation of HTMX from SQL

Certain HTMX files are generated by make from the SQL database if their .sql text file source changes. (These HTMX files are marked read-only so that an editor doesn't forget and edit them directly.) The Makefile notices that the .sql file has changed by noticing that the date modified for the .sql file is newer than the corresponding zero-length .touch file, loads the .sql file into the database, expands a template file for the modified table which generates one or more .htmx files, and executes touch filename.touch so the reload will be done only once. Updating the date modified on filename.touch also causes make rules that depend on this file to be executed.

Additional dependency rules for compiling .htmx files into .html are also present in the Makefile. The .touch file is listed as a dependency in a Makefile rule to cause HTML files to be rebuilt if they may depend on the table contents. There are about 10 such rules in Makefile. For example, the Makefile rule

  # .. rebuild about-multics for count of pages, graphics, or changes
  $(OBJ_DIR)/about-multics.html:  changes.touch pages.touch multhumbs.touch

declares that about-multics.htmx should be re-expanded into about-multics.html if any of the "changes", "pages", or "multhumbs" database tables change, in order to ensure that the counts of changes, pages, or images are re-computed. This rule adds reasons to regenerate its target to those declared in the standard rule for .html files, and ensures that about-multics.html is regenerated after its prerequisites are generated. (If we discover that we have to make twice to cause an output file to be correct, this is a sign that a dependency rule is missing.)

3.2.1 HTMX files generated from SQL

Filename	Table	SQL file	Template
sites.htmx	sites	loadsites.sql	glostpt.htmt
site-timeline.htmx	sites	loadsites.sql	glostpt.htmt
mgloss.htmx	mgloss	g1.sql	glostpt.htmt
mga.htmx	mgloss	g1.sql	glostpt.htmt
mgb.htmx	mgloss	g1.sql	glostpt.htmt
mgc.htmx	mgloss	g1.sql	glostpt.htmt
mgd.htmx	mgloss	g1.sql	glostpt.htmt
mge.htmx	mgloss	g1.sql	glostpt.htmt
mgf.htmx	mgloss	g1.sql	glostpt.htmt
mgg.htmx	mgloss	g1.sql	glostpt.htmt
mgh.htmx	mgloss	g1.sql	glostpt.htmt
mgi.htmx	mgloss	g1.sql	glostpt.htmt
mgj.htmx	mgloss	g1.sql	glostpt.htmt
mgk.htmx	mgloss	g1.sql	glostpt.htmt
mgl.htmx	mgloss	g1.sql	glostpt.htmt
mgm.htmx	mgloss	g1.sql	glostpt.htmt
mgn.htmx	mgloss	g1.sql	glostpt.htmt
mgo.htmx	mgloss	g1.sql	glostpt.htmt
mgp.htmx	mgloss	g1.sql	glostpt.htmt
mgq.htmx	mgloss	g1.sql	glostpt.htmt
mgr.htmx	mgloss	g1.sql	glostpt.htmt
mgs.htmx	mgloss	g1.sql	glostpt.htmt
mgt.htmx	mgloss	g1.sql	glostpt.htmt
mgu.htmx	mgloss	g1.sql	glostpt.htmt
mgv.htmx	mgloss	g1.sql	glostpt.htmt
mgw.htmx	mgloss	g1.sql	glostpt.htmt
mgx.htmx	mgloss	g1.sql	glostpt.htmt
mgy.htmx	mgloss	g1.sql	glostpt.htmt
mgz.htmx	mgloss	g1.sql	glostpt.htmt
biblio.htmx	biblio,bibtypes,authors,aka	loadbib.sql	glostpt.htmt
mspmtoc.htmx	biblio	loadbib.sql	mspmtoc.htmt
multicians.htmx	m,mxawards,awards,publist	loadm.sql,mxawards.sql,publist.sql	multicians.htmt
changes.htmx	changes	loadc.sql	changes.htmt
sitemap.htmx	pages	loadpages.sql	sitemap.htmt

3.2.2 HTMX files that fetch values from SQL

Filename	Table	SQL file	Values
multics.htmx	homeslider	home-slider.sql	sliding picture file
about-multics.html	changes	loadc.sql	count of changes
articles.html	pages	loadpages.sql	count of articles
changes-old.html	changes	loadc.sql	counts of changes by year
mac50-videos.html	multhumbs	multhumbs.sql	list of videos
multics-images.html	multhumbs	multhumbs.sql	thumbnail image paths
multics-stories.html	pages	loadpages.sql	tables of stories
people-pictures.html	multhumbs	multhumbs.sql	image paths
site-timeline.html	s	loadsites.sql	site names and dates, data for pie and bar charts
sites.html	s	loadsites.sql	site information
source-index.html	msource	msource.sql	source file names and info
chrono.htmx	chrono	chrono.sql, chrono2.sql	dates and events, thumbnails

(mult-links.html also includes a count of mailing list members in grpsio.csv described in section 2.4.2, obtained by using the *shell builtin in mult-links.htmx to count the lines in the CSV file.)

The following Multics specific expressions appear in many HTMX files and are replaced by the results of an SQL query.

Expression	Table	SQL file
{! ... !}	extref	loadext.sql
{@ ... @}	pages	loadpages.sql
{[ ... }}	multicians	loadm.sql
{{ ... }}	mgloss	g1.sql
bibliolink macro calls	biblio	loadbib.sql
mtblink macro calls	biblio	loadbib.sql

3.3 Makefile Rule for Sending files to the Web Host Computer(s)

The Makefile invokes the rsync command to publish files from the personal computer to the web host computer(s). rsync doesn't copy files that it doesn't need to. Files are sent securely so they can't be intercepted or tampered with, and the file transmission is compressed so it will go quickly.

The rsync command is issued by make instead of typed directly. The Makefile gets the rsync local and remote path names from Makefile variables. rsync uses SSH keys in $HOME/.ssh to be able to send files to the web host computer(s).

Specific Makefile rules can be executed by specifying their name as an argument to make. The command make install invokes rsync:

# make install pushes modified files to the primary website and also backs up the source install: all incrementversion vnfile expandfile dashboard.csvt > $(OBJ_DIR)/dashboard.csv $(LOGACTION) updated multicians rsync -avzu --blocking-io -e ssh --exclude .DS_Store $(OBJ_DIR)/ $(WEBACCT):$(WEBPATH) rsync -avzu --blocking-io -e ssh --exclude .DS_Store --exclude $(CONFIG) . $(WEBACCT):$(WEBPATHSRC) date

This rule means "Use rsync to send any modified object files to the web host computer, and also update a copy of the source files on the web host." It also updates a file dashboard.csv which visitors can look at to see counts of files on the site and the time of last update.

See Using Unix tools with expandfile for an explanation of rsync args.

3.4 Other Makefile Rules

The Makefile has other interesting rules.

3.6.1 Version Number File

The rule that pushes files to the ISP calls the shell script incrementversion to update vnfile. The contents of the file vnfile look like this:

6491 2024-04-20 14:50

The sequential number is incremented every time a make install is done, so this is the number of installs done since some time in 2006, when I first set up this file. It is followed by the date and time of the push. The shell script incrementversion updates these values. vnfile is included into an HTML comment in multics.html when that file is built. (The sequential number is not the same as the number of file changes, since a single install may install many files, or I may do repeated installs to get a single change just right.)

3.4.2 Dashboard File

The rule that pushes files to the ISP executes expandfile dashboard.csvt > $(OBJ_DIR)/dashboard.csv to update dashboard.csv. The file dashboard.csv is created by expanding the template dashboard.csvt. It contains two lines, a list of column headers and a list of corresponding column values:

  changect,asof,multicians,multiciansmail,multiciansurl,multiciansdec,glossent,npages,\
   npdf,alldoc,alldoc-online,images,mspm,mspm-online,nsites
  5388,"2020-07-02 13:35",2039,749,169,90,839,466,601,4994,2669,639,838,54,84

This file repeats the version number, and then has counts of the number of various items on multicians.org, such as number of multicians. It is intended to assist in the construction of other sites that refer to our site's contents. A file with this format can be read using the Expandfile builtin *bindcsv.

3.4.3 Subdirectories

Each subdirectory has a sub-Makefile which is invoked via a "subdirectory target" in the Makefile. The $SUBTGTS variable names these targets, and this variable is listed as a dependency for the "all" target. The result of this setup is that just typing make will also invoke the sub-Makefiles and make whatever is out of date. Most subdirectories have symbolic links to standard .htmi files in the parent directory, as well as the image and thumbnail directories. Subdirectories have a configx.htmi file that may override some path definitions in config2.htmi so that subdirectory files refer correctly to other files on the site. If you are setting up a subdirectory, it is best to clone the Makefile from one that already works, and make the same symbolic links.

3.4.4 Special build instructions

Several rules in the Makefile create special files which are published to the host ISP, using Unix utility commands in addition to expandfile. For example, make_mx_tar is a shell script that crates a gzipped tarfile listing all files that should be shipped to mirror sites. Generating it automatically saves me from having to remember to edit another file whenever I add a page to the site.

File	Function
sitemap.xml.gz	Gzipped XML sitemap used by Google crawler
multicians.procmail	Multicians email forwarding file (not visible on web)
make_mx_tar	shell script that creates a tarfile of the site, for mirrors (not visible on web)

3.4.5 Error checking performed by the build process

File building rules check for errors and alert the editor of any problems whenever make rebuilds a file. For ordinary HTML file creation, the auxiliary program checknonempty is invoked to check for the case where a target file is nonexistent or empty after expandfile is invoked. (This can happen when expandfile aborts due to a missing include file.) If this happens, make exits with an error. After you fix the original error, make sure that the HTML file was not deleted.

Newly created HTML files are also checked by the HTML Tidy utility, which prints a line if there are any warnings or errors. When this happens, the editor should investigate the failing file and correct the problem.

For a few cases where SQL files are reloaded, the make recipe runs custom shell scripts to check that foreign key relationships are satisfied. In particular, the script checkbib.sh ensures that the tables loaded by loadbib.sql have authors entries for each document, that aka rows are present for each author, and that m rows for each author exist. If any errors occur, the editor should investigate loadbib.sql and correct the problem.

The rule that pushes files to the ISP executes check_missing_extref.sh, which checks output files for the sequence ###, which is produced when a {@path anchor@} sequence is executed and does not find path. This warns that we have generated a dead intra-website link. (Some of these are OK, and represent unfinished improvements.)

The rule that pushes files to the ISP also executes check_quotes.sh, which checks HTMX files for characters such as curly quotes that display poorly in browsers. (This can happen if text is copied and pasted from mail messages or Word documents.) These characters should be replaced by HTML entities.

3.5 Source file suffixes

These suffixes are my convention for indicating the use of the content in its file name. expandfile doesn't interpret files differently depending on their names. Macros defined in mxlib.htmi have some suffix dependencies. The Makefile generates .html from .htmx files.

Suffix	Meaning
htmx	HTML eXtended, generates HTML
htmt	HTMX Template, generates HTMX
htmi	HTMX Include file
html	HTML
tpt	template, generates non-HTML files such as site map
css	style sheet
csv	Comma Separated Values
csvt	template that generates CSV file
e	text file (emacs)
faq	Usenet source, posted to alt.os.multics
gif	bitmap graphic
htmq	test htmx file
jpg	JPEG graphic
js	JavaScript
mk	Makefile include
pl	Perl language
png	portable network graphic
procmail	mail host path for procmail
py	Python
sh	shell script for /bin/sh
sql	MySQL source
txt	text file

3.6 Standard Template Expansion with Wrappers

Most pages' HTMX files conform to a standard pattern. Each file contains

• Introductory comments in %[** ... ]%
• *set statements for title, description, keywords, etc
• An optional *set statement for headingdate (If this value is not supplied, the wdate from the pages table is used. If that is not supplied either, the date modified of the source file is used.)
• %[*include,=mxlib.htmi]% which contains *block definitions for macros.
• Comments including copyright date and revision history.
• Optional *sqlloop statements that do SQL queries and expand template blocks, appending the expansions to a variable which is used in the body.
• Optional %[*block,&extrastyle,^END]% followed by items for the HEAD, ending with a line with just END
• Other optional %[*block,&...,^END]% constructs containing iterator blocks expanded for each row by *sqlloop, ending with END
• %[*block,&body,^ENDBODY]% followed by body HTML, ending with a line with just ENDBODY. (The body may contain references to variables and invocations of builtin functions.)
• The last line is *%[include,=wrapper.htmi]% -- everything else just set things up: this statement actually generates output.

Expansion of normal HTMX files proceeds as follows. expandfile

1. Reads config2.htmi to set configuration constants, MySQL configuration, etc.
2. Ignores comments in the input HTMX file.
3. Sets variables and blocks. (Does not expand variable references when blocks are defined.)
4. Optionally executes *sqlloop builtins to execute SQL queries that create variables containing the expansion of an iterator template block for each query row.
5. Includes and expands the wrapper, which outputs standard HTML structure and expands variable references, including %[body]%. If an expanded variable like body contains variable references, these are expanded also. Page wrappers begin by querying the pages table:
   1. Executes SELECT wdate FROM pages WHERE sourcefile='%[mainfilename]%'
   2. Executes SELECT crumbs FROM pages WHERE sourcefile='%[mainfilename]%'
   3. Executes SELECT l2navfile FROM pages WHERE sourcefile='%[mainfilename]%'
   4. Outputs the HEAD section
      1. Expands title, description, and keywords obtained from the *set statements, and minwidth, maxwidth, and contentwidth obtained from config2.htmi
      2. Executes *include,=mxstdfmt.htmi, which includes standard CSS boilerplate for the site
      3. Expands extrastyle if it was defined in the source file. This block may include JavaScript files and define STYLE settings local to the file.
   5. Outputs the BODY section:
      1. Includes class2head.htmi, which defines the page's standard top bar (which will use the wdate and crumbs from above) and drop-down menus.
      2. Expands the body block, set by the HTMX file, and variable references contained in it. This block may include result variables set by *sqlloop.
      3. Expands the extratail if the file supplied it.
      4. Formats and outputs tail navigation, including l2navfile if it was obtained from pages.
   6. Closes the BODY and HTML.

Alternate wrappers support variations for special page formats, such as MSPM sections or Glossary pages.

3.6.1 Why do this?

Expanding items that expand other items may seem indirect and convoluted. However, this scheme spares the writer of simple pages from the details of the site implementation, reduces the chances of a simple change breaking a page, and enables global changes when site design or HTML standards change. In practice, a one-line change can be made and published in about a minute; adding a new page to the site, cross-linked to other pages, visible to Google, and findable by standard navigation, can be done in only a few more minutes.

3.7 Additional Functions of Website Host

HTML pages, whether hosted on the primary site or mirror sites, invoke CGI functions on the primary host site.

The primary host provides additional functions for the support of multicians.org:

• Inbound email filtering and forwarding for Multicians, invoked by procmail
• Web server CGI support for form-based outbound mail to Multicians and the editor
• Web server CGI support for form-based signup and registration editing requests by Multicians, emailed to the editor

Protected mail addresses are in a file called hiddenmail.sql which is not published on the web. This SQL file is loaded into a MySQL database on the primary host site. The database is not accessible from the Web; CGIs on the primary host site access it and return values to web pages, after checking that the request is valid. Mail forwarding addresses are not visible to web crawlers (unless the ISP account gets hacked).

3.8 Tools

Tools needed to maintain Multics site provided by others

These are installed in /usr/bin by the OS distribution or /usr/local/bin by package managers.

• Shell (might work with Cygwin, haven't tried it for years)
• Perl language, requires CPAN modules DBI and DBD::mysql
• Remote copy command rsync
• Remote access commands ssh (or PuTTY)
• Unix commands make, touch, chmod, diff, cp, cat, ls, echo, mv, sh, grep, sed, ed, gzip, git
• SQL (MySQL), including command line interface and Perl DBI (currently no version 4/5 features are used)
• HTML-aware text editor, such as emacs
• Photoshop, Affinity Photo, or Gimp
• ImageMagick tool convert
• JQuery 3.6.1. - https://jquery.com
• EasySlider 1.7. - Alen Grakalic - https://github.com/atierant/easySlider (2011)
• fancybox jquery plugin: see http://fancyapps.com/fancybox/3/
• Optional: HTML Tidy (HTML5 version) - https://github.com/htacg/tidy-html5
• Optional: Integrity link checker (Mac)
• Optional: free geolocation database from https://maxmind.com/ (Needs license account.)

3.8.1 Tools needed to maintain Multics site provided by editor

See expandfile web pages. These are mostly installed in the /tools subdirectory of the source directory.

• HTMX language (Perl): expandfile, expandfile.pm, readbindsql.pm, readbindxml.pm
• HTMX Site macros and templates, especially htmxlib.htmi and mxlib.htmi
• MySQL schemas
• Configuration file config2.htmi
• HTMX helper programs (Perl) called by *shell: filemodshort, dbcounts, filesizek, nargs, filedaysold, filedmodiso, checknonempty, gifsize, gifsize2
• Special page generators (Perl/SQL) generating HTMX: printgloss3.pl, printsites2.pl, gen_site_timeline.pl
• Special page generators (Perl/SQL) generating SQL: addwp.htmt
• Generate USENET postings: auto1a.sh, post.pl (and a USENET account somewhere, I use a free one at eternal-september.org)

3.8.2 The /mxs/tools directory

Tools are distributed in a subdirectory of the source directory, but they need to be linked to from your $HOME/bin to run correctly, and $HOME/bin should be on your shell PATH. These tools include:

----- wtlog.sh, calls $HOME/wtx/wtlog (Perl)
----- expandfile_config (script, calls expandfile with config file)
----- check_missing_extref.sh
----- printsites2.pl
----- gen_site_timeline.pl
----- auto1a.sh, calls post.pl

3.8.3 Web Host Programs and data files in cgi-bin directory provided by editor

• Configuration file mxconfig.htmi
• mxmail.cgi, mxmailform.cgi, ckem.cgi, thvvutil.pm, challenge.pm
• mxregister.cgi, mxregister4.cgi
• Template files for mxmail and mxregister
• special.pmrc for spam checking messages sent via the web form, using procmail

Some of these tools invoke the subroutine version of expandfile and link to expandfile.pm in $HOME/bin

3.8.4 Web Host Programs and data provided by web host and site admin

• Apache web server, including .htaccess support, NCSA combined web log, CGI Perl execution, DEFLATE, Content Security Policy, Strict Transport Security, image caching, nosniff, sameorigin, XSS protection.
• cgiwrap support (Pair written)
• procmail support for incoming mail and forwarding, invoked by qmail (Pair installed)
• Shell access via ssh (freeBSD, Pair configured)
• scp/rsync access (freeBSD, Pair configured)
• SQL (MySQL), including command line interface and Perl DBI and CGI execution. (Pair installed and configured)
• CPAN and library modules that support GeoIP. (Pair installed)

3.8.5 Web Host Runtime Functions Provided by Editor

These functions are installed in the js subdirectory of the object directory.

• Pie Chart: piecanvas.js used to display statistics in site-timeline.html
• Timeline: timelinecanvas.js used to display graphical time lines of events in history.html and b2.html

3.9 Performance

Expandfile is written in an interpretive language, so it is not blazingly fast, but it is fast enough. Re-expanding all 467 pages on the site takes about 1 minute. Pages that have a lot of *shell calls are slower: for example, a page like people-pictures.htmx that has to involve a lot of *shell calls to Perl helper program gifsize2 has to launch a shell and a Perl interpreter for each of over 100 pictures. If I built gifsize2 into expandfile this could be a lot faster. Not worth the effort at this time.

3.10 Git

The maintainer's Multics source directory contains a local Git repository that contains all the source .htmx and {.sql:} files. Doing a make install does a git status. After making and installing a set of changes, the editor should do a git commit -a with a brief message describing the changes.

The Makefile rule that pushes files to the ISP ends with a git status which reminds the editor to add or commit changes to the git repository.

--- the SQL source for mail addresses hiddenmail.sql is not included (see 3.7).
--- various temporary files, scratch files, obsolete files are named in .gitignore and don't go into the repo.
--- future: upstream repo on github, and a way for others to request changes and build their own copy.

3.11 Files and their purpose

(by THVV unless indicated)

3.11.1 Tools invoked by Makefile

auto1a.sh (shell script to post FAQ to USENET)
checknonempty (shell script halt make with error if a file is zero length)
config2.htmi (configuration constants for every page in the website)
expandfile (see Expandfile)
gen_site_timeline.pl (generate the site-timeline chart from sites.sql)
incrementversion (shell script to add one to the version number in vnfile)
nextpostingexpires (Perl program to generate expire date for USENET summary)
post.pl (post monthly update to USENET)
printgloss4.pl (generate 27 glossary htmx files from g1.sql)
printsites2.pl (generate sites.htmx from sites.sql)

3.11.2 Tools invoked by *shell in HTMX files

dbcounts (Perl: queries SQL databases for counts of various kinds of document)
get-facebook.sh (Shell script calls get-facebook-fan-count.pl)
filemodshort (Perl: comes with expandfile)
filemodyear (Perl: comes with expandfile)
filesizek (Perl: comes with expandfile)
firstletter (Perl: comes with expandfile)
gifsize2 (Perl: comes with expandfile)
nargs (Perl: comes with expandfile)

3.11.3 Files used to support ISP mail and signup on multicians.org host /cgi directory

errtpt.txt (template for error message if update fails)
mform-thanks.txt (template for successful multicians update)
mformtpt.txt (template for multicians update form)
mxaddrtag.htmi (editor email boilerplate for all HTML files)
mxclass2head.htmi (copy of class2 head for HTML pages)
mxh1style.htmi (copy of class2 head for HTML pages)
mxmailerr.txt (template for error message if mail fails)
mxolwrapper.htmi (wrapper file for all pages created by CGIs)
mxregerr.txt (template for error message if registration fails)
mxregister.cgi (Perl: sends update mail when mform.html for is submitted)
mxregister4.cgi (Perl: Ajax form checker called from mform.html)
mxstdfmt.htmi (copy of standard format for HTML pages)
mxtextnav.htmi (copy of tail text navigation for HTML pages)
reloadm (shell script invoked to update host database)
special.pmrc (Special procmail filter, for checking user outbound mail)
thvvutil.pm (utility functions in Perl)

3.12 Displaying graphics files on the site

There are five main methods the site uses to display graphics on a web page, depending on the desired appearance and visitor experience. These methods are supported by Expandfile macros. The macros generate HTML IMG tags to display the graphics: they make it easy for page authors by looking up the the image size at compilation time, and using additional versions of graphics to support High DPI displays. Using the macros makes page creation easier and generates sharp looking graphics.

Graphic images in jpg, png, and gif format are stored in the directory /mulimg, and are used at compile time and at page display time. Every thumbnail is stored in /thumbnails150 as a 150x150 thumbnail to be displayed in a 75x75 space.

There are five methods used to display graphics on a page, supported by different macros. Examine the source of existing pages to see how these macros are used. (For example, picnics.htmx uses both the second and third methods to display pictures.)

1. Display a graphic in a DIV, possibly floated to the left or right. Use the getimgdiv macro to do this. Given a graphic name such as xxx.jpg, this macro looks for both xxx.jpg and xxx-2x.jpg. If both are found, an IMG tag that uses the SRCSET attribute is generated.

2. Display a graphic in a DIV, possibly floated to the left or right, and if the graphic is clicked, display a larger version of the graphic in a popup with a Multics banner and a "close" control. In the extrastyle block of a multicians.org source page, include the file use-dhtml-lightbox.htmi. At the top of the body block of a multicians.org source page, include the file dhtml-lightbox.htmi. Use the getimgpopdiv macro to insert a graphic in a DIV block, with an optional CSS tag. Use the getimgpopdiv75 macro if the base graphic is a 150x150 thumbnail displayed in a 75x75 space. These macros look for the given file name and a -2x version of the file, and generate the appropriate IMG tags and popups.

3. Some pages display galleries of thumbnails. (cisl-wake.html, cno-pictures.html, lcs25.html, lcs35.html, people-pictures.html, picnics.html, etc.) Clicking on any thumbnail opens a popup, with arrow keys that allow the visitor to go to the next item in the gallery. In the extrastyle block of a multicians.org source page, include the file usefancybox.htmi to include the JQuery JavaScript file fancybox.js. Create a DIV container for the gallery with a unique ID tag; inside the DIV, place a UL container. Then insert a call to the getfancybox_li macro for each thumbnail, specifying the thumbnail/image name and the caption, to generate an LI element for the thumbnail. This method and the "popdiv" method can be used on the same page. The thumbnails are displayed using CSS sprites, to speed up page loading. lightbox_li is used with an older image popup display used in several pages that display image galleries. Other pages use use-dhtml-lightbox.htmi and dhtml-lightbox.htmi and generate IMG tags with the macro getimgpopdiv. (For example, Picnics uses both.)

4. On the home page multics.html, a 300x244 window with a sequence of sliding images is displayed by easySlider.js. The pictures and their attributes are specified in the homeslider table, queried when the home page is compiled. The query generates a packed graphic image with CSS sprite definitions for each individual picture. Each picture is 600x488, so that they will look sharp on a High DPI display. References to the pictures are in individual LI items in a UL element: EasySlider handles the formatting and picture changing. When the home page is compiled, a caption specified by the homeslider table is overlaid on each graphic: this caption's definition may invoke command line programs with *shell to get counts from the database, to produce legends like "84 Multics Stories."

5. The Image Gallery page multics-images.html displays all thumbnails on the site, grouped by general type. It uses CSS sprites, so that instead of loading 462 small graphics, it loads five large ones, which is much faster. Each thumbnail links to a page that contains the graphic. All thumbnail links are generated at compile time by five SQL queries of the multhumbs table.

See How I use Affinity Photo for a cookbook on processing graphics, sharpening, etc.

3.12.1 Generating 2x Graphics from Adobe Illustrator files Using Affinity Designer and Affinity Photo

Here is my recipe for generating .png graphics from .ai files. (So I don't forget.) The goal is to take a diagram in Illustrator and make 2x and 1x versions of the graphic as PNGs.

1. Read the xxx.ai file into Affinity Designer
2. In the dialog box, accept the font substitution, set bold, set the DPI to 144, leave colorspace at 'estimate'
3. Select all, export PNG, set Area to "selection with background", leave size alone
4. Save with name to mulimg/xxx-2x.png
5. Read in xxx-2x.png into Affinity Photo
6. Resize to half size, sharpen, and save as mulimg/xxx.png (remember to make it PNG, not JPG)
7. Edit the source file to invoke a macro such as getimgdiv referencing mulimg/xxx.png
8. Recompile. On a machine with a HiDPI display, browsers will load xxx-2x.png and display it in 1/2 the CSS dimensions, because the SRCSET selects the 2x file; the browser knows the screen resolution and uses all the pixels to make a sharp image. On a machine without a HiDPI display, browsers will load xxx.png and display it, because the SRCSET selects the 1x.
9. Generate a thumbnail for the image using gth2x xxx-2x.png and move /mulimg/thumb2-xxx-2x.png to /thumbnails150/xxx.png.
10. Update multhumbs.sql to use the new thumbnail.
11. Verify that everything displays correctly.
12. If there are /mulimg/xxx.gif or /thumbnails150/xxx.gif files, remove them.

4. Expandfile Documentation

• Expandfile
• Expandfile Tutorial
• Expandfile Multics Arguments
• Expandfile Wrappers
• Expandfile Macros
• Using Unix Tools with expandfile