Multics > About
20 Apr 2024

Maintenance tasks

Tom Van Vleck

This file describes maintenance tasks for multicians.org.

Design principles for multicians.org are described in Design Principles for the Multicians Website.

The site build process and tools for multicians.org are documented in Build process for the Multicians Website.

Creating Computer History Websites contains my thoughts on building sites like this.

1. Roles -- Who Does What

2. Hosting for multicians.org

Hosting for this site is provided by the hosting site admin, who pays the domain name registration fees and web hosting fees as a contribution to the online community. The web site has been hosted at Pair Networks Inc in Pittsburgh PA since 2001. Pair is referred to as the Information Service Provider (ISP).

We are limited in both storage space and bandwidth as a result of the ISP account parameters. The editor will take whatever steps are necessary to avoid exceeding the (fairly high) limits imposed by the ISP, since this could cost the editor a lot more money than he can spare.

The ISP account has a bandwidth limit. Usage over this limit incurs additional charges. The ISP has a policy that forgives high usage due to "slashdotting" outside of our control, but more than one such event in a month could be a problem.

We are also bound by the ISP's Acceptable Use Policies. So far this has not been a problem, partly because the site editor moderates the site content.

The advantages of hosting at Pair Networks are

Pair has been in business since 1996. In 2017 Pair Networks was bought by Libsyn, a podcast hosting company, from Kevin Martin, the CEO of Pair. (https://www.sec.gov/Archives/edgar/data/1667489/000165495418000043/lsyn_8k.htm) Pair's service remained good.

Pair supports

In December 2022, multicians.org was moved to a new account at Pair Networks. This is a "Virtual Private Server" account rather than a shared server account. (Saved a little money. Moved from FreeBSD to Ubuntu. Raised bandwidth limit.)

3. Tasks for Site Editor

Files on the site include graphics and PDFs, HTML provided by others, and HTML compiled from HTMX and SQL.

The open source expandfile tool translates HTMX to HTML.

Wrappers

Translation is invoked using the Unix make command for any file whose source is newer than its object.

After translation, the Makefile uses HTML Tidy to check for errors. Cure these and re-install.

Web pages on multicians.org are written in .htmx source files that contain extended HTML for the page's contents. The source file ends by invoking a wrapper that formats the page with standard layout, headings, and CSS. expandfile creates a .html object file.

Some web pages on multicians.org are generated from .sql source files that contain structured data. Template files (.htmt or .tpt) are expanded to generate .htmx files that are in turn expanded into .html. Don't edit the .htmx files -- edit the SQL and invoke make. See Build process for the Multicians Website -- Generation of HTMX from SQL for info on which SQL tables generate which HTML files.

Many web pages on multicians.org use reference constructs to generate hyperlinks, described in Multics Features.

3.1 Updating the Main Page of multicians.org

Edit multics.htmx in the source directory.

Execute make and then check the local copy of the compiled files. Fix any errors or warnings. Then execute make install to publish the new story.

make will automatically update the main page if other pages change that would change totals in the main page. See Makefile.

3.1.1 Adding a photo to the sliding gallery on the main page

Prepare the photo. Make two versions: a 600x488 pixel -2x version, and a 300x244 version.

Edit home-slider.sql in the source directory. Add a line like 

  (8.0,' class="starthidden"','sc_rdlo','articles.html','slider-6880-system-2x.jpg',
     '%[narticles]% Articles','68/80 Multics system'),

with the fields 

 ordinal FLOAT,            -- order of items if more than one selected
 liclass VARCHAR(64),      -- class for the LI
 cssclass VARCHAR(64),     -- class for the picture -- encoding caption text color and position (see SQL source)
 target VARCHAR(255),      -- link target
 filename VARCHAR(255),    -- picture file name
 oltitle VARCHAR(255),     -- overlay title, may contain HTMX var
 description VARCHAR(255), -- TITLE attribute

Execute make to regenerate the page. The source file multics.htmx generates a shell script pack-slider.sh and executes it. This script invokes ImageMagick convert to generate a packed sprite file. multics.htmx uses *sqlloop to generate a CSS class definition for each sprite that picks out the photo, and also generates an LI element for each sliding photo that references the CSS class. The LI elements contain captions from the SQL table that may refer to totals (such as number of Multicians) determined at compile time. The generated multics.html contains a UL element containing these LI elements. multics.htmx also invokes EasySlider, adapted from http://cssglobe.com/easy-slider-17-numeric-navigation-jquery-slider/, which displays one picture at a time from the UL, with a caption overlaid on it. If a visitor clicks on the picture, their browser takes them to the target page.

3.2 Adding or Updating a Multician's Entry in multicians.html

3.2.1 Visitor Interface

A site visitor may use the Multician add/update form at mform.html to send mail to the site editor.

If the visitor has JavaScript enabled, changing the person name in mform.htmx sends an XHR (Ajax) request to mxregister4.cgi at www.multicians.org to look for existing database entries in the m table and fill in the rest of the form. If multiple database entries match the partial person name, the form code presents a pick list; if the visitor picks one entry, a new XHR message is sent.

When the visitor submits the completed form, this action invokes mxregister.cgi at www.multicians.org, which sends mail to the editor with SQL rows for database input. (It doesn't update the database.) The mail looks like this: 

  loadm.sql
  ('fooch,melvint', 'Fooch, Melvin T', 'User (MIT): stuff', '', '', 'mail 2020-04-01 11:54', '', 'Y', '', null),

  hiddenmail.sql
  ('fooch,melvint', 'fooch@supertool.com', '', '2020-04-01 11:54'),

Alternatively, a Multician may send a mail message to the editor instead of using the form, or the editor may discover a Multician in some other publication, web page, or forum.

3.2.1 Editor Tasks

The editor reads email messages sent by visitors requesting additions or changes to the Multicians list. For each request, the editor verifies the form info, including whether the submitter is actually a Multics contributor. To do this, the editor may know the sender personally or know someone who does, or the editor may send mail to the submitter and ask for details. The editor can send a standard form letter that says, "Sorry, liking Multics doesn't make you a Multician."

The editor checks the URL if supplied, and may correspond with the submitter if the URL has a problem.

If a Multician's name or "did" contains apostrophes, they must be replaced by '. Loose ampersands must be replaced by &. Accented characters like the accented o in Corby's name must be replaced by character escapes such as ó.

The editor adjusts the submitted "did" field to match terminology in other entries.. e.g. OU versus Oakland University, UC versus Calgary. Formatting tags such as <cite> and <em> in "did" fields canot be used: the visitor's browser will remove them when parsing the Ajax XML message. It is OK to use HTMX constructs such as {@page title@} to link to other site pages.

The editor also checks the mail info and adds or updates the corresponding row in hiddenmail.sql in the source directory. When a new multicians.org email address is added, the site editor must also log into the web account hosting control panel and add a mail forwarding recipe. See "mail reflector maintenance" below.

3.2.3 Installing Changes

If a request is accepted, the editor edits loadm.sql and edits hiddenmail.sql if necessary, and recompiles and uploads the changed pages. The editor can also edit loadc.sql to add a notice like "added an entry for Melvin Fooch." The update operation can be done in a minute or two.

When the editor executes make after updating loadm.sql or hiddenmail.sql, the Makefile will rsync updated copies of the m, mmail, and mgemail tables to the web hosting computer and then execute the remote command reloadm on the hosting computer via SSH. The reloadm command loads the SQL database on the hosting site with new copies of the m and mmail tables. Its output looks like this: 

  total 892
  -rw-r--r--  1 thvv  users    7897 Apr 25 16:28 grpmail.sql
  -rw-r--r--  1 thvv  users   52710 Mar 31 19:44 hiddenmail.sql
  -rw-r--r--  1 thvv  users  258790 Apr 25 16:39 loadm.sql
  reloaded m on (host)
  reloaded mmail on (host)

The editor sends a welcome/thanks email to each new or updated Multician, and may suggest that the person join the Multicians mailing list, LinkedIn group, etc.

3.2.4 Other Maintenance of the Multicians List

If a Multician has won an award or medal, the editor should add or update a row in the mxawards table in mxawards.sql in the source directory. For an award type not seen before, the editor must also add a row describing the award in the awards table.

If a Multician has died, the editor updates the "comment" field of the Multicians table to indicate "deceased" and a date, and removes the email address. When the Multicians list is generated, the person's entry will be flagged as deceased and totals will be adjusted.

3.3 Adding or Updating Multics Site Information

Verify the site info and add or update the site's row in loadsites.sql in the source directory.

Update the site history if there is one. Write a new site history page if you have enough information to make it interesting.

Execute make and then check the local copy of the compiled files. Fix any errors or warnings. Then execute make install to publish the new story.,

3.4 Adding or Updating the Bibliography

Contributors send documents to the editor, or send URLs to online documents, or the editor discovers documents on the Web or scans paper documents for storage on-site.

If a document file is provided, find space for a copy, and ensure that the document is readable, searchable, and small enough to load quickly. (Ask contributors to provide non-proprietary file formats: e.g. no Word or PowerPoint; PDF with OCR text is OK.) (Providing copies on the site avoids issues of other sites taking the document down or changing its URL in the future.) Ensure that the rights of the document creator are respected, and that the content is accurate and informational.

Add or update a row in table biblio in loadbib.sql in the source directory. Give this row a document identifier like "Smith100" with an author name and unique sequence number.

For each author of the document, add a row to table "authors" giving the document identifier, the author name as it appears in the document, and the ordinal position of the author name in the citation.

For each author name, ensure that there is a row in table "aka" that links the document-style name with the canonical "name" field in loadm.sql.

Executing make will load the SQL tables and regenerate the bibliography, the MTB index, the MCB index, the MSPM TOC, the MSB index, the MCR index, the MAB index, and the publication counts in the Multicians list. The Makefile will execute the shell script checkbib.sh to check that there are no missing rows in the bibliography tables and that they match the Multicians table -- fix any errors and recompile. Then execute make install to publish the new files.

3.5 Adding or updating glossary entries

Edit or add paragraphs to g1.sql in the source directory. Use special tags in other files on the site to generate links to glossary entries and Multicians list entries, and to mark commands, pathnames, and code. One oddity: to insert a backslash in the output, you have to put EIGHT backslashes in the database file, because it gets expanded three times.

Each glossary entry is created by one or more rows in g1.sql. The build process will reload the database and run printgloss4.pl to regenerate 27 .htmx files comprising the index and one page per letter.

A glossary row in g1.sql looks like 

  ('useroutput', -- sortkey     (lowercase)
  'useroutput',  -- tag	        used after hash in HTML A tags
  'u',		 -- letter      letter for glossary page
  'user_output', -- term        thing being defined
  1,		 -- ord         paragraph number: must be sequential starting with 1
  'DMW',	 -- attribution who defined
  'The standard process output {{IOswitch I/O switch}} that commands normally write to.
     It is connected to the SYSOUT file in PL/I.
     The attachment of this switch can be manipulated using such commands as
     {:file_output:}, {:terminal_output:}, and {:revert_output:}.', -- definition
  'glossdef',	 -- how text is formatted
  ''),		 -- notes

Glossary entries can contain {{ ... }} constructs to crosslink to other entries, and can contain formatting macros for text, e.g. {:revert_output:}.

The 5th field in the entry allows multi-paragraph entries, for example if several different contributors supply definitions of the same term.

Execute make and then check the local copy of the glossary. Fix any errors or warnings. Then execute make install to publish the new copy.

Use the formatting macros provided by expandfile to generate links whenever possbile.

3.6 Adding a New Web Page

Pages may be added to the site to provide additional technical and historical articles, personal stories, event reports, or other information about Multics.

The editor should ensure that the context of the page is clear to a broad audience, and that each new page fits into the rest of the site. See Design Principles for the Multicians Website.

For consistency, use formatting macros like {:xxx:} to generate simple formatting.

Use the macros ( invoked by *callv) in mxlib.htmi that generate external links to the Multics source at MIT (mitsourcearc, mitsourcefile, and mitinfoseg) and to Multics-related files at bitsavers.org (bitsaversmultics). These macros do not reference SQL; they use constant pathnames in config2.htmi.

Also use SQL lookup macros like {{glosstag anchor}} (reference to Glossary), {[multiciantag name]} (reference to Multicians list), {@pageref anchor@} (intra-site link), and {!extref anchor!} (link to another site) to correctly generate intra-site and external links.

If the added page should be displayed on the dropdown and alternate menus, update menubar.sql. Recompiling the site will update all .html files. Find the major heading that the new page should be added below and add a row for it. We mostly display 3 levels in the dropdown and 2 in the alternate menus. We do not display the whole list of sites in the alt menubar because it would be too long.

Try to format the page so that it will work well on all screen sizes. For example, the old Sites page displayed a table of 84 site names as a kind of index, but this table was cut off on the right on a small screen. I rewrote this so that the table adjusts the number of columns to fit on the screen.

Contributed Text

If a contributor sends information in plain text, convert it to simple HTMX. Change any & characters to &amp;. change any < to &lt;, change any > to &gt;, change any { to \{\, and change any \ to \\. If the contributor provided plain text with empty lines as paragraph separators, convert those to </p><p>. Convert bulleted lists to

<ul>  <li> ... </li> </ul>

You can use HTML Tidy on your converted file to have it generate canonical HTML, with closing tags for entities.

If you're provided text in Microsoft Word, just copy the plain text to the clipboard and paste it into a simple text file.

In either case, turn it into a HTMX file by wrapping the contributor's text in a

%[*block,&body,^END]% ... user's text ... END %[*include,=pagewrapper.htmi]%

.. and setting the variables title, description, keywords, and headingdate with the *set builtin. Make it look like other .htmx files on the site. Remove Microsoft "curly" quotes and replace them by HTML entities or plain quotes. Use checkquotes.sh to make sure you have no bad characters.

Contributed HTML

If a contributor provides a story in HTML, turn it into a HTMX file by removing the HEAD section and <BODY> tags, defining a few standard variables with *set, and wrapping the body text in a

%[*block,&body,^END]% ... user's text ... END %[*include,=pagewrapper.htmi]%

Remove Microsoft "curly" quotes and replace them by HTML entities or plain quotes. Use checkquotes.sh to make sure you have no bad characters.

If a contributor insists on not having his HTML tampered with, see "Class 4" pages below.

CSS Definitions

The standard site CSS definitions are included by the page wrapper; if the story uses specialized CSS constructs, create a

%[*block,&extrastyle,^END]% <style>   user style definitions   ... </style> END

The extrastyle block can be used to insert any HTML into the HEAD section,including JavaScript references or functions. Use <p> and </p> around paragraphs and similar simple markup ... HTML Tidy should find no errors or warnings in the converted file.

Definition of Variables

At the top of the HTMX file, specify

%[*set,&title,="(page title)"]% %[*set,&description,="(page description)."]% %[*set,&keywords,="(keywords)"]% %[*set,&headingtitle,="(short page title, without 'multics')"]% %[** Copyright (c) (year), (author) **]% %[*include,=mxlib.htmi]%

Content

If the story is a long one, using H2 and H3 headings may make it easier to follow. (H1 is the page title, generated by the wrapper from the headingtitile or titile variables.) References to other pages on the site, or to external documents, may help readers understand the relation of this page to other information on the Web -- use the linkage macros.

If possible, include pictures. Getting useful pictures with sufficient pixels may require a mail discussion with the submitter. Contributor supplied pictures will often need size, color, and contrast adjustment to display well in HTML. Use macros to generate IMG tags for pictures, as described below.

Content on the site is public and can be read and re-used by any Web user. If a contributor is concerned about AI language models using content they provide, the site maintainers should attempt to segregate such information into its own pages and mark some pages as disallowed for AI crawlers (this protection can be bypassed by web crawlers that don't respect conventions about robots.txt).

Adding a Story to the Site

Create new files storyname.html in the object directory and storyname.htmx in the source directory.

Do the other "new page" operations below. Adding the page to loadpages.sql with a nonblank "site" entry and a kind of "people" triggers its inclusion in the stories index. Set the nextstory field of an existing page to point to this one, and the nextstory field of the new entry to the old value, so the "Next Story" link works. (touch the source of the existing page so its navigation links will be regenerated also.)

Publishing

Spell check the local copy of the file and correct any errors. Execute make and then check the local copy of the compiled files, especially the "Next Story" links. Fix any errors or warnings. Then execute make install to publish the new story.

3.7 Adding or updating other class 2 and class 3 pages

Edit or add filename.htmx in the source directory. Don't edit the HTML object files directly.

Search the rest of the site to see if links to the new page should be added.

Use the wrapper macros: set title, description, keywords, and headingdate, then define a block called body that contains the HTML of the content. At the bottom, include pagewrapper.htmi to cause the content to be formatted with standard headings and footers. If necessary, use optional blocks extrastyle for extra lines in the HEAD section, and extratail to add content to the footer.

Use the getimgtag, getimgdiv, getfancybox_li, or getimgpopdiv macros for references to images. For new images, make a 150x150 thumbnail and a -2x version of the file if possible, and update multhumbs.sql. If the images have picture of Multicians, add the appropriate 'gallery' field to the multhumbs row.

When a new HTML page is added,

Search the rest of the site to see if links to the new page should be added, and if so, add links referencing the pages table entry.

Spell check the local copy of the new file and correct any errors. Execute make. Fix any errors or warnings. If you made major updates, use W3 validator to check the new local copy of the file. Also use Site Grader to check your site's loading speed, security, and conformance to specs. Then execute make install to publish the new or updated story.

3.8 Adding or updating class 4 pages (formatted by others)

Either convert the file to a class 3 page, or

Place the HTML file directly in the object directory. Edit it to insert navigation to the rest of the site at the bottom, so that if a visitor finds the file from a search engine, they can find the rest of the site.

Add a row for the file to loadpages.sql and an entry in makemxtarextrafiles in the source directory. Indicate the HTML file as both source and object.

Optionally edit the HTML to add mxstyle.css for uniform heading styles if this doesn't mess up the appearance of the document; discuss this with the file author.

Change mail addresses in the HTML to HREF links via the Multicians database, so that the submitter's mail address cannot be scraped by spammers. The site maintainers are responsible for keeping users' mail addresses private.

Verify that the local copy of the file looks OK. Execute make install to publish the page.

3.9 Managing Images

Images are stored in directory /mulimg in the object directory. (This directory is linked to from the source directory so that image macros can get the size of the image.) They can be .jpg, .gif, or .png files.

3.10 Adding or Updating an Offsite Link

For an HTML link to a file not on multicians.org used in article text, edit loadext.sql to add or modify a row in the external links table. The row specifies an internal tag and the link target URL. In the page source file, generate an HTML link to the external file, using the {!tag anchor text!} syntax. The globe icon, etc will be generated by expandfile.

Multicians home page URLs are specified in the url column of a Multician's row in the m table defined in loadm.sql.

External URLs of items in the bibliography are specified in the bhref column of a row in the biblio table defined in loadbib.sql.

After editing the SQL files, execute make install to recompile source files that reference them. (In some cases it may be necessary to touch a source file to cause it to be re-expanded.)

Check every few months for dead links. I use the Mac tool Integrity, and search for better URLS for failing links if possible.

3.11 Maintaining Links in HTML files

Use the bitsaversmultics HTMX macro to generate HTML links to scanned Multics manuals at bitsavers.org,
e.g. %[*callv,bitsaversmultics,="AN61A_storageSysPLM_Sep78.pdf",="AN61, Storage System PLM"]%.

Use the mitsourcefile and mitsourcearc macros to generate links to Multics source at MIT.
e.g. %[*callv,mitsourcefile,="Multics/ldd/include/flagbox.incl.pl1",="flagbox"]%.

Generate other external links using the HTMX construct {!tag anchor text!} which looks up "tag" in the external links table and inserts an HTML link with globe icon and TITLE tag.

Generate multicians links using {[tag anchor text]} which looks up "tag" in the multicians table and inserts an HTML link with TITLE tag.

Generate glossary links using {{tag text}} which looks up "tag" in the glossary table and inserts an HTML link with TITLE tag.

Generate internal links using {@tag text@} which looks up "tag" in the pages table and inserts an HTML link with TITLE tag.

Use {:text:} to format items with the "cmd" class, {=text=} to format items with the "pathname" class, {+text+} to format items with the "code" class, all defined in mxstyle.css.

3.12 Adding or updating a PDF file

Ensure that each PDF file on the site has OCR text so that indexers can see its content. If not, try using a utility such as Acrobat to add it. (See below for use of VueScan.) Do not obsess over OCR accuracy.. some faded documents or those with unusual fonts will convert poorly. The pdffonts utility (part of Poppler) will show if a PDF file has OCR text.

Place the file in the object directory, with no corresponding source file.

Add its name to loadpages.sql. For the "source" use something like ../multics/foo.pdf -- that is, the file is both source and object. This entry ensures that it is in sitemap.html, and that the date modified displayed is that of the PDF file.

Add its name to makemxtarextrafiles in the source directory. This ensures that the PDF file will be included in the TAR of the whole site.

Execute make install to publish the page.

3.12.1 Adding OCR Text to a PDF using VueScan

These directions are for the Mac, using VueScan Pro.

  1. open VueScan
  2. change Input Options to Professional
  3. Set Task: scan to file
  4. Set Source: file
  5. Set Mode: flatbed
  6. cange to Output Options:
  7. Check PDF OCR text
  8. Back to Input Options:
  9. Set "batch scan" to "list" and "batch list" to "1-*"
  10. click the @ sign next to PDF file name, and pick the file from the file picker
  11. click scan
  12. it should create an output file yyyy-mm-dd-nnnn.pdf -- rename this file to something useful
  13. go to 10 till all files done

3.13 For All Changes

Run W3C Validator on each new class 1-3 HTML file. It checks for more errors than HTML Tidy. (Both Tidy and W3C Validator fuss about some things that we allow on the site: for example, the Facebook link code and the Google search code use non-standard HTML, and we use a few deprecated CSS constructs necessary for correct display in old browsers like MSIE.)

Update loadc.sql with a one-line description of each change, and execute make to generate changes.html describing what's new. Changing this file will regenerate the HTML site map, RSS feed, and Google sitemap.xml.gz files. (Some minor edits, typo corrections, etc. do not need a line in 'changes.' For intermal machinery changes, enter a line in loadc.sql indicating that we did make a change but that changes.html need not be updated.)

The htmx, sql and other source files are stored in a local git repository (graphics, generated files, and binary files are not included). The make install command will do a git status to remind you to do a git commit -a with a summary of changes. Currently this repository is local to the site editor's machine.

To make a tarfile of the whole site, first touch makemxtarextrafiles and then do a make install. This sends the shell script make_mx_tar to the ISP. Run that script from the ISP account's command line.

To manage the Google Custom Search box, log into https://cse.google.com/cse/all. Add and delete sites that are searched.

3.14 Run Lighthouse After Major Changes

Occasionally run Google Chrome Audit (Lighthouse) after significant changes.

4. Monitoring Implementation Features

Editors and admins must keep aware of and respond to the following issues:

5. Smoke Test by Site Editor

Before updating the hosting site, check the appearance of the local copy of the site. Check major updates in Firefox/Mac, Safari/Mac, Chrome/Mac. (I probably ought to check Chrome and Edge on Windows also, but I do not have a Windows machine.) Also check when major browser updates to Chrome, Firefox, Safari, or Edge are released. Run the W3C HTML5 Validator on a sample of pages, in case the validator changes its criteria or the HTML language changes (which happens sometimes).

Check for:

If all looks OK, execute make install.

Occasionally check that the actual online site loads correctly and looks right, to make sure that nothing is down or misconfigured at the ISP. Especially check this after making any kind of change to the web server configuration, .htaccess, or file access controls.

6. Details to Check on Hosting Site When Deploying

Hosting site admins should ensure the following.

Hosting site admins can execute shell script make_mx_tar to generate a tarfile for transmission to mirror sites when needed.

7. Web Report Maintenance

A CRON job running on the main hosting site at the ISP pair.net runs daily at 3AM. (This time is chosen so that Pair's logextractor has time to prepare the input.) The CRON job executes a script called daily which in turn runs /wtx/gen_swt2.sh.

Pair's log processing extracts Apache Combined Format web server logs from the shared server's log for domains hosted on the ISP account. The daily logs are placed in the directory /www_logs in the account's home directory.

The directory /wtx has Super Webtrax (SWT) installed. This program has been in use since 2006. It is available from github. The script /wtx/gen_swt2.sh

The output from the CRON job is automatically mailed to the site administrator.

Recovering from Problems With the Web Report

(I find that the web report job almost always works fine, unless I am traveling somewhere with no Internet access.)

If the daily web reporting job encounters problems, the mail message from the CRON job will contain error messages or be missing. The process the site administrator follows will depend on what went wrong.

If the mail message from the cron job did not arrive, is the hosting server up? Was it up when the job should have started? Perhaps the job ran OK but the host mail server was down or backlogged, or the mail was sent OK but was spam trapped before reception.

It the mail message from the cron job did arrive, see if it looks OK. It should end with a "done".

If the hosting server is down, check the ISP web page for outages, or contact support. If it was down and came back up, you can just wait a day, the job should process all available logs.

If the message from the cron job has lines like 

  Configuration loaded Fri Nov 27 05:53:39 EST 2020, webtrax version S24
  3.11 real         2.14 user         0.07 sys
  load data
  perl ./tools/logvisits3.pl -sql -config swtconfig.htmi templog
  time: command terminated abnormally
  98.05 real        97.68 user         0.20 sys
  User defined signal 1
  **** failed: perl ./tools/logvisits3.pl -sql -config swtconfig.htmi templog
  *** swt failed, error 127
  *** moving comb.20201126.gz to badlogs and bailing out
  *** recovery will depend on how far swt got

This is an example of a process in the cron job running so long that Pair's "reaper" utility killed it. Often a retry will work.

Log into the hosting server. crontab -l should show daily job and that file should exist. Check dates modified in wtx: if there are some for today, then the job did something.

Check /badlogs to see if /wtx/gen_swt2 encountered a failure and moved the processed log to /badlogs.

In /www_logs, gen_swt2 executes nice combinelogs to combine multiple webserver logs into comb.mmddyy.gz check /www_logs for comb.mmddyy.gz -- if found, then the cron job did not pick up the logs

Look at the script swt to see how far it got. It executes functions from swtfunct.sh. Check /wtx for a file templog -- if it is there, then the CRON job failed loading the database in function loadusage. Check /wtx for a file comb.* -- if it is there then the CRON job probably crashed.

Try to rerun the job.. a second run may succeed because of caching results. Check /badlogs for a file comb.mmddyy.gz. If it is found, there should be a problem message in the mail from the CRON job, saying that the job has saved the input file it was working on in /badlogs. If this happens, fix the problem and then copy the bad log file to /wtx as www.mmddyy.gz and execute ./daily.

8. Site Maintenance Tasks for Hosting Site Admin

The main hosting site is currently at the ISP pair.net. The account has a control panel that manages incoming email, domain names, billing, and other details. The domain name multicians.org is registered at pairdomains.com and the Pair account has its own password and control panel. Specific settings are documented elsewhere.

Currently our hosting account uses a "Virtual Private Server" running in a virtual machine on an all-SSD server at Pair. The VPS is running Ubuntu Linux. The SQL datbases are on a separate shared server at Pair.

multicians.org mail reflector maintenance

The procmail script multicians.procmail in the home directory of the account hosting multicians.org redirects selected mail messages.

See multicians-org.html for details.

The redirector script is automatically generated by expandfile with procmail.tpt from the multicians table by the Makefile when loadm.sql is updated.

When a new multicians.org address is added to hiddenmail.sql, it must also be added to the list of valid entries in the ISP's mail recipe control panel. This allows our site hosting to do SMTP Reject of spam mail sent to invalid addresses at multicians.org and greatly decreases the mail load on the server.

Mail Forwarding Repairs

Very occasionally, Pair screws up the mail forwarding. This happened in 2004, 2007, 2014, and 2019 (often when I was traveling). Usually this happens when some other problem, such as disk errors or a spam attack, causes web server load to spike. Inexperienced sysadmins then panic, notice multiple procmail processes running (checking incoming mail for spam), decide that procmail is the problem, and either break .procmailrc or set access on /usr/local/bin/procmail to null.

If they choose the first option, then all incoming mail including spam gets dumped into a single file with no filtering. If they choose the second option, then all incoming mail is bounced with a permanent fatal error from MAILER-DAEMON.

The site editor must then call Pair support on the phone and teach them their jobs, which takes about an hour on hold. The right solution is usually to fix any I/O problems, and then wait 10 or 15 minutes for procmail to clean out the mail queue.

Recovery from such a mess depends on how they panicked: in the first case, this means taking the huge mailbox and piping it through procmail; in the second case, the mails sent are lost forever, but Pair support can grep the qmail log to find out the senders of messages that were discarded.

Domain name management

Pairdomains

The domain name multicians.org is registered at pairdomains.com. Some domain name operations need to be done partly with the domain registrar site and partly with the Pair control panel.

Pair Account Control Panel

Pair website management supports https access and automatic renewal, using free Let's Encrypt certificates, in the Pair control panel. So far, it has worked with no problems. It is valuable to enable this feature, since Google search rankings penalize non-https sites.

The "domain management" panel of the Pair account control panel also provides controls for setting up SPF and DKIM records for hosted domins. The domain control panel sets up the server mechaisms and displays the content of the records to be added to the DNS in the domain registration site.

DNS records for mail delivery

The site's DNS is not hosted at Pair.com, it is at Pairdomains.com. You have to switch back and forth between two different accounts, with different account names and passwords, to manage the site.

multicians.org has SPF and DKIM records associated with it in its DNS registration to allow mail sent via the Pair servers to be marked as "not spoofed." (gmail.com and yahoo.com started rejecting mail messages without these headers in February 2024.)

multicians.org has a DMARC record in its DNS registration that specifies a minimal policy, because some ISPs appear to bounce mail if no policy is specified.

Search Engine Optimization

Most "search engine optimization" activity is not relevant to multicians.org (and some of it is bullshit or scam). Keyword stuffing and fake links to our site from link farms are a very bad idea. Ensure that META DESCRIPTION tags on eac page are accurate and incorporate the words that visitors might use in a query.

Because we have a fairly high search rank, we get occasional requests to put a link on our pages to some "school page." These are scams. Most of them seem to be copied from a book somewhere. Ignore them.

Google Webmaster Tools

We have a Google Webmaster Tools account, now called Google Search Console. They sometimes send warnings that our pages have problems that they detected when indexing our site. Some of these are issues that should be fixed.. others are not useful. I have been ignoring most of the "click targets too close together" because the other click targets nearby point to the same thing.

We don't use Google Analytics, because it would inject JavaScript into our pages, and because visitors may not wish to have Google tracking their web browsing. For the same reason, we removed the code that creates a Facebook "like".

9. Social Media

alt.os.multics FAQ Maintenance

Currently the site admin causes a short FAQ posting to alt.os.multics on the first of every month. At some point we will stop doing this, when USENET is no longer interesting.

In order to post to USENET you need an account at some USENET service. Currently we have a free account at etermal-september.org.

To post the short FAQ, execute make auto2 on the first of each month. This operation invokes a Perl program that updates the alt.os.multics FAQ with totals from the website.

Groups.io Multicians group administration

If applicants meet the group criteria (Multics contributor or Multics user) accept them. Otherwise send a form letter. Use the 'admin' control panel of groups.io to approve new applications. Add the visitor's subscription email to grpmail.sql (nothing uses this table yet).

LinkedIn Multicians group administration

If applicants meet the group criteria (Multics contributor, including user) accept them.

10. Backup

Local Backup

Site admin: back up local computer to multiple backup devices stored in multiple places, continuously as files change; do complete backups to multiple destinations occasionally. Back up the source and object directories that make up multicians.org, and tools and configuration data for the local computer.

Web Backup

The site admin uses rsync (invoked by make install) to copy the object directory to his account at the web host.
The site admin also uses rsync to copy the source directory to his account at the web host; these files require password access to view. Some files in the source directory contain Multician mail addresses and should not be web accessible.

It would be possible to put almost all of the HTMX source of the website (except the SQL file with mail addresses) into a public git repository. People could clone the source, modify it, and send pull requests to the site admin. The people I have discussed this with have not showed much interest.

The expandfile tool source is backed up to a public repository on GitHub. expandfile usage and installation instructions are documented in the site admin's website.

If these locations don't work, try the Wayback Machine.

Archive Copy at Computer History Museum

CHM is keeping a .tar.gz file of the multicians website for static presentation in their permanent archives. (Mail with Dag Spicer, Senior Curator, 12 Sep 2021.) The site administrator creates a tarfile using make_mx_tar occasionally, copies the tarfile to a CHM Dropbox folder, and notifies CHM.