spring cleaning

README.md

@@ -0,0 +1,58 @@
+---
+title: Website Publication Files
+cuid: 26e4-a0
+trls: [ nl ]
+lang: en
+---
+
+This describes how I publish selected files from this repository to the Web. Actually, these files _are_ the publication mechanism.
+
+<!--summary-above-->
+
+
+## The Engine
+The static site is generated with [Web Origami](https://weborigami.org). 
+
+`pipeline.ori` contains the main pipeline. `site.ori` selects some keys from `pipeline.ori` to create the final site. The reason for this split is: there are a bunch of steps in the pipeline which should not be exposed in the final build, but that do need to be accessible for inspection during development/debugging.
+
+Both files take an argument which specifies whether to build the site in public or private mode. In public mode, only files with a certain tag are included in the output.
+
+Public mode is the default. Run: `ori site.ori()` or `ori pipeline.ori()`.
+
+For private mode, run: `ori 'site.ori("all")'` or `ori 'pipeline.ori("all")'`.
+
+<trl-group>
+<trl-alt lang="en">
+
+## Markup Language
+I have developed a text markup format which suits my needs better than Markdown. The parser works and I can render HTML with it, but the feature set is still very limited. For example, I can only render single works in italic. That might seem like an odd sort of limitation, but it's because I've started from somewhat different first principles than the assumptions the HTML/Markdown worfklow is based on. As we all know, the last 10% takes the most work. So to start with, I'm formatting the source files in Markdown.
+
+</trl-alt>
+<trl-alt lang="nl">
+
+## Markup-taal
+Ik heb een formaat ontwikkeld voor tekst die beter bij mijn doelen past dan Markdown.
+</trl-alt>
+
+
+
+</trl-group>
+
+## Internal Links
+![link to image](/ad_pubfiles_pub.d/images/bird.webp)
+
+Internal links need to be absolute from the root of the directory. This SHOULD be fixed in a future version.
+
+
+
+## Web Origami, Public and Private Site Versions
+I want to make only select files public. Files whose name contains the tag `pub` should be included in the public build. The public build should be the default. To include all files, pass 'all' to the site.ori file. That sort of reduces the risk of accidentally publishing private files.
+
+
+<p>—private-below—</p>
+
+EXCEPT NOT REALLY! In this case, because this file is hardlinked to `README.md` inside a publicised git repository. You have been warned!
+
+## Organisation
+
+${ada_pubfiles-worklog.md/}

_atest.md

@@ -1 +0,0 @@
-WILL BE SEEN IN FINAL OUTPUT!

_origami-questions-deep-filters.md

@@ -1,44 +0,0 @@
----
-title: foobar
----
-
-(see test.ori. This is confusing!)
-So I was wondering the following, but I believe by default the path `..` returns a shallow tree by default, so the question is moot. That is: I was stuck on how to apply a regex to a nested tree of file paths, but I was erroneously assuming the tree was deep, but it's not.
-
-My original solution therefore works because the input is already just the top-level flat directory contents:
-
-```
-{
-    all: ..
-    filtered: Tree.filter(raw, (val, key) => key.endsWith('.md')) //yes, should really use globKeys for this, but actually I'm doing a more complex comparison with AND here
-}
-```
-
-Nevertheless, I can't figure out if its possible to achieve the goal stated in the subject.
-
-`globKeys` supports a limited set of glob patterns. I guess `mask` with `regExpKeys` is probably the closest, but it's not going to work across deep paths?
-
-given this directory structure:
-
-```
-/src
-    readme.md
-    installation.md
-    node_modules/
-        package/
-            readme.md
-```
-
-I only want `src/readme.md` and `src/installation.md`.
-
-I came up with this, except is it going to be a deep operation?
-
-```
-{
-     all: ..
-    selected: Tree.map(all, Tree.regExpKeys({"^[^\/]+?\/.*\.md$": false}))
-}
-```
-the regex returns false if the string ends with `.md` but also has a folder name with slash at the beginning. So these are excluded from the result.
-
-huh: would you have to do some rewriting of keys to a flat map with concatenated paths?

_testfile.ori.md

@@ -1,9 +0,0 @@
----
-title: ok
-id: testit
-date: today
----
-
-# My file
-
-${<_index.md>}

ada_public-site.ori

@@ -1,3 +0,0 @@
-{
-  files: site.ori('')/filenames
-}

adb_private-site.ori

@@ -1,3 +0,0 @@
-{
-  files: site.ori("all")/filenames
-}

allfiles.sh

@@ -1,2 +0,0 @@
-fd -e md --color=never
-#fd -e txt

another_pubfile.md

@@ -1,9 +0,0 @@
----
-title: Another Public File
-cuid: 26ef-a0
----
-
-This file is also public. But not all of it.
-<!--private-below-->
-This part will not be shown.
-Everything following here will be hidden from private view!

anotherfile.md

@@ -1,9 +0,0 @@
----
-title: Another File
-cuid: 26e4-a2
----
-
-Just some text, in this file.
-<!--summary-above-->
-This file should only be shown if site was built with `site.ori("all")`,
-because it does not have `_pub` in its filename.

echo.sh

@@ -1,3 +0,0 @@
-while read line
-do echo "$line"
-done

pageSummaries.js

@@ -1,6 +0,0 @@
-//return title and html ...
-export default (doc) => {
-
-
-}
-

parseHtml.js

@@ -1,14 +0,0 @@
-import {fromHtml} from 'hast-util-from-html';
-import {visit} from 'unist-util-visit';
-
-export default async function(value) {
-  const tree = fromHtml(value);
-  const links = [];
-  visit(tree, [{tagName: 'link'}, {tagName: 'img'}], function(node) {
-    links.push(node);
-  })
-  const hrefs = links.map(link => link.properties.href || link.properties.src)
-
-  return hrefs;
-  // return tree;
-}

pipeline.ori

@@ -3,59 +3,58 @@
   to keep parts of the pipeline private while still allowing better debugging.
   serve/watch pipeline.ori for debugging.
   server or copy site.ori to build final output.
+
+  See README.md/#
 */
 (all) => {
-    /*
+  /*
-      list is all the files in current directory.
+    list is all the files in parent directory.
-      allfiles is all source files: ending with `.md` and not starting with `_`.
+    allfiles is all source files: ending with `.md` and not starting with `_`.
-      pubfiles is the subset containing the string `_pub`.
+    pubfiles is the subset containing the string `_pub`.
-      files selects either of these depending on the argument passed to this file.
+    files selects either of these depending on the argument passed to this file.
 
-    NOTES:
+  NOTES:
 
-    - if markdown files are empty, Origami.document() will error.
+  - if markdown files are empty, Origami.document() will error.
-    */
+  */
-
-    /*
-      this is the external filename method. 
-    */
-  //  allfiles: allfiles.sh() //reads MD files.
-  //  pubfiles: pubfiles.sh(allfiles)
-    (list): ..
-    allfiles: Tree.filter(list, (val, key) => key.endsWith('.md') && !key.startsWith('_'))
-    documents: Tree.map(allfiles, {value: (value) => Origami.document(value)}) //if I don't use the verbose syntax I get the filename characters as part of value!
-    withfndata: Tree.map(documents, { value: (value, key) => addFilenameData.js(value, Origami.slash.remove(key))})
-
-    pubfiles: Tree.filter(withfndata, (val, key) => val.fnd.tags?.includes('pub'))
-
-    /*
-      Now convert to html.
-    */
-    asHtml: Tree.map(all ? withfndata: pubfiles, {
-      value: (value) => Origami.mdHtml(value)
-      key: (value, key) => `${value.fnd.name}`
-    })
-
-    /*
-    after converting to html, we can extract a 'summary' from the file.
-    With markdown, the title is taken from the YAML frontmatter,
-    and ends up in a separate field of the document.
-    So we dont have to worry about the title being part of the summary.
-    */
-    withSummary: Tree.map(asHtml, extractSummary.js)
-
-    /*
-    Removing private content coming below the html paragraph `<p>—private-below—</p>`
-    Only remove this if in public mode. So if this file is called as `site.ori("all")`,
-    then final output is `withSummary`, otherwise it's `privateRemoved`.
-    */
-    privateRemoved: Tree.map(withSummary, removePrivate.js)
-    final: (all ? withSummary : privateRemoved )
-  
 
   /*
-  slightly complex. This adds a property `html` to the page with the rendered content.
+    this is the external filename method. 
   */
+  //  allfiles: allfiles.sh() //reads MD files.
+  //  pubfiles: pubfiles.sh(allfiles)
+  (list): ..
+  allfiles: Tree.filter(list, (val, key) => key.endsWith('.md') && !key.startsWith('_'))
+  documents: Tree.map(allfiles, {value: (value) => Origami.document(value)}) //if I don't use the verbose syntax I get the filename characters as part of value!
+  withfilenamedata: Tree.map(documents, { value: (value, key) => addFilenameData.js(value, Origami.slash.remove(key))})
+  //files now have a key `fnd` for 'filenamedata`. This contains address, title, and tags parsed from the filename.
+
+  pubfiles: Tree.filter(withfilenamedata, (val, key) => val.fnd.tags?.includes('pub'))
+
+  /*
+    Now convert to html.
+  */
+  asHtml: Tree.map(all ? withfilenamedata: pubfiles, {
+    value: (value) => Origami.mdHtml(value)
+    key: (value, key) => `${value.fnd.name}`
+  })
+
+  /*
+  after converting to html, we can extract a 'summary' from the file.
+  With markdown, the title is taken from the YAML frontmatter,
+  and ends up in a separate field of the document.
+  So we dont have to worry about the title being part of the summary.
+  */
+  withSummary: Tree.map(asHtml, extractSummary.js)
+
+  /*
+  Removing private content coming below the html paragraph `<p>—private-below—</p>`
+  Only remove this if in public mode. So if this file is called as `site.ori("all")`,
+  then final output is `withSummary`, otherwise it's `privateRemoved`.
+  */
+  privateRemoved: Tree.map(withSummary, removePrivate.js)
+  final: (all ? withSummary : privateRemoved )
+  
   renderedPages: Tree.map(final, {value: (page) => {
     html: <page.ori>(page, all)
     ...page
@@ -68,14 +67,11 @@
   pagesInFolders: Tree.map(renderedPages, pagesInFoldersWithTranslations.js)
 
 
-  linksByFile: Tree.map(Tree.map(renderedPages, (a) => a/html),getLinkedFilesFromHtml.js) 
+  linksByFile: Tree.map(Tree.map(renderedPages, (a) => a/html), getLinkedFilesFromHtml.js) 
   uniqueLinks: Tree.flat(linksByFile)  → (a) => [...new Set(a)]
   linksAsTree: pathsToObjs.js(uniqueLinks)
   onlyLinkedFiles: Tree.mask(list, linksAsTree)
 
-  /*
-  Now, I think I have enough to build both the individual pages and the index page!
-  */
   index.html: indexPage.ori(final, all)
   ...onlyLinkedFiles
 }

pubfiles.sh

@@ -1 +0,0 @@
-rg '_pub' || true

readFiles.js

@@ -1,11 +0,0 @@
-import {promises} from 'fs';
-const {readFile} = promises;
-export default async function readFiles(filelist) {
-  const filenames = filelist.split('\n').filter(name => name !== '')
-  const files = {};
-  for (const file of filenames) {
-    const contents = await readFile(file, 'utf-8');
-    files[file] = contents;
-  }
-  return files;
-}

test-arg.sh

@@ -1 +0,0 @@
-echo "$1"

test-stdin.sh

@@ -1,3 +0,0 @@
-while read line
-do echo "$line"
-done

test.ori

@@ -1,30 +0,0 @@
-{
-  test: {
-    "readme.md": "hello world"
-    "index.js" : "import nest from './a/nested.js'"
-    a: {
-      "nested.md": "myfile"
-      "nested.js": "export default nest = ''"
-      b: {
-        "deeper.md" : "deep inside"
-      }
-    }
-  }
-  raw: ..
-  files: Tree.filter(raw, (val, key) => key.endsWith('.md') && /^[a-p]+_/.test(key))
-  regexpd: Tree.mask(test, Tree.regExpKeys({"^[^\/]+?\/.*\.md$": false}))
-  //regexpdtop: Tree.mask(test, Tree.globKeys({"**": {"*.md": true}}))
-  regexpdtest: Tree.mask(test, {
-    ...Tree.constant(true) 
-    ...Tree.regExpKeys({
-    "^[^\/]+?\/.*\.md$": false
-  })})
-  regexpdfiles: Tree.mask(files, {
-    ...Tree.constant(true) 
-    ...Tree.regExpKeys({
-    "^[^\/]+?\/.*\.md$": false
-  })})
-  paths: Tree.paths(test)
-  regexpaths: Tree.map(Tree.filter(paths, (value) => /^[^\/]+?\/.*\.md$/.test(value)), {key: (value, key) => value, value: (value) => true})
-  excluded: Tree.mask(paths, regexpaths)
-}

test.sh

@@ -1 +0,0 @@
-echo "foobar"

testUnpackFiles.js

@@ -1,8 +0,0 @@
-export default async (value) => {
-  if (value.unpack) {
-    const unpacked = await value.unpack();
-    console.log(unpacked);
-  }
-  return value;
-}
-

testinclude.ori

@@ -1,6 +0,0 @@
-{
-  test: testfile.ori.md/
-  all: .
-  allmd: Tree.filter(all, (val, key) => key.endsWith('.md') && !key.startsWith('_'))
-  docs: Tree.map(allmd, {value: (val) => Origami.unpack(val)})
-}