Xac
About
What is Xac?
Xac is a content management system (CMS) that generates a site based on content in markdown files. It basically does 4 things:
- it uses Pandoc to insert markdown content into custom user templates
- it supports arbitrary content insertion via plugins (see below)
- it optionally interpolates user-defined variables in other files
- it passes through everything else
The output can be static (X)HTML pages, dynamic PHP pages, or whatever else you want. It is entirely determined by the templates that you provide.
Why Markdown?
Markdown is simple. It lets you focus on the content and keep it separate from the presentation. It’s easy to read and easy to write. With Pandoc, it can be converted to many other formats too.
Of course, you could also write in other formats and convert them to markdown.
Why should I use Xac?
I never said you should. I’m not trying to convince you to use it either. I wrote it for myself and, as usual, I am sharing it in case someone else thinks it’s useful or interesting.
Usage
Xac requires the following:
- the Xac Python modules
- Pandoc
- at least 1 HTML template file
- an input directory of files to publish
To create content, just create some plaintext markdown files with the “.markdown” extension in the input directory. They will be inserted into the template and published on the server. Files without the markdown extension will be published directly on the server, unless you configure file interpolation.
The easiest way to learn how Xac works it to explore the examples included with the source code. The comments and documentation strings in those files will hopefully help to make it clear, and I will continue to improve them gradually.
The minimal example is the easiest to understand, but it’s not very interesting.
The “default server”. so named because it uses the same configuration as http://xyne.archlinux.ca, is a bit more complex and difficult to understand, but it also provides a real working example. It will show you how to create custom XHTML inserts, default directory listings for missing index files, basic plugin usage, a news feed, and custom CSS for different areas of the site.
For information about Pandoc usage and markdown syntax, consult the Pandoc User’s Guide.
Plugins
Overview
By default, Xac passes the input markdown files to Pandoc exactly as they are, but sometimes you will want to add something extra to the output file, such as custom syntax highlighting or a graph. To do that, Xac uses plugins.
The plugin system in Xac is simple. When Xac reads a markdown file, it scans for CDATA-like blocks of code, for example:
This is some markdown text before the cdata-like block.
<![FOO[input
input input input
input input input
]FOO]>
This is some markdown after the cdata-like block.<![FOO[ is the
start tag and
]FOO]> is the end
tag. Everything between is data, and it can be anything except the end
tag.
When Xac encounters such a block, it extracts the data from the tags and stores it in a dictionary. The key of the dictionary is a hash of the entire block, and the block is replaced in the text with that hash.
When the input is sent to Pandoc, the inserted hashes pass through. It’s just plain text to Pandoc. The hashes can then be replaced with output created by plugins that act on the original data in the block.
For example, you could place a dot-language graph in the block, and then use a plugin to create an SVG image of that graph and insert it directly into the HTML output. This is exactly what the graphviz plugin (see below) does.
If you want to escape the CDATA-like block, just add another exclamation mark. If there is more than one, one will be removed and the block will be re-inserted into the text without further changes. For example, to display the code above, the following was using in the markdown file:
This is some markdown text before the cdata-like block.
<!![FOO[input
input input input
input input input
]FOO]>
This is some markdown after the cdata-like block.This is some crazy bullshit right here <![FOO[input input input input input input input ]FOO]>
Xac plugins are just Python functions that accept 5 arguments:
- the Pandoc-generated output
- the path of the markdown source file
- a dictionary of hashes to data
- the output type (normally “html”)
- a dictionary of plugin options
Implementation
The plugin goes through the data in the dictionary, processes it, and then replaces the hash in the output with the generated content. The plugin options can be used to configure the inserted objects, such as setting their class for styling purposes or changing command-line arguments to external applications that generate the content.
What follows are examples using the included plugins. They should help clarify how the plugins work. Note that if you are reading this page on the default server then you must have the matching optional dependencies installed to publish this page. If you do not, you will not see the expected inserts below. If you have already tried to publish this page without all of the optional dependencies installed, you can just re-extract the files from the source archive. The included files contain the expected plugin output.
The following is the list of plugins provided by Xac. The tags in the
following examples are the default tags used by the default
server. You can change or override these as you like. To see
how these tags were set, look at the
get_default_post_pandoc_handlers function in
Xac/defaults.py.
Insert File
One of the simplest plugins is the insert_file.py
plugin. It will read the contents of a local file and insert them into
the output so they appear verbatim (HTML escaping is used
internally).
Simply insert the path to the file that you want to insert in a
FILE datablock. For example, to insert the input markdown
file for this page:
<![FILE[%SELF%]FILE]>This results in the following:
The datablock may contain one of the following:
- The tag
%SELF%, which will be replaced by the path to the markdown source file. - A relative path to file. This path is resolved relative to the directory of the markdown source file.
- An absolute path to a file.
GNU’s source-highlight
The source_highlight plugin uses GNU’s source-highlight
for syntax highlighting. Here’s the plugin code, highlighted with the
plugin itself:
#!/usr/bin/env python3 from Xac.common import * default_options = { 'args' : ['--css', 'code.css'], 'class' : 'xac_code' } def source_highlight(text, chunks, outtype, options=default_options): """Highlight code with GNU Source-highlight.""" if outtype == 'html': cmd = ['source-highlight', '-f', 'xhtml', '--no-doc'] + options['args'] try: cls = ' class="%s"' % options['class'] except KeyError: cls = '' else: raise TypeError("unsupported output type: %s", outtype) for tag, data in chunks.items(): i = data.find('\n') lang = data[:i] data = data[i+1:] if outtype == 'html': replacement = ('<div%s>' % cls) + pipe2(cmd + ['-s', lang], data) + '</div>' else: raise TypeError("unsupported output type: %s", outtype) text = text.replace(tag, replacement) return text
This was inserted using the following tags:
<![CODE[python
]CODE]>Note the use of the name after the opening tag to indicate the desired language. See
source-highlight --lang-listfor a complete list of supported languages.
This is useful if your version of Pandoc does not include support for syntax highlighting, or if simply you prefer GNU’s source-highlight. The default server includes support for highlighting Pandoc codeblocks delimited with “~” too.
The code hopefully also helps to clarify how the plugin works.
Highlighting Files
As of 2013-01-04 it is also possible to insert the highlighted
content of external files using CODEFILE tags. For example,
the following snippet inserts the default publish.py script
with Python syntax highlighting:
<![CODEFILE[python
publish.py
]CODEFILE]>#!/usr/bin/env python3 # -*- coding: utf-8 -*- from Xac.defaults import * from Xac.plugins.news_feed import gen_news_feed import Xac import os import sys # Your name. NAME = 'Foo Bar' # The site URL. SRV_URL = 'http://example.com' # The directory in which to store automatically generated content. It is a good # idea to keep this in a separate directory that it can be easily cleaned up. # # This should always map to the root of the server. AUTOGEN_DIR = 'files/autogen' # The server path of dynamically generated plugin content. DYN_PATH = 'etc/dyn' # The local path to the dynamic plugin content, usually a subpath of AUTOGEN_DIR. DYN_REAL_PATH = os.path.join(AUTOGEN_DIR, DYN_PATH) # The server path of the dynamic plugin content. DYN_SRV_PATH = os.path.join('/', DYN_PATH) # The temporary working directory for plugins that generate intermediate content. DYN_TMP_DIR = 'tmp' # The prepublication directory in which to stage files for publication. PREPUB_DIR = os.getenv('XAC_PREPUB_DIR', default='prepub') # The publication directory that will be uploaded to the server. PUB_DIR = os.getenv('XAC_PUB_DIR', default='pub') # The templates directory in which to look for the default templates. TEMPLATES_DIR = os.getenv('XAC_TEMPLATES_DIR', default="files/templates") # The templates CSS directory in which to look for the template CSS files. TEMPLATE_CSS_DIR = os.getenv('XAC_TEMPLATE_CSS_DIR', default="files/template_css") # The theme color of the site. THEME_COLOR = '#76249C' # The list of file directories to publish. The first component of the tuple is # the local path. The second component is the relative path from the server # root. None indicates that the files will be published in the server root. # # The hierarchy of each directory is recreated at the relative server path. # For example, given the local files # # foo/bar/a.txt # foo/bar/b.txt # # ('foo', 'baz') would create the following files on the server: # # baz/bar/a.txt # baz/bar/b.txt # # Files with the '.markdown' extension will be passed to Pandoc along with # template files returned by the get_pandoc_template function (see the Xac class # documentation). # # Non-markdown files can be interpolated using a custom get_var_delimiters # function. This function accepts a path and returns either None or a tuple # of variable delimiters. If it returns None for a given path, that file is # just symlinked into the server. If it returns a tuple of delimiters, then # the file is parsed and all configuration variables in config_vars will be # interpolated if they are flanked by the delimiters. # # This will hopefully be made clear with an example. The default # get_var_delimiters function returns ('@@@','@@@') for all files with a ".css" # extension. When Xac publishes a CSS file, it therefore checks the file for # "@@@foo@@@", where foo is any variable in the config_vars dictionary. All # instances of "@@@foo@@@" are replaced by the value for foo in the dictionary. # This enables CSS files to be easily customized using variables, but it could # be used for any file. # See how the news directories are used below. NEWS_DIRS = ('files/news', 'news') TEMPLATE_CSS_DIRS = (TEMPLATE_CSS_DIR, 'etc/css') FILE_DIRS = ( ('files/main', None), ('files/forum', 'forum'), (AUTOGEN_DIR, None), TEMPLATE_CSS_DIRS, ) # This is an *iterable* of patterns that will be matched against server paths. # Anything that matches is passed through without further action. Use this to # include a forum or .markdown files on the server. Ignored paths will still # be included in the nav bar, but not their contents. # Note the trailing slash for directories. Also be careful not to remove the # trailing comma for 1-tuples, as they will cease to be tuples. The pattern # would then be matched against the separate letters of the string. IGNORED = ( '/forum/', ) # This is just an example of some of the things that can be done by creating # custom functions for the Xac object. This will change the theme color and # banner for different areas of the site to make them more distinct. See the # comments in Xac.defaults for more details. areas = ( { 'path' : '/', 'name' : 'Main', 'color' : THEME_COLOR, 'banner' : '/etc/img/banner.png' }, { 'path' : '/news/', 'name' : 'News', 'color' : THEME_COLOR, 'banner' : '/etc/img/banner.png' }, { 'path' : '/foo/', 'name' : 'Foo', 'color' : '#1793D1', 'banner' : '/etc/img/banner.png' }, ) # The footer is inserted directly into the HTML template. # Here is a simple example. footer_fields = ( ('Contact', 'foo@example.com'), ) footer = r'<dl>' for t, d in footer_fields: footer += r'<dt>%s</dt><dd>%s</dd>' % (t,d) footer += r'</dl>' # Create the default Xac object. xac = Xac.Xac( file_dirs = FILE_DIRS, get_pandoc_template = get_default_get_pandoc_template(TEMPLATES_DIR), get_pandoc_args = get_default_get_pandoc_args(areas, AUTOGEN_DIR, TEMPLATE_CSS_DIRS, footer), config_vars = get_default_config_vars(THEME_COLOR), get_var_delimiters = get_default_var_delimiters, post_pandoc_handlers = get_default_post_pandoc_handlers(DYN_REAL_PATH, DYN_SRV_PATH, DYN_TMP_DIR), final_pandoc_handler = default_final_pandoc_handler, prepub_dir = PREPUB_DIR, pub_dir = PUB_DIR, ignored = IGNORED ) # Generate the news index and atom feed, and sort old new articles into # subdirectories of the news directory. def manage_news(): gen_news_feed( NEWS_DIRS[0], #src os.path.join(AUTOGEN_DIR, NEWS_DIRS[1]), #dest NEWS_DIRS[1], #srv SRV_URL, #url author=NAME, title=NEWS_DIRS[1].title(), last_update=xac.watchlist_mtime, xac=xac ) def main(): manage_news() xac.publish_html() #insert_maps() if __name__ == '__main__': if sys.argv[1:]: for path in sys.argv[1:]: os.chdir(path) main() else: main()
This uses the same filepath resolution function as insert_file.py.
tex2png
The tex2png plugin uses tex2png to insert
PNG images of (La)TeX math. It can be used with both custom cdata-like
blocks as described above, or with standard $ delimiters
when using Pandoc’s --gladtex option.
Equations such as
Using the plugin options it is possible to create even more complicated inserts using full LaTeX document input.
Graphviz
The graphviz plugin supports the insertion of arbitrary graphs using graphviz. Any layout can be used by changing the graphs layout attribute. The inserted graphs are in SVG format but it would be easy to modify the pluin to support PNG and other image formats.
The following graph is taken from the Pacserve project page.
The graph is inserted in the page by pasting a dot file between the following tags:
<![GRAPHVIZ[
]GRAPHVIZ]>Babel
The babel plugin uses the openbabel Python module to create and insert chemical structures using InChIs. This is just a simple example of what can be done with the module. A more complex example would be to insert a full table of chemical data, including data from online databases.
<![INCHI[
InChI=1S/C4H5As/c1-2-4-5-3-1/h1-5H
]INCHI]>
<![INCHI[
1S/C38H44O2/c1-9-11-13-15-29-23-30(16-14-12-10-2)25-31(24-29)26-32-27-33(17-19-37(3,4)5)35(36-39-21-22-40-36)34(28-32)18-20-38(6,7)8/h23-25,27-28,36H,9-12,21-22,26H2,1-8H3
]INCHI]>
Aha
The aha plugin uses aha to convert ANSI output to HTML. It can therefore be used to display colorized console output.
Here’s an example of the output of armh:
$ armd --show changes --days 5 --color --table - rng-tools 3-2 ^ aurploader 2012.7.23.7-1 2012.7.24.1-1 ^ bind 9.9.1.P1-2 9.9.1.P2-1 ^ dbus 1.6.2-1 1.6.4-1 ^ dbus-core 1.6.2-2 1.6.4-1 ^ dnsutils 9.9.1.P1-1 9.9.1.P2-1 ^ filesystem 2012.6-4 2012.7-1 ^ haskell-blaze-html 0.4.3.3-5 0.5.0.0-1 ^ haskell-highlighting-kate 0.5.1-5 0.5.1-6 ^ haskell-pandoc 1.9.4.2-1 1.9.4.2-2 ^ iana-etc 2.30-2 2.30-3 ^ imagemagick 6.7.8.4-1 6.7.8.6-1 ^ initscripts 2012.06.3-2 2012.07.5-1 ^ iptables 1.4.14-2 1.4.14-3 ^ libcups 1.5.3-6 1.5.4-1 ^ libpng 1.5.11-1 1.5.12-1 ^ libsystemd 186-2 187-2 ^ mpg123 1.14.3-1 1.14.4-1 ^ netcfg 2.8.5-3 2.8.8-1 ^ pm2ml 2012.7.20-1 2012.7.25-1 ^ pngcrush 1.7.31-1 1.7.33-1 ^ python-distribute 0.6.27-1 0.6.28-1 ^ python-numpy 1.6.1-1 1.6.2-1 ^ python2-distribute 0.6.27-1 0.6.28-1 ^ python2-numpy 1.6.1-1 1.6.2-1 ^ reflector 2012.7.15-1 2012.7.26-1 ^ systemd-tools 186-2 187-2 ^ texlive-bin 2012.0-2 2012.0-3 ^ whois 5.0.17-1 5.0.18-1 ^ xac 2012.7.20-1 2012.7.28-1 ^ xf86-input-evdev 2.7.0-2 2.7.1-1 ^ xf86-input-keyboard 1.6.1-2 1.6.2-1 ^ xf86-input-mouse 1.7.2-1 1.8.0-1 + aha-git 20120729-1 + haskell-blaze-markup 0.5.1.0-1 + valgrind 3.7.0-4
The above output was generated by placing command line output between the following tags:
<![ANSI[
]ANSI]>- Contact
- echo xyne.archlinux.org | sed 's/\./@/'
- Validation
- XHTML 1.0 Strict CSS level 3 Atom 1.0