.mixin tutorial .mixin markdown

. copy basic.css

.title Livetext: A smart processor for text

*[This README is currently mangled. Fixes coming soon!]

Livetext is simply a tool for transforming text from one format into another. The source file has commands embedded in it, and the output is dependent on those commands.

Why is this special? It’s very flexible, very extensible, and it’s extensible _[in Ruby].

.section Why Livetext?

Livetext grew out of several motivations. One was a desire for a markup language that would permit me to write articles (and even books) in my own way and on my own terms. I’ve done this more than once (and I know others who have, as well).

I liked Softcover, but I found it to be very complex. I never liked Markdown much – I find it very dumb, and it’s not extensible at all. (In fairness to Markdown, it does serve a different purpose in everyday life.)

I wanted something that had the basic functionality of all my _[ad hoc] solutions but allowed extensions. Then my old solutions would be like subsets of the new format. This was a generalization similar to the way we began several years ago to view HTML as a subset of XML.

.section What is Livetext really?

Here goes: .list It’s a text transformer It’s Ruby-based (later on, more language agnostic) It’s (potentially) agnostic about output format It’s designed to be flexible, extensible, and easy It’s designed to be “plugin” oriented It’s like an old-fashioned text formatter (but extensible) It’s like a macro processor (but not) It’s like markdown and others (but not) It’s like erb or HAML (but not) It’s powerful but not too dangerous It’s not necesarily a markdown replacement It’s definitely not a softcover replacement It could possibly augment markdown, softcover, others .end

.section How does it work?

A Livetext file is simply a text file which may have commands interspersed. A command is simply a period followed by a name and optional parameters (at the beginning of a line).

The period will be configurable later if you want to use another character. The names are (for now) actual Ruby method names, so names such as ‘to_s and `inspect are currently not allowed.

At present, I am mostly emitting “dumb HTML” or Markdown as output. In theory, you can write code (or use someone else’s) to manipulate text in any way and output any format. Technically, you could even emit PDF, PNG, or SVG formats.

. Idea: Make an RMagick DSL as an example.

It’s possible to embed comments in the text. Later it will be possible to pass them through to the output in commented form.

The command ‘.end is special, marking the end of a body of text. Some commands may operate on a block of lines rather than just a few parameters. (A text block is like a here-document.) There is no method name corresponding to the `.end command.

The file extension I’ve chosen is ‘.lt3 (though this may change). *Note: The source for this README is a `.lt3 file which uses its own little _[ad hoc] library (called “tutorial.rb). Refer to the repo to see these.

.section Syntax, comments, and more

At first, my idea was to provide predefined commands and allow user-defined commands (to be distinguished by a leading ‘. or `.. marker). So the single and double dots were both legal.

However, my concept at present is that the double dots (currently unused) may be used for subcommmands.

User-defined commands may be added to the standard namespace. There are plans to permit commands beginning with a specified character other than the period (to be stored in their own namespace.

When a leading period is followed by a space, that line is a comment. When it is follwed by a name, that name is typically understood to be a method name. Any remaining text on the line is treated as a parameter list to be accessed by that method. Some methods accept a text block (multiple lines of text terminated by a ‘.end tag).

.section Boldface and italics

Very commonly we want to format short words or phrases in italics, boldface, or a monospaced (fixed width) font. The Markdown spec provides ways to do this that are fairly intuitive; but I personally don’t like them. My own notation works a different way.

First of all, note that these don’t work across source lines; they’re strictly intra-line. You may need (for example) an italicized phrase that spans across a newline; at present, you’ll need a workaround for that.

I find that most short items I want to format are single tokens. Therefore I use a prefixed character in front of such a token: Underscore for italics, asterisk for boldface, and backtick for “code font.” The formatting ends when the first blank space is encountered, without any kind of suffixed character.

I also find it’s common to want to terminate such a string with some kind of naturally-occurring punctuation mark. If we double the initial delimiter, it will be understood to terminate at the first period, comma, or right parenthesis.

Of course, there are cases where this won’t work; a formatted string may contain spaces, or it may exclude characters before the blank space. In this case, we can use an opening bracket after the prefix and a closing bracket at the end of the string.

This means that it can be difficult to include brackets inside a formatted token. The solution is simply to escape with a backslash.

A delimiter character sitting by itself need not be escaped. It will be output as a literal.

A delimiter character that is already inside another string need not be escaped. These cannot be nested (though there is a way to accomplish this using functions).

Most of this is summarized in this example (taken from one of the testcases):

.testcase basic_formatting

.section Standard methods

The module ‘Livetext::Standard contains the set of standard or predefined methods. Their names are essentially the same as the names of the dot-commands, with occasional exceptions. (For example, it is impractical to use the name `def as a method name, so the module has a `_def method instead.) Here is the current list:

.xtable

`comment    ::  Start a comment block
`errout     ::  Write an error message to STDERR
`def        ::  Define a new method inline
`set        ::  Assign values to variables for later interpolation
`include    ::  Include an outside text file (to be interpreted as Livetext)
`mixin      ::  Mix this file of Ruby methods into the standard namespace
`copy       ::  Copy this input file verbatim (no interpretation)
`r          ::  Pass a single line through without processing
`raw        ::  Pass this special text block (terminated with ``__EOF__) directly into output without processing 
`func       ::  Define a function to be invoked inline
`say        ::  Print a message to the screen
`banner     ::  Print a "noticeable" message to the screen
`quit       ::  End processing and exit
`nopass     ::  Don't pass lines through (just honor commands)
`include    ::  Read and process another file (typically a `.lt3 file)
`debug      ::  Turn on debugging
`nopara     ::  Turn off the "blank line implies new paragraph" switch
`newpage    ::  Start a new output page

.end

.section Examples from the tests

Here are some tests from the suite. The file name reflects the general purpose of the test.

.testcase hello_world .testcase comments_ignored_1 .testcase block_comment .testcase def_method .testcase simple_vars .testcase simple_include .testcase simple_mixin .testcase simple_copy .testcase copy_is_raw .testcase raw_text_block

.section Writing custom methods

Suppose you wanted to write a method called ‘chapter that would simply output a chapter number and title with certain heading tags and a horizontal rule following. There is more than one way to do this.

The simplest way is just to define a method inline with the rest of the text. Here’s an example.

.code

.comment
This example shows how to define
a simple method "chapter" inline
.end

. This is also a comment, by the way.
.def chapter
   params = _args
   raise "chapter: expecting at least two args" unless params.size > 1
   num, *title = params     # Chapter number + title
   title = title.join(" ")  # Join all words into one string
   text = <<-HTML
   <h3>Chapter #{num}</h3>
   <h2>#{title}</h2>
   <hr>
   HTML
   _puts text
.end
. Now let's invoke it...
.chapter 1 Why I Went to the Woods
It was the best of times, and you can call me Ishmael. The clocks
were striking thirteen.

.end

What can we see from this example? First of all, notice that the part between ‘.def and `.end (the body of the method) really is just Ruby code. The method takes no parameters because parameter passing is handled inside the Livetext engine and the instance variable `@_args is initialized to the contents of this array. We usually refer to the `@_args array only through the method `_args which returns it.

The ‘_args method is also an iterator. If a block is attached, that block will be called for every argument.

We then create a string using these parameters and call it using the ‘_puts method. This really does do a `puts call, but it applies it to wherever the output is currently being sent (defaulting to STDOUT).

All the “helper” methods start with an underscore so as to avoid name collisions. These are all stored in the ‘Livetext::UserAPI module (which also has some methods you will never use).

Here is the HTML output of the previous example:

.code

<h3>Chapter 1</h3>
<h2>Why I Went to the Woods</h2>
<hr>
It was the best of times, and you can call me Ishmael. The clocks
were striking thirteen.

.end

What are some other helper methods? Here’s a list.

.xtable

‘_args

Returns an array of arguments for the method (or an enumerator for that array)

‘_data

A single “unsplit” string of all arguments in raw form

‘_body

Returns a string (or enumerator) giving access to the text block (preceding “.end)

‘_puts

Write a line to output (STDOUT or wherever)

‘_print

Write a line to output (STDOUT or wherever) without a newline

‘_formatting

A function transforming boldface, italics, and monospace (Livetext conventions)

‘_passthru

Feed a line directly into output after transforming and substituting

.end

Note that the last three methods are typically _not called in your own code. They could be, but it remains to be seen whether something that advanced is useful.

.section More examples

Suppose you wanted to take a list of words, more than one per line, and alphabetize them. Let’s write a method called ‘alpha for that. This exercise and the next one are implemented in the test suite.

.testcase example_alpha

I’ll let that code stand on its own. Now suppose you wanted to allow columnar output. Let’s have the user specify a number of columns (from 1 to 5, defaulting to 1).

.testcase example_alpha2

What if we wanted to store the code outside the text file? There is more than one way to do this.

Let’s assume we have a file called ‘mylib.rb in the same directory as the file we’re processing. (Issues such as paths and security have not been addressed yet.) We’ll stick the actual Ruby code in here (and nothing else).

.code # File: mylib.rb

def alpha

cols = _args.first
cols = "1" if cols == ""
cols = cols.to_i
raise "Columns must be 1-5" unless cols.between?(1,5)
text = _body.join
text.gsub!(/\n/, " ")
words = text.split.sort
words.each_slice(cols) do |row| 
  row.each {|w| _print '%-15s' % w }
  _puts 
end

end .end

Now the ‘.lt3 file can be written this way:

.code

.mixin mylib
Here is an alphabetized list:

.alpha 3
fishmonger anarchist aardvark glyph gryphon
halcyon zymurgy mataeotechny zootrope
pareidolia manicotti quark bellicose anamorphic
cytology fusillade ectomorph
.end

And that is all.

.end

The output, of course, is the same.

You can define variables in Livetext, defined with ‘.set and referenced with a “$. Later there will be a few predefined variables. Variables are just string values.

.testcase simple_vars

If a variable needs to contain spaces, you can double-quote it.

.testcase more_complex_vars

Livetext permits user-defined functions (as well as defining a few predefined ones). Call a function with ‘$$ and (if applicable) pass a single string parameter between brackets.

.testcase functions

There is an important feature that has not yet been implemented (the ‘require method). Like Ruby’s “require, it will grab Ruby code and load it; however, unlike “mixin, it will load it into a customized object and associate a new sigil with it. So for example, the command ‘.foobar would refer to a method in the `Livetext::Standard class (whether predefined or user-defined). If we did a `require on a file and associated the sigil `# with it, then `#foobar would be a method on that new custom object. I plan to implement this later.

.section Issues, open questions, and to-do items

This list is not prioritized yet.

.list ~[Add versioning information ~[Clean up code structure Add RDoc ~[Think about command line executable ~[Write as pure library in addition to executable ~[Package as gem Document: ‘require `include `copy `mixin `errout and others Need ~much better error checking and corresponding tests Worry about nesting of elements (probably mostly disallow) Think about UTF-8 Document API fully Add `_raw_args and let `_args honor quotes Support quotes in `.set values Support “namespaced” variables (`[.set code.font=“whatever”]) ~[Support functions (“$$func)] Support function namespacing Create predefined variables (e.g., `[$_source_file], `$[_line]) Create predefined functions (e.g., `[$$_date]) More support for markdown Allow turning on/off: formatting, variable interpolation, function interpolation? `.require with file and sigil parameters Investigate “common intermediate format” - output renderers all read it Comments passed through (e.g. as HTML comments) `.run to execute arbitrary Ruby code inline? Concept of `.proc (guaranteed to return no value, produce no output)? Exceptions?? Ruby `$SAFE levels? Warn when overriding existing names? Think about passing data in (erb replacement) ~]Allow custom ending tag on `raw method ~[Ignore first blank line after `[.end]? (and after raw-tag?) Allow/encourage custom `passthru method? Must have sane support for CSS Support for Pygments and/or other code processors Support for gists? arbitrary links? other remote resouces? Small libraries for special purposes (books? special Softcover support? blogs? PDF? RMagick?) Experiment with idea of special libraries having pluggable output formats (via Ruby mixin?) Imagining a lib that can run/test code fragments as part of document generation Create vim (emacs?) syntax files Someday: Support other languages (Elixir, Python, …) `.pry method? `.irb method? Other debugging features Feature to “break” to EOF? `.meth? method ending in `? takes a block that may be processed or thrown away (`.else perhaps?) `.dump to dump all variables and their values `.if and `[.else]? Make any/all delimiters configurable HTML helper? (in their own library?) .end