XML first and XML last workflows with InDesign come with several limitations. Technologies for automated typesetting promise to be faster and better suited for maintaining XML quality. For this purpose, the advantages and disadvantages of XSL-FO, PrintCSS and LaTeX are discussed in this contribution.

Introduction

In traditional print-based publishing workflows, BITS is often created after the final PDF. These workflows can be characterized by the usage of established typesetting applications such as Adobe InDesign. The typesetting process includes typically a considerable number of manual work which is primarily focused on creating the print-ready PDF. The XML is converted ex post and receives usually a lower level of attention. Historically, PDF was considered as the main product by scholarly publishers and the only raison d'être of XML was storing metadata and citations. This situation changed fundamentally when XML became more and more valuable to libraries, abstracting and indexing services and web platforms. When the industry started to require full text XML, the shortcomings of this print-based workflow model eventually became visible.

Creating XML with a print-oriented page layout software comes with certain challenges. First of all, you need to create a sequential document flow even if the reading order in the layout is less distinctive. Since page layout software is primarily designed for (typo-)graphical representation, you need to inject semantics, for example by associating style names with XML elements. Furthermore, you need to link related objects such as cross references and citations. These and other tasks make the traditional typesetting process more complex, time-consuming and error-prone. Although this XML last approach provides a practical solution for print products with high layout standards, it may be too expensive for more standardized products.

For this purpose, XML first promises to be a more efficient alternative. XML First means not necessarily that the manuscript has to be authored with an XML editor but that the XML is available after the editorial stage and is carried through the entire typesetting process. When Adobe introduced the XML import and export interfaces with InDesign CS4, it finally seemed possible to combine data quality with high typographic standards by embedding the XML as hidden layer of the page layout.

Unfortunately, it turned out that InDesign’s XML interfaces were not well-engineered and Adobe showed no interest in investing in their improvement. The XML import and export was not agnostic to the XML structure[1] and you could easily corrupt the data with simple text corrections[2]. For tables and images, InDesign required a specific XML structure enriched with proprietary attributes. Additionally, InDesign is not very efficient in XML processing, so large chunks of XML data can render the application almost unusable. Even though there are workarounds, such as flattening the XML structure, the results could not deliver on the initial promise to seamlessly carry XML through the process. For that reason, le-tex moved to IDML (InDesign Markup Language) for XML conversion.

Given the problems of XML first and XML last workflows with InDesign or traditional page layout software in general, it’s obvious that the question of InDesign alternatives arises. What is the status of automatic typesetting technologies? Are they able to lower the burden of maintaining data and typographic quality at the same time?

Automated Typesetting

The promise of automated typesetting is simple: With a snap of your fingers, a proof or print-ready PDF can be generated from structured data. Since the typesetting is fully automated, a layout change does not affect the underlying XML structure. In an ideal world, the only task would be the markup of the manuscript and the typesetting machine would automatically create the final PDF.

Even if you start with a perfect XML document, there is preliminary work needed to ensure that the results meet the expected requirements. General layout options need to be adjusted, like choosing the layout template and page size. In some contexts, it’s necessary to insert processing instructions to improve the layout, for example by rotating an image or table. Since hyphenation is automatically executed based on the stored dictionary, manual corrections are inevitable to avoid the hyphenation of names. If the typesetting system is not capable of simultaneously maintaining the baseline grid and avoiding orphans and widows, manual interventions are required, too.

However, a large amount of work is needed to initially set up the typesetting machine. The layout must be implemented with a specific style sheet language. XML nodes must be mapped to their respective layout elements. This mapping can range from simple lookup-tables to complex transformations. Even for an apparently simple element like a chapter heading, you need to create and format multiple blocks of information:

• title of the chapter
• chapter number
• chapter subtitle
• chapter author(s)

Not to forget, you may need to create slightly different information for running titles, table of contents and PDF bookmarks. When setting up an automatic typesetting workflow, there are many things to consider. Additionally, each information must be formatted in a way that it fits perfectly on a printed page. This imposes a challenging task compared to formatting a reflowable e-book. Considering the setup effort, typesetting automation only pays for itself after a considerable amount of printed pages.

Perhaps the most important significant decision is the selection of a suitable typesetting technology. Since there are significant differences in terms of setup effort, extensibility and quality, it is worth taking a look at the typesetting technologies available.

InDesign Server

InDesign is most commonly used in a traditional desktop environment. Several tasks can be automated, for example with XML import and export, templates, scripts and plugins. However, the desktop version allows no full automatization, so users must manually start automation tasks.

For automating the creation of many documents, Adobe offers a server variant of the product which should not be left unmentioned:: InDesign Server is a layout engine without a built-in user interface intended for server use only. It offers support for multiple instances, error capturing and logging and can be run via SOAP, COM or InDesign plugins[3].

Fig. 1

InDesign desktop and server components

InDesign Server is built on the same code base as the desktop version and the methods for automating tasks do not differ significantly between the two:

1. Templates: many tasks can be automated with a template, for example layout, pagination, running headers, styles.
2. Scripting: With Extend Script and its successor UXP Scripting, InDesign offers two scripting APIs to automate tasks. This is basically Adobe’s flavor of JavaScript that can access InDesign functions through an object model. With InDesign Server, it’s not possible to work with interactive or user interface components of the scripting API.
3. Plugins: When it becomes necessary to access low-level functions of InDesign, you can write plugins with C++ but these are naturally harder to develop and maintain.

Unfortunately, InDesign Server shares the disadvantages of its desktop sibling when it comes to XML processing. Both InDesign versions offer only XSLT 1.0 and a limited subset of XPath 1.0. This can turn simple tasks like grouping, filtering and sorting of nodes or text manipulation into inconvenient programming tasks. Critical tasks would rather be implemented outside of InDesign with pre- or post-processing in XSLT 2.0/3.0.

Another drawback is performance: You can run multiple instances of InDesign Server on a single server, but InDesign Server is not a multi-threaded application[4]. An InDesign Server instance can run only in a single thread. When it comes to scripting, even non-blocking actions are executed sequentially. For example, if you have a script to create cross references, each cross reference is created after the other.

InDesign server provides only limited feedback in case of errors. Even though every InDesign job generates a log file, many script errors are not logged but cause the process just to abruptly stop. For debugging, you would take the InDesign file and run the tasks manually in your desktop version of InDesign. Furthermore, you would insert logging statements throughout your script code to track the execution flow and capture relevant information.

One major advantage of InDesign Server is that you can continue to work with your accustomed designers to create layout templates. Skilled programmers are only required for scripting and system integration. For whom certain InDesign features are essential and who can afford the high license costs should consider InDesign Server. From an XML user's perspective, working with InDesign in an automated fashion is less error-prone but the application lacks of modern technologies like XSLT 3.0 and XPath 3.0. From my experience, programming and debugging InDesign scripts can be very cumbersome and time-consuming. Eventually, you would expect higher performance gains from an automated typesetting system.

XSL-FO

XSL-FO (XSL Formatting Objects) is an XML-based markup language that provides a vocabulary for formatting print media. XSL-FO was published as W3C Recommendation in 2001[5] and the latest version was published in 2006[6]. An XSL-FO based typesetting process consists of two steps:

1. convert from XML input document to FO XML, typically with XSLT
2. pass the FO file to a FO formatter that creates the final PDF

There are three widely adopted FO formatters: Antenna House Formatter, RenderX XEP and Apache FOP, the latter being released open source.

Alike InDesign’s parent pages (formerly known as master pages), XSL-FO provides elements to create page templates (page master). A page master can be used to specify the margins and regions on a page. The page model of XSL-FO defines a main area and four surrounding boxes with margins. While the region body is for content, the other regions can be used to specify marginal columns, running titles and page numbers.

Fig. 2

XSL-FO page model

These page masters can be later applied to page sequences. In this sense, a page sequence references a page master to apply its style to a series of pages, for example the frontmatter.

Here is a simple “hello world” example which specifies a page master that is later referenced by a page sequence.

<fo:root xmlns:fo="http://www.w3.org/1999/XSL/Format">
  <fo:layout-master-set>
    <fo:simple-page-master master-name="myPageMaster" 
      page-width="210mm" page-height="297mm">
      <fo:region-body/>
    </fo:simple-page-master>
  </fo:layout-master-set>
  <fo:page-sequence master-reference="myPageMaster">
    <fo:flow flow-name="xsl-region-body">
      <fo:block font-size="12pt">Hello World!</fo:block>
    </fo:flow>
  </fo:page-sequence>
</fo:root>

Since XSL-FO is based on XML, it’s easy to integrate into XML workflows and brings Unicode support by default. Many of the typographic attributes are derived from CSS. Because XSL-FO is a pure formatting vocabulary, it’s agnostic towards the XML input document. Only an XSLT stylesheet is needed to transform your XML input document to XSL-FO.

XSL-FO had a number of typographical limitations which prevented a wider adoption in the publishing sector. To share some examples: In XSL-FO it’s not possible to specify a baseline grid. You cannot control how footnotes are numbered and you couldn’t switch between one column to multiple columns within the same region.

XSL-FO implementers like Antenna House addressed these issues with proprietary extensions[7]. These extensions added features to the FO formatter that are not part of the original specification. Many XSL-FO based book typesetting projects wouldn’t have come to fruition without these extensions. But they make it also more difficult to switch from one FO formatter to the other.

Another issue for XSL-FO users is that the standard has not been actively maintained since 2013. The former member of the XSL-FO working group Liam Quin stated that not enough people took part and e-book publishers showed a greater interest in CSS for paged media[8].

When XSL-FO was published, CSS had only a minimal set of properties for print media[9] and LaTeX lacked support of Unicode and OpenType fonts. For these reasons, XML developers considered it as an obvious choice for paged media at the time.

“CSS is never going to fix it. … In the XML world, the obvious tool is XSL, the Extensible Style Language, not the Transformation language.”[10]

Norman Walsh, 2004

However, CSS and LaTeX evolved since then: The W3C introduced the Paged Media Module Level 3[11], shortly known as “PrintCSS” and current TeX engines such as XeTeX and LuaTeX support Unicode and OpenType fonts.

PrintCSS

PrintCSS appears to be the natural successor of XSL-FO. The CSS working group has many members and stronger support from the industry. In contrast to the heavyweight XSL-FO, PrintCSS is much easier to learn, especially if you’ve worked with regular CSS before.

XML and HTML documents can be styled with PrintCSS and media queries allow to apply CSS depending on the output device. CSS media queries make it possible to include formatting instructions for screen and print in a single stylesheet. Here is how you would specify a type area with PrintCSS:

@media print { 
  @page {
    size: 210mm 297mm;
    margin: 30mm 30mm 60mm 30mm;
  }
}

The page model of PrintCSS specifies a main area for the content flow and page margin boxes that can contain generated content. PrintCSS comes with pseudo classes for selecting pages. You can address left and right pages, the first page and blank pages. For the numbering of pages, tables and figures, you can create counters. Furthermore, PrintCSS allows to specify footnotes, generate strings for running titles and it adds more options for controlling page breaks. For a more detailed introduction, I recommend Rachel Andrew's article Designing For Print With CSS[12].

Fig. 3

PrintCSS page model

Designing layouts with PrintCSS is very straightforward, but it inherits many limitations from its predecessor XSL-FO. Like XSL-FO, you cannot define a baseline grid or specify whether to omit table headers at page breaks with PrintCSS. It’s not possible to limit the number of consecutive lines that may end with a hyphenated word. Another issue is that you cannot enforce line breaking on special characters or control the hyphenation character which is required in certain situations, for example for URLs. Properties to generate PDF bookmarks or PDF tags for accessibility are also not part of the standard. Many of the parts that are missing in the PrintCSS specification are addressed by proprietary vendor extensions. The PrintCSS renderers from Antenna House[13] and Prince XML[14] already come with a number of extensions that are prefixed with

-ah-

-prince-

Typesetting requires certain text snippets to be shortened or manipulated, e.g., for the table of contents or running titles. CSS is very limited in this regard, so it’s more practicable to have an XSLT preprocessing that creates the information.

Although PrintCSS is a promising technology, the specification lacks some features needed for professional typesetting. Since 2018 the status has remained unchanged as “W3C Working Draft”. Obviously, the slow progress on the specification doesn’t stop you from creating books with PrintCSS as long as the missing features can be replaced with vendor extensions. The downside is that these extensions tie you to one vendor and increase the costs of switching to another PrintCSS renderer. A solution might be that the manufacturers of PrintCSS renderers coordinate together to unify the naming of their proprietary extensions and propose it to the CSS Working Group.

LaTeX

Despite being developed in the 1980’s (the underlying TeX system even in the 1970s), LaTeX has always been an important part of the scientific publishing ecosystem. LaTeX is a collection of macros that simplify and extend the use of TeX for the average user. Here is a minimal LaTeX document:

\documentclass{article}
\begin{document}
  \section{Hello world!}
\end{document}
      

The first line of a LaTeX document specifies the document class, which defines the overall layout and formatting of the document. The document class declaration is followed by the preamble. It typically includes any necessary packages, commands and formatting instructions that are not already defined by the document class. The document body can be structured with chapters and sections.

If you want to create PDFs from XML with LaTeX, there are two common methods to get there:

1. parsing XML with xmltex[15]
2. generating LaTeX with XSLT

The first method requires xmltex, an XML parser implemented in TeX that associates XML elements and attributes with LaTeX instructions. However, the configuration is very sophisticated and xmltex works only with standard LaTeX, so you need to map Unicode characters to TeX instructions. If you want to use modern TeX engines such as LuaTeX or XeTeX, it is recommended to generate LaTeX code with XSLT.

However, the implementation with XSLT can become very complex when documents include tables and mathematical equations or they differ in the XML schema. To address these problems, le-tex developed the XProc/XSLT library xml2tex[16] that can be configured to convert arbitrary XML to LaTeX. For example, le-tex’s probably most popular Open-Source tool docx2tex[17] includes xml2tex as library.

An XML-based configuration can be used to control various aspects of the LaTeX output. You can create LaTeX instructions based on XPath matching patterns or regular expressions. There are separate sections for configuring the preamble, front and backmatter and a character map where Unicode characters can be mapped to LaTeX. Here is a minimal example which maps a BITS title element to a chapter macro:

<?xml version="1.0" encoding="UTF-8"?>
<set xmlns="http://transpect.io/xml2tex">  
  <template context="/book/book-body/book-part/book-part-meta/title-group/title">
    <rule break-after="2" name="chapter" type="cmd">
      <param/>
    </rule>
  </template>
</set>

For more demanding tasks, you can extend the configuration with XSLT. There is no additional configuration needed for tables and formulas. Instead, there exist modules to convert CALS and HTML tables and MathML formulas to LaTeX.

Regarding the layout capabilities of LaTeX, the main advantages are the sophisticated algorithms for line-breaking and spacing and the beautiful typesetting of equations. LaTeX is open source and has a rich ecosystem of freely available packages that further extend the functionality of LaTeX. For example, there are packages that add a baseline grid, dictionaries for hyphenation or layout options for equations.

On the other hand, there are sometimes packages that overlap in their feature set or cause conflicts in certain contexts. If you have a document with many float objects, the TeX processor sometimes has problems to position them properly on the page and rather outputs them at once at the end of a section.

An important advantage of LaTeX over XSL-FO and PrintCSS is that it is based on a programming language. In this sense, you are not limited to the features of the rendering software, rather you can override them. For instance, the chapter macro of LaTeX contains just the title as the only argument and is automatically numbered. Because the built-in macros from LaTeX are not sufficient for every use case, it’s a common method to override them or introduce new macros with more functionality.

le-tex used both methods for its new typesetting system: xerif[18] is the combination of transpect conversion pipelines, in particular those that include xml2tex, and a LaTeX macro package that provides customizable components for advanced publishing requirements. Where it’s applicable, common LaTeX macros for character markup, footnotes, indices etc. were used. When necessary, new macros were introduced like a new chapter macro that defines also subtitle, author, quotes, shortened running header, and the entries in the table of contents and PDF bookmarks.

\begin{heading}{section}
  \tpNumber{1.1.1}
  \tpTitle{\textsc{Shakespeare} of Stratford – The Life of an Author}
  \tpBMNumber{1.1.1}
  \tpRunNumber{}
  \tpRunTitle{Shakespeare of Stratford}
  \tpBMTitle{Shakespeare of Stratford}
  \tpTocTitle{\textsc{Shakespeare} of Stratford – The Life of an Author}
  \tpTocNumber{1.1.1}
  \begin{tpQuote}
    \tpQuoteText{He was not of an age, but for all time}
    \tpQuoteSource{\textsc{Ben Jonson}}
  \end{tpQuote}
\end{heading}

Thanks to the extensibility of LaTeX, le-tex could address other issues as well. For example, xerif introduces a new TeX table model named htmltabs[19] that is based on HTML structure and supports CSS properties for table formatting. Additionally, xerif includes style classes for figures and tables and introduces a baseline grid and new line breaking methods. As an outlook, it is also planned to implement a PrintCSS processor for xerif.

Fig. 4

xerif consists of two main components: the transpect xml2tex module and the LuaTeX rendering module that is based on a customizable TeX framework.

To sum up the pros and cons, LaTeX is open source, offers many packages for various purposes, the functionality can be flexibly extended and it is widely used. In contrast to PrintCSS, creating layouts with LaTeX requires a steep learning curve. The extensibility comes with a cost as well. TeX is hard to master and it lacks the concepts of modern programming languages. At least, with LuaTeX you can use Lua for many purposes.

Add-Ons for Layout Optimization

When typesetting publications automatically, there are sometimes different, better ways of arranging text, images, and boxes than the actual output of the renderer. Apart from setting the static layout parameters, it is often not possible to influence the dynamic rendering of the output. For example, there is usually no way to specify that the font size and spacing should be reduced if a table doesn't fit on the page, and that the table should be rotated if that isn't possible either.

This limitation can be circumvented by implementing a multi-pass process. For instance, the application can render the layout in multiple passes and evaluate the intermediate results by assigning scores for different layout parameters. With TeX, it is possible to control the rendering process by setting penalties[20]. For instance, if you want to minimize the number of paragraphs where the last line remains at the top of the following page (called "widow"), you can set a penalty for that. While you can turn widows on or off in many applications, the point here is that you can use a scoring system to achieve the probably best result.

In order to improve the rendering of automatic typesetting systems, various vendors offer add-ons or solutions based on these technologies. For InDesign, Typefi[21] offers plugins such as AutoFit which enables to align the width of layout objects vertically or horizontally. The typesetting application speedata Publisher[22] is based on LuaTeX and uses Lua to measure the size of various layout elements to create the best fitting layout. The W3C Print & Page Layout Community Group (PPLCG) provides XSLT extension functions[23] to get the area tree from an XSL formatter and make layout decisions based on it.

Conclusion

If the typesetting process depends on specific InDesign features and you want to stick with your trusted designers, InDesign Server may be an option unless XML features, licensing costs, or performance becomes crucial. While the other technologies discussed in this article cannot surpass InDesign in terms of typographic features, they are well suited to produce books and magazines faster and more cost-effectively. In contrast to InDesign, the technologies require not a particular XML structure and the data cannot be corrupted by the typesetting process. Furthermore, the rendering output can be enhanced by various add-ons and extensions.

It has been shown that the specifications of XSL-FO and PrintCSS lack some important features, but through proprietary vendor extensions productive use is still possible, albeit associated with a slight lock-in effect. For XSL-FO and PrintCSS, the features actually available are limited by the capabilities of the rendering software. LaTeX also lacks some features, but provides a thriving ecosystem of available packages and programming interfaces to extend its functionality.

In my opinion, XSL-FO will eventually be superseded by PrintCSS. An active working group, numerous implementations, and a flatter learning curve make PrintCSS a more compelling alternative. Limitations of PrintCSS with regard to content generation can be partially overcome with XSLT. If you want to have more control over the typesetting process and are not afraid of programming, then LaTeX is certainly the solution with which to achieve the best results. However, there is no such thing as the perfect typesetting technology, but rather the solution that is most practical for a particular purpose.

References

1.

Kraetke, M. (2015) XML und InDesign I – Tücken des XML-Imports [GERMAN]. Available at https://xporc​.net/2015​/05/14/xml-und-indesign-i-tuecken-des-xml-imports/ (Accessed: March 8, 2023).

2.

Kraetke, M. (2015) XML und InDesign III – Das seltsame Doppelleben von XML und Satz [GERMAN]. Available at https://xporc​.net/2015​/08/27/das-seltsame-doppelleben-von-xml-und-satz/ (Accessed: March 8, 2023).

3.

Adobe (2023) InDesign Server SDK 18.2.1. Interfacing with InDesign Server. Available at https://developer​.adobe​.com/console/servicesandapis/id (Accessed: May 24, 2023)

4.

Adobe (2021) Frequently Asked Questions. Available at https://www​.adobe.com​/in/products/indesignserver/faq.html (Accessed: May 24, 2023)

5.

W3C (2001) Extensible Stylesheet Language (XSL) Version 1.0. Available at https://www​.w3.org/TR​/2001/REC-xsl-20011015/slice6​.html#fo-section (Accessed: March 8, 2023).

6.

W3C (2006) Extensible Stylesheet Language (XSL) Version 1.1. Available at https://www​.w3.org/TR/xsl11/#fo-section (Accessed: March 8, 2023).

7.

Antenna House Inc (2022) Mastering antenna house XSL formatting extensions. Available at: https://www​.antennahouse​.com/xsl-extensions-guide (Accessed: March 8, 2023).

8.

Quin , L. (2013) Re: [xsl] xsl 2.0? XSLT mailing list , Paged media. Available at: https://www​.biglist.com/lists/lists​.mulberrytech​.com/xsl-list​/archives/201311/msg00012.html (Accessed: March 8, 2023).

9.

W3C (1998) Cascading Style Sheets, level 2 CSS2 Specification, Paged Media. Available at https://www​.w3.org/TR​/1998/REC-CSS2-19980512/page.html (Accessed: March 8, 2023).

10.

Walsh, N. (2004) webarch.pdf. Available at https://norman​.walsh​.name/2004/12/07/webarchPdf (Accessed: March 8, 2023).

11.

W3C (2018) CSS Paged Media Module Level 3. Available at https://www​.w3.org/TR/css-page-3/ (Accessed: March 8, 2023).

12.

Rachel Andrew (2015) Designing For Print With CSS. Available at https://www​.smashingmagazine​.com/2015/01​/designing-for-print-with-css/ (Accessed: March 8, 2023).

13.

Antenna House (2021) Introduction to CSS for Paged Media, p. 85, Available at https://www​.antenna.co​.jp/AHF/help/en/ahf-ext.html (Accessed May 25, 2023)

14.

Prince XML (2022) CSS Properties. Available at https://www​.princexml.com/doc/css-props/ (Accessed May 25, 2023)

15.

David Carlisle (2000) xmltex: A non validating (and not 100% conforming) namespace aware XML parser implemented in TeX. Available at https://packages​.oth-regensburg​.de/ctan/macros​/xmltex/base/manual.html (Accessed: March 8, 2023)

16.

le-tex transpect (2023) xml2tex source code at GitHub. Available at https://github​.com/transpect/xml2tex (Accessed: March 8, 2023)

17.

le-tex transpect (2023) docx2tex source code at GitHub. Available at https://github​.com/transpect/docx2tex (Accessed: March 25, 2023)

18.

le-tex (2023) xerif. Automatic is better. Available at https://www​.le-tex.de/en/xerif.html (Accessed: March 8, 2023)

19.

le-tex transpect (2022) xerif source code at GitHub. Available at https://github​.com/transpect/xerif (Accessed: March 8, 2023)

20.

WikiBooks (2023) TeX/Penalty. Available at https://en​.wikibooks​.org/wiki/TeX/penalty (Accessed: May 25, 2023)

21.

Typefi (2023) Typefi AutoFit. Available at https://www​.typefi.com​/products-services​/design-tools/autofit/ (Accessed: May 25, 2023)

22.

speedata (2023) speedata Publisher. Available at https://www​.speedata.de/de/ (Accessed: May 25, 2023)

23.

Print & Page Layout Community Group (2014) XSLTExtensions. Available at https://www.w3.org/community/ppl/wiki/XSLTExtensions (Accessed: May 25, 2023)

Martin Kraetke works as Lead Content Engineer at le-tex in Leipzig. He develops software with XSLT and XProc, speaks at conferences, and blogs about publishing technologies.