Good Enough: A Meditation on the Past, Present and Future of EPUB

VW Beetle The Rallye Monte Carlo Historique XVII started from Paisley Abbey on 23rd January 2014, the only UK starting point for this event
  • Sumo

This is a guest post from Dave Cramer, co-chair of the W3C’s EPUB 3 Community Group, and member of the CSS Working Group, in addition to being Senior Digital Publishing Technology Specialist, Hachette Book Group .


It all started with a rant to some of my #eprdctn friends (who are my real-life friends too; I love this community):

11:27 dauwhe: I've been thinking about "don't break the web" and epub
11:27 dauwhe: browsers have to support every feature forever, more or less
11:27 dauwhe: even the mistakes
11:27 dauwhe: if even one in ten thousand web sites uses something, it won't be removed
11:30 dauwhe: is epub in the same situation? I think 3.x is. As much was we want to fix our mistakes, we can't.
11:31 dauwhe: and people see no value in fixing mistakes. 3.1 is better in every way than 3.0.1, but nobody cares.
11:31 dauwhe: does 3.1 allow new kinds of books? no.
11:31 dauwhe: does 3.1 guarantee that you can use script now? no.
11:31 dauwhe: so where's the incentive to switch?
11:32 dauwhe: Even with EPUB4 or WP or PWP, how can we make it compelling enough for people to want to use them?
11:32 dauwhe: James Patterson works in OEB 1.0. Everything since has been minor details.

Why was I thinking about this? Late last year, work began quietly on a proposal to undo some of the changes in EPUB 3.1, create a new EPUB 3.0.2 spec, and then abandon EPUB 3.1. The stated motivation was for EPUB 3.0.2 to be entirely backward-compatible with EPUB 3.0.1: all existing EPUB 3.0.1 files would automatically be valid EPUB 3.0.2s. I believe one of the unstated motivations was for existing files to be automatically compatible with the “newest” version of EPUB, without having to make any changes (especially to the package version attribute).

This bothered me, both rejecting the good work in EPUB 3.1, and the idea of going backwards in version. But there were good arguments that everyone was doing the right thing. EPUB 3.1 made sense because we were removing features which hardly anyone used, and weren’t well supported—did any of you use bindings? But it also makes sense not to make old content invalid. So who’s right? What should we do? Can we make EPUB better without breaking anything?

Mistakes were made

Consider this HTML fragment:

<font color=red>hello</font>

Firefox 57.0.4, released two weeks ago, knows exactly what to do with this markup, even though the font element was deprecated twenty years ago, and removed entirely four years ago. The text is indeed red. On the web, the old and the new coexist, if uneasily.

The web has to be careful, because mistakes are permanent.

font was a mistake. People realized that, and specs changed. But why does font still work? Because of one rule: don’t break the web. If even three in ten thousand web sites use a feature, and removing a feature might render those sites unreadable, browsers won’t remove it. The web has to be careful, because mistakes are permanent.

The slow pace of EPUB

Twenty years into the age of ebooks, our community is trying to figure out what to do with our mistakes. But we don’t even agree on what our mistakes are, let alone what to do about them. Was EPUB 3.1 a mistake? Was ncx? Was epub:switch? At least we’re finally talking seriously about backward compatibility, about how to “manage” change.

We work in publishing because we love books. We work with ebooks because we’re both idealists and gluttons for punishment. We are full of frustration with the present and hope for the future. We want things to be better; we want change; we need change.

After more than six years of EPUB 3, EPUB 2 is alive and well.

But does anyone else want change? Laura Brady wrote an article on the slow adoption of EPUB 3 just last month; the very same EPUB 3 which became an official recommendation in October of 2011. After more than six years of EPUB 3, EPUB 2 is alive and well. Even my employer, one of the largest publishers in the world, makes EPUB 3s which are as close to EPUB 2 as possible. The ncx refuses to die.

There are two problems. First, EPUB 2 is often good enough. Most books are better without video. It’s hard to use some new HTML5 features due to poor implementations in older reading systems. Many accessibility benefits have spotty support—how many reading systems render MathML, or expose ARIA attributes?

Reading system support is far more variable than the specifications are.

Second, the lines between versions of EPUB are often blurred. Many of us remember fixed-layout EPUB 2, or EPUB 2 with audio. Our EPUB 3 files include the dreaded ncx. Reading system support is far more variable than the specifications are. Most EPUB 3.0.1 files are indistinguishable from EPUB 3.0 files.

Moving Forward

The question remains: how do we move forward? Perhaps we can learn something from the web, and make old and new coexist more easily. But first, we need to be aware of how different EPUB-Land is from Web World:

  1. The EPUB ecosystem depends on formal validation, unlike the web. One interesting aspect of this is that many formally invalid EPUBs actually work (at least when side-loaded). Most common reading systems already support package version="3.1". You can use regular HTML without the “X” in some places. And, of course, we all know about the millions valid EPUBs which don’t work.
  2. There are many more EPUB Reading Systems than browsers, making research, testing, and QA that much harder, and sometimes impossible.
  3. Browsers are well-tested and constantly upgraded. And they have dev tools.
  4. EPUBs are sold and displayed by someone else. Web sites don’t have intermediaries rewriting their content and generally changing their behavior randomly.
  5. All the major browser vendors participate in the web standards process. Most major EPUB reading systems do not participate in EPUB standards.
  6. EPUB is often not used directly, but converted into something else. What percentage of EPUBs created today are converted to Mobi formats by Amazon?

But, acknowledging all those differences, what should we do?

What is the interoperable core of EPUB? What features can be trusted everywhere?

1. Find the EPUB that is, not the EPUB we want.

Before we change EPUB any more, we need to find out what is supported today. EPUB creators are less interested in what should work than in what does work. Let’s create a version of EPUB that is thoroughly tested, that includes only features that are widely implemented, that is fact instead of fiction.

To do this we need tests, and we need lots of reading systems that pass the tests. What is the interoperable core of EPUB? What features can be trusted everywhere? Let’s find out.

2. More nuanced validation.

EPUBCheck is the pillar of our community, the center of our workflows, the true arbiter of what EPUB is and what EPUB isn’t. But this sophisticated tool is too often used in a simplistic way. No errors or warnings is good; literally anything else is bad.

We’re afraid to allow EPUBCheck to provide more information by default. Even the specs say that EPUB creators should be alerted if an EPUB contains deprecated features. But the spec also says that alert should be less severe than an error or warning, and so no one ever sees the alert.

What would happen if we used EPUBCheck differently? Instead of telling EPUBCheck that I’m an EPUB 3, what if EPUBCheck recognized what features I actually used, and then described my level of compatibility? Most of our EPUB3s work as EPUB 2. Why not say so? What we’re really checking for is a match between the content and the reading system, not between the content and the spec.

We’ve also talked about the possibility of a strict mode for EPUBCheck, where all obsolete or deprecated features would be forbidden. And there are errors of structure—is the package invalid in the XML sense?—and errors of features—content MathML isn’t part of the spec. Is there value in recognizing that an EPUB is structurally well-formed, even though it may include unrecognized features?

3. Trust HTML

One of the strengths of HTML is that it knows how to handle almost anything. Consider this fragment:

<epub-secrets>
<h-7 style="color: red;">Editors</h-7>
<simple-list>
<simple-list-item>Laura Brady</simple-list-item>
</simple-list>
</epub-secrets>

This works perfectly well in my browser. HTML has a rule for unknown elements: ignore them, but process their children. This is what makes it possible for past and future to co-exist. I could actually implement what you see above using HTML custom elements. So it might be magical in a cutting-edge browser, but still readable in a tech-challenged executive’s old copy of Internet Explorer.

Eventually, EPUB should get out of the business of policing HTML and CSS—those languages are both very robust, and have well-defined error handling. Does your EPUB contain epub:switch? No problem—web engines, even if they don’t understand that element, won’t crash. They know how to fail gracefully.

4. Make change compelling

Creating EPUB 3.2, or 3.0.2, is not going to make more people adopt a new version. The cost is too high; the benefits too small or too uncertain. Even if you think you can just run a script to update something small, do you really want to risk touching every single book you’ve ever published? Do you want to redo expensive QA?

If we want people to upgrade, we need to create something worth upgrading to. I would love it if script worked everywhere (a very hard problem), if I had persistent local storage, if I could have images that filled the reader’s screen.

EPUB has been a field where dreams are crushed—we built it, and they didn’t come.

5. Build, then specify

Yes, let’s imagine and then create great new features. But let’s make sure they work first, that people want them, that they’re well designed, that reading systems can support them. EPUB has been a field where dreams are crushed—we built it, and they didn’t come. The web community learned a long time ago that a specification isn’t real until it’s implemented, and you can’t discover all the problems without using it. We must learn this lesson, and require multiple interoperable implementations before specifications are final.

EPUB has evolved, but largely in ways we don’t recognize. CSS Flexbox now works in iBooks, because the underlying browser engine was updated. How can our specs recognize when things like that happen? The state of the art will evolve, no matter what we do.

What Do You Think?

How can EPUB move forward without abandoning existing books? How do we create specifications that match reality? How can we get our community more involved in shaping the future of ebooks? I would love to hear your thoughts, in comments, on twitter, in the community group, anywhere!

Acknowledgments

Thanks to Tzviya Siegman, Rachel Comerford, Benjamin Young, Matt Garrish, Daniel Glazman, Baldur Bjarnason, and Brian O’Leary for comments and ideas.


DAVE CRAMER has been typesetting print and digital books since the 20th century. A fascination with XML and automated pagination led to work on print typesetting with HTML and CSS. Complaining about EPUB led to a parallel career in standards work, where Dave edits various ebook and CSS specifications. He dreams of bringing the rich history of print design and typography to ebooks and the web.

10 Responses to “Good Enough: A Meditation on the Past, Present and Future of EPUB”

  1. I know I’m a pariah in that I don’t code except under extreme duress. But I like and agree with the basic thread of what Dave is saying.

    I just want something that works in Kindle, iBooks, Kobo, and even Nook. For authors and tiny publishers, like myself, this is the bottom line.

    What prompted this comment is that I still have no way to make a well-designed table readable by the four basic e-readers. Most of the problem is Kindle, BUT Kindle is two-thirds of most author’s sales. And tables? They are still the only real solution for tabular matter that I know of.

  2. Keith Snyder says:

    AMEN.

    Digital revolutions get to a point where the technical people forget content is actually the point. CDs at 44.1kHz didn’t all get dumped and replaced by music lovers when 96kHz recording became possible—because the clearly superior technical superiority made no difference to anyone who thought they were buying (or selling) MUSIC.

    Books are words that readers read, typeset in a way that makes the reading easier and/or more pleasant. Sometimes they’re also pictures that readers look at. For the vast majority of books, anything else is either an app, or a book with unnecessary tinsel glued on it.

    EPUB3 doesn’t need replacing or refining. The platforms it runs on need to not suck.

  3. twobyte says:

    ePub as a standard is just fine whatever version it is. What needed is a reading apps (Kindle, ibooks etc) and devices (Kindle, Nook Kobo others) to declare their capabilities, for example that it is able to handle FlexBox etc, so that the author of ePub file can control the presentation of the content using CSS queries. The default set of such capabilities can be the lowest denominator of all current reading apps and devices or something like that. It seems that this route of adoption is more realistic and pragmatic for both sides: content authors and reading apps providers.

  4. JayPanoz says:

    > for example that it is able to handle FlexBox etc

    You mean like feature queries, which first appeared 2013? https://hacks.mozilla.org/2016/08/using-feature-queries-in-css/

    Cos it already is implemented in Reading apps using rendering engines (through web browsers, web views, etc.) which support @supports…

    (Spoiler: won’t necessarily work as expected for flexbox, grid, etc. though, because of the column-box model).

  5. twobyte says:

    “Column box” model is an implementation detail – I hope you understand what I mean by that – it is nowhere required by ePub standard, so it may change with time to something better. Because ePub is mostly a container for the content, it should stay like that and do not interfere with content and presentation which are governed by other standards. I hope that W3C will play along this lines after absorbing IDPF for good. Have a good day!

  6. MURATA Makoto says:

    Having read “Find the EPUB that is, not the EPUB we
    want” and “More nuanced validation”, I am glad that
    leadership of Publishing@W3C are more serious about
    testing and epubcheck.

    But there is another important thing. Fixing bugs
    in specifications is extremely important. W3C has
    published errata, new editions, and minor versions
    for fixing bugs. Such documents do not introduce
    any new features. But they improve readabiity, make
    ambiguous descriptions clear, and correct technical
    mistakes. Development of such documents is boring,
    but is crutial for interoperability. I think that
    Publishing@W3C has to patiently do such boring
    works.

    There are many unclear descriptions in EPUB 3.0.1.
    For example, I recently had a discussion whether
    vertically-scrollable manga is representable as
    fixed-layout EPUB. Wording in EPUB 3.1 carefully
    uses “viewport area” and “content display area”, and
    is quite clear. I was thus able to conclude that
    vertically-scrollable manga requires reflow rather
    than fixed layout in EPUB 3.1. But EPUB 3.0.1 is
    very unclear and I thus have no ideas about 3.0.1.

    Do external registries of vocabularies, which were
    introduced as part of 3.1, apply to EPUB 3.0.1
    documents? What is the order of encryption and
    compression in 3.0.1? All these issues are unclear
    in EPUB 3.0.1. Addressing these issues helps
    interoperability. I think that Publishing@W3C is
    oblidged to address them.

    Although I proposed 3.2, I am now wondering 3.0.1
    (second edition) is better. It should provide no
    new features, but should handle bugs

  7. JayPanoz says:

    Since you decided to focus on the spoiler for some reason…

    > “Column box” model is an implementation detail – I hope you understand what I mean by that

    Well, actually I don’t. But maybe that’s because I had to implement EPUB rendition and know it’s not a detail at all. And also know that there’s a lot more to the “de facto standard” (spec as implemented in the real world) that what’s written in the spec.

    A few remarks:

    – it is tied to the fragmentation model/logic currently implemented in each rendering engine;
    – with print, it’s the only cross-platform and visible practical use of those fragmentation logics;
    – iOS pagination API is based on columns (they updated Webkit in order to make it happen, and it automagically maps column breaks to page breaks or else that wouldn’t even work);
    – paged overflow was based on columns;
    – CSS regions were a superset of columns;
    – the paged media implementation that never was in Blink circa 2013 was based on regions;
    – all are tied to the fragmentation logic browser vendors implement.

    So if say you have a bug in columns, it means there is probably something to fix at the fragmentation level, and all current and future specs will benefit. If we don’t focus on such implementation details we can focus on at present, I’m not holding my breath as regards better specs – which will take 5–10 years anyway e.g. flexbox/grid.

    And BTW, this doesn’t invalidate the following

    > it should stay like that and do not interfere with content and presentation which are governed by other standards.

    Since it’s not interfering, it’s helping the whole ecosystem benefit. Web designers can obviously use flexbox, grid, etc. in columns. In that regard, the “implementation detail” is just a use case for the specs governed by the CSSWG – cf. this article by a Firefox developer: https://medium.com/@patrickbrosset/css-grid-css-multi-columns-7664f59bb60c

    Which is also the reason why, for instance, we report bugs, feature requirements, etc. to the spec editors: issues related to multicol very often outscope the spec itself and end up impacting a lot of others e.g. overflow, CSS break, etc. because they are related to fragmentation.

    And I’d say it is part of “Find the EPUB that is, not the EPUB we want.” Because this also has consequences for a lot of features in the spec. FWIW, we’re long past theoretical purity, we know little tiny “implementation details” can have huge impacts at the interoperability/compatibility level, which in turn impacts the spec.

  8. Robert Nagle says:

    I had a hard time following this article because I couldn’t find any URL detailing the differences between 3.01 and 3.1. I finally found this summary of changes between epub 3.01 and 3.1.

  9. Bill Seitz says:

    Reading systems could probably “upgrade” old ebooks if it weren’t for DRM, right?

  10. Liam Quin says:

    Sorry for a late reply; DRM makes no difference to the reader’s ability to “upgrade” old content: the reader needs to be able to access the content in order to render it. The best way is not to upgrade the content but to have backwards-compatible rendering, and for authors/publishers/developers of new books to be able to rely on fallback, as Dave Cramer suggests.

    Of course, the readers cannot “upgrade” a book and let you redistribute the result – not many publishers would want that, even for free books, because people would be calling support for problems caused by faulty upgrades… or woud be blaming the author or publisher for problems.