Testing, Testing

By Robert Kendall

(also in SIGLINK Newsletter, Vol. 6, No. 3)
[cover date: Oct. 1997; publication date: Aug. 1998]

When poets and fiction writers venture into the domain of electronic text, they want to think of themselves as creators of literature, not software. Yet the issues of software design are by no means incompatible with those of artistic creation. After all, every type of writing, hypertextual or not, is information technology. The technological underpinnings of written language have become transparent simply because they have reached such a high level of refinement. We don't think of the many centuries of development that went into our now-ubiquitous technique for recording language and presenting it in such ingeniously ergonomic media as the book.

Some of the concerns and practices of the software engineer are actually quite familiar to the creative writer. Even writers working in the "low-tech" medium of print routinely rely on an established procedure for successfully implementing any technology--product testing. Writers fashion vehicles for conveying something to the reader, but the vehicle may not always run properly. Before they end up in print, most pieces of writing are submitted to the critical eye of an editor, an agent, or a few trusted friends for feedback.

Of course, beta-testing software is something a little different. The software developer, working with a technology that's less mature and reliable, must focus primarily on the objective process of finding program elements that simply don't function properly. While a test reader of a written text can help in objectively "debugging" it of spelling mistakes or factual errors, its author is intent mostly on soliciting an opinion. Whether the piece "works" or not is a matter of the impression it makes and how it's interpreted. One reader's bug may be another's killer feature.

Hypertext is a true hybrid of text and software, so we should expect some of the more objective concerns of software development to come to the fore in working with the medium. Yet it may not be obvious how integral these concerns can be to the writer's creative enterprise.

The hypertext reader of course still forms a subjective interpretation of the text, but she also puts together something more tangible: a text realization consisting of a specific set of textual components viewed in a specific order, as a result of navigational choices. This poses a challenge unique to the creation of hypertext. It's relatively easy to ensure that the reader's options at any given point will be meaningful on a purely local level, but it's much more difficult to foresee the cumulative results of all the individual choices and to gauge whether every possible text realization will be satisfying as a whole. Unlike authors of linear writing, hypertext authors don't know what configuration of words will actually meet the reader's eye--and therefore in a sense don't really know what they've written--until they observe a representative sampling of text realizations for themselves.

The need arises here for empirical data. The normal mechanisms of editorial feedback may therefore blur into the more scientific procedures of software usability testing. The text realization process can be objectively monitored and even recorded as a reading history or log. While an author can write off many criticisms of textual content--a complaint about an unconvincing character, for example--as a matter of incompatible taste, it's much harder to similarly dismiss problems in a text realization. In fact, an author can observe a reading take shape and decide that it doesn't hold together without even knowing the reader's opinion on the matter.

The hypertext test cycle typically begins with the author repeatedly reading through the work. This can uncover such problems as dead ends, loops that are difficult to break out of, or text that is too hard to get to. The author can keep a running record of problem areas and then, if the work is in HTML, call up a directory listing in the browser to look for nodes that are consistently left unread at the end of each reading. In this process the author tries to assume the role of someone coming to the text for the first time and to anticipate the choices that a first-time reader might make. Assumptions about how people will navigate may not always be accurate, though, so for many authors the next step is to recruit testers of the work.

To get some insights into this process of testing the hypertext structure, as opposed to merely garnering feedback about textual content, I conducted an informal survey of testing practices among hypertext poets and fiction writers. I found that the most common type of reader/author interaction takes the traditional form of listening carefully to readers' comments, but many authors feel the need for a more "eyes-on" approach to studying the text realization process.

Deena Larsen has a cadre of about ten "beta-testers" who are willing to read her rough drafts while she looks over their shoulders and notes the routes they take. Larsen explains, "I have 'trained' them to say things like: I want to follow this thread because . . . I don't understand why that word linked to this space . . . " This gives her a better sense of "where readers will want to go and why."

Marjorie Luesebrink (who writes under the pen name M.D. Coverley) also employs this approach regularly. She says that comments about the work made after a reading often aren't that helpful because "we don't yet have enough language to talk about our experience in hypertext abstractly" and because it is often impossible to reconstruct a specific problem situation. "The body language of delight, or the sign of fatigue or frustration" can convey more than words about what is working and what isn't. Another benefit of direct observation, which I've experienced in testing my own work, is that the author can note problems that may not even be apparent to the reader but nonetheless keep her from seeing the work in its best light.

Public presentations of a hypertext can provide valuable experience with text realizations. Judy Malloy reports gaining insight into problems with The Yellow Bowl (forthcoming from Eastgate Systems) by watching readers at an open studio exhibition of it. Michael Joyce says he pays "close attention to how a text unfolds" when he gives public readings of his work and lets audience members make the navigational choices. The Web can provide another means of observing readers, though at a distance. Both Mark Amerika and Judy Malloy have found it instructive to analyze the page access logs of their hypertext fiction. Amerika cautions that it is prohibitively time consuming to fully reconstruct each reader's behavior from this data, though.

What sort of problems can this testing uncover? It often leads to revisions that give readers more flexibility. Reader responses have prompted both Larsen and Luesebrink to add more navigational aids to their work. When Malloy found that many readers of her Web piece l0ve0ne weren't entering it from the opening page (presumably arriving at internal nodes through search engines), she added a "home" link to each page. Testing can also be necessary for multimedia synchronization. In adding music to Califia (forthcoming from Eastgate Systems), Luesebrink found that determining how fast people generally read was necessary for the timing of the audio.

Particularly interesting are the far-reaching structural revisions that can emerge from observing readings unfold in unexpected ways on the large scale. Carolyn Guyer says that some early test readings of Sister Stories (an unpublished work written by her, Michael Joyce, and Rosemary Joyce) revealed that the original linking scheme weighted readings heavily toward the more difficult text--nonfiction material dealing with Aztec culture. The authors revised the link structure to allow easier access to other parts of the work.

Luesebrink had designed Califia assuming that readings would center around the most prominent of the story's three narrators. While women usually stuck with this female character, temporarily digressing from her path to other material, Luesebrink unexpectedly found that men tended to follow the story's male narrator. This led her to flesh out the male perspective. Malloy redesigned The Yellow Bowl to shift the reader back and forth between two parallel narrative threads when she found that readers tended to stick with one thread and miss the other. I encountered the opposite problem in my work A Life Set for Two (Eastgate Systems, 1996), finding that readers sometimes shifted too frequently among parallel versions of the text that represented different moods. I altered the interface to discourage them from changing "mood gears" with every new node, and thus promoted a more-natural metamorphosis of emotional states.

Serious attention to usability and beta-testing can obviously pay off for hypertext poets and fiction writers, and the software industry has a lot to teach us here. We should not be afraid to develop methodical beta-testing procedures and to suspend the sense of propriety that forbids looking over a reader's shoulder. It's also important to distinguish where we can between subjective feedback about content and objective observation of text realizations--that is, between soliciting opinions and testing the hypertext structure. Obviously authors can't hope to satisfy the personal tastes of every reader, but they can work to increase the chances that every possible text realization will meet their own artistic standards.

The testing effort would benefit immensely from better tracking systems that don't just monitor where readers have been but automatically flag potential problem situations, such as looping or excessive backtracking. It would also be nice if log files could accommodate comments by test readers indicating exactly what problems they encountered and where.

Improved methods for learning how people read our hypertexts and more intelligent tracking mechanisms could also lead to advances in hypertext systems design. If the software could recognize potential problem situations during a reading, it could also dynamically present solutions--such as adding links on the fly to afford a way out of a loop or to offer contrasting text if a reading becomes too heavily weighted toward one facet of the work. The better we understand the multilinear reading process, the better our hypertexts will be on all levels.