In this paper we present Pàtrà an integrated document architecture which incorporates handwritten illustrations captured and rendered in a temporal fashion synchronized with audio, video, text, and image data. The architecture of Pàtrà permits non-linear growth in the form of multiple hierarchically organized play streams. Semantic meta-data is also an integral part of Pàtrà which serves a useful purpose of organizing such documents in a collection. We have developed an email application in which the users are provided with an authoring and rendering environment to compose, view, and reply to messages in the form of Pàtrà.