The stage of production I'm currently at is that the AR action script and Flash is complete with all variables coming from Max/MSP assigned to specific tasks. The Max patch is all set up to change image sets and sizes depending on the graphics tablets coordinates. The Photo album is all set up with the tablet in place and the interface laid upon the top of it.
I have setup all the symbols so that they are in fact double sided to add some more interactivity and choice to the application. The one side is affected in size by the users choice with the albums interface system, whereas the other side of the symbol keeps the image at a static thumbnail size. This enables certain images to remain static in size whilst others are altered and positioned to the users choice. All images can be static or custom depending on choice.
Now I am in the midst of putting together all the paperwork and presentation material needed for Tuesday 1st June, the presentation day. In the next post I'll upload the showreel which will show how the album works and displays Augmented Reality web images and explains visually how this project works after all this development.
So after I setup the graphics tablet to control image sizes and sets depending on x.y coordinates I needed to create a visual interface to aid the user in the choice of images and sizes.
I came up with a printed interface that sits directly over the tablet which disguises it and includes set points for the user to interact with using the tablets pen.
This is the interface that sits in the book. It works by the user simply placing the pen over the desired box for image set 1,2, or 3. This will trigger a bang in Max.MSP and send the selected set to flash for display in AR. The size is selected by dragging the pen up or down within the size box and will change the image size to small, medium and large.
The interface also corresponds to an on screen interface providing the user with feedback as to which set and size is currently selected. The image below illustrates this...
As mentioned previously I need to find a more physical and reliable method of controlling the incoming web content that is to be displayed using Augmented Reality. I stumbled upon something I hadn't seen since I was a kid. it was (at the time a really cool) book that featured in the cartoon inspector gadget, which was basically a computer embeded within a book that feautred screens and button interface. Now despite this not being a real technology and a cartoon after all, I decided to take some inspiration from the idea and create a physical control system to be embedded within the book.
My idea for the control system was to incorporate a Wacom graphics tablet to the book in order to control directly from the book itself. Luckily I have an A5 size tablet which fits perfectly on the inside of the front cover and can allow me to overlay a page over it to hide what it actually is and to place an interface design over the top.
The first thing I had to do was to connect the tablet to Max/MSP, again fortunately an external object for the detection of a graphics tablet has been created for max enabling the detection of all the buttons and surface area for use within Max. I set up the patch so that specific x/y coordinates of the tablets drawing area would enable image set selection and image size selection.
Here is a an image of the patch detecting the coordinates and outputing the values to select size and set.
Basically it detects the x and y axis coordinates of where the tablet pen is positioned on the drawing area and if they match a specific coordinate it will trigger a bang that will send a pre defined value to the flash server and be sent as a variable to Flash that will in turn control the image size and image set. The image set has been defined within Max/MSP and is loaded via an external text file. The user has to define images to each set from the external file and that will be automatically loaded into Max/MSP upon start up and then depending on the coordinate position of the tablet will select Sets 1,2 or 3.
This image shows the rest of the patch which features 24 text objects which will load images 1-24 from the 3 sets of images and below that is the flash server featuring an array of outputs sent to flash in the form of an array and takes values from each to load the specific image sets and sizes to be displayed in AR.
The next step is to create an interface for the tablet within the book itself so users know which set they are selecting as well as image sizing and create a corresponding on screen interface to enable feedback from selections.
After attempting to control the Augmented Reality content using additional AR markers to no success I decided to connect the Flash content to Max/MSP using the Flash Server object that enables communication between both softwares.
The flash server available for download from http://www.nullmedium.de/dev/flashserver/ enables such communication. However this particular version of flash server is scripted in ActionScript 2.0 and my entire project is AS 3.0. fortunately however a former iDat student Alex Ryley re wrote the application and converted into AS3 and made it available for use. Credit goes to Alex Ryley for the server object in AS3 and is available here: http://www.alexryley.net/2009/01/maxmsp-flashserver/
Once I had the As3 version of the flash server I set about creating a Max/MSP patch that would connect to Flash and allow communication between the two. Originally I had the idea of potentially using the voice controlled system as featured previously to change images sets and manipulate sizes, but again even with specific one word commands, it proved to be too unreliable and if used in a busy noisy area would prove troublesome. I needed to come up with a new method to physically control the content from within the book with reliability and ease
After taking into consideration the thoughts from the last meeting I've set to work focusing my efforts away from the virtual aspect of the project and how it can be controlled and changed physically from within the book itself or another method away from interacting with the computer.
I have tried first of all to use the symbols that load the AR content to control images sets, sizes and overall user interaction for the change and manipulation of web content. So far however to no avail, the symbols tend to still pick up content and not work alone as a method to change content without issues. I can't seem to find another suitable way to change content using physical items within Flash so I may have to attempt to use another application to work along side the Action Script content which is Max/MSP...
I had one last meeting with Gianni today before the final hand in and presentations of the project. I discussed my progress so far with him and what I had achieved with the Augmented Reality and the web URLs and explained my issues with using flickr and the plans to find an alternative method of displaying web based images.
The idea of how the photo album itself was going to work was also discussed and the idea that book should act as an interface for not only displaying the images in a physical manner but how it could be used to change images and user interaction without the user having to use the computer as a method of main interaction.
Gianni suggested a few ideas that could work, such as using a mobile phone to assign images and then using the book to manipulate and display the images, or perhaps some alternative method to make the book itself more "playful" in its use.
So now I have the AR markers displaying multiple images the next step is to decide where the web images that will be displayed will come from. I was hoping to use flickr as a source of web images for display in the photo album that could constantly change if the public timeline was used or have specific user galleries be displayed.
I tried to add the flickr api to the existing action script project with minimal success. The issue was with displaying multiple images on multiple markers with the type of code structure that the flickr api was designed with. Because of this I couldn't find a possible way to display a flickr api stream onto the AR Markers successfully so I have decided to set up the project to allow users to simply define image Urls to the project via an external source such as a text file or xml file that allows content to be defined and changed by the user.
After successfully loading an image to a single marker I needed to load multiple images to multiple markers. This is where I had some difficulty. I managed to find an extension of the FLARtoolkit called FLARManager, that is designed to enable multiple marker for multiple content display. I was lucky enough to find a further tutorial that showed how to create one 3D cube just like before but by using multiple markers it would create multiple instances of the one cube and assign a new color to each new instance. I needed to use something similar to this but rather than create multiple cubes, I needed multiple flat planes that would allow me to assign a different image to each rather than a colour. The tutorial was from http://rawkes.com/blog/2010/01/26/tracking-multiple-augmented-reality-markers-flarmanager-and-papervision#download
I managed to combine the work I had done previously with the image and the marker with the tutorial above and managed to get multiple markers to display the image from the previous post. The problem was that each marker displayed that one image so if I had three images on screen at once all three would be displaying the same image. I needed to find a method of assigning an image to each marker. After days of struggling and frustration at the lack of progress I went to see Ben Reynhart at Mutant Labs. He helped me out massively with this issue and showed me what was going wrong. I was close with my code in the sense that I had to assign an image to the detected marker, but just wasn't calling the function correctly. After seeing him it worked perfectly and each time a numbered marker was detected the corresponding image I set up would display to that marker alone! Next job was to replace the external image with web images, fortunately the BitMapFile material of PaperVision3D used to display the extenal images supports http urls so it was just a case of replacing the file paths with urls.
So having no experience in using Augmented Reality i needed to find a good foundation to work from and build upon if I was to create a sucessful Augmented Reality Photo Album. Fortunately I found a few very useful tutorials about the basis of creating an AR application. I found a website called gotoandlearn.com this site is run by an Adobe flash developer Lee Brimelow who provides two AR tutorials to show the basics.
The first tutorial helped me to understand how to setup an action script project that would enable me to display a 3D cube on top of an AR marker and to create a webcam window as well as adding additional classes Papervision3D & FLARToolkit basically these two external libraries are what makes AR possible within action script. Papervision3D enables the creation and rendering of 3D objects directly from within Flash/action script without having to import basic 3D renders from another source. The FLARToolkit is what allows the Augmented Reality to occur and for the recognition of AR markers so that content can be associated with it.
I followed the first tutorial http://www.gotoandlearn.com/play?id=105 and this enabled me to learn how to setup AR and to add content in the form of a basic 3D cube. From here I found another tutorial that was alot more related to the area of content that my project was going to be based on. The tutorial again from gotoandlearn was an extended AR tutorial showing how to display a video on a flat 3D plane using an AR marker http://www.gotoandlearn.com/play?id=114 from this I was able to develop the code so instead of loading a video file it would allow me to load an external image file for display on the AR marker. Below is an screen shot of what I achieved.
The image moves with the symbol even when rotated so it acts in the same manner as a traditional photograph.
The next stage was to add more markers and more images to each marker to build a collection of images.
I had my first meeting back with Gianni after the Easter break and explained how the idea had changed from the public physical environment to the new Augmented Reality idea. I explained my idea to him and the idea of creating a printed book that would enable constantly updatable content to be displayed within it. Gianni seemed happy with this idea but suggested that rather setting the physical book in stone, in terms of printed symbols, why not create a photo album that allows the user to change the web content being displayed as well as being able to change the physical symbols so the book itself also becomes more interactive. The idea to make the AR photo album act like a real photo album that gives the user the choice to move AR symbols around in the album as well being able to completely remove markers that would in turn remove digital content from the physical environment with the option to add it again at a later time.
At this point of the meeting I had followed a number of tutorials regarding Augmented Reality to create using action script a method of creating and displaying a 3D cube on on AR marker. I had been successful in setting up the action script file so that it displays a webcam window and connects to the relevant technology to enable me to display and create AR material. I will post further info about the AR technology and my progression with using it.
So after deciding to change the current project from a using a physical environment and spoken words to create digital content in the form of tweets I have decided to stick with the idea of physical and digital interaction and came up with the idea of developing an augmented reality photo album.
The idea behind this project is to still take digital web content and display it in a physical form, in this case a printed book. I intend to use web images rather than tweets to create an online photo album, the likes of flickr, photo bucket etc, but be able to display and store the web content within a physical book that can be consumed and distributed in a traditional way.
I hope to print the book with a number of printed AR Makers that will constantly update depending on content being published to the web. This will enable a physical album to be constantly updated with current web content despite the fact the book itself is in finished form.
Augmented reality is a method that will allow the book to remain in its original form and be viewed via a webcam or camcorder to display the content within it. This also looks at the idea of unauthorised use of content and restricted use of content in terms of the physical book itself will feature black and white markers that really don't mean anything when viewed on its own, but place it under a webcam and view it via that it will reveal digital web content that can be interchangeable dependant on the current content published to the web.
Below is an example of a 3D model being projected onto an AR marker:
So after a couple of weeks of working on the current project idea, I have stumbled into some problems and major flaws in my idea. The major issue with the project is the unreliability of the aka.listen speech recognition object for max/MSP. The main issue with speech recognition is that it is not entirely developed enough to understand every word spoken and often takes numerous attempts for it to pick up a word, be it the intended word or not.
The other issue is the fact that specific words have to be assigned in the first place in order for any word to be recognised. I can't seem to find a solution to efficiently adding enough words to make the project autonomous so that it is not pre determined. There seems to be no solution after searching days for one to add an entire dictionary or similar to the speech recognition object it seems there is no real solution to this issue, so after much thought I've got no choice than to change or seriously adapt the project to make it more reliable and feasible in its creation.
So in my previous post I made reference to a number of digital to physical projects that involve web content of different kinds. I also commented on a project idea that involved creating digital content from a physical environment and that project idea is the following.
My idea is to create a physical installation that takes sounds (in particular speech) from an environment and transforms them into online digital content. The plan is to use speech from a public environment and transform that into online content via twitter. The context behind this idea is to use speech from a public environment and convert it to tweets, due to the idea of twitter being a public environment for thoughts and ideas.
I came across two patches for Max/MSP interactive software that would possibly help me to create a physical to digital project. The two patches are Twitter for Max/MSP and "aka.Listen". Twitter for max/MSP allows the user to directly tweet to twitter via the max/MSP environment. This would enables me to connect other ways to communicate with twitter using this max object. The aka.listen object is a speech recognition application that uses Mac OSX's speech command system to enable the user to use voice control within max/MSP.
So far I have successfully managed to connect the two objects so that when a user speaks it records what they say and adds the speech to text recognition into a text box ready to be sent to Twitter. I need to explore these further as at the moment I have to assign specific keywords to the aka.listen object in order for words to be recognised by voice, it does not automatically detect spoken words but only specified words or phrases so it seems. This could cause some issues with the project plan due to the fact I want it to be unrestricted in terms of content production and not be pre-determined as this would defeat the object of the project itself.
My next step is to explore whether there is another way for the speech recognition to detect spoken words with out having to input specific phrases, failing that I may try and find a way to add an entire dictionary to the patch so that it automatically loads all available words within the english language...
Here are a few more digital to physical (and vice versa) projects. The first was a project referred to me by Gianni called 'Listening Post'. This project is another example of a digital to physical art project. It works by displaying both visually and audibly, content from the web that was posted seconds before being displayed. It is passed through a variety of filtering mechanisms and they appear on 231 text displays and a sound system.
This project enables digital web content to displayed both visually and audibly into a physical environment. The Video Below demonstrates how the project works fully and how it displays the content.
This project as well as the projects in the previous post gave me the idea of using twitter to create a physical to digital project that holds the context of creating digital content using the surrounding physical environment and by using in particular public environments to create the public content category that twitter and tweets fall into. The idea of tweeting being a public act for all to see and to involve with on the web could involve the idea of using a public physical environment to create digital public content such as twitter tweets.
So I had another meeting with Gianni about project progression and ideas. He felt that the idea of a project involving autonomous art controlled by twitter didn't really have a strong context in terms of what the project is trying to achieve and wasn't really focused enough.
I shared a few more ideas with Gianni that were different to the autonomous art project that involved bringing digital content into a physical element. All of which still involved using Twitter as a source of user generated material.
The References I showed Gianni all involved using twitters digital presence and bringing that from the web into a physical form. I found a blog containing three pages of twitter art projects that bring twitter from digital to physical. the blogs are found at:
The projects that stood out from the three pages of blog posts are the following:
The first is the 'Chalkbot' a twitter art project that is part of Nikes Tour de France - Lance Armstrong campaign. It featured a huge road printer that would print users tweets regarding the race at points on the road itself. The project according to Nike features a "pneumatic robot and software system. The system includes a text message interface, web based queue and approval system for tour officials, onboard machine and nozzle control, spray mechanism, camera and GPS capture system, and Twitter integration." This project brings digital content and takes it from a virtual environment and transforms it into a physical print. Below is a clip of how the Chalkbot works. This is a great example of publicly displaying digital content in a physical approach on a large scale.
The next project of interest was an art installation that audibly displays current tweets containing the word art though a selection of toy robots that were customised using text to speech software. This is another example of bringing web based content into a physical environment.
This project involves users tweeting to twitter via a computer and then having them printed out on small label strips. Once the tweets are sent to print, they are removed from the web and only exist in the physical arena. The user gets a message on twitter stating that their tweet has successfully been sent to the physical world and removed from the digital environment.
These are just three great examples of digital to physical web content projects, be sure to check out the three web links at the beginning of this post to see the rest of the projects featuring twitter.
Just a couple of references suggested to me from the previous group meeting, some relevant projects here.... Life Writer is an art installation created by Christa Sommerer and Laurent Mignonneau.
The project is an interactive art installation which consists of an old fashion type writer table and chair and a paper reel attached to the type writer which acts as a projection screen. Users of the installation type in the conventional way on the type writer and the output on the paper projector is a collection of virtual insects that behave and move depending on the inputted text from the user. The letters and words inputted act as a genetic code for the insects.
The insects will eat letters to stay alive and also they reproduce. Their movement speed is also determined by the input text. The user can also interact with the outputted insects by altering the typewriters paper scroll. Below are a few pics and Clip of the Life Writer.
The second reference is entitled "Biomorphic Type" and is a project that uses human responses such including heartrate, breathing, viscosity of the eye, brain activity and muscle activity to enable a typographic that responds to these senses and manipulates the appearance of the type when a user is inputting text whilst connected to the sensory equipment.
There isn't a massive amount of web coverage for this project but some info can be found here:
Just to update we had our first group meeting today with Gianni and shared current project progress and ideas with each other. Some great ideas shared and likewise useful feedback for additions and changes to current ideas.
Suggestions made to me about my current idea involving using twitter as a form of generative art production tool were a number of ideas that could be used to determine words, phrases etc from twitter to make an art project from. Suggestions such as using the graphic visualisation tool Processing to pull in twitter feeds that could in turn be visualised randomly in one manner or another. Another was to look at the length of words being used rather than specific words or letters as a process of deciding how art forms will be made.
A range of references were suggested also that could be useful in the production and progression of ideas and development of my project. I will post those in a separate post....
After I met with Gianni and had time to look over some of the references and advice he had given me, I decided that creating a web based authoring tool and social network for the collaboration of art was going to be a tall order.
I had the idea of still using a social network based website that would allow users to interact and collaborate but I needed to refine the process of production and the reasoning behind it.
The idea is to create a social network based website that uses twitter feeds or similar to create autonomous art. Basically it would work in the sense that users would be able to log into my website using their existing twitter details and from within the website be able to 'tweet' and write updates, but instead of posting them in the normal sense, art would be created from the combination of words or letters that the user types and submits. The art would be created using software called 'Quartz Composer' by Apple. This software allows the creation of animations, graphics etc that can respond to real time data such as audio feeds, xml feeds etc. It is used in the creation of iTunes Visualisers and screensavers.
This process of production would be autonomous in the sense that peoples written tweets would determine the outcome of the art being displayed, but could potentially effect the process in which users tweet and may even write tweets to gain a specific outcome.
During the meeting with Gianni he suggested I looked at a selection of references related to my area of exploration. These included a range of art project websites to specific projects and installations.
The first reference suggested was a website http://www.rhizome.org/ this site contains a vast amount of collaborative projects and art projects that could possibly link to my project idea.
Secondly he suggested checking out former iDat student Adrian Wards Auto Illustrator. A graphic tool application that according to the website signwave:
Auto-Illustrator is an experimental, semi-autonomous, generative software artwork and a fully functional vector graphic design application to sit alongside your existing professional graphic design utilities.
Use it to explore a wide range of generative and procedural techniques in the production of your own graphic designs. Discover how easy it is to produce complex designs in an exciting and challenging environment that questions how contemporary software should behave.
This project shows relevance in the area of my project due to its process of outcome being generative and undetermined. This follows on from what Gianni suggested in the meeting to explore, the area of process and whether I want the outcome of my project to be random or be pre-determined.
The final reference is a project entitled Poetic Generator. http://www.poietic-generator.net/ The idea is a collaborative web based project in which users will be allocated a slot in which they provide art work upon a 20 x 20 pixel window to be randomly added to a number of other users to create an autonomous collaborative art work.
So after submitting my project proposal I had my first meeting with Gianni. We spoke about what I was wanting to achieve from my project and we both decided that what my initial proposal could be too ambitious in terms of the web based authoring tool for the creation of art. However Gianni went on give me helpful advice in terms of the process of creation. He suggested I think about the process of the project and whether I wanted the creation process to be
Random Outcome that allows the user the freedom to create without boundaries
Pre-determined outcome that restricts the user to follow rules in order to create a specific outcome.
As well as this Gianni suggested I studied a number of key references as well as the idea of Collaborative Practice and Authorship essentially the key context for the project.
From this session I will go on to investigate alternative more realistic methods of producing user generative and collaborative content for the project.
