The end is nigh...

OK, so the official Summer of Code coding period has ended, so what better time to sum up the status of my project and the general experience I have had.

Project Status

The last two weeks have been pretty hectic, as I have been trying to make my finished code as complete and polished as possible. Generally, I feel like all the required functionality of the virtual keyboard and keymapper is there, but I am sure that there will still be bugs hiding in there. Here is a summary of the the work I achieved in the last couple weeks:

Virtual Keyboard:

• New special events - added support for submit, cancel, delete, clear, and cursor movement, to add further control to VK users.
• Added support to preview all keypresses - means that the VK display will show all key presses (not just those with ASCII codes)
• Proper initialisation - throughout development I had been loading the VK pack the first time the VK was run, because the EventManager was constructed before we knew what screen resolution we were running on. To get round this I added an EventManager::init() method (where the VK pack is loaded), and called it from scummvm_main after the screen had been initialised.
• Tons of memory-related bug-fixes (thanks to Valgrind!) as well as optimization and cleaning up of drawing code.

Keymapper:

• Remap GUI dialog - which allowed keymaps to be re-configured
• Stack-based active keymap system - taking inspiration from the CursorManager I realized it made a lot of sense to push and pop keymaps off a stack of active keymaps. Not only would it make it more intuitive for keymap switching, but also gave us the option to inherit key mappings from keymaps lower in the active stack - a very powerful and flexible option.
• Parent keymap feature - linking into the stack-based keymap selection I made it an option for a Keymap to have a parent keymap. This would allow the automatic mapping algorithm to be more intelligent in assigning keys, considering that the keymap would usually inherit actions from its parent when it was active.
• HardwareKey types - I realized that in many situations it made sense to say this key is a "shoulder button", and that this action should be assigned to a shoulder button. So now HardwareKey's have a type and a preferred action, whereas Action's have a type and preferred key. Again, this provides us with a lot more flexibility!
• Refactoring - got rid of redundant KeymapManager class and put its core functionality (saving, loading & automatic mapping) into the Keymap class

The one major thing that I simply did not have time for was proper documentation of all the virtual keyboard and keymapping features. This is something that I will work on over the next few weeks (at a somewhat more leisurely pace! :). I feel that overall I have achieved what was set out for me at the start of the project, and hope that I can do the required work to get my code merged into the trunk sometime soon.

My GSoC experience

Firstly, I would like to say that I have thoroughly enjoyed my summer working with the ScummVM team. I will be the first to say that we didn't exactly get off on the right foot, and that my participation has been a bit 'patchy', but after all is said and done I hope that I have made a valuable contribution to this project! Has been great having Joost as a mentor, was always there to answer my questions and encourage my progress. Was also very helpful to hear Eugene and Max's opinions when I was unsure about my project's direction. The only regret I have is that I feel that I could have maybe pushed myself to acheive more if I had gone for a more ambitious project, but then again I would not have wanted it to all end in failure! At the end of the day, I think the GSoC is a wonderful experience and will be returning next year, maybe even for ScummVM (if you'll take me! ;) So, thats it for now. Over the coming weeks I will write proper documentation, and continue working towards making my code stable enough for ScummVM. Thank you, and good night!

Latest developments

Time for another quick update on my progress recently. Generally, I think I am still on track with my plan of action that I set out a couple weeks ago.

Virtual Keyboard:

• Major refactoring of GUI code into a separate VirtualKeyboardGUI class. This meant VirtualKeyboard class only handled the detection of keypresses and the queuing of resulting events.
  • This abstraction should make it easier to modify VK to use ScummVM's GUI rather than being responsible for its own drawing like present.
• Display functionality complete - now a virtual keyboard pack can specify a rectangular area that will be used to display the virtual key presses that have occured.
  • This was implemented by firstly creating a KeyPressQueue support class that not only kept track of the queue of KeyPress's that the VK was generating, but also maintained a string representation of these.
  • Then I added methods to the VirtualKeyboardGUI class to draw this String in the required area. I had to include textbox-like functionality e.g. handling when the string was too big for the area, drawing a caret, etc.

Keymapper:

• Saving/loading of keymaps to config file is implemented (but not tested extensively and saving of keymaps is not integrated yet)
• Simple automatic mapping algorithm has been implemented
• Keymapper class now uses a stack-based implementation for switching between active keymaps.
  • This will make it easier for switching as we can push and pop keymaps onto the active stack (like ScummVM's CursorManager does)
  • The pushKeymap() method also takes an inherit boolean argument which says whether the Keymapper should use iterate down the active Keymap stack when a mapping for the given key is not found. This should be very powerful feature because it could allow the new keymap to only remap one or two keys, and get the rest of its mappings from the last active map.
• Started proper testing by integrating Keymapper into SDL backend. Just added method that sets up a simple HardwareKeySet and a simple Keymap.

Next steps:

• Develop the automatic mapping algorithm, considering the new stack-based implementation of the keymapper. My current thoughts:
  • The automatic mapping algorithm should look at the maps that have already been added, so that it can use HardwareKeys that either have not been mapped yet, or at least have been mapped to a Action with low priority.
  • This could be further improved by defining which keymaps will be parents of other keymaps. Then we only have to check the mappings of the parent Keymaps.
  • In fact, the idea of keymaps having parents, and therefore being arranged in a tree structure, does seem like a good idea.
• More extensive testing of these new features
• Keymap dialog

So, finger's crosses, I should have most of this done by the weekend. That leaves next week to really finish and clean up the code, test thoroughly and write documentation. OK...back to work!

Keymapper coming along nicely...

Thought it was a good time for a quick update. I've made a lot of progress on the keymapper since my last update, with most of the infrastructure in place now. Most of the interface to the rest of the system is implemented (in the Keymapper class). The KeymapManager (where the loading/saving and automatic mapping will be done) still needs work, and will probably be quite tricky! Anyway, here is a summary of my major progress and any design decisions I have made since my last post:

• KeymapManager now uses Domain classes to separate out global and game-specific keymaps - this was inspired by ScummVM's ConfigManager. Each domain has a default keymap, which incorporates the idea of a "super-global" keymap. Also, this means that a game can provide a specialised version of a global keymap, because keymaps are only requested by name (not domain) and if a keymap of the given name exists in the game-specific domain it is given priority over a global keymap with that name.
• Keymapper has initGame and cleanupGame methods. The idea here is that these methods need to be called before and after a new game is run. In initGame the keymapper looks at the ConfigManager's active domain (to obtain the gameid). In cleanupGame the keymapper tells the KeymapManager to free up all the game specific keymaps in has loaded. This should provide support for when we have to return to launcher feature.
  
• Where it comes to actually adding keymaps to the Keymapper we have addGlobalKeyMap(name, map) and addGameKeyMap(name, map). All these methods do is register the new Keymap with the KeymapManager (which is where the loading / automatic mapping of the keymap will occur).
• So the idea is that whenever part of the system needs a specific keymap they will call one of these methods in their initialisation code. Then when they actually want that keymapping to be used they call switchMapping(name).
• The actual mapping of the keys is done (see Keymapper::mapKey). The key point here is that both KEYUP and KEYDOWN events are mapped, and the mapped event is modified to make sure it correlates with the type of event that prompted it.
• The instantiation of the Keymapper and the calling of mapKey has been integrated into the DefaultEventManager.
  

So I am well on track with my schedule at the moment. For the next few days I will go back to working on the Virtual Keyboard, as there are a couple of features that are still needed.

One last thing...thought I'd better let you know that I'm going to stay with my gran up in Scotland from tomorrow till Monday, which means I probably won't be online much. But, I'll be taking my laptop and should be able to work without it.

More boring planning...

So its been a while since my last update, and for most of the time I've been busy planning the rest of my project. I'll get on to that later, but first let me sum up exactly where I got to with my coding.

Virtual Keyboard

• Dragging feature
  • made it possible to drag the keyboard around the screen when part of it that was not a key was clicked
  • also made it snap to the sides of the screen
• Fixed bug in event delivery
  • found that when many key presses were delivered after keyboard was shut the SCUMM game that I was testing with would ignore the first few keypresses and only register the final one
  • fixed this by implementing a delay in the event manager's delivery of queued events
• Handling of GUI and VK displaying at same time
  • before I would unconditionally clear the overlay before displaying the keyboard, but if the GUI had been displaying then bad things would happen once the keyboard was closed
  • to get round this, when the GUI was active I would save the overlays contents when the keyboard was opened
  • then this would be the background to blit the keyboard into, every time it had to be redrawn
  • and when the keyboard was hidden, the original overlay contents would be written back to the overlay
• Documentation
  • added comments to public interface of classes
  • added explanation of XML file format to VirtualKeyboardParser.h, which will later be added to wiki

After doing this, I started researching the keymapper task, so that I could get a better idea of all the stuff I have left to do. After reading up and asking questions on the mailing list I was ready to come up with a breakdown of my design.

Keymapper breakdown:

I chose to break down the keymapper task into four main classes that I would need to implement.

• Mapping class:
• will store details of a particular key mapping and offer simple methods to carry out the mapping
• will also provide methods to edit the mapping (which will be called by the Edit Mapping GUI dialog)
• Keymapper class
• will provide interface to rest of system, and will be accessible via EventManager
• will store a pointer to the active Mapping class
• and other state such as the active gameid, etc...
• will provide method to switch to a different Mapping (which will be called by engines or other parts of system, e.g. main menu dialog)
• engine will call methods on game start that will allow it to specify game-specific Mappings (and the UserActions they require)
• backend will call methods (or possibly subclass?) that will allow it to specify which HardwareKey's are available
• MappingManager class:
• will be responsible for managing all the different mappings that are available
• involves the loading and saving of mappings to the standard ini file
• will load global mappings on creation, and will load game-specific mappings when prompted by engine calling methods on Keymapper class
• if no valid game-specific mappings can be loaded from the ini file the automatic mapping algorithm will be run
• Edit Mapping GUI dialog:
• will be implemented using the existing GUI toolkit
• using the descriptions from the UserAction and HardwareKey structs will make the dialog easily understandable
• the will be a number of label / button pairs
• each label will contain the UserAction description text
• each button will contain the mapped HardwareKey description text
• clicking the button will start an event loop, and the first event that is recognised to have come from a particular HardwareKey will re-assign that HardwareKey to the UserAction in question

Now that I had a good understanding of the second part of my project I came up with a rough TODO list.

TODO list:

Virtual Keyboard

• prepare a complete virtual keyboard pack
• waiting on original images from rob (IRC: sanguineh)
• add textbox to show what user has entered - 2 days
• maybe a special part of the keyboard image could be marked with a rectangular imagemap area
• then this area will be filled with text as user clicks keys
• will then need left/right arrows and backspace keys to alter this text
• shift key functionality - 2-3 days
• currently the keyboard only offers functionality of a caps lock key because mode can be changed to caps but then must be switched back to normal
• should be easy to offer a key that switches the mode but only until another key is pressed, at which point the mode is switched back to the original mode
• this leads to a separate point - do we require the virtual keyboard to offer functionality of control / alt keys?
• if so, then shift functionality should probably be done without mode switching but by implementing modifier keys in the keyboard - I think this is the most flexible approach

Keymapper

• Mapping class - 1-2 days
• Keymapper class - 3-4 days
• MappingManager class - 4-5 days
• Edit Mapping GUI dialog - 2-3 days
• Integration into system for testing purposes - 2-3 days

The last part of my planning extravaganza was to plan how I would spend my time over the remaining weeks.

Plan of Action

I have decided to get going on the Keymapper class in the next few days, before hopefully coming back to finish the Virtual Keyboard once I have some good images available. 18th July - 23rd July

• Mapping class - complete
• Keymapper class - complete
• MappingManager class - basic outline done

24th July - 28rd July

• Finish remaining features of Virtual Keyboard

29th July - 8th August

• Bulk of the MappingManager class
• Integration with rest of system

8th August - 11th August

• Edit Mapping GUI Dialog

12 August onwards

• Cleanup and finishing off code
• Writing documentation
• Preparing for final evaluations

Done! Sorry for boring you guys with all my planning, but I guess this benefits us all! Time to get back to the code...

Progress Update

Time for a quick update on the huge progress I have made in the past few days.

Virtual Keyboard Parser

Thanks to Vicent's great work implementing the XMLParser base class this was a relatively simple task. There was a lot of error handling code to write, and I added a couple of new features to the base class. Apart from that it didn't take me long at all to have a parser that was parsing the file format that I had earlier settled upon.

Keyboard's main loop

This involved writing the event loop that would be run while the virtual keyboard was visible. It involved catching all the mouse click events, and passing them to the processClick() method. I also had to ensure the screen was being updated (so that the mouse cursor continued to move) and that EVENT_QUIT events were handled properly (so that closing the window had an effect).

Displaying the keyboard

The main challenge here was implementing keycolor transparency for the keyboard bitmap. To do this I created a subclass of Surface, SurfaceKeyColored, that provided a blit() method. This method would be passed a source Surface and a transparency color, and any pixels in the source Surface that was not the same color as the transparency color, would be copied into the Surface. Once I had this class, I could grab the overlay into an instance of SurfaceKeyColored, then blit the keyboard bitmap's Surface into it, before copying the SurfaceKeyColored's pixels back into the overlay. Below are some images demonstrating it in action:

Imagemap testing

After I had the bitmap's being displayed and the event loop implemented, I could test my image map code that I had written last week. The processClick() method of my VirtualKeyboard class simply had to pass the mouse co-ordinates to the corresponding ImageMap class and it would return the ID string (ie. the target attribute) of the matching area. I created a few image map areas for some keys of my test image in GIMP (making sure to use rectangle and polygon areas), then used the exported HTML in my keyboard pack XML file. After a couple of bugfixes I got the ImageMap working perfectly, proving that my Polygon::contains() method was correct.

Delivery of virtual events

I had written the VirtualKeyboard class to save all the key presses it created into a queue. But, once it was no longer visible it had to insert these key press events into the main event queue. I decided that the VirtualKeyboard should have a pollEvent() method, that would return an event if the keyboard was closed and there was a key press in the queue. Then all I had to do was modify the DefaultEventManager to first poll the VirtualKeyboard, and then poll the backend, but only if the VirtualKeyboard did not return an event. The DefaultEventManager class was also made responsible for showing the virtual keyboard when a special button press was made. While writing this code, it was decided that the VirtualKeyboard class should be instantiated from within the DefaultEventManager class, because they were very closely related.

Next...

Tomorrow I will try to create a complete keyboard pack, with multiple modes and resolutions, together with complete image maps, and event specifications. This will allow me to fully test my current code, as well as test the little bits of code that I am still to write.

XML file format

Before I start work on writing the parser for the virtual keyboard's XML file, I need to tie down the format of the file. Yesterday I came up with the following as my first idea:

<keyboard modes="normal,caps,symbols1,symbols2">
  <mode name="normal" resolutions="320x200, 640x480">
    <layout resolution="320x200" bitmap="normal_320x200.bmp">
      <map>
        <area shape="rect" coords="84,64,246,146">
          <event type="key" code="97" ascii="A" modifiers="shift" />
        </area>
        <area shape="poly" coords="2,12,25,9,34,8,19,22">
          <event type="mode_switch" name="caps" />
        </area>
        ...
      </map>
    </layout>
    <layout resolution="640x480" bitmap="normal_320x200.bmp">
      ...
    </layout>
  </mode>
  <mode name="caps" resolutions="320x200, 640x480">
    ...
  </mode>
</keyboard>

Here each mode tag refers to a different keyboard display, and each mode has a number of layouts - which provide different resolutions for each mode. Doing it this way - rather than having layout as the parent of mode - was a pretty arbitrary choice, with a slight advantage of being more flexible.

The main flaw I noticed here was that the event tags would have to be repeated for each different layout resolution. If we can assume that all different layout resolutions will contain the same events then we can do this:

<keyboard modes="normal,caps,symbols1,symbols2">
  <mode name="normal" resolutions="320x200, 640x480">
    <event name="A" type="key" code="97" ascii="A" modifiers="shift" />
    <event name="shift" type="mode_switch" name="caps" />
    ...
    <layout resolution="320x200" bitmap="normal_320x200.bmp">
      <map>
        <area shape="rect" coords="84,64,246,146" target="A" />
        <area shape="poly" coords="2,12,25,9,34,8,19,22" target="shift" />
        ...
      </map>
    </layout>
    <layout resolution="640x480" bitmap="normal_320x200.bmp">
      ...
    </layout>
  </mode>
  <mode name="caps" resolutions="320x200, 640x480">
   ...
  </mode>
</keyboard>

Where the target of each area refers to an event tag by its name property. This is nice because it means the map sections now use pure HTML IMGMAP syntax. This means the target property can be specified when using an image map tool, and then the output just pasted into the file without any additions.

Therefore I think this format will be the basis for my first attempt at the parser. However, any comments on it will be much appreciated!

Revised schedule

We are now in the 6th week of the Summer of Code and through a number of circumstances, both in and out of my control, I am pretty behind schedule. My coding in the last few days has shown great promise that I can quickly get back on top of things, but I think it is time for me to lay down a revised schedule for the coming months. The first stage of my schedule is pretty tight - but I have ample time at the moment and am willing to work all the hours in the day to make up for lost time.

Now - July 12th

• Write all the code for virtual keyboard
• ALREADY DONE
  • implementation of image map functionality (ie. polygon/rectangle intersection code)
  • Skeletal version of main virtual keyboard class
  • XML file format has been defined
• TO DO
  • XML parser - a case of extending Vicent's XML parser to parse my file format = 2 days
  • Keyboard main loop - to handle mouse clicks when keyboard visible and draw the keyboard to the overlay layer = 1-2 days
  • modification of event manager to receive events from keyboard and handle mode switching of keyboard = 2 days
  • integration of virtual keyboard into rest of system = 1-2 days
  • Other stuff - extracting from zip file & creation of example keyboard pack = 1-2 days

July 13th - July 14th

• Prepare mid-term evaluation material

July 15th - July 20th

• Clean up of existing code
• Make sure code is commented clearly
• Extensive testing of virtual keyboard

July 20th - July 25th

• Detailed research into keymapping task

The following is slightly vague, due to my lack of knowledge

July 25th - August 2nd

• Writing of general keymapper interface
• involves providing the link between engines and backends
• working out how each is going to tell the keymapper what it needs to know

August 2nd - August 9th

• Writing of algorithm to automatically map actions to buttons
• gui to allow user to change these mappings
• saving of these mappings to ini file

August 10th - August 20th

• Additional key mapping features

August 20th - Sept 1st

• prepare for final deadline
• Lots of testing, code cleanup and documentation

PS - progress report coming tomorrow (including details of XML file format)