I'm long overdue for another official post about the status of the LogicMail 2.0 project. So far, most of my commentary has been scattered between the forums, ticket system, mailing lists, and private E-Mails. As such, I think its about time I explained where everything currently stands.

As many of you have observed, I've been working towards a 2.0 release for far too long. There are really three main reasons for this:

• 2.0 really is significantly more complex than 1.x
  _{To put things in perspective, using the SLOC metric (as calculated by this tool, the final 1.1 release comes in at 13,579 SLOC. Meanwhile, as of today, trunk is at 31,427 SLOC and growing.}
• I do this as part-time adventure
  _{A few hours here and there is plenty for following up on E-Mails, fixing minor bugs, or adding small features. However, the bigger stuff requires me to be able to dedicate whole days of development time. While I can do this on occasion, I am limited in how often I get the chance.}
• As a direct side-effect of taking so long, the following additional things tend to happen:
  • More time is spent dealing with end-user issues and requests
  • New BlackBerry platform features come out that I want to take advantage of

Despite everything mentioned above, I am optimistic about being able to finally get 2.0 to the beta stage some time this year. The end is in sight, and I actually know what else needs to get done. In short, the following are my must-haves for 2.0 being "feature complete":

• Various performance and memory management enhancements (connection read/write #221, IMAP parsing #222, cache loading #213, memory management #212)
• Incremental loading of mailbox data (#211)
• Running in the background and auto-fetching new data (#79, #78)

Anyone who follows the development builds has probably noticed some of the slush after build 199 and hopefully ending in build 202. This is where I implemented #221. It has made actual network loading much faster, but has had the side effect of making the post-processing seem slower. Hopefully #222 (a work in progress) should fix this.

Now, we all really want 2.0 to actually get to a beta release sometime this year. As far as I'm concerned, the beta milestone is where the application is feature-complete, and most of the more major bugs are fixed (and/or well documented). In order to get there, I really need to focus on the core features. This may mean that every little user-request or issue won't get acted upon immediately. However, if they are documented in well-written tickets, they can still be mostly addressed prior to a final 2.0 release (or moved over to 2.1).

If you follow any of the usual enthusiast and official websites, its hard to miss what's happened this past week. Amongst all the buzz, they also finally remembered to release an actual SDK to go along with it.

So what did this mean for LogicMail?

• I wanted a BlackBerry 6 build, to take immediate advantage of some new APIs:
  • StandardTitleBar
  • ToolbarManager
• The number of existing LogicMail builds was growing out of control.
  • I already had separate builds for OS 4.2, 4.6, 4.7, and 5.0
  • I also had separate touchscreen builds for 4.7 and 5.0.
  • The first 6.0 device is a hybrid slider, making it unclear what to do

So I started thinking to myself... Why do I actually have separate touchscreen and non-touchscreen builds? What am I actually saving by doing that? Apparently, only about 32kb! In an era where other developers can't figure out how to write a toy app in less than 1M, and I'm only in the 300-400kb range, it seemed silly to fuss over that little memory.

If you now look at the main page of this site, you'll notice that I've done the following:

• Combined the touchscreen and non-touchscreen builds (which icons or fields to use will now just be decided at runtime)
• Added an OS 6.0 build, using those new APIs

Of course its best to see screenshots:

I just spent the better part of a week at the BlackBerry Developer's Conference, and had an overall good experience. Had good times hanging out with Ronen from BerryReview, Jessica from CNET, Simon from IntoMobile, Monica from BlackBerry Motion, and even got to see the infamous Crackberry Kevin! Of course I also met up with Eddie from Weatherbug and Travis from GWAVA, both of whom I first met at last year's conference.

A lot of exciting things are going on with the BlackBerry platform, many of which are continuations of what we started to hear about last year. As a developer, one of the most interesting items is the completely rewritten Eclipse Plug-in 1.1 that is now in beta. The original plug-in was considered to be a forceful mashing of the JDE and Eclipse, and did have its quirks. The new plug-in has been rewritten from the ground up, and integrates much better into the whole Eclipse framework. I've already installed it, of course, and will soon be migrating the LogicMail trunk over to using it. I recommend checking out the full list of Tools Announcements from the conference.

Many new features are going into BlackBerry OS 5.0, and I fully intend to embrace some of them. (Don't worry, I'll still jump through the hoops to support 4.2) Some of the more notable features include:

• Network API - This fixes the #1 gripe of us developers, and will help magically solve the "make LogicMail auto-detect which network connection to use" problem. I might still have to solve it manually for older devices, though.
• FilePicker API - This will provide a more polished alternative to my current attempt at the same thing, for saving/loading attachments
• BrowserField API version 2 - This will provide a clean and easy to use mechanism for displaying HTML content. The current Browser Field is an abomination of a hack to get working, and I'm looking forward to having a cleaner option for newer devices.
• AutoCompleteField API - I'm sure I'll find some place to use this, possible for choosing message recipients, depending on how it works.

While they weren't giving out free phones anymore, at least this year's presentations came with a lot more sample code. I'm drooling at getting back to the code now, and also eagerly awaiting the BlackBerry Bold 9700 coming to my carrier (T-Mobile) on Monday.

Now I just need to find an unlocked BlackBerry Storm 9550 for testing, to cover all the bases...

As many users want support for the BlackBerry Storm to be useful, I've been working on improving the UI for touch-screen devices. While I still have a long way to go, I've finally put all the necessary hooks into the code itself to make actually doing all of this possible. So, here are some pretty pictures, before I discuss some of the gory details:

Quick question: Has anyone actually seen an example of a tree widget that actually works well on a touch-screen device? Particularly one like the LogicMail folder tree, where each node can be expanded/collapsed as well as clicked?

One of my constant side-projects that has finally come to fruition is support for building LogicMail against multiple versions of the BlackBerry API. The challenge is keeping a single code-base, while continuing to implement new features that become available. The initial drive for this, of course, is offering a better UI on the BlackBerry Storm (OS 4.7 and up).

Towards achieving this goal, two approaches were initially investigated: using the preprocessor in RAPC to separate chunks of code, or using a fancier approach based on multiple libraries and class inheritance. While tempting up front, I wound up disregarding the preprocessor approach due to its sheer inelegance and the quirkiness of Eclipse IDE support for it. So what I've implemented is a variation on the library approach. I separate out the code for different API versions when working in Eclipse, but merge it all together in multiple full-up builds when actually using Ant to compile LogicMail on the build server. Got it? :-)

My first implementation of this approach involved some of the UI screens. I wanted to offer specific functionality on touch-screen devices when necessary, while maintaining as much common code as possible with the base implementation. Of course the API to even find out if you have a touchscreen doesn't exist until 4.7, so that posed a bit of a challenge. What I essentially did was completely refactor how screens are created, from an inheritance-based approach to a composition-based one. This had the effect of separating the screen implementations from the RIM API. A somewhat confusing UML diagram attempts to explain this:

P.S. I've also updated the development page with instructions on how to use this new structure when working with the source code.

As some of you may have noticed, I now have a second automated build of LogicMail up and running. On the main page, its referred to as the "Maintenance branch," and it reports its version as LogicMail 1.1. By now you may be wondering what this is, and how it relates to LogicMail 1.0.x and LogicMail 2.0 (a.k.a. "trunk" or 1.9). Well, I realized that 2.0 is still a ways away from being ready, and that I was starting to get more contributions for 1.0.x.

The build process for 1.0.x was a bit of a pain, and debugging it was even harder, mainly because it was configured using this hack for Netbeans 5.5. Now that RIM actually has a usable Eclipse plug-in, life has gotten a lot easier.

So, what exactly is the LogicMail 1.1 branch, you may ask? Well, I basically took the 1.0 branch, and did the following:

• Updated the project structure and build files to closely match 2.0 (trunk)
  • Development is now in Eclipse, with the same setup instructions as for 2.0 (trunk)
  • Simple builds through Ant is now possible
• Configured the Bitten automated build system on this site to build it whenever updated
• Began backporting the internationalization updates from 2.0 (trunk)

Going forward, the goals for 1.1 are simple:

• Full internationalization support
• Integration of any contributed simple features
• Occasional backport of simple things from 2.0
• Continued bug fixes
• Regularly available automated builds, avoiding the need for formal releases
• Keep everyone happy, while major work continues to target 2.0 (trunk)

One of the best (or worst) things about developing an E-Mail application is dealing with internationalization issues. The early Internet protocols tend to assume that 7-bit ASCII US English text is the default, and everything else is the exceptional case. To this end, there have been many character sets and encoding formats created over the years. Today, we've at least managed to consolidate all the character sets into a standard known as Unicode.

Now the BlackBerry API only claims to support the following character sets:

• ISO-8859-1
• UTF-8 (Unicode)
• UTF-16BE (Unicode)
• US-ASCII

In addition, it has been discovered that East Asia localized BlackBerry devices also support GB2312.

To make matters more interesting, Vietnamese characters will only display correctly if they are represented as decomposed Unicode characters. Meanwhile, most Vietnamese text on the Internet is represented as canonically composed Unicode characters.

Users of LogicMail have occasionally run into issues where their non-English E-Mails do not render correctly. The most recent example involved a message that used the CP1251 character set for Cyrillic. However, I suspect that other similar issues have been noticed and have gone unreported.

Assuming you haven't fallen asleep yet, you are probably just wondering whether I can just wave my hand over the code and make your E-Mail render legibly. Well, I can, but there are some drawbacks. Basically, I need to include two things in the LogicMail code:

1. Unicode translation tables for all unsupported character sets
2. Unicode compatibility decomposition normalization (NFKD) tables

Believe it or not, Java SE 6 already includes libraries for this. But, the BlackBerry API does not. Including this in its complete form would dramatically increase the size of the LogicMail application, most likely by an unreasonable amount.

So what do I want to know from you? What unsupported character sets do you all normally receive E-Mails written in? If I can scope the effort to just a few translation tables, the size penalty may be acceptable. (If you don't know where to look, check the raw message headers.)

Yes, that's right. I've finally gotten automated builds working again. This time I've decided to use the Bitten continuous integration system, which better integrates with Trac.

It took a lot of frustrating work, spending too much time playing with Apache Ant build configurations, using BlackBerry Ant Tools, upgrading the Python installation on the server, and upgrading Trac itself for the heck of it, but its now all fully functional!

The links for downloading builds are on the main wiki page as before. The actual build details can be found on the Timeline and Build Status tabs above.

Now some of you are probably eager to go ahead and give it a try. Well, I'll warn you that its still not really ready for production use. At this point, I'd recommend only running it on a BlackBerry Simulator. If you want to use a real device, I suggest a spare or secondary unit.

To actually track the progress towards 2.0, I've created the LogicMail 2.0 Progress page. I'll eventually scrub out all the relevant tickets, but for now this will serve as a more convenient checklist of sorts.

As a developer, I constantly have to ask myself this question. LogicMail 1.0.x currently supports 4.0 and 4.1. Most shipping devices run 4.2 or 4.3, the latest version that actually works on most of them is 4.5, and the newest BlackBerry models run 4.6 (Bold) and 4.7 (Storm).

Collecting statistics from the actual application is impossible, since I really don't want LogicMail to ever "phone home." Collecting statistics from the SourceForge download page isn't useful either, since it only offers 4.0 and 4.1. Also, I suspect a large number of people are accidentally downloading the 4.0 build. However, one place I actually can collect statistics from is the OTA download page. Ever since I moved that from SourceForge to my own web server, I gained the ability to do data mining on the log files. Thanks to the User-Agent string on every hit to "LogicMail.jad," I now have some useful numbers:

(Collected between 12-OCT-2008 and 20-NOV-2008)

From this its obvious that very few people still run 4.1, and almost no one runs 4.0. Also, a few Storm users have tried it. I've been considering making 4.2 the minimum version for LogicMail 2.0, and this data validates that thought. Making 4.2 the base version allows me to depend on 4.2-and-up capabilities for some core features, like using the FileConnection APIs for local mail storage. I'm also considering offering additional builds for newer versions that incorporate extra features, such as SVG images and field borders that are now supported in 4.6.

One of the big features of LogicMail 2.0 will be internationalization. That means I'd like the user interface to work in more than just US English. However, I'm personally only fluent in US English, so I'm going to need help with this one. I know I have a lot of users all over the world, so it shouldn't be too hard to find a few of you to help out with this aspect of the project. So what do you need to help?

• Be familiar with the technical terms used in E-Mail clients between US English and your native language
• Be able to edit a text file in Unicode
  • Being able to build and run LogicMail in the BlackBerry simulator is also useful, but not strictly necessary
  • If you're curious, this is the actual file you will need to translate
• Be willing to revisit your translation as I add features and need more text translated

So if you're interested and willing, please write me at octo@logicprobe.org and we'll get started.

I've just released LogicMail 1.0.3, which is another bug fix release along the 1.0 branch. The fixed bugs include two IMAP server compatibility issues (#113, #114), and one memory issue (#115) I discovered as a result of some things I learned at the recent conference. For every good bug report I get, several poorly described and impossible to reproduce bug reports are written, so I hope that many of those issues are magically covered under the things I've fixed in this release.

As I'm now back home and settled from the conference, I've started integrating some of the changes I made to the trunk on my laptop. I've basically been reorganizing the source tree to better fit with how RIM's Eclipse plugin wants to look at your project. As such, the code on the trunk now has the following top-level structure:

This new organization is primarily designed to make the RIM Eclipse plugin happy. The plugin is currently not very flexible on how you organize your project, and is easily confused. (I really don't think RIM even yet tested it on projects whose source is littered with hidden source-control support directories.) In order to make all the application code available to the unit-testing project, I had to basically make everything but the main() method into a library project. Also, so it wouldn't throw tons of unused-method warnings on the J2MEUnit itself, I had to move that into its own library project.

One thing to keep in mind is that all of this complexity is only to support working and debugging in the development IDE. I intend to keep actual official builds fully ant-based, and ant gives me far more flexibility. When building through ant, the only generated artifacts will be a complete "LogicMail" CLDC Application, and a complete "LogicMailTests" CLDC Application.

The BlackBerry Developer Conference just ended yesterday afternoon, and I have to say I'm extremely glad that I went. It was an excellent conference all around, even if it was the first one. I learned a ton of useful information about development on the BlackBerry platform, made some good connections, and got a free BlackBerry Bold!

It'll take a good week or two to finish digesting all the information I learned, review all the conference proceedings, and follow up on various things. However, right now I'm already thinking of a few interesting tidbits that should directly benefit LogicMail:

• The ALX format can let me build a single download for the desktop application loader that can include builds for multiple BlackBerry OS versions.
• The new Eclipse Plug-In is greatly improved, and I now know how to use it to profile for memory and performance issues.
• They're now supporting a preprocessor in the build tools which will make it much easier to support newer BlackBerry features and devices from a single code base.

Here's the presentation I gave at the conference, and an older presentation I gave about LogicMail itself at the local Linux User's Groups:

The upcoming BlackBerry Developer Conference begins this Monday, and I'm going to be there in person. It should be a great opportunity to network with the rest of the BlackBerry software world, meet a lot of interesting people, and learn a lot of useful things.

I should also mention that I'm actually going to be presenting at the conference, so look for the following in the session catalog:

If any of you will be there, and would like to run into me in person, just look for the badge! Of course, I'll also be regularly checking my E-Mail.

P.S. I finally put my real name on the main page, since a few too many of you were calling me by my randomly chosen online nick.

I know I've been fairly quiet these past few months on discussion of actual progress on 2.0, and really haven't even been updating related tickets on this website. However, I've actually been making significant progress on the actual implementation. Since far too many highly desired features have been pushed off into the 2.0 plan, I feel its only fair to keep everyone updated on where things are going.

To give you all an idea of just how big of a change 2.0 will be, I'll start by explaining just how simple 1.0 really is. LogicMail 1.0 is basically a UI coupled almost directly to a collection of mail (IMAP/POP/SMTP) clients. There isn't much of a data model (beyond what's necessary for the pieces to talk to each other), and there certainly aren't any persistent program threads running in the background when you're not downloading something. Now don't get me wrong, developing and testing that mail client code has been a lot of work. E-Mail really is a royal pain, between different server behaviors, far too many data encoding formats, complex message structure, etc. However, what you all really want is a full-featured mail-handling application. With 2.0, I intend to give that to you.

The 2.0 project all started with a million variations on the simple feature request... "How do I get LogicMail to notify me when I get a new message? It should be pretty simple to implement, right?" Now that you've read the preceding paragraph, it should be obvious that it really isn't that simple. Just take a look at the design page to get an idea of how I began to flesh out my ideas for how LogicMail should be built. If you've read a famous book by Fred Brooks on software engineering, this is probably already starting to look like the Second System Effect

Okay, now to let you know what I've actually done thus far... (Yes, I'll try to give the brief version of what's in my notebook.)

• Retooled the build process to be fully Ant-based and not IDE-dependent. (and switched to Eclipse as the preferred IDE)
• Implemented a fully threaded, request-driven, asynchronous mail protocol engine to sit on top of the IMAP/POP/SMTP client code
• Implemented a complete mail object model to sit at the core of the application, providing a user-visible object based request/event driven facade on top of the mail protocol engine.
• Reimplemented most of the 1.0 functionality on the new 2.0 architecture
• Did initial implementation of IMAP IDLE!!! (currently only works for the last-selected folder, and doesn't do keepalives yet, but its a start)
• Did the draft implementation of a new account setup wizard, to hopefully ease the currently awkward setup process.

In terms of code-size, SLOCCount shows the 2.0 trunk to be approximately 5,000 SLOC larger than the 1.0 branch.

Of course at this point, if you want to try 2.0, the best way is to build from source and run it in a BlackBerry Simulator. Its far from ready for prime-time, but I'll eventually get to the point where I start offering up technology-preview releases. The code is still littered with "TODO" comments, and error handling/recovery gets a lot more complicated on the 2.0 architecture.

Now for some screenshots:

New home screen	Account setup wizard	Folder selection dialog

Welcome to the LogicMail Development Blog

Since everyone else is blogging these days, and since I really should be keeping all of you better informed of what I'm up to, I've decided to add a blog to this website.

Assuming I regularly update it, this blog should give me a way to keep you all posted on what I'm up to with LogicMail development. It should also give you a good way to talk back and start a discussion on anything I post. I know we already have a Discussion section on this site, and that should still be everyone's primary outlet for help requests and general project discussions. This blog will supplement that by letting me more regularly talk back to the community and provide my own updates.