My second blog

To content | To menu | To search

Sunday, April 20 2014

Shutting down my (now redundant) comm-central clone on github

This is just a public service announcement: I'm shutting down the comm-central repository on github I've been maintaining since 2011. Please use the official version: they share the same hashes so it's just a matter of modifying the url in your .git/config. Plus, the official repo has all the cool tags baked in.

(Turns out that due to yet-another hg repository corruption, the repo has been broken since early March. Since no one complained, I suspect no one's really using this anymore. I'm pretty sure the folks at Mozilla are much better at avoiding hg repository corruptions than I am, so here's the end of it!).

Friday, April 4 2014

Freedom of speech in the Mozilla Community

I read this morning Andrew Truong's blog on planet:

I guess it's okay to speak out about how we truly feel when somebody resigns over a controversial topic but not to speak out during the controversy? We should ALWAYS speak out. Freedom of Speech.

The reason why I didn't speak up is, just like many other Mozillians I suspect, for fear of retribution. Seeing the attacks perpetrated on Brendan, a well-respected member of the Mozilla community, I can only imagine the torrent of hatemail that I'm going to get for publishing this blog post. (Update: so far, I didn't get any hatemail. It seems like the fears were unjustified.) The other reason is, I didn't want to further encourage a debate which I hoped would fade off after a few days. I somehow hoped we could go back to "business as usual". We were unable to, and thus I see no reason to hold back this blog post anymore.

I live in Europe. We tend to have a different opinion on these matters. But in any case, before you go any further down, I should just mention that I do support the right for anyone to marry the one they love, and I voted according to that belief for the last few elections in my country. I am a straight person, though.

I'm going to quote a few blog posts that resonate with me, and add a few comments.

Daniel wrote:

Who said "Mozilla Community"? Who said Openness? Pfffff. I've been a Mozillian for fourteen years and I'm not even sure I still recognize myself in today's Mozilla Community. Well done guys, well done. What's the next step? 100% political correctness? Is it still possible to have a legally valid personal opinion while being at Mozilla and express it in public?

From private conversations I had with other (mostly European) Mozillians, I know for certain that people have opinions that they are afraid to express in the Mozilla Community. Some people are religious, and will take great care _not_ to reveal that fact. Some people may have other beliefs that do not align with the dominant, Silicon-Valley progressive ideology. They also make sure that these are not apparent. Andrew Truong mentions freedom of speech. I believe there is freedom of speech in the Mozilla community as long as you happen to have the right opinions.

In my personal case, I fortunately happen to side with the prevalent ideology for most points, but I am now very afraid of slipping and expressing an opinion that is not considered progressive enough. I am now afraid of what is going to happen to me then: will I be kicked out? Will people call out for my name being removed from about:credits? Will people call on Twitter for my being ousted from Mozillians.org?

Ben Moskowitz wrote:

For the record, I don’t believe Brendan Eich is a bigot. He’s stubborn, not hateful. He has an opinion. It’s certainly not my opinion, but it was the opinion of 52% of people who voted on Prop 8 just six years ago, and the world is changing fast.

Again, my European point of view on the matter is simple: I interact in my daily life with many people who do not support same-sex marriage. I'm pretty sure some bosses up my hierarchy do not support that. I believe that, ultimately, there will be less and less such people, and that society will change enough that being against same-sex marriage will be a thing of the past. I also believe that as long as my bosses do their jobs, I'm fine with that. I do not care about the personal life of my president; I just care for him to run the country according to the values that his party adheres to. Same thing goes for Brendan.

Christie, who in a much better position that I am to speak about that, says:

Certainly it would be problematic if Brendan’s behavior within Mozilla was explicitly discriminatory, or implicitly so in the form of repeated microagressions. I haven’t personally seen this (although to be clear, I was not part of Brendan’s reporting structure until today). To the contrary, over the years I have watched Brendan be an ally in many areas and bring clarity and leadership when needed. Furthermore, I trust the oversight Mozilla has in place in the form of our chairperson, Mitchell Baker, and our board of directors.

The way people demanded a public apology reminded me of the glorious times of Soviet Russia and Communist China. What next? Should Brendan be photoshopped out of all the pictures? If we leave the matter as is, the only reasonable thing left to do is to add an extra round of interviews when hiring people: the political interview. There, we should make sure that the people we hire share the "right" political opinion. Otherwise, it seems like they is no space for them in the Mozilla Community.

Andrew Sullivan wrote:

If this is the gay rights movement today – hounding our opponents with a fanaticism more like the religious right than anyone else – then count me out. If we are about intimidating the free speech of others, we are no better than the anti-gay bullies who came before us.

While being a strong supporter of the movement, I really feel sad today: what legitimacy is there now for people who've been dragging through the mud an opponent, instead of treating them like a decent, human being? Is this what Mozilla is about?

(Comments are disabled on this post.)

Monday, December 24 2012

So I figured out I should try out the Addon SDK

So I figured out I should try out the Addon SDK. Jeff Griffiths and Erik Vold finally convinced me at the latest MozCamp that it was worth a try, so for my latest project (a lightweight revival of FireGPG), I decided to go the Addon SDK route. I'm a Thunderbird addons developer, so although I'm very familiar with all the Gecko and Thunderbird parts, I'm not familiar with Firefox-specific extensions. I thought the Addon SDK would save me the trouble of finding out which parts of the UI to overlay, and so on.

The short version is that it's been a very pleasant experience, and I believe it did save me a lot of time, but some of the shortcomings of the Addon SDK are really frustrating. Don't get me wrong: all in all, I believe it's an amazing experience for a newcomer, and I'm sure it's made developing addons much easier and much more fun than it used to be a few years ago.

The good parts:

  • having well-written documentation is... quite a shocking experience for someone who's used to developing Thunderbird addons; what a joy!
  • the tutorial and the various guides are well-written all throughout,
  • there's an easy escape door as soon as you want to do something for which there is no API: require("chrome"),
  • converting a JSM module (the ipccode library in my case) into a RequireJS-style module was really seamless,
  • the cfx utility is really great for launching a Firefox with a fresh profile and the addon pre-installed; I could do it before with shell scripts, but having it all integrated is really life-changing!

The weird parts:

  • figuring out the two-way communication dance between content side and add-on side is quite strange for someone who's not used to it,
  • it took me a short while to understand that as soon as I wanted to manipulate a page's contents, I had to do it through a content script,
  • it feels weird to create a new file (contentScriptFile) just for a dozen lines,
  • the creation of workers and the message passing is slow: I can feel the various steps taking place; also, I'm unsure about the performance hit of creating so many workers all of the time;
  • the final XPI ends up being around 200kB -- so much for just a few dozen lines of code?
  • there's no clear explanation on how one is supposed to do two-way communication: should I used tab.attach, a page-mod, the worker that's created when using context-menu? The tutorials aren't well decided on whether to advertise port-style or postMessage-style communications, which adds to the confusion;
  • I don't want to source bin/activate as it messes up my PS1; does it do anything useful besides changing the PATH?
  • cfx warned me about mozrunner being installed system-wide (I do use mozregression), but I couldn't tell if that was a severe problem or not.

The frustrating parts:

  • I cannot localize the attributes in my HTML file: the placeholder attribute of my <input> field and the value attribute of my button will remain in English...
  • I wanted to do a two-way communication between add-on side and content-side starting from a right-click event, and it tooks me three hours to figure out how to do so -- I guess I was just confused by the variety of options;
  • I'm stuck because the context-menu module won't work in GMail's composition area.

For the last three items, bugs have been filed (respectively bug 824489, bug 824348 and bug 824482). I'm really hoping for the latter to be fixed, since it basically makes half of my addon useless. In the event that no one has the time to fix it, I'll probably have to rewrite the addon to not use the SDK.

I would like to thank KWierso for his help in #jetpack, he was instrumental in achieving what I wanted to do. My warmest congratulations to the Addon SDK team for making the addon landscape usable, and my best encouragements for fixing the various shortcomings!

PS: for the curious, the code lives on GitHub.

Saturday, December 3 2011

Leveraging pdf.js in Thunderbird

Thunderbird Conversations 2.2.1 has been released.

Open this PDF in a new tab

You can now view PDF attachments inside Thunderbird using pdf.js! You don't need an external viewer anymore. That's the pdf.js magic.

PDF Viewer

Clicking the icon opens a new tab with a mini-viewer (very basic functionality). It works pretty well, and the plan is to add a viewer inside the conversation, so that you can get some sort of inline pdf viewer, just below the email. That's a reasonably sized chunk of work, but I'm really busy right now, so it's not going to happen anytime soon; therefore, anyone coming with a patch would be extremely well-received :).

Saturday, November 12 2011

MozCamp 2011 demo addons

EDIT: the slides are available at http://www.xulforum.org/mozcamp2011/presentation/

I'm just writing it down so that it's easy for other people to find it. Here are the demo addons for my MozCamp talk this afternoon.

http://www.xulforum.org/mozcamp2011/

  • Right-click, save link as.
  • In Thunderbird, Tools > Add-ons
  • Wrench menu (next to the search field, top right), Install add-on from file
  • Restart

Both add a new button to the main toolbar so you need to right-click the main toolbar and hit customize.

Monday, September 19 2011

Git mirror of comm-central

Mozilla is the only project where I use hg, and because I'm more comfortable with git, I decided to follow Chris Double's instructions and create a git mirror of comm-central. It now lives at https://github.com/protz/comm-central. It will be updated regularly.

Sunday, September 11 2011

Restartless addons for Thunderbird Conversations

OEmbed addon

Thunderbird Conversations can now have its own addons! I just created a sample plugin for Conversations that uses oembed to analyze links to Flickr in emails. Once it's found a link, it displays a small version of the image below the email, with the title of the picture and its author. You can click the picture to open the original link to Flickr in a new tab. I packaged this plugin as a Thunderbird addon.

The good thing is this addon is restartless! As always, it's been a little bit of a fight to get it to work (I ♥ you restartless), but I managed to get it right. Enough talk, download it now.

Some remarks:

  • this addon of course requires Thunderbird Conversations to be installed first,
  • this addon uses the plugin infrastructure I described earlier,
  • it will not uninstall properly: there was no support for removing a plugin in Conversations, I just added support for that in the trunk version of Conversations,
  • I will not support this in any way. This is a proof of concept, and I'd love to see other people pick up the project, enhance it, and come up with new ideas in creative ways.
  • the source lives on GitHub if you want to fork it!

Comments are welcome!

Tuesday, August 23 2011

Collecting telemetry data for Thunderbird

To all of our testing community: we are now trying to gather anonymous performance data on Thunderbird, using the Telemetry infrastructure that the Firefox folks have so kindly pioneered. I've written about that on tb-planning, but in short: please hit Preferences > Advanced > General > Config Editor and set toolkit.telemetry.enabled to true. Thanks!

Thursday, August 18 2011

Dealing with users

Thunderbird Conversations now has more than 50000 users, and with users, comes a lot of feedback. Right now, too much of the feedback goes like this:

Hi, I installed Thunderbird Conversations, but it doesn't take into account option X, which is really vital (usually all caps here) for any sane user / which sucks and I'm going to uninstall this piece of shit / which makes your addon useless / etc. etc. Optional: what in your right mind did you think? Frequent: if you don't fix this, I'm going to uninstall the addon!

Option X is usually one very, very specific option that I believe is used by a minority of users: detach or delete attachment, download headers only (allegedly for “security reasons"), put sent messages in the folder of the message being replied to. I wasn't even aware of the existence of such options. Even if these were heavily used, I'm not trying to replicate every single option Thunderbird has. If I did, I would end up... reimplementing Thunderbird, except there would be a conversation view. What's the point? If I were to offer all these options, Thunderbird Conversations would become a bloated mess of options with a UI that's cluttered with all these extra actions that seem to be so vital. These users would be happy, but that's not the Thunderbird I'm dreaming of.

I'm trying to offer a new, more efficient, simpler, leaner way to read and reply to your emails. I'm already reimplementing a significant chunk of Thunderbird (I'm just re-doing entirely message display and message composition, mind you), and there's no way I can take into account all these options without becoming crazy. And I'm the only person working on this, to boot.

How can I make it clearer that my goal is not and never will be to reimplement every single Thunderbird feature? How can I tell people that yes, their favorite option that apparently their life depends on will not be implemented and that's just the way it is, they don't have to spam every other communication channel to insult me, write a super negative review on AMO, and encourage other users not to touch, and I'm quoting “this piece of $*§$%#”?

They just have to uninstall the addon.

Monday, August 15 2011

Personas in Thunderbird

There does not seem to be an easy way to set your Thunderbird to use a persona, so until we make it more discoverable in the UI, here's a snippet you can copy and paste in your error console (Tools > Error Console) to make your Thunderbird use a nice Persona! Just copy the line below in Tools > Error Console, and hit Enter. This will open a new tab allowing you to navigate getpersonas.com.

top.opener.document.getElementById("tabmail").openTab("contentTab", { contentPage: "https://www.getpersonas.com" });

Wednesday, August 10 2011

MimeMessage API update

This is just a quick update to let all Thunderbird extension developers know that I've landed a series of improvements to the MimeMessage API. These improvements will be available in Thunderbird 8.

MimeMessage updated API

If you are a Thunderbird addon developer, and you want to inspect the structure of a message, MsgHdrToMimeMessage is what you need. It will give you a nice hierarchical javascript-based representation of the structure of the message. Notable improvements are:

  • you can now inspect encrypted messages (but you have to request that explicitly through the examineEncryptedParts option),
  • detached attachments are now supported,
  • uuencoded attachments are now supported,
  • if the message is stored on a remote IMAP server, you can use the partsOnDemand option that will not retrieve the full message in order to speed things up (the drawback is that you will get inaccurate attachment sizes, for instance).

Tuesday, June 28 2011

Thunderbird Conversations 2.0 released

Update: fixed the broken link for the manual

It is my pleasure to announce that after almost one year in development, Thunderbird Conversations 2.0 has been released! Thunderbird Conversations goes along Thunderbird 5, so go get the latest Thunderbird and...

Download now!

Thunderbird Conversations is a Thunderbird addon that, as the name implies, equips your favorite email program with a fully-featured conversation view. This addon has been developed in close collaboration with the Thunderbird team, and much effort went into Thunderbird to make this happen.

Conversation teaser left

Highlights include:

  • a quick reply feature,
  • tight integration with your Thunderbird workflow,
  • contact tooltips,
  • support for viewing a conversation in a tab,
  • integration with other extensions (enigmail, lightning),
  • quote collapsing,
  • and many more.

Quick reply

While this is still an experiment, we believe that it is sufficiently stable for you to try it out, hence the 2.0 version number. Documentation is available on my personal website (should be integrated in the Thunderbird FLOSS Manual when the next version comes out).

We welcome any kind of feedback. For general discussion, the right place is the mozilla-labs google group. For bugs, please direct them to our GitHub bug tracker.

Known issues:

  • many improvements have been made to Thunderbird to enhance its interaction with Conversations, which means only Thunderbird 5.0 will be compatible with Conversations 2.0,
  • big messages (with attachments) that just arrived will be slow to render if you try to display them immediately: we have a fix both on the Thunderbird and the Conversations side, but we had to back it out due to a regression — we hope to have the fix included in Thunderbird 6,
  • the print command (from the menu) doesn't print the conversation – please use the print icon from the conversation view, or use a recent Thunderbird 7 nightly, the fix should land soon,
  • if you customized the default font settings, it might be that the conversation view has too small fonts: in that case, please revert your font settings to the defaults and read the thread on this topic.

Before reporting a bug, please try out both a recent nightly of Conversations (labeled "master", available at http://jonathan.xulforum.org/files/gcv-nightlies/) and a recent build of Thunderbird 7, known as "Daily". If the issue doesn't go away, we'd be glad to hear about it!

Thursday, May 19 2011

Wanna work on Thunderbird Conversations? We need help!

I started working on Thunderbird Conversations pretty much 18 months ago, and it's grown from a quick hack to a fully-fledged re-implementation of the whole message-reading process in Thunderbird. It now integrates very nicely with the rest of Thunderbird, and I've put a lot of effort into making sure we comply with the bazillion options Thunderbird has. I believe it is a very viable replacement for the standard message reader, and I do have great hopes that the next version (compatible with the upcoming Thunderbird 3.3) will fulfill most people's desires when it comes to a conversation view.

A poor choice of icon and colors

The development has been quiet for quite some time now, as Thunderbird Conversations is slowly becoming more and more feature complete. There's some big, low-priority items that I tackle when I feel really motivated, but apart from that, I'd say this works really fine on a day-to-day basis. However, I'm pretty much alone on this. Coding is fine, but I'm bad at design, and I need a hand to help me polish the UI before the final release.

A nice window, but not consistent with the original design Here's a list of easy items that you can tackle if you have an hour or two:

  • bug 157: experiment with a nicer look for nested blockquotes. Right now, we have this ugly set of colors that feels so 1998-ish, and we could probably use something more modern.
  • bug 182 and bug 184: there's a few windows that might pop up at some point ("Give feedback", "Indexing warning", "Setup assistant"). I designed these by shamelessly stealing the graphics from mozillamessaging.com. They could probably use some visual love, and be more consistent with the rest of the Conversations UI. We also have a phishing warning, but the icon doesn't blend well. This could use some help too.
  • bug 189: make sure people with a dark theme still can use Conversations. Right now, we assume the default text color is black. This is not always the case, and it breaks things for people who have dark themes. We don't necessarily want to respect the system colors, but we should at least make sure Conversations is usable.
  • bug 257: the icons in the conversation toolbar are not aligned.

All these items are probably trivial for someone who's good at UI. I'm pretty bad at it, and most of these items would require me to spend too much time on them, for a less-than-satisfactory result in the end. So if you're good at these things, just drop me an email, or find me on irc.mozilla.org in #maildev (:protz). I'll provide guidance and help you get started with the Conversations source code.

Thanks!

Tuesday, March 29 2011

A synchronous API for asynchronous calls to mozStorage

Are you an extension developer using mozStorage? Do you wish you could switch your code to using asynchronous queries easily? Are you looking for a simple key/value storage system for your Firefox extension? I've written a small library that uses coroutines and the Javascript 1.7 Iterator stuff present in Gecko. It gives you access to a very simple key/value storage API, that's asynchronous, but that reads like synchronous code. If you're interested, read on :-).

I've previously complained about the lack of a simple, asynchronous API for storing key/value pairs when working in chrome. I now consider this problem solved with my new SimpleStorage API, that I developed as part of a larger "thunderbird-stdlib" project.

This is how you can write boring code:

Cu.import("resource://myext/modules/SimpleStorage.js");

let ss = SimpleStorage.createCpsStyle("myExt"); //myExt is the name of your storage "table".
ss.has("myKey", function (aHas) {
  ss.set("myKey", "myVal", function (aWasSet) {
    ss.get("myKey", function (aVal) {
      // ... do stuff ...
    });
  });
});

This can easily become hard to manage. Here comes the new version, using iterators as coroutines:

let ss = SimpleStorage.createIteratorStyle("myExt");
SimpleStorage.spin(function anon () {
  if (yield ss.has("myKey")) {
    yield ss.set("myKey", "myVal");
    let v = yield ss.get("myKey");
    yield ss.remove("myKey");
    ...
    dump("Done");
    yield SimpleStorage.kWorkDone;
    // Nothing ever gets executed past that line. Don't use return!
  }
});
dump("Woo!");

The code is now all sprinkled with calls to yield. Basically, you need to wrap everything with spin and make sure you prefix every call to a database function with yield. And also, don't forget the final yield kWorkDone.

The execution is not synchronous, in the sense that everytime you see yield, the anon function is suspended, the spin function takes care of running the query in the background, and when the result (asynchronously) arrives, anon is restarted with the result. The good thing is it makes code more readable, abstracts away the asynchronous nature of the storage queries, and allows you to write your code in a more concise way.

The cons are:

  • this is counter-intuitive: only what's nested in a call to spin executes in sequence — this means that "Woo!" will probably be printed before "Done",
  • this requires that all the yield statements be topmost in the anon function ; if you move into another function, that one will have to wrap its own calls to yield, i.e. wrap everything inside another spin ; the inner function might even be scheduled after the outer function is done,
  • this seems dangerous because of some rare cases that might happen after all,
  • it's easy to forget to return the final call to yield,
  • there's also the possibility of hardly understandable error messages.

Does that sound acceptable for you? The code lives in my thunderbird-stdlib repository. There is documentation available. The code has been scarcely documented. Let's just say that I'm using function-expressions heavily.

Comments are welcome! This is just a prototype, and I'd be delighted to know how other extension authors feel about this. I'm pretty sure I'm not the only person who wants to store key/value pairs when working within an extension... I initially had planned on offering a third promises-style API, but I got busy with other stuff, and I don't think I will release it in the near future, so I figured out I might officially release this "as is".

Note for the curious

You can actually use spin to make your own asynchronous tasks look synchronous. Here's a useless example:

function myAsyncFunction (aReturn) {
  // do something asynchronous, like...
  setTimeout(function () {
     // blah, blah, blah
    aReturn(42); // this is how you return, don't use return!
  });
}

SimpleStorage.spin(function () {
  let result = yield myAsyncFunction; // you always yield a function that takes one parameter which is itself a function to send the result back
  // result is now 42
  yield SimpleStorage.kWorkDone;
});

If you're reading the implementation

The schema below might come in handy. Or not. But at least I tried!

SimpleStorage

In the actual implementation of the spin function, action is called asyncFunction, and step 5 is done using a recursive function, because I assume (wrongly, I think) we have proper tail calls.

Tuesday, March 15 2011

Thunderbird Conversations 2.0 alpha 3 released

There's a new release of Thunderbird Conversations out there waiting for you to give it a try! (Pending review from the AMO team). Thunderbird Conversations is an addon for Thunderbird that enables a conversation view in the style of Gmail. You can read more about it in our introductory blog post

There have been numerous improvements since the last release, and this will be the last alpha. The software is stable enough so that I can start labeling the builds as "beta". There might be a feature or two coming up (we're thinking about forwarding an entire conversation, in a nice HTML formatting), but that's pretty much all. If you want to share your thoughts, we'd be happy to hear about your feedback on the google groups thread.

The Gallery View This small link will show up if the message you're viewing contains any attached images.

Notable improvements include:

  • conversation tabs will stay there, even if you close Thunderbird,
  • we now have a new gallery view,
  • support for marking a message as junk from the conversation UI,
  • support for phishing warnings,
  • support for Lightning events notifications (thanks go to Fallen),
  • several improvements to the quick reply,
  • fixed bugs in the setup assistant.

Just click this button to open the conversation in a new tab This is the button that will allow you to open the conversation in a new tab. This allows you to have more space to read message, and you can also resize the quick reply area to have more space for replying.

There's also support for spell checking the quick reply, thanks to a series of patches I committed to Thunderbird. To set the language, just right-click in the quick reply area, and pick the language you want.

Spell checking support Spell checking is controlled by the context menu (right-click on the quick reply area), just like in Firefox.

As a side note, all of you who were running Thunderbird Conversations 2.0 alpha 2 should definitely check their preferences: there's a checkbox that the setup assistant used to disable in 2.0 alpha 2. This is not necessary anymore, so you should revert the change. To do so, just go to Preferences > Advanced > Reading and display, and check that "Automatically mark as read" has the value you want it to be.

As usual, the juicy bits are on GitHub. Come over there if you wish to contribute!

Monday, March 14 2011

Basic MimeMessage demo

Edit: I added some screenshots and fixed a couple problems with the extension.

I've been talking for quite some time now about the new MimeMessage representation that allows you to conveniently inspect the structure of a Mime Message. It looks like the concept of callback is unclear to many of our extension developers. Coming from the functional programming community, I probably didn't realize I needed to be a little more specific. To tackle this issue, I've put together a very basic extension that should hopefully get you started :-).

The glodebug button

  • Get the extension.
  • Make sure you have dump enabled.
  • Launch Thunderbird from a terminal.
  • Customize the main toolbar and add the "Glodebug" button.
  • Select any message, and hit the glodebug button.

The result is a ton of colored information about the selected message in the console. The first half of overlay.js extracts the information through the MimeMessage representation. The other half uses a Gloda query to examine the information stored in the index. The last bit kicks a reindexing of the selected message in, so that in case you're writing a Gloda plugin, you can see how it performs.

Glodebug console output

I recommend reading my previous blog post if you're interested in doing that kind of stuff with Thunderbird. I hope it will help some of our extension developers!

As a side note, this is the extension I'm using when I need to debug Gloda stuff, hence the name.

Sunday, February 6 2011

Fosdem 2011 slides

My presentation about Thunderbird addons and Gloda is now available online. Here's a few links:

And don't forget: find us on irc.mozilla.org #maildev: we're a friendly bunch :-). Thanks for your attention!

Saturday, January 15 2011

Thunderbird Conversations git subdmodule update

If you've been using the Git repository for Thunderbird Conversations, please note that the thunderbird-stdlib project has been spun off. While it's far for complete yet, it already hosts the SimpleStorage wrapper I've asked for a few blog posts earlier. This means Thunderbird Conversations now requires a checkout of thunderbird-stdlib to live in its source tree. Fortunately, git submodules make this easy to deal with: just run

git submodule init
git submodule update

from the root of your Thunderbird Conversations source tree. In the future, if you notice while pulling that the .gitmodules file has been updated, make sure you run git submodule update again.

Thanks for your attention!

PS: the SimpleStorage wrapper still lacks a Promises-style API, which is why I haven't blogged about it yet. Currently, there's only a callback-based API, and an iterator-style one, which Thunderbird Conversations now uses.

Thursday, January 13 2011

Thunderbird Conversations 2.0 alpha 2 released

I'm glad to announce that a new version of Thunderbird Conversations has been approved on AMO. If you've been reluctant to use Thunderbird Conversations because some bug would annoy you, there's good chances it's been fixed in the new version. If it hasn't, come and let us know about it on our bugtracker!

Highlights from the Changelog include:

  • quick reply will now quote the text you're replying to,
  • new "send & archive" feature,
  • fix character encoding issues in quick reply,
  • fix keyboard shorcuts sometimes not working when the conversation view has focus,
  • and much more.

The next release will include more feature work, but there's no ETA yet.

Monday, January 3 2011

Thunderbird: the global picture

I've been working on Thunderbird Conversations for more than a year now, and I've learned a lot about Thunderbird internals over the past months. I thought I'd share some thoughts on the design of Thunderbird Conversations, in the hope that it helps would-be extension authors better grasp the design and the relationship between Gloda, libmime, the message headers, and the message database.

This is the first blog post in a series. In this post, I'll talk how various parts of Thunderbird interact together. In another post, I'll talk about the Thunderbird Conversations design.

Thunderbird fellows, if I'm talking nonsense in this post, please make the right amendments in the comments. Thanks!

An introduction

The mere fact of displaying a message involves a complex pipeline that's responsible from fetching the message to rendering it onto the users' screen. I'll focus on a small portion of the pipeline, and I believe this is the one most extension authors will be interested in.

I'll detail the following situations:

  • displaying a message,
  • examining the structure of a message,
  • indexing (by gloda) of a message,
  • manipulation of a gloda message.

The basics

At the core of Thunderbird is the message store. These are the big 4GB+ files in your "Mail" or "ImapMail" folders in your profile directory. As the name implies, they store all the messages that you have chosen to keep on your computer (this depends on your synchronization and storage settings). Presently, we have one big file for each folder.

Each folder has an summary file (some kind of an index): these are the .msf files that go along each one of the big files mentioned above. When you choose to open a given folder in Thunderbird, we do not re-read every single message in the folder. The point of the summary file is that it contains just the right amount of information so that one can build the message list from it, that is, the part of the screen that lists the contents of the selected folder. Summary files are usually rather small files.

The summary files contain only basic information: sender, recipients, date, and so on. Programatically speaking, they translate into nsIMsgDbHdr instances: all the information that's stored in the .msf files is reflected onto nsIMsgDbHdrs.

There are different execution pipelines, depending on what happens. Here's the simple case where the user chooses to display a message.

Standard "message display" pipeline.

Gloda schema

Fig. 1. A big schema that you should read alongside the explanations!

How do we display the currently selected message in the message list?

What Thunderbird initially holds is a message header (extension authors, this is gFolderDisplay.selectedMessage, a nsIMsgDbHdr). The message store is queried to figure out whether we have this message locally, or if we need to download it. A download can happen if the message just arrived (think about IMAP), or if the user chose not to synchronize the folder the message is in. A partial download will occur, that is, we won't download heavy attachments, but seek so that we just read the associated metadata. Once we have downloaded the message body and all the attachments that are to be displayed alongside the message (text/plain attachments, for instance), libmime kicks in.

libmime is an old, ancient piece of software that lies at the core of Thunderbird. It is responsible for two completely unrelated tasks: parsing a message, and at the same time outputting the corresponding HTML for rendering. Because the two tasks correspond to distinct, separate logics, they naturally are horribly entangled in the libmime code ;-). Yay!

Libmime will parse the message structure, decode attachments, take care of various message encodings, output <hr>s to separate inline attachments, use <fieldset>s for attachment headers, and so on.

Finally, the output of libmime is fed into a XUL <browser> element whose task is to render HTML. This is the message reader, that sits in the bottom right corner of Thunderbird, and this is the part that's responsible for rendering the final HTML.

I am an extension, give me a representation of the message

If you are an extension author, there's good hope you'll want to examine a message at some point. What you hold is usually a nsIMsgDbHdr. There's absolutely no way you can disguise yourself as a libmime consumer, and talk to libmime directly. The ancient API is really hard to work with, plus, the data is not presented in an easily understandable way. What you'll probably do is use a module called mimemsg.js, that abstracts the libmime API away and offers you a nice, hierarchical vision of a message.

The pipeline goes like this: you call MsgHdrToMimeMessage, a function that, as the name implies, translates a nsIMsgDbHdr into a MimeMessage instance. The really nice thing is, mimemsg.js will talk to libmime for you, and will build a hierarchical representation of the message. Once it's done (this is asynchronous), the callback you passed to MsgHdrToMimeMessage will be called with the resulting MimeMessage instance as its second parameter.

let msgHdr = gFolderDisplay.selectedMessage;
MsgHdrToMimeMessage(msgHdr, null, function (aMsgHdr, aMimeMessage) {
  // do something with aMimeMessage:
  dump(aMimeMessage.coerceBodyToPlainText());
  dump(aMimeMessage.allUserAttachments.length);
  dump(aMimeMessage.size);
}, true);

Fig. 2. MsgHdrToMimeMessage example. Read the documentation here.

MimeMessage, MimeBody, MimeMessageAttachment and MimeUnknown are all Javascript classes that represent parts of a message. They all have interesting properties: for instance, MimeMessage has some allAttachments and allUserAttachments properties, a parts property, a body property. MimeMessageAttachment has a url property, a size property, a isReal property, and so on. mimemsg.js is modern JS, is really well commented, easily readable, so if your are interested, you should definitely take a look.

So I don't need Gloda, right? What is Gloda anyway?

Gloda is a global index, kind of like the msf files, except it is Thunderbird-wide, uses an modern storage format, and is searchable efficiently. Gloda runs in the background, and indexes messages as they go: that is to say, it reads the interesting properties (sender, recipient, attachments, body) and stores them in a SQLite database that the user can search later.

Internally, when performing an indexing task, Gloda also starts with message headers. It requests a download of the message if not present in the offline store, obtains a MimeMessage representation of the message, and extracts relevant information from it. It inserts the information into a file called "global-messages.sqlite", and goes on indexing other messages. Indexing happens when new messages arrive, or in the background, in the case of an initial indexing.

I'm an extension author, how am I interested?

Gloda allows you to perform complex searches: this is the spine of Thunderbird Conversations. Gloda allows us to find out about the entire conversation even though only one message of it is currently available (because, say, the rest of the conversation is in other folders). Besides messages, Gloda also allows you to find contacts.

This is not the only reason you might want to use gloda. Gloda contains information that's much richer than what's stored in the message headers. For instance, the allUserAttachments property of MimeMessages is also stored on Gloda messages (it's the attachmentInfos noun). That kind of information is not available with a bare nsIMsgDbHdr. Finally, while asking for a MimeMessage might require the message to be downloaded, querying Gloda is all done locally.

So now let's assume you hold a nsIMsgDbHdr and you want more information than what's stored in the .msf file (remember, that's what the nsIMsgDbHdr corresponds to). Your first reflex should be to find the corresponding GlodaMessage if present. If that doesn't work, you can re-trigger a download of the message (this will be slower) by requesting a MimeMessage representation. This should be your fallback plan.

Gloda.getMessageCollectionForHeaders([gFolderDisplay.selectedMessage], {
  onItemsAdded: function (aItems) {},
  onItemsModified: function () {},
  onItemsRemoved: function () {},
  onQueryCompleted: function (aCollection) {
    let [glodaMsg] = aCollection.items;
    dump("This message has "+glodaMsg.attachmentInfos.length+"\n");
    let domNode = myDocument.getElementsByTagName("img")[0];
    dump("We assume the first attachment is an image so we can display it\n");
    domNode.setAttribute("src", glodaMsg.attachmentInfos[0].url);
  },
}, null);

Fig. 3. Obtaining a GlodaMessage starting with a nsIMsgDbHdr

EDIT: as Andrew kindly pointed out, I'm being over-optimistic since I assume the message has been indexed already. The user might be running with indexing disabled, or the indexer might be busy doing some background task, and if the message just arrived, it might fail to index it in a timely manner. Here's a complete version with a fallback plan:

Gloda.getMessageCollectionForHeaders([gFolderDisplay.selectedMessage], {
  onItemsAdded: function (aItems) {},
  onItemsModified: function () {},
  onItemsRemoved: function () {},
  onQueryCompleted: function (aCollection) {
    if (aCollection.items.length) {
      let [glodaMsg] = aCollection.items;
      dump("This message has "+glodaMsg.attachmentInfos.length+"\n");
      let domNode = myDocument.getElementsByTagName("img")[0];
      dump("We assume the first attachment is an image so we can display it\n");
      domNode.setAttribute("src", glodaMsg.attachmentInfos[0].url);
    } else {
      MsgHdrToMimeMessage(gFolderDisplay.selectedMessage, null, function (aMsgHdr, aMimeMsg) {
        // do the same thing with the MimeMessage API
        dump("This message has "+aMimeMsg.allUserAttachments.length+"\n");
        let domNode = myDocument.getElementsByTagName("img")[0];
        dump("We assume the first attachment is an image so we can display it\n");
        domNode.setAttribute("src", aMimeMsg.allUserAttachments[0].url);
      }, true); // true means force the message into being downloaded... this might take some time!
    }
  },
}, null);

Fig. 4. Complete sample that first tries to find the requested information with Gloda, and fallbacks to a complete streaming of the message with MimeMessage if the message isn't found in Gloda.

Gloda doesn't have the information I'm interested in

I can hear astute readers telling me that manipulating MimeMessages will always be better that manipulating GlodaMessages: although it might be slower to obtain a MimeMessage, it has all the information: all the data, all the headers... GlodaMessages only have what's needed for indexing and searching.

Surprise! You can write a Gloda plugin that will attach the custom information you want in the Gloda index. That way, you'll never have to re-stream a message to find out about a X-Whatever header: you just have to make sure you insert the X-Whatever information in the Gloda index, and voilà, you're good to go.

For instance, Thunderbird Conversations packages a gloda plugin that searches for bugzilla information and attaches it to the gloda message. The Gloda plugin we package is passed the MimeMessage representation Gloda obtained, extracts relevant information from it, and attaches it to the Gloda message that's to be stored in the database.

I want complex information about the message, and I also want to display it

You'll have to use Gloda or mimemsg.js to obtain the metadata you're interested in, and then trigger the streaming again into an iframe. This might result in the message being streamed twice if you go the MimeMessage way, but there's no easy way to work around this right now. Walking the MimeMessage and synthetizing the HTML to be rendered yourself is error-prone, so for the time being, you're better off leaving it all to libmime (remember, libmime not only decodes the raw messages but also decides how to display it).

I walked into the gloda land, and I want to display one of the messages I just found

GlodaMessage instances have a folderMessage property (possibly null) that represents the original nsIMsgDbHdr if, say, you want to display it. That's true, Gloda sometimes remembers dead messages.

That's all folks. I'll try to write another blog post detailing the issues I've faced and the design choices I've made when writing Thunderbird Conversations. Thanks for reading!

- page 1 of 2