Mozilla’s Networking Team in 2011

Mozilla created a networking team as part of the platform engineering group in April of 2011. There are nine of us now, spread out around the world, with me managing. We’re responsible for Gecko’s networking stack, including network protocol support, security, and caching. I want to share some of the things we did in 2011.

We landed a SPDY implementation for the upcoming Firefox 11. SPDY will remain pref’d off until we’ve completed testing and various reviews, but it works quite well already. We’re working with Google to properly standardize SPDY.

We added support for the latest WebSockets specification. Firefox 11 currently has WebSockets enabled in standard form, without our vendor prefix.

Firefox 11 also includes SSL performance improvements. We can now negotiate SSL connections in parallel, on multiple threads, instead of serially on a single SSL thread.

We’re looking forward to increased IPv6 usage and we want to make sure that Gecko handles it well. In the past year we’ve improved IPv6 auto-detection, proxy support, and security. These fixes are spread out among a number of Firefox releases.

We made a number of improvements to HTTP pipelining in 2011, largely related to batching and ordering requests efficiently. HTTP pipelining is interesting because it’s an improvement to an existing, widely-used technology, but it has compatibility issues. Pipelining is currently only in use for our mobile products, where the risk/reward ratio is clearly in its favor, but we may enable it for desktop products next year.

Performance on Android is a priority for us. One optimization for Android that we’re particularly happy with is a major DNS performance improvement, which is detailed in a previous post. We also did the work necessary to enable our disk cache on Android.

Networking performance can be hard to test due to the variety of network conditions users can be operating under. In order to add flexibility to our testing capabilities we developed a system called NeckoNet. NeckoNet is a software suite including, among other things, a web server (Apache), talos, and a modified netem kernel module. We provide a Linux VM with all of this properly set up. Using NeckoNet, we can adjust bandwidth, packet loss and latency for network tests.

We’ve also made many other bug fixes and optimizations, conducted security reviews, and spent time planning future work.

In case you want to follow our work in the future, I’ll point out some of the ways in which we communicate (in addition to B.M.O.). Mozilla’s main networking wiki page has links to a number of pages we use to stay organized, including quarterly goals. Starting in January 2012 our team meetings will be open, with dial-in information posted on the mozilla.dev.planning newsgroup the day before the meeting. These meetings happen at 10am Pacific every other Tuesday. We’re also going to try to blog about what we’re doing more often. Any blog posts will be syndicated to Planet Mozilla.

Improving DNS Performance in Firefox for Android

Mozilla has been working hard to improve Firefox on Android. The following is a guest post from Steve Workman of Mozilla’s networking team which describes an effort to improve DNS performance. – Josh

It started with some crashes on Android that were due to getaddrinfo being called from multiple threads. The problem was that the version of getaddrinfo supplied by Bionic (Android’s minimal-but-fast libc implementation) in pre-Honeycomb Android isn’t thread-safe. This is because fopen/fclose etc. aren’t thread-safe. Multiple accesses were being made to a file pointer when reading the local hosts file, resulting in crashes.

Why were we calling getaddrinfo on multiple threads? Calls to getaddrinfo can block until a response is received from a DNS server. This can take a while, especially if there is a problem and we wait for the timeout. Making parallel getaddrinfo calls allows us to cut down on waiting and get more done at once. Sockets can be opened sooner, HTTP requests can be sent sooner, and ultimately your content can be received and displayed sooner. Not being able to make parallel calls to getaddrinfo would be a serious performance regression, especially on mobile where round trip times are generally longer.

First we needed a quick fix for the crash – a performance regression is better than a crash regression. We temporarily serialized calls to getaddrinfo and prefetching (predictive DNS resolution) was disabled.

After that, we decided to provide our own thread-safe version of getaddrinfo, bypassing Bionic’s. Our implementation would have mmap‘d access to the local hosts file, using open/close directly, thus providing a thread-safe function. However, since we were dealing with a library-exposed function, it meant calls to functions and use of structures which were not exposed; at least not officially. After a few failed attempts in which we were trying to get away with dependencies on some unofficially exposed symbols, we finally pulled in a pretty complete version of the host resolver from Gingerbread. This added to our library size a bit, but it allowed for parallel calls to getaddrinfo on Android again. Given the potential for such calls to block for the duration of a DNS request, we believe this is a good tradeoff.

This change is currently scheduled to ship in Firefox 11.

–
Steve Workman

2011 Favorite Restaurants (in the U.S.)

It has been a little over a year since I put together my first list of my top ten favorite restaurants in the U.S., along with some honorable mentions. Things have changed over the past year, though there was no movement in the top four.

1. La Belle Vie (510 Groveland Avenue, Minneapolis, MN)
2. Grocery (288 Smith Street, Brooklyn, NY)
3. Burma Superstar (309 Clement Street, San Francisco, CA)
4. Hard Times Café (1821 Riverside Avenue, Minneapolis, MN)
5. Frankie’s 457 (457 Court Street, Brooklyn, NY)
6. Amber Indian (2290 West El Camino Real, Mountain View, CA‎)
7. Saad’s Halal Restaurant (4500 Walnut Street, Philadelphia, PA)
8. Jasmine Deli (2532 Nicollet Avenue, Minneapolis, MN‎)
9. Boqueria (53 West 19th Street, New York, NY)
10. Maialino (2 Lexington Avenue, New York, NY)

Worth mentioning:

• Tertulia (359 6th Avenue, New York, NY)
• Tamarind (41 East 22nd Street, New York NY)
• Bandaloop (2 Ocean Avenue, Kennebunkport, ME)
• Rice (292 Elizabeth Street, New York, NY)
• Dirt Candy (430 East 9th Street, New York, NY)
• Farmicia (15 South 3rd Street, Philadelphia, PA)
• Lucali’s (575 Henry Street, Brooklyn, NY)
• Ralph’s Italian Restaurant (760 South 9th Street, Philadelphia, PA)
• Broder (2508 SE Clinton St, Portland, OR)
• Tartine Bakery (600 Guerrero Street, San Francisco, CA)

Incredible 2011 NBA Finals

I’m not a big basketball fan. I rarely watch during the regular season. However, the NBA finals are fantastic every time I watch and this year was no exception – loved it! Energetic, passionate, and incredibly talented play. Jason Terry was my favorite player in the end. His performance in game six when Nowitzki had a tough night was inspiring.

Congrats Mavs!

Visiting the Mozilla Community in Indonesia

Bali. Photo by Yofie Setiawan.

I visited Indonesia with fellow Mozilla engineers David Mandelin and David Anderson at the beginning of May. It was a great chance to explore a new place and get to know some of our Mozilla community members.

After the 22-hour journey to Jakarta (New York -> Hong Kong -> Jakarta) I had one day to relax and explore by myself before meeting my co-workers and hosts. Indonesia’s energy and enthusiasm was apparent right from the start. Jakarta proved to be quite the buzzing metropolis, its population frantically engaged in economic and cultural expansion. Despite the pace of modernization, some parts of the city might have existed just the same in any number of past decades. I took a car to Sunda Kelapa in the evening and watched Pinisi being loaded at sunset. Simultaneously beautiful and eery calls to prayer drifted across the water from the many minarets surrounding the harbor.

Students and community members at the Firefox event in Malang.

I met up with my co-workers and our first Indonesian hosts, Yofie and Andi, the next morning. We headed to the airport for our flight to Surabaya, where the first Firefox event on our agenda was held. Between one and two hundred people attended and I think we met over half of them by the time the evening was over. We really enjoyed talking and taking pictures with community members.

After the event in Surabaya we attended events in Malang, Makassar, and Bali. Each was fantastic and unique, but energy, enthusiasm and kindness were constants. In Malang we had some particularly good technical discussions because the event was at a university. David Mandelin and I also got to play basketball in the university’s gym after the event! David Anderson and I had a great time exploring Makassar together on foot, and the food we ate there was amazing. We had a gourmet Indonesian lunch, complete with banana fritters and fresh fruit for dessert. Bali was somewhat over-developed for tourists, with nothing less than a huge American-style mall, but there is no denying its natural beauty. And the people, like everywhere else in Indonesia, really make the place something special.

David Mandelin, Yofie Setiawan, David Anderson, Andi Darmawan, Josh Aas

I can’t thank our wonderful hosts Yofie, Andi, RaRa, Mamie, and Viking, enough. We learned so much from them and we couldn’t have asked for better company throughout our visit. It’s great to know that we have such an organized and vibrant community in Indonesia because it is clear that the country is only going to become more and more influential in the world.

(Lots more pictures on my Flickr page!)

Wells Fargo Rewards is a Sleazy Program

Here is how the rewards program works. First my banker bothers me about enrolling until I do. Undoubtedly he got commission for that which ultimately I paid for in higher fees. Then I get charged $19.00 per year for the rewards program enrollment. Over time I built up points by spending money on my credit and debit cards. Then I have to go to a totally different website from the normal WellsFargo.com site, WellsFargoRewards.com, register a new account there, then log in and select cash rewards. This is hundreds of dollars that they’ve been sitting on for years.

They took my money up front and again every year automatically, just for this program. Then they make it non-automatic and non-obvious for me to reap the benefits of what I paid for. The whole process involves wasting my time (talking to my banker about it, signing up and redeeming points) just to get cash I’m entitled to. I don’t want to play this stupid, sleazy game. If they are sitting on all of this money for me why can’t they just lower my fees and interest rates with it? Because they’re hoping I’m too forgetful and/or inept to redeem, but they’ll never forget to charge me the fee!

Due to things like this (there’s more but I won’t bore you with it) and their part in the financial/mortgage crisis over the past few years I’m done doing business with Wells Fargo.

Firefox 4 for Mac OS X: Under the Hood

Firefox 4 will be an exciting release and we’ve made a number of improvements specific to Mac OS X. Users will benefit primarily in terms of speed, stability, and security. We’ve come a long way since Firefox 3 for Mac OS X.

First, we’ve switched from a ppc/i386 universal binary to an i386/x86_64 universal binary. The default architecture on Mac OS X 10.6 will be x86_64. The default architecture on Mac OS X 10.5 will be i386. You will be able to run in i386 mode on Mac OS X 10.6 if you choose to do so but you will not be able to run in x86_64 mode on Mac OS X 10.5. Performance is the primary motivation for the move to x86_64. These numbers comparing Firefox 4b7 i386 to Firefox 4b7 x86_64 on Mac OS X 10.6.4 give some idea of the kinds of gains we’re seeing from the architecture change alone:

• Cold startup: x86_64 is ~26% faster
• Warm startup: x86_64 is ~5% faster
• MS Psychadelic Browsing Demo: x86_64 is ~540% faster
• MS Speed Reading Demo: x86_64 is ~35% faster

A big part of this is the availability of more CPU registers, but there are a number of other factors in play such as the ABI and the caching of system libraries. If most of your other applications are x86_64, and this is the case on most Mac OS X 10.6 systems, then x86_64 system libraries are more likely to be cache-hot. Your mileage may vary depending on your exact system configuration.

We dropped ATSUI for text rendering and moved to Harfbuzz and Core Text. The move to Harfbuzz for many operations was done for security reasons and in order to expose advanced typographic features. Font handling is difficult in general, and even more so in web browsers. We’d prefer to depend on open source font code if possible because we can patch it quickly and participate in improving it.

We enabled OpenGL accelerated layer composition. This stage in the rendering pipeline is where we composite independently-rendered regions of a web page for your screen. Accelerating it helps us most when resizing images and video. GPUs are much better at performing those sorts of transformations than CPUs. For more information, see this post from Joe Drew. We hope to accelerate the rest of our rendering pipeline on Mac OS X soon.

We also added support for the Cocoa NPAPI event model and the Core Animation NPAPI drawing model. These specifications are a big step forward for browser plugins on Mac OS X. They are easier to develop for, properly documented, and designed with IPC in mind. As of version 10.1, Adobe’s Flash plugin supports Cocoa NPAPI. Which leads me to the next improvement…

Firefox 4 will run many plugins out-of-process on Mac OS X. All of your plugins will be out-of-process if you’re running the x86_64 version of Firefox. If you’re running the i386 version of Firefox we’ll run some popular plugins, such as Flash 10.1+, out-of-process but others will run in-process for performance and user experience reasons. And yes – the x86_64 version of Firefox will be able to use i386 plugins.

Those are the major Mac OS X-specific changes but we’ve also made a large number of minor improvements for Mac OS X. Combined with all of the great cross-platform improvements like our new JavaScript engine, WebM, and better HTML5 support, Firefox 4 should take the web to a whole new level for our users.

Favorite Restaurants (in the U.S.)

It’ll be interesting to look back on this in a year or two and see how much I still agree with.

1. La Belle Vie (510 Groveland Avenue, Minneapolis, MN)
2. Grocery (288 Smith Street, Brooklyn, NY)
3. Burma Superstar (309 Clement Street, San Francisco, CA)
4. Hard Times Café (1821 Riverside Avenue, Minneapolis, MN)
5. Saad’s Halal Restaurant (4500 Walnut Street, Philadelphia, PA)
6. Tartine Bakery (600 Guerrero Street, San Francisco, CA)
7. Amber Indian (2290 West El Camino Real, Mountain View, CA‎)
8. Jasmine Deli (2532 Nicollet Avenue, Minneapolis, MN‎)
9. Frankie’s 457 (457 Court Street, Brooklyn, NY)
10. Boqueria (53 West 19th Street, New York, NY)

Worth mentioning:

• BINO Brooklyn (276 Smith Street, Brooklyn, NY)
• Bandaloop (2 Ocean Avenue, Kennebunkport, ME)
• Rice (292 Elizabeth Street, New York, NY)
• Dirt Candy (430 East 9th Street, New York, NY)
• Bliss Café (191 Bedford Avenue, Brooklyn, NY)
• Farmicia (15 South 3rd Street, Philadelphia, PA)
• South Brooklyn Pizza (451 Court Street, Brooklyn, NY)
• Ralph’s Italian Restaurant (760 South 9th Street, Philadelphia, PA)
• Broder (2508 SE Clinton St, Portland, OR)
• Millennium (580 Geary Street, San Francisco, CA)

Goodbye iPhone, Hello Nexus One

I got an iPhone 3G two years ago and I loved it. It was fast, easy to use, and beautiful. Since then I’ve fallen out of love and every time it takes the phone 5-10 seconds to show a new text message I have to resist the urge to smash it. I was never very pleased with my high cell phone bill or the accompanying contract, or the fact that it’ll probably be a long time before I’m allowed to run Firefox on the phone.

I thought long and hard about getting an iPhone 4 because it would solve some major problems (speed, screen quality) and despite the other issues the user experience is just amazing. However, I decided to go with a solution that also solves the other problems. In total I wanted better phone performance, a nicer screen, better network service, the ability to run Firefox, I don’t want a contract, and I want to pay significantly less for service than I do now. I chose to go with a Nexus One from Google on T-Mobile and so far I’m pretty happy.

The T-Mobile plan I chose has no contract, 1000 any time minutes, unlimited text messaging, and unlimited internet for $70. The AT&T plan I had has a 2-year contract, 900 any time minutes, 1500 text messages, and unlimited internet for $105. T-Mobile reportedly has great service in New York City, so this is an easy choice.

The Nexus One is sold unlocked, it’s fast, thin, has a beautiful screen, gets Android updates quickly, and runs the only two third party apps I really care about – MLB At Bat and Firefox. My only two issues are that the Android OS is generally less polished than iPhone’s iOS and MLB At Bat for Android is junk compared to the iPhone edition.

Five Years with Mozilla

Me working at the Mozilla office in July of 2005. Image from ZDNet UK, click for the article about our old office.

I started working for Mozilla in February of 2005, shortly before I graduated from college. I started full-time in June of 2005. Since then we’ve grown from around 20 employees, most of us in one room, to over 250. Firefox’s market share has grown from about 8.5% globally to over 28%. Much has changed but I’m still as proud to be with Mozilla as I was when I started.

I’m most proud of our strong and consistent commitment to our principles, which were the reason I wanted to work for Mozilla in the first place. This is evidenced by our commitment to an open video format that is freely available to everyone, which is just the most recent example of us favoring our principles over short-term gains.

I wanted to write this primarily to express my gratitude for the opportunity to work on such an incredible project with such excellent co-workers and community members. I couldn’t have asked for a better context in which to grow and I’m as excited about our future as I am about what we’ve already accomplished. Thanks!

(For fun, some old blog posts and articles from when I was hired: 1, 2, 3, 4, 5, 6)