February 04, 2016

Claudio Saavedra: Thu 2016/Feb/04

Igalia WebKit

We've opened a few positions for developers in the fields of multimedia, networking, and compilers. I could say a lot about why working in Igalia is way different to working on your average tech-company or start-up, but I think the way it's summarized in the announcements is pretty good. Have a look at them if you are curious and don't hesitate to apply!

February 04, 2016 12:53 PM

February 03, 2016

Michael Catanzaro: On Subresource Certificate Validation

Igalia WebKit

Ryan Castellucci has a quick read on subresource certificate validation. It is accurate; I fixed this shortly after joining Igalia. (Update: This was actually in response to a bug report from him.) Run his test to see if your browser is vulnerable.

Epiphany, Xombrero, Opera Mini and Midori […] were loading subresources, such as scripts, from HTTPS servers without doing proper certificate validation. […] Unfortunately Xombrero and Midori are still vulnerable. Xombrero seems to be dead, and I’ve gotten no response from them. I’ve been in touch with Midori, but they say they don’t have the resources to fix it, since it would require rewriting large portions of the code base in order to be able to use the fixed webkit.

I reported this to the Midori developers in late 2014 (private bug). It’s hard to understate how bad this is: it makes HTTPS completely worthless, because an attacker can silently modify JavaScript loaded via subresources.

This is actually a unique case in that it’s a security problem that was fixed only thanks to the great API break, which has otherwise been the cause of many security problems. Thanks to the API break, we were able to make the new API secure by default without breaking any existing applications. (But this does no good for applications unable to upgrade.)

(A note to folks who read Ryan’s post: most mainstream browsers do silently block invalid certificates, but Safari will warn instead. I’m not sure which behavior I prefer.)

By Michael Catanzaro at February 03, 2016 08:36 PM

February 01, 2016

Michael Catanzaro: On WebKit Security Updates

Igalia WebKit

Linux distributions have a problem with WebKit security.

Major desktop browsers push automatic security updates directly to users on a regular basis, so most users don’t have to worry about security updates. But Linux users are dependent on their distributions to release updates. Apple fixed over 100 vulnerabilities in WebKit last year, so getting updates out to users is critical.

This is the story of how that process has gone wrong for WebKit.

Before we get started, a few disclaimers. I want to be crystal clear about these points:

  1. This post does not apply to WebKit as used in Apple products. Apple products receive regular security updates.
  2. WebKitGTK+ releases regular security updates upstream. It is safe to use so long as you apply the updates.
  3. The opinions expressed in this post are my own, not my employer’s, and not the WebKit project’s.

Browser Security in a Nutshell

Web engines are full of security vulnerabilities, like buffer overflows, null pointer dereferences, and use-after-frees. The details don’t matter; what’s important is that skilled attackers can turn these vulnerabilities into exploits, using carefully-crafted HTML to gain total control of your user account on your computer (or your phone). They can then install malware, read all the files in your home directory, use your computer in a botnet to attack websites, and do basically whatever they want with it.

If the web engine is sandboxed, then a second type of attack, called a sandbox escape, is needed. This makes it dramatically more difficult to exploit vulnerabilities. Chromium has a top-class Linux sandbox. WebKit does have a Linux sandbox, but it’s not any good, so it’s (rightly) disabled by default. Firefox does not have a sandbox due to major architectural limitations (which Mozilla is working on).

For this blog post, it’s enough to know that attackers use crafted input to exploit vulnerabilities to gain control of your computer. This is why it’s not a good idea to browse to dodgy web pages. It also explains how a malicious email can gain control of your computer. Modern email clients render HTML mail using web engines, so malicious emails exploit many of the same vulnerabilities that a malicious web page might. This is one reason why good email clients block all images by default: image rendering, like HTML rendering, is full of security vulnerabilities. (Another reason is that images hosted remotely can be used to determine when you read the email, violating your privacy.)

WebKit Ports

To understand WebKit security, you have to understand the concept of WebKit ports, because different ports handle security updates differently.

While most code in WebKit is cross-platform, there’s a large amount of platform-specific code as well, to improve the user and developer experience in different environments. Different “ports” run different platform-specific code. This is why two WebKit-based browsers, say, Safari and Epiphany (GNOME Web), can display the same page slightly differently: they’re using different WebKit ports.

Currently, the WebKit project consists of six different ports: one for Mac, one for iOS, two for Windows (Apple Windows and WinCairo), and two for Linux (WebKitGTK+ and WebKitEFL). There are some downstream ports as well; unlike the aforementioned ports, downstream ports are, well, downstream, and not part of the WebKit project. The only one that matters for Linux users is QtWebKit.

If you use Safari, you’re using the Mac or iOS port. These ports get frequent security updates from Apple to plug vulnerabilities, which users receive via regular updates.

Everything else is broken.

Since WebKit is not a system library on Windows, Windows applications must bundle WebKit, so each application using WebKit must be updated individually, and updates are completely dependent on the application developers. iTunes, which uses the Apple Windows port, does get regular updates from Apple, but beyond that, I suspect most applications never get any security updates. This is a predictable result, the natural consequence of environments that require bundling libraries.

(This explains why iOS developers are required to use the system WebKit rather than bundling their own: Apple knows that app developers will not provide security updates on their own, so this policy ensures every iOS application rendering HTML gets regular WebKit security updates. Even Firefox and Chrome on iOS are required to use the system WebKit; they’re hardly really Firefox or Chrome at all.)

The same scenario applies to the WinCairo port, except this port does not have releases or security updates. Whereas the Apple ports have stable branches with security updates, with WinCairo, companies take a snapshot of WebKit trunk, make their own changes, and ship products with that. Who’s using WinCairo? Probably lots of companies; the biggest one I’m aware of uses a WinCairo-based port in its AAA video games. It’s safe to assume few to no companies are handling security backports for their downstream WinCairo branches.

Now, on to the Linux ports. WebKitEFL is the WebKit port for the Enlightenment Foundation Libraries. It’s not going to be found in mainstream Linux distributions; it’s mostly used in embedded devices produced by one major vendor. If you know anything at all about the internet of things, you know these devices never get security updates, or if they do, the updates are superficial (updating only some vulnerable components and not others), or end a couple months after the product is purchased. WebKitEFL does not bother with pretense here: like WinCairo, it has never had security updates. And again, it’s safe to assume few to no companies are handling security backports for their downstream branches.

None of the above ports matter for most Linux users. The ports available on mainstream Linux distributions are QtWebKit and WebKitGTK+. Most of this blog will focus on WebKitGTK+, since that’s the port I work on, and the port that matters most to most of the people who are reading this blog, but QtWebKit is widely-used and deserves some attention first.

It’s broken, too.

QtWebKit

QtWebKit is the WebKit port used by Qt software, most notably KDE. Some cherry-picked examples of popular applications using QtWebKit are Amarok, Calligra, KDevelop, KMail, Kontact, KTorrent, Quassel, Rekonq, and Tomahawk. QtWebKit provides an excellent Qt API, so in the past it’s been the clear best web engine to use for Qt applications.

After Google forked WebKit, the QtWebKit developers announced they were switching to work on QtWebEngine, which is based on Chromium, instead. This quickly led to the removal of QtWebKit from the WebKit project. This was good for the developers of other WebKit ports, since lots of Qt-specific code was removed, but it was terrible for KDE and other QtWebKit users. QtWebKit is still maintained in Qt and is getting some backports, but from a quick check of their git repository it’s obvious that it’s not receiving many security updates. This is hardly unexpected; QtWebKit is now years behind upstream, so providing security updates would be very difficult. There’s not much hope left for QtWebKit; these applications have hundreds of known vulnerabilities that will never be fixed. Applications should port to QtWebEngine, but for many applications this may not be easy or even possible.

Update: As pointed out in the comments, there is some effort to update QtWebKit. I was aware of this and in retrospect should have mentioned this in the original version of this article, because it is relevant. Keep an eye out for this; I am not confident it will make its way into upstream Qt, but if it does, this problem could be solved.

WebKitGTK+

WebKitGTK+ is the port used by GTK+ software. It’s most strongly associated with its flagship browser, Epiphany, but it’s also used in other places. Some of the more notable users include Anjuta, Banshee, Bijiben (GNOME Notes), Devhelp, Empathy, Evolution, Geany, Geary, GIMP, gitg, GNOME Builder, GNOME Documents, GNOME Initial Setup, GNOME Online Accounts, GnuCash, gThumb, Liferea, Midori, Rhythmbox, Shotwell, Sushi, and Yelp (GNOME Help). In short, it’s kind of important, not only for GNOME but also for Ubuntu and Elementary. Just as QtWebKit used to be the web engine for choice for Qt applications, WebKitGTK+ is the clear choice for GTK+ applications due to its nice GObject APIs.

Historically, WebKitGTK+ has not had security updates. Of course, we released updates with security fixes, but not with CVE identifiers, which is how software developers track security issues; as far as distributors are concerned, without a CVE identifier, there is no security issue, and so, with a few exceptions, distributions did not release our updates to users. For many applications, this is not so bad, but for high-risk applications like web browsers and email clients, it’s a huge problem.

So, we’re trying to improve. Early last year, my colleagues put together our first real security advisory with CVE identifiers; the hope was that this would encourage distributors to take our updates. This required data provided by Apple to WebKit security team members on which bugs correspond to which CVEs, allowing the correlation of Bugzilla IDs to Subversion revisions to determine in which WebKitGTK+ release an issue has been fixed. That data is critical, because without it, there’s no way to know if an issue has been fixed in a particular release or not. After we released this first advisory, Apple stopped providing the data; this was probably just a coincidence due to some unrelated internal changes at Apple, but it certainly threw a wrench in our plans for further security advisories.

This changed in November, when I had the pleasure of attending the WebKit Contributors Meeting at Apple’s headquarters, where I was finally able meet many of the developers I had interacted with online. At the event, I gave a presentation on our predicament, and asked Apple to give us information on which Bugzilla bugs correspond to which CVEs. Apple kindly provided the necessary data a few weeks later.

During the Web Engines Hackfest, a yearly event that occurs at Igalia’s office in A Coruña, my colleagues used this data to put together WebKitGTK+ Security Advisory WSA-2015-0002, a list of over 130 vulnerabilities disclosed since the first advisory. (The Web Engines Hackfest was sponsored by Igalia, my employer, and by our friends at Collabora. I’m supposed to include their logos here to advertise how cool it is that they support the hackfest, but given all the doom and gloom in this post, I decided perhaps they would perhaps prefer not to have their logos attached to it.)

Note that 130 vulnerabilities is an overcount, as it includes some issues that are specific to the Apple ports. (In the future, we’ll try to filter these out.) Only one of the issues — a serious error in the networking backend shared by WebKitGTK+ and WebKitEFL — resided in platform-specific code; the rest of the issues affecting WebKitGTK+ were all cross-platform issues. This is probably partly because the trickiest code is cross-platform code, and partly because security researchers focus on Apple’s ports.

Anyway, we posted WSA-2015-0002 to the oss-security mailing list to make sure distributors would notice, crossed our fingers, and hoped that distributors would take the advisory seriously. That was one month ago.

Distribution Updates

There are basically three different approaches distributions can take to software updates. The first approach is to update to the latest stable upstream version as soon as, or shortly after, it’s released. This is the strategy employed by Arch Linux. Arch does not provide any security support per se; it’s not necessary, so long as upstream projects release real updates for security problems and not simply patches. Accordingly, Arch almost always has the latest version of WebKitGTK+.

The second main approach, used by Fedora, is to provide only stable release updates. This is more cautious, reflecting that big updates can break things, so they should only occur when upgrading to a new version of the operating system. For instance, Fedora 22 shipped with WebKitGTK+ 2.8, so it would release updates to new 2.8.x versions, but not to WebKitGTK+ 2.10.x versions.

The third approach, followed by most distributions, is to take version upgrades only rarely, or not at all. For smaller distributions this may be an issue of manpower, but for major distributions it’s a matter of avoiding regressions in stable releases. Holding back on version updates actually works well for most software. When security problems arise, distribution maintainers for major distributions backport fixes and release updates. The problem is that this not feasible for web engines; due to the huge volume of vulnerabilities that need fixed, security issues can only practically be handled upstream.

So what’s happened since WSA-2015-0002 was released? Did it convince distributions to take WebKitGTK+ security seriously? Hardly. Fedora is the only distribution that has made any changes in response to WSA-2015-0002, and that’s because I’m one of the Fedora maintainers. (I’m pleased to announce that we have a 2.10.7 update headed to both Fedora 23 and Fedora 22 right now. In the future, we plan to release the latest stable version of WebKitGTK+ as an update to all supported versions of Fedora shortly after it’s released upstream.)

Ubuntu

Ubuntu releases WebKitGTK+ updates somewhat inconsistently. For instance, Ubuntu 14.04 came with WebKitGTK+ 2.4.0. 2.4.8 is available via updates, but even though 2.4.9 was released upstream over eight months ago, it has not yet been released as an update for Ubuntu 14.04.

By comparison, Ubuntu 15.10 (the latest release) shipped with WebKitGTK+ 2.8.5, which has never been updated; it’s affected by about 40 vulnerabilities fixed in the latest upstream release. Ubuntu organizes its software into various repositories, and provides security support only to software in the main repository. This version of WebKitGTK+ is in Ubuntu’s “universe” repository, not in main, so it is excluded from security support. Ubuntu users might be surprised to learn that a large portion of Ubuntu software is in universe and therefore excluded from security support; this is in contrast to almost all other distributions, which typically provide security updates for all the software they ship.

I’m calling out Ubuntu here not because it is specially-negligent, but simply because it is our biggest distributor. It’s not doing any worse than most of our other distributors.

Debian

Debian provides WebKit updates to users running unstable, and to testing except during freeze periods, but not to released version of Debian. Debian is unique in that it has a formal policy on WebKit updates. Here it is, reproduced in full:

Debian 8 includes several browser engines which are affected by a steady stream of security vulnerabilities. The high rate of vulnerabilities and partial lack of upstream support in the form of long term branches make it very difficult to support these browsers with backported security fixes. Additionally, library interdependencies make it impossible to update to newer upstream releases. Therefore, browsers built upon the webkit, qtwebkit and khtml engines are included in Jessie, but not covered by security support. These browsers should not be used against untrusted websites.

For general web browser use we recommend Iceweasel or Chromium.

Chromium – while built upon the Webkit codebase – is a leaf package, which will be kept up-to-date by rebuilding the current Chromium releases for stable. Iceweasel and Icedove will also be kept up-to-date by rebuilding the current ESR releases for stable.

(Iceweasel and Icedove are Debian’s de-branded versions of Firefox and Thunderbird, the product of an old trademark spat with Mozilla.)

Debian is correct that we do not provide long term support branches, as it would be very difficult to backport security fixes. But it is not correct that “library interdependencies make it impossible to update to newer upstream releases.” This might have been true in the past, but for several years now, we have avoided requiring new versions of libraries whenever it would cause problems for distributions, and — with one big exception that I will discuss below — we ensure that each release maintains both API and ABI compatibility. (Distribution maintainers should feel free to get in touch if we accidentally introduce some compatibility issue for your distribution; if you’re having trouble taking our updates, we want to help. I recently worked with openSUSE to make sure WebKitGTK+ can still be compiled with GCC 4.8, for example.)

The risk in releasing updates is that WebKitGTK+ is not a leaf package: a bad update could break some application. This seems to me like a good reason for application maintainers to carefully test the updates, rather than a reason to withhold security updates from users, but it’s true there is some risk here. One possible solution would be to have two different WebKitGTK+ packages, say, webkitgtk-secure, which would receive updates and be used by high-risk software like web browsers and email clients, and a second webkitgtk-stable package that would not receive updates to reduce regression potential.

Recommended Distributions

We regularly receive bug reports from users with very old versions of WebKit, who trust their distributors to handle security for them and might not even realize they are running ancient, unsafe versions of WebKit. I strongly recommend using a distribution that releases WebKitGTK+ updates shortly after they’re released upstream. That is currently only Arch and Fedora. (You can also safely use WebKitGTK+ in Debian testing — except during its long freeze periods — and Debian unstable, and maybe also in openSUSE Tumbleweed. Just be aware that the stable releases of these distributions are currently not receiving our security updates.) I would like to add more distributions to this list, but I’m currently not aware of any more that qualify.

The Great API Break

So, if only distributions would ship the latest release of WebKitGTK+, then everything would be good, right? Nope, because of a large API change that occurred two and a half years ago, called WebKit2.

WebKit (an API layer within the WebKit project) and WebKit2 are two separate APIs around WebCore. WebCore is the portion of the WebKit project that Google forked into Blink; it’s too low-level to be used directly by applications, so it’s wrapped by the nicer WebKit and WebKit2 APIs. The difference between the WebKit and WebKit2 APIs is that WebKit2 splits work into multiple secondary processes. Asides from the UI process, an application will have one or many separate web processes (for the actual page rendering), possibly a separate network process, and possibly a database process for IndexedDB. This is good for security, because it allows the secondary processes to be sandboxed: the web process is the one that’s likely to be compromised first, so it should not have the ability to access the filesystem or the network. (Remember, though, that there is no Linux sandbox yet, so this is currently only a theoretical benefit.) The other main benefit is robustness. If a web site crashes the renderer, only a single web process crashes (corresponding to one tab in Epiphany), not the entire browser. UI process crashes are comparatively rare.

Intermission: Certificate Verification

Another advantage provided by the API change is the opportunity to handle HTTPS connections more securely. In the original WebKitGTK+ API, applications must handle certificate verification on their own. This was a serious mistake; predictably, applications performed no verification at all, or did so improperly. For instance, take this Shotwell bug which is not fixed in any released version of Shotwell, or this Banshee bug which is still open. Probably many more applications are affected, because I have not done a comprehensive check. The new API is secure by default; applications can ignore verification errors, but only if they go out of their way to do so.

Remember that even though WebKitGTK+ 2.4.9 was released upstream over eight months ago, Ubuntu 14.04 is still on 2.4.8? It’s worth mentioning that 2.4.9 contains the fix for that serious networking backend issue I mentioned earlier (CVE-2015-2330). The bug is that TLS certificate verification was not performed until an HTTP response was received from the server; it’s supposed to be performed before sending an HTTP request, to prevent secure cookies from leaking. This is a disaster, as attackers can easily use it to get your session cookie and then control your user account on most websites. (Credit to Ross Lagerwall for reporting that issue.) We reported this separately to oss-security due to its severity, but that was not enough to convince distributions to update. But most applications in Ubuntu 14.04, including Epiphany and Midori, would not even benefit from this fix, because the change only affects WebKit2; remember, there’s no certificate verification in the original WebKitGTK+ API. (Modern versions of Epiphany do use WebKit2, but not the old version included in Ubuntu 14.04.) Old versions of Epiphany and Midori load pages even if certificate verification fails; the verification result is only used to change the status of a security indicator, basically giving up your session cookies to attackers.

Removing WebKit1

WebKit2 has been around for Mac and iOS for longer, but the first stable release for WebKitGTK+ was the appropriately-versioned WebKitGTK+ 2.0, in March 2013. This release actually contained three different APIs: webkitgtk-1.0, webkitgtk-3.0, and webkit2gtk-3.0. webkitgtk-1.0 was the original API, used by GTK+ 2 applications. webkitgtk-3.0 was the same thing for GTK+ 3 applications, and webkit2gtk-3.0 was the new WebKit2 API, available only for GTK+ 3 applications.

Maybe it should have remained that way.

But, since the original API was a maintenance burden and not as stable or robust as WebKit2, it was deleted after the WebKitGTK+ 2.4 release in March 2014. Applications had had a full year to upgrade; surely that was long enough, right? The original WebKit API layer is still maintained for the Mac, iOS, and Windows ports, but the GTK+ API for it is long gone. WebKitGTK+ 2.6 (September 2014) was released with only one API, webkit2gtk-4.0, which was basically the same as webkit2gtk-3.0 except for a couple small fixes; most applications were able to upgrade by simply changing the version number. Since then, we have maintained API and ABI compatibility for webkit2gtk-4.0, and intend to do so indefinitely, hopefully until GTK+ 4.0.

A lot of good that does for applications using the API that was removed.

WebKit2 Adoption

While upgrading to the WebKit2 API will be easy for most applications (it took me ten minutes to upgrade GNOME Initial Setup), for many others it will be a significant challenge. Since rendering occurs out of process in WebKit2, the DOM API can only be accessed by means of a shared object injected into the web process. For applications that perform only a small amount of DOM manipulation, this is a minor inconvenience compared to the old API. For applications that use extensive DOM manipulation — the email clients Evolution and Geary, for instance — it’s not just an inconvenience, but a major undertaking to upgrade to the new API. Worse, some applications (including both Geary and Evolution) placed GTK+ widgets inside the web view; this is no longer possible, so such widgets need to be rewritten using HTML5. Say nothing of applications like GIMP and Geany that are stuck on GTK+ 2. They first have to upgrade to GTK+ 3 before they can consider upgrading to modern WebKitGTK+. GIMP is working on a GTK+ 3 port anyway (GIMP uses WebKitGTK+ for its help browser), but many applications like Geany (the IDE, not to be confused with Geary) are content to remain on GTK+ 2 forever. Such applications are out of luck.

As you might expect, most applications are still using the old API. How does this work if it was already deleted? Distributions maintain separate packages, one for old WebKitGTK+ 2.4, and one for modern WebKitGTK+. WebKitGTK+ 2.4 has not had any updates since last May, and the last real comprehensive security update was over one year ago. Since then, almost 130 vulnerabilities have been fixed in newer versions of WebKitGTK+. But since distributions continue to ship the old version, few applications are even thinking about upgrading. In the case of the email clients, the Evolution developers are hoping to upgrade later this year, but Geary is completely dead upstream and probably will never be upgraded. How comfortable are you with using an email client that has now had no security updates for a year?

(It’s possible there might be a further 2.4 release, because WebKitGTK+ 2.4 is incompatible with GTK+ 3.20, but maybe not, and if there is, it certainly will not include many security fixes.)

Fixing Things

How do we fix this? Well, for applications using modern WebKitGTK+, it’s a simple problem: distributions simply have to start taking our security updates.

For applications stuck on WebKitGTK+ 2.4, I see a few different options:

  1. We could attempt to provide security backports to WebKitGTK+ 2.4. This would be very time consuming and therefore very expensive, so count this out.
  2. We could resurrect the original webkitgtk-1.0 and webkitgtk-3.0 APIs. Again, this is not likely to happen; it would be a lot of work to restore them, and they were removed to reduce maintenance burden in the first place. (I can’t help but feel that removing them may have been a mistake, but my colleagues reasonably disagree.)
  3. Major distributions could remove the old WebKitGTK+ compatibility packages. That will force applications to upgrade, but many will not have the manpower to do so: good applications will be lost. This is probably the only realistic way to fix the security problem, but it’s a very unfortunate one. (But don’t forget about QtWebKit. QtWebKit is based on an even older version of WebKit than WebKitGTK+ 2.4. It doesn’t make much sense to allow one insecure version of WebKit but not another.)

Or, a far more likely possibility: we could do nothing, and keep using insecure software.

By Michael Catanzaro at February 01, 2016 10:40 PM

Xabier Rodríguez Calvar: Web Engines Hackfest according to me

Igalia WebKit

And once again, in December we celebrated the hackfest. This year happened between Dec 7-9 at the Igalia premises and the scope was much broader than WebKitGTK+, that’s why it was renamed as Web Engines Hackfest. We wanted to gather people working on all open source web engines and we succeeded as we had people working on WebKit, Chromium/Blink and Servo.

The edition before this I was working with Youenn Fablet (from Canon) on the Streams API implementation in WebKit and we spent our time on the same thing again. We have to say that things are much more mature now. During the hackfest we spent our time in fixing the JavaScriptCore built-ins inside WebCore and we advanced on the automatic importation of the specification web platform tests, which are based on our prior test implementation. Since now they are managed there, it does not make sense to maintain them inside WebKit too, we just import them. I must say that our implementation is fairly complete since we support the current version of the spec and have almost all tests passing, including ReadableStream, WritableStream and the built-in strategy classes. What is missing now is making Streams work together with other APIs, such as Media Source Extensions, Fetch or XMLHttpRequest.

There were some talks during the hackfest and we did not want to be less, so we had our own about Streams. You can enjoy it here:

You can see all hackfest talks in this YouTube playlist. The ones I liked most were the ones by Michael Catanzaro about HTTP security, which is always interesting given the current clumsy political movements against cryptography and the one by Dominik Röttsches about font rendering. It is really amazing what a browser has to do just to get some letters painted on the screen (and look good).

As usual, the environment was amazing and we had a great time, including the traditional Street Fighter‘s match, where Gustavo found a worthy challenger in Changseok :)

Of course, I would like to thank Collabora and Igalia for sponsoring the event!

And by the way, quite shortly after that, I became a WebKit reviewer!

By calvaris at February 01, 2016 11:06 AM

January 31, 2016

Manuel Rego: Deep Dive into Grid Layout Placement

Igalia WebKit

During the last months as part of my work in Igalia I’ve been focused on finishing the new/missing bits of the CSS Grid Layout Blink’s implementation related to items placement. In summary the task was mostly related to 2 things:

  • Support implicit grid before explicit. So the grid can add tracks on demand not only on the growing direction (usually right/bottom) but also on the opposite one.

  • Fix unknown named grid lines resolution. This is the case when an item is placed in a line called “foo”, but no lines with that name exist in the grid.

These might seem quite simple tasks, but they implied quite a lot of changes in the underlying implementation. I ended up refactoring all the code to resolve grid positions in order to complete them. I even wrote a document explaining the whole plan and all the technical details, where you can find links to all the related patches.

Now that we’ve finished this, it’s time to explain how you can use it. Although my colleague Sergio already wrote about this in 2014, the specification has changed since that time, so I think it’s better to try to explain the whole thing from scratch. This post is a kind of summary with examples of Placing Grid Items” section of the CSS Grid Layout spec.

Grid lines

This is probably one of the most important concepts of the grid layout spec. The grid lines are the ones dividing horizontally and vertically a grid. And they’re actually numbered, starting at 1.

Let’s use a 3x2 grid as example to explain how this works:

<div style="display: grid;
            grid-template-columns: 300px 200px 100px;
            grid-template-rows: 100px 50px;">
</div>

Numbered grid lines example Numbered grid lines example

Grid placement properties

In order to position the items inside the grid container, you need to use the grid placement properties. These properties are:

  • grid-column-start: Set the first vertical line for the item.
  • grid-column-end: Set the last vertical line for the item.
  • grid-row-start: Set the first horizontal line for the item.
  • grid-row-end: Set the last horizontal line for the item.

With these properties you define the area where the grid item will be placed. In order to do that, you use the line numbers.

The initial value for these properties is auto, which makes possible that items are automatically placed looking for empty cells inside the grid. For more information about this, please review a previous post on the matter.

On top of that, there’re some handy shorthands:

  • grid-column: Shorthand for grid-column-start and grid-column-end properties.
  • grid-row: Shorthand for grid-row-start and grid-row-end properties.
  • grid-area: Shorthand to set the 4 placement properties in just one declaration.

So, imagine that you add the following item in the previous grid:

<div style="grid-column-start: 2; grid-column-end: 4;
  grid-row-start: 1; grid-row-end 2;">
</div>

Probably easier to read if you use some shorthands:

<div style="grid-column: 2 / 4; grid-row: 1 / 2;"></div>

This means that the grid item will be placed taking the 2nd and 3rd columns in the first row.

Place item using line numbers example Place item using line numbers example

Cell spanning

Previous item was spanning 2 columns (2nd and 3rd ones) referencing the lines. You could do the same using the span keyword, together with the number of cells you want to span.

So, you could place the item in the very same position using:

<div style="grid-column: 2 / span 2; grid-row: 1;"></div>

Place item using span example Place item using span example

Note that here you’re not setting the end line for the row. This means that grid-row-end takes a value of auto. In this case auto defaults to a span of one.

Negative line numbers

So far we’ve only seen positive numbers, but lines have also negative indexes. Negative numbers allow you to reference the lines starting from the end of the grid.

Following with the same example, you could place the item again in the same position using the negative indexes:

<div style="grid-column: -3 / -1; grid-row: -3 / -2;"></div>

Place item using negative line numbers example Place item using negative line numbers example

This might be really useful in some situations. For example, if you want to be sure that the item is in the last column, independently of the number of tracks, you’ll just need to set: grid-column-end: -1;.

Named grid lines

Not only that, but you can also name the grid lines, so you don’t need to remember the specific number to reference to them.

Let’s modify the definition of the grid, keeping the size of tracks but adding names to the lines:

<div style="display: grid;
            grid-template-columns: [start] 300px [main] 200px [aside] 100px [end];
            grid-template-rows: [top] 100px [middle] 50px [bottom];">
</div>

And again if we want to place the item in the same position, we just need to reference the name of the lines:

<div style="grid-column: main / end; grid-row: top / middle;"></div>

Place item using line names example Place item using line names example

One line can have several names, you just need to set them in the definition: grid-template-rows: [top start] 100px [middle center] 50px [bottom end];.

Also the names of the lines can be repeated. To reference them you’ve to use a number that can be again positive or negative. Let’s use a different example to showcase this:

<div style="display: grid;
            grid-template-columns: [col] 200px [gap] 50px [col] 200px [gap] 50px [col] 200px [gap];">
</div>

And imagine that you place some items like this:

<div style="grid-column: col 2;"></div>
<div style="grid-column: col 1 / gap 2;"></div>
<div style="grid-column: col -1;"></div>

Place items with repeated named grid lines example Place items with repeated named grid lines example

And of course, you can span to a named grid line:

<div style="grid-column: col 2 / span 2 gap;"></div>

Place item spanning to named grid line example Place item spanning to named grid line example

Grid areas

Better still, you can define grid areas and place items directly on them. You have to use the grid-template-areas property to put names to the different areas in your grid. And you could use the grid-area shorthand directly to place the items.

Let’s use a bigger grid (5x4) to show an example:

<div style="display: grid;
            grid-template-columns: 100px 100px 100px 100px 100px;
            grid-template-rows: 100px 100px 100px 100px;
            grid-template-areas:
                'title  title  title  title  aside'
                'nav    main   main   main   aside'
                'nav    main   main   main   aside'
                'footer footer footer footer footer';">
</div>

And position one item in each of the areas:

<div style="grid-area: title;"></div>
<div style="grid-area: nav;"></div>
<div style="grid-area: main;"></div>
<div style="grid-area: aside;"></div>
<div style="grid-area: footer;"></div>

Place items inside grid areas example Place items inside grid areas example

Grid areas & Named grid lines

One interesting thing about areas and placement is that grid areas create implicit names for the grid lines surrounding them. These implicit names use the “-start” and “-end” suffixes. And you can reference those lines when placing an item, instead of using the whole area.

E.g. the “title” area from previous example creates 4 implicit names for the lines (2 in each axis):

  • Left line: “title-start
  • Right line: “title-end
  • Top line: “title-start
  • Bottom line: “title-end

Following with that example you could place an item using the implicit names:

<div style="grid-column: main-start / aside-end;
            grid-row: title-start / nav-end;"></div>

Place item using implicit names from grid areas example Place item using implicit names from grid area

And the same can happen the other way around. If you name lines using those suffixes, you’re actually creating implicit areas. So, we could just create the very same grid using named grid lines:

<div style="display: grid;
            grid-template-columns: [title-start nav-start footer-start] 100px [main-start nav-end] 100px 100px 100px [aside-start title-end main-end] 100px [aside-end footer-end];
            grid-template-rows: [title-start aside-start] 100px [nav-start main-start title-end] 100px 100px [footer-start nav-end main-end aside-end] 100px [footer-end];">
</div>

All the examples of items positioned in this section will be exactly in the same place with this new grid.

Implicit grid

With the grid definition properties (grid-template-columns, grid-template-rows and grid-template-areas) you determine the explicit number of tracks (columns and rows) in your grid. However, grid spec allows you to place items outside of the explicit grid. In order to support that, implicit tracks are created automatically, the size of these tracks is controlled by grid-auto-columns and grid-auto-rows properties. In the following examples I’ll use red color to highlight the implicit lines.

This time let’s use a simple 2x1 grid:

<div style="display: grid;
            grid-template-columns: 200px 200px;
            grid-auto-columns: 100px;">
</div>

And imagine that you place an item in the 5th column (grid-column: 5;). As the grid only has 2 columns, 3 implicit columns will be added in order to position the item.

Implicit grid example Implicit grid example

Again you can also create implicit tracks with items that span several cells. For example, if an item takes 3 columns starting on the 2nd one (grid-column: 2 / span 3);

Implicit grid with span example Implicit grid with span example

Originally, the implicit tracks could only be added at the end. But now it’s possible to add implicit tracks before the explicit grid. For example, if you place an item using grid-column: -5; it’ll add 2 columns on the left and it’ll be placed in the -2nd column.

Implicit grid before explicit example Implicit grid before explicit example

Implicit grid & Named grid lines

But this is not the only way to create implicit tracks, you can also create them if you use undefined named grid lines. This is more a way to show mistakes on the user CSS than a feature itself, but maybe someone finds it useful. The idea is that all the lines in the implicit grid will take any random name you might need to position an item.

The basic example is placing items referencing a nonexistent line called “foo”. For example you will create 3 implicit columns (1 before and 2 after the explicit grid) with the following items:

<div style="grid-column: foo;"></div>
<div style="grid-column: 2 / span 2 foo;"></div>
<div style="grid-column: -1 foo;"></div>

Implicit grid using undefined named grid lines example Implicit grid using undefined named grid lines example

Note that the simplest example grid-column: foo is being placed in the 4th column (adding an extra empty column just after the explicit grid). This is because first line that is considered to be called “foo” is the first implicit line (line 4), so last line of the grid (line 3) is not included.

Also, the last item grid-column: -1 foo is placed on the -1th column (maybe you was not expecting that). This is because of you start looking for a line named “foo” from the edge of the explicit grid. So, you ignore lines -1, -2 and -3 (as they’re not called “foo”) and consider line -4 (first line on the implicit grid) to have that name.

This is a bit trickier if the line actually exists, as you’ve to count it too in order to place the item. Specially it’s complex if you’re using span to a named grid line, but there’re not enough lines. In that case only implicit lines in the search direction are considered to have that name.

Again, hopefully an example can help to understand this. Let’s add a name to the middle line in the previous example:

<div style="display: grid;
            grid-template-columns: 200px [middle] 200px;
            grid-auto-columns: 100px;">
</div>

And now, let’s use place a few items referencing that “middle” line:

<div style="grid-column: 3 middle;"></div>
<div style="grid-column: span 2 middle / 5;"></div>
<div style="grid-column: -4 / span middle;"></div>

Implicit grid using more named grid lines than available example Implicit grid using more named grid lines than available example

The strange case here is grid-column: span 2 middle / 5;, as you can see it takes from -1th column to 4th column (both included). The item ends at line 5, and it has to span 2 lines called “middle” to find the start. You could think that it should count line 4 and line 2, but, as explained before, you have to start counting lines from the edge of the explicit grid. So you actually count line 2 and then you’ve to consider the implicit lines on the left to find the start position (line -4).

Special cases

Grid placement properties have a few special situations that are resolved by the conflict handling section of the spec.

For example, if you place an item where the end line is before than the start line, both lines are swapped. Thus, something like grid-column: 5 / 2; would become grid-column: 2 / 5;.

Another situation is the one in which you have span in both the start and end positions. The span for the end position is discarded. So, grid-column: span 2 / span 3; would become grid-column: span 2;. Which will use the grid placement algorithm to find an empty area (of 2 columns in this particular example) to position itself.

Last one is the case when you only have a span to named grid line. In that case, it’s replaced by span 1. E.g. grid-column: span foo; will become grid-column: span 1;.

Placement special cases example Placement special cases example

Recap

If you have read that far it seems you’re really interested on CSS Grid Layout. The main conclusion is that the specification is really flexible regarding how to place items on the grid. As you can see there’re quite a lot of different ways to position an item in the very same place. Probably each person will get used to a few of them and just forget about the rest.

IMHO the basic stuff (line indexes, spans, line names and areas) is quite straightforward. And the implicit grid is not that hard either. Then negative indexes bring some fun. However undefined named grid lines behavior is really tricky (hopefully not something you should care about anyway). But it’s true that I’m biased as I’ve been dealing with this for a long time.

Last, I thought it would be nice to finish this with a big example which uses most of the things described in this post. If you got it right and don’t get confused at all you’re mastering grid layout placement. Congrats! 😄

This is the definition of the grid container used in this example:

<div style="display: grid;
            grid-template-columns: [left] 200px [main-start] 100px [center] 100px 100px [main-end] 50px 50px [right];
            grid-template-rows: [top title-start] 50px [title-end main-start] 200px [main-end center] 150px 100px [bottom];
            grid-auto-columns: 50px;
            grid-auto-rows: 50px;">
</div>

And in the next picture you can see how different items will be placed.

Placement big example Placement big example

You can try it live in our examples repository: http://igalia.github.io/css-grid-layout/grid-placement.html

Status

As commented on the introduction Blink’s implementation should support now (Chrome 50+) all these different placement possibilities. Igalia has been working on this implementation and we’re porting it now to WebKit too.

On the other side, Gecko has already support for it too, in this case developed by Mozilla.

Igalia and Bloomberg working together to build a better web Igalia and Bloomberg working together to build a better web

Finally, as usual I’d like to highlight one more time that all this work has been done as part of a collaboration between Igalia and Bloomberg. Big thanks for your support!

January 31, 2016 11:00 PM

January 26, 2016

CSS Font Features

Surfin’ Safari

Font features are a way to enable advanced text styles and effects as designed by the font author. A font may support a number of features: some examples include different kinds of ligatures, tabular numbers, or small caps. Because the designer explicitly created forms for these variants in the font file, using font features is the highest quality way of achieving these effects.

Comparison with and without features font-variant-numeric: diagonal-fractions, font-variant-caps: small-caps, and font-variant-numeric: oldstyle-nums

Enabling font features in web content for both iOS and OS X is easy: simply apply one or more of the following CSS properties to your text:

  • font-variant-ligatures
  • font-variant-position
  • font-variant-numeric
  • font-variant-alternates (partial support)
  • font-variant-east-asian

In addition, the font-variant property now additionally accepts the same values as the above properties, which means that all existing usages of font-variant: small-caps will automatically use the higher-quality font feature if it is available. If the font doesn’t support small-caps, WebKit synthesizes a version for you, which unfortunately won’t be as high-quality as a font file which includes a small-caps variant.

While the above properties cover the most common situations, there are far more font features than could be represented by CSS keywords. In these situations, the font-feature-settings property can be used to enable any OpenType feature. This property accepts a comma-separated list of feature name strings to enable. For example: font-feature-settings: 'swsh', 'flac' will enable both swashes and flattened accents. These string identifiers are specific to the OpenType font format, so they only work with OpenType fonts.

All of these properties can also be added to an @font-face block in CSS, which will make all uses of the web font use the appropriate font feature. This default behavior can be overridden by specifying the same properties on the styled element itself.

Support for these font features is available in WebKit nightly builds, as well as the iOS 9.3 Beta and OS X 10.11.4 Beta. If you have questions, comments or other feedback, please feel free to reach me at mmaxfield@apple.com and @litherum, or contact Jonathan Davis at web-evangelist@apple.com, @jonathandavis, or @WebKit.

By Myles Maxfield at January 26, 2016 02:00 PM

January 16, 2016

10 Years of Web Inspector

Surfin’ Safari

Ten years ago today we introduced the first version of Web Inspector to the world. That time was the wild west of web development, when your best debugging aid was alert(). While the earliest version of Web Inspector was limited (it didn’t even have a console), it did expose two things that developers didn’t have much insight into — the DOM tree and matching style rules.

Web Inspector being open source also marked a pivotal moment for the WebKit project. It was one of the rare cases when major UI would become open source from Apple. You have to remember, WebKit was a nascent open source project at the time — only being released to the public 6 months earlier at WWDC 2005. We were still figuring out project goals and what direction the project would take. Ultimately, we made the final call to open source Web Inspector — and it has been a great decision ever since.

As the years progressed, we rapidly added to the tool — exposing more features and evolving the UI to adapt to new developer workflows. To commemorate the past 10 years of Web Inspector, we put together this timeline of major events and past blog posts for posterity.

  1. January 7, 2003

    Safari Unveiled

    Apple unveiled Safari, the fastest and easiest to use web browser on OS X.

  2. June 23, 2003

    Safari 1.0 Released

    At WWDC 2003, Apple released Safari 1.0, completing a successful public beta program.

  3. April 29, 2005

    Safari 2 Released

    It was included with the OS X 10.4 Tiger release.

  4. June 7, 2005

    WebKit Open Sourced

    At WWDC 2005, Apple announced that the WebKit engine behind Safari available as open source.

  5. January 15, 2006

    Web Inspector Landed

    The first version landed in open source. Did you know it used a native NSOutlineView for the DOM tree as a pseudo-plugin inside the HTML user interface?

  6. January 16, 2006

    Web Inspector Introduced

    Web Inspector

    The first version let you browse the live DOM hierarchy in a compact HUD window. It showed all the CSS rules that matched the focused node and highlighted it on the page.

  7. June 28, 2006

    Drosera Introduced

    Drosera JavaScript Debugger

    Drosera, named after the largest genera of bug eating plants, let you attach and debug JavaScript for any WebKit application — not just Safari.

  1. June 11, 2007

    Safari 3 Released

    At WWDC 2007, Apple released Safari 3 for OS X 10.5 Leopard, and beta versions for Windows XP, and Windows Vista.

  2. June 20, 2007

    New Version for Safari 3

    New Web Inspector on Windows

    With a completely redesigned interface, that also worked on Windows. It supported docking to the page, introduced the network waterfall, inline JavaScript and HTML error reporting and a lot more.

  3. March 18, 2008

    Safari 3.1 Released

    Apple released Safari 3.1 for OS X 10.5 Leopard, Windows XP, and Windows Vista.

  4. June 11, 2008

    Safari 4 Released

    Safari 4 featured the Nitro engine which ran JavaScript up to 4.5 times faster than Safari 3.

  5. September 30, 2008

    Web Inspector Redesign

    Web Inspector Redesign

    The standalone Drosera JavaScript debugger was replaced with a new JavaScript integrated debugger. The redesign introduced the split Console, console auto-completion, style property editing and more.

  6. November 3, 2009

    Web Inspector Updates

    Web Inspector Updates

    Introduced a number of exciting new features like the Storage panel, CSS syntax highlighting, Watch Expressions and Conditional Breakpoints.

  1. April 14, 2010

    More Web Inspector Updates

    Web Inspector Timeline Panel

    Introduced the Timeline, Audits and Console panels along with many other improvements.

  2. June 7, 2010

    Safari 5 Released

    At WWDC 2010, Apple released Safari 5 for OS X 10.7 Lion, Windows XP, and Windows Vista.

  3. Febuary 24, 2011

    Style Editing Enhanced

    Showed off improved editing of CSS rules and jumping to the CSS resource from the Styles sidebar.

  4. April 20, 2011

    Understanding Stack Traces

    Documented the ways a developer can deal with exceptions and stack traces in Web Inspector.

  1. June 11, 2012

    Safari 6 Announced

    Apple released a developer preview of Safari 6 which included a reimagination of Web Inspector that aligned the design and user experience with Xcode 4.

  2. July 25, 2012

    Safari 6 Released

    Apple released Safari 6 with OS X 10.8 Mountain Lion.

  3. June 10, 2013

    Safari 7 Announced

    Apple released a developer preview of Safari 7 with a revised Web Inspector user interface.

  4. June 11, 2013

    State of Web Inspector

    Announced the next version of Web Inspector, released as a developer preview at WWDC 2013.

  5. October 22, 2013

    Safari 7 Released

    Apple released Safari 7 with OS X 10.9 Mavericks.

  6. June 8, 2014

    Safari 8 Announced

    Apple released Safari 8 with a developer preview of OS X 10.10 Yosemite.

  7. June 10, 2014

    Accessibility Node Inspector

    Explained how to diagnose and debug problems using new accessibility inspection features in Web Inspector.

  8. October 16, 2014

    Safari 8 Released

    Apple released Safari 8 with OS X 10.10 Yosemite.

  1. June 8, 2015

    Safari 9 Announced

    Apple released of Safari 9 with a developer preview OS X 10.11 El Capitan.

  2. June 24, 2015

    Console Improvements

    Demonstrated improvements to the Console by giving developers quicker access to useful data and modernizing it to work better with the new changes in JavaScript.

  3. August 3, 2015

    User Interface Changes

    Explained the new tab-based user interface. Like the tabs in Safari, they can be rearranged to fit your workflow and personal development needs. This marked the return of dedicated Elements and Network tabs. The changes were covered in Using Safari to Deliver and Debug a Responsive Web Design from WWDC 2015.

  4. August 20, 2015

    Visual Styles Sidebar Previewed

    Showed off a preview of the new Visual Styles sidebar which allows designers and developers new to CSS to easily edit styles with intelligent editing widgets.

  5. August 24, 2015

    JavaScript Type and Code Coverage Profiling

    Added two great tools designed to make debugging JavaScript programs easier: the Code Coverage Profiler and the Type Profiler. Both tools made understanding and debugging your JavaScript programs better than ever.

  6. August 31, 2015

    Styles Sidebar Refinements

    Introduced the various enhancements we made to Web Inspector’s Style sidebar in Safari 9 — such as text filtering and intelligent editing.

  7. September 8, 2015

    Rendering Frames Timeline Introduced

    Introduced a new tool to help locate and resolve page rendering performance issues. The Timelines tab added a new timeline presenting a frame-based, task-specific view of existing profiler/instrumentation data — in addition to the familiar event-based timelines.

  8. September 30, 2015

    Safari 9 Released

    Apple released Safari 9 with OS X 10.11 El Capitan and for OS X 10.10 Yosemite.

  1. October 6, 2015

    Keyboard Shortcuts Documented

    Documented the various keyboard shortcuts for many commonly used actions.

  2. December 1, 2015

    Breakpoint Options Documented

    Documented the Debugger tab and the many options available to enhance standard breakpoints.

  3. December 31, 2015

    Architecture of Web Inspector

    A brief tour through its architecture, history and main features for anyone interested in understanding or hacking on Web Inspector in WebKit.

  1. January 11, 2016

    Safari 9.1 Beta Released

    Apple released a Safari 9.1 beta as part of the OS X 10.11.4 beta, containing many improvements to Web Inspector, including the new Visual Styles sidebar.

  2. January 16, 2016

    10 Year Anniversary

    As always, the latest features and enhancements can be found in WebKit Nightly Builds. Help us make Web Inspector even better by sending us feedback on Twitter (@xeenon, @jonathandavis), or by filing a bug.

By Timothy Hatcher at January 16, 2016 02:00 PM

December 31, 2015

Diego Pino: Architecture of the Web Inspector

Igalia WebKit

In Igalia we have been contributing to the WebKit project for many years. Starting with WebKitGTK+ and progressively reaching other areas such as WebCore, improving accessibility, implementing new APIs, tooling, fixing tons of bugs, etc. The Web Inspector is another area were we have contributed, although much more modestly. This is a post I wanted to write since a long time ago. It’s a brief tour through the Web Inspector. Here I discuss its architecture, history and main features. Hopefully this information can be useful for other people who would like to start hacking on this important component of the WebKit project.

What’s the Web Inspector?

The Web Inspector is a tool that is part of WebKit and allows you to inspect web pages. I mean inspect in a broad sense. It doesn’t only allow you to debug your JavaScript code, but it also includes a rich set of tools, usually divided into panels, which provide very valuable information such as loading times of resources (files, images), CSS properties and DOM elements edition, visual identification of elements within a web page, and many things more. It basically answers the question what’s going on inside a web page?

The difference between the Web Inspector and other tools, such as Firefox’s famous Web Developer extension, is that the Web Inspector is not an external plugin, but it’s part of WebKit. That means that every WebKit port features the Web Inspector. Nowadays, all major open-source browsers include their own Web Inspector alike tool. Chrome features its Developer tools and Firefox has its own Developer tools too.

Throughtout the years

The Web Inspector was shipped in WebKit for the first time in January 2006. Since then it has gone through big and small changes.

The first big change came in June 2007. There was a big redesign, the network panel was included for the first time, syntax highlighting, better error reporting, etc.

One year later, in September 2008, the Inspector went through another big redesign and more panels were included (Elements, Resources, Scripts, Profile, Database).

November 2009 brought better edition of DOM elements, inline creation of CSS rules and selectors, a widget for color editing, JSON and CSS syntax highlighting, etc.

In June 2012 a brand-new Web Inspector was introduced but only for Safari 6. Exactly one year later, Safari 6’s Web Inspector was open-sourced. During some time the new and old versions of the Web Inspector lived together in the codebase. This new inspector brought a major visual redesign and new layout elements: toolbar, navigation bar, quick console, content, sidebar, etc. The panels were structured in: Resources, Timeline, Debugger, Styles, Layers and Node. The current Web Inspector it’s still based on this release.

Two months before Apple open-sourced the new Web Inspector, Google forked WebKit and created Blink. The Web Inspector is known as Developer Tools in Chrome. Since Blink was forked two months before the new Web Inspector release, Chrome’s Developer Tools are actually based in the old Web Inspector code. Although probably it has gone through many changes since the last two years. The bottom line is that WebKit’s Web Inspector and Chrome’s Developer Tools look in fact very similar.

A first glimpse

The Web Inspector source code lives at Source/WebInspectorUI. Most of it is composed of HTML, CSS and JavaScript although there are parts in C++ to bridge with WebCore and JavaScriptCore. According to OpenHub, WebKit’s source code is 20% JavaScript. That’s a big amount of code, although not all JavaScript code in WebKit is part of the Web Inspector. There’s JavaScript also in the Layout tests and JavaScriptCore.

The WebInspectorUI, which represents the frontend, is structured in several folders:

  • Base/: The main classes (Bootstrap.js, WebInspector.js, EventListener.js, Object.js, etc).
  • Configurations/: XCode configuration files.
  • Controllers/: Catch events and call Model classes to perform business logic. Important files: XXXManager.js (TimelineManager.js, DebuggerManager.js, StorageManager.js, LogManager.js).
  • Images/: Images for icons, visual elements, etc. The images for GTK are different due to license restrictions.
  • Localications/: Contains file with localized strings in English.
  • Models/: Classes that perform business logic (KeyboardShortcut.js, LogObjet.js, Timeline.js, Color.js, etc).
  • Protocol/: Observers that respond to notifications emitted by the backend.
  • Scripts/: Several scripts in Perl, Ruby and Python to perform automatic tasks (update CodeMirror, update resources, minimize CSS, etc).
  • Tools/PrettyPrinting/: External tool for pretty print source code in the Console tab.
  • UserInterface/External:
    • CodeMirror/ : Very powerful text editor implemented in JavaScript for the browser. Used for code editing (JavaScript, CSS) inside Web Inspector.
    • ESLint/ : Check JavaScript syntax.
    • Esprima/ : JavaScript parser, for code completion in the editor.
  • Versions/: Description of protocols for different iOS Versions.
  • View/: Classes for visual element objects.

Getting started

Think of the Web Inspector as a web application that lives inside WebKit. It’s possible to modify one of its elements (HTML, CSS of JavaScript) and see the change reflected in the UI just by typing the following command:

$ Tools/Scripts/build-webkit --inspector-frontend

But this only works in not CMake ports (Apple). It has the inconvenient of no updating localized strings too. In other ports, such as WebKitGTK+, there’s no –inspector-frontend flag so it’s necessary to build WebKit. As usual, only the changed files are built:

$ Tools/Scripts/build-webkit --gtk
====================================================================
   WebKit  is  now built   (00m:18s).
   To  run MiniBrowser with    this    newly-built code,   use the
   "../Tools/Scripts/run-minibrowser"  script.
====================================================================

Once it’s built, open MiniBrowser and the Inspector (Ctrl+I) to see your changes:

$ Tools/Scripts/run-minibrowser   --gtk

There’s a permanent tag with open bugs for the Web Inspector and the URL http://webkit.org/new-inspector-bug, can be used to file out new bugs related to the Web Inspector.

How to debug?

Definitively when starting hacking in a new project it’s fundamental to be able to see what’s going on inside. If you try to inspect a Web Inspector variable using console.log(var) nothing will be printed out. It’s necessary to build WebKit in debug mode and enable the flag developerExtrasEnabled:

$ Tools/Script/build-webkit --gtk --debug

In the case of the WebKitGTK+ port, developerExtrasEnabled is always set to TRUE.

It’s also possible to do the same in release mode, just by removing an #ifdef block in WebKit2/UIProcess/gtk/WebInspectorProxyGtk.cpp:

#ifndef NDEBUG
   //  Allow   developers  to  inspect the Web Inspector   in  debug   builds
   //  without changing    settings.
   preferences->setDeveloperExtrasEnabled(true);
   preferences->setLogsPageMessagesToSystemConsoleEnabled(true);
#endif

Now all console.log(var) messages will be printed out in the shell. With this setting enabled, it’s also possible to open a new Web Inspector to inspect the Web Inspector. With the Web Inspector open, righ-click on one of its elements and select Inspect Element, just like in a normal web page.

Remote debugging

The Web Inspector is a multi-tier application. It’s divided into 3 layers: a frontend, a backend and a target. This division detaches the inspector from the inspected browser. In other words, it’s possible to use the inspector to inspect a browser running in a remote device. This can be useful to debug an iPhone web application or a WebKitGTK+ based browser running in an embedded environment, such as the RaspberryPi.

On the browser to be inspected, first define the WEB_INSPECTOR_SERVER variable:

$ export WEB_INSPECTOR_SERVER=127.0.0.1:9222
$ Tools/Script/run-minibrowser --gtk

On the client side, open WebKit:

$ Tools/Script/run-minibrowser --gtk

Go to the server URL, in this case http://127.0.0.1:9222, and you will see a Web Inspector which is actually inspecting a remote browser.

Architecture of the Web Inspector

As mentioned before, the Web Inspector is a 3-tier application divided into several layers:

  • Frontend, also known as the client.
  • Target, also known as the debugee.
  • Backend, also known as the server.

The frontend (WebInspectorUI/UserInterface/) is Web Inspector’s user interface. It’s what the user sees and interacts with. It’s implemented in HTML, CSS and JavaScript. Sometimes this frontend is referred as the debug client, whereas the backend is referred as the server.

The target is the program being debugged. In normal operation of the Web Inspector, the target program is the WebKit loaded in the browser. In remote debugging mode, the inspected target is a WebKit loaded in a remote machine.

The backend (JavaScriptCore/inspector/ and /WebCore/inspector) is what mediates between the target and the frontend. It gives access to the elements that live in JavaScriptCore and WebCore. For instance, in order to be able to debug JavaScript code within a web page, it’s necessary to have access to JavaScriptCore’s stackframe. In consequence, JavaScriptCore has to provide hooks to the Web Inspector so it can access the inspected properties. The same happens with DOM elements. Showing up a DOM element properties requires that WebCore provides this information to the inspector.

Frontend

Let’s dive into the frontend first. If you grep for HTML code hoping to find the layout of the inspector elements, you’re not going to find any code actually. The reason why it’s because all the layout elements in the inspector are via DOM operations (createElement, appendChild,…). For instance, in UserInterface/Views/SidebarPanel.js:

WebInspector.SidebarPanel = class SidebarPanel extends WebInspector.Object
{
    constructor(identifier, displayName, element, role, label)
    {
        super();
        ...
        this._element = element || document.createElement("div");
        this._element.classList.add("panel", identifier);

        this._contentElement = document.createElement("div");
        this._contentElement.className = "content";
        this._element.appendChild(this._contentElement);
        ...
     }
}

Everything starts with Main.html (UserInterface/Main.html). This file loads all the elements that compose the inspector UI. First it loads several external components (CodeMirror, JSLint), then CSS files and after that all the JavaScript files that form the inspector. These files can be classified into different types. Some are architectural elements (Base/XXX.js), others implement business logic operations (Model/XXX.js), other implements programmatic logic (Controllers/XXX.js) and others implement visual elements and widgets (View/XXX.js). Usually UI classes have a CSS file of the same associated, for instance Views/SidebarPanel.js and Views/SidebarPanel.css.

The WebInspector namespace (UserInterface/Base/WebInspector.js) is the central element of the frontend. Everything is going to be accessed from there:

var WebInspector = {}; // Namespace

As the model, controller and view classes are processed they are going to hook themselves to the WebInspector namespace. Here’s the definition of Views/DOMDetailsSidebarPanel.js:

WebInspector.DOMDetailsSidebarPanel = class DOMDetailsSidebarPanel extends WebInspector.DetailsSidebarPanel
{
    constructor(identifier, displayName, singularDisplayName, element, dontCreateNavigationItem)
    {
        super(identifier, displayName, singularDisplayName, element, dontCreateNavigationItem);
        this.element.addEventListener("click", this._mouseWasClicked.bind(this), true);
        this._domNode = null;
    }
    ...
}

It’s a classical Model-View-Controller pattern. The view accesses the controller to execute business logic operations, which are implemented by models. On the same files there’s:

_mouseWasClicked(event)
{
    if (this._domNode && this._domNode.ownerDocument) {
        var mainResource = WebInspector.frameResourceManager.resourceForURL(this._domNode.ownerDocument.documentURL);
        if (mainResource)
            var parentFrame = mainResource.parentFrame;
    }

    WebInspector.handlePossibleLinkClick(event, parentFrame);
}

Where frameResourceManager is an instance of Controllers/FrameResourceManager.js. Views can also access Models, not usually to perform operations on them but to query them. It’s the case on the same file of inspect method:

inspect(objects)
{
   // Iterate over the objects to find a WebInspector.DOMNode to inspect.
   for (var i = 0; i < objects.length; ++i) {
      if (objects[i] instanceof WebInspector.DOMNode) {
         nodeToInspect = objects[i];
         break;
      }
   }
}

Where DOMNode is a model class (./Models/DOMNode.js).

One thing that characterizes the frontend code is that it tends to quickly adopt new JavaScript features implemented by JavaScriptCore. All the code is structured in classes, makes use of inheritance, there are getter and setter methods, there are for-of loops, makes use of Map and Set classes, etc. Any new ES2015 feature that lands in JavaScriptCore, if it’s convenient and simplifies code, makes its room into the inspector. And it makes sense to do it, as it’s guaranteed the latest JavaScriptCore version is going to be there. It also makes the inspector a helpful codebase to understand new ES2015 features.

WebKit Remote Debugging Protocol

Until this point, most of the frontend architecture is covered. I mentioned earlier that another layer of the inspector is the backend. The backend is what mediates between the target program and the frontend. It consists of several C++ classes that expose properties of WebCore (WebCore/inspector) and JavaScriptCore (JavaScriptCore/inspector) to the inspector. But how is possible that C++ classes and JavaScript classes can exchange information?

The answer to that is the WebKit Remote Debugging Protocol, a JSON formatted protocol than enables communication between the frontend and the backend, and vice versa. This protocol is based on the JSON-RPC 2.0 specification. Currently there’s an attempt, under the RemoteDebug initiative, to standardize all the remote debugging protocols that major browsers use. Remote debugging is a bidirectional protocol: clients send asynchronous requests to the server, the server responds to these request and/or generates notifications. The protocol is divided into a different number of domains.

  • Console: Defines methods and events for interaction with the JavaScript console.
  • Debugger: Exposes JavaScript debugging functions; allows setting and removing breakpoints, stepping through execution, exploring stack traces, etc.
  • DOM: Exposes DOM read/write operations.
  • DOM Debugger: Allows setting breakpoints on particular DOM operations and events. JavaScript execution will stop on these operations as if there was a regular breakpoint set.
  • Network: Allows tracking network activities of the page; exposes information about HTTP and WebSocket requests and responses, their headers, bodies, raw timing, etc.
  • Page: Actions and events related to the inspected page.
  • Runtime: Exposes JavaScript runtime by means of remote evaluation and mirror objects.
  • Timeline: Provides its clients with instrumentation records that are generated during the page runtime.

Each domain defines a number of commands it implements and events it generates. For instance, when setting a breakpoint in the frontend’s console, the following message is sent:

{
    "id": 10, // <-- command sequence number generated by the caller
    "method": "Debugger.setBreakpointByUrl", // <-- protocol method
    "params": { // <-- named parameters map
        "lineNumber": 23,
        "url": "http://www.webkit.org/index.html"
    }
}

For this command, the backend will generate the following response:

{
    "id": 10, // <-- same id as in the command
    "result": { // <-- command result
        "breakpointId": "http://www.webkit.org/index.html:23",
        "locations": [
            {
                "lineNumber": 23,
                "columnNumber": 10
            }
        ]
    }
}

Frontend-to-backend communication: an example

Let’s use the clearMessages command and messagesCleared event defined in the Console domain (JavaScriptCore/inspector/protocol/Console.json) to illustrate how frontend-to-backend communication works:

{
    domain: Console,
    description: Console domain defines methods and events for interaction with the JavaScript console...
    commands: [ 
    {
        name: clearMessages,
        description: Clears console messages collected in the browser.
    },
    events: [ 
    {
        name: messagesCleared,
        description: Issued when console is cleared. This happens either upon <code>clearMessages</code> command or after page navigation.
    }]
}

In the frontend, the LogManager class (Controllers/LogManager.js) sends a clearMessages command through ConsoleAgent:

requestClearMessages()
{
    this._clearMessagesRequested = true;

    ConsoleAgent.clearMessages();
}

Domain commands are implemented in the backend by agents, which are located at WebCore/inspector/agents/ and JavaScriptCore/inspector/agents/, depending on what information available in the backend they need to access. Agents are accessible from the frontend through the Window object. In Main.js (UserInterface/Base/Main.js):

// Enable the Console Agent after creating the singleton managers.
if (window.ConsoleAgent)
    ConsoleAgent.enable();

The glue code that communicates the frontend with the backend is implemented by a set of dispatcher classes. These classes are generated automatically during the build process, out of the definition of the protocol domains. Here is an excerpt of JavaScriptCore/DerivedSources.make:

INSPECTOR_GENERATOR_SCRIPTS = \
    $(JavaScriptCore)/inspector/scripts/codegen/__init__.py \
    $(JavaScriptCore)/inspector/scripts/codegen/cpp_generator_templates.py \
    $(JavaScriptCore)/inspector/scripts/codegen/cpp_generator.py \
    $(JavaScriptCore)/inspector/scripts/codegen/generate_cpp_backend_dispatcher_header.py \
    $(JavaScriptCore)/inspector/scripts/codegen/generate_cpp_backend_dispatcher_implementation.py \
    $(JavaScriptCore)/inspector/scripts/codegen/generate_cpp_frontend_dispatcher_header.py \
    $(JavaScriptCore)/inspector/scripts/codegen/generate_cpp_frontend_dispatcher_implementation.py \
    $(JavaScriptCore)/inspector/scripts/codegen/generate_cpp_protocol_types_header.py \
    $(JavaScriptCore)/inspector/scripts/codegen/generate_cpp_protocol_types_implementation.py \
    $(JavaScriptCore)/inspector/scripts/codegen/generate_js_backend_commands.py \
    $(JavaScriptCore)/inspector/scripts/codegen/generator_templates.py \
    $(JavaScriptCore)/inspector/scripts/codegen/generator.py \
    $(JavaScriptCore)/inspector/scripts/codegen/models.py \
    $(JavaScriptCore)/inspector/scripts/generate-inspector-protocol-bindings.py \
    $(JavaScriptCore_SCRIPTS_DIR)/generate-combined-inspector-json.py \

# Inspector Backend Dispatchers, Frontend Dispatchers, Type Builders
InspectorFrontendDispatchers.h : CombinedDomains.json $(INSPECTOR_GENERATOR_SCRIPTS)
    $(PYTHON) $(JavaScriptCore)/inspector/scripts/generate-inspector-protocol-bindings.py --framework JavaScriptCore --outputDir . ./CombinedDomains.json

This is something common in WebKit, where there are many classes for which there is not existing code in the Source/ directory but are generated automatically and placed at Release/DerivedSources/. The InspectorBackendDispatchers.h (WebKitBuild/Release/DerivedSources/JavaScriptCore/inspector/InspectorBackendDispatchers.h) implements an interface for the domain commands, while the InspectorFrontDispatchers.h implements an interface for the domain notifications:

class JS_EXPORT_PRIVATE ConsoleBackendDispatcherHandler {
public:
    virtual void enable(ErrorString&) = 0;
    virtual void disable(ErrorString&) = 0;
    virtual void clearMessages(ErrorString&) = 0;
    virtual void setMonitoringXHREnabled(ErrorString&, bool in_enabled) = 0;
    virtual void addInspectedNode(ErrorString&, int in_nodeId) = 0;
protected:
    virtual ~ConsoleBackendDispatcherHandler();
};

The dispatching of a command is done by matching a command name with a method name and calling that method.

void ConsoleBackendDispatcher::dispatch(long callId, const String& method, Ref<InspectorObject>&& message)
{
    Ref<ConsoleBackendDispatcher> protect(*this);
  
    if (method == "clearMessages")
        clearMessages(callId, message);
    else if
        ...
    else
        m_backendDispatcher->reportProtocolError(&callId, BackendDispatcher::MethodNotFound, makeString('\'', "Console", '.', method, "' was not found"));
}

InspectorConsoleAgent.cpp (JavaScriptCore/inspector/agents/InspectorConsoleAgent.cpp) implements the clearMessages() method. Once it has finished it will send a notification back to the frontend through the frontendDispatcher class.

void InspectorConsoleAgent::clearMessages(ErrorString&)
{   
    m_consoleMessages.clear();
    m_expiredConsoleMessageCount = 0;
    m_previousMessage = nullptr;

    m_injectedScriptManager->releaseObjectGroup(ASCIILiteral("console"));

    if (m_frontendDispatcher && m_enabled)
        m_frontendDispatcher->messagesCleared();
}

Backend-to-frontend communication: response

The frontend dispatcher is the mechanism by which the backend can send information to the backend. The frontend dispatcher implements the protocol notifications of a domain:

// DO NOT EDIT THIS FILE. It is automatically generated from 
// CombinedDomains.json by the script: Source/JavaScriptCore/inspector/
// scripts/generate-inspector-protocol-bindings.py

class JS_EXPORT_PRIVATE ConsoleFrontendDispatcher {
public:
    ConsoleFrontendDispatcher(FrontendChannel* frontendChannel) : 
        m_frontendChannel(frontendChannel) { }
    void messageAdded(RefPtr<Inspector::Protocol::Console::ConsoleMessage> 
        message);
    void messageRepeatCountUpdated(int count);
    void messagesCleared();
private:
    FrontendChannel* m_frontendChannel;
};

In order to react to backend notifications, the frontend needs to register observers of backend events. This registration happens in Main.js:

// Register observers for events from the InspectorBackend.
if (InspectorBackend.registerConsoleDispatcher)
    InspectorBackend.registerConsoleDispatcher(new WebInspector.ConsoleObserver);
if (InspectorBackend.registerInspectorDispatcher)
    InspectorBackend.registerInspectorDispatcher(new WebInspector.InspectorObserver);
if (InspectorBackend.registerPageDispatcher)
    InspectorBackend.registerPageDispatcher(new WebInspector.PageObserver);

The frontend class ConsoleObserver.js (UserInterface/Protocol/ConsoleObserver.js) will react to messagesCleared event and trigger some programmatic or business logic:

WebInspector.ConsoleObserver = class ConsoleObserver
{
    // Events defined by the "Console" domain.
    messagesCleared()
    {
        WebInspector.logManager.messagesCleared();
    }
};

Localization

Web Inspector strings are localized. Localized strings are stored at localizedStrings.js (Localizations/en.lproj/localizedStrings.js). All UI strings are wrapped by the WebInspector.UIString() method, so they are printed localized:

grep -Rn "WebInspector.UIString(\"Resource" 

The contents of localizedStrings.js are not created manually but by running the script update-webkit-localizable-strings. This script parses all the strings marked to be localized and updates localizedStrings.js.

Bear this in mind if you send a patch with a new or modified string.

Sending a patch

When sending a patch subject should be prefixed by Web Inspector:. Before the patch is sent, some style checkers are run (Tools/Scripts/check-webkit-style), to verify the patch complies Web Inspector coding style. It’s also possible to run the script manually:

    Tools/Scripts/check-webkit-style
    Tools/Scripts/prepare-ChangeLog --bug xxxxx
    # edit WebCore/ChangeLog

As usual, modify the updated ChangeLogs and run webkit-patch upload to send your patch.

Community and resources

There’s a very convenient Wiki with very valuable sources of information such as pointers to blog posts, how to debug, open bugs, etc. The Web Inspector has also it’s own IRC channel at freenode: #webkit-inspector. Most of the work in Web Inspector is carried by Timothy Hatcher, Joseph Pecoraro and Brian Burg, with contributions of other WebKit hackers and other collaborators. I can tell patches and bug reports are very welcomed and reviews go very fast.

I also want to mention this post from Brian Burg that discusses Web Inspector architecture and I used as a basis for this post.

Summary

The Web Inspector is an important component of the WebKit project. It’s an application structured in 3 layers: a frontend, a backend and a target. This abstraction allows detaching the inspector from the target (inspected WebKit browser).

The frontend is a web application composed of HTML + CSS + JavaScript. It implements a Model-View-Controller pattern and makes heavy use of ECMA2015 features. The backend exposes information of WebKit’s WebCore and JavaScriptCore elements.

Communication between frontend and backend is provided by several dispatchers that communicate both parts through the WebKit Remote Debugging Protocol (a JSON-RPC 2.0 based protocol). The protocol defines several domains. Each domain defines commands and events which are the messages the frontend and the backend exchange with each other. Actual implementation of backend commands is provided by agent classes that live in WebCore and JavaScriptCore side. On the frontend side, several observer classes can listen and respond backend notifications.

It has been a rather long post. I hope it can serve as a starting point for anyone interested in understanding or hacking in this important component of the WebKit project.

December 31, 2015 10:00 PM

December 20, 2015

MathML at the Web Engines Hackfest 2015

Frédéric Wang

Hackfest

Two weeks ago, I travelled to Spain to participate to the second Web Engines Hackfest which was sponsored by Igalia and Collabora. Such an event has been organized by Igalia since 2009 and used to be focused on WebkitGTK+. It is great to see that it has now been extended to any Web engines & platforms and that a large percentage of non-igalian developers has been involved this year. If you did not get the opportunity to attend this event or if you are curious about what happened there, take a look at the wiki page or flickr album.

Last day of the hackfest
Photo from @webengineshackfest licensed under Creative Commons Attribution-ShareAlike

I really like this kind of hacking-oriented and participant-driven event where developers can meet face to face, organize themselves in small collaboration groups to efficiently make progress on a task or give passionate talk about what they have recently been working on. The only small bemol I have is that it is still mainly focused on WebKit/Blink developments. Probably, the lack of Mozilla/Microsoft participants is probably due to Mozilla Coincidental Work Weeks happening at the same period and to the proprietary nature of EdgeHTML (although this is changing?). However, I am confident that Igalia will try and address this issue and I am looking forward to coming back next year!

MathML developments

This year, Igalia developer Alejandro G. Castro wanted to work with me on WebKit's MathML layout code and more specifically on his MathML refactoring branch. Indeed, as many people (including Google developers) who have tried to work on WebKit's code in the past, he arrived to the conclusion that the WebKit's MathML layout code has many design issues that make it a burden for the rest of the layout team and too complex to allow future improvements. I was quite excited about the work he has done with Javier Fernández to try and move to a simple box model closer to what exists in Gecko and thus I actually extended my stay to work one whole week with them. We already submitted our proposal to the webkit-dev mailing list and received positive feedback, so we will now start merging what is ready. At the end of the week, we were quite satisfied about the new approach and confident it will facilitate future maintenance and developements :-)

Main room
Photo from @webengineshackfest licensed under Creative Commons Attribution-ShareAlike

While reading a recent thread on the Math WG mailing list, I realized that many MathML people have only vague understanding of why Google (or to be more accurate, the 3 or 4 engineers who really spent some time reviewing and testing the WebKit code) considered the implementation to be unsafe and not ready for release. Even worse, Michael Kholhase pointed out that for someone totally ignorant of the technical implementation details, the communication made some years ago around the "flexbox-based approach" gave the impression that it was "the right way" (indeed, it somewhat did improve the initial implementation) and the rationale to change that approach was not obvious. So let's just try and give a quick overview of the main problems, even if I doubt someone can have good understanding of the situation without diving into the C++ code:

  1. WebKit's code to stretch operator was not efficient at all and was limited to some basic fences buildable via Unicode characters.
  2. WebKit's MathML code violated many layout invariants, making the code unreliable.
  3. WebKit's MathML code relied heavily on the C++ renderer classes for flexboxes and has to manage too many anonymous renderers.

The main security concerns were addressed a long time ago by Martin Robinson and me. Glyph assembly for stretchy operators are now drawn using low-level font painting primitive instead of creating one renderer object for each piece and the preferred width for them no longer depends on vertical metrics (although we still need some work to obtain Gecko's good operator spacing). Also, during my crowdfunding project, I implemented partial support for the OpenType MATH table in WebKit and more specifically the MathVariant subtable, which allows to directly use construction of stretchy operators specified by the font designer and not only the few Unicode constructions.

However, the MathML layout code still modifies the renderer tree to force the presence of anonymous renderers and still applies specific CSS rules to them. It is also spending too much time trying to adapt the parent flexbox renderer class which has at the same time too much features for what is needed for MathML (essentially automatic box alignment) and not enough to get exact placement and measuring needed for high-quality rendering (the TeXBook rules are more complex, taking into account various parameters for box shifts, drops, gaps etc).

During the hackfest, we started to rewrite a clean implementation of some MathML renderer classes similar to Gecko's one and based on the MathML in HTML5 implementation note. The code now becomes very simple and understandable. It can be summarized into four main functions. For instance, to draw a fraction we need:

  • computePreferredLogicalWidths which sets the preferred width of the fraction during the first layout pass, by considering the widest between numerator and denominator.
  • layoutBlock and firstLineBaseline which calculate the final width/height/ascent of the fraction element and position the numerator and denominator.
  • paint which draws the fraction bar.

Perhaps, the best example to illustrate how the complexity has been reduced is the case of the renderer of mmultiscripts/msub/msup/msubsup elements (attaching an arbitrary number of subscripts and superscripts before or after a base). In the current WebKit implementation, we have to create three anonymous wrappers (a first one for the base, a second one for prescripts and a third one for postscripts) and an anonymous wrapper for each subscript/superscript pair, add alignment styles for these wrappers and spend a lot of time maintaining the integrity of the renderer tree when dynamic changes happen. With the new code, we just need to do arithmetic calculations to position the base and script boxes. This is somewhat more complex than the fraction example above but still, it remains arithmetic calculations and we can not reduce any further if we wish quality comparable to TeXBook / MATH rules. We actually take into account many parameters from the OpenType MATH table to get much better placement of scripts. We were able to fix bug 130325 in about twenty minutes instead of fighting with a CSS "negative margin" hack on anonymous renderers.

MathML dicussions

The first day of the hackfest we also had an interesting "breakout session" to define the tasks to work on during the hackfest. Alejandro briefly presented the status of his refactoring branch and his plan for the hackfest. As said in the previous section, we have been quite successful in following this plan: Although it is not fully complete yet, we expect to merge the current changes soon. Dominik Röttsches who works on Blink's font and layout modules was present at the MathML session and it was a good opportunity to discuss the future of MathML in Chrome. I gave him some references regarding the OpenType MATH table, math fonts and the MathML in HTML5 implementation note. Dominik said he will forward the references to his colleagues working on layout so that we can have deeper technical dicussion about MathML in Blink in the long term. Also, I mentioned noto-fonts issue 330, which will be important for math rendering in Android and actually does not depend on the MathML issue, so that's certainly something we could focus on in the short term.

Álex and Fred during the MathML breakout session
Photo from @webengineshackfest licensed under Creative Commons Attribution-ShareAlike

Alejandro also proposed to me to prepare a talk about MathML in Web Engines and exciting stuff happening with the MathML Association. I thus gave a brief overview of MathML and presented some demos of native support in Gecko. I also explained how we are trying to start a constructive approach to solve miscommunication between users, spec authors and implementers ; and gather technical and financial resources to obtain a proper solution. In a slightly more technical part, I presented Microsoft's OpenType MATH table and how it can be used for math rendering (and MathML in particular). Finally, I proposed my personal roadmap for MathML in Web engines. Although I do not believe I am a really great speaker, I received positive feedback from attendees. One of the thing I realized is that I do not know anything about the status and plan in EdgeHTML and so overlooked to mention it in my presentation. Its proprietary nature makes hard for external people to contribute to a MathML implementation and the fact that Microsoft is moving away from ActiveX de facto excludes third-party plugin for good and fast math rendering in the future. After I went back to Paris, I thus wrote to Microsoft employee Christian Heilmann (previously working for Mozilla), mentioning at least the MathML in HTML5 Implementation Note and its test suite as a starting point. MathML is currently on the first page of the most voted feature requested for Microsoft Edge and given the new direction taken with Microsoft Edge, I hope we could start a discussion on MathML in EdgeHTML...

Conclusion

This was a great hackfest and I'd like to thank again all the participants and sponsors for making it possible! As Alejandro wrote to me, "I think we have started a very interesting work regarding MathML and web engines in the future.". The short term plan is now to land the WebKit MathML refactoring started during the hackfest and to finish the work. I hope people now understand the importance of fonts with an OpenType MATH table for good mathematical rendering and we will continue to encourage browser vendors and font designers to make such fonts wide spread.

The new approach for WebKit MathML support gives good reason to be optmimistic in the long term and we hope we will be able to get high-quality rendering. The fact that the new approach addresses all the issues formulated by Google and that we started a dialogue on math rendering, gives several options for MathML in Blink. It remains to get Microsoft involved in implementing its own OpenType table in EdgeHTML. Overall, I believe that we can now have a very constructive discussion with the individuals/companies who really want native math support, with the Math WG members willing to have their specification implemented in browsers and with the browser vendors who want a math implementation which is good, clean and compatible with the rest of their code base. Hopefully, the MathML Association will be instrumental in achieving that. If everybody get involved, 2016 will definitely be an exciting year for native MathML in Web engines!

By fredw at December 20, 2015 04:15 PM

December 15, 2015

More Responsive Tapping on iOS

Surfin’ Safari

WebKit on iOS has a 350 millisecond delay before single taps activate links or buttons. WebKit has this delay because we also allow users to double tap to zoom, which is a great way to zoom in on content that is well-sized for large desktop displays, but appears too small on mobile devices. However, when a user has tapped once, WebKit cannot tell if the user intends on tapping again to trigger a double tap gesture. Since double tapping is defined as two taps within a short time interval (350ms), WebKit must wait for this time interval to pass before we’re sure that the user intended to tap only once.

We know that responsive tapping is really important to web developers — so much so that many are willing to employ JavaScript frameworks to avoid the delay using touch handlers. Instead of waiting for WebKit to fire a click after a delay, these libraries prevent the default behavior of the touchend event and call click() immediately so that the element is clicked the moment the user stops touching the element. While this may make a link feel fast, it can also reduce responsiveness in other ways, including page load time and scrolling. To address this, we baked fast tapping optimizations into WebKit so well-scaled mobile web pages will be able to achieve responsive tapping out of the box without the drawbacks of third-party frameworks.

On web pages optimized for mobile viewports, elements such as links and form controls are scaled to fit well on smaller screens. As such, double tapping on these elements increases the page scale by only a small amount. Since double tapping provides little value on these web pages, we’ve implemented a mechanism for removing the delay for single taps by disabling double tap gestures.

The following examples show the amount of time required to tap a button 10 times in a page with <meta name="viewport" content="initial-scale=1.0, width=device-width">, with and without the fast-tap optimization. The left video shows the old behavior, while the right video shows the new behavior.


When Fast-Tap Optimizations Apply

When content is scaled to fit well on a mobile device, WebKit disables double tapping in favor of fast single tapping. Using the viewport meta tag, there are a few ways WebKit determines whether content is well-proportioned, for both scalable and unscalable web pages.

Since users cannot zoom in unscalable viewports (using meta tags with user-scalable=no or a minimum-scale equal to the maximum-scale) WebKit assumes that if a page has an unscalable viewport, all content is already well-proportioned relative to the screen. On these pages, WebKit removes the ability to double-tap on elements altogether and dispatches all taps immediately as clicks. We implemented this behavior in Bug #149968.

However, we believe web pages should be scalable when possible. For this reason, viewports that have width=device-width will have fast single tapping when the user is at initial scale. We added this optimization in Bug #150604.

Styling Fast-Tap Behavior

Even if a web page cannot use one of the viewport configurations above, it can still opt in to the fast-tap optimization. We’ve implemented the manipulation value of the new touch-action CSS property in Bug #149854. By default, elements have touch-action: auto, which indicates that WebKit may enable any touch behaviors such as panning, pinching and double-tapping. Putting touch-action: manipulation; on a clickable element makes WebKit consider touches that begin on the element only for the purposes of panning and pinching to zoom. This means WebKit does not consider double-tap gestures on the element, so single taps are dispatched immediately. Single taps on an element become fast when any of the element’s ancestors have the touch-action: manipulation. Note that putting touch-action: manipulation on any node opts all children of the node into the fast-tap optimization.

Additionally, single taps on an element with touch-action: manipulation are fast for all zoom scales, whereas our optimizations on device-width viewports only trigger when the viewport is at initial scale.

If you have questions, comments or other feedback regarding fast-tapping in mobile Safari, feel free to reach out to Jon Davis @jonathandavis or WebKit @webkit on Twitter, or email me directly at wenson_hsieh@apple.com.

By Wenson Hsieh at December 15, 2015 02:00 PM

December 01, 2015

Breakpoint Options

Surfin’ Safari

In his classic book on graphics and assembly language, Michael Abrash writes, “the best optimizer is between your ears”. The best debugger is also between your ears, and the best development tools should provide powerful and flexible features to help developers to analyze program flow and locate errors.

Web Inspector provides a number of options for customizing breakpoints, many of which will be familiar to users of modern IDEs like Xcode and Microsoft Visual Studio. This post will focus on the Debugger tab, customizing breakpoints with conditions and actions for logging and executing scripts, and creating probe expressions.

The Debugger Tab

The debugger tab’s left sidebar lists breakpoints by source file, including two global breakpoints for pausing automatically whenever an exception is thrown.

The following options are provided for filtering the breakpoint list:

Enable/disable all breakpoints ⌘Y Toggle all breakpoints on or off, including global breakpoints
Breakpoint filter Only show scripts with breakpoints
Issues filter Only show scripts with exceptions, console errors, and warnings
Text filter Only show files matching the filter text

A breakpoint’s color in the sidebar or text editor reflects its current state:

Enabled Default
Auto-continue Breakpoint will not pause the debugger when evaluated
Disabled Breakpoint will not be evaluated
Inactive Unresolved breakpoint, or all breakpoints disabled

Right-clicking a breakpoint in the sidebar or source code view’s gutter opens the breakpoint editor. From here we can customize the runtime behavior of breakpoints by setting conditions and actions.

Conditional Breakpoints

A conditional breakpoint is only triggered when the specified conditional expression evaluates to true. The expression is evaluated in the breakpoint’s scope. The condition field supports autocompletion for JavaScript built-in types.

Breakpoint evaluation can also be controlled by setting an ignore count, causing the debugger to ignore the breakpoint a number of times before stopping. Both are useful when debugging code which runs inside a loop, and event callbacks that fire frequently.

If both an ignore count and conditional expression are specified, the ignore count takes precedence. To demonstrate, consider the following breakpoint in the body of a loop:

for (var i = 0; i < 10; ++i) {
   // do some work
}

The breakpoint action is triggered for odd values of i, starting with the sixth loop iteration. Running the program results in the following console output:

5
7
9

Breakpoint Actions

Stopping script execution to inspect program state isn’t always practical. Repeatedly pausing in the debugger can be distracting, and when debugging mouse event handlers or asynchronous callbacks it may even introduce unexpected program behavior. Instead of simply pausing the current script, breakpoints can run one or more actions when triggered.

  1. Right-click the breakpoint and choose Edit Breakpoint
  2. Click Add Action
  3. Set the new action’s type
  4. Set to automatically continue to avoid pausing in the debugger
  5. Optional. Add additional actions

The following actions are available:

Log Message Default. Log a plain text message to the console
Evaluate JavaScript Execute custom JavaScript directly in the inspected page
Play a Sound Beep
Probe Expression Sample the value of an expression

Evaluate JavaScript

The Evaluate JavaScript action is a powerful and flexible tool. Like conditional expressions, the script action has access to any variables, functions, and objects in the breakpoint’s scope. Script actions are often used to log data to the console, but they can do much more.

Script actions can be used to modify program state at runtime, output the current call stack, measure execution time, and even record timeline profiles. The Console API contains many functions that are well-suited for these tasks:

console.log(a, b, ...) Output arguments to the Console
console.profile(name) Start profiling (creates a timeline recording)
console.profileEnd() Stop profiling
console.table(object) Output an object to the Console, formatted as a table:
console.time(name) Start a timer
console.timeEnd(name) Stop a timer, output timestamp to the Console
console.trace() Output the call stack to the Console:

Probe Expressions

Probe expressions make it easy to observe state changes in an interactive program, without being interrupted by the debugger or sifting through console output. Adding a Probe Expression action creates a Breakpoint Probe, which can contain multiple expressions.

In the example below, a breakpoint with two probe expressions has been added to the keydown handler in a Tetris game:

After creating the breakpoint, a Probes panel is added to the debugger’s right sidebar. Probe samples are added to the data table at runtime each time the breakpoint is evaluated:

Probes can be edited directly from the sidebar, without opening the Edit Breakpoint dialog. Each probe’s entry in the sidebar panel has a toolbar with the following commands:

Add a new probe expression (creates a breakpoint action)
Clear probe samples
Delete the probe (removes associated breakpoint actions)

Conclusion

The Debugger sidebar is under active development, and we’re currently exploring new ways of organizing breakpoints to make debugging workflows smoother.

The latest features and enhancements can be found in WebKit Nightly Builds. Help us make Web Inspector even better by sending us feedback on Twitter (@xeenon@jonathandavis, @matt_a_baker), or by filing a bug.

By Matt Baker at December 01, 2015 02:00 PM

November 05, 2015

Targeting Domains with Content Blockers

Surfin’ Safari

Content Blockers extensions are enjoying a tremendous success on iOS and Mac. Developers are being creative with the API and users love the speed and privacy benefits. If you are interested in writing your own Content Blocker, I suggest starting with the introduction and the official documentation. Content Blocker Extensions are different than traditional Safari extensions. Rather than have the engine (in this case, WebKit) ask extension code how to behave, the extension tells the engine how to behave in advance. Each Content Blocker extension defines a list of rules to follow, in order. Each rule can be activated only when certain conditions are met. Those conditions are defined in the “trigger” of each rule. In this blog post, we are going to explore one particularly popular type of triggers: activation based on domain names.

Domain names and URL-filter

One of the most popular kind of trigger is activated based on matching certain domains. Typically, the author wants to prevent the browser from disclosing information to those domains. The trigger specification does not have an explicit way of matching requests of a certain domain. Instead, the rule “url-filter” allow matching of a regular expression over the complete URL of every request. There are many ways to write a regular expression that match a domain name. In general, the runtime performance is great regardless of how the expression is written. What happens in the backend is that the Content Blocker engine compiles the rules into a small number of optimized finite state machines. Different ways of writing the same expression generates the same machines. Some definitions are more generic than others; it is the engine’s job to optimize them. The bad news is different regular expressions can cause very different compile times. Some extensions compile in milliseconds while others compile in seconds. The more work we give to the optimizer, the longer the compilation process takes. On iOS, this translates into it taking longer after a user turns a content blocker on for it to apply to loaded pages. In the following sections, we will see how the definition of URL-filters affects the compile time. The running example is to match “a-tracker.com” and all its subdomains.

Generic domain matching

Before going through the Content Blocker engine, URLs are canonicalized. The most direct way of matching a domain is to match the exact sequence of characters that can appear in a valid canonical URL. It would look something like this:

[
    {
        "trigger": {
            "url-filter": "^[a-z][-+.0-9a-z]*:/+([!$%&'()*+,-.0-9:;=a-z_~]*@)?([-%0-9a-z_]+\.)*a-tracker\.com[:/]"
        },
        "action": {
            "type": "block"
        }
    }
]

Compiling fifty thousand patterns of this form takes about 10 seconds in the worst case. A bit long. The reason compiling such pattern takes a long time is its complexity. The optimizer has to spend a fair amount of time figuring the most efficient way to handle all the character sets.

Let’s simplify the expression for http-like URLs

The URL specifications give plenty of freedom on what can be defined. In most cases, we only care about the subset that matches URLs with the HTTP scheme. We can improve the pattern by making the structure stricter! First, the scheme and domain names are always lowercase in a canonical URL. We can change the trigger to:

"trigger": {
    "url-filter": "^[a-z][-+.0-9a-z]*://+([!$%&'()*+,-.0-9:;=A-Za-z_~]*@)?([-%0-9a-z_]+\.)*a-tracker\.com[:/]",
    "url-filter-is-case-sensitive":true
}

This is already 20% faster! Next, let’s simplify the scheme definition. When compiling “^[a-z][-+.0-9a-z]*://+“, the compiler generates code matching specifically the characters defined here. Matching the first character is easy, it just needs to be between “a” and “z” inclusive. Matching the next characters is a bit more complicated because more possibilities need to be tested. There are two options to make that part trivial. If it is okay to restrict the pattern to HTTP/HTTPS, using the pattern “^https?://+” is effective. If you need to match any scheme, the pattern “^[^:]+://” is a good option. Instead of looking for a character in the range from “a” to “z”, the compiler can assume the input can be skipped until the first character “:”. This is a simpler operation, which makes it simpler to optimize. We can use the same idea to match anything in front of the domain. Instead of matching the username, password and subdomains exactly, we can just exclude anything past the domain name. We end up with a simplified expression like this:

"trigger": {
    "url-filter": "^[^:]+://+([^:/]+\.)?a-tracker\.com[:/]",
    "url-filter-is-case-sensitive":true
}

Compiling fifty thousand patterns like this takes 2.7 seconds in the worst case, almost four times faster than the original pattern!

Conclusion

The Content Blocker compiler is there to optimize the performance of running Content Blockers. Such optimizations can take time. If your extension has several thousand rules, it is worth looking into using simple regular expressions to reduce the compile time. Doing so ensures that users can enjoy the benefits of your extension as soon as it is activated. For domain matching, my advice is to use triggers of the form:

"trigger": {
    "url-filter": "^[^:]+://+([^:/]+\.)?domain-to-match\.tld[:/]",
    "url-filter-is-case-sensitive":true
}

For other cases, keep in mind the following:

  • Use “url-filter-is-case-sensitive” when possible. It halves the number of characters to consider.
  • Do not forget to escape special characters. In particular “.” can make a pattern much more generic than it needs to.
  • Sometimes, it is simpler to define what character should not match instead of listing everything that can match.

I would be happy to answer questions about anything related to Content Blockers in WebKit. You can find me on twitter at @awfulben. Brian can answer questions regarding the use of Content Blockers in Safari. As usual, you can also get in touch with @jonathandavis with any other questions.

By Benjamin Poulain at November 05, 2015 02:00 PM

October 26, 2015

Introducing Slot-Based Shadow DOM API

Surfin’ Safari

We’re pleased to announce that basic support for the new slot-based shadow DOM API we proposed in April is now available in the nightly builds of WebKit after r190680. Shadow DOM is a part of Web Components, a set of specifications that were initially proposed by Google to enable the creation of reusable widgets and components on the Web. Shadow DOM, in particular, provides a lightweight encapsulation for DOM trees by allowing a creation of a parallel tree on an element called a “shadow tree” that replaces the rendering of the element without modifying the underlying DOM tree. Because a shadow tree is not an ordinary child of the “host” element to which it is attached, users of components cannot accidentally poke into it. Style rules are also scoped, meaning that CSS rules defined outside of a shadow tree do not apply to elements inside the shadow tree and rules defined inside the shadow tree do not apply to elements outside of it.

Style Isolation

One major benefit of using shadow DOM is style isolation. To see how, let’s say we want to create a custom progress bar. We can use two nested div’s to show the bar and another div with the text to show the percentage as follows:

<style>
.progress { position: relative; border: solid 1px #000; padding: 1px; width: 100px; height: 1rem; }
.progress > .bar { background: #9cf; height: 100%; }
.progress > .label { position: absolute; top: 0; left: 0; width: 100%;
    text-align: center; font-size: 0.8rem; line-height: 1.1rem; }
</style>
<template id="progress-bar-template">
    <div class="progress" role="progressbar" aria-valuenow="0" aria-valuemin="0" aria-valuemax="100">
        <div class="bar"></div>
        <div class="label">0%</div>
    </div>
</template>
<script>
function createProgressBar() {
    var fragment = document.getElementById('progress-bar-template').content.cloneNode(true);
    var progressBar = fragment.querySelector('div');
    progressBar.updateProgress = function (newPercentage) {
        this.setAttribute('aria-valuenow', newPercentage);
        this.querySelector('.label').textContent = newPercentage + '%';
        this.querySelector('.bar').style.width = newPercentage + '%';
    }
    return progressBar;
}
</script>

Note the use of the template element, which allows authors to include a snippet of HTML that can be instantiated later by cloning the content. This was the first feature of Web Components we implemented in WebKit that later got merged into the HTML5 specification. A template element can appear anywhere in a document (e.g. between table and tr elements), and content inside template elements is inert and does not run scripts or load images and other types of subresources. Then the user of this custom progress bar can instantiate it and update the progress as follows:

var progressBar = createProgressBar();
container.appendChild(progressBar);
...
progressBar.updateProgress(10);

The problem with this progress bar implementation is that its two internal divs are freely accessible to its users and its style rules are not scoped to the progress bar. For example, the style rules defined for the progress bar will apply to content outside the progress bar with the class name progress:

<section class="project">
    <p class="progress">Pending an approval</p>
</section>

Similarly, style rules defined for other elements could override rules in the progress bar:

<style>
.label { font-weight: bold; }
</style>

While we could work around these problems by using a custom element name such as custom-progressbar to scope rules and then initialize all other properties by all: initial, Shadow DOM provides a much more elegant solution. The idea here is to introduce an encapsulation layer at the outer div so that users of the progress bar don’t see its internals (such as divs created for the label and the bar) and styles defined for the progress bar don’t interfere with the rest of the page and vice versa. To do that, we first create a ShadowRoot on the progress bar by calling attachShadow({mode: 'closed'}), and then append various nodes needed for its implementation under it. Let’s say we’re still using a div to “host” this shadow root, then we can create a new div and attach a shadow root as follows:

<template id="progress-bar-template">
    <style>
        .progress { position: relative; border: solid 1px #000; padding: 1px; width: 100px; height: 1rem; }
        .progress > .bar { background: #9cf; height: 100%; }
        .progress > .label { position: absolute; top: 0; left: 0; width: 100%;
            text-align: center; font-size: 0.8rem; line-height: 1.1rem; }
    </style>
    <div class="progress" role="progressbar" aria-valuenow="0" aria-valuemin="0" aria-valuemax="100">
        <div class="bar"></div>
        <div class="label">0%</div>
    </div>
</template>
<script>
function createProgressBar() {
    var progressBar = document.createElement('div');
    var shadowRoot = progressBar.attachShadow({mode: 'closed'});
    shadowRoot.appendChild(document.getElementById('progress-bar-template').content.cloneNode(true));
    progressBar.updateProgress = function (newPercentage) {
        shadowRoot.querySelector('.progress').setAttribute('aria-valuenow', newPercentage);
        shadowRoot.querySelector('.label').textContent = newPercentage + '%';
        shadowRoot.querySelector('.bar').style.width = newPercentage + '%';
    }
    return progressBar;
}
</script>

Notice that the style element is inside the template element and cloned into the shadow root along with the divs. This allows the style rules defined inside the shadow root to be scoped. Style rules defined outside a shadow root do not apply to elements inside the shadow root either. Tip: while debugging your code, you may find it helpful to use shadow DOM’s open mode so that you can access the newly created shadow root via the shadowRoot property of the host element. e.g. {mode: DEBUG ? 'open' : 'closed'}

Composition with Slots

At this point, you might be wondering why this had to be done in DOM instead of CSS. Styling is a presentational concept, so why should we add new elements to the DOM? In fact, the first public working draft of the CSS Scoping Module Level 1 defines the @scope rule, which enables exactly that. So why did we need to add another mechanism to isolate styles? One motivation was to allow elements used in the implementation of components to be hidden from node traversal APIs such as querySelectorAll and getElementsByTagName. Because nodes inside a shadow root are not found by these APIs by default, users of components that utilize shadow DOM do not need to worry about how each component is implemented. Each component is presented as an opaque element whose implementation details are encapsulated in its shadow DOM. Note that shadow DOM does not provide a cross-origin iframe-like security boundary. Scripts can easily bypass the shadow DOM boundary if needed. Another reason we need a DOM-based solution is for composition. Let’s say we have a list of contacts:

<ul id="contacts">
    <li>
        Commit Queue
        (<a href="mailto:commit-queue@webkit.org">commit-queue@webkit.org</a>)<br>
        One Infinite Loop, Cupertino, CA 95014
    </li>
    <li>
        Niwa, Ryosuke
        (<a href="mailto:rniwa@webkit.org">rniwa@webkit.org</a>)<br>
        Two Infinite Loop, Cupertino, CA 95014
    </li>
</ul>

and we would like to add a fancy UI for each contact’s information in the list when scripts are enabled:

Instead of copying all of this text over to our own shadow DOM, we can use named slots to render the text elsewhere in our shadow DOM without modifying the DOM as follows:

<template id="contact-template">
    <style>
        :host { border: solid 1px #ccc; border-radius: 0.5rem; padding: 0.5rem; margin: 0.5rem; }
        b { display: inline-block; width: 5rem; }
    </style>
    <b>Name</b>: <slot name="fullName"><slot name="firstName"></slot> <slot name="lastName"></slot></slot><br>
    <b>Email</b>: <slot name="email">Unknown</slot><br>
    <b>Address</b>: <slot name="address">Unknown</slot>
</template>
<script>
window.addEventListener('DOMContentLoaded', function () {
    var contacts = document.getElementById('contacts').children;
    var template = document.getElementById('contact-template').content;
    for (var i = 0; i < contacts.length; i++)
        contacts[i].attachShadow({mode: 'closed'}).appendChild(template.cloneNode(true));
});
</script>

Conceptually, slots are holes in a shadow DOM that will be filled by children of its host element. Each element can be assigned into a slot of a specific name by the slot attribute as follows:

<ul id="contacts">
    <li>
        <span slot="fullName">Commit Queue</span>
        (<a slot="email" href="mailto:commit-queue@webkit.org">commit-queue@webkit.org</a>)<br>
        <span slot="address">One Infinite Loop, Cupertino, CA 95014</span>
    </li>
</ul>

Here, we’re attaching a shadow root to the li, and each span with a slot attribute is assigned to the slot of the same name inside the shadow DOM. Let’s take a closer look at the shadow DOM template:

<b>Name</b>:
<slot name="fullName">
    <slot name="firstName"></slot>
    <slot name="lastName"></slot>
</slot><br>
<b>Email</b>: <slot name="email">Unknown</slot><br>
<b>Address</b>: <slot name="address">Unknown</slot>

In this template, we have slots named fullName, which contains two other slots named firstName and lastName, and two additional slots named email and address. The fullName slot is taking the advantage of fallback content, and showing firstName and lastName only if there were no nodes assigned to the fullName slot. Even though there is exactly one node assigned to each slot in this example, multiple elements with the same slot attribute value can be assigned to a single slot, and they will appear in the order they appeared as the children of the host element. You can also use an unnamed default slot that will be filled by all of the host’s children that don’t have a slot attribute specified. When a Web browser renders this content, the content of the li element is replaced by the shadow DOM, and slots inside of it are replaced by their assigned node as if rendering the following DOM instead:

<ul id="contacts">
    <li>
        <!--shadow-root-start-->
            <b>Name</b>:
            <slot name="fullName">
                <!--slot-content-start-->
                    <span slot="fullName">Commit Queue</span>
                <!--slot-content-end-->
            </slot><br>
            <b>Email</b>:
            <slot name="email">
                <!--slot-content-start-->
                    <a slot="email" href="mailto:commit-queue@webkit.org">commit-queue@webkit.org</a>
                <!--slot-content-end-->
            </slot><br>
            <b>Address</b>:
            <slot name="address">
                <!--slot-content-start-->
                    <span slot="address">One Infinite Loop, Cupertino, CA 95014</span>
                <!--slot-content-end-->
            </slot>
        <!--shadow-root-end-->
    </li>
</ul>

As you can see, slot-based composition is a powerful tool that allows widgets to pull in the page content without cloning or modifying the DOM. With it, widgets can respond to changes made to its child nodes without MutationObservers or an explicit notification via script. In essence, composition turns the DOM into a communication medium between components.

Styling the Host Element

There is one more thing to note in the previous example, which had a mysterious pseudo-class :host:

<template id="contact-template">
    <style>
        :host { border: solid 1px #ccc; border-radius: 0.5rem; padding: 0.5rem; margin: 0.5rem; }
        b { display: inline-block; width: 5rem; }
    </style>
...
</template>

This pseudo class, as its name suggests, matches the host element of the shadow DOM in which this rule appears. By default, author style rules defined outside the shadow DOM have a higher precedence over rules defined in the shadow DOM. This allows a component to define its “default style”, and let users of the component override as needed. In addition, a component can use !important to force a certain style, such as width and display type, without which it cannot function properly with. Any !important rules defined inside a shadow DOM have a higher precedence over regular and !important rules defined outside the shadow DOM.

Future Work

There is still a lot of work left for Web Components. For styling, we would like to allow styling nodes assigned to a slot inside a shadow DOM. There is also a desire for components to respond to the document theme as well as exposing a stylable part to their users like CSS pseudo elements. In the longer term, we would like to see an imperative DOM API to manipulate slot assignments as we proposed a while ago. To complement shadow DOM, we’re also interested in custom elements. The custom elements API allows authors to associate a JavaScript class with a particular element name in HTML documents, and it’s a great way to attach shadow DOM and other custom behaviors idiomatically. Unfortunately, there are a few conflicting proposals on when and how to create a custom element. To help steer the discussion in W3C, we’re planning to prototype it in WebKit. For packaging and delivering Web Components, we’ve been working on ES6 modules. Like Mozilla, we believe modules will radically change the way authors strcuture their pages. We would also like to eventually design an API to create a fully isolated web component with an iframe-like security boundary on top of shadow DOM and custom elements. To conclude, we’re really excited about bringing a major feature of Web Components to WebKit, and we’ll keep you posted about more features coming your way. If you have any questions, please feel free to contact myself, @WebKit, or Jon Davis.

By Ryosuke Niwa at October 26, 2015 02:00 PM

December 15, 2014

Web Engines Hackfest 2014

Gustavo Noronha

For the 6th year in a row, Igalia has organized a hackfest focused on web engines. The 5 years before this one were actually focused on the GTK+ port of WebKit, but the number of web engines that matter to us as Free Software developers and consultancies has grown, and so has the scope of the hackfest.

It was a very productive and exciting event. It has already been covered by Manuel RegoPhilippe Normand, Sebastian Dröge and Andy Wingo! I am sure more blog posts will pop up. We had Martin Robinson telling us about the new Servo engine that Mozilla has been developing as a proof of concept for both Rust as a language for building big, complex products and for doing layout in parallel. Andy gave us a very good summary of where JS engines are in terms of performance and features. We had talks about CSS grid layouts, TyGL – a GL-powered implementation of the 2D painting backend in WebKit, the new Wayland port, announced by Zan Dobersek, and a lot more.

With help from my colleague ChangSeok OH, I presented a description of how a team at Collabora led by Marco Barisione made the combination of WebKitGTK+ and GNOME’s web browser a pretty good experience for the Raspberry Pi. It took a not so small amount of both pragmatic limitations and hacks to get to a multi-tab browser that can play youtube videos and be quite responsive, but we were very happy with how well WebKitGTK+ worked as a base for that.

One of my main goals for the hackfest was to help drive features that were lingering in the bug tracker for WebKitGTK+. I picked up a patch that had gone through a number of iterations and rewrites: the HTML5 notifications support, and with help from Carlos Garcia, managed to finish it and land it at the last day of the hackfest! It provides new signals that can be used to authorize notifications, show and close them.

To make notifications work in the best case scenario, the only thing that the API user needs to do is handle the permission request, since we provide a default implementation for the show and close signals that uses libnotify if it is available when building WebKitGTK+. Originally our intention was to use GNotification for the default implementation of those signals in WebKitGTK+, but it turned out to be a pain to use for our purposes.

GNotification is tied to GApplication. This allows for some interesting features, like notifications being persistent and able to reactivate the application, but those make no sense in our current use case, although that may change once service workers become a thing. It can also be a bit problematic given we are a library and thus have no GApplication of our own. That was easily overcome by using the default GApplication of the process for notifications, though.

The show stopper for us using GNotification was the way GNOME Shell currently deals with notifications sent using this mechanism. It will look for a .desktop file named after the application ID used to initialize the GApplication instance and reject the notification if it cannot find that. Besides making this a pain to test – our test browser would need a .desktop file to be installed, that would not work for our main API user! The application ID used for all Web instances is org.gnome.Epiphany at the moment, and that is not the same as any of the desktop files used either by the main browser or by the web apps created with it.

For the future we will probably move Epiphany towards this new era, and all users of the WebKitGTK+ API as well, but the strictness of GNOME Shell would hurt the usefulness of our default implementation right now, so we decided to stick to libnotify for the time being.

Other than that, I managed to review a bunch of patches during the hackfest, and took part in many interesting discussions regarding the next steps for GNOME Web and the GTK+ and Wayland ports of WebKit, such as the potential introduction of a threaded compositor, which is pretty exciting. We also tried to have Bastien Nocera as a guest participant for one of our sessions, but it turns out that requires more than a notebook on top of a bench hooked up to   a TV to work well. We could think of something next time ;D.

I’d like to thank Igalia for organizing and sponsoring the event, Collabora for sponsoring and sending ChangSeok and myself over to Spain from far away Brazil and South Korea, and Adobe for also sponsoring the event! Hope to see you all next year!

Web Engines Hackfest 2014 sponsors: Adobe, Collabora and Igalia

Web Engines Hackfest 2014 sponsors: Adobe, Collabora and Igalia

By kov at December 15, 2014 11:20 PM

December 08, 2014

How to build TyGL

University of Szeged

This is a follow-up blog post of our announcement of TyGL - the 2D-accelerated GPU rendering port of WebKit.

We have been received lots of feedback about TyGL and we would like to thank you for all questions, suggestions and comments. As we promised lets get into some technical details.

read more

By szilard.ledan at December 08, 2014 12:47 PM

November 12, 2014

Announcing the TyGL-WebKit port to accelerate 2D web rendering with GPU

University of Szeged

We are proud to announce the TyGL port (link: http://github.com/szeged/TyGL) on the top of EFL-WebKit. TyGL (pronounced as tigel) is part of WebKit and provides 2D-accelerated GPU rendering on embedded systems. The engine is purely GPU based. It has been developed on and tested against ARM-Mali GPU, but it is designed to work on any GPU conforming to OpenGL ES 2.0 or higher.

The GPU involvement on future graphics is inevitable considering the pixel growth rate of displays, but harnessing the GPU power requires a different approach than CPU-based optimizations.

read more

By zoltan.herczeg at November 12, 2014 02:18 PM

October 22, 2014

Fuzzinator reloaded

University of Szeged

It's been a while since I last (and actually first) posted about Fuzzinator. Now I think that I have enough new experiences worth sharing.

More than a year ago, when I started fuzzing, I was mostly focusing on mutation-based fuzzer technologies since they were easy to build and pretty effective. Having a nice error-prone test suite (e.g. LayoutTests) was the warrant for fresh new bugs. At least for a while.

read more

By renata.hodovan at October 22, 2014 10:38 PM

September 25, 2014

Measuring ASM.JS performance

University of Szeged

What is ASM.JS?

Now that mobile computers and cloud services become part of our lives, more and more developers see the potential of the web and online applications. ASM.JS, a strict subset of JavaScript, is a technology that provides a way to achieve near native speed in browsers, without the need of any plugin or extension. It is also possible to cross-compile C/C++ programs to it and running them directly in your browser.

In this post we will compare the JavaScript and ASM.JS performance in different browsers, trying out various kinds of web applications and benchmarks.

read more

By matyas.mustoha at September 25, 2014 10:40 AM

August 28, 2014

CSS Shapes now available in Chrome 37 release

Adobe Web Platform

Support for CSS Shapes is now available in the latest Google Chrome 37 release.

chromelogo

What can I do with CSS Shapes?

CSS Shapes lets you think out of the box! It gives you the ability to wrap content outside any shape. Shapes can be defined by geometric shapes, images, and even gradients. Using Shapes as part of your website design takes a visitor’s visual and reading experience to the next level. If you want to start with some tutorials, please go visit Sarah Soueidan’s article about Shapes.

Demo

The following shapes use case is from the Good Looking Shapes Gallery blog post.

Without CSS Shapes
the_11_guesses_no_shapes
With CSS Shapes
the_11_guesses_shapes

In the first picture, we don’t use CSS Shapes. The text wraps around the rectangular image container, which leads to a lot of empty space between the text and the visible part of the image.

In the second picture, we use CSS Shapes. You can see the wrapping behavior around the image. In this case the white parts of the image are transparent, thus the browser can automatically wrap the content around the visible part, which leads to this nice and clean, visually more appealing wrapping behavior.

How do I get CSS Shapes?

Just update your Chrome browser to the latest version from the Chrome/About Google Chrome menu, or download the latest stable version from https://www.google.com/chrome/browser/.

I’d like to thank the collaboration of WebKit and Blink engineers, and everyone else in the community who has contributed to this feature. The fact that Shapes is shipping in two production browsers — Chrome 37 now and Safari 8 later this year — is the upshot of the open source collaboration between the people who believe in a better, more expressive web. Although Shapes will be available in these browsers, you’ll need another solution for the other browsers. The CSS Shapes Polyfill is one method of achieving consistent behavior across browsers.

Where should I start?

For more info about CSS Shapes, please check out the following links:

Let us know your thoughts or if you have nice demos, here or on Twitter: @AdobeWeb and @ZoltanWebKit.

By Zoltan Horvath at August 28, 2014 05:12 PM

May 13, 2014

Good-Looking Shapes Gallery

Adobe Web Platform

As a modern consumer of media, you rarely crack open a magazine or a pamphlet or anything that would be characterized as “printed”. Let me suggest that you take a walk on the wild side. The next time you are in a doctor’s office, or a supermarket checkout lane, or a library, thumb though a magazine. Most of the layouts you’ll find inside can also be found on the web, but not all of them. Layouts where content hugs the boundaries of illustrations are common in print and rare on the web. One of the reasons non-rectangular contour-hugging layouts are uncommon on the web is that they are difficult to produce.

They are not difficult to produce anymore.

The CSS Shapes specification is now in the final stages of standardization. This feature enables flowing content around geometric shapes (like circles and polygons), as well as around shapes defined by an image’s alpha channel. Shapes make it easy to produce the kinds of layouts you can find in print today, with all the added flexibility and power that modern online media affords. You can use CSS Shapes right now with the latest builds of WebKit and Blink based browsers, like Safari and Chrome.

Development of CSS Shapes has been underway for about two years, and we’ve been regularly heralding its progress here. Many of those reports have focused on the evolution of the spec and implementations, and they’ve included examples that emphasized basics over beauty. This article is an attempt to tilt the balance back towards good-looking. Listed below are simple shapes demos that we think look pretty good. Everyone on Adobe’s CSS Shapes engineering team contributed at least one.

There’s a live CodePen.io version of each demo in the gallery. Click on the demo screenshot or one of the handy links to take a look. You’ll want to view the demos with a browser that supports Shapes and you’ll need to enable CSS Shapes in that browser. For example you can use a nightly build of the Safari browser or you can enable shapes in Chrome or Chrome Canary like this:

  1. Copy and paste chrome://flags/#enable-experimental-web-platform-features into the address bar, then press enter.
  2. Click the ‘Enable’ link within that section.
  3. Click the ‘Relaunch Now’ button at the bottom of the browser window.

A few of the demos use the new Shapes Polyfill and will work in most browsers.

And now, without further ado, please have a look through our good-looking shapes gallery.


Ozma of Oz

ozma-demo-screenshot

This demo reproduces the layout style that opens many of the chapters of the L. Frank Baum books, including Ozma of Oz.  The first page is often dominated by an illustration on the left or right. The chapter’s text conforms to the illustration, but not too tightly. The books were published over 100 years ago and they still look good print.  With CSS Shapes they can still look good on the web.

Top Cap

topcap-demo-screenshot

The conventional “drop-cap” opens a paragraph by enlarging and highlighting the first letter, word or phrase. The drop-cap’s goal is to draw your attention to where you can start reading. This demo delivers the same effect by crowning the entire opening paragraph with a “top cap” that funnels your attention into the article. In both cases, what’s going on is a segue from a graphic element to the text.

Violator

monsters-demo-screenshot

A violator is small element that “violates” rectangular text layout by encroaching on a corner or a small part of an edge. This layout idiom is common in short-form magazines and product packaging. That “new and improved” banner which blazes through the corner of thousands of consumer products (whether or not they are new or improved) – it’s a violator.

Column Interest

columns-demo-screenshot

When a print magazine feels the need to incorporate some column layout melodrama, they often reach for this idiom. The shape spans a pair of columns, which creates visual interest in the middle of the page. Without it you’d be faced with a wall of attention sapping text and more than likely turn the page.

Caption

Screenshot of the wine jug caption demo.

The old-school approach for including a caption with an image is to put the caption text alongside or below the image. Putting a caption on top of an image requires a little more finesse, since you have to ensure that the text doesn’t obscure anything important and that the text is rendered in a way that preserves readability.  The result can be relatively attractive.

This photograph was taken by Zoltan Horvath who has pointed out that I’ve combined a quote about tea with a picture of a ceremonial wine jug.  I apologize for briefly breaching that beverage boundary. It’s just a demo.

Paging

Screenshot of the paging demo.

With a layout like this, one could simple let the content wrap and around the shape on the right and then expand into the usual rectangle.  In this demo the content is served up a paragraph at a time, in response to the left and right arrow keys.

Note also: yes in fact the mate gourd is perched on exactly the same windowsill as the previous demo. Zoltan and Pope Francis are among the many fans of yerba mate tea.

Ersatz shape-inside

Screenshot of the ersatz shape-inside demo.

Originally the CSS Shapes spec included shape-inside as well as shape-outside. Sadly, shape-inside was promoted to “Level 2″ of the spec and isn’t available in the current implementations. Fortunately for shape insiders everywhere, it’s still sometimes possible to mimic shape-inside with an adjacent pair of carefully designed shape-outside floats. This demo is a nice example of that, where the text appears inside a bowl of oatmeal.

Animation

animation-demo-screeenshot

This is an animated demo, so to appreciate it you’ll really need to take a look at the live version. It is an example of using an animated shape to draw the user’s attention to a particular message.  Of course one must use this approach with restraint, since an animated loop on a web page doesn’t just gently tug at the user’s attention. It drags at their attention like a tractor beam.

Performance

performance-demo-screenshot

Advertisements are intended to grab the user’s attention and a second or two of animation will do that. In this demo a series of transition motions have been strung together into a tiny performance that will temporarily get the reader’s attention. The highlight of the performance is – of course – the text snapping into the robot’s contour for the finale. Try and imagine a soundtrack that punctuates the action with some whirring and clanking noises, it’s even better that way.

By hmuller at May 13, 2014 05:38 PM

April 24, 2014

Adobe Web Platform Goes to the 2014 WebKit Contributors’ Meeting

Adobe Web Platform

Last week, Apple hosted the 2014 WebKit Contributors’ Meeting at their campus in Cupertino. As usual it was an unconference-style event, with session scheduling happening on the morning of the first day. While much of the session content was very specific to WebKit implementation, there were topics covered that are interesting to the wider web community. This post is a roundup of some of these topics from the sessions that Adobe Web Platform Team members attended.

CSS Custom Properties for Cascading Variables

Alan Stearns suggested a session on planning a new implementation of CSS Custom Properties for Cascading Variables. While implementations of this spec have been attempted in WebKit in the past, they never got past the experimental stage. Despite this, there is still much interest in implementing this feature. In addition, the current version of the spec has addressed many of the issues that WebKit contributors had previously expressed. We talked about a possible issue with using variables in custom property values, which Alan is investigating. More detail is available in the notes from the Custom Properties session.

CSS Regions

Andrei Bucur presented the current state of the CSS Regions implementation in WebKit. The presentation was well received and well attended. Notably, this was one of the few sessions with enough interest that it had a time slot all to itself.

While CSS Regions shipped last year in iOS 7 and Safari 6.1 and 7, the implementation in WebKit hasn’t been standing still. Andrei mentioned the following short list of changes in WebKit since the last Safari release:

  • correct painting of fragments and overflow
  • scrollable regions
  • accelerated content inside regions
  • position: fixed elements
  • the regionoversetchange event
  • better selection
  • better WebInspector integration
  • and more…

Andrei’s slides outlining the state of CSS Regions also contain a roadmap for the feature’s future in WebKit as well as a nice demo of the fix to fragment and overflow handling. If you are following the progress of CSS Regions in WebKit, the slides are definitely worth a look. (As of this writing, the Regions demo in the slides only works in Safari and WebKit Nightly.)

CSS Shapes

Zoltan Horvath, Bear Travis, and I covered the current state of CSS Shapes in WebKit. We are almost done implementing the functionality in Level 1 of the CSS Shapes Specification (which is itself a Candidate Recommendation, the last step before becoming an official W3C standard). The discussion in this session was very positive. We received good feedback on use cases for shape-outside and even talked a bit about the possibilities for when shape-inside is revisited as part of CSS Shapes Level 2. While I don’t have any slides or demos to share at the moment, we will soon be publishing a blog post to bring everyone up to date on the latest in CSS Shapes. So watch this space for more!

Subpixel Layout

This session was mostly about implementation. However, Zalan Bujtas drew an interesting distinction between subpixel layout and subpixel painting. Subpixel layout allows for better space utilization when laying out elements on the page, as boxes can be sized and positioned more precisely using fractional units. Subpixel painting allows for better utilization of high DPI displays by actually drawing elements on the screen using fractional CSS pixels (For example: on a 2x “Retina” display, half of a CSS pixel is one device pixel). Subpixel painting allows for much cleaner lines and smoother animations on high DPI displays when combined with subpixel layout. While subpixel layout is currently implemented in WebKit, subpixel painting is currently a work in progress.

Web Inspector

The Web Inspector is full of shiny new features. The front-end continues to shift to a new design, while the back-end gets cleaned up to remove cruft. The architecture for custom visual property editors is in place and will hopefully enable quick and intuitive editing of gradients, transforms, and animations in the future. Other goodies include new breakpoint actions (like value logging), a redesigned timeline, and IndexedDB debugging support. The Web Inspector still has room for new features, and you can always check out the #webkit-inspector channel on freenode IRC for the latest and greatest.

Web Components

The Web Components set of features continues to gather interest from the browser community. Web Components is made up of four different features: HTML Components, HTML Imports, Shadow DOM, and HTML Templates. The general gist of the talk was that the Web Components concepts are desirable, but there are concerns that the features’ complexity may make implementation difficult. The main concerns seemed to center around performance and encapsulation with Shadow DOM, and will hopefully be addressed with a prototype implementation of the feature (in the works). You can also take a look at the slides from the Web Components session.

CSS Grid Layout

The WebKit implementation of the CSS Grid Layout specification is relatively advanced. After learning in this session that the only way to test out Grid Layout in WebKit was to make a custom build with it enabled, session attendees concluded that it should be turned on by default in the WebKit Nightlies. So in the near future, experimenting with Grid Layout in WebKit should be as easy as installing a nightly build.

More?

As I mentioned earlier, this was just a high-level overview of a few of the topics at this year’s WebKit Contributors’ Meeting. Notes and slides for some of the topics not mentioned here are available on the 2014 WebKit Meeting page in the wiki. The WebKit project is always welcoming new contributors, so if you happen to see a topic on that wiki page that interests you, feel free to get in touch with the community and see how you can get involved.

Acknowledgements

This post would not have been possible without the notes and editing assistance of my colleagues on the Adobe Web Platform Team that attended the meeting along with me: Alan Stearns, Andrei Bucur, Bear Travis, and Zoltan Horvath.

By Bem Jones-Bey at April 24, 2014 05:23 PM

March 18, 2014

QtWebKit is no more, what now?

Gustavo Noronha

Driven by the technical choices of some of our early clients, QtWebKit was one of the first web engines Collabora worked on, building the initial support for NPAPI plugins and more. Since then we had kept in touch with the project from time to time when helping clients with specific issues, hardware or software integration, and particularly GStreamer-related work.

With Google forking Blink off WebKit, a decision had to be made by all vendors of browsers and platform APIs based on WebKit on whether to stay or follow Google instead. After quite a bit of consideration and prototyping, the Qt team decided to take the second option and build the QtWebEngine library to replace QtWebKit.

The main advantage of WebKit over Blink for engine vendors is the ability to implement custom platform support. That meant QtWebKit was able to use Qt graphics and networking APIs and other Qt technologies for all of the platform-integration needs. It also enjoyed the great flexibility of using GStreamer to implement HTML5 media. GStreamer brings hardware-acceleration capabilities, support for several media formats and the ability to expand that support without having to change the engine itself.

People who are using QtWebKit because of its being Gstreamer-powered will probably be better served by switching to one of the remaining GStreamer-based ports, such as WebKitGTK+. Those who don’t care about the underlying technologies but really need or want to use Qt APIs will be better served by porting to the new QtWebEngine.

It’s important to note though that QtWebEngine drops support for Android and iOS as well as several features that allowed tight integration with the Qt platform, such as DOM manipulation through the QWebElement APIs, making QObject instances available to web applications, and the ability to set the QNetworkAccessManager used for downloading resources, which allowed for fine-grained control of the requests and sharing of cookies and cache.

It might also make sense to go Chromium/Blink, either by using the Chrome Content API, or switching to one its siblings (QtWebEngine included) if the goal is to make a browser which needs no integration with existing toolkits or environments. You will be limited to the formats supported by Chrome and the hardware platforms targeted by Google. Blink does not allow multiple implementations of the platform support layer, so you are stuck with what upstream decides to ship, or with a fork to maintain.

It is a good alternative when Android itself is the main target. That is the technology used to build its main browser. The main advantage here is you get to follow Chrome’s fast-paced development and great support for the targeted hardware out of the box. If you need to support custom hardware or to be flexible on the kinds of media you would like to support, then WebKit still makes more sense in the long run, since that support can be maintained upstream.

At Collabora we’ve dealt with several WebKit ports over the years, and still actively maintain the custom WebKit Clutter port out of tree for clients. We have also done quite a bit of work on Chromium-powered projects. Some of the decisions you have to make are not easy and we believe we can help. Not sure what to do next? If you have that on your plate, get in touch!

By kov at March 18, 2014 07:44 PM

February 25, 2014

Improving your site’s visual details: CSS3 text-align-last

Adobe Web Platform

In this post, I want to give a status report regarding the text-align-last CSS3 property. If you are interested in taking control of the small visual details of your site with CSS, I encourage you to keep reading.

The problem

First, let’s talk about why we need this property. You’ve probably already seen many text blocks on pages that don’t quite seem visually correct, because the last line isn’t justified with the previous lines. Check out the example paragraph below:

Example of the CSS3 text-align-last property

In the first column, the last line isn’t justified. This is the expected behavior, when you apply the ‘text-align: justify’ CSS property on a container. On the other hand, in the second column, the content is entirely justified, including the last line.

The solution

This magic is the ‘text-align-last’ CSS3 property, which is set to justify on the second container. The text-align-last property is part of the CSS Text Module Level 3 specification, which is currently a working draft. The text-align-last property describes how the last line of a block or a line right before a forced line break is aligned when ‘text-align’ is ‘justify’, which means you gain full control over the alignment of the last line of a block. The property allows several more options, which you can read about on WebPlatform.org docs, or the CSS Text Module Level 3 W3C Specification.

A possible use case (Added April – 2014)

After looking at the previous example (which was rather focusing on the functionality of the property), let’s move on to a more realistic use case. The feature is perfect to make our multi-line captions look better. Check out the centered, and the justified image caption examples below.

centertext_align__simple_justify

And now, compare them with a justified, multi-line caption, where the last line has been centered by text-align-last: center.
text_align_last_center

I think the proper alignment of the last line gives a better overlook to the caption.

Browser Support

I recently added rendering support for the property in WebKit (Safari) based on the latest specification. Dongwoo Joshua Im from Samsung added rendering support in Blink (Chrome). If you like to try it out in WebKit, you’ll need to make a custom developer build and use the CSS3 text support build flag (--css3-text).

The property is already included in Blink’s developer nightlies by default, so after launching your latest Chrome Canary, you only need to enable ‘Enable experimental Web Platform features’ under chrome://flags, and enjoy the full control over your last lines.

Developer note

Please keep in mind that both the W3C specification and the implementations are under experimental status. I’ll keep blogging about the feature and let you know if anything changes, including when the feature ships for production use!

By Zoltan Horvath at February 25, 2014 04:58 PM

January 05, 2014

Funding MathML Developments in Gecko and WebKit (part 2)

Frédéric Wang

As I mentioned three months ago, I wanted to start a crowdfunding campaign so that I can have more time to devote to MathML developments in browsers and (at least for Mozilla) continue to mentor volunteer contributors. Rather than doing several crowdfunding campaigns for small features, I finally decided to do a single crowdfunding campaign with Ulule so that I only have to worry only once about the funding. This also sounded more convenient for me to rely on some French/EU website regarding legal issues, taxes etc. Also, just like Kickstarter it's possible with Ulule to offer some "rewards" to backers according to the level of contributions, so that gives a better way to motivate them.

As everybody following MathML activities noticed, big companies/organizations do not want to significantly invest in funding MathML developments at the moment. So the rationale for a crowdfunding campaign is to rely on the support of the current community and on the help of smaller companies/organizations that have business interest in it. Each one can give a small contribution and these contributions sum up in enough money to fund the project. Of course this model is probably not viable for a long term perspective, but at least this allows to start something instead of complaining without acting ; and to show bigger actors that there is a demand for these developments. As indicated on the Ulule Website, this is a way to start some relationship and to build a community around a common project. My hope is that it could lead to a long term funding of MathML developments and better partnership between the various actors.

Because one of the main demand for MathML (besides accessibility) is in EPUB, I've included in the project goals a collection of documents that demonstrate advanced Web features with native MathML. That way I can offer more concrete rewards to people and federate them around the project. Indeed, many of the work needed to improve the MathML rendering requires some preliminary "code refactoring" which is not really exciting or immediately visible to users...

Hence I launched the crowdfunding campaign the 19th of November and we reached 1/3 of the minimal funding goal in only three days! This was mainly thanks to the support of individuals from the MathML community. In mid december we reached the minimal funding goal after a significant contribution from the KWARC Group (Jacobs University Bremen, Germany) with which I have been in communication since the launch of the campaign. Currently, we are at 125% and this means that, minus the Ulule commision and my social/fiscal obligations, I will be able to work on the project during about 3 months.

I'd like to thank again all the companies, organizations and people who have supported the project so far! The crowdfunding campaign continues until the end of January so I hope more people will get involved. If you want better MathML in Web rendering engines and ebooks then please support this project, even a symbolic contribution. If you want to do a more significant contribution as a company/organization then note that Ulule is only providing a service to organize the crowdfunding campaign but otherwise the funding is legally treated the same as required by my self-employed status; feel free to contact me for any questions on the project or funding and discuss the long term perspective.

Finally, note that I've used my savings and I plan to continue like that until the official project launch in February. Below is a summary of what have been done during the five weeks before the holiday season. This is based on my weekly updates for supporters where you can also find references to the Bugzilla entries. Thanks to the Apple & Mozilla developers who spent time to review my patches!

Collection of documents

The goal is to show how to use existing tools (LaTeXML, itex2MML, tex4ht etc) to build EPUB books for science and education using Web standards. The idea is to cover various domains (maths, physics, chemistry, education, engineering...) as well as Web features. Given that many scientific circles are too much biased by "math on paper / PDF" and closed research practices, it may look innovative to use the Open Web but to be honest the MathML language and its integration with other Web formats is well established for a long time. Hence in theory it should "just work" once you have native MathML support, without any circonvolutions or hacks. Here are a couple of features that are tested in the sample EPUB books that I wrote:

  • Rendering of MathML equations (of course!). Since the screen size and resolution vary for e-readers, automatic line breaking / reflowing of the page is "naturally" tested and is an important distinction with respect to paper / PDF documents.
  • CSS styling of the page and equations. This includes using (Web) fonts, which are very important for mathematical publishing.
  • Using SVG schemas and how they can be mixed with MathML equations.
  • Using non-ASCII (Arabic) characters and RTL/LTR rendering of both the text and equations.
  • Interactive document using Javascript and <maction>, <input>, <button> etc. For those who are curious, I've created some videos for an algebra course and a lab practical.
  • Using the <video> element to include short sequences of an experiment in a physics course.
  • Using the <canvas> element to draw graphs of functions or of physical measurements.
  • Using WebGL to draw interactive 3D schemas. At the moment, I've only adapted a chemistry course and used ChemDoodle to load Crystallographic Information Files (CIF) and provide 3D-representation of crystal structures. But of course, there is not any problem to put MathML equations in WebGL to create other kinds of scientific 3D schemas.

WebKit

I've finished some work started as a MathJax developer, including the maction support requested by the KWARC Group. I then tried to focus on the main goals: rendering of token elements and more specifically operators (spacing and stretching).

  • I improved LTR/RTL handling of equations (full RTL support is not implemented yet and not part of the project goal).
  • I improved the maction elements and implemented the toggle actiontype.
  • I refactored the code of some "mrow-like" elements to make them all behave like an <mrow> element. For example while WebKit stretched (some) operators in <mrow> elements it could not stretch them in <mstyle>, <merror> etc Similarly, this will be needed to implement correct spacing around operators in <mrow> and other "mrow-like" elements.
  • I analyzed more carefully the vertical stretching of operators. I see at least two serious bugs to fix: baseline alignment and stretch size. I've uploaded an experimental patch to improve that.
  • Preliminary work on the MathML Operator Dictionary. This dictionary contains various properties of operators like spacing and stretchiness and is fundamental for later work on operators.
  • I have started to refactor the code for mi, mo and mfenced elements. This is also necessary for many serious bugs like the operator dictionary and the style of mi elements.
  • I have written a patch to restore support for foreign objects in annotation-xml elements and to implement the same selection algorithm as Gecko.

Gecko

I've continued to clean up the MathML code and to mentor volunteer contributors. The main goal is the support for the Open Type MATH table, at least for operator stretching.

  • Xuan Hu's work on the <mpadded> element landed in trunk. This element is used to modify the spacing of equations, for example by some TeX-to-MathML generators.
  • On Linux, I fixed a bug with preferred widths of MathML token elements. Concretely, when equations are used inside table cells or similar containers there is a bug that makes equations overflow the containers. Unfortunately, this bug is still present on Mac and Windows...
  • James Kitchener implemented the mathvariant attribute (e.g used by some tools to write symbols like double-struck, fraktur etc). This also fixed remaining issues with preferred widths of MathML token elements. Khaled Hosny started to update his Amiri and XITS fonts to add the glyphs for Arabic mathvariants.
  • I finished Quentin Headen's code refactoring of mtable. This allowed to fix some bugs like bad alignment with columnalign. This is also a preparation for future support for rowspacing and columnspacing.
  • After the two previous points, it was finally possible to remove the private "_moz-" attributes. These were visible in the DOM or when manipulating MathML via Javascript (e.g. in editors, tree inspector, the html5lib etc)
  • Khaled Hosny fixed a regression with script alignments. He started to work on improvements regarding italic correction when positioning scripts. Also, James Kitchener made some progress on script size correction via the Open Type "ssty" feature.
  • I've refactored the stretchy operator code and prepared some patches to read the OpenType MATH table. You can try experimental support for new math fonts with e.g. Bill Gianopoulos' builds and the MathML Torture Tests.

Blink/Trident

MathML developments in Chrome or Internet Explorer is not part of the project goal, even if obviously MathML improvements to WebKit could hopefully be imported to Blink in the future. Users keep asking for MathML in IE and I hope that a solution will be found to save MathPlayer's work. In the meantime, I've sent a proposal to Google and Microsoft to implement fallback content (alttext and semantics annotation) so that authors can use it. This is just a couple of CSS rules that could be integrated in the user agent style sheet. Let's see which of the two companies is the most reactive...

By fredw at January 05, 2014 06:45 PM

December 11, 2013

WebKitGTK+ hackfest 5.0 (2013)!

Gustavo Noronha

For the fifth year in a row the fearless WebKitGTK+ hackers have gathered in A Coruña to bring GNOME and the web closer. Igalia has organized and hosted it as usual, welcoming a record 30 people to its office. The GNOME foundation has sponsored my trip allowing me to fly the cool 18 seats propeller airplane from Lisbon to A Coruña, which is a nice adventure, and have pulpo a feira for dinner, which I simply love! That in addition to enjoying the company of so many great hackers.

Web with wider tabs and the new prefs dialog

Web with wider tabs and the new prefs dialog

The goals for the hackfest have been ambitious, as usual, but we made good headway on them. Web the browser (AKA Epiphany) has seen a ton of little improvements, with Carlos splitting the shell search provider to a separate binary, which allowed us to remove some hacks from the session management code from the browser. It also makes testing changes to Web more convenient again. Jon McCan has been pounding at Web’s UI making it more sleek, with tabs that expand to make better use of available horizontal space in the tab bar, new dialogs for preferences, cookies and password handling. I have made my tiny contribution by making it not keep tabs that were created just for what turned out to be a download around. For this last day of hackfest I plan to also fix an issue with text encoding detection and help track down a hang that happens upon page load.

Martin Robinson and Dan Winship hack

Martin Robinson and Dan Winship hack

Martin Robinson and myself have as usual dived into the more disgusting and wide-reaching maintainership tasks that we have lots of trouble pushing forward on our day-to-day lives. Porting our build system to CMake has been one of these long-term goals, not because we love CMake (we don’t) or because we hate autotools (we do), but because it should make people’s lives easier when adding new files to the build, and should also make our build less hacky and quicker – it is sad to see how slow our build can be when compared to something like Chromium, and we think a big part of the problem lies on how complex and dumb autotools and make can be. We have picked up a few of our old branches, brought them up-to-date and landed, which now lets us build the main WebKit2GTK+ library through cmake in trunk. This is an important first step, but there’s plenty to do.

Hackers take advantage of the icecream network for faster builds

Hackers take advantage of the icecream network for faster builds

Under the hood, Dan Winship has been pushing HTTP2 support for libsoup forward, with a dead-tree version of the spec by his side. He is refactoring libsoup internals to accomodate the new code paths. Still on the HTTP front, I have been updating soup’s MIME type sniffing support to match the newest living specification, which includes specification for several new types and a new security feature introduced by Internet Explorer and later adopted by other browsers. The huge task of preparing the ground for a one process per tab (or other kinds of process separation, this will still be topic for discussion for a while) has been pushed forward by several hackers, with Carlos Garcia and Andy Wingo leading the charge.

Jon and Guillaume battling code

Jon and Guillaume battling code

Other than that I have been putting in some more work on improving the integration of the new Web Inspector with WebKitGTK+. Carlos has reviewed the patch to allow attaching the inspector to the right side of the window, but we have decided to split it in two, one providing the functionality and one the API that will allow browsers to customize how that is done. There’s a lot of work to be done here, I plan to land at least this first patch durign the hackfest. I have also fought one more battle in the never-ending User-Agent sniffing war, in which we cannot win, it looks like.

Hackers chillin' at A Coruña

Hackers chillin’ at A Coruña

I am very happy to be here for the fifth year in a row, and I hope we will be meeting here for many more years to come! Thanks a lot to Igalia for sponsoring and hosting the hackfest, and to the GNOME foundation for making it possible for me to attend! See you in 2014!


By kov at December 11, 2013 09:47 AM

October 15, 2013

Funding MathML Developments in Gecko and WebKit

Frédéric Wang

update 2013-10-15: since I got feedback, I have to say that my funding plan is independent of my work at MathJax ; I'm not a MathJax employee but I have an independent contractor status. Actually, I already used my business to fund an intern for Gecko MathML developments during Summer 2011 :-)

Retrospect

Since last April, I have been allowed by the MathJax Consortium to dedicate a small amount of my time to do MathML development in browsers, until possibly more serious involvement later. At the same time, we mentioned this plan to Google developers but unfortunately they just decided to drop the WebKit MathML code from Blink, making external contributions hard and unwelcome...

Hence I have focused mainly on Gecko and WebKit: You can find the MathML bugs that have been closed during that period on bugzilla.mozilla.org and bugs.webkit.org. For Gecko, this has allowed me to finish some of the work I started as a volunteer before I was involved full-time in MathJax as well as to continue to mentor MathML contributors. Regarding WebKit, I added a few new basic features like MathML lengths, <mspace> or <mmultiscripts> while I was getting familiar with the MathML code and WebKit organization/community. I also started to work on <semantics> and <maction>. More importantly, I worked with Martin Robinson to address the design concerns of Google developers and a patch to fix these issues finally landed early this week.

However, my progress has been slow so as I mentioned in my previous blog post, I am planning to find a way to fund MathML developments...

Why funding MathML?

Note: I am assuming that the readers of this blog know why MathML is important and are aware of the benefits it can bring to the Web community. If not, please check Peter Krautzberger's Interview by Fidus Writer or the MozSummit MathML slides for a quick introduction. Here my point is to explain why we need more than volunteer-driven development for MathML.

First the obvious thing: Volunteer time is limited so if we really want to see serious progress in MathML support we need to give a boost to MathML developments. e-book publishers/readers, researchers/educators who are stuck outside the Web in a LaTeX-to-PDF world, developers/users of accessibility tools or the MathML community in general want good math support in browsers now and not to wait again for 15 more years until all layout engines catch up with Gecko or that the old Gecko bugs are fixed.

MathML

There are classical misunderstandings from people thinking that non-native MathML solutions and other polyfills are the future or that math on the Web could be implemented via PNG/SVG images or Web Components. Just open a math book and you will see that e.g. inline equations must be correctly aligned with the text or participate in line wrapping. Moreover we are considering math on the Web not math on paper, so we want it to be compatible with HTML, SVG, CSS, Javascript, Unicode, Bidi etc and also something that is fast and responsive. Technically, this means that a clean solution must be in the core rendering engine, spread over several parts of the code and must have strong interaction with the various components like the HTML5 parser, the layout tree, the graphic and font libraries, the DOM module, the style tree and so forth. I do not see any volunteer-driven Blink/Gecko/WebKit feature off the top of my head that has this characteristic and actually even SVG or any other kind of language for graphics have less interaction with HTML than MathML has.

The consequence of this is that it is extremely difficult for volunteers to get involved in native MathML and to do quick progress because they have to understand how the various components of the Blink/Gecko/WebKit code work and be sure to do things correctly. Good mathematical rendering is already something hard by itself, so that is even more complicated when you are not writing an isolated rendering engine for math on which you can have full control. Also, working at the Blink/Gecko/WebKit level requires technical skills above the average so finding volunteers who can work with the high-minded engineers of the big browser companies is not something easy. For instance, among the enthusiastic people coming to me and willing to help MathML in Gecko, many got stuck when e.g. they tried to build the Firefox source or do something more advanced and I never heard back from them. In the other direction, Blink/Gecko/WebKit paid developers are generally not familiar with MathML and do not have time to learn more about it and thus can not always provide a relevant review of the code, or they may break something while trying to modify code they do not entirely understand. Moreover, both the volunteers and paid staff have only a small amount of time to do MathML stuff while the other parts of the engine evolve very quickly, so it's sometimes hard to keep everything in sync. Finally, the core layout engines have strong security requirements that are difficult to satisfy in a volunteer-driven situation...

Beyond volunteer-driven MathML developments

At that point, there are several options. First the lazy one: Give up with native math rendering, only focus on features that have impact on the widest Web audience (i.e. those that would allow browser vendors to get more market share and thus increase their profit), thank the math people for creating the Web and kindly ask them to use whatever hacks they can imagine to display equations on the Web. Of course as a Mozillian, I think people must decide the Web they want and thus exclude this option.

Next there is the ingenuous option: Expect that browser companies will understand the importance of math-on-the-Web and start investing seriously in MathML support. However, Netscape and Microsoft rejected the <MATH> tag from the 1995 HTML 3.0 draft and the browser companies have kept repeating they would only rely on volunteer contributions to move MathML forward, despite the repeated requests from MathML folks and other scientific communities. So that option is excluded too, at least in the short to medium term.

So it remains the ambitious option: Math people and other interested parties must get together and try to fund native MathML developments. Despite the effort of my manager at MathJax to convince partners and raise funds, my situation has not changed much since April and it is not clear when/if the MathJax Consortium can take the lead in native MathML developments. Given my expertise in Gecko, WebKit and MathML, I feel the duty to do something. Hence I wish to reorganize my work time: Decrease my involvement in MathJax core in order to increase my involvement in Gecko/WebKit developments. But I need the help of the community for that purpose. If you run a business with interest for math-on-the-Web and are willing to fund my work, then feel free to contact me directly by mail for further discussion. In the short term, I want to experiment with Crowd Funding as discussed in the next section. If this is successful we can think of a better organization for MathML developments in the long term.

Crowd Funding

Wikipedia defines Crowd funding as "the collective effort of individuals who network and pool their money, usually via the Internet, to support efforts initiated by other people or organizations". There are several Crowd Funding platforms with similar rule/interface. I am considering Catincan which is specialized in Open Source Crowd Funding, can be used by any backer/developer around the world, can rely on Bugzilla to track the bug status and seems to have good process to collect the fund from backers and to pay developers. You can easily login to the Catincan Website if you have a GitHub, Facebook or Google account (apparently Persona is not supported yet...). Finally, it seems to have a communication interface between backers and developers, so that everybody can follow the development on the funded features.

One distinctive feature of catincan is that only well-established Open Source projects can be funded and only developers from these projects can propose and work on the new features ; so that backers can trust that the features will be implemented. Of course, I have been working on Gecko, WebKit and MathML projects so I hope people believe I sincerely want to improve MathML support in browsers and that I have the skills to do so ;-)

As said in my previous blog post, it is not clear at all (at least to me) whether Crowd Funding can be a reliable method, but it is worth trying. There are many individuals and small businesses showing interest in MathML, without the technical knowledge or appropriate staff to improve MathML in browsers. So if each one fund a small amount of money, perhaps we can get something.

One constraint is that each feature has 60 days to reach the funding goal. I do not have any idea about how many people are willing to contribute to MathML and how much money they can give. The statistical sample of projects currently funded is too small to extract relevant information. However, I essentially see two options: Either propose small features and split the big ones in small steps, so that each catincat submission will need less work/money and improvements will be progressive with regular feedback to backers ; or propose larger features so they look more attractive and exciting to people and will require less frequent submissions to catincat. At the beginning, I plan to start with the former and if the crowd funding is successful perhaps try the latter.

Status in Open Source Layout Engines

Note: Obviously, Open Source Crowd Funding does not apply to Internet Explorer, which is the one main rendering engine not mentioned below. Although Microsoft has done a great job on MathML for Microsoft Word, they did not give any public statement about MathML in Internet Explorer and all the bug reports for MathML have been resolved "by design" so far. If you are interested in MathML rendering and accessibility in Internet Explorer, please check Design Science blog for the latest updates and tools.

Blink

Note: I am actually focusing on the history of Chromium here but of course there are other Blink-based browsers. Note that programs like QtWebEngine (formerly WebKit-based) or Opera (formerly Presto-based) lost the opportunity to get MathML support when they switched to Blink.

Alex Milowski and François Sausset's first MathML implementation did not pass Google's security review. Dave Barton fixed many issues in that implementation and as far as I know, there were not any known security vulnerabilities when Dave submitted his last version. MathML was enabled in Chrome 24 but Chrome developers had some concerns about the design of the MathML implementation in WebKit, which indeed violated some assumptions of WebKit layout code. So MathML was disabled in Chrome 25 and as said in the introduction, the source code was entirely removed when they forked.

Currently, the Chromium Dashboard indicates that MathML is shipped in Firefox/Safari, has positive feedback from developers and is an established standard ; but the Chromium status remains "No active development". If I understand correctly, Google's official position is that they do not plan to invest in MathML development but will accept external contributions and may re-enable MathML when it is ready (for some sense of "ready" to be defined). Given the MathML story in Chrome, it seems really unlikely that any volunteer will magically show up and be willing to submit a MathML patch. Incidentally, note the interesting work of the ChromeVox team regarding MathML accessibility: Their recent video provides a good overview of what they achieve (where Volker Sorge politely regrets that "MathML is not implemented in all browsers").

Although Google's design concerns have now been addressed in WebKit, one most serious remark from one Google engineer is that the WebKit MathML implementation is of too low quality to be shipped so they just prefer to have no MathML at all. As a consequence, the best short term strategy seems to be improving WebKit MathML support and, once it is good enough, to submit a patch to Google. The immediate corollary is that if you wish to see MathML in Chrome or other Blink-based browsers you should help WebKit MathML development. See the next section for more details.

Chromatic chromatic

Actually, I tried to import MathML into Blink one day this summer. However, there were divergences between the WebKit and Blink code bases that made that a bit difficult. I do not plan to try again anytime soon, but if someone is interested, I have published my script and patch on GitHub. Note there may be even more divergences now and the patch is certainly bit-rotten. I also thought about creating/maintaining a "Chromatic" browser (Chrome + mathematics) that would be a temporary fork to let Blink users benefit from native MathML until it is integrated back in Blink. But at the moment, that would probably be too much effort for one person and I would prefer to focus on WebKit/Gecko developments for now.

WebKit

The situation in WebKit is much better. As said before, Google's concerns are now addressed and MathML will be enabled again in all WebKit releases soon. Martin Robinson is interested in helping the MathML developments in WebKit and his knowledge of fonts will be important to improve operator stretching, which is one of the biggest issue right now. One new volunteer contributor, Gurpreet Kaur, also started to do some work on WebKit like support for the *scriptshifts attributes or for the <menclose> element. Last but not least, a couple of Apple/WebKit developers reviewed and accepted patches and even helped to fix a few bugs, which made possible to move development forward.

WebKineTic

When he was still working on WebKit, Dave Barton opened bug 99623 to track the top priorities. When the bugs below and their related dependencies are fixed, I think the rendering in WebKit will be good enough to be usable for advanced math notations and WebKit will pass the MathML Acid 1 test.

  • Bug 44208: For example, in expression like sin(x), the "x" should be in italic but not the "sin". This is actually slightly more complicated: It says when the default mathvariant value must be normal/italic. mathvariant is more like the text-transform CSS property in the sense that it remaps some characters to the corresponding mathematical characters (italic, bold, fraktur, double-struck...) for example A (mathvariant=fraktur A) should render exactly the same as 𝔄 (U+1D504). By the way, there is the related bug 24230 on Windows, that prevents to use plane 1 characters. The best solution will probably be to implement mathvariant correctly. See also Gecko's ongoing work by James Kitchener below.
  • Bug 99618: Implement <mmultiscripts>, that allows expressions like C614 or Rij;j=12S;i. As said in the introduction, this is fixed in WebKit Nightly.
  • Bug 99614: Support for stretchy operators like in (z1+z2¯3)4. Currently, WebKit can only stretch operators vertically using a few Unicode constructions like ⎛ (U+239B) + ⎜ (U+239C) + ⎝ (U+239D) for the left parenthesis. Essentially only similar delimiters like brackets, braces etc are supported. For small sizes like ( or for large operators like n2 it is necessary to use non-unicode glyphs in various math fonts, but this is not possible in WebKit MathML yet. All of this will require a fair amount of work: implementing horizontal stretching, font-specific stuff, largeop/symmetric properties etc
  • Bug 99620: Implement the operator dictionary. Currently, WebKit treats all the operators the same way, so for example it will use the same 0.2em spacing before and after parenthesis, equal sign or invisible operators in e.g. f(x)=x2. Instead it should use the information provided by the MathML operator dictionary. This dictionary also specifies whether operators are stretchy, symmetric or largeop and thus is related to the previous point.
  • Bug 119038: Use the code for vertical stretchy operators to draw the radical symbols in roots like 23. Currently, WebKit uses graphic primitives which do not give a really good rendering.
  • Bug 115610: Implement <mspace> which is used by many MathML generators to do some spacing in mathematical formulas. As said in the introduction, this is fixed in WebKit Nightly.

In order to pass the Mozilla MathML torture tests, at least displaystyle and scriptlevel must be implemented too, probably as internal CSS properties. This should also allow to pass Joe Java's MathML test, although that one relies on the infamous <mfenced> that duplicates the stretchy operator features and is implemented inconsistently in rendering engines. I think passing the MathML Acid 2 test will require slightly more effort, but I expect this goal to be achievable if I have more time to work on WebKit:

  • Bug 115610: Implement <mspace>. Fixed!
  • Bug 120164: Implement negative spacing for <mspace> (I have an experimental patch).
  • Bug 85730: Implement <mpadded>, which is also used by MathML generators to do some tweaking of formulas. I have only done some experiments, that would be a generalization of <mspace>
  • Bug 85733: Implement the href attribute ; well I guess the name is explicit enough to understand what it is used for! I only have some experimental patch here too. That would be mimicing what is done in SVG or HTML.
  • Bug 120059 and bug 100626: Implement <maction> (at least partially) and <semantics>, which have been asked by long-time MathML users Jacques Distler and Michael Kohlhase. I have patches ready for that and this could be fixed relatively soon, I just need to find time to finish the work.

In general passing the MathML Acid 2 test is not too hard, you merely only need to implement those few MathML elements whose exact rendering is clearly defined by the MathML specification. Passing the MathML Acid 3 test is not expected in the medium term. However, the score will naturally increase while we improve WebKit MathML implementation. The priority is to implement what is currently known to be important to users. To give examples of bugs not previously mentioned: Implementing menclose or fixing various DOM issues like bugs 57695, 57696 or 107392.

More advanced features like those mentioned in the next section for Gecko are probably worth considering later (Open type MATH, linebreaking, mlabeledtr...). It is worth noting that Apple has already done some work on accessibility (with MathML being readable by VoiceOver in iOS7), authoring and EPUB (MathML is enabled in WebKit-based ebook readers and ibooks-author has an integrated LaTeX-to-MathML converter).

Gecko

Mathzilla

In general I think I have a good relationship with the Mozilla community and most people have respect for the work that has been done by volunteers for almost 15 years now. The situation has greatly improved since I joined the project, at that time some people claimed the Mozilla MathML project was dead after Roger Sidge's departure. One important point is that Karl Tomlinson has worked on repairing the MathML support when Roger Sidge left the project. Hence there is at least one Mozilla employee with good knowledge of MathML who can review the volunteer patches. Another key ingredient is the work that has recently been made by Mozilla to increase engagement of the volunteer community like good documentation on MDN, the #Introduction channel, Josh Matthews' mentored bugs and of course programs like GSOC. However, as said above, it is one thing to attract enthusiastic contributors and another thing to get long-term contributors who can work independently on more advanced features. So let's go back to my latest Roadmap for the Mozilla MathML Project and see what has been accomplished for one year:

  • Font support: Dmitry Shachnev created a Debian package for the MathJax fonts and Mike Hommey added MathJax and Asana fonts in the list of suggested packages for Iceweasel. The STIX fonts have also been updated in Fedora and are installed by default on Mac OS X Lion (10.7). For Linux distributions, it would be helpful to implement Auto Installation Support. The bug to add mathematical fonts to Android has been assigned in June but no more progress has happened so far. Henri Sinoven opened a bug for FirefoxOS but there has not been any progress there either. I had some patches to restore the "missing MathML fonts" warning (using an information bar) but it was refused by Firefox reviewers. However, the code to detect missing MathML font could still be used for the similar bug 648548, which also seems inactive since January. There are still some issues on the MathJax side that prevent to integrate Web fonts for the native MathML output mode. So at the moment the solution is still to inform visitors about MathML fonts or to add MathML Web fonts to your Web site. Khaled Hosny (font and LaTeX expert) recently updated my patches to prepare the support for Open Type fonts and he offered to help on that feature. After James Kitchener's work on mathvariant, we realized that we will probably need to provide Arabic mathematical fonts too.
  • Spacing: Xuan Hu continued to work on <mpadded> improvements and I think his patch is close to be accepted. Quentin Headen has done some progress on <mtable> before focusing on his InstantBird GSOC project. He is still far from being able to work on mtable@rowspacing/columnspacing but a work around for that has been added to MathJax. I fixed the negative space regression which was missing to pass the MathML Acid 2 test and is used in MathJax. Again, Khaled Hosny is willing to help to use the spacing of the Open Type MATH, but that will still be a lot of work.
  • <mlabeledtr>: A work around for native MathML has been added in MathJax.
  • Linebreaking: No progress except that I have worked on fixing a bug with intrinsic width computation. The unrelated printing issues mentioned in the blog post have been fixed, though.
  • Operator Stretching: No progress. I tried to analyze the regression more carefully, but nothing is ready yet.
  • Tabular elements: As said above, Quentin Headen has worked a bit on cleaning up <mtable> but not much improvements on that feature so far.
  • Token elements: My patch for <ms> landed and I have done significant progress on the bad measurement of intrinsic width for token elements (however, the fix only seems to work on Linux right now). James Kitchener has taken over my work on improving our mathvariant support and doing related refactoring of the code. I am confident that he will be able to have something ready soon. The primes in exponents should render correctly with MathJax fonts but for other math fonts we will have to do some glyph substitutions.
  • Dynamic MathML: No progress here but there are not so many bugs regarding Javascript+MathML, so that should not be too serious.
  • Documentation: It is now possible to use MathML in code sample or directly in the source code. The MathML project pages have been entirely migrated to MDN. Also, Florian Scholz has recently been hired by Mozilla as a documentation writer (congrats!) and will in particular continue the work he started as a volunteer to document MathML on MDN.

I apologize to volunteers who worked on bugs that are not mentioned above or who are doing documentation or testing that do not appear here. For a complete list of activity since September 2012, Bugzilla is your friend. There are two ways to consider the progress above. If you see the glass half full, then you see that several people have continued the work on various MathML issues, they have made some progress and we now pass the MathML Acid 2 test. If you see the glass half empty, then you see that most issues have not been addressed yet and in particular those that are blocking the native MathML to be enabled in MathJax: bug 687807, bug 415413, the math font issues discussed in the first point, and perhaps linebreaking too. That is why I believe we should go beyond volunteer-driven MathML developments.

Most of the bugs mentioned above are tested by the MathML Acid 3 tests and we will win a few points when they are fixed. Again, passing MathML Acid 3 test is not a goal by itself so let's consider what are the big remaining areas it contains:

  • Improving Tabular Elements and Operator Stretching, which are obviously important and used a lot in e.g. MathJax.
  • Linebreaking, which as I said is likely to become fundamental with small screens and ebooks.
  • Elementary Mathematics (you know addition, subtraction, multiplication, and division that kids learn), which I suspect will be important for educational tools and ebooks.
  • Alignment: This is the one part of MathML that I am not entirely sure is relevant to work on in the short term. I understand it is useful for advanced layout but most MathML tools currently just rely on tables to do that job and as far as I know the only important engine that implements that is MathPlayer.

Finally there are other features outside the MathML rendering engines that I also find important but for which I have less expertise:

  • Transferring MathML that is implementing copy/cut/drag and paste. Currently, we can do that by treating MathML as normal HTML5 code or by using the "show MathML source" feature and copying the source code. However, it would be best to implement a standard way to communicate with other MathML applications like Microsoft Word, Mathematica, Mapple, Windows' Handwriting panel etc I wrote some work-in-progress patches last year.
  • Authoring MathML: Essentially implementing things like deletion, insertion etc maybe simple MathML token creation ; in Gecko's core editor, which is used by BlueGriffon, KompoZer, SeaMonkey, Thunderbird or even MDN. Other things like integrating Javascript parsers (e.g. ASCIIMath) or equation panels with buttons like are probably better done at the higher JS/HTML/XUL level. Daniel Glazman already wrote math input panels for BlueGriffon and Thunderbird.
  • MathML Accessibility: This is one important application of MathML for which there is strong demand and where Mozilla is behind the competitors. James Teh started some experimental work on his NVDA tool before the summit.
  • EPUB reader for FirefoxOS (and other mobile platforms): During the "Co-creating Action Plans" session, the Mozilla Taipei people were thinking about missing features for FirefoxOS and this idea about EPUB reader was my modest contribution. There are a few EPUB readers relying on Gecko and it would be good to check if they work in FirefoxOS and if they could be integrated by default, just like Apple has iBooks. BTW, there is a version of BlueGriffon that can edit EPUB books.

Conclusion

I hope I have convinced some of the readers about the need to fund MathMLin browsers. There is a lot of MathML work to do on Gecko and WebKit but both projects have volunteers and core engineers who are willing to help. There are also several individuals / companies relying on MathML support in rendering engines for their projects and could support the MathML developments in some way. I am willing to put more of my time on Gecko and WebKit developments, but I need financial help for that purpose. I'm proposing catincan Crowd Funding in the short term so that anyone can contribute at the appropriate level, but other alternatives to fund the MathML development can be found like asking Peter Krautzberger about native MathML funding in MathJax, discussing with Igalia about funding Martin Robinson to work more on WebKit MathML or contacting me directly to establish some kind of part-time consulting agreement.

Please leave a comment on this blog or send me a private mail, if you agree that funding MathML in browsers is important, if you like the crowd funding idea and plan to contribute ; or if you have any opinions about alternative funding options. Also, please tell me what seem to be the priority for you and your projects among what I have mentioned above (layout engines, features etc) or among others that I may have forgotten. Of course, any other constructive comment to help MathML support in browsers is welcome. I plan to submit features on catincan soon, once I have more feedback on what people are interested in. Thank you!

By fredw at October 15, 2013 08:05 PM

October 07, 2013

Post-Summit Thoughts on the MathML Project

Frédéric Wang

I'm back from a great Mozilla Summit 2013 and I'd just like to write a quick blog post about the MathML booths at the Innovation Fairs. I did not have the opportunity to talk with the MathML people who ran the booth at Santa Clara yet. However, everything went pretty well at Brussels, modulo of course some demos failing when done in live... If you are interested, the slides and other resources are available on my GitHub page.

Many Mozillians did not know about MathML or that it had been available in Gecko since the early days of the Mozilla project. Many people who use math (or just knowing someone who does) were curious about that feature and excited about the MathML potentials. I appreciated to get this positive feedback from Mozillians willing to use math on the Web and related media, instead of the scorn or hatred I sometimes see by misinformed people. I expect to provide more updates on LaTeXML, MediaWiki Math and MathJax when their next versions are released. The Gecko MathML support improves slowly but there has been interesting work by James Kitchener recently that I'd like to mention too.

MathZilla on blackboard

Let's do an estimation à la Fermi: only a few volunteers have been contributing regularly and simultaneously to MathML in Gecko while most Mozilla-funded Gecko projects have certainly development teams that are 3 times as large. Let's be optimistic and assume that these volunteers have been able to dedicate a mean of 1 work day per week, compared to 5 for full-time staff. Given that the Mozilla MathML project will celebrate its 15 years next May, that means that the volunteer work transposed in terms of paid-staff time is only 15 35 = 1 year. To be honest, I'm disregarding here the great work made by the Mozilla NZ team around 2007 to repair MathML after the Cairo migration. But still, what we have achieved in quality and completeness with such limited resources and time is really impressive.

As someone told me at the MathML booth, it's really frustating that something that is so important for the small portion of math-educated people is ignored because it is useless for the vast majority of people. This is not entirely true, since even elementary mathematics taught at school like the one of this blog post are not easily expressed with standard HTML and even less in a way accessible to people with visual disabilities. However, it summarizes well the feeling MathML folks had when they tried to convince Google to accept the volunteer work on MathML, despite its low quality.

As explained at the Summit Sessions, Mozilla's mission is different and the goal is to give people the right to control the Web they want. The MathML project is perhaps one of the oldest and successful volunteer-driven Mozilla project that is still active and demonstrates concretely the idea of the Mozilla's mission with e.g. the work of Roger Sidge who started to write the MathML implementation when Netscape opened its source code or the one of Florian Scholz who made MDN one of the most complete Web resource for MathML.

Mozilla Corporation has kept saying they don't want to invest in MathML developments and the focus right now is clearly on other features like FirefoxOS. Even projects that have a larger audience than the MathML support like the mail client or the editor are not in the priorities so someone else definitely need to step in for MathML. I've tried various methods, with more or less success, to boost the MathML developments like mentoring a GSoC project, funding a summer internship or relying on mentored bugs. I'm now considering crowd funding to help the MathML developments in Gecko (and WebKit). I don't want to do another Fermi estimation now but at first that looks like a very unreliable method. The only revenue generated by the MathML project so far are the 2 100 π 100 = 2 3.14 = 6.28 dollars to the Mozilla Fundation via contributions to my MathML-fonts add-on, so it's hard to get an idea of how much people would contribute to the Gecko implementaton. However, that makes sense since the only people who showed interest in native MathML support so far are individuals or small businesses (e.g. working on EPUB or accessibility) and I think it's worth trying it anyway. That's definitely something I'll consider after MathJax 2.3 is released...

By fredw at October 07, 2013 04:18 PM

August 27, 2013

HTML Alchemy – Combining CSS Shapes with CSS Regions

Adobe Web Platform

Note: Support for shape-inside is only available until the following nightly builds: WebKit r166290 (2014-03-26); Chromium 260092 (2014-03-28).


I have been working on rendering for almost a year now. Since I landed the initial implementation of Shapes on Regions in both Blink and WebKit, I’m incredibly excited to talk a little bit about these features and how you can combine them together.

Mad_scientist

Don’t know what CSS Regions and Shapes are? Start here!

The first ingredient in my HTML alchemy kitchen is CSS Regions. With CSS Regions, you can flow content into multiple styled containers, which gives you enormous creative power to make magazine style layouts. The second ingredient is CSS Shapes, which gives you the ability to wrap content inside or outside any shape. In this post I’ll talk about the “shape-inside” CSS property, which allows us to wrap content inside an arbitrary shape.

Let’s grab a bowl and mix these two features together, CSS Regions and CSS Shapes to produce some really interesting layouts!

In the latest Chrome Canary and Safari WebKit Nightly, after enabling the required experimental features, you can flow content continuously through multiple kinds of shapes. This rocks! You can step out from the rectangular text flow world and break up text into multiple, non-rectangular shapes.

Demo

If you already have the latest Chrome Canary/Safari WebKit Nightly, you can just go ahead and try a simple example on codepen.io. If you are too lazy, or if you want to extend your mouse button life by saving a few button clicks, you can continue reading.


iBuyd3

In the picture above we see that the “Lorem ipsum” story flows through 4 different, colorful regions. There is a circle shape on each of the first two fixed size regions. Check out the code below to see how we apply the shape to the region. It’s pretty straightforward, right?
#region1, #region2 {
    -webkit-flow-from: flow;
    background-color: yellow;
    width: 200px;
    height: 200px;
    -webkit-shape-inside: circle(50%, 50%, 50%);
}
The content flows into the third (percentage sized) region, which represents a heart (drawn by me, all rights reserved). I defined the heart’s coordinates in percentages, so the heart will stretch as you resize the window.
#region3 {
    -webkit-flow-from: flow;
    width: 50%;
    height: 400px;
    background-color: #EE99bb;
    -webkit-shape-inside: polygon(11.17% 10.25%,2.50% 30.56%,3.92% 55.34%,12.33% 68.87%,26.67% 82.62%,49.33% 101.25%,73.50% 76.82%,85.17% 65.63%,91.63% 55.51%,97.10% 31.32%,85.79% 10.21%,72.47% 5.35%,55.53% 14.12%,48.58% 27.88%,41.79% 13.72%,27.50% 5.57%);
}

The content that doesn’t fit in the first three regions flows into the fourth region. The fourth region (see the retro-blue background color) has its CSS width and height set to auto, so it grows to fit the remaining content.

Real world examples

After trying the demo and checking out the links above, I’m sure you’ll see the opportunities for using shape-inside with regions in your next design. If you have some thoughts on this topic, don’t hesitate to comment. Please keep in mind that these features are under development, and you might run into bugs. If you do, you should report them on WebKit’s Bugzilla for Safari or Chromium’s issue tracker for Chrome. Thanks for reading!

By Zoltan Horvath at August 27, 2013 04:00 PM

August 06, 2013

WebGL, at last!

Brent Fulgham

It's been a long time since I've written an update -- but my lack of blog posting is not an indication of a lack of progress in WebKit or the WinCairo port. Since I left my former employer (who *still* hasn't gotten around to updating the build machine I set up there), we've:

  • Migrated from Visual Studio 2005 to Visual Studio 2010 (and soon, VS2012)
  • Enabled New-run-webkit-tests
  • Updated the WinCairo Support Libraries to support 64-bit builds
  • Integrated a ton of cURL improvements and extensions thanks to the TideSDK guys 
  • and ...
... thanks to the hard work of Alex Christensen, brought up WebGL on the WinCairo port.  This is a little exciting for me, because it marks the first time (I can recall) where the WinCairo port actually gained a feature that was not already part of the core Apple Windows port.



The changes needed to see these circa-1992 graphics in all their three-dimensional glory are already landed in the WebKit tree.  You just need to:

  1. Enable the libEGL, libGLESv2, translator_common, translator_glsl, and translator_hlsl for the WinCairo build (they are currently turned off).
  2. Make the following change to WTF/wtf/FeatureDefines.h: 

Brent Fulgham@WIN7-VM ~/WebKit/Source/WTF/wtf
$ svn diff
Index: FeatureDefines.h
===================================================================
--- FeatureDefines.h    (revision 153733)
+++ FeatureDefines.h    (working copy)
@@ -245,6 +245,13 @@
 #define ENABLE_VIEW_MODE_CSS_MEDIA 0
 #endif

+#define ENABLE_WEBGL 1
+#define WTF_USE_3D_GRAPHICS 1
+#define WTF_USE_OPENGL 1
+#define WTF_USE_OPENGL_ES_2 1
+#define WTF_USE_EGL 1
+#define WTF_USE_GRAPHICS_SURFACE 1
+
 #endif /* PLATFORM(WIN_CAIRO) */

 /* --------- EFL port (Unix) --------- */

Performance is a little ragged, but we hope to improve that in the near future.

We have plenty of more plans for the future, including full 64-bit support (soon), and hopefully some improvements to the WinLauncher application to make it a little more useful.

As always, if you would like to help out,

By Brent Fulgham (noreply@blogger.com) at August 06, 2013 05:53 AM

May 15, 2013

CSS Level 3 Text Decoration on WebKit and Blink – status

Bruno de Oliveira Abinader

It’s been a while since I wrote the last post about progress on implementing CSS Level 3 Text Decoration features in WebKit. I’ve been involved with other projects but now I can finally resume the work in cooperation with my colleague from basysKom, Lamarque Souza. So far we have implemented:

  • text-decoration-line (link)
  • text-decoration-style (link)
  • text-decoration-color (link)
  • text-underline-position (link)

These properties are currently available under -webkit- prefix on WebKit, and guarded by a feature flag - CSS3_TEXT – which is enabled by default on both EFL and GTK ports. On Blink, plans are to get these properties unprefixed and under a runtime flag, which can be activated by enabling the “Experimental WebKit Features” (updated to “Experimental Web Platform Features” in latest builds) flag – see chrome://flags inside Google Chrome/Chromium). There are still some Skia-related issues to fix on Blink to enable proper dashed and dotted text decoration styles to be displayed. In the near future, we shall also have the text-decoration shorthand as specified on CSS Level 3 specification.

See below a summary of things I plan to finish in the near future:

  • [webkit] Property text-decoration-line now accepts blink as valid value
  • [blink] Fix implementation of dashed and dotted styles on Skia
  • [blink] Fix an issue where previous Skia stroke styles were used when rendering paint decorations
  • [blink] Implement CSS3_TEXT as a runtime flag
  • [blink] Property text-decoration-line now accepts blink as valid value
  • [blink] Implement support for text-decoration shorthand
  • [webkit] Implement support for text-decoration shorthand

Note: Please do not confuse text-decoration‘s blink value with Blink project :)

Stay tuned for further updates!

By Bruno Abinader at May 15, 2013 04:52 PM

May 03, 2013

Firefox Nightly passes the Acid2 test

Frédéric Wang

Some updates on the MathML Acid Tests... First the patch for bug 717546 landed in Nightly and thus Gecko is now the first layout engine to pass the MathML Acid2 test. Here is a screenshot that should look familiar:

MathML Acid2, Nightly

As you know, Google developers forked Webkit and decided to remove from Blink all the code (including MathML) on which they don't plan to work in the short term. As a comparison, here is how the MathML Acid2 test looks like in Chrome Canary:

MathML Acid 2 Test, Canary

Next, someone reported that Firefox Mac got more errors in the MathML Acid3 test. I was already aware of some shortcomings anyway and thus took the opportunity to rewrite the tests with a better error tolerance. The changes also fixed some measurement issues with auto resizing on mobile platforms or when the zoom level is not set to the default. I also made the tests for stretchy operators more reliable and as a consequence, Gecko lost two points: the new score is 60/100. I still need to review and describe the tests and hope I won't find more mistakes.

Finally, I also added a MathML Acid1 test. It does not really look like the "classical" Acid1 test and is not "automated", in the sense that a reader must carefully (and in a subjective way) check the basic requirements. But at least it provides a small test in the spirit of CSS Acid 1: all 100%-conformant HTML 5 agents should be able to render these very elementary MathML expressions. Note that the formulas in the MathML Acid1 test are supposed to express mathematical properties of boxes from the CSS Acid1 test.

By fredw at May 03, 2013 12:43 PM

March 27, 2013

Freeing the Floats of the Future From the Tyranny of the Rectangle

Adobe Web Platform

With modern web layout you can have your content laid out in whatever shape you want as long as it’s a rectangle. Designers in other media have long been able to have text and other content lay out inside and around arbitrarily complex shapes. The CSS Exclusions, CSS Shapes Level 1, and CSS Shapes Level 2 specifications aim to bring this capability to the web.

While these features aren’t widely available yet, implementation is progressing and it’s already possible to try out some of the features yourself. Internet Explorer 10 has an implementation of the exclusions processing model, so you can try out exclusions in IE 10 today.

At Adobe we have been focusing on implementing the shapes specification. We began with an implementation of shape-inside and now have a working implementation of the shape-outside property on floats. We have been building our implementation in WebKit, so the easiest way to try it out yourself is to download a copy of Chrome Canary. Once you have Canary, enable Experimental Web Platform Features and go wild!

What is shape-outside?

“Now hold up there,” you may be thinking, “I don’t even know what a shape-outside is and you want me to read this crazy incomprehensible specification thing to know what it is!?!”

Well you’ll be happy to know that it really isn’t that complex, especially in the case of floats. When an element is floated, inline content avoids the floated element. Content flows around the margin box of the element as defined by the CSS box model. The shape-outside CSS property allows you to tell the browser to use a specified shape instead of the margin box when wrapping content around the floating element.

CSS Exclusions

The current implementation allows for rectangles, rounded rectangles, circles, ellipses, and polygons. While this gives a lot of flexibility, eventually you will be able to use a SVG path or the alpha channel of an image to make it easier to create complex shapes.

How do I use it?

First, you need to get a copy of Chrome Canary and then enable Experimental Web Platform features. Once you have that, load up this post in Chrome Canary so that you can click on the images below to see a live example of the code. Even better, the examples are on Codepen, so you can and should play with them yourself and see what interesting things you can come up with.

Note that in this post and the examples I use the unprefixed shape-outside property.
If you want to test these examples outside of my Codepen then you will need to use the prefixed -webkit-shape-outside property or use (which is a built in option in Codepen).

We’ll start with a HTML document with some content and a float. Currently shape-outside only works on floating elements, so those are the ones to concentrate on. For example: (click on the image to see the code)

HTML without shape-outside

You can now add the shape-outside property to the style for your floats.

.float {
  shape-outside: circle(50%, 50%, 50%);
}

A circle is much more interesting than a standard rectangle, don’t you think? This circle is centered in the middle of the float and has a radius that is half the width of the float. The effect on the layout is something like this:

shape-outside circle

While percentages were used for this circle, you can use any CSS unit you like to specify the shape. All of the relative units are relative to the dimensions of element where the shape-outside is specified.

Supported shapes

Circles are cool and all, but I promised you other shapes, and I will deliver. There are four types of shapes that are supported by the current shape-outside implementation: rectangle, circle, ellipse, and polygon.

rectangle

You have the ability to specify a shape-outside that is a fairly standard rectangle:

shape-outside: rectangle(x, y, width, height);

The x and y parameters specify the coordinates of the top-left corner of the rectangle. This coordinate is in relation to the top-left corner of the floating element’s content box. Because of the way this interacts with the rules of float positioning, setting these to anything other than 0 causes an effect that is similar to relatively positioning the float’s content. (Explaining this is beyond the scope of this post.)

The width and height parameters should be self-explanatory: they are the width and height of the resulting rectangle.

Where things get interesting is with the six-argument form of rectangle:

shape-outside: rectangle(x, y, width, height, rx, ry);

The first four arguments are the same as explained above, but the last two specify corner radii in the horizontal (rx) and vertical (ry) directions. This not only allows the creation of rounded rectangles, you can create circles and ellipses as well. (Just like with [border-radius][border-radius].)

Here’s an example of a rectangle, a rounded rectangle, a circle, and an ellipse using just rectangle syntax:

shape-outside rectangle

If you’re reading this in Chrome Canary with exclusions turned on, play around with this demo and see what other things you can do with the rectangles.

circle

I already showed you a simple circle demo and you’ll be happy to know that’s pretty much all there is to know about circles:

shape-outside: circle(cx, cy, radius);

The cx and cy parameters specify the coordinates of the center of the circle. In most situations you’ll want to put them at the center of your box. Just like with rectangles moving this around can be useful, but it behaves similarly to relatively positioning the float’s content with respect to the shape.

The radius parameter is the radius of the resulting circle.

In case you’d like to see it again, here’s what a circle looks like:

shape-outside circle

While it is possible to create circles with rounded rectangles as described above, having a dedicated circle shape is much more convenient.

ellipse

Sometimes, you need to squish your circles and that’s where the ellipse comes in handy.

shape-outside: ellipse(cx, cy, rx, ry);

Just like a circle, an ellipse has cx and cy to specify the coordinates of its center and you will likely want to have them at the center of your float. And just like all the previous shapes, changing these around will cause the float’s content to position relative to your shape.

The rx and ry parameters will look familiar from the rounded rectangle case and they are exactly what you would expect: the horizontal and vertical radii of the ellipse.

Ellipses can be used to create circles (rx = ry) and rounded rectangles can be used to create ellipses, but it’s best to use the shape that directly suits your purpose. It’s much easier to read and maintain that way.

Here’s an example of using an ellipse shape:

shape-outside ellipse

polygon

Now here’s where things get really interesting. The polygon `shape-outside` allows you to specify an arbitrary polygonal shape for your float:

shape-outside: polygon(x1 y1, x2 y2, ... , xn yn);

The parameters of the polygon are the x and y coordinates of each vertex of the shape. You can have as many vertices as you would like.

Here’s an example of a simple polygon:

shape-outside triangle

Feel free to play with this and see what happens if you create more interesting shapes!

Putting content in the float

The previous examples all had divs without any content just to make it easier to read and understand the code, but a big motivation for shape-outside is to wrap around other content. Interesting layouts often involve wrapping text around images as this final example shows:

shape-outside with images

As usual, you should take a look and play with the code for this example of text wrapping around floated images. This is just the beginning of the possibilities, as you can put a shape outside on any floating element with any content you want inside.

Next steps

We are still hard at work on fixing bugs in the current implementation and implementing the rest of the features in the CSS Shapes Level 1 specification. We welcome your feedback on what is already implemented and also on the spec itself. If you are interested in becoming part of the process, you can raise issues with the current WebKit implementation by filing bugs in the WebKit bugzilla. If you have issues with the spec, those are best raised on the www-style mailing list. And of course, you can leave your feedback as comments on this post.

I hope that you enjoy experimenting with shape-outside and the other features we are currently working on.

By Bem Jones-Bey at March 27, 2013 05:10 PM

March 02, 2013

MathML Acid Tests

Frédéric Wang

There has recently been discussion in the Mozilla community about Opera switch from Presto to Webkit and the need to preserve browser competition and diversity of rendering engines, especially with mobile devices. Some people outside the community seem a bit skeptic about that argument. Perhaps a striking example to convince them is to consider the case of MathML where basically only Gecko has a decent native implementation and the situation in the recent eBooks workshop illustrates that very well: MathML support is very important for some publishers (e.g. for science or education) but the main eBook readers rely exclusively on the Webkit engine and its rudimentary MathML implementation. Unfortunately because there is currently essentially no alternatives on mobile platforms, developers of eBook readers have no other choices than proposing a partial EPUB support or relying on polyfill....

After Google's announce to remove MathML from Chrome 25, someone ironized on twitter about the fact that an Acid test for MathML should be written since that seems to motivate them more than community feedback. I do not think that MathML support is something considered important from the point of view of browser competition but I took this idea and started writing MathML versions of the famous Acid2 and Acid3 tests. The current source of these MathML Acid tests is available on GitHub. Of course, I believe that native MathML implementation is very important and I expect at least that these tests could help the MathML community ; users and implementers.

Here is the result of the MathML Acid2 test with the stable Gecko release. To pass the test we only need to implement negative spacing or at least integrate the patch I submitted when I was still active in Gecko developments (bug 717546).

MathML Acid2 test ; Gecko

And here is the score of the MathML Acid 3 test with the stable Gecko release. The failure of test 18 was not supposed to happen but I discovered it when I wrote the test. That will be fixed by James Kitchener's refactoring in bug 827713. Obviously, reaching the score of 100/100 will be much more difficult to achieve by our volunteer developers, but the current score is not too bad compared to other rendering engines...

MathML Acid 3 ; Gecko

By fredw at March 02, 2013 06:19 PM

August 01, 2012

WebKit CSS3 text-decoration properties (preview)

Bruno de Oliveira Abinader

WebKit currently supports CSS Text Level 2.1 version of text-decoration property (link). This version treats only about the decoration line types (underline, overline, line-through and blink – the latter is not supported on WebKit).

The draft version of CSS Text Level 3 upgrades text-decoration (link) property as a shorthand to 3 newly added properties, named text-decoration-line (link), text-decoration-style (link) and text-decoration-color (link), and also adds text-decoration-skip (link) property.

Among other WebKit stuff I’ve been doing lately, this feature implementation is one of the most cool ones I’m enjoying implementing. I’ve grabbed the task of implementing all of these CSS3 text-decoration* properties on WebKit, and results are great so far!

As you can see below, these are the new text decoration styles (solid, double, dotted, dashed and wavy – the latter still requires platform support) available:

Text decoration style layout test results on Qt platform

And also specific text decoration colors can be set:

Text decoration color layout test results on Qt platform

These features (with exception to text-decoration-skip property) are already implemented on Firefox, thus it gets easier to compare results with different web engines. It is important to notice since CSS3 specification is still in development, all these properties have a -webkit- prefix (ie. -webkit-text-decoration), so text-decoration still maintains CSS2.1 specification requirements. The patches are being reviewed and will soon land upstream, let’s hope it will be soon!

By Bruno Abinader at August 01, 2012 09:58 PM

April 11, 2012

A guide for Qt5/WebKit2 development setup for Nokia N9 on Ubuntu Linux

Bruno de Oliveira Abinader

As part of my daily activities at basysKom on QtWebKit maintenance and development for Nokia devices, it is interesting to keep a track on latest developments circa QtWebKit. There is currently a promising project of a Qt5/WebKit2-based browser called Snowshoe mainly developed by my fellow friends from INdT which is completely open-source. This browser requires latest Qt5 and QtWebKit binaries and thus requires us to have a functional build system environment. There is a guide available on WebKit’s wiki (link) which is very helpful but lacks some information about compilation issues found when following the setup steps. So I am basing this guide from that wiki page and I hope that it gets updated soon :)

On this guide it is assumed the following:

  • All commands are issued on a Linux console. I am not aware of how this guide would work on other systems.
  • All commands are supposed to be issued inside base directory, unless expressely said otherwise (ie. cd <QT5_DIR>).
  • You might want to check if you have git and rsync packages installed in your system.

1. Install Qt SDK

In order to build Qt5 and QtWebKit for Nokia N9, you need to set up a cross-compiler. Thankfully, Qt SDK already comes with a working setup. Please download the online installer from Qt Downloads section (link).

NOTE: The offline installer comes with an outdated version of the MADDE target, which can be updated by running the script below and chosing “Update components” when asked:

$ ~/QtSDK/SDKMaintenanceTool

2. Directory setup

It is suggested (and actually required by some build scripts) to have a base directory which holds Qt5, Qt Components and WebKit project sources. The suggested base directory can be created by running:

$ mkdir -p ~/swork

NOTE: You can actually choose another directory name, but so far it is required by some scripts to have at least a symbolic link pointing to <HOME_DIR>/swork.

3. Download convenience scripts

3.1. browser-scripts

$ git clone https://github.com/resworb/scripts.git browser-scripts

3.2. rsync-scripts

$ wget http://trac.webkit.org/attachment/wiki/SettingUpDevelopmentEnvironmentForN9/rsync-scripts.tar.gz?format=raw
$ tar xzf rsync-scripts.tar.gz

4. Download required sources

4.1. testfonts

$ git clone git://gitorious.org/qtwebkit/testfonts.git

4.2. Qt5, QtComponents and WebKit

The script below when successfully run will create ~/swork/qt5, ~/swork/qtcomponents and ~/swork/webkit directories:

$ browser-scripts/clone-sources.sh --no-ssh

NOTE: You can also manually download sources, but remember to stick with the directory names described above.

5. Pre-build hacks

5.1. Qt5 translations

Qt5 translations are not being properly handled by cross-platform toolchain. This happens mainly because lrelease application is called to generate Qt message files, but since it is an ARMEL binary your system is probably not capable of running it natively (unless you have a misc_runner kernel module properly set, then you can safely skip this step). In this case, you can use lrelease from your system’s Qt binaries without any worries.

If you have a Scratchbox environment set, it is suggested for you to stop its service first:

$ sudo service scratchbox-core stop

Now you can manually generate Qt message files by running this:

$ cd ~/swork/qt5/qttranslations/translations
$ for file in `ls *ts`; do lrelease $file -qm `echo "$file" | sed 's/ts$/qm/'`; done

5.2. Disable jsondb-client tool

QtJsonDB module from Qt5 contains a tool called jsondb-client, which depends on libedit (not available on MADDE target). It is safe to disable its compilation for now:

$ sed -i 's/jsondb-client//' ~/swork/qt5/qtjsondb/tools/tools.pro

5.3. Create missing symbolic links

Unfortunately Qt5 build system is not robust enough to support our cross-compilation environment, so some symbolic links are required on MADDE to avoid compilation errors (where <USER> is your system user name):

$ ln -s ~/swork/qt5/qtbase/include ~/QtSDK/Madde/sysroots/harmattan_sysroot_10.2011.34-1_slim/home/<USER>/swork/qt5/qtbase
$ ln -s ~/swork/qt5/qtbase/mkspecs ~/QtSDK/Madde/sysroots/harmattan_sysroot_10.2011.34-1_slim/home/<USER>/swork/qt5/mkspecs

6. Build sources

You can execute the script that will build all sources using cross-compilation setup:

$ browser-scripts/build-sources.sh --cross-compile

If everything went well, you now have the most up-to-date binaries for Qt5/WebKit2 development for Nokia N9. Please have a look at WebKit’s wiki for more information about how to update sources after a previous build and information on how to keep files in sync with device. The guide assumes PR1.1 firmware for N9 device, which is already outdated, so I might come up next with updated instructions on how to safely sync files to your PR1.2-enabled device.

That’s all for now, I appreciate your comments and feedback!

By Bruno Abinader at April 11, 2012 07:18 AM

March 10, 2012

WebKitGTK+ Debian packaging repository changes

Gustavo Noronha

For a while now the git repository used for packaging WebKitGTK+ has been broken. Broken as in nobody was able to clone it. In addition to that, the packaging workflow had been changing over time, from a track-upstream-git/patches applied one to a import-orig-only/patches-not-applied one.

After spending some more time trying to unbreak the repository for the third time I decided it might be a good time for a clean up. I created a new repository, imported all upstream versions for series 1.2.x (which is in squeeze), 1.6.x (unstable), and 1.7.x (experimental). I also imported packaging-related commis for those versions using git format-patch and black magic.

One of the good things about doing this move, and which should make hacking the WebKitGTK+ debian package more pleasant and accessible can be seen here:


kov@goiaba ~/s/debian-webkit> du -sh webkit/.git webkit.old/.git
27M webkit/.git
1.6G webkit.old/.git

If you care about the old repository, it’s on git.debian.org still, named old-webkit.git. Enjoy!

By kov at March 10, 2012 05:32 PM

December 07, 2011

WebKitGTK+ hackfest \o/

Gustavo Noronha

It’s been a couple days since I returned from this year’s WebKitGTK+ hackfest in A Coruña, Spain. The weather was very nice, not too cold and not too rainy, we had great food, great drinks and I got to meet new people, and hang out with old friends, which is always great!

Hackfest black board, photo by Mario

I think this was a very productive hackfest, and as usual a very well organized one! Thanks to the GNOME Foundation for the travel sponsorship, to our friends at Igalia for doing an awesome job at making it happen, and to Collabora for sponsoring it and granting me the time to go there! We got a lot done, and although, as usual, our goals list had many items not crossed, we did cross a few very important ones. I took part in discussions about the new WebKit2 APIs, got to know the new design for GNOME’s Web application, which looks great, discussed about Accelerated Compositing along with Joone, Alex, Nayan and Martin Robinson, hacked libsoup a bit to port the multipart/x-mixed-replace patch I wrote to the awesome gio-based infrastructure Dan Winship is building, and some random misc.

The biggest chunk of time, though, ended up being devoted to a very uninteresting (to outsiders, at least), but very important task: making it possible to more easily reproduce our test results. TL;DR? We made our bots’ and development builds use jhbuild to automatically install dependencies; if you’re using tarballs, don’t worry, your usual autogen/configure/make/make install have not been touched. Now to the more verbose version!

The need

Our three build slaves reporting a few failures

For a couple years now we have supported an increasingly complex and very demanding automated testing infrastructure. We have three buildbot slaves, one provided by Collabora (which I maintain), and two provided by Igalia (maintained by their WebKitGTK+ folks). Those bots build as many check ins as possible with 3 different configurations: 32 bits release, 64 bits release, and 64 bits debug.

In addition to those, we have another bot called the EWS, or Early Warning System. There are two of those at this moment: one VM provided by Collabora and my desktop, provided by myself. These bots build every patch uploaded to the bugzilla, and report build failures or passes (you can see the green bubbles). They are very important to our development process because if the patch causes a build failure for our port people can often know that before landing, and try fixes by uploading them to bugzilla instead of doing additional commits. And people are usually very receptive to waiting for EWS output and acting on it, except when they take way too long. You can have an idea of what the life of an EWS bot looks like by looking at the recent status for the WebKitGTK+ bots.

Maintaining all of those bots is at times a rather daunting task. The tests require a very specific set of packages, fonts, themes and icons to always report the same size for objects in a render. Upgrades, for instance, had to be synchronized, and usually involve generating new baselines for a large number of tests. You can see in these instructions, for instance, how strict the environment requirements are – yes, we need specific versions of fonts, because they often cause layouts to change in size! At one point we had tests fail after a compiler upgrade, which made rounding act a bit different!

So stability was a very important aspect of maintaining these bots. All of them have the same version of Debian, and most of the packages are pinned to the same version. On the other hand, and in direct contradition to the stability requirement, we often require bleeding edge versions of some libraries we rely on, such as libsoup. Since we started pushing WebKitGTK+ to be libsoup-only, its own progress has been pretty much driven by WebKitGTK+’s requirements, and Dan Winship has made it possible to make our soup backend much, much simpler and way more featureful. That meant, though, requiring very recent versions of soup.

To top it off, for anyone not running Debian testing and tracking the exact same versions of packages as the bots it was virtually impossible to get the tests to pass, which made it very difficult for even ourselves to make sure all patches were still passing before committing something. Wow, what a mess.

The explosion^Wsolution

So a few weeks back Martin Robinson came up with a proposed solution, which, as he says, is the “nuclear bomb” solution. We would have a jhbuild environment which would build and install all of the dependencies necessary for reproducing the test expectations the bots have. So over the first three days of the hackfest Martin and myself hacked away in building scripts, buildmaster integration, a jhbuild configuration, a jhbuild modules file, setting up tarballs, and wiring it all in a way that makes it convenient for the contributors to get along with. You’ll notice that our buildslaves now have a step just before compiling called “updated gtk dependencies” (gtk is the name we use for our port in the context of WebKit), which runs jhbuild to install any new dependencies or version bumps we added. You can also see that those instructions I mentioned above became a tad simpler.

It took us way more time than we thought for the dust to settle, but it eventually began to. The great thing of doing it during the hackfest was that we could find and fix issues with weird configurations on the spot! Oh, you build with AR_FLAGS=cruT and something doesn’t like it? OK, we fix it so that the jhbuild modules are not affected by that variable. Oh, turns out we missed a dependency, no problem, we add it to the modules file or install them on the bots, and then document the dependency. I set up a very clean chroot which we could use for trying out changes so as to not disrupt the tree too much for the other hackfest participants, and I think overall we did good.

The aftermath

By the time we were done our colleagues who ran other distributions such as Fedora were already being able to get a substantial improvements to the number of tests passing, and so did we! Also, the ability to seamlessly upgrade all the bots with a simple commit made it possible for us to very easily land a change that required a very recent (as in unreleased) version of soup which made our networking backend way simpler. All that red looks great, doesn’t it? And we aren’t done yet, we’ll certainly be making more tweaks to this infrastructure to make it more transparent and more helpful to the users (contributors and other people interested in running the tests).

If you’ve been hit by the instability we caused, sorry about that, poke mrobinson or myself in the #webkitgtk+ IRC channel on FreeNode, and we’ll help you out or fix any issues. If you haven’t, we hope you enjoy all the goodness that a reproducible testing suite has to offer! That’s it for now, folks, I’ll have more to report on follow-up work started at the hackfest soon enough, hopefully =).

By kov at December 07, 2011 11:34 PM

November 29, 2011

Accelerated Compositing in webkit-clutter

Gustavo Noronha

For a while now my fellow Collaboran Joone Hur has been working on implementing the Accelerated Compositing infrastructure available in WebKit in webkit-clutter, so that we can use Clutter’s powers for compositing separate layers and perform animations. This work is being done by Collabora and is sponsored by BOSCH, whom I’d like to thank! What does all this mean, you ask? Let me tell me a bit about it.

The way animations usually work in WebKit is by repainting parts of the page every few milliseconds. What that means in technical terms is that an area of the page gets invalidated, and since the whole page is one big image, all of the pieces that are in that part of the page have to be repainted: the background, any divs, images, text that are at that part of the page.

What the accelerated compositing code paths allow is the creation of separate pieces to represent some of the layers, allowing the composition to happen on the GPU, removing the need to perform lots of cairo paint operations per second in many cases. So if we have a semi-transparent video moving around the page, we can have that video be a separate texture that is layered on top of the page, made transparent and animated by the GPU. In webkit-clutter’s case this is done by having separate actors for each of the layers.

I have been looking at this code on and off, and recently joined Joone in the implementation of some of the pieces. The accelerated compositing infrastructure was originally built by Apple and is, for that reason, works in a way that is very similar to Core Animation. The code is still a bit over the place as we work on figuring out how to best translate the concepts into clutter concepts and there are several bugs, but some cool demos are already possible! Bellow you have one of the CSS3 demos that were made by Apple to demo this new functionality running on our MxLauncher test browser.

You can also see that the non-Accelerated version is unable to represent the 3D space correctly. Also, can you guess which of the two MxLauncher instances is spending less CPU? ;) In this second video I show the debug borders being painted around the actors that were created to represent layers.

The code, should you like to peek or test is available in the ac2 branch of our webkit-clutter repository: http://gitorious.org/webkit-clutter/webkit-clutter/commits/ac2

We still have plenty of work to do, so expect to hear more about it. During our annual hackfest in A Coruña we plan to discuss how this work could be integrated also in the WebKitGTK+ port, perhaps by taking advantage of clutter-gtk, which would benefit both ports, by sharing code and maintenance, and providing this great functionality to Epiphany users. Stay tuned!

By kov at November 29, 2011 05:55 PM

October 09, 2011

Tests Active

Brent Fulgham


Looking back over this blog, I see that it was around a year ago that I got the initial WinCairo buildbot running. I'm very pleased to announce that I have gotten ahold of a much more powerful machine, and am now able to run a full build and tests in slightly under an hour -- a huge improvement over the old hardware which took over two hours just to build the software!

This is a big step, because we can now track regressions and gauge correctness compared to the other platforms. Up to now, testing has largely consisted of periodic manual runs of the test suite, and a separate set of high-level tests run as part of a larger application. This was not ideal, because it was easy for low-level functions in WebKit that I rarely use to be broken and missed.

All is not perfect, of course. Although over 12,000 tests now run (successfully) with each build, that is effectively two thirds of the full test suite. Most of the tests I have disabled are due to small differences in the output layout. I'm trying to understand why these differences exist, but I suspect many of them simply reflect small differences in Cairo compared to the CoreGraphics rendering layer.

If any of you lurkers are interested in helping out, trying out some of the tests I have disabled and figuring out why they fail would be a huge help!

By Brent Fulgham (noreply@blogger.com) at October 09, 2011 02:43 AM

July 14, 2011

An Unseasonable Snowfall

Brent Fulgham

A year or two ago I ported the Cocoa "CallJS" application to MFC for use with WebKit. The only feedback I ever got on the topic was a complaint that it would not build under the Visual Studio Express software many people used.

After seeing another few requests on the webkit-help mailing list for information on calling JavaScript from C++ (and vice-versa), I decided to dust off the old program and convert it to pure WINAPI calls so that VS Express would work with it.

Since my beloved Layered Window patches finally landed in WebKit, I also incorporated a transparent WebKit view floating over the main application window. Because I suck at art, I stole appropriated the Let It Snow animation example to give the transparent layer something to do.

Want to see what it looks like?

By Brent Fulgham (noreply@blogger.com) at July 14, 2011 06:34 PM

July 10, 2011

Updated WebKit SDK (@r89864)

Brent Fulgham

I have updated the WebKitSDK to correspond to SVN revision r8984.

Major changes in this revision:
* JavaScript engine improvements.
* Rendering improvements.
* New 'Transparent Web View' support.
* General performance and memory use improvements.

This ZIP file also contains updated versions of Zlib, OpenSSL, cURL, and OpenCFLite.

Note that I have stopped statically linking Cairo; I'm starting to integrate some more recent Cairo updates (working towards some new rendering features), and wanted to be able to update it incrementally as changes are made.

This package contains the same Cairo library (in DLL form) as used in previous versions.

As usual, please let me know if you encounter any problems with this build.

[Update] I forgot to include zlib1.dll! Fixed in the revised zip file.

By Brent Fulgham (noreply@blogger.com) at July 10, 2011 04:24 AM

July 05, 2011

WinCairoRequirements Sources Archive

Brent Fulgham

I've posted the 80 MB source archive of the requirements needed to build the WinCairo port of WebKit.

Note that you do NOT need these sources unless you plan on building them yourself or wish to archive the source code for these modules. The binaries are always present in the WinCairoRequirements.zip file, which is downloaded and unzipped to the proper place when you execute the update-webkit --wincairo command.

By Brent Fulgham (noreply@blogger.com) at July 05, 2011 07:39 PM

June 28, 2011

Towards a Simpler WinCairo Build

Brent Fulgham


For the past couple of years, anyone interested in trying to build the WinCairo port of WebKit had to track down a number of support libraries, place them in their development environment's include (and link search) paths, and then cross their fingers and hope everything built.

To make things a little easier, I wrapped up the libraries and headers I use for building and posted them as a zip file on my .Mac account. This made things a little easier, but you still had to figure out where to drop the files and figure out if I had secretly updated my 'requirements.zip' file without telling anyone. Not ideal.

A couple of days ago, while trolling through the open review queue, I ran across a Bug filed by Carl Lobo, which automated the task of downloading the requirements file when running build-webkit --wincairo. This was a huge improvement!

Today, I hijacked Carl's changes and railroaded the patch through the review process (making a few modifications along the way):

  • I renamed my requirements file WinCairoRequirements.zip.

  • I added a timestamp file, so that build-webkit --wincairo can check to see if the file changed, and download it if necessary.

  • I propagated Carl's changes to update-webkit, so that now by adding the --wincairo argument it will update the WinCairoRequirements file.


I'm really excited about this update. If you've been wanting to try out the WinCairo port of WebKit, this would be a great time to try it out. I'd love to hear your experiences!

By Brent Fulgham (noreply@blogger.com) at June 28, 2011 04:42 AM

June 14, 2011

Benchmarking Javascript engines for EFL

Lucas De Marchi

The Enlightenment Foundation Libraries has several bindings for other languages in order to ease the creation of end-user applications, speeding up its development. Among them, there’s a binding for Javascript using the Spidermonkey engine. The questions are: is it fast enough? Does it slowdown your application? Is Spidermonkey the best JS engine to be used?

To answer these questions Gustavo Barbieri created some C, JS and Python benchmarks to compare the performance of EFL using each of these languages. The JS benchmarks were using Spidermonkey as the engine since elixir was already done for EFL. I then created new engines (with only the necessary functions) to also compare to other well-known JS engines: V8 from Google and JSC (or nitro) from WebKit.

Libraries setup

For all benchmarks EFL revision 58186 was used. Following the setup of each engine:

  • Spidermonkey: I’ve used version 1.8.1-rc1 with the already available bindings on EFL repository, elixir;
  • V8: version 3.2.5.1, using a simple binding I created for EFL. I named this binding ev8;
  • JSC: WebKit’s sources are needed to compile JSC. I’ve used revision 83063. Compiling with CMake, I chose the EFL port and enabled the option SHARED_CORE in order to have a separated library for Javascript;

Benchmarks

Startup time: This benchmark measures the startup time by executing a simple application that imports evas, ecore, ecore-evas and edje, bring in some symbols and then iterates the main loop once before exiting. I measured the startup time for both hot and cold cache cases. In the former the application is executed several times in sequence and the latter includes a call to drop all caches so we have to load the library again from disk

Runtime – Stress: This benchmark executes as many frames per second as possible of a render-intensive operation. The application is not so heavy, but it does some loops, math and interacts with EFL. Usually a common application would do far less operations every frame because many operations are done in EFL itself, in C, such as list scrolling that is done entirely in elm_genlist. This benchmark is made of 4 phases:

  • Phase 0 (P0): Un-scaled blend of the same image 16 times;
  • Phase 1 (P1): Same as P0, with additional 50% alpha;
  • Phase 2 (P2): Same as P0, with additional red coloring;
  • Phase 3 (P3): Same as P0, with additional 50% alpha and red coloring;

The C and Elixir’s versions are available at EFL repository.

Runtime – animation: usually an application doesn’t need “as many FPS as possible”, but instead it would like to limit to a certain amount of frames per second. E.g.: iphone’s browser tries to keep a constant of 60 FPS. This is the value I used on this benchmark. The same application as the previous benchmark is executed, but it tries to keep always the same frame-rate.

Results

The first computer I used to test these benchmarks on was my laptop. It’s a Dell Vostro 1320, Intel Core 2 Duo with 4 GB of RAM and a standard 5400 RPM disk. The results are below.

Benchmarks on Dell 1320 laptop

First thing to notice is there are no results for “Runtime – animation” benchmark. This is because all the engines kept a constant of 60fps and hence there were no interesting results to show. The first benchmark shows that V8’s startup time is the shortest one when considering we have to load the application and libraries from disk. JSC was the slowest and  Spidermonkey was in between.

With hot caches, however, we have another complete different scenario, with JSC being almost as fast as the native C application. Following, V8 with a delay a bit larger and Spidermonkey as the slowest one.

The runtime-stress benchmark shows that all the engines are performing well when there’s some considerable load in the application, i.e. removing P0 from from this scenario. JSC was always at the same speed of native code; Spidermonkey and V8 had an impact only when considering P0 alone.

 

Next computer to consider in order to execute these benchmarks was  a Pandaboard, so we can see how well the engines are performing in an embedded platform. Pandaboard has an ARM Cortex-A9 processor with 1GB of RAM and the partition containing the benchmarks is in an external flash storage drive. Following the results for each benchmark:

 

Benchmarks on Pandaboard

Once again, runtime-animation is not shown since it had the same results for all engines. For the startup tests, now Spidermonkey was much faster than the others, followed by V8 and JSC in both hot and cold caches. In runtime-stress benchmark, all the engines performed well, as in the first computer, but now JSC was the clear winner.

 

There are several points to be considered when choosing an engine to be use as a binding for a library such as EFL. The raw performance and startup time seems to be very near to the ones achieved with native code. Recently there were some discussions in EFL mailing list regarding which engine to choose, so I think it would be good to share these numbers above. It’s also important to notice that these bindings have a similar approach of elixir, mapping each function call in Javascript to the correspondent native function. I made this to be fair in the comparison among them, but depending on the use-case it’d  be good to have a JS binding similar to what python’s did, embedding the function call in real python objects.

By Lucas De Marchi at June 14, 2011 05:25 PM

April 29, 2011

Collection of WebKit ports

Holger Freyther

WebKit is a very successfull project. It is that in many ways. The code produced seems to very fast, the code is nice to work on, the people are great, the partys involved collaborate with each other in the interest of the project. The project is also very successfull in the mobile/smartphone space. All the major smartphone platforms but Windows7 are using WebKit. This all looks great, a big success but there is one thing that stands out.

From all the smartphone platforms no one has fully upstreamed their port. There might be many reasons for that and I think the most commonly heard reason is the time needed to get it upstreamed. It is specially difficult in a field that is moving as fast as the mobile industry. And then again there is absolutely no legal obligation to work upstream.

For most of today I collected the ports I am aware of, put them into one git repository, maybe find the point where they were branched, rebase their changes. The goal is to make it more easy to find interesting things and move them back to upstream. One can find the combined git tree with the tags here. I started with WebOS, moved to iOS, then to Bada and stopped at Android as I would have to pick the sourcecode for each android release for each phone from each vendor. I think I will just be happy with the Android git tree for now. At this point I would like to share some of my observations in the order I did the import.

Palm


Palm's release process is manual. In the last two releases they call the file .tgz but forgot to gzip it, in 2.0.0 the tarball name was in camel case. The thing that is very nice about Palm is that they provide their base and their changes (patch) separately. From looking at the 2.1.0 release it looks that for the Desktop version they want to implement Complex Font rendering. Earlier versions (maybe it is still the case) lack the support for animated GIF.

iOS


Apple's release process seems to be very structured. The source can be downloaded here. What I think is to note is that the release tarball contains some implementations of WebCore only as .o file and Apple has stopped releasing the WebKit sourcecode beginning with iOS 4.3.0.

Bada


This port is probably not known by many. The release process seems to be manual as well, the name of directories changed a lot between the releases, they come with a WML Script engine and they do ship something they should not ship.

I really hope that this combined tree is useful for porters that want to see the tricks used in the various ports and don't want to spend the time looking for each port separately.

By zecke (noreply@blogger.com) at April 29, 2011 07:20 PM

February 13, 2011

How to make the GNU Smalltalk Interpreter slower

Holger Freyther

This is another post about a modern Linux based performance measurement utility. It is called perf, it is included in the Linux kernel sources and it entered the kernel in v2.6.31-rc1. In many ways it is obsoleting OProfile, in fact for many architectures oprofile is just a wrapper around the perf support in the kernel. perf comes with a few nice application. perf top provides a statistics about which symbols in user and in kernel space are called, perf record to record an application or to start an application to record it and then perf report to browse this report with a very simple CLI utility. There are tools to bundle the record and the application in an archive, a diff utility.

For the last year I was playing a lot with GNU Smalltalk and someone posted the results of a very simplistic VM benchmark ran across many different Smalltalk implementations. In one of the benchmarks GNU Smalltalk is scoring last among the interpreters and I wanted to understand why it is slower. In many cases the JavaScriptCore interpreter is a lot like the GNU Smalltalk one, a simple direct-threaded bytecode interpreter, uses computed goto (even is compiled with -fno-gcse as indicated by the online help, not that it changed something for JSC), heavily inlined many functions.

There are also some differences, the GNU Smalltalk implementation is a lot older and in C. The first notable is that it is a Stack Machine and not register based, there are global pointers for the SP and the IP. Some magic to make sure that in the hot loop the IP/SP is 'local' in a register, depending on the available registers also keep the current argument in one, the interpreter definition is in a special file format but mostly similar to how Interepreter::privateExecute is looking like. The global state mostly comes from the fact that it needs to support switching processes and there might be some event during the run that requires access to the IP to store it to resume the old process. But in general the implementation is already optimized and there is little low hanging fruits and most experiments result in a slow down.

The two important things are again: Having a stable benchmark, having a tool to help to know where to look for things. In my case the important tools are perf stat, perf record, perf report and perf annotate. I have put a copy of the output to the end of this blog post. The stat utility provides one with number of instructions executed, branches, branch misses (e.g. badly predicted), L1/L2 cache hits and cache misses.

The stable benchmark helps me to judge if a change is good, bad or neutral for performance within the margin of error of the test. E.g. if I attempt to reduce the code size the instructions executed should decrease, if I start putting __builtin_expect.. into my code the number of branch misses should go down as well. The other useful utility is to the perf report that allows one to browse the recorded data, this can help to identify the methods one wants to start to optimize, it allows to annotate these functions inside the simple TUI interface, but does not support searching in it.

Because the codebase is already highly optimized any of my attempts should either decrease the code size (and the pressure on the i-cache), the data size (d-cache), remove stores or loads from memory (e.g. reorder instructions), fix branch predictions. The sad truth is that most of my changes were either slow downs or neutral to the performance and it is really important to undo these changes and not have false pride (unless it was also a code cleanup or such).

So after about 14 hours of toying with it the speed ups I have managed to make come from inlining a method to unwind a context (callframe), reordering some compares on the GC path and disabling the __builtin_expect branch hints as they were mostly wrong (something the kernel people found to be true in 2010 as well). I will just try harder, or try to work on the optimizer or attempt something more radical...



$ perf stat gst -f Bench.st
219037433 bytecodes/sec; 6025895 sends/sec

Performance counter stats for 'gst -f Bench.st':

17280.101683 task-clock-msecs # 0.969 CPUs
2076 context-switches # 0.000 M/sec
123 CPU-migrations # 0.000 M/sec
3925 page-faults # 0.000 M/sec
22215005506 cycles # 1285.583 M/sec (scaled from 70.02%)
40593277297 instructions # 1.827 IPC (scaled from 80.00%)
5063469832 branches # 293.023 M/sec (scaled from 79.98%)
70691940 branch-misses # 1.396 % (scaled from 79.98%)
27844326 cache-references # 1.611 M/sec (scaled from 20.02%)
134229 cache-misses # 0.008 M/sec (scaled from 20.03%)

17.838888599 seconds time elapsed


PS: The perf support probably works best on Intel based platforms and the biggest other problem is that perf annotate has some issues when the code is included from other c files.

By zecke (noreply@blogger.com) at February 13, 2011 08:56 PM

January 17, 2011

Using systemtap userspace tracing...

Holger Freyther

At the 27C3 we were running a GSM network and during the preparation I noticed a strange performance problem coming from the database library we are using running. I filled our database with some dummy data and created a file with the queries we normally run and executed time cat queries | sqlite3 file as a mini benchmark. I also hacked this code into our main routine and ran it with time as well. For some reason the code running through the database library was five times slower.

I was a bit puzzled and I decided to use systemtap to explore this to build a hypothesis and to also have the tools to answer the hypothesis. I wanted to find out if if it is slow because our database library is doing some heavy work in the implementation, or because we execute a lot more queries behind the back. I was creating the below probe:


probe process("/usr/lib/libsqlite3.so.0.8.6").function("sqlite3_get_table")
{
a = user_string($zSql);
printf("sqlite3_get_table called '%s'\n", a);
}


This probe will be executed whenever the sqlite3_get_table function of the mentioned library will be called. The $zSql is a variable passed to the sqlite3_get_table function and contains the query to be executed. I am converting the pointer to a local variable and then can print it. Using this simple probe helped me to see which queries were executed by the database library and helped me to do an easy optimisation.

In general it could be very useful to build a set of probes (I think one calls set a tapset) that check for API misusage, e.g. calling functions with certain parameters where something else might be better. E.g. in Glib use truncate instead of assigning "" to the GString, or check for calls to QString::fromUtf16 coming from Qt code itself. On second thought this might be better as a GCC plugin, or both.

By zecke (noreply@blogger.com) at January 17, 2011 12:41 PM

December 17, 2010

In the name of performance

Holger Freyther

I tend to see people doing weird things and then claim that the change is improving performance. This can be re-ordering instructions to help the compiler, attempting to use multiple cores of your system, writing a memfill in assembly. On the one hand people can be right and the change is making things faster, on the other hand they could use assembly to make things look very complicated, justify their pay, and you might feel awkward to question if it is making any sense.

In the last couple of weeks I have stumbled on some of those things. For some reason I found this bug report about GLIBC changing the memcpy routine for SSE and breaking the flash plugin (because it uses memcpy in the wrong way). The breakage is justified that the new memcpy was optimized and is faster. As Linus points out with his benchmark the performance improvement is mostly just wishful thinking.

Another case was someone providing MIPS optimized pixman code to speed-up all drawing which turned out to be wishful thinking as well...

The conclusion is. If someone claims that things are faster with his patch. Do not simply trust him, make sure he refers to his benchmark, is providing numbers of before and after and maybe even try to run it yourself. If he can not provide this, you should wonder how he measured the speed-up! There should be no place for wishful thinking in benchmarking. This is one of the areas where Apple's WebKit team is constantly impressing me.

By zecke (noreply@blogger.com) at December 17, 2010 01:48 PM

December 16, 2010

Benchmarking QtWebKit-V8 on Linux

University of Szeged

For some time it has been possible to build and run QtWebKit on Linux using Google's V8 JavaScript engine instead of the default JavaScriptCore. I thought it would be good to see some numbers comparing the runtime performance of the two engines in the same environment and also measuring the performance of the browser bindings.

read more

By andras.becsi at December 16, 2010 01:04 PM

October 23, 2010

Easily embedding WebKit into your EFL application

Lucas De Marchi

This is the first of a series of posts that I’m planning to do using basic examples in EFL, the Enlightenment Foundation Libraries. You may have heard that EFL is reaching its 1.0 release. Instead of starting from the very beginning with the basic functions of these libraries, I decided to go the opposite way, showing the fun stuff that is possible to do. Since I’m also an WebKit developer, let’s put the best of both softwares together and have a basic window rendering a webpage.

Before starting off, just some remarks:

  1. I’m using here the basic EFL + WebKit-EFL (sometimes called ewebkit). Developing an EFL application can be much simpler, particularly if you use an additional library with pre-made widgets like Elementary. However, it’s good to know how the underlying stuff works, so I’m providing this example.
  2. This could have been the last post in a series when talking about EFL since it uses at least 3 libraries. Don’t be afraid if you don’t understand what a certain function is for or if you can’t get all EFL and WebKit running right now. Use the comment section below and I’ll make my best to help you.

Getting EFL and WebKit

In order to able to compile the example here, you will need to compile two libraries from source: EFL and WebKit. For both libraries, you can either get the last version from svn or use the last snapshots provided.

  • EFL:

Grab a snapshot from the download page. How to checkout the latest version from svn is detailed here, as well as some instructions on how to compile

  • WebKit-EFL:

A very detailed explanation on how to get WebKit-EFL up and running is available on trac. Recently, though, WebKit-EFL started to be released too. It’s not detailed in the wiki yet, but you can grab a snapshot instead of checking out from svn.

hellobrowser!

In the spirit of “hello world” examples, our goal here is to make a window showing a webpage rendered by WebKit. For the sake of simplicity, we will use a default start page and put a WebKit-EFL “widget” to cover the entire window. See below a screenshot:

hellobrowser - WebKit + EFL

The code for this example is available here. Pay attention to a comment in the beginning of this file that explains how to compile it:

gcc -o hellobrowser hellobrowser.c \
     -DEWK_DATADIR="\"$(pkg-config --variable=datadir ewebkit)\"" \
     $(pkg-config --cflags --libs ecore ecore-evas evas ewebkit)

The things worth noting here are the dependencies and a variable. We directly depend on ecore and evas from EFL and on WebKit. We define a variable, EWK_DATADIR, using pkg-config so our browser can use the default theme for web widgets defined in WebKit. Ecore handles events like mouse and keyboard inputs, timers etc whilst evas is the library responsible for drawing. In a later post I’ll detail them a bit more. For now, you can read more about them on their official site.

The main function is really simple. Let’s divide it by pieces:

    // Init all EFL stuff we use
    evas_init();
    ecore_init();
    ecore_evas_init();
    ewk_init();

Before you use a library from EFL, remember to initialize it. All of them use their own namespace, so it’s easy to know which library you have to initialize: for example, if you call a function starting by “ecore_”, you know you first have to call “ecore_init()”. The last initialization function is WebKit’s, which uses the “ewk_” namespace.

    window = ecore_evas_new(NULL, 0, 0, 800, 600, NULL);
    if (!window) {
        fprintf(stderr, "something went wrong... :(\n");
        return 1;
    }

Ecore-evas then is used to create a new window with size 800×600. The other options are not relevant for an introduction to the libraries and you can find its complete documentation here.

    // Get the canvas off just-created window
    evas = ecore_evas_get(window);

From the Ecore_Evas object we just created, we grab a pointer to the evas, which is the space in which we can draw, adding Evas_Objects. Basically an Evas_Object is an object that you draw somewhere, i.e. in the evas. We want to add only one object to our window, that is where WebKit you render the webpages. Then, we have to ask WebKit to create this object:

    // Add a View object into this canvas. A View object is where WebKit will
    // render stuff.
    browser = ewk_view_single_add(evas);

Below I demonstrate a few Evas’ functions that you use to manipulate any Evas_Object. Here we are manipulating the just create WebKit object, moving to the desired position, resizing to 780x580px and then telling Evas to show this object. Finally, we tell Evas to show the window we created too. This way we have a window with an WebKit object inside with a little border.

    // Make a 10px border, resize and show
    evas_object_move(browser, 10, 10);
    evas_object_resize(browser, 780, 580);
    evas_object_show(browser);
    ecore_evas_show(window);

We need to setup a bit more things before having a working application. The first one is to give focus to the Evas_Object we are interested on in order to receive keyboard events when opened. Then we connect a function that will be called when the window is closed, so we can properly exit our application.

    // Focus it so it will receive pressed keys
    evas_object_focus_set(browser, 1);
 
    // Add a callback so clicks on "X" on top of window will call
    // main_signal_exit() function
    ecore_event_handler_add(ECORE_EVENT_SIGNAL_EXIT, main_signal_exit, window);

After this, we are ready to show our application, so we start the mainloop. This function will only return when the application is closed:

    ecore_main_loop_begin();

The function called when the application is close, just tell Ecore to exit the mainloop, so the function above returns and the application can shutdown. See its implementation below:

static Eina_Bool
main_signal_exit(void *data, int ev_type, void *ev)
{
    ecore_evas_free(data);
    ecore_main_loop_quit();
    return EINA_TRUE;
}

Before the application exits, we shutdown all the libraries that were initialized, in the opposite order:

    // Destroy all the stuff we have used
    ewk_shutdown();
    ecore_evas_shutdown();
    ecore_shutdown();
    evas_shutdown();

This is a basic working browser, with which you can navigate through pages, but you don’t have an entry to set the current URL, nor “go back” and “go forward” buttons etc. All you have to do is start adding more Evas_Objects to your Evas and connect them to the object we just created. For a still basic example, but with more stuff implemented, refer to the EWebLauncher that we ship with the WebKit source code. You can see it in the “WebKitTools/EWebLauncher/” folder or online at webkit’s trac. Eve is another browser with a lot more features that uses Elementary in addition to EFL, WebKit. See a blog post about it with some nice pictures.

Now, let’s do something funny with our browser. With a bit more lines of code you can turn your browser upside down. Not really useful, but it’s funny. All you have to do is to rotate the Evas_Object WebKit is rendering on. This is implemented by the following function:

// Rotate an evas object by 180 degrees
static void
_rotate_obj(Evas_Object *obj)
{
    Evas_Map *map = evas_map_new(4);
 
    evas_map_util_points_populate_from_object(map, obj);
    evas_map_util_rotate(map, 180.0, 400, 300);
    evas_map_alpha_set(map, 0);
    evas_map_smooth_set(map, 1);
    evas_object_map_set(obj, map);
    evas_object_map_enable_set(obj, 1);
 
    evas_map_free(map);
}

See this screenshot below and  get the complete source code.

EFL + WebKit doing Politreco upside down

By Lucas De Marchi at October 23, 2010 09:53 PM

October 02, 2010

Deploying WebKit, common issues

Holger Freyther

From my exposure to people deploying QtWebKit or WebKit/GTK+ there are some things that re-appear and I would like to discuss these here.

  • Weird compile error in JavaScript?
  • It is failing in JavaScriptCore as it is the first that is built. It is most likely that the person that provided you with the toolchain has placed a config.h into it. There are some resolutions to it. One would be to remove the config.h from the toolchain (many things will break), or use -isystem instead of -I for system includes.
    The best way to find out if you suffer from this problem is to use -E instead of -c to only pre-process the code and see where the various includes are coming from. It is a strategy that is known to work very well.

  • No pages are loaded.
  • Most likely you do not have a DNS Server set, or no networking, or the system your board is connected to is not forwarding the data. Make sure you can ping a website that is supposed to work, e.g. ping www.yahoo.com, the next thing would be to use nc to execute a simple HTTP 1.1 get on the site and see if it is working. In most cases you simply lack networking connectivity.

  • HTTPS does not work
  • It might be either an issue with Qt or an issue with your system time. SSL Certificates at least have two dates (Expiration and Creation) and if your system time is after the Expiration or before the Creation you will have issues. The easiest thing is to add ntpd to your root filesystem to make sure to have the right time.

    The possible issue with Qt is a bit more complex. You can build Qt without OpenSSL support, you can make it link to OpenSSL or you can make it to dlopen OpenSSL at runtime. If SSL does not work it is most likely that you have either build it without SSL support, or with runtime support but have failed to install the OpenSSL library.

    Depending on your skills it might be best to go back to ./configure and make Qt link to OpenSSL to avoid the runtime issue. strings is a very good tool to find out if your libQtNetwork.so contains SSL support, together with using objdump -x and search for _NEEDED you will find out which config you have.

  • Local pages are not loaded
  • This is a pretty common issue for WebKit/GTK+. In WebKit/GTK+ we are using GIO for local files and to determine the filetype it is using the freedesktop.org shared-mime-info. Make sure you have that installed.

  • The page only displays blank
  • This is another issue that comes back from time to time. It only appears on WebKit/GTK+ with the DirectFB backend but sadly people never report back if and how they have solved it. You could make a difference and contribute back to the WebKit project.


    In general most of these issues can be avoided by using a pre-packaged Embedded Linux Distribution like Ångström (or even Debian). The biggest benefit of that approach is that someone else made sure that when you install WebKit, all dependencies will be installed as well and it will just work for your ARM/MIPS/PPC system. It will save you a lot of time.

    By zecke (noreply@blogger.com) at October 02, 2010 06:12 AM

    August 28, 2010

    WebKit

    Lucas De Marchi

    After some time working with the EFL port of WebKit, I’ve been nominated as an official webkit developer. Now I have super powers in the official repository :-), but I swear I intend to use it with caution and responsibility. I’ll not forget Uncle Ben’s advice: ”with great power comes great responsibility”.

    I’m preparing a post to talk about WebKit, EFL, eve (a new web browser based on WebKit + EFL) and how to easily embed a browser in your application. Stay tuned.

    By Lucas De Marchi at August 28, 2010 03:15 AM

    August 10, 2010

    Coscup2010/GNOME.Asia with strong web focus

    Holger Freyther

    On the following weekend the Coscup 2010/GNOME.Asia is taking place in Taipei. The organizers have decided to have a strong focus on the Web as can be seen in the program.

    On saturday there are is a keynote and various talks about HTML5, node.js. The Sunday will see three talks touching WebKit/GTK+. There is one about building a tablet OS with WebKit/GTK+, one by Xan Lopez on how to build hybrid applications (a topic I have devoted moiji-mobile.com to) and a talk by me using gdb to explain how WebKit/GTK+ is working and how the porting layer interacts with the rest of the code.

    I hope the audience will enjoy the presentations and I am looking forward to attend the conference, there is also a strong presence of the ex-Openmoko Taiwan Engineering team. See you on Saturday/Sunday and drop me an email if you want to talk about WebKit or GSM...

    By zecke (noreply@blogger.com) at August 10, 2010 04:32 PM

    July 16, 2010

    Cross-compiling QtWebKit for Windows on Linux using MinGW

    University of Szeged

    In this post I'll show you how to configure and compile a MinGW toolchain for cross-compilation on Linux, then how to build Qt using this toolchain and finally compile the Qt port of WebKit from trunk.

    read more

    By andras.becsi at July 16, 2010 09:31 AM

    September 06, 2008

    Skia graphics library in Chrome: First impressions

    Alp Toker

    With the release of the WebKit-based Chrome browser, Google also introduced a handful of new backends for the browser engine including a new HTTP stack and the Skia graphics library. Google’s Android WebKit code drops have previously featured Skia for rendering, though this is the first time the sources have been made freely available. The code is apparently derived from Google’s 2005 acquisition of North Carolina-based software firm Skia and is now provided under the Open Source Apache License 2.0.

    Weighing in at some 80,000 lines of code (to Cairo’s 90,000 as a ballpark reference) and written in C++, some of the differentiating features include:

    • Optimised software-based rasteriser (module sgl/)
    • Optional GL-based acceleration of certain graphics operations including shader support and textures (module gl/)
    • Animation capabilities (module animator/)
    • Some built-in SVG support (module (svg/)
    • Built-in image decoders: PNG, JPEG, GIF, BMP, WBMP, ICO (modules images/)
    • Text capabilities (no built-in support for complex scripts)
    • Some awareness of higher-level UI toolkit constructs (platform windows, platform events): Mac, Unix (sic. X11, incomplete), Windows, wxwidgets
    • Performace features
      • Copy-on-write for images and certain other data types
      • Extensive use of the stack, both internally and for API consumers to avoid needless allocations and memory fragmentation
      • Thread-safety to enable parallelisation

    The library is portable and has (optional) platform-specific backends:

    • Fonts: Android / Ascender, FreeType, Windows (GDI)
    • Threading: pthread, Windows
    • XML: expat, tinyxml
    • Android shared memory (ashmem) for inter-process image data references

    Skia Hello World

    In this simple example we draw a few rectangles to a memory-based image buffer. This also demonstrates how one might integrate with the platform graphics system to get something on screen, though in this case we’re using Cairo to save the resulting image to disk:

    #include "SkBitmap.h"
    #include "SkDevice.h"
    #include "SkPaint.h"
    #include "SkRect.h"
    #include <cairo.h>
     
    int main()
    {
      SkBitmap bitmap;
      bitmap.setConfig(SkBitmap::kARGB_8888_Config, 100, 100);
      bitmap.allocPixels();
      SkDevice device(bitmap);
      SkCanvas canvas(&device);
      SkPaint paint;
      SkRect r;
     
      paint.setARGB(255, 255, 255, 255);
      r.set(10, 10, 20, 20);
      canvas.drawRect(r, paint);
     
      paint.setARGB(255, 255, 0, 0);
      r.offset(5, 5);
      canvas.drawRect(r, paint);
     
      paint.setARGB(255, 0, 0, 255);
      r.offset(5, 5);
      canvas.drawRect(r, paint);
     
      {
        SkAutoLockPixels image_lock(bitmap);
        cairo_surface_t* surface = cairo_image_surface_create_for_data(
            (unsigned char*)bitmap.getPixels(), CAIRO_FORMAT_ARGB32,
            bitmap.width(), bitmap.height(), bitmap.rowBytes());
        cairo_surface_write_to_png(surface, "snapshot.png");
        cairo_surface_destroy(surface);
      }
     
      return 0;
    }

    You can build this example for yourself linking statically to the libskia.a object file generated during the Chrome build process on Linux.

    Not just for Google Chrome

    The Skia backend in WebKit, the first parts of which are already hitting SVN (r35852, r36074) isn’t limited to use in the Chrome/Windows configuration and some work has already been done to get it up and running on Linux/GTK+ as part of the ongoing porting effort.

    The post Skia graphics library in Chrome: First impressions appeared first on Alp Toker.

    By alp at September 06, 2008 12:11 AM

    June 12, 2008

    WebKit Meta: A new standard for in-game web content

    Alp Toker

    Over the last few months, our browser team at Nuanti Ltd. has been developing Meta, a brand new WebKit port suited to embedding in OpenGL and 3D applications. The work is being driven by Linden Lab, who are eagerly investigating WebKit for use in Second Life.

    While producing Meta we’ve paid great attention to resolving the technical and practical limitations encountered with other web content engines.


    uBrowser running with the WebKit Meta engine

    High performance, low resource usage

    Meta is built around WebKit, the same engine used in web browsers like Safari and Epiphany, and features some of the fastest content rendering around as well as nippy JavaScript execution with the state of the art SquirrelFish VM. The JavaScript SDK is available independently of the web renderer for sandboxed client-side game scripting and automation.

    It’s also highly scalable. Some applications may need only a single browser context but virtual worlds often need to support hundreds of web views or more, each with active content. To optimize for this use case, we’ve cut down resource usage to an absolute minimum and tuned performance across the board.

    Stable, easy to use cross-platform SDK

    Meta features a single, rock-solid API that works identically on all supported platforms including Windows, OS X and Linux. The SDK is tailored specifically to embedding and allows tight integration (shared main loop or operation in a separate rendering thread, for example) and hooks to permit seamless visual integration and extension. There is no global setup or initialization and the number of views can be adjusted dynamically to meet resource constraints.

    Minimal dependencies

    Meta doesn’t need to use a conventional UI toolkit and doesn’t need any access to the underlying windowing system or the user’s filesystem to do its job, so we’ve done away with these concepts almost entirely. It adds only a few megabytes to the overall redistributable application’s installed footprint and won’t interfere with any pre-installed web browsers on the user’s machine.

    Nuanti will be offering commercial and community support and is anticipating involvement from the gaming industry and homebrew programmers.

    In the mid term, we aim to submit components of Meta to the WebKit Open Source project, where our developers are already actively involved in maintaining various subsystems.

    Find out more

    Today we’re launching meta.nuanti.com and two mailing lists to get developers talking. We’re looking to make this site a focal point for embedders, choc-full of technical details, code samples and other resources.

    The post WebKit Meta: A new standard for in-game web content appeared first on Alp Toker.

    By alp at June 12, 2008 09:35 AM

    April 21, 2008

    Acid3 final touches

    Alp Toker

    Recently we’ve been working to finish off and land the last couple of fixes to get a perfect pixel-for-pixel match against the reference Acid3 rendering in WebKit/GTK+. I believe we’re the first project to achieve this on Linux — congratulations to everyone on the team!


    Epiphany using WebKit r32284

    We also recently announced our plans to align more closely with the GNOME desktop and mobile platform. To this end we’re making a few technology and organisational changes that I hope to discuss in an upcoming post.

    The post Acid3 final touches appeared first on Alp Toker.

    By alp at April 21, 2008 02:38 AM

    April 06, 2008

    WebKit Summer of Code Projects

    Alp Toker

    With the revised deadline for Google Summer of Code ’08 student applications looming, we’ve been getting a lot of interest in browser-related student projects. I’ve put together a list of some of my favourite ideas.

    If in doubt, now’s the time to submit proposals. Already-listed ideas are the most likely to get mentored but students are free to propose their own ideas as well. Proposals for incremental improvements will tend to be favoured over ideas for completely new applications, but a proof of concept and/or roadmap can help when submitting plans for larger projects.

    Update: There’s no need to keep asking about the status of an application on IRC/private mail etc. It’s a busy time for the upstream developers but they’ll get back in touch as soon as possible.

    The post WebKit Summer of Code Projects appeared first on Alp Toker.

    By alp at April 06, 2008 08:40 PM

    March 27, 2008

    WebKit gets 100% on Acid3

    Alp Toker

    Today we reached a milestone with WebKit/GTK+ as it became the first browser engine on Linux/X11 to get a full score on Acid3, shortly after the Acid3 pass by WebKit for Safari/Mac.

    Acid3
    Epiphany using WebKit r31371

    There is actually still a little work to be done before we can claim a flawless Acid3 pass. Two of the most visible remaining issues in the GTK+ port are :visited (causing the “LINKTEST FAILED” notice in the screenshot) and the lack of CSS text shadow support in the Cairo/text backend which is needed to match the reference rendering.

    It’s amazing to see how far we’ve come in the last few months, and great to see the WebKit GTK+ team now playing an active role in the direction of WebCore as WebKit continues to build momentum amongst developers.

    Update: We now also match the reference rendering.

    The post WebKit gets 100% on Acid3 appeared first on Alp Toker.

    By alp at March 27, 2008 09:06 PM

    March 15, 2008

    Bossa Conf ’08

    Alp Toker

    Am here in the LHR lounge. In a couple of hours, we take off for the INdT Bossa Conference, Pernambuco, Brazil via Lisbon. Bumped in to Pippin who will be presenting Clutter. Also looking forward to Lennart‘s PulseAudio talk amongst others.

    If you happen to be going, drop by on my WebKit Mobile presentation, 14:00 Room 01 this Monday. We have a small surprise waiting for Maemo developers.

    WebKit Mobile

    The post Bossa Conf ’08 appeared first on Alp Toker.

    By alp at March 15, 2008 03:29 AM