May 23, 2024

Patrick Griffis: Introducing the WebKit Container SDK

Igalia WebKit

Developing WebKitGTK and WPE has always had challenges such as the amount of dependencies or it’s fairly complex C++ codebase which not all compiler versions handle well. To help with this we’ve made a new SDK to make it easier.

Current Solutions

There have always been multiple ways to build WebKit and its dependencies on your host however this was never a great developer experience. Only very specific hosts could be “supported”, you often had to build a large number of dependencies, and the end result wasn’t very reproducable for others.

The current solution used by default is a Flatpak based one. This was a big improvement for ease of use and excellent for reproducablity but it introduced many challenges doing development work. As it has a strict sandbox and provides read-only runtimes it was difficult to use complex tooling/IDEs or develop third party libraries in it.

The new SDK tries to take a middle ground between those two alternatives, isolating itself from the host to be somewhat reproducable, yet being a mutable environment to be flexible enough for a wide range of tools and workflows.

The WebKit Container SDK

At the core it is an Ubuntu OCI image with all of the dependencies and tooling needed to work on WebKit. On top of this we added some scripts to run/manage these containers with podman and aid in developing inside of the container. It’s intention is to be as simple as possible and not change traditional development workflows.

You can find the SDK and follow the quickstart guide on our GitHub: https://github.com/Igalia/webkit-container-sdk

The main requirements is that this only works on Linux with podman 4.0+ installed. For example Ubuntu 23.10+.

In the most simple case, once you clone https://github.com/Igalia/webkit-container-sdk.git, using the SDK can be a few commands:

source /your/path/to/webkit-container-sdk/register-sdk-on-host.sh
wkdev-create --create-home
wkdev-enter

From there you can use WebKit’s build scripts (./Tools/Scripts/build-webkit --gtk) or CMake. As mentioned before it is an Ubuntu installation so you can easily install your favorite tools directly like VSCode. We even provide a wkdev-setup-vscode script to automate that.

Advanced Usage

Disposibility

A workflow that some developers may not be familiar with is making use of entirely disposable development environments. Since these are isolated containers you can easily make two. This allows you to do work in parallel that would interfere with eachother while not worrying about it as well as being able to get back to a known good state easily:

wkdev-create --name=playground1
wkdev-create --name=playground2

podman rm playground1 # You would stop first if running.
wkdev-enter --name=playground2

Working on Dependencies

An important part of WebKit development is working on the dependencies of WebKit rather than itself, either for debugging or for new features. This can be difficult or error-prone with previous solutions. In order to make this easier we use a project called JHBuild which isn’t new but works well with containers and is a simple solution to work on our core dependencies.

Here is an example workflow working on GLib:

wkdev-create --name=glib
wkdev-enter --name=glib

# This will clone glib main, build, and install it for us. 
jhbuild build glib

# At this point you could simply test if a bug was fixed in a different versin of glib.
# We can also modify and debug glib directly. All of the projects are cloned into ~/checkout.
cd ~/checkout/glib

# Modify the source however you wish then install your new version.
jhbuild make

Remember that containers are isoated from each other so you can even have two terminals open with different builds of glib. This can also be used to test projects like Epiphany against your build of WebKit if you install it into the JHBUILD_PREFIX.

To Be Continued

In the next blog post I’ll document how to use VSCode inside of the SDK for debugging and development.

May 23, 2024 04:00 AM

May 22, 2024

Release Notes for Safari Technology Preview 195

Surfin’ Safari

Safari Technology Preview Release 195 is now available for download for macOS Sonoma and macOS Ventura. If you already have Safari Technology Preview installed, you can update it in System Settings under General → Software Update.

This release includes WebKit changes between: 278097@main…278429@main.

Accessibility

Resolved Issues

  • Fixed updating table accessibility text when its caption dynamically changes. (278164@main) (127263464)

  • Fixed updating aria-describedby text after the targeted element changes its subtree. (278318@main) (127390465)

CSS

Resolved Issues

  • Fixed carrying analogous components forward when interpolating colors. (278379@main) (127170141)

Media

Resolved Issues

  • Fixed broken audio playback for a WebM file with a Vorbis track. (278665@main) (124880261)

Rendering

Resolved Issues

  • Fixed pseudo-elements not getting captured for View Transitions. (278123@main) (126964779)

  • Fixed View Transition rendering when the root element is not captured in the new snapshot. (278120@main) (127217265)

Scrolling

Resolved Issues

  • Fixed the cursor not updating as content scrolls under it on some pages. (278297@main) (122347347)

Web Extensions

Resolved Issues

  • Fixed Content Script CSS inconsistently overriding style attributes. (278387@main) (126916972)

Web Inspector

Resolved Issues

  • Fixed an issue where the Web Inspector viewport might appear cut off. (278208@main) (117272735)

  • Fixed overflow: scroll elements to scroll as expected when highlighting an element from the DOM tree. (278614@main) (124554999)

May 22, 2024 10:20 PM

May 21, 2024

WPE WebKit Blog: Update on the Layer Based SVG Engine (LBSE) in WebKit

Igalia WebKit

This blog entry gives an update on what we at Igalia have done on upstreaming and development of LBSE in WebKit in the last seven months. For an explanation of what LBSE is and how it is related to WPE, see this previous entry as a refresher.

Thanks to generous funding by Wix, which extensively uses SVG in their products and has a broad knowledge of the SVG peculiarities across browser engines, LBSE has made great progress in the past seven months. During this period, several advanced SVG painting features were implemented (e.g. clip-paths, masks, gradients, patterns), along with important performance improvements that expanded the new engine’s capabilities and stability. All this was possible thanks to Wix’s decision to address their problems by funding upstream work at the core of WebKit, instead of accepting the status-quo and implementing case-by-case fixes and workarounds on their end. As a result, WebKit-based browsers now benefit from the results of this fruitful collaboration, which we’ll try to explain in more detail in this blog post.

Project kick off and WebKit Contributors Meeting

In October 2023 we started the project mostly by thinking about the design and roadmap. We also did some general SVG bug fixing. For example, visual overflow computation for SVG renderers was corrected, which fixed quite a few SVG pixel tests. Various visual bugs were also fixed, such as unnecessary repainting when viewBox is used on <svg> elements, and incorrect clipping for outermost <svg> elements

Also in the same month, we attended the WebKit Contributors Meeting, where we presented a talk on the LBSE (slides are available here). The feedback on LBSE at the meeting was very positive. Giving the talk early on in the process actually helped us since we needed to have a good design in place.

Supporting SVG resources

The main focus in October 2023 was introducing the SVG resource concept, as already outlined in the WebKit Contributors Meeting talk. Thus, we started with adding a base SVG resource class: RenderSVGResourceContainer . This class was kept as simple as possible, with no support for resource invalidation or repainting logic. The main task of RenderSVGResourceContainer is to take care of registration so that the resource can be looked up by its clients.

For the first SVG resource to implement, we chose the SVG <clip-path> element, so we landed RenderSVGResourceClipper. To comply with the specification, the RenderSVGResourceClipper implementation produces 1-bit masks and uses a special rendering mode:

  • fill-opacity/stroke-opacity/opacity set to 1
  • masker/filter not applied when rendering the children
  • fill set to solid black and stroke set to none

The initial implementation did not use caching of ImageBuffers for clipping, but relied on Porter-Duff DestinationIn/SourceOver compositing operations to achieve the same effect, but faster. By integrating RenderSVGResourceClipper properly into RenderLayer, it aligned SVG clipping with HTML/CSS clipping.

Finally, the implementation prefers a pure clipping solution internally, as in legacy rendering, but for more complicated clip-paths (for example when the clip-path involves text content), a fallback to a mask is done.

Resource invalidation handling

After introducing the first SVG resource (RenderSVGResourceClipper), we noticed some issues with handling invalidations for it, such as adding to clip-path contents. In the legacy engine, invalidations have been handled through layouting. This caused various problems: for one, it could cause calling the setNeedsLayout method from within layout, which meant the invalidation chain depended on the DOM order.

In November 2023, an implementation landed that avoided using layout for resource invalidation. Instead, on dynamic updates, the style system is used to determine the appropriate action:

  • For non resources, changes that cause renderer geometry changes, like changing the x value of a <rect> element, still require a relayout. For visual changes not affecting geometry, like changing the fill color of a <rect>, a repaint action is enough.
  • For resources, the resource is invalidated/updated with the change and any of its clients are repainted using the new resource.

Support for masks

With improved support for SVG resource invalidation, in late November 2023 we were ready to upstream support for the next SVG resource, RenderSVGResourceMasker.

Like the support for clip-path, RenderSVGResourceMasker started out without caching image buffers and relied on creating temporary image buffers at rendering time. Mask content invalidations/changes were supported out of the box since we had improved resource invalidation handling (see above).

Support for gradients

In early January 2024, support for SVG gradients was upstreamed. Gradients are a kind of SVG resource that is a bit different to the previously implemented clipping paths and masks because it is a paint server, so a helper class for that called SVGPaintServerHandling and a base class RenderSVGResourcePaintServer were introduced. The main difference is in invalidation: paint servers simply need a repaint of all its clients on invalidation, whereas clipping paths/masks may need to do more work; i.e., masks underlying image buffers need to be updated before its clients can be repainted.

Support for patterns and markers

By the end of January 2024, support for SVG patterns was upstreamed. In the first implementation, no image buffer caching was implemented in order to keep things clean and simple. This implementation is different from the legacy implementation because the pattern contents are being rendered through pattern content layers (see RenderLayer::paintSVGResourceLayer). To make this work, RenderSVGResourcePattern has to set up the graphics context matrix correctly before calling paintSVGResourceLayer.

Around that same time, we implemented the next to last SVG resource on our TODO list, namely SVG markers.

SVG filters

In February 2024, Apple started work on supporting SVG filters in LBSE. A first iteration managed to fix a lot of the official SVG filter tests, but it turned out a filter regression had to be fixed first. Moreover, the initial work uncovered issues with the HTML/CSS filters implementation that need to be fixed in general. Finally, another reason why this support takes more time than some other features is that there is a strong requirement to make the support efficient in both memory usage and overall (re)painting speed. Still, the early results are very promising!

Cycle detection

It is quite easy in SVG to cause direct circular references:

<svg>
    <defs>
        <pattern id="p" xlink:href="#p" />
    </defs>
    ...
</svg>

It is also possible to cause indirect circular references:

<svg>
    <defs>
        <mask id="z" />
            <rect mask="url(#z)" />
        </mask>
    </defs>
    <ellipse mask="url(#z)" />
</svg>

The legacy engine solved this in an ad-hoc way in various places in the engine; it tried to break cycles before rendering, but still needed cycle protections in various places, since the solution was never unified or complete.

In February 2024 we provided a unified solution for LBSE by introducing SVGVisitedRendererTracking; see this commit for more. In the new approach, we don’t attempt to remove cycles, but detect them everywhere upon usage and stop processing in well-defined ways, all centralized in SVGVisitedRendererTracking.

Nested mask/pattern slowness

In April 2024, we addressed the slowness problems with nested masks/patterns. As an example, consider this for nested masks:

<svg>
    <defs>
        <mask id="z" />
            <rect id="1" mask="url(#y)" />
            <rect id="2" mask="url(#y)" />
            ...
        </mask>
        <mask id="y" />
            <rect id="1" mask="url(#x)" />
            <rect id="2" mask="url(#x)" />
            ...
        </mask>
        ...
    </defs>
    <ellipse mask="url(#z)" />
</svg>

For this example, the complexity can be increased at will by adding more masks and contents per mask.

The solution was twofold:

  • For masks, we realized bounding box calculations for a mask were not affected by masks used in the mask contents, so we could cut off bounding box calculations for nested masks.
  • For both masks and patterns, we added caching of image buffers per resource client so nested masks/patterns that are already encountered can reuse the image buffer cache.

See optimizations here for nested masks and patterns.

Next steps

For the short and mid-term, the plan is to make LBSE at least as good as legacy in regards to test coverage; i.e., all tests that pass in legacy should pass in LBSE. We have made a lot of progress over the last seven months just because of the amount of SVG resources that were implemented, but for example ,we will need to have SVG filters in place to pass this goal.

Another goal is to make sure LBSE passes all security requirements, as failing that would be a blocker to replacing the current engine. Fortunately, we are already taking this into account in several ways, such as adopting a lot of good smart pointer practices.

Finally, a big goal will be for LBSE to perform well on certain benchmarks like MotionMark, since WebKit has a golden rule to never ship a performance regression. So far there has not been an explicit focus on performance, and we know there are likely optimizations possible in RenderLayer usage, both in reducing the number of RenderLayer objects we create in certain situations as well as a possible reduction in complexity of RenderLayer for LBSE usage.

All in all, we are very pleased with the results and the progress we made in the last seven months. We at Igalia look forward to finishing the work to get the new engine in a shippable state in the near future!

May 21, 2024 12:00 AM

May 20, 2024

Frédéric Wang: Time travel debugging of WebKit with rr

Igalia WebKit

Introduction

rr is a debugging tool for Linux that was originally developed by Mozilla for Firefox. It has long been adopted by Igalia and other web platform developers for Chromium and WebKit too. Back in 2019, there were breakout sessions on this topic at the Web Engines Hackfest and BlinkOn.

For WebKitGTK, the Flatpak SDK provides a copy of rr, but recently I was unable to use the instructions on trac.webkit.org. Fortunately, my colleague Adrián Pérez suggested using a direct build without flatpak or the bubblewrap sandbox, and that indeed solved my problem. I thought it might be interesting to share this information with others, so I decided to write this blog post.

Disclaimer: The build instructions below may be imperfect, will likely become outdated, and are in any case not a replacement for the official ones for WebKitGTK development. Use them at your own risk!

CMake configuration

The approach that worked for me was thus to perform a direct build from my system. I came up with the following configuration step:

cmake -S. -BWebKitBuild/Release \
   -DCMAKE_BUILD_TYPE=Release \
   -DCMAKE_INSTALL_PREFIX=$HOME/WebKit/WebKitBuild/install/Release \
   -GNinja -DPORT=GTK -DENABLE_BUBBLEWRAP_SANDBOX=OFF \
   -DDEVELOPER_MODE=ON -DDEVELOPER_MODE_FATAL_WARNINGS=OFF \
   -DENABLE_TOOLS=ON -DENABLE_LAYOUT_TESTS=ON

where:

  • The -B option specifies the build directory, which is traditionnaly called WebKitBuild/ for the WebKit project.
  • CMAKE_BUILD_TYPE specifies the build type, e.g. optimized release builds (Release, corresponding to --release for the offical script) or debug builds with assertions (Debug, corresponding to --debug) 1.
  • CMAKE_INSTALL_PREFIX specifies the installation directory, which I place inside WebKitBuild/install/ 2.
  • The -G option specifies the build system generator. I used Ninja, which is the default for the offical script too.
  • -DPORT=GTK is for building WebKitGTK. I haven’t tested rr with other Linux ports.
  • -DENABLE_BUBBLEWRAP_SANDBOX=OFF was suggested by Adrián. The bubblewrap sandbox probably does not make sense without flatpak, so it should be safe to disable it anyway.
  • I extracted the other -D flags from the official script, trying to stay as close as possible to what it provides for WebKit development (being able to run layout tests, building the Tools/, ignoring fatal warnings, etc).

Needless to say, the advantage of using flatpak is that it automatically downloads and install all the required dependencies. But if you do your own build, you need to figure out what they are and perform the setup manually. Generally, this is straightforward using your distribution’s package manager, but there can be some tricky exceptions 3.

While we are still at the configuration step, I believe it’s worth sharing two more tricks for WebKit developers:

  • You can use -DENABLE_SANITIZERS=address to produce Asan builds or builds with other sanitizers.
  • You can use -DCMAKE_CXX_FLAGS="-DENABLE_TREE_DEBUGGING" in release builds if you want to get access to the tree debugging functions (ShowRenderTree and the like). This flag is turned on by default for debug builds.

Building and running WebKit

Once the configure step is successful, you can build and install WebKit using the following CMake command 2.

cmake --build WebKitBuild/Release --target install

When that operation completes, you should be able to run MiniBrowser with the following command:

LD_LIBRARY_PATH=WebKitBuild/install/Release/lib ./WebKitBuild/Release/bin/MiniBrowser

For WebKitTestRunner, some extra environment variables are necessary 2:

TEST_RUNNER_INJECTED_BUNDLE_FILENAME=$HOME/WebKit/WebKitBuild/Release/lib/libTestRunnerInjectedBundle.so LD_LIBRARY_PATH=WebKitBuild/install/Release/lib ./WebKitBuild/Release/bin/WebKitTestRunner filename.html

You can also use the official scripts, Tools/Script/run-minibrowser and Tools/Script/run-webkit-tests. They expect some particular paths, but a quick workaround is to use a symbolic link:

ln -s $HOME/WebKit/WebKitBuild $HOME/WebKit/WebKitBuild/GTK

Using rr for WebKit debugging

rr is generally easily installable from your distribution’s package manager. However, as stated on the project wiki page:

Support for the latest hardware and kernel features may require building rr from Github master.

Indeed, using the source has always worked best for me to avoid mysterious execution failures when starting the recording 4.

If you are not familiar with rr, I strongly invite you to take a look at the overview on the project home page or at some of the references I mentioned in the introduction. In any case, the first step is to record a trace by passing the program and arguments to rr. For example, to record a trace for MiniBrowser:

LD_LIBRARY_PATH=WebKitBuild/install/Debug/lib rr ./WebKitBuild/Debug/bin/MiniBrowser https://www.igalia.com/

After the program exits, you can replay the recorded trace as many times as you want. For hard-to-reproduce bugs (e.g. non-deterministic issues or involving a lot of manual steps), that means you only need to be able to record and reproduce the bug once and then can just focus on debugging. You can even turn off your machine after hours of exhausting debugging, then continue the effort later when you have more time and energy! The trace is played in a deterministic way, always using the same timing and pointer addresses. You can use most gdb commands (to run the program, interrupt it, and inspect data), but the real power comes from new commands to perform reverse execution!

Before coming to that, let’s explain how to handle programs with multiple processes, which is the case for WebKit and modern browsers in general. After you recorded a trace, you can display the pids of all recorded processes using the rr ps command. For example, we can see in the following output that the MiniBrowser process (pid 24103) actually forked three child processes, including the Network Process (pid 24113) and the Web Process (24116):

PID     PPID    EXIT    CMD
24103   --      0       ./WebKitBuild/Debug/bin/MiniBrowser https://www.igalia.com/
24113   24103   -9      ./WebKitBuild/Debug/bin/WebKitNetworkProcess 7 12
24115   24103   1       (forked without exec)
24116   24103   -9      ./WebKitBuild/Debug/bin/WebKitWebProcess 15 15

Here is a small debugging session similar to the single-process example from Chromium Chronicle #13 5. We use the option -p 24116 to attach the debugger to the Web Process and -e to start debugging from where it exited:

rr replay -p 24116 -e
(rr) break RenderFlexibleBox::layoutBlock
(rr) rc # Run back to the last layout call
Thread 2 hit Breakpoint 1, WebCore::RenderFlexibleBox::layoutBlock (this=0x7f66699cc400, relayoutChildren=false) at /home/fred/src-obj/WebKit/Source/WebCore/rendering/RenderFlexibleBox.cpp:420
(rr) # Inspect anything you want here. To find the previous Layout call on this object:
(rr) cond 1 this == 0x7f66699cc400
(rr) rc
Thread 2 hit Breakpoint 1, WebCore::RenderFlexibleBox::layoutBlock (this=0x7f66699cc400, relayoutChildren=false) at /home/fred/src-obj/WebKit/Source/WebCore/rendering/RenderFlexibleBox.cpp:420
420     {
(rr) delete 1
(rr) watch -l m_style.m_nonInheritedFlags.effectiveDisplay # Or find the last time the effective display was changed
Thread 4 hit Hardware watchpoint 2: -location m_style.m_nonInheritedFlags.effectiveDisplay

Old value = 16
New value = 0
0x00007f6685234f39 in WebCore::RenderStyle::RenderStyle (this=0x7f66699cc4a8) at /home/fred/src-obj/WebKit/Source/WebCore/rendering/style/RenderStyle.cpp:176
176     RenderStyle::RenderStyle(RenderStyle&&) = default;

rc is an abbreviation for reverse-continue and continues execution backward. Similarly, you can use reverse-next, reverse-step and reverse-finish commands, or their abbreviations. Notice that the watchpoint change is naturally reversed compared to normal execution: the old value (sixteen) is the one after intialization, while the new value (zero) is the one before initialization!

Restarting playback from a known point in time

rr also has a concept of “event” and associates a number to each event it records. They can be obtained by the when command, or printed to the standard output using the -M option. To elaborate a bit more, suppose you add the following printf in RenderFlexibleBox::layoutBlock:

@@ -423,6 +423,8 @@ void RenderFlexibleBox::layoutBlock(bool relayoutChildren, LayoutUnit)
     if (!relayoutChildren && simplifiedLayout())
         return;

+    printf("this=%p\n", this);
+

After building, recording and replaying again, the output should look like this:

$ rr -M replay -p 70285 -e # replay with the new PID of the web process.
...
[rr 70285 57408]this=0x7f742203fa00
[rr 70285 57423]this=0x7f742203fc80
[rr 70285 57425]this=0x7f7422040200
...

Each printed output is now annotated with two numbers in bracket: a PID and an event number. So in order to restart from when an interesting output happened (let’s say [rr 70285 57425]this=0x7f7422040200), you can now execute run 57425 from the debugging session, or equivalently:

rr replay -p 70285 -g 57425

Older traces and parallel debugging

Another interesting thing to know is that traces are stored in ~/.local/share/rr/ and you can always specify an older trace to the rr command e.g. rr ps ~/.local/share/rr/MiniBrowser-0. Be aware that the executable image must not change, but you can use rr pack to be able to run old traces after a rebuild, or even to copy traces to another machine.

To be honest, most the time I’m just using the latest trace. However, one thing I’ve sometimes found useful is what I would call the “parallel debugging” technique. Basically, I’m recording one trace for a testcase that exhibits the bug and another one for a very similar testcase (e.g. with one CSS property difference) that behaves correctly. Then I replay the two traces side by side, comparing them to understand where the issue comes from and what can be done to fix it.

The usage documentation also provides further tips, but this should be enough to get you started with time travel debugging in WebKit!

  1. RelWithDebInfo build type (which yields an optimized release build with debug symbols) might also be interesting to consider in some situations, e.g. debugging bugs that reproduce in release builds but not in debug builds. 

  2. Using an installation directory might not be necessary, but without that, I had trouble making the whole thing work properly (wrong libraries loaded or libraries not found).  2 3

  3. In my case, I chose the easiest path to disable some features, namely -DUSE_JPEGXL=OFF, -DUSE_LIBBACKTRACE=OFF, and -DUSE_GSTREAMER_TRANSCODER=OFF

  4. Incidentally, you are likely to get an error saying that perf_event_paranoid is required to be at most 1, which you can force using sudo sysctl kernel.perf_event_paranoid=1

  5. The equivalent example would probably have been to watch for the previous style change with watch -l m_style, but this was exceeding my hardware watchpoint limit, so I narrowed it down to a smaller observation scope. 

May 20, 2024 10:00 PM

May 13, 2024

WebKit Features in Safari 17.5

Surfin’ Safari

Happy May! It’s time for another release of Safari — our third significant update of 2024. With just a month until WWDC24 and the unveiling of what’s coming later this year, we are happy to get these 7 features and 22 bug fixes into the hands of your users today.

CSS

There are several exciting new CSS features in Safari 17.5, including text-wrap: balance, the light-dark() color function, and @starting-style, plus the ability to use feature queries with @import rules. Let’s look at how you can put each one to use.

Text wrap balance

On the web, with its flexible container widths, inconsistent lengths of content, and variation between browsers, it can feel impossible to avoid having text wrap in such a way that too few words end up all by themselves on a very short last line.

Very long text headline wrapping using the normal algorithm — which leaves a single word on the last line, all by itself

When type was set by hand, typographers would painstakingly avoid this undesirable result by manually moving content around. Over the decades, web developers have tried a series of different tricks to avoid orphans in CSS, in HTML, in JavaScript, and in content management systems. None work very well. The attempts usually feel hacky, laborious, and fragile.

To solve this and other frustrations, the CSS Working Group has defined three new options that you can use to change how text will wrap. You can switch from default wrapping to another style with text-wrap. WebKit for Safari 17.5 adds support for the first of these new options — balancing.

The text-wrap: balance rule asks the browser to “balance” the lines of text and make them all about the same length.

A very long headline wrapped using text-wrap: balance, so each of the three lines are the same length as each other — and none of them fill all the horizontal space available

You can see how now the text no longer fills the containing block — there’s a large amount of space on the right of the words. This is expected, and something you’ll want to think about as you decide when to use text-wrap: balance.

Where exactly each line of text will break when using text-wrap: balance may be slightly different in each browser. The CSS Text level 4 web standard leaves it up to each browser engine team to decide which algorithm they want to use in determining how exactly to wrap balanced text.

It can be computationally expensive for the browser to count characters and balance multiple lines of text, so the standard allows browsers to limit the number of lines that are balanced. Chromium browsers balance 6 or fewer lines, Firefox balances 10 or fewer, while Safari/WebKit balances an unlimited numbers of lines.

For now, Safari does not balance text if it’s surrounding a float or initial letter. And Safari disables the balancer if the content contains preserved tabs or soft hyphens.

Text wrap shorthands and longhands

The text-wrap property is actually a shorthand for two longhand properties: text-wrap-style and text-wrap-mode.

The text-wrap-mode property provides a mechanism for expressing whether or not text should wrap.

text-wrap-mode: wrap; /* initial value */
text-wrap-mode: nowrap;

The wrap value turns it on, and the nowrap value turns it off, just like the values for white-space. (In fact, text-wrap-mode is the newly introduced longhand of white-space.) WebKit added support for text-wrap-mode: wrap and nowrap in Safari 17.4.

The text-wrap-style property selects how to wrap. The initial value is auto — asking text to wrap in the way it has for decades. Or, you can choose a value to switch to another “style” of wrapping.

WebKit for Safari 17.5 adds support for text-wrap-style: balance, stable, and auto.

text-wrap-style: auto; /* initial value */
text-wrap-style: balance;
text-wrap-style: stable;

Of course, the text-wrap shorthand is a way to combine text-wrap-mode and text-wrap-style and declare them together. If you write text-wrap: balance it’s the same as text-wrap: wrap balance, meaning: “yes, please wrap, and when you do, please balance the text”.

Full support will eventually include three properties and six values. No browser supports everything yet, so be sure to look up support for the text-wrap, text-wrap-mode, and text-wrap-style properties, as well as the balance, pretty, stable, auto, wrap, and nowrap values.

The balance, pretty, and stable values will simply fall back to auto in browsers without support, so progressive enhancement is easy. You can use these values today, no matter how many of your users don’t yet have a browser with support. They will simply get auto-wrapped text, just like they would if you didn’t use text-wrap. Meanwhile, those users with support will get an extra boost of polish.

Dark mode and the light-dark() color function

More and more, users expect websites and web apps to support dark mode. Since Safari 12.1, the prefers-color-scheme media query has given you the ability to write code like this:

body {
  background: white;
  color: black;
}
@media (prefers-color-scheme: dark) {
  body {
    background: darkslategray;
    color: white;
  }
}

Or perhaps you’ve used variables to define colors for both light and dark mode at once, making it easier to use them everywhere.

:root {
  --background: white;
  --text: black;
}
@media (prefers-color-scheme: dark) {
  :root {
    --background: darkslategray;
    --text: white;
  }
}
body {
  background: var(--background);
  color: var(--text);
}

Well, now there’s a new option — the light-dark() function. It makes defining colors for dark mode even easier.

First, inform the browser you are providing a design for both light and dark modes with the color-scheme property. This prompts the browser to switch the default user agent styles when in dark mode, ensuring the form controls appear in dark mode, for example. It’s also required for light-dark() to work correctly.

:root {
  color-scheme: light dark;
}

Then, any time you define a color, you can use the light-dark() function to define the first color for light mode, and the second color for dark mode.

color: light-dark(black, white);
background-color: light-dark(white, darkslategray);

You can still use variables, if you’d like. Perhaps you want to structure your code like this.

:root {
  color-scheme: light dark;
  --background: light-dark(black, white);
  --text: light-dark(white, darkslategray);
}
body {
  background: var(--background);
  color: var(--text);
}

An often-asked question when learning about light-dark() is “does this only work for colors?” Yes, this function only works for colors. Use the prefers-color-scheme media query to define the rest of your color-scheme dependent styles.

Starting style

WebKit for Safari 17.5 adds support for @starting-style. It lets you define starting values for a particular element. This is needed to enable a transition when the element’s box is created (or re-created).

.alert {
  transition: background-color 2s;
  background-color: green;
  @starting-style {
    background-color: transparent;
  }
}

In the above example, the background-color will transition from transparent to green when the element is added to the document.

Many developers are excited to use @starting-style along with display: none interpolation. To do so, WebKit also needs to support animation of the display property, which has not yet shipped in Safari. You can test this use case today in Safari Technology Preview.

Features queries for importing CSS

WebKit for Safari 17.5 adds the supports() syntax to @import rules. Now you can conditionally import CSS files based on whether or not there’s support for a certain feature.

@import <url> supports(<feature>);

For example, you could load different stylesheets based on whether or not CSS Nesting is supported.

@import "nested-styles.css" supports(selector(&));
@import "unnested-styles.css" supports(not selector(&));

Or you could load certain CSS files when a browser does not have support for Cascade Layers. (Note that any @import rules with layer() will automatically be ignored in a browser without layer support.)

@import url("reset.css") layer(reset);
@import url("framework.css") layer(framework);
@import url("custom.css") layer(custom);

@import url("unlayered-fallback-styles.css") supports(not at-rule(@layer));

Or simply test for a feature. Here, these layout styles will only be loaded if Subgrid is supported.

@import url("layout.css") supports(grid-template-columns: subgrid);

WebCodecs

WebKit for Safari 17.5 adds support for AV1 to WebCodecs when an AV1 hardware decoder is available.

WebGL

WebKit for Safari 17.5 adds WebGL support for EXT_conservative_depth and NV_shader_noperspective_interpolation.

WKWebView

WKWebView adds support for logging MarketplaceKit errors to the JavaScript console. This will make errors easier to debug.

Bug Fixes and more

In addition to these new features, WebKit for Safari 17.5 includes work polishing existing features.

Accessibility

  • Fixed a bug preventing VoiceOver word echoing in some text fields. (122451549) (FB13592798)

Animations

  • Fixed flickering with multiple accelerated animations and direction changes. (117815004)

Authentication

  • Fixed excludeCredentials property being ignored during a passkey registration request. (124405037)

CSS

  • Fixed the proximity calculation for implicit @scope. (124640124)
  • Fixed the Grid track sizing algorithm logical height computation avoid unnecessary grid item updates. (124713418)
  • Fixed any @scope limit making the element out of scope. (124956673)

Forms

  • Fixed native text fields becoming invisible in dark mode. (123658326)
  • Fixed fallback native <select> rendering in dark mode. (123845293)

Media

  • Fixed scrolling for an element when a video element with pointer-events: none is placed over it. (118936715)
  • Fixed HTML5 <audio> playback to continue to the next media activity when in the background. (121268089) (FB13551577)
  • Fixed AV1 to decode in hardware on iPhone 15 Pro. (121924090)
  • Fixed audio distortion over internal speakers when streaming content in web browsers. (122590884)
  • Fixed firing loadeddata events for <audio> and <video> on page load. (124079735) (FB13675360)

Rendering

  • Fixed adjusting the size of the scrollable area when changing betwen non-overlay and overlay scrollbars. (117507268)
  • Fixed flickering when showing a layer on a painted background for the first time by avoiding async image decoding. (117533495)
  • Fixed line breaking before or between ruby sequences. (122663646)

Web API

  • Fixed mousemove events in an iframe when the mouse is clicked from outside the iframe and then moves into it while the button is held down. (120540148) (FB13517196)

Web Apps

  • Fixed several issues that caused Web Push to not show notifications when the web app or Safari was not already running. (124075358)

Web Inspector

  • Fixed info and debug buttons not appearing in the Console Tab until new console messages are displayed. (122923625)

WebRTC

  • Fixed WebCodecs to correctly use the VP9 hardware decoder. (123475343)
  • Fixed no incoming video in Teams VA. (124406255)
  • Fixed the camera pausing occasionally when torch is enabled. (124434403)

Updating to Safari 17.5

Safari 17.5 is available on iOS 17.5, iPadOS 17.5, macOS Sonoma 14.5, macOS Ventura, macOS Monterey and in visionOS 1.2.

If you are running macOS Ventura or macOS Monterey, you can update Safari by itself, without updating macOS. On macOS Ventura, go to  > System Settings > General > Software Update and click “More info…” under Updates Available.

To get the latest version of Safari on iPhone, iPad, or Apple Vision Pro, go to Settings > General > Software Update, and tap to update.

Feedback

We love hearing from you. To share your thoughts on Safari 17.5, find us on Mastodon at @jensimmons@front-end.social and @jondavis@mastodon.social. Or send a reply on X to @webkit. You can also follow WebKit on LinkedIn. If you run into any issues, we welcome your feedback on Safari UI, or your WebKit bug report about web technologies or Web Inspector. Filing issues really does make a difference.

Download the latest Safari Technology Preview on macOS to stay at the forefront of the web platform and to use the latest Web Inspector features.

You can also find this information in the Safari 17.5 release notes.

May 13, 2024 06:30 PM

May 09, 2024

Release Notes for Safari Technology Preview 194

Surfin’ Safari

Safari Technology Preview Release 194 is now available for download for macOS Sonoma and macOS Ventura. If you already have Safari Technology Preview installed, you can update it in System Settings under General → Software Update.

This release includes WebKit changes between: 277150@main…278096@main.

Accessibility

Resolved Issues

  • Fixed range input not firing an input event when incremented or decremented via accessibility APIs. (277182@main) (85707481)
  • Fixed setting aria-hidden on a slot not hiding the slot’s assigned nodes. (277359@main) (108762653)
  • Fixed aria-hidden=true to be ignored on the <body> and <html> elements. (277677@main) (123049663)

Canvas

New Feature

  • Added support for CanvasRenderingContext2D.filter. (278000@main) (51303686)
  • Added support for willReadFrequently. (277792@main) (126739379)

Resolved Issues

  • Fixed OffscreenCanvas failing to render to the placeholder with nested workers. (277328@main) (126069375)
  • Fixed losing the contents layer of the placeholder canvas of OffscreenCanvas when switching off the tab. (277654@main) (126070648)

Deprecations

  • Removed support for OffscreenCanvasRenderingContext2D’s commit() method. (278047@main) (126758254)

CSS

Resolved Issues

  • Fixed an incorrect difference between implicit and explicit initial values for custom properties. (277670@main) (124573975)
  • Fixed excluding -apple-pay-button from applying to any element that supports appearance: auto and is not a button. (277268@main) (126107516)
  • Fixed missing color interpretation methods added to CSS color specifications. (277481@main) (126444371)

JavaScript

New Features

  • Added support for the v flag with RegExp.prototype[Symbol.matchAll]. (277160@main) (126017731)
  • Added support for Unicode 15.1.0 characters in RegExp. (277786@main) (126863692)

Resolved Issues

  • Fixed RegExp.prototype.@@split to update the following legacy RegExp static properties: RegExp.input, RegExp.lastMatch, RegExp.lastParen, RegExp.leftContext, RegExp.rightContext, and RegExp.$1, ... RegExp.$9. (277559@main) (99865597)
  • Fixed logical assignment expressions to throw a syntax error when the left side of the assignment is a function call. (277536@main) (126540636)
  • Fixed throwing a syntax error for nested duplicate-named capturing groups in RegEx. (277928@main) (126863735)
  • Fixed Intl implementation to ensure canonicalizing “GMT” to “UTC” based on a spec update. (277997@main) (127061600)

Networking

Resolved Issues

  • Fixed incorrect Sec-Fetch-Site value for navigation of a nested document. (277662@main) (109358563)

Rendering

Resolved Issues

  • Fixed incorrect View Transition snapshot position when page is scrolled. (277371@main) (123668892)
  • Fixed Masonry to correctly perform intrinsic sizing. (277403@main) (124240255)
  • Fixed visual overflow causing shifts in View Transitions. (277183@main) (125882228)
  • Fixed snapshot scaling when animating View Transitions. (277636@main) (125931538)
  • Fixed capturing the root background showing a white area at bottom in View Transitions. (272224@main) (125931851)
  • Fixed incorrect grid item positioning with out-of-flow sibling. (277300@main) (126207467)
  • Fixed visual overflow rect to be recomputed at every frame of View Transitions. (277482@main) (126240398)
  • Fixed break-word with a float discarding text. (277461@main) (126309547)
  • Fixed rendering 3D transforms in View Transitions. (277639@main) (126513916)
  • Fixed image capture rectangle not including positioned descendants. (277714@main) (126660740)

SVG

New Features

  • Added animation support for the d property in SVG. (277299@main) (126206354)

Resolved Issues

  • Fixed clipping an SVG when an SVGFilter is applied to its root. (265135@main) (118938065)

Web Animations

Resolved Issues

  • Fixed updating an animation when changing the value of a transform property while that property is animated with an implicit keyframe. (277257@main) (126126617)
  • Fixed animating with color-mix. (277256@main) (126148201)

Web API

New Features

  • Added support for serializing shadow roots through getHTML() as well as the corresponding shadow root serializable feature. (277374@main) (125513986)

Resolved Issues

  • Fixed throwing exceptions in navigation methods if in a detached state. (277487@main) (123898636)
  • Fixed a minor issue in URL’s host setter. (277531@main) (124363495)
  • Fixed GeolocationCoordinates to expose a toJSON() method. (277347@main) (126183686)
  • Fixed GeolocationPosition to expose a toJSON() method. (277413@main) (126247408)
  • Fixed setting CustomEvent.target when dispatching an event. (277435@main) (126311287)

Web Extensions

Resolved Issues

  • Fixed permissionState to be requested if an extension has the webNavigation permission. (277308@main) (126212951)

WebGL

New Features

  • Enabled support for the following approved extensions: EXT_render_snorm, OES_sample_variables, and OES_shader_multisample_interpolation. (277775@main) (126863775)

May 09, 2024 08:38 PM

April 30, 2024

JetStream 2.2

Surfin’ Safari

JetStream 2 is a suite of JavaScript and WebAssembly performance benchmarks that runs 64 subtests. We recently fixed several minor issues we discovered that impact running the benchmark.

One issue happened when running JetStream on fast hardware where the resolution of time to run a subtest, as measured by the timing code, occasionally returned 0. This caused problems when calculating subtest scores. Two other issues were found in the sub-test segmentation, where the code didn’t create the initial Float32Array for the test and another race condition issue was fixed in the task queue code for that test.

Two other issues running JetStream via the command-line were found and fixed. The first was specific to the Spider Monkey command-line shell and the second was to fix the Wasm command-line script when running with both the D8 and Spider Monkey shells.

We are introducing JetStream 2.2, which fixes the issues mentioned above and includes the same subtests as JetStream 2.1 as well as the original JetStream 2. The fixes improve the stability of the benchmark. We measured JetStream 2.2’s benchmark scores from Chrome, Safari, and Firefox and found them to be as stable as JetStream 2.1. Measurements were performed using a 16“ MacBook Pro with an M3 processor. The system was not running other applications and the browser app being tested, configured without extensions enabled, was restarted before each JetStream run. A total of 10 runs were performed for each browser app JetStream version combination. Scores produced from JetStream 2.2 are not comparable to other versions of any JetStream benchmark.

April 30, 2024 08:42 PM

April 24, 2024

Release Notes for Safari Technology Preview 193

Surfin’ Safari

Safari Technology Preview Release 193 is now available for download for macOS Sonoma and macOS Ventura. If you already have Safari Technology Preview installed, you can update it in System Settings under General → Software Update.

This release includes WebKit changes between: 276610@main…277149@main.

Accessibility

Resolved Issues

  • Fixed hidden elements targeted by aria-labelledby to expose their entire subtree text, not just their direct child text. (276864@main) (125634439)
  • Fixed accessible name computation for elements with visibility: visible inside a container with visibility: hidden. (277004@main) (125738704)

CSS

Resolved Issues

  • Fixed the Grid track sizing algorithm logical height computation avoid unnecessary grid item updates. (276633@main) (124713418)
  • Fixed the style adjuster for @starting-style incorrectly invoking with a null element. (276993@main) (125837628)

Forms

Resolved Issues

  • Fixed the value attribute not getting displayed in an input element with type="email" and the multiple attribute. (276895@main) (125221858)

JavaScript

Resolved Issues

  • Fixed inconsistent output of Function.prototype.toString for accessor properties. (276904@main) (125739577)

Rendering

Resolved Issues

  • Fixed intrinsic inline size calculators to account for whitespace before an empty child with nonzero margins. (276875@main) (122586712)
  • Fixed overlapping elements with flex box when height: 100% is applied on nested content. (276880@main) (125572851)
  • Fixed block containers that are scroll containers to default to unsafe alignment. (276929@main) (125742095)

Web API

New Features

  • Added support for PopStateEvent’s hasUAVisualTransition. (277001@main) (125849073)

Resolved Issues

  • Fixed cloning of ShadowRoot nodes following a DOM Standard clarification. (277066@main) (125917138)

Web Inspector

Resolved Issues

  • Fixed Console and code editor completion not auto-scrolling the suggestion into view. (277034@main) (124979790)
  • Fixed search in the DOM tree view unexpectedly chaning the text display. (277073@main) (125797803)

April 24, 2024 11:23 PM

April 19, 2024

Help us invent CSS Grid Level 3, aka “Masonry” layout

Surfin’ Safari

If you’ve been making websites for years, you know how frustrating it was to lay out a web page with CSS floats. Managing sizes and placement was tedious and time consuming. Being creative was often impossible. CSS Grid greatly eased that pain with Grid Level 1 in 2017, and now with Grid Level 2, aka Subgrid. But even with the powerful CSS of today, not every layout imaged by designers is possible. In fact, when CSS Grid shipped, one of the most commonly asked questions was: “how do I write CSS to accomplish a masonry layout?” Sadly, for the last seven years the answer has been — you can’t.

What is masonry layout?

What do we mean by the term “masonry layout”? Basically it’s the pattern seen in the following image — where content packs together like a brick or stone wall. That’s where it gets the name “masonry”. It’s also frequently called “waterfall layout”, as a metaphor for how content flows down the page like a waterfall.

Two dozen photos of different aspect ratios laid out using a

This layout is popular because it solves a few problems that other layouts do not.

  1. It allows for content of different aspect ratios, and avoids the need to crop or truncate content in order to turn everything into uniform rectangles.
  2. It distributes content across the page (instead of flowing down each column, one by one). This follows the natural reading order as you scroll the page. And it lets the website lazy-load additional content at the bottom without moving existing content around.

This layout creates uniformly-sized columns, without any rows. It’s quite possible that because this layout has required JavaScript, anything more creative or complex has been too hard to pull off — and we’ve been left with an expectation that masonry layout should only ever be a simple pattern with uniformly-sized columns. Let’s see what’s possible if we build it into CSS instead.

Inventing masonry

A mechanism in CSS for “masonry layout” was first proposed by Mozilla in January 2020 as an extension of CSS Grid, and implemented as an experiment behind a flag in Firefox Nightly. In 2022, Apple started implementing this CSS Grid Level 3 proposal in Safari Technology Preview (where it’s currently on by default), and we’ve been helping to move the web standard along to reach maturity.

However, there are big questions still being asked about how CSS should handle masonry-style layouts. Some people remain skeptical that this capability should be part of CSS Grid, and want it to instead be its own separate display type. Others are questioning whether or not this kind of layout is needed on the web at all — they aren’t sure that well-known websites will use it. With such fundamental disagreements at play, no browser can ship. We must first come to consensus in the CSS Working Group.

This is where we need your help. We’d like real-world web designers and developers to weigh into the discussion, and express what it is that you want. Your input really can make a difference.

In this article, we’ll walk through how the CSS Grid Level 3 proposal works, and how you can use its new capabilities. We’ll show you why we believe these features should be part of CSS Grid, and explain what the alternative would be if the CSS Working Group creates display: masonry instead. And then, we’ll ask you to join the debate to help move us forward. Please do read to the end.

Four demos

To show why we at Apple believe this capability should be part of CSS Grid, we created four demonstrations. If you’d like, try them yourself at webkit.org/demos/grid3. View these demos in a browser that supports Grid Level 3 — currently Safari Technology Preview or Firefox after you’ve turned on the feature flag.

Note there’s a control panel for each demo, with the relevant layout code printed to the page. Turn on “Number items” to see the relationship between the HTML order of content and the layout placement of that content.

Screenshot showing the controls of the demo — here with the numbers turned on

Each demo has a multitude of variations. Switch between variations from the dropdown menu, which changes only the CSS. The HTML stays the same.

Screenshot showing the controls of the demo — here with the dropdown of various alternative layouts showing

Creating a classic masonry / waterfall layout

First, let’s take a look at how to build a classic masonry/waterfall layout. In this gallery of photos, each image is wrapped with a figure element, and the figures are direct children of a main element.

<main>
  <figure><img src="photo-1.jpg"></figure>
  <figure><img src="photo-2.jpg"></figure>
  <figure><img src="photo-3.jpg"></figure>
</main>

We start by applying display: grid to the main element to create the Grid container. Then we can define grid-template-columns however we’d like.

In this case, let’s use grid-template-columns: repeat(auto-fill, minmax(14rem, 1fr)) to ask the browser to repeat a looped definition to create a set of flexible columns that are each of a minimum of 14rem. This results in uniformly-sized columns, typical of the classic masonry/waterfall layout. The gap: 1rem; rule creates a 1rem-wide space between the items — both between the columns, and horizontally between items.

And then, we’ll define the “rows” with the masonry value. (It’s likely the name of this value will change before this ships in browsers — more on that at the end of this article. For now, masonry is what works.)

main { 
  display: grid;
  grid-template-columns: repeat(auto-fill, minmax(14rem, 1fr));
  gap: 1rem;
  grid-template-rows: masonry;
}

The grid-template-rows: masonry rule tells the browser: “Please do not create rows. Instead pack the content into a masonry/waterfall-like pattern.”

That’s it! In four lines of CSS, with zero media queries or container queries, we’ve created a flexible layout that works on screens of all sizes. And there’s no need to crop content to force everything into same-sized boxes.

masonry layout of photos

In graphic design, a layout that has uniformly-sized columns and no rows is often called a “symmetrical columnar grid”. For centuries, columnar grids were the dominant type of grid used in page design.

Leveraging Grid’s full power to define columns

Now let’s dive into the advantages of combining the full power of CSS Grid with masonry/waterfall packing. CSS Grid provides many options for defining grid our columns. Using fr units to create a symmetrical grid is only one option of many.

How could these possibilities be used for a masonry/waterfall-style layout? Let’s try mixing fixed-sized columns with flexible columns. We could make the first and last column fixed-sized, while the middle columns are flexible, changing in both size and number.

Specifically, the first and last columns are exactly 14 characters wide, while the middle columns are flexible (at least 28 characters wide) and change in number to fill the available space.

main { 
  display: grid;
  grid-template-columns: 14ch repeat(auto-fill, minmax(28ch, 1fr)) 14ch;
  grid-template-rows: masonry;
  gap: 1rem;
}

This is just one of many, many possibilities.

CSS Grid allows for a lot of creativity with its options for defining grid tracks:

  • fixed sizes defined in any unit (px, em, rem, cqi, lh, ch, ic, cap, vw, svh, and many more)
  • max-content and min-content
  • the full power of fr units
  • minmax() functions
  • %-sized
  • auto

These options in CSS Grid allow you to create something much more dynamic and flexible in interesting ways. You can create two stages of flexibility, because the fr-unit sized columns grow and shrink in a separate stage from the minmax()-sized columns. The max-content and min-content values let you size the columns based on the content size, rather than sizing the content based on the column size. The fr units can easily be used to create compound or asymmetrical grids, where the columns are different sizes. The options are endless.

By adding the ability to pack content in a masonry/waterfall pattern to CSS Grid, we maintain the full power of Grid for defining our columns in whichever manner we like.

For example, let’s use grid-template-columns: repeat(auto-fill, minmax(8rem, 1fr) minmax(16rem, 2fr)) minmax(8rem, 1fr) to create a pattern of alternating narrow and wider columns, where all the columns are flexible. More columns are added when there’s space. And there’s always an odd number of columns, ending with a narrow one.

three web browser windows next to each other, showing how the layout adjusts for narrow, medium and wider windows

Even when we define columns using only the simple fr unit, the full power provided in CSS Grid means different columns can be set to different sizes. For fun, let’s use fr units to define a set of columns sized to inject the vibes of the golden ratio by using the Fibonacci sequence in grid-template-columns: 1fr 1fr 2fr 3fr 5fr 8fr;

layout of photos where the columns on the left are very narrow, getting bigger and bigger as them move to the right, in a fibonacci sequence

In a more practical example, let’s use max-content when defining our columns. Content-based sizing is an incredibly power feature of CSS Grid. This demo of a mega menu layout uses grid-template-columns: repeat(auto-fill, minmax(max-content, 30ch)); to ensure that every column is big enough to fit every link without wrapping text.

a menu with a ton of links, like in a website footer, laid out with Grid Level 3

Mega menus have been hard to code, especially across multiple screen sizes. With CSS Grid Level 3, it becomes incredibly easy. A few lines of code creates a dynamic layout which adds and removes columns one at a time as space allows — without any media/container queries, and with wrapping prevention.

Many of these examples could never be created with masonry as a separate display type. The discussion of display: masonry is to only allow symmetrical columns (columns that are the same size as each other), much like multicolumn layout does today.

Leveraging Grid’s ability to let content span columns

CSS Grid also lets us span items across multiple columns. Let’s use that capability to see what interesting options might emerge. How about making every 5th image span two grid columns, while the rest of the images are span one column.

same photo layout, now with random photos being bigger

What if instead, we put a wider class on specifically on images that have a wider aspect-ratio, to make those images span multiple columns. We can also change the styling a bit, making the corners square instead of round, and reducing the grid gap to zero. This gives us a different way to pack photos of different aspect ratios together.

another layout of image, this time where wider images are wider

We also experimented with combining the classic masonry/waterfall layout of photos with View Transitions. When a user clicks/taps on any photo, it grows to span multiple columns. The browser automagically animates the transition. (This demo requires Safari Technology Preview 192 or later.)

These variations of the Photos and Mega Menu demos are just a small taste of all of the many possibilities you get when leveraging the full power of CSS Grid in the column direction, while simultaneously turning off rows.

Columnar vs. Modular Grids

What happens when we keep experimenting? Let’s let go of thinking about “masonry”, and start imagining Grid Level 3 purely as an expansion of Grid. At its core, CSS Grid Level 3 provides a mechanism for turning off rows. It lets us create a columnar grid — a grid that’s made up of columns alone.

By contrast, a modular grid is a grid where everything is lined up in both columns and rows. They became popular in the 20th century during the dominance of modernism in graphic design. CSS Grid Level 1 is really good at making modular grids… that’s what it wants to do. In fact, float-based layouts also encouraged the creation of modular grids on the web, since you had to make all your content the same height to get your floats to clear. Images need to be the same aspect ratio. Text has to be the same length. This is often accomplished on the back-end with policies enforced by the content management system, or on the front-end by CSS that truncates/crops the content.

It’s incredible common for websites to do some variation of this sort of modular grid, laid out here with CSS Grid Level 1.

layout of text, in ridged sets of boxes, row after row, all the same height

Of course, this example is overly simplistic. The article ledes look bare with no teaser images. The uniformity is so strict and formal, the design lacks life. Real websites find other ways to breathe life back into the design.

But what if the layout itself could also provide some vitality and interest? What will happen if we use CSS Grid to create a columnar grid as easily as it creates a modular grid? What if we don’t truncate content, and instead let it be the size that it wants to be — and get the layout to work for the content, rather than forcing the content to work for the layout?

A classic masonry/waterfall layout with various lengths of text looks like this, which is already more engaging since a user can read more about each article:

same article teaser text, this time laid out in columns set with masonry

Although, that’s still a bit visually repetitive. Symmetrical columnar grids often are. We need the rest of the power of CSS Grid to do something more interesting. Let’s make the latest article much bigger, and have it span four columns. A handful of other recent articles can be medium-sized and span two columns. While the older content is smaller, spanning one column.

same text, this time laid out with much more dynamic use of 'masonry-style layout plus spanning columns

Now this otherwise visually boring text is starting to look fairly lively. If we were to add images to each of these articles, it would be it even more dynamic.

Let’s experiment with mixing images and text together on a webpage for a museum. The first grid item is a header that introduces the museum, and provides navigation to other resources. The rest of the content consists of pieces of artwork and their information: title, artist, year, medium, catalog number and location.

Because the paintings are gorgeous, the content looks pretty great in a classic masonry/waterfall layout.

layout of cards, each with a painting and text

Let’s see what else we can do by utilizing two more powerful features of CSS Grid — subgrid and explicit placement.

Using subgrid and explicit placement

The functionality provided by subgrid in CSS Grid Level 2 is incredible, and it’s finally supported in most browsers.

Instead of listing the painting’s metadata in a single left-aligned column, let’s see how we might better use the available space. By using subgrid, we can put the year and catalog number on the right of each card — and line up this data for one painting with the same data for the other paintings.

close-up of this layout, with the Grid Inspector showing, with lines marking columns showing how subgrid works

By adding this new functionality to CSS Grid Level 3, we get the benefit of existing developer tools. You can use the Grid Inspector in Safari Technology Preview today as you try out grid-template-rows: masonry.

If masonry is its own display type, and not part of CSS Grid, it will not get the benefit of subgrid.

We can also use the power of CSS Grid Level 1 to explicitly place the header into the last two columns, moving it to the top right corner of the page with grid-column: -3 / -1.

In just a few lines of layout code, we are using the full power of CSS Grid Levels 1, 2, and 3 to create flexible columns that change in number to accommodate the available size — without using any media queries or container queries.

Hopefully you can see the advantages of fully combining a mechanism for masonry/waterfall layouts with CSS Grid — providing many more creative possibilities than masonry alone.

The Debate

So let’s get into the debate that’s been blocking the CSS Working Group from moving forward. Our hope is that web designers and developers chime in (post to social media, write blog posts) with your thoughts about which direction CSS should take.

Some people, including those of us at Apple, like having “Masonry” be part of CSS Grid. We believe this functionality is a mechanism to expand CSS Grid — allowing it to finally create columnar grids as well as modular grids. And we want this functionality to be mixed with all the other features of Grid, including the powerful options for defining a columns, track spanning, explicit placement, and subgrid.

Other people instead believe Masonry should be its own separate display type. At first glance, defining Masonry with a new display type might make a lot of sense. You do get a tidy separation between layout types.

display: block;
display: inline;
display: flexbox;
display: grid;
display: masonry;

The CSS Working Group has not discussed how the syntax for a separate Masonry display type would work, but perhaps it would be patterned after Multicolumn layout.

main { 
  display: masonry;
  columns: 28ch;
}

Or perhaps the syntax would be patterned after Grid, but with significant limitations:

main {
  display: masonry;
  masonry-columns: repeat(5, minmax(28ch, 1fr)); 
                   /* where only one repeating width is allowed */
}

Either way, it’s clear that advocates of this option want Masonry to be limited to a symmetrical grid — where all the columns are the same size as each other. None of the rest of CSS Grid’s track sizing capabilities would be allowed.

Making masonry a simple and separate layout type would avoid the work necessary to keep Grid and Masonry working together in combination — both now and in the long term. Doing this would simplify the layout model, make it easier to implement in browsers, reduce the potential for performance traps, and allow the feature sets of Grid and Masonry to diverge.

Conversely, we believe the effort needed to add this capability to CSS Grid is worth the many benefits to be had. The CSS Grid Level 3 specification has already been written, and implemented in two browser engines. And yes, while making CSS Grid more complex will make it harder to extend in the future, we believe there’s an advantage to having these two types of grid layouts intertwined. This way the CSS Working Group will always define all new additions for both modular and columnar grids. There won’t be something added to display: grid that will be left out of display: masonry, or vice versa. For example, many developers want CSS Grid Level 4 to provide a mechanism for styling grid areas and grid lines — perhaps a way to add a background color to a track, or create a rule line in a gap. It’d be great to ensure that will work for both modular and columnar grids from Day 1.

Another argument made by advocates of display: masonry is that that masonry is conceptually a fundamentally different layout type from CSS Grid, and therefore should have its own display type. They often describe CSS Grid as inherently being about “lining things up in two-dimensions”, and since masonry only lines things up in one dimension, “it’s not a grid”. (In fact, some have advocated that Masonry is more like Flexbox, since “both line things up in one direction”.)

In many ways, your perspective on this question might depends on what you imagine a grid is.

What is a grid?

Grids are an incredibly important aspect of graphic design. Grids are used to line up text, images and other content in a regular pattern. They help readability and usability by making things predictable.

You can trace their use through thousands of years of history.

two photos of book pages, showing historic columnar layouts from long ago[Left] Antoine Vérard published a French translation of Boccaccio in 1498. Seen in Thirty Centuries of Graphic Design, by James Craig and Bruce Barton, 1987. [Right] A Russian illustrated sheet proclaiming that ‘Hops are head above all other fruit’. Seen in Books: A Living History by Martyn Lyons, 2011.

It wasn’t until the 20th century that European and American modernists started promoting the idea that “proper” graphic design grids should line content up in both directions — in rows as well as columns.

spread from a book, showing two different newspaper home pages, where articles line up in a modular grid — in both row and column directionsMassimo Vignelli is especially well-known for promoting the idea that lining things up in both columns and rows is a superior practice of graphic design. Two examples from The Vignelli Canon, 2010.

Even today, there is a lot of debate about which type of grid is the best grid or the only legitimate grid. Many designers claim a 12 column grid is the only correct way to design a web page — or 12 columns for “desktop”, 8 columns for “tablet”, and 4 columns for “phones”. At times designers have gotten quite religious about their ideas of what a “proper grid” looks like.

Mark Boulton argued for years that symmetrical columnar grids are incredibly formulaic and boring. He promoted the use of asymmetrical compound grids in design for the web. Today, luckily CSS Grid Level 1 makes it incredibly easy to create both asymmetrical grids and compound grids, giving designers the freedom to create what they want. But only if they also want all their grids to be a modular grids.

Both modular and columnar grids are in fact grids. And CSS Grid deserves the ability to also create columnar grids.

photograph of an open book, showing many many hand-drawn diagrams of how different layouts can use different numbers of columns Ideas for designing columnar grids, in Editing by Design, by Jan V. White.

We believe there’s an opportunity for CSS to enable a rich history of design grids to come to the web — and would be greatly disappointed to see the new masonry feature limited to only allowing symmetrical columnar grids.

But what do you think?

We want to hear from you

This is where you come in. Try some demos of your own. Write about your thoughts on your own blog. Comment at the CSS Working Group in this issue.

  • Should “masonry”/“waterfall” be part of CSS Grid?
  • Do you want the capabilities to define a columnar grid with CSS Grid — to use subgrid, spanning, explicit placement, and all the many options for track sizing? Or do you only want the ability to define a classic masonry layout with equal-sized columns?
  • Will you use this? What might you create with it?
  • Do you have links to demos you’ve made? We want to see your ideas and use cases.
  • Are there things you want to do that you can’t do with this model?

Often, thinking about something theoretically and actually seeing it in use can be very different. To make sure the CSSWG gets the design of this feature correct, we need developers to gain some hands-on experience, and articulate your thoughts about what it’s like.

The WebKit team has been working on Masonry for a year and a half. It was first enabled by default in Safari Technology Preview 163 in February 2023. There’s a bit more polish needed, and details to work out (naming being one). But we would like to ship this feature soon. To do so, these fundamental questions need to be resolved.

Thank you for your help!

P.S. About the name…

It’s likely masonry is not the best name for this new value. Names in CSS are usually simple words that directly describe the result they create — like center, under, contain, clip, revert, ltr, always, break-word, hidden, allow-end, scale-down, wrap, smooth.

The word “masonry” is more of a metaphor, where the meaning has to be explained with a backstory. Such a term is harder to remember for developers who do not speak English. And arguably, the syntax could just as easily be grid-template-rows: waterfall instead, since that’s the dominant word for this layout used in certain regions, not masonry.

Plus, once you start to write a lot of code using this feature, it’s likely you’ll come to the realization that we did — this really isn’t about the layout used by Pinterest or other similar sites. This is a mechanism for telling the browser, “please create a grid, but without any rows.

Perhaps the best syntax could be grid-template-rows: none; to convey “please do not give me any rows”. Sadly, it’s too late to use this name, because none is the default value for grid-template-* and means “please give me only implicit rows, no explicit ones”.

Instead we could use the name off to convey “please turn off the grid in the row direction, and give me only columns”.

main { 
  display: grid;
  grid-template-columns: repeat(auto-fill, minmax(14rem, 1fr));
  grid-template-rows: off; 
}

The CSSWG is debating this name in this issue. If you have ideas or preferences for a name, please join that discussion.

Meanwhile, masonry is the value that is currently implemented in Safari Technology Preview, since that’s what the Editor’s Draft currently uses. And so that’s what we used in our demos above, and what you should use in yours. But do expect the name of this value to change in the future. And perhaps prepare for a future where we call this “columnar grid” or “Grid Level 3” instead of “Masonry”.

April 19, 2024 10:00 PM

April 16, 2024

Philippe Normand: From WebKit/GStreamer to rust-av, a journey on our stack’s layers

Igalia WebKit

In this post I’ll try to document the journey starting from a WebKit issue and ending up improving third-party projects that WebKitGTK and WPEWebKit depend on.

I’ve been working on WebKit’s GStreamer backends for a while. Usually some new feature needed on WebKit side would trigger work …

By Philippe Normand at April 16, 2024 08:15 PM

April 10, 2024

Release Notes for Safari Technology Preview 192

Surfin’ Safari

Safari Technology Preview Release 192 is now available for download for macOS Sonoma and macOS Ventura. If you already have Safari Technology Preview installed, you can update it in System Settings under General → Software Update.

This release includes WebKit changes between: 276247@main…276863@main.

Authentication

Resolved Issues

  • Fixed setting the cancel flag once the cancel completes regardless of a subsequent request occurring. (276368@main) (124727713)

CSS

New Features

  • Added support for View Transitions. (276426@main) (123128491)

Resolved Issues

  • Fixed setting white-space to a non-default value dynamically on a whitespace or a new line. (276277@main) (92559818)
  • Fixed the proximity calculation for implicit @scope. (276345@main) (124640124)
  • Fixed the Grid track sizing algorithm logical height computation avoid unnecessary grid item updates. (276633@main) (124713418)
  • Fixed any @scope limit making the element out of scope. (276359@main) (124956673)
  • Fixed the contrast of Menu and MenuText system colors. (276597@main) (125270664)
  • Fixed keeping the shorthand value for CSS gap as-is in serialized and computed values. (276794@main) (125335787)

Deprecations

  • Removed WEBKIT_KEYFRAMES_RULE and WEBKIT_KEYFRAME_RULE in CSSRule. (276264@main) (97084520)
  • Removed the SVGAnimateColorElement interface. (276683@main) (122586568)

Editing

Resolved Issues

  • Fixed certain text being duplicated or misplaced. (276317@main) (123642870)
  • Fixed word completion suggestion interfering with a website’s suggestion results. (276344@main) (124727588)

JavaScript

Resolved Issues

  • Fixed Object.groupBy and Map.groupBy to work for non-objects. (276736@main) (125485685)
  • Fixed Array.fromAsync to not call the Array constructor twice. (276752@main) (125509304)

Media

New Features

  • Added support for MSE in workers. (276389@main) (123052315)

Resolved Issues

  • Fixed scrolling for an element when a video element with pointer-events: none is placed over it. (276807@main) (118936715)
  • Fixed allowing a video’s currentTime to be further than the gap’s start time. (276761@main) (124186726)
  • Fixed sampleRate and numberOfChanges to be required and non-zero in a valid AudioEncoderConfig. (276413@main) (125107934)
  • Fixed media elements appending the same media segment twice. (276821@main) (125386530)

Rendering

Resolved Issues

  • Fixed adjusting the size of the scrollable area when changing betwen non-overlay and overlay scrollbars. (276439@main) (117507268)
  • Fixed flickering when showing a layer on a painted background for the first time by avoiding async image decoding. (276513@main) (117533495)
  • Fixed line breaking before or between ruby sequences. (276353@main) (122663646)

Web API

New Features

  • Added support for URL.parse(). (276656@main) (125376520)
  • Added support for shadowRootDelegatesFocus and shadowRootClonable to <template>. (276631@main) (125401993)

Resolved Issues

  • Fixed URL text fragment directives not fully stripped from JavaScript. (276370@main) (107326333)
  • Fixed inserting a <picture> element displaying the same image twice. (276754@main) (123795045)

Web Extensions

New Features

  • Updated to use the web extension architecture in open-source WebKit code. Web extension authors are encouraged to test your extensions and report issues. (123908710)

Web Inspector

Resolved Issues

  • Fixed console clearing unexpectedly when Web Inspector reopens. (276705@main) (124171190)

April 10, 2024 09:30 PM

Optimizing WebKit & Safari for Speedometer 3.0

Surfin’ Safari

The introduction of Speedometer 3.0 is a major step forward in making the web faster for all, and allowing Web developers to make websites and web apps that were not previously possible. In this article, we explore ways the WebKit team made performance optimizations in WebKit and Safari based on the Speedometer 3.0 benchmark.

In order to make these improvements, we made an extensive use of our performance testing infrastructure. It’s integrated with our continuous integration, and provides the capability to schedule A/B tests. This allows engineers to quickly test out performance optimizations and catch new performance regressions.

Improving Tools

Proper tooling support is the key to identifying and addressing performance bottlenecks. We defined new internal JSON format for JavaScriptCore sampling profiler output to dump and process them offline. It includes a script which processes and generates analysis of hot functions and hot byte codes for JavaScriptCore. We also added FlameGraph generation tool for the dumped sampling profiler output which visualizes performance bottlenecks. In addition, we added support for JITDump generation on Darwin platforms to dump JIT related information during execution. And we improved generated JITDump information for easy use as well. These tooling improvements allowed us to quickly identify bottlenecks across Speedometer 3.0.

Improving JavaScriptCore

Revising Megamorphic Inline Cache (IC)

Megamorphic IC offers faster property access when one property access site observes many different object types and/or property names. We observed that some frameworks such as React contain a megamorphic property access. This led us to continuously improve JavaScriptCore’s megamorphic property access optimizations: expanding put megamorphic IC, adding megamorphic IC for the in operation, and adding generic improvements for megamorphic IC.

Revising Call IC

Call IC offers faster function calls by caching call targets inline. We redesigned Call IC and we integrated two different architectures into different tiers of Just-In-Time (JIT) compilers. Lower level tiers use Call IC without any JIT code generation and the highest level tier uses JIT code generatiton with the fastest Call IC. There is a tradeoff between code generation time and code efficiency, and JavaScriptCore performs a balancing act between them to achieve the best performance across different tiers.

Optimizing JSON

Speedometer 3.0 also presented new optimization opportunities to our JSON implementations as they contain more non-ASCII characters than before. We made our fast JSON stringifier work for unicode characters. We also analyzed profile data carefully and made JSON.parse faster than ever.

Adjusting Inlining Heuristics

There are many tradeoffs when inlining functions in JavaScript. For example, inline functions can more aggressively increase the total bytecode size and may cause memory bandwidth to become a new bottleneck. The amount of instruction cache available in CPU can also influence how effective a given inlining strategy is. And the calculus of these tradeoffs change over time as we make more improvements to JavaScriptCore such as adding new bytecode instruction and changes to DFG’s numerous optimization phases. We took the release of the new Speedometer 3.0 benchmark as an opportunity to adjust inlining heuristics based on data collected in modern Apple silicon Macs with the latest JavaScriptCore.

Make JIT Code Destruction Lazy

Due to complicated conditions, JavaScriptCore eagerly destroyed CodeBlock and JIT code when GC detects they are dead. Since these destructions are costly, they should be delayed and processed while the browser is idle. We made changes so that they are now destroyed lazily, during idle time in most cases.

Opportunistic Sweeping and Garbage Collection

In addition, we noticed that a significant amount of time goes into performing garbage collection and incremental sweeping across all subtests in both Speedometer 2.1 and 3.0. In particular, if a subtest allocated a large number of JavaScript objects on the heap, we would often spend a significant amount of time in subsequent subtests collecting these objects. This had several effects:

  1. Increasing synchronous time intervals on many subtests due to on-demand sweeping and garbage collection when hitting heap size limits.
  2. Increasing asynchronous time intervals on many subtests due to asynchronous garbage collection or timer-based incremental sweeping triggering immediately after the synchronous timing interval.
  3. Increasing overall variance depending on whether timer-based incremental sweeping and garbage collection would fall in the synchronous or asynchronous timing windows of any given subtest.

At a high level, we realized that some of this work could be performed opportunistically in between rendering updates — that is, during idle time — instead of triggering in the middle of subtests. To achieve this, we introduced a new mechanism in WebCore to provide hints to JavaScriptCore to opportunistically perform scheduled work after the previous rendering update has completed until a given deadline (determined by the estimated remaining time until the next rendering update). The opportunistic task scheduler also accounts for imminently scheduled zero delay timers or pending requestAnimationFrame callbacks: if it observes either, it’s less likely to schedule opportunistic work in order to avoid interference with imminent script execution. We currently perform a couple types of opportunistically scheduled tasks:

  • Incremental Sweeping: Prior to the opportunistic task scheduler, incremental sweeping in JavaScriptCore was automatically triggered by a periodically scheduled 100 ms timer. This had the effect of occasionally triggering incremental sweeping during asynchronous timing intervals, but also wasn’t aggressive enough to prevent on-demand sweeping in the middle of script execution. Now that JavaScriptCore is knowledgable about when to opportunistically schedule tasks, it can instead perform the majority of incremental sweeping in between rendering updates while there aren’t imminently scheduled timers. The process of sweeping is also granular to each marked block, which allows us to halt opportunistic sweeping early if we’re about to exceed the deadline for the next estimated rendering update.
  • Garbage Collection: By tracking the amount of time spent performing garbage collection in previous cycles, we’re able to roughly estimate the amount of time needed to perform the next garbage collection based on the number of bytes visited or allocated since the last cycle. If the remaining duration for performing opportunistically scheduled tasks is longer than this estimated garbage collection duration, we immediately perform either an Eden collection or full garbage collection. Furthermore, we integrated activity-based garbage collections into this new scheme to schedule them at appropriate timing.

Overall, this strategy yields a 6.5% total improvement in Speedometer 3.0*, decreasing the time spent in every subtest by a significant margin, and a 6.9% total improvement in Speedometer 2.1*, significantly decreasing the time spent in nearly all subtests.

* macOS 14.4, MacBook Air (M2, 2022)

Various Miscellaneous Optimizations for Real World Use Cases

We extensively reviewed all Speedometer 3.0 subtests and did many optimizations for realistic use cases. The examples include but are not limited to: faster Object.assign with empty objects, improving object spread performance, and so on.

Improving DOM code

Improving DOM code is Speedometer’s namesake, and that’s exactly what we did. For example, we now store the NodeType in the Node object itself instead of relying on a virtual function call. We also made DOMParser use a fast parser, improved its support of li elements, and made DOMParser not construct a redundant DocumentFragment. Together, these changes improved TodoMVC-JavaScript-ES5 by ~20%. We also eliminated O(n^2) behavior in the fast parser for about ~0.5% overall progression in Speedometer 3.0. We also made input elements construct their user-agent shadow tree lazily during construction and cloning, the latter of which is new in Speedometer 3.0 due to web components and Lit tests. We devirtualized many functions and inlined more functions to reduce the function call overheads. We carefully reviewed performance profile data and removed inefficiency in hot paths like repeated reparsing of the same URLs.

Improving Layout and Rendering

We landed a number of important optimizations in our layout and rendering code. First off, most type checks performed on RenderObject are now done using an inline enum class instead of virtual function calls, this alone is responsible for around ~0.7% of overall progression in Speedometer 3.0.

Improving Style Engine

We also optimized the way we compute the properties animated by Web Animations code. Previously, we were enumerating every animatable properties while resolving transition: all. We optimized this code to only enumerate affected properties. This was ~0.7% overall Speedometer 3.0 progression. Animating elements can now be resolved without fully recomputing their style unless necessary for correctness.

Speedometer 3.0 content, like many modern web sites, uses CSS custom properties extensively. We implemented significant optimizations to improve their performance. Most custom property references are now resolved via fast cache lookups, avoiding expensive style resolution time property parsing. Custom properties are now stored in a new hierarchical data structure that reduces memory usage as well.

One key component of WebKit styling performance is a cache (called “matched declarations cache”) that maps directly from a set of CSS declarations to the final element style, avoiding repeating expensive style building steps for identically styled elements. We significantly improved the hit rate of this cache.

We also improved styling performance of author shadow trees, allowing trees with identical styles to share style data more effectively.

Improving Inline Layout

We fixed a number of performance bottlenecks in inline layout engine as well. Eliminating complex text path in Editor-TipTap was a major ~7% overall improvement. To understand this optimization, WebKit has two different code paths for text layout: the simple text path, which uses low level font API to access raw font data, and the complex text path, which uses CoreText for complex shaping and ligatures. The simple text path is faster but it does not cover all the edge cases. The complex text path has full coverage but is slower than the simple text path.

Previously, we were taking the complex text path whenever a non-default value of font-feature or font-variant was used. This is because historically the simple text path wouldn’t support these operations. However, we noticed that the only feature of these still missing in the simple text path was font-variant-caps. By implementing font-variant-caps support for the simple text path, we allowed the simple text path to handle the benchmark content. This resulted in 4.5x improvement in Editor-TipTap subtest, and ~7% overall progression in Speedometer 3.0.

In addition to improving the handling of text content in WebKit, we also worked with CoreText team to avoid unnecessary work in laying out glyphs. This resulted in ~0.5% overall progression in Speedometer 3.0, and these performance gains will benefit not just WebKit but other frameworks and applications that use CoreText.

Improving SVG Layout

Another area we landed many optimizations for is SVG. Speedometer 3.0 contains a fair bit of SVG content in test cases such as React-Stockcharts-SVG. We were spending a lot of time computing the bounding box for repaint by creating GraphicsContext, applying all styles, and actually drawing strokes in CoreGraphics. Here, we adopted Blink’s optimization to approximate bounding box and made ~6% improvement in React-Stockcharts-SVG subtest. We also eliminated O(n^2) algorithm in SVG text layout code, which made some SVG content load a lot quicker.

Improving IOSurface Cache Hit Rate

Another optimization we did involve improving the cache hit rate of IOSurface. An IOSurface is a bitmap image buffer we use to paint web contents into. Since creating this object is rather expensive, we have a cache of IOSurface objects based on their dimensions. We observed that the cache hit rate was rather low (~30%) so we increased the cache size from 64MB to 256MB on macOS and improved the cache hit rate by 2.7x to ~80%, improving the overall Speedometer 3.0 score by ~0.7%. In practice, this means lower latency for canvas operations and other painting operations.

Reducing Wait Time for GPU Process

Previously, we required a synchronous IPC call from the Web Process to the GPU process to determine which of the existing buffers had been released by CoreAnimation and was suitable to use for the next frame. We optimized this by having the GPUP just select (or allocate) an appropriate buffer, and direct all incoming drawing commands to the right destination without requiring any response. We also changed the delivery of any newly allocated IOSurface handles to go via a background helper thread, rather than blocking the Web Process’s main thread.

Improving Compositing

Updates to compositing layers are now batched, and flushed during rendering updates, rather than computed during every layout. This significantly reduces the cost of script-incurred layout flushes.

Improving Safari

In addition to optimizations we made in WebKit, there were a handful of optimizations for Safari as well.

Optimizing AutoFill Code

One area we looked at was Safari’s AutoFill code. Safari uses JavaScript to implement its AutoFill logic, and this execution time was showing up in the Speedometer 3.0 profile. We made this code significantly faster by waiting for the contents of the page to settle before performing some work for AutoFill. This includes coalescing handling of newly focused fields until after the page had finished loading when possible, and moving lower-priority work out of the critical path of loading and presenting the page for long-loading pages. This was responsible for ~13% progression in TodoMVC-React-Complex-DOM and ~1% progression in numerous other tests, improving the overall Speedometer 3.0 score by ~0.9%.

Profile Guided Optimizations

In addition to making the above code changes, we also adjusted our profile-guided optimizations to take Speedometer 3.0 into account. This allowed us to improve the overall Speedometer 3.0 score by 1~1.6%. It’s worth noting that we observed an intricate interaction between making code changes and profile-guided optimizations. We sometimes don’t observe an immediate improvement in the overall Speedometer 3.0 score when we eliminate, or reduce the runtime cost of a particular code path until the daily update of profile-guided optimizations kicks. This is because the modified or newly added code has to benefit from profile-guided optimizations before it can show a measurable difference. In some cases, we even observed that a performance optimization initially results in a performance degradation until the profile-guided optimizations are updated.

Results

With all these optimizations and dozens more, we were able to improve the overall Speedometer 3.0 score by ~60% between Safari 17.0 and Safari 17.4. Even though individual progressions were often less than 1%, over time, they all stacked up together to make a big difference. Because some of these optimizations also benefited Speedometer 2.1, Safari 17.4 is also ~13% faster than Safari 17.0 on Speedometer 2.1. We’re thrilled to deliver these performance improvements to our users allowing web developers to build websites and web apps that are more responsive and snappier than ever.

April 10, 2024 05:00 PM

April 08, 2024

WPE WebKit Blog: WPE WebKit 2.44 highlights

Igalia WebKit

The WPE team released WPE WebKit 2.44 a few weeks ago. Let’s have a look at the most significant changes to the port for this release cycle.

DisplayLink is a WebCore feature that improves resource utilization and improves synchronization with vertical screen retrace. 2.44 adds an implementation of this feature for the WPE port that improves rendering performance.

Improved hardware-acceleration video decoding and rendering

When WebKit is using GStreamer 1.24 or newer, video playback can use the new support for DRM modifiers in the DMA-BUF sink. This improves video decoding and rendering, as it allows for zero-copy negotiation with the video decoders.

WebCodec API supported

WPE now supports the WebCodecs API, which allows web developers low-level access to video frames and audio chunks, a feature of importance for multimedia applications that need finer grain control over what gets played on the browser.

Other noteworthy changes

  • Support for the JPEG2000 image format has been removed. WebKit was the only major engine still supporting the format, which these days is rarely used. As a consequence, OpenJPEG is no longer a dependency. JPEG2000 should not be confused with JPEG-XL, which is still supported.
  • Support usage of libbacktrace, enabled by default at build time. This library provides quality stacktraces that can help developers and deployers more efficiently debug crashes in WPE-powered browsers.
  • Many memory and stability improvements, particularly on the multimedia backends.

For a complete list of changes, please check the releases page.

April 08, 2024 12:00 AM

March 28, 2024

Release Notes for Safari Technology Preview 191

Surfin’ Safari

Safari Technology Preview Release 191 is now available for download for macOS Sonoma and macOS Ventura. If you already have Safari Technology Preview installed, you can update it in System Settings under General → Software Update.

This release includes WebKit changes between: 274942@main…276246@main.

Accessibility

New Features

  • Added support for ariaBrailleLabel and ariaBrailleRoleDescription element reflection properties. (275591@main) (123926949)

Resolved Issues

  • Fixed a bug preventing VoiceOver word echoing in some text fields. (275694@main) (122451549) (FB13592798)
  • Fixed time input accessibility by adding labels to subfields. (275301@main) (122590568)
  • Fixed datetime values being exposed to assistive technologies in the wrong timezone. (275265@main) (123522296)
  • Fixed time control accessibility by adding a label to the meridiem component. (276195@main) (123781149)
  • Fixed wrong datetime value being exposed to assistive technologies for datetime-local inputs. (275548@main) (123803281)
  • Fixed the computed role for these elements: dd, details, dt, em, hgroup, option, s, and strong. (276240@main) (124641956)

Authentication

Resolved Issues

  • Fixed excludeCredentials property being ignored during a passkey registration request. (276006@main) (124405037)

CSS

Resolved Issues

  • Fixed getComputedStyle() to work with functional pseudo-elements like ::highlight(). (274846@main) (117864743)
  • Fixed changing color scheme to update gradients with system colors or light-dark(). (275645@main) (121285450)
  • Fixed :empty selector to work with animations. (275832@main) (122838142)
  • Fixed starting style to inherit from its parent’s after-change style. (275061@main) (123302667)
  • Fixed CSS container style queries to allow !important. (275104@main) (123374708)
  • Fixed preserving whitespace when serializing custom properties. (275236@main) (123491915)
  • Fixed updating style correctly for non-inherited custom property mutations. (275445@main) (123645196)
  • Fixed element removed by parent to end up losing the last remembered size. (275606@main) (123975513)
  • Fixed resolving container units in style queries. (275689@main) (124058441)

Forms

Resolved Issues

  • Fixed native text fields becoming invisible in dark mode. (275363@main) (123658326)
  • Fixed <select multiple> scrollbars to match the used color scheme. (275505@main) (123807167)
  • Fixed fallback native <select> rendering in dark mode. (275532@main) (123845293)

JavaScript

Resolved Issues

  • Fixed String.prototype.replace to not take the fast path if the pattern is RegExp Object and the lastIndex is not numeric. (275255@main) (101122567)
  • Fixed Symbol.species getters to not share a single JS Function. (275064@main) (120416817)
  • Fixed try/catch to not intercept errors originated in [[Construct]] of derived class. (275353@main) (121959506)
  • Fixed TypedArray sorting methods to have a special-case for camparator returning false. (276130@main) (122093956)
  • Fixed emitReturn() to load this value from arrow function lexical environment prior to the TDZ check. (275425@main) (122430056)
  • Fixed NFKC normalization to work with Latin-1 characters. (275062@main) (123328161)
  • Fixed parsing of private names with Unicode start characters. (275152@main) (123425805)
  • Fixed instanceof to not get RHS prototype when LHS is primitive. (275318@main) (123629166)
  • Fixed bracket update expression to resolve property key at most once. (275531@main) (123872374)
  • Fixed bracket compound assignement to resolve the property key at most once. (276014@main) (124420301)

Lockdown Mode

Resolved Issues

  • Fixed: Updated the trusted list of fonts to address missing glyphs on some sites. (275705@main) (123658932)

Media

New Features

  • Added support for WebCodecs AV1 when an AV1 hardware decoder is available. (275174@main) (123433815)

Resolved Issues

  • Fixed audio distortion over internal speakers when streaming content in web browsers. (275262@main) (122590884)
  • Fixed playback for MSE videos on some sites. (275443@main) (123528095)
  • Fixed firing loadeddata events for <audio> and <video> on page load. (275997@main) (124079735) (FB13675360)

SVG

Resolved Issues

  • Fixed returning the initial value for the SVG gradient stop-color if it is not rendered in the page. (274997@main) (123262508)
  • Fixed the SVG marker segment calculations if the marker path consists of sub-paths. (275167@main) (123434203)

Web API

Resolved Issues

  • Fixed cssTextsetter to change the style attribute when the serialization differs. (276176@main) (29861252) (FB5535475)
  • Fixed mousemove events in an iframe when the mouse is clicked from outside the iframe and then moves into it while the button is held down. (275157@main) (120540148) (FB13517196)
  • Fixed getGamepads() to no longer trigger an insecure contexts warning. (274962@main) (123039555)

Deprecations

  • Removed HashChangeEvent‘s non-standard initHashChangeEvent() method. (276232@main) (124736521)

Web Inspector

Resolved Issues

  • Fixed expanded sections of Storage to not collapse. (275783@main) (107687975)
  • Fixed CSS font property values marked !important not getting overridden when using the interactive editing controls. (275389@main) (112080113)
  • Fixed runtimes to be aligned in the Audit tab. (275298@main) (121810292)
  • Fixed info and debug buttons not appearing in the Console Tab until new console messages are displayed. (275914@main) (122923625)
  • Fixed remembering the message type selection in the Console tab. (275143@main) (122924275)
  • Fixed autocomplete for the text-indent property suggesting prefixed properties instead of each-line or hanging. (274979@main) (123240715)
  • Fixed background autocompletion suggestion to include repeating-conic-gradient. (275160@main) (123428709)

WebRTC

Resolved Issues

March 28, 2024 02:16 PM

March 20, 2024

Jani Hautakangas: Bringing WebKit back to Android.

Igalia WebKit

It’s been quite a while since the last blog post about WPE-Android, but that doesn’t mean WPE-Android hasn’t been in development. The focus has been on stabilizing the runtime and implementing the most crucial core features to make it easier to integrate new APIs into WPEView.

Main building blocks #

WPE-Android has three main building blocks:

  • Cerbero
    • Cross-platform build aggregator that is used to build WPE WebKit and all of its dependencies to Android.
  • WPEBackend-Android
    • Implements Android specific graphics buffer support and buffer sharing between WebKit UIProcess and WebProcess.
  • WPEView
    • Allows displaying web content in activity layout using WPEWebKit.

WPE-Android high-level design

What’s new #

The list of all work completed so far would be quite long, as there have been no official releases or public announcements, with the exception of the last blog post. Since that update, most of the efforts have been focused on ‘under the hood’ improvements, including enhancing stability, adding support for some core WebKit features, and making general improvements to the development infrastructure.

Here is a list of new features worth mentioning:

  • Based on WPE WebKit 2.42.1, the project was branched for the 2.42.x series. Future work will continue in the main branch for the next release.
  • Dropped support for 32-bit platforms (x86 and armv7). Only arm64-v8a and x86_64 are supported
  • Integration to Android main loop so that WPE WebKit GLib main loop is driven by Android main loop
  • Process-Swap On Navigation aka PSON
  • Added ASharedMemory support to WebKit SharedMemory
  • Hardware-accelerated multimedia playback
  • Fullscreen support
  • Cookies management
  • ANGLE based WebGL
  • Cross process fence insertion for composition synchronization with Surface Flinger
  • WebDriver support
  • GitHub Actions build bots
  • GitHub Actions WebDriver test bots

Demos #

WPEWebkit powered web view (WPEView) #

Demo uses WPE-Android MiniBrowser sample application to show basic web page loading and touch-based scrolling, usage of a simple cookie manager to clear the page’s date usage, and finally loads the popular “Aquarium sample” to show a smooth (60FPS) WebGL animation running thanks to HW acceleration support.

WebDriver #

Demo shows how to run WebDriver test with with emulator. Detailed instructions how to run WebDriver test can be found in README.md

Test requests a website through the Selenium remote webdriver. It then replaces the default behavior of window.alert on the requested page by injecting and executing a JavaScript snippet. After loading the page, it performs a click() action on the element that calls the alert. This results in the text ‘cheese’ being displayed right below the ‘click me’ link.

Test contains three building blocks:

  • WPE WebDriver running on emulator
  • HTTP server serving test.html web page
  • Selenium python test script executing the test

What’s next #

We have ambitious plans for the coming months regarding the development of WPE Android, focusing mainly on implementing additional features, stabilization, and performance improvements.

As for implementing new features, now that the integration with the WPE WebKit runtime has reached a more robust state, it’s time to start adding more of the APIs that are still missing in WPEView compared to other webviews on Android and to enable other web-facing features supported by WebKit. This effort, along with adding support for features like HTTP/2 and the remote Web Inspector, will be a major focus.

As for stabilization and performance, having WebDriver support will be very helpful as it will enable us to identify and fix issues promptly and thus help make WPE Android more stable and feature-complete. We would also like to focus on conformance testing compared to other web views on Android, which should help us prioritize our efforts.

The broader goal for this year is to develop WPE-Android into an easy-to-use platform for third-party projects, offering a compelling alternative to other webviews on the Android platform. We extend many thanks to the NLNet Foundation, whose support for WPE Android through a grant from the NGI Zero Core fund will be instrumental in helping us achieve this goal!

Try it yourself #

WPE-Android is still considered a prototype, and it’s a bit difficult to build. However, if you want to try it out, you can follow the instructions in the README.md file. Additionally, you can use the project’s issue tracker to report problems or suggest new features.

March 20, 2024 12:00 AM

March 19, 2024

Introducing Natural Input for WebXR in Apple Vision Pro

Surfin’ Safari

Apple Vision Pro is here, and with it, a lot of excitement about the possibilities for WebXR in visionOS. Support is in progress, where you can test it today.

WebXR now includes a more natural and privacy-preserving method for interaction — the new transient-pointer input mode — available for Safari 17.4 in visionOS 1.1. Let’s explore how natural input for WebXR works, and how to leverage it when developing a WebXR experience for Apple Vision Pro.

Background

WebXR lets you transform 3D experiences created in WebGL into immersive, spatial experiences available directly in the browser. Defined in web standards governed by the W3C, WebXR is an excellent choice for presenting immersive content to your users.

One challenge, though, is that because it’s completely immersive — and rendered entirely through WebGL — it’s not possible to provide interaction via DOM content or the traditional two-dimensional input available on a typical web page via a mouse or trackpad. This, combined with the fact that a spatial experience requires spatial input, means WebXR needs a totally new interaction model.

The interaction model of visionOS, known as natural input, uses a combination of eyes and hands for input. A user simply looks at a target and taps their fingers to interact. (Or uses the alternatives provided by accessibility features.)

The initial web standards for WebXR assumed all input would be provided by persistent hardware controllers. Since the natural input interaction model of visionOS differs from XR platforms which rely on listening to physical controllers and button presses, many existing WebXR experiences won’t work as intended on Apple Vision Pro.

We’ve been collaborating in the W3C to incorporate support for the interaction model of visionOS into WebXR. And we’re excited to help the WebXR community add support to popular WebXR frameworks.

Using Natural Input interactions

Since WebXR in visionOS requires the use of spatial inputs, rather than a trackpad, touch, or mouse, and the DOM isn’t visible within a WebXR session, inputs are provided as part of the XRSession itself. Events related to the input, e.g. select, selectstart and selectend are then dispatched from the session object. The XRInputSources are available within the xrSession.inputSources array. Because the default WebXR input in visionOS is transient, that array is empty — until the user pinches. At that moment, a new input is added to the array, and the session fires an inputsourceschange event followed by a selectstart event. You can use these for detecting the start of the gesture. To differentiate this new input type, it has a targetRayMode of transient-pointer.

Ongoing interaction

The XRInputSource contains references to two different positions in space related to the input: the targetRaySpace and the gripSpace. targetRaySpace represents the user’s gaze direction, this space begins with its origin between the user’s eyes and points to what the user was looking at the start of the gesture. The targetRaySpace is initially set to the direction of the user’s gaze but is updated with the movement of their hand rather than their eyes — that is, a movement of the hand to the left will move that ray to the left as well. The gripSpace represents the location of the user’s pinching fingers at the current point in time.

The targetRaySpace can be used for finding what the user wanted to interact with when they started the gesture typically by raycasting into the scene and picking the intersected object and the gripSpace can be used for the positioning and orientation of objects near the user’s hand for interaction purposes, e.g. to flip a switch, turn a dial or pick up an item from the virtual environment. To learn more about finding the pose of the input, see Inputs and input sources on MDN.

End of interaction

Three events are fired from the session object when the user releases the pinch.

First is a select and selectEnd event, both referencing the input as the event.inputSource object. The session then fires a final inputsourceschange event, indicating that this inputSource has been removed.

Benefits for simplicity and privacy

As each input represents a single, tracked point in space per pinch, and because it exists only for the duration of the user’s pinch, significantly less data about a user’s motion is required overall.

Combining transient inputs with hand tracking

WebXR on Safari in visionOS continues to support full hand tracking as well, supplying hand joint information for the duration of the experience. If the call to navigator.xr.requestSession has included hand-tracking as an additional feature and this is granted by the user, the first two inputs in the inputSources list will be standard tracked-pointers supplying this joint information. For information on requesting features in an XRSession, refer to the documentation at MDN. Because these inputs persist for the duration of the session, any transient-pointer inputs will appear further down the list. The hand inputs are supplied for pose information only and do not trigger any events.

Testing natural input on visionOS

Using the visionOS simulator

The new transient-pointer input mode works in the visionOS simulator, so you can test your Web XR experience without needing Apple Vision Pro. To get started:

  1. Download Xcode from the Mac App Store.
  2. Select the visionOS SDK for download and installation.
    Checklist of which platforms you would like to develop for — macOS and visionOS are checked, ready to press the download & install button.
  3. Launch Xcode to complete the installation process. This installs a new application named “Simulator”. Learn more about using the visionOS Simulator.
  4. Open the Simulator app on macOS.
  5. Create a new visionOS simulator or open an existing one.

If you have a website open in Safari on macOS, you can easily trigger that page to open in Safari in visionOS Simulator. On Mac, select Develop > Open Page With > Apple Vision Pro. (If the Develop menu is not present in the menu bar, enable features for web developers.)

Enabling WebXR support

Whether you have Apple Vision Pro or are using visionOS Simulator, you’ll want to turn on support for WebXR. From the Home View, go to Settings > Apps > Safari > Advanced > Feature Flags and enable WebXR Device API.

Identifying potential issues in existing WebXR scenes

WebXR is provided in visionOS 1.1 for testing purposes only. If you encounter bugs in the browser, please don’t attempt to target a user-agent string to work around them. Instead, report them to us via Feedback Assistant.

When hand-tracking is enabled I can’t select anything.

Your experience probably is only looking at events which correspond to the first two inputs in the inputSources array. Many examples in the popular framework Three.js only act on events related to inputs at index 0 and 1. When hand tracking is enabled, the inputs corresponding to the hand tracking data appear in entries 0 and 1 but the transient-pointer style inputs appear in entries 2 and 3.

When I attach an object to the controller it appears at the user’s face.

The input’s targetRaySpace originates between the user’s eyes and points in the direction of their gaze. To place an object in the user’s hand, instead attach objects to the input’s gripSpace position, which corresponds to the location of the user’s pinch.

The selection ray to interact with objects lags between the position of my current selection and my last one.

Some frameworks attempt to smooth out controller motion by interpolating the position of the controller each frame. This interpolated motion should be reset if a controller is disconnected and reconnected. You may need to file an issue with the framework author.

Let us know

We can’t wait to see what you make. To share your thoughts on WebXR, find us on Mastodon at @ada@mastodon.social, @jensimmons@front-end.social, @jondavis@mastodon.social, or on X at @zachernuk. Or send a reply on X to @webkit. You can also follow WebKit on LinkedIn. If you run into any issues, we welcome your feedback (to include a sysdiagnose from Apple Vision Pro), or your WebKit bug report about Web Inspector or web platform features. Filing issues really does make a difference. Thanks.

March 19, 2024 04:30 PM

March 18, 2024

Implementing Vertical Form Controls

Surfin’ Safari

Safari 17.4 adds vertical writing mode support for form control elements across macOS, iOS, iPadOS, and visionOS.

all the form controls, typeset vertically

Setting written text vertically is commonly observed in East Asian languages. For example, Chinese, Japanese, and Korean (CJK) may be written vertically and read top-to-bottom, flowing in lines from right to left. Similarly, Traditional Mongolian is a vertical script that flows in lines from left to right.

Support for vertical text has been available in browsers for several years using the CSS writing-mode property. However, until recently, support for the vertical-lr and vertical-rl values for form controls was inconsistent among all browsers. Consequently, as part of Interop 2023, the industry committed to working towards vertical writing mode support in form controls. We are excited to see cross-browser support improve considerably. And we are proud that Safari 17.4 brings support for vertical writing modes to form controls everywhere you find Safari.

Using vertical writing modes with form controls

Adopting vertical form controls is as simple as adding a CSS declaration using the writing-mode property and applying it to your controls. For a right-to-left block flow direction, as observed in CJK languages, use writing-mode: vertical-rl. Or, for a left-to-right block flow direction, use writing-mode: vertical-lr.

button, textarea, progress, meter, input, select {
    writing-mode: vertical-rl;
}

Support for vertical writing mode is available for the following elements: <button>, <textarea>, <progress>, <meter>, <input>, and <select>. WebKit is committed to supporting vertical writing modes on any new controls moving forward, including <input type="checkbox" switch>, which was also added in Safari 17.4.

Note that in all browsers, any popup UIs associated with form controls, such as the menu for a <select> element, or the color picker for an <input type="color"> are still displayed horizontally.

Implementation

Adding vertical writing mode support for form control elements in WebKit was a large undertaking that involved over 10 unique types of controls. Below are some of the changes that were needed to support this feature in WebKit.

Style

WebKit’s user-agent stylesheet made frequent use of physical properties, such as width, height, and margin. In order to support vertical writing modes, this usage was updated to use logical properties, such as inline-size and block-size, where appropriate.

Layout

Form controls in WebKit make heavy use of custom layout code. While the interfaces were designed to support logical properties, the implementations often assumed a horizontal writing mode. We updated the methods to compute logical widths to ensure they considered the writing mode.

Additionally, custom baseline adjustment logic, which makes controls such as checkboxes and radio buttons look great alongside text, was updated to use the central baseline for vertical writing modes. This ensures that controls continue to look great alongside text in a vertical writing mode.

Rendering

Rendering form controls with a vertical writing mode required unique changes depending on the control and system capabilities.

On macOS, WebKit’s form controls match the look and feel of the operating system. However, macOS itself does not support vertical writing modes. Consequently, some controls, such as <progress>, are simply rotated after obtaining the system image in order to support vertical rendering.

two progress bars, laid out horizontally and vertically

Other controls cannot simply be rotated due to details such as shadows, for example <select>, so we used a custom fallback rendering approach.

the select menu typeset vertically

If a control was already custom painted in WebKit, it was updated to use logical coordinates rather than physical coordinates. We used this approach throughout the iOS rendering code and it was also necessary in our rewrite of the listbox (<select multiple>) on macOS.

the select multiple form control, typeset vertically

Finally, some controls, such as checkbox and radio buttons, did not require any rendering changes. As “glyph-like” controls, they look the same in all writing modes.

checkboxes and radio buttons on iOS typeset vertically

WebKit remains committed to improving internationalization support in our engine. By adding vertical writing mode support for form control elements in Safari 17.4, we hope to empower authors to create the best content for their local and global communities.

A special thanks goes to Tim Nguyen, who kicked off the vertical form controls project in WebKit, and Alan Baradlay, whose deep knowledge of CSS layout proved invaluable in driving the project to the finish line.

Feedback

We love hearing from you. Send a tweet to @webkit to share your thoughts on this feature. Find us on Mastodon at @jensimmons@front-end.social and @jondavis@mastodon.social. You can also follow WebKit on LinkedIn. If you run into any issues, we welcome your WebKit bug reports on WebKit features like this. Reporting issues makes an enormous difference.

You can also download the latest Safari Technology Preview to try out new web platform features like this before they appear in a Safari beta.

March 18, 2024 03:00 PM

March 11, 2024

Speedometer 3.0: The Best Way Yet to Measure Browser Performance

Surfin’ Safari

As announced on browserbench.org today, in collaboration with other browser engine developers, Apple’s WebKit team is excited to introduce Speedometer 3.0, a major update that better reflects the Web of today. It’s built together by the developers of all major browser engines: Blink, Gecko, and WebKit with hundreds of contributions from companies like Apple, Google, Intel, Microsoft, and Mozilla. This post is a deep dive into how the collaborative Speedometer project improved the benchmark’s measurements methods and test content.

To recap history, in 2014, the WebKit team at Apple released the Speedometer browser benchmark, designed to measure the responsiveness of websites and web apps.

The original Speedometer simulated user interactions in web applications, driving TodoMVC sample apps written using different JavaScript frameworks to add, complete, and remove todo items. It was unlike other DOM or web app benchmarks publicly available at the time. These older benchmarks were mostly collections of micro-benchmarks, and didn’t reflect how DOM APIs were used in real web apps, or how individual APIs interacted with the rest of the web browser engine. Speedometer quickly became an important tool for performance measurement and tuning not just in WebKit but also in other browser engines.

In 2018 the WebKit team, in collaboration with Google’s Chrome team, released Speedometer 2.0, updated to use the latest frameworks and libraries available at the time. The Speedometer benchmark has since gained even more popularity among browser engines as a guide for optimization, and among independent testers and reviewers to compare different devices, operating systems, and browsers.

Today’s release of Speedometer 3.0 marks a major step forward in web browser performance testing. It improves the accuracy of measurement and measures the performance of a wide variety of contents.

Cross-Browser Collaboration

Speedometer 3.0’s release is a result of the collaboration among browser developers to improve the Web as a whole together. Much as Interop 2024 represents joint work to test and improve standards compliance, Speedometer 3.0 is a joint effort to test and improve browser performance.

Where previous Speedometer versions were developed as part of the WebKit project, Speedometer 3.0 has been developed and released under a joint multi-stakeholder governance model including the three major engine browsers: Blink, Gecko, and WebKit, and the repository has received hundreds of open source contributions since the original announcement in December 2022. This collaboration better ensures fairness in measurement and workload composition. And together, the group created a shared vision for the benchmark.

Improved Test Harness

We’ve improved the way Speedometer measures runtime performance. Prior Speedometer versions measured the time to run a test script synchronously as “sync” time; and the time until a zero-delay timer scheduled at the end of “sync” work fires as “async” time, as shown in the following diagram:

In Speedometer 2, async time was measured as between when a timer is scheduled at the end of sync time and when the timer fires. That sometimes captures layout and paint and other tasks running after synchronous script execution but before the timer fires.

However, this method sometimes misses important work that browser engines do in response to script-driven changes, because synchronous tasks and the zero-delay timer are scheduled without considering the timing of rendering updates. It also didn’t capture any work frameworks delay until the next requestAnimationFrame (rAF) callback, a common technique in modern frameworks. The following diagram illustrates how important work could be missed by the time calculations.

In Speedometer 2, it was possible that requestAnimationFrame, layout, and paint to happen after the async timer had fired in some cases. In such cases, we fail to capture the time browser spends updating the rendering of a web page.

Speedometer 3.0 takes advantage of the fact that all browser engines have adopted the HTML5 event loop model for updating the webpage rendering. It measures test scripts within a requestAnimationFrame callback as “sync” time, and the time to fire zero-delay timer scheduled in a second requestAnimationFrame as “async” time:

In Speedometer 3, we schedule two requestAnimationFrame callbacks. The first one runs and measures sync time, and the second one schedules a timer to measure the async time. Because browsers are expected to update the rendering after invoking requestAnimationFrame callback but before running any other tasks, we always capture the time browser spends updating the rendering of a web page.

Because the zero-delay timer is now scheduled in a second requestAnimationFrame, it’s guaranteed to be fired after all the zero-delay timers scheduled during the synchronous portion of the test had fired. Thanks to HTML5’s event loop processing model, browser engines update the rendering of web pages after all requestAnimationFrame are called before the next zero-delay timer fires. These changes greatly improved Speedometer’s ability to accurately measure the runtime of synchronous work and asynchronous work browsers do in response to script that handles user events.

The test harness has also been rewritten to use modern JavaScript features like modules, native promises, let & const, async & await, and class syntax, which were not widely available at the time Speedometer 1.0 was first written.

Like its precursors, Speedometer 3.0 sums up the runtime taken to simulate user actions such as adding todo items, completing them, and removing them per each workload, and computes the geometric mean of the totals across different workloads. The final score is calculated as the arithmetic mean of the reciprocal of the geometric mean:

In Speedometer 3, each workload can have multiple sync and async times. The total time for all sync and async time is calcualted for each workload, and the score is calculated as the reciprocal of the geometric mean of the totals across workloads.

Adjustment to Score

Since Speedometer’s benchmark content was last updated in 2018, web browsers have gotten increasingly better at handling Speedometer content. Moreover, new hardware, such as Apple Silicon Macs, continues to push the boundary of what’s possible in computing. Where originally scores were scaled to be under 100, modern browsers now can score over 500 on the fastest devices. To make scores easier to compare and to make room for future improvements, we’ve adjusted the score so that a typical web browser will get a score in the 20-30 range to start out.

Updated UI Frameworks

Now let’s take a look at the test content in Speedometer 3. Like the past versions of Speedometer, version 3.0 includes TodoMVC-based todo apps that emulate adding, completing, and removing todo items. To better represent the modern Web, the most widely used JavaScript UI frameworks were identified from the HTTP Archive in March 2023:

In the order of popularity, React, Vue.js, React, Backbone.js, AngularJS, Next.js, Angular, Nuxt.js, lit-html, Knockout.js, Alpine.js, Marionette.js, Gatsby, Svelte, React Redux

The monthly downloads in NPM was also taken into account to find frameworks with high momentum:

In the order of NPM monthly downloads, React, React Redux, Next.js, Vue.js, Angular, Preact, lit-html, Backbone.js, Nuxt.js, Svelte, AngularJS, Gatsby, Alpine.js, Knockout.js, Marionette.js

Based on these data points, we’ve included the following JavaScript frameworks in our todo apps: Angular, Backbone, jQuery, Lit, Preact, React, React+Redux, Svelte, and Vue. For each framework, the most commonly used version at the time was picked. Todo implementations written in vanilla JavaScript using ES5, ES6, and web components are also included.

Complex DOM Versions

In addition, Speedometer 3.0 includes “complex DOM” versions of some of the TodoMVC applications. In these complex DOM versions, each todo app is embedded inside a UI structure which mimics a web application with many deeply nested DOM nodes and plenty of CSS rules. Even though the benchmark still emulates the same set of operations, doing so in the context of more DOM elements and CSS rules adds work and captures additional performance bottlenecks.

In order to ensure the variety of performance scenarios to be tested, Speedometer 3.0 includes 6 simple DOM todo applications and 6 complex DOM todo applications.

Complex DOM workloads has ribbon menus, side bar tree view, search field, and other complex UI elements surrounding the todo app.

Broader Content

Together, these changes to todo apps dramatically improved the coverage of the benchmark. But Speedometer 3.0 takes it a step further and includes entirely new kinds of applications.

Speedometer 3.0 includes two test apps that mimic typical news sites, built using the popular single page application frameworks Next.js and Nuxt. It emulates user actions such as clicking on menu items and navigating to another page in the single page app setup.

News sites workload mimics a popular news site with navigation menu, hero images, headlines, and a summary of articles.

Speedometer 3.0 also includes four charting applications based on Observable Plot, chart.js, React stockcharts, and WebKit’s performance dashboards. Observable Plot and React Stockcharts are based on D3 and test manipulating SVG-based graphics. Chart.js and WebKit’s performance dashboards test drawing canvas-based graphics.

Charting workloads draw bar graphs of the number of airports in each U.S. state for example.

Finally, Speedometer 3.0 has added two text editing applications: a JavaScript code editor built with CodeMirror and a WYSIWYG editor built with TipTap. In both scenarios, it emulates the steps to create a new editable region, loading a large amount of text, and syntax highlighting or boldening text:

CodeMirror workload, for example, loads React's codebase and enables syntax highlighting

The addition of these new applications dramatically broadens the scope of what Speedometer 3.0 measures, and provide new opportunities for browser engines to optimize a broad spectrum of features like JavaScript, style, layout, graphics, and DOM.

Future Work

Today marks a remarkable milestone for the Web platform. Speedometer 3.0 sets a whole new standard for measuring web browser performance. As browser developers optimize their engines, this will expand the horizon of what Web developers can achieve in the years to come. Because the goal of the Speedometer benchmark is to reflect the real-world Web as much as possible, we’re expecting this benchmark to evolve over time. We’ll be regularly updating the list of frameworks to be tested, and periodically updating the tested framework’s versions to reflect the real world usage. You can
try Speedometer 3 benchmark on browserbench.org. If you have any feedback or questions, feel free to file issues on Github.

March 11, 2024 04:00 PM

March 06, 2024

Release Notes for Safari Technology Preview 190

Surfin’ Safari

Safari Technology Preview Release 190 is now available for download for macOS Sonoma and macOS Ventura. If you already have Safari Technology Preview installed, you can update it in System Settings under General → Software Update.

This release includes WebKit changes between: 274067@main…274941@main.

Authentication

New Features

  • Added support for CTAP to set a pin. (274385@main) (113573055)

CSS

New Features

  • Added support for the safe keyword in flexbox properties. (274304@main) (118000717)
  • Added support for CSS style container queries. (274481@main) (122800215)

Resolved Issues

  • Fixed getComputedStyle() to work with functional pseudo-elements like ::highlight(). (274846@main) (117864743)

History

Resolved Issues

  • Fixed the properties of History to throw a SecurityError when not in a fully active Document. (274260@main) (118750576)

HTML

New Features

  • Added support for the shadowrootclonable attribute and aligned with declarative shadow root standards changes. In particular, web developers will now have to set this attribute if they want their declarative shadow root to be clonable. (274727@main) (123006751)

JavaScript

Resolved Issues

  • Fixed several issues:
    • direct eval() in a default value expression inside a rest parameter creates a variable in the environment of the function rather than the separate one of the parameters;
    • a ReferenceError is thrown when accessing a binding, which is defined inside rest parameter, in eval(), or a closure created in a default value expression of a preceding parameter, but only if there is a var binding by the same name;
    • a closure, created in the default value expression inside a rest parameter, is created in a different VariableEnvironment of the function than its counterparts in preceding parameters which causes the incorrect environment to be consulted when querying or modifying parameter names that are “shadowed” by var bindings. (274109@main) (121961421)

Media

Resolved Issues

  • Fixed pseudo-element font size calculation to fix subtitle size in fullscreen mode. (274480@main) (122584350)

Deprecations

  • Removed non-standard VTTRegion.track. (274936@main) (123172214)

Networking

Resolved Issues

  • Fixed upgrading inactive or passive subresource requests and fetches in would-be mixed security contexts to match standards. (274409@main) (101678657)
  • Fixed loading WebArchives with a non-persistent datastore. (274565@main) (122290562)

Rendering

Resolved Issues

  • Fixed backdrop-filter to apply to the border area of an element with a border-radius. (274757@main) (122295068)

SVG

New Features

  • Added support for the turn unit in <angle>. (274778@main) (120840743)

Resolved Issues

  • Fixed the UA stylesheet for links in SVGs to apply cursor: pointer matching standards. (274422@main) (122715957)

URLs

Resolved Issues

  • Fixed the URL parser to parse URLs that include authority and a backslash after the host. (274915@main) (119219832)

Web API

Resolved Issues

  • Fixed changing HTMLCanvasElement width or height causing intermediate buffer allocations. (274135@main) (122309325)
  • Fixed canvas captureStream stuttering with WebGL. (274454@main) (122471664)
  • Fixed DOM Range to correctly account for CDATASection nodes. (274346@main) (122608224)

Web Inspector

Deprecations

  • Disabled the network throttling experimental feature. (274112@main) (122327408)

March 06, 2024 10:50 PM

March 05, 2024

WebKit Features in Safari 17.4

Surfin’ Safari

Just like Safari 15.4 and Safari 16.4, this March’s release of Safari 17.4 is a significant one for web developers. We’re proud to announce another 46 features and 146 bug fixes.

You can experience Safari 17.4 on iOS 17.4, iPadOS 17.4, macOS Sonoma 14.4, macOS Ventura, macOS Monterey, and in visionOS 1.1.

Architectural improvements

It’s always exciting to ship new features that you can use while building websites and web apps for your users. WebKit engineers also work on many important projects beyond implementing new web platform features. Recently, much effort has gone into multiple infrastructure projects that strengthen WebKit for the long-term.

We completed the final installment of our multi-year long rewrite of our inline layout engine (more on that later). We built two new iOS frameworks with hundreds of new APIs to support functionality used by web browsers, including multiprocess, JIT, and advanced keyboard & touch event access — and we are pivoting WebKit to use these new frameworks. We’re working on several other large projects that deepen security and privacy. And we’ve been hard at work to make Safari even faster. For many years, Safari has held the crown of the world’s fastest browser. It’s important to us to keep pushing the boundaries of speed, as the websites you build continue to get more complex. Hundreds of recent changes result in Safari 17.4 showing a significant performance bump.

Web Apps

Safari 17.4 brings two improvements to web apps on Mac.

First, Safari adds support for the shortcuts manifest member on macOS Sonoma. This gives you a mechanism in the manifest file for defining custom menu commands that will appear in the File menu and the Dock context menu.

Web Kittens web app open on macOS, with the File menu showing and four custom shortcuts listedOn Mac, our Web Kittens web app includes four shortcuts. You can see them listed in the File menu: New Kitten, Discover, Messages, and Notifications. They each open a menu item by going to the appropriate URL.

A web app shortcut consists of a name, (the words you’d like to appear in the menu), and a url. When a user activates the command, it opens the specified URL inside the web app.

"shortcuts": [
  {
    "name": "New Kitten",
    "url": "/new-kitten"
  },
  {
    "name": "Discover",
    "url": "/discover"
  }
]

Users can set up custom keyboard shortcuts for app menu commands in System Settings > Keyboard > Keyboard Shortcuts > App Shortcuts. By default, macOS does not assign web app shortcuts any keyboard commands.

Second, Safari 17.4 now supports the categories manifest member on macOS Sonoma. This member provides you with a mechanism for telling the browser which categories your web app belongs in. On Mac, when a user creates a Launchpad folder that contains web apps, the folder is automatically named accordingly.

Launchpad on macOS showing two app icons in a group titled Social Networking

Form elements

Switch control

The switch is a popular interface for many use cases, but until now, there was no easy way to put a switch on the web. Instead developers might use a checkbox input field, remove the visual look of the checkbox with appearance: none, and write custom styles to create something that looks like a switch.

Now, with WebKit for Safari 17.4, HTML supports a native switch. If you code <input type="checkbox" switch>, the browser will simply create a switch for you, and map it to role=switch and related ARIA features.

Try this demo in Safari 17.4. Currently, in other browsers you will see three checkboxes.

Extending the current HTML checkbox provides several benefits and honors the W3C’s HTML Design Principles. First, this design degrades gracefully — which means you can use <input type="checkbox" switch> today. Browser that have support will show a switch, while browsers that do not have support will show a checkbox. No user will get a broken experience, and you don’t have to wait until all of your users have a browser with support in order to utilize this on your website or web app. This design also does not reinvent the wheel. It matches the way form controls have always worked on the web, and feels just like the code you’re used to. It’s an incremental evolution of the web. And as a simple solution, it avoids needless complexity.

The accent-color property can be used to change the background color of the switch in the “on” state. And, exactly like other form controls, you can use appearance: none to remove the system default styling and apply your own, perhaps while leveraging :before and :after.

In the future, there will likely be multiple pseudo-elements to make it even easier to style the switch with your custom styles. You can try out ::track and ::thumb in Safari Technology Preview today and let us know what you think of this approach. See how they work in this demo, after enabling the “::thumb and ::track pseudo-elements” feature flag. (These pseudos are waiting to ship until there is a more comprehensive plan for styling form controls proposed, discussed and resolved on at the CSS Working Group.)

Vertical writing modes

From the beginning, the web has always been interactive. Even before any method of custom styling was available, form controls and input fields provided the means for users to communicate back to the website and to each other. The web was also originally designed in an era when the Latin alphabet (used by many languages, including English) was the presumed default, with its horizontal top-to-bottom writing mode.

For thirty years, form controls have presumed a horizontal writing mode. Typesetting in a vertical writing mode for languages like Chinese, Japanese, Korean, and Mongolian did not include vertical form controls. Now that’s changed. Starting in Safari 17.4, vertical form controls are supported. This includes meter, range, progress and other form controls that could make for great UI in any language when laid out in a vertical format.

screenshot of all the form controls, now laid out in a vertical directionTry this demo of vertical form controls in a browser that has support.

Horizontal Rules inside Select

You can use an <hr> element (a horizontal rule) inside a <select> element to draw a separator line. WebKit shipped support in Safari 17.0 on macOS. Now, Safari 17.4 on iOS 17.4, iPadOS 17.4, and in visionOS 1.1 also has support.

Safari window floating in front of mountains in Vision Pro, with a a select menu open on a web page, showing lines between list items

Try a demo and read the story of how hr in select was supported years ago, went away, and is now restored.

And more

WebKit for Safari 17.4 also adds support for the showPicker() method for <input type="date"> on macOS.

CSS

Inline Layout

One of the infrastructure projects that’s been underway in WebKit during the last several years is the complete rewrite of our inline layout engine. Safari 17.4 marks the completion of this project and the retirement of the twenty-one year-old legacy line layout engine.

Inline layout is integral to displaying content on the web. It determines the size and layout — wrapping, justification, spacing, and baseline alignment — of all text and other inline-level content. As developers, we often focus on the invisible block boxes on a web page and write CSS to layout those boxes using Flow, Tables, Flexbox or Grid. The content inside those boxes is placed using complex inline layout algorithms that developers often don’t need to think much about.

We’ve been shipping our new inline layout engine incrementally for many years. As more and more of the new engine was complete, more and more of the content on web pages was positioned by the new engine. This means users have been benefiting from WebKit’s new inline layout engine for a while. The legacy system was only triggered if there were something that hadn’t been yet implemented in the new engine. The last major step needed was the reimplementation of Ruby — and now it’s also a proper inline layout feature, fixing past inconsistencies.

Projects like these can be disruptive. Often browser teams will choose to not ship any new features while a multi-year rewrite project is in progress. WebKit instead chose to keep shipping new features, often implementing them twice — once in the legacy line layout engine, and again in the new inline layout engine. Now that this work is done, we no longer have to implement anything twice. This work also let us go through a large number of bugs reported on bugs.webkit.org, confirm they are no longer a problem, and close them as fixed.

We’re excited for WebKit’s future with this new engine. This investment results in increased interoperability by aligning to the latest web standards, fewer inline layout bugs, better performance, improvements to stability, and the ability to implement new features far more easily. The completion of inline layout also marks the beginning of rewriting the layout engine for other formatting contexts, starting with Flexbox.

Align content everywhere

When Flexbox shipped, it brought a powerful new tool to the web — box alignment. The align-content property made it possible to easily vertically center content inside a box! Or you could use it to align content to the bottom to the box, to align baselines of text, and more. When CSS Grid shipped, box alignment became possible in a second layout mode. Since 2017, you’ve been able to align the direct children of both Flexbox and Grid containers.

Now, we are proud to be the first browser shipping support for align-content inside two more formatting contexts — block layout and table layout. This means if all you want to do is align a box’s content in the block direction, you don’t need to involve Flexbox or Grid. Plus, you can now mix alignment with floats, and you can use it inside table cells.

div { align-content: center; } /* one-line vertical centering */

In addition, we updated the handling of align-content and justify-content on scroll containers in WebKit for Safari 17.4. Now, for example, you can use CSS to set the initial scroll position to the end rather than the start of the content.

div { overflow: auto; align-content: unsafe end; } /* end-aligned scroller */

Be sure to test alignment on scroll containers across browsers, as many are still in the process of updating to the specified behavior. Those that have not yet updated may clip content.

CSS Scoping

Websites today can be complex, with large teams working across multiple projects, relying on pattern libraries or frameworks to keep everything organized and consistent. It can become tough for large teams to handle how their CSS cascades. Tools like Cascade Layers and :has() have changed the game, allowing developers to apply styles more masterfully. Yet, developers often want a way to scope styles to the individual component they’re working on, without worrying about the big picture or preventing unintended consequences.

CSS Scoping was created to provide several more powerful options for organizing and structuring CSS. (Note, there have been many debates over many years on how style scoping might work. Search results for “CSS scoping” often yield old, unimplemented or completely different ideas.)

WebKit for Safari 17.4 adds supports the @scope rule and expands the capabilities of the :scope pseudo-class. Scoping changes how the cascade works in some surprising ways, so do be sure to read about its impact before deploying widely.

If your project is making heavy use of components, constructed independently and loaded in random order, scoping can help you by ensuring certain styles only apply to the contents of a specific element, and never to anything else on the page.

By default, all CSS on a project applies universally. It has a “scoping root” of <html>. And the :root pseudo-element refers to the root element in the DOM — the html element. CSS Scoping lets you use <style> @scope to reset the scoping root to a certain element, to the parent of the <style> element.

<article id="my-component">
  <style>
    @scope {
      h1 { font-size: 4rem; }
    }
  </style>
  <h1>This is 4rem text.</h1>
</article>

<h1>This will not be styled by the CSS above.</h1>

In this case, because <article> is the direct parent of <style> @scope, all of the styles defined inside @scope will only impact article and the content inside article. Nothing outside article is affected.

But that’s not all CSS Scoping can do. Let’s imagine we want to apply styles to a sidebar, but we don’t want those styles to apply to everything in the sidebar. We can use @scope to create a donut of sorts — with a hole in the middle where the styles don’t apply.

@scope (aside) to (section) {
  h2 {
    font-size: 3rem;
  }
}
<aside id="my-sidebar">
  <h2>This is 3rem text.</h2>
  <section>
    <h2>This is not styled by the CSS above.</h2>
  </section>
</aside>
You can try this demo in a browser with support.

By defining a scoping root with a scope-start selector (aside) and a scoping limit with a scope-end selector (section), we can effectively stop the cascading of the styles.

Also, anytime you use CSS Scoping, it radically changes what happens when there’s a tie in specificity.

Since the creation of CSS, when multiple selectors have equal specificity, the one that appears last in the CSS cascade is the one that gets applied. For example, if this is your CSS:

.blue h1 { color: blue; }
.yellow h1 { color: yellow; }

Then this is your result.

<section class="blue">
  <section class="yellow">
    <h1>This headline is yellow.</h1>
  </section>
</section>

<section class="yellow">
  <section class="blue">
    <h1>This headline is yellow.</h1>
  </section>
</section>

The headline is always yellow, because .yellow comes later in the CSS file. The order in the HTML does not matter.

But with scoping, the selector that applies to an element that’s closer in the DOM to the scoping root is the one that will apply when their specificities are tied.

Let’s use @scope instead of descendant selectors:

@scope (.blue) { 
  h1 { color: blue; }
}
@scope (.yellow) {
  h1 { color: yellow; }
}

Now, the headline color is determined by the DOM order in HTML, not the cascade order in CSS:

<section class="blue">
  <section class="yellow">
    <h1>This headline is yellow.</h1>
  </section>
</section>

<section class="yellow">
  <section class="blue">
    <h1>This headline is blue!</h1>
  </section>
</section>

The headline is yellow when .yellow is the closer ancestor, and it’s blue when .blue is the closer ancestor.

This is a fundamental change to how CSS works, so don’t get caught off guard. Use CSS Scoping with a lot of thought and care.

Note that a selector like .blue h1 { } has higher specificity than a selector like @scope (.yellow){ h1 { }}. The specificity of the scoping root’s selector is not added to the specificity of the selectors inside the @scope rule, unlike Nesting. And .blue h1 is higher specificity than h1.

WebKit for Safari 17.4 also expands the purpose of the :scope pseudo-class. When used inside a @scope block, :scope matches the block’s defined scope root. This provides a way to apply styles to the root of the scope from inside the @scope block itself. In the following example, :scope applies a border to the article element.

<article id="my-component">
  <style>
    @scope {
      :scope { border: 1px solid black; }    
      h1 { font-size: 4rem; }
    }
  </style>
  <h1>This is 4rem text.</h1>
</article>
You can try this demo in a browser with support.

White space and text wrap

For years, the white-space property in CSS has provided a mechanism for doing two things at once: 1) defining whether and how white space is collapsed, and 2) defining whether and how lines wrap. The CSS Working Group has since noted that this was likely a mistake, to handle two different qualities in one property. With the introduction of text-wrap, the CSSWG has rethought how the long and shorthand versions of these properties combine into an architecture that makes more sense and gives us needed flexibility.

Now the white-space property is a shorthand for two new longhand properties: white-space-collapse and text-wrap-mode, both added in WebKit for Safari 17.4. These longhands let you change the collapsing and wrapping modes independently, each without affecting the other.

The white-space-collapse property controls how white space is collapsed. By default, it’s set to collapse, causing strings of multiple spaces to become a single space. You can change the value instead to preserve in order to keep all the spaces, or use other the values: preserve-breaks, preserve-spaces, or break-spaces. These values all behave as they have for years with the white-space property.

The new text-wrap-mode property provides a mechanism for setting whether or not text should wrap. The wrap value turns it on, and the nowrap value turns it off.

This work sets the stage for the text-wrap shorthand and it’s longhands text-wrap-style and text-wrap-mode, some of which you can currently test in Safari Technology Preview.

Percentages in spacing

WebKit for Safari 17.4 adds support for percentages in letter-spacing and word-spacing. This lets you define spacing as a percentage of the element’s font-size — and keeps tracking the font-size even when it grows or shrinks on descendant elements.

Styling grammar and spelling errors

WebKit for Safari 17.4 adds support for the ::spelling-error and ::grammar-error pseudo-elements. These make it possible to create your own custom styling for text that is marked by the browser as misspelled or grammatically incorrect.

Alt text for generated content

The vast majority of content on the web is communicated through HTML, but CSS does have the ability to insert content into the page. Until now, sometimes this kind of content could not be made accessible. Now in WebKit for Safari 17.4, you can provide alternative text with accessible content fallback — content: "foo" / "alt-text";

For example, perhaps we want to prefix certain links with the little ⓘ icon to let users know this item leads to more detailed information. That symbol might be read by screenreader as “Circled Latin Small Letter I” or “Information source combining enclosing circle”, neither of which do a good job communicating the intended purpose. Perhaps a better experience would be to simply hear “Info:”.

.info::before {
  content: "ⓘ" / "Info:";
}

Previously, the -webkit-alt property served this function. It has been deprecated in favor of the new content alt text syntax. The new syntax is also more expressive as it allows for cascading, and allows you to chain multiple strings and attr() as alternative text.

Transitions

When CSS Transitions were created, they allowed authors to create a gradual timed transition between old and new values by interpolation. Sometimes, however, interpolation is not possible. For example, there’s no meaningful intermediary value between float: left and float: right, so, transitions ignored these properties. They simply jumped from the first state to the second immediately, without any ability to define when the jump should happen.

Yet, web developers have wanted a way to at least be able to define when the transition should happen for discrete properties. So the CSS Working Group figured out a way to make that possible. Now, you can tell the browser that you want an element to be capable of transitioning discrete property values, which lets you control their transition timing using the easing functions.

WebKit for Safari 17.4 adds support for the transition-behavior property. The transition-behavior: allow-discrete rule lets you enable transitions between discrete property values, so that you can control their timing via transition.

li {
  list-style: disc;
  color: blue;
  transition: all 2s, list-style 0.5s step-end;
  transition-behavior: allow-discrete;
}
li:hover {
  list-style: square;
  color: red;
}
Try this demo code in a browser with support. Toggle transition-behavior off to see the difference.

:has()

The :has() pseudo-class provides tremendous value. We keep making it more and more powerful by adding support for additional pseudo-classes within :has(). WebKit for Safari 17.4 adds support for :has(:any-link), :has(:link), and :has(:-webkit-any-link), making it possible to select an element depending on whether or not it contains a link.

And more

WebKit for Safari 17.4 adds support for CSS custom properties to the ::backdrop pseudo-element, allowing variables to be applied to the backdrop behind dialog elements and other top layer items.

WebKit for Safari 17.4 also adds offset-position support for circle() and ellipse().

And WebKit for Safari 17.4 makes -apple- prefixed pseudo-elements no longer valid.

Web API

This release of Safari adds support for an assortment of small Web API additions that give you extra tools in your developer toolkit.

With support for the element.checkVisibility() method, you can determine the visibility of an element across a variety of conditions including how CSS properties such as display, visibility, and opacity are applied.

WebKit for Safari 17.4 also extends its Declarative Shadow Root support. The Element.prototype.setHTMLUnsafe(), ShadowRoot.prototype.setHTMLUnsafe(), and Document.parseHTMLUnsafe() methods, as well as the ShadowRoot clonable property are now available. The setHTMLUnsafe() methods work similar to setting an element’s innerHTML property, enabling unsanitized DOM tree mutation but with additional support for declarative shadow roots. The parseHTMLUnsafe() method similarly parses unsanitized HTML with declarative shadow root support and returns a document. And the clonable read-only boolean property allows you to detect if a ShadowRoot is clonable.

WebKit for Safari 17.4 adds support for the CustomStateSet interface for custom element state management. This interface includes methods to add(), delete(), or detect if the element has() a given state, and more. Importantly, these states added to a custom element can be styled using the :state() pseudo-class by users of the custom element.

The DOMMatrixReadOnly interface now supports the scaleNonUniform() method that creates a new DOMMatrix scaling on X, Y, and Z axes. The X axis scaling factor must be specified, but the Y and Z axes default to 1. The scaling is centered at the given origin that defaults to (0, 0, 0).

Lastly, WebKit for Safari 17.4 adds support for AbortSignal.any() giving you a convenient way to combine abort signals such as user input (e.g. a user clicks a cancel button) and a timeout to send an abort signal to an async operation.

JavaScript

New JavaScript features in Safari 17.4 add new expressiveness and convenience with promise resolvers, improved internationalization formatting, ArrayBuffer ownership management, and Array grouping features.

WebKit for Safari 17.4 adds support for the Promise.withResolvers static method. It allows developers the convenience of creating a promise and configure the resolution and rejection handlers after it has been created. The method returns the promise along with the resolution and rejection functions.

const { promise, resolve, reject } = Promise.withResolvers();

The TimeZoneOffset format is now available for Intl.DateTimeFormat. It allows you to specify the difference of the local time to UTC time in positive or negative hours and minutes depending on whether the local time is ahead or behind.

new Intl.DateTimeFormat("en-US", {
    dateStyle: 'long',
    timeStyle: 'long',
    timeZone: '-0800'
}).format(new Date())

Additionally, Number.prototype.toLocaleString and Intl.NumberFormat have been updated so the string representation correctly aligns with recent specification changes.

There’s also new expressive API for managing the concept of ownership for ArrayBuffers. ArrayBuffer.prototype.transfer creates a new ArrayBuffer with the same contents and properties as the target ArrayBuffer (such as being resizable) and detaches it from the original ArrayBuffer. You can use ArrayBuffer.prototype.transferToFixedLength() to guarantee a non-resizable ArrayBuffer with the same content as the buffer. ArrayBuffer.prototype.detached will tell you if the buffer has been transferred and is detached.

WebKit for Safari 17.4 also adds the Array grouping feature that includes Object.groupBy and Map.groupBy methods. These methods give you powerfully simple tools for grouping datasets.

const todos = [
    { task: "Water the flowers", context: "home", estimate: "5 minutes" },
    { task: "Get the TPS report done", context: "work", estimate: "45 minutes" },
    { task: "Find new insurance", context: "home", estimate: "180 minutes" },
    { task: "Fix a website bug", context: "work", estimate: "25 minutes" },
    { task: "Answer emails", context: "anywhere", estimate: "10 minutes" }
];

let contextual_tasks = Object.groupBy(todos, ({ context }) => context);
console.log(contextual_tasks);

let tasks_by_time = Map.groupBy(todos, ({ estimate }) => {
    return parseInt(estimate.split(' ')[0]) < 15 ? "short" : "long";
});
console.log(tasks_by_time);

Media

Additional codecs

WebKit for Safari 17.4 adds support for several audio and video codecs.

First, WebKit for Safari 17.4 on iOS, iPadOS and in visionOS adds support for WebM. While the WebM container (with both the VP8 and VP9 video codecs) has been fully supported on macOS since Safari 14.1, support on iOS and iPadOS was limited to VP8 in WebRTC. Now, WebM is fully supported everywhere.

The Vorbis audio codec is also now supported in WebKit on iOS 17.4, iPadOS 17.4 and in visionOS 1.1.

And WebKit for Safari 17.4 expands what WebCodecs can do with the addition of support for the HEVC codec.

Source prioritization

When support for video embedding arrived in HTML5 with the <video> and <source> elements, the web standard specified that the first file that’s recognized by the browser should be chosen and played. This put the burden on the developer to make sure the best files were listed before lesser-quality files.

<video>
   <source src="movie.webm">
   <source src="movie.av1">
   <source src="movie.mov">
</video>
Are you sure the first format listed is always a better choice than the rest?

This made sense in a simpler time, when there were just a few codecs available. Now, there are many codecs with different qualities. It’s not always possible for developers to know which file is the best one for a user to stream. And it can be impossible to put them in one specific order that’s best for all users.

A browser might easily be capable of playing several of the files offered, but one of those files could be compressed with a codec that the user’s device can decode using hardware, while the rest might only be decoded by software alone.

It’s definitely a better user experience to use hardware decoding. Doing so significantly impacts power usage and makes a battery last longer. So now, in WebKit for Safari 17.4, the best file for the user is chosen, instead of defaulting to the first file that has support. Video codecs with hardware decoding support on various Apple devices include VP9, h.264, HEVC and AV1.

WebVTT

WebKit for Safari 17.4 adds support for HTML character entities to WebVTT (Web Video Text Tracks Format), the technology used to add subtitles and captions to video files on the web. HTML entities are a way to write special characters without having the browser mistakenly think they are part of the HTML code. For example, &middot; represents the “·” character.

MediaStream

WebKit for Safari 17.4 adds support whiteBalanceMode to MediaStream. In photography, adjusting white balance is a technique for compensating for the fact that “white” is a different color under different lighting conditions. Sunlight is very blue, while indoor lights tend to be quite orange. Our brains automatically adjust, so as humans, we rarely notice. But cameras need technology to help them adjust color temperature so that the resulting photo or video has the kind of coloring people expect. Now modes for white balance are available for the MediaStream Image Capture API on the web.

SVG

WebKit for Safari 17.4 adds support for kernelUnitLengthX and kernelUnitLengthY to SVGFESpecularLightingElement.

WebGL

WebKit for Safari 17.4 adds support for four new WebGL extensions: EXT_clip_control, EXT_depth_clamp, EXT_polygon_offset_clamp, and WEBGL_polygon_mode.

Web Assembly

WebKit for Safari 17.4 enables extended constant expressions to support more advanced WebAssembly linking.

Web Inspector

Web Inspector for Safari 17.4 has two new features. First, when a page attempts to load a font URL blocked by Lockdown Mode, a message is logged to the Console.

Second, Web Inspector now groups load errors for source maps. Source map files are used for converting a combined or minified file back into its original state. Grouping load errors helps reduce noise while debugging. You can disable this behavior in Web Inspector Settings under Experimental settings.

Changes to Safari

Safari 17.4 itself includes three changes to the UI and user experience. First, you can now configure the Favorites Bar to show your bookmarks with only their icons. Edit the name of the bookmark in the favorites bar, and remove the name. The icon will remain.

Second, Safari 17.4 now supports webpage translation inside <iframe> elements.

And third, Safari 17.4 adds support for Apple Cash virtual card numbers and showing the user their Apple Cash balance when using AutoFill.

Safari Extensions

Safari 17.4 includes a change to web extensions that allows extensions to open Private Browsing windows even when they don’t have access to Private Browsing.

Web Authentication

WebKit for Safari 17.4 adds support for WebAuthn’s PublicKeyCredentials.getClientCapabilities() function. Use it to find out which WebAuthn features are supported. It returns a Promise of a record<DOMString, boolean> containing capabilities and their values.

Bug Fixes and more

In addition to all the new features, WebKit for Safari 17.4 includes work polishing existing features.

Accessibility

  • Fixed exposing the correct <summary> element role. (13661104)
  • Fixed non-accessible content within iframes with ARIA roles. (104611075)
  • Fixed VoiceOver word echo on text inputs with a combobox role. (112488137)
  • Fixed an issue where innerHTML and innerText changes to labels did not update their corresponding input element’s accessibility title. (113872525)
  • Fixed <details> and <summary> elements not included in VoiceOver form controls menu or list. (117308226)
  • Fixed comboboxes not notifying assistive technologies when aria-activedescendant changes. (117747058)
  • Fixed toggling accessibility preferences to correctly update form control appearance. (117914468)
  • Fixed: Removed the default ARIA-level heading for a heading role, matching removal from ARIA specifications. (119059172)
  • Fixed text missing from accessibility labels for many common shadow DOM scenarios. (120223342)

Browser Changes

  • Fixed loading a ⌘Click fragment link in a background tab. (119079650)

CSS

  • Fixed the default link color contrast for the dark color scheme. (61149466)
  • Fixed getComputedStyle() for invalid pseudo-elements. (98504661)
  • Fixed querySelector() to not throw an exception for -webkit- prefixed pseudo-elements. (99299129)
  • Fixed :user-invalid triggering while typing a date. (110687369)
  • Fixed: Updated text-transform: full-size-kana to align with Unicode 15. (111508663)
  • Fixed contain: inline-size breaking grid-template-rows: auto. (113915953)
  • Fixed svh and dvh units being unexpectedly equal when the Safari tab bar is not visible. (115085360)
  • Fixed mixed-blend-mode to blend correctly against the root background. (115688282)
  • Fixed backdrop-filter with many interoperability improvements. (115703346)
  • Fixed oklab and oklch lightness value clamping. (116195533)
  • Fixed flex layout invalidation in cases where the content of a flex item changes or style changes impact the preferred widths computation of its items. (117181858)
  • Fixed selection gaps to get painted with the expected ::selection pseudo-element color. (117796745)
  • Fixed parsing and serialization of -webkit- prefixed pseudo-elements. (118081134)
  • Fixed ::backdrop to be allowed after ::slotted(). (119015204)
  • Fixed to allow :checked and :indeterminate to match at the same time. (119075969)
  • Fixed grid with size containment and min-height not sizing row correctly. (119736473)
  • Fixed computing values of basic shape rect() and xywh() as the equivalent inset(). (119739406)
  • Fixed poor performance with :has(+ :not(.class)) pseudo-class selector. (119819247)
  • Fixed CSS content computed value serialization. (120061551)
  • Fixed pseudo-element parsing in getComputedStyle() and KeyframeEffect.prototype.pseudoElement so they require them starting with :: (or : for 4 legacy pseudo-elements). (120170550)
  • Fixed CSS linear() easing. (120290721)
  • Fixed named at-rule container getting skipped when the container is named in a :host selector. (120428386)
  • Fixed :not(:has(:not(foo))) getting misclassified as scope breaking. (120492012)
  • Fixed the name for a ::slotted pseudo-element in a container query getting resolved against the wrong scope. (122224135)
  • Made -apple- prefixed pseudo-elements no longer valid. (120268884)

Forms

  • Fixed <select> not refreshing the dropdown after an <option> is removed on iPad. (88292987)
  • Fixed text-indent to affect the selected file(s) label for file inputs. (105223868)
  • Fixed dir=auto to work for hidden, password, submit, reset, and button input types, made dirname work for password and submit input types, and removed dirname support from number input types. (113127508)
  • Fixed serialization of autocomplete with a webauthn token. (116107937)
  • Fixed <option> elements outside of an <optgroup> getting added to the preceding group. (117930480)

Fullscreen

  • Fixed viewport units to be correct after entering and exiting fullscreen mode on iOS, iPadOS, and in visionOS. (120496571)

HTML

  • Fixed the system-ui font family within <canvas>. (117231545)
  • Fixed <progress> to use the page’s preferred rendering update interval. (118976548)
  • Fixed missing support for the direction attribute in the list of attributes whose values are matched case-insensitively with attribute selectors. ( (119432066)

JavaScript

  • Fixed stringification algorithm of the Function constructor to match specifications. (102065151)
  • Fixed block-level function declarations to have correct scope in global code and aligned the detection of hoistable block-level legacy function declarations with the spec. (113880075)
  • Fixed an edge case with detecting a semantic error in generators. (117497786)
  • Fixed Temporal API to throw TypeErrors for unexpected primitives. (117992134)
  • Fixed Temporal options handling to align with the specification. (118088676)
  • Fixed Temporal.Now.timeZone() to be updated to timeZoneId(). (118674314)

Loading

  • Fixed Link-stylesheet elements to not fire load events for non-text/css and non-2XX responses. (116112223)
  • Fixed link-stylesheet elements to not fire load events for non-2XX responses such as 3XX responses that do not redirect. (116331826)

Lockdown Mode

  • Fixed Lockdown Mode disabling on sites with COOP and COEP HTTP headers. (119503109)

Media

  • Fixed WebVTT regions to position according to specifications. (23091897)
  • Fixed pausing MediaRecorder continuing to call ondataavailable at every timeslice event. (115979604)
  • Fixed an HEVC decoder issue when translating annexb data. (116768196)
  • Fixed WebVTT to treat negative percentages as invalid values. (117615681)
  • Fixed object-fit: fill on <video> elements. (118020922)
  • Fixed WebRTC calls not unmuting automatically after using Siri sometimes losing incoming audio. (118461093)
  • Fixed white bars across the top and bottom of fullscreen video playback while using Light Mode. (118530255)
  • Fixed the always empty video.buffered attribute. (118550061)
  • Fixed WebVTT to correctly parse region id settings. (118551267)
  • Fixed VideoEncoder produces no frames with latencyMode “realtime” when framerate/bitrate are not given. (118725549)
  • Fixed AV1-in-MP4 codec string not shown in Show Media Stats. (118850797)
  • Fixed getDisplayMedia frameRate always at 30 regardless of constraints. (118874132)
  • Fixed returning to fullscreen from picture-in-picture breaking subsequent touch input. (119832557)
  • Fixed HLS video captions where there are multiple text tracks available. ( (119839950)
  • Fixed fullscreen video not scaling to display size when the Safari window is in Full Screen App Mode. (119893556)
  • Fixed handling key renewal requests that cause playback errors for some DRM content. (120230860)
  • Fixed camera and mic activation failure due to media capability granting and activation order. (120510826)
  • Fixed paint-on captions shifting during playback. (120847946)
  • Fixed videos shifting up and down when fullscreen overlay controls appear or disappear. (120848395)
  • Fixed volume slider flickering when adjusting volume in Safari in visionOS. (120855936)
  • Fixed blocked encrypted sampled not getting enqueued after a CDM is attached to a SourceBuffer. (120879185)
  • Fixed video playback on Twitter.com in Safari in visionOS. (121391975)
  • Fixed[Netflix.com content that can become zoomed-in and cropped when in fullscreen mode. (121822831)
  • Fixed pseudo-element font size calculation to fix subtitle size in fullscreen mode. (122584350)

Rendering

  • Fixed incorrectly oriented Traditional Mongolian script characters.(93426525)
  • Fixed resizing behavior with writing-mode: vertical-rl or direction: rtl. (102620110)
  • Fixed opacity and rendering the root element background image. (115396444)
  • Fixed the color of the drop shadow to preserve its alpha channel. (115812347)
  • Fixed filters with outsets to repaint the entire filterRegion if GraphicsStyles are used. (115817290)
  • Fixed compositing the filter style transparency layers to not clip the destination context. (115901634)
  • Fixed a bug where the returned transform from getComputedStyle was incorrect. (117523629)
  • Fixed handling images with color spaces not supported by the backend to fallback to render in sRGB. (118238178)
  • Fixed check boxes and radio buttons to avoid floats. (118660695)
  • Fixed rendering for a <div> within a transformed parent <div> with overflow: hidden. (118901069)
  • Fixed rendering issues when editing text. (119833765)
  • Fixed offsetHeight and offsetWidth are 0 for an inline box wrapping a block. (119955792)
  • Fixed a floating element causing a list item bullet to be orphaned on constrained lines. (120022893)
  • Fixed incorrect inline box (hugging) outline painting in vertical writing modes. (120217559)
  • Fixed incorrect ch unit value in vertical-rl and vertical-lr when text-orientation is not upright. (120293590)
  • Fixed graphics artifacts when scrolling a Heroku app. (120373474)
  • Fixed overflow: hidden to not prevent CSS Subgrid from applying. (120848131)
  • Fixed the repaint area for underline text decorations. (121082290)
  • Fixed align-content and justify-content on scroll containers causing overflowing content to become inaccessible. (121366949)
  • Fixed rendering floats and an out-of-flow <br> element with clear. (121444267)
  • Fixed a line break at gaps between two inline elements in a container with white-space: nowrap. (121859917)
  • Fixed cropped first letter for custom fonts that report negative advance width. (121891210)
  • Removed margin-trim behavior for floats to match specification changes. (115794102)

Safari Extensions

  • Fixed sending an error back to the caller if an error occurs for scripting.executeScript(). (107996753)
  • Fixed an issue where scripts may not be removed after calling scripting.unregisterContentScripts(). (113171510)

Scrolling

  • Fixed unusable horizontal scrollbars for right-to-left, vertical-rl, or flexbox reverse mode elements. (104944522)
  • Fixed a scrollTo() followed by an animated scroll ending at the wrong scroll position. (117608836)
  • Fixed wheel overflow behavior with Shadow DOM elements. (118496293)
  • Fixed keyboard scrolling beyond the page getting stuck at a bad scroll offset. (120053910)

Storage

  • Fixed cases where website data is unexpectedly evicted. (119818267)

SVG

  • Fixed applying rx or ry exclusively via CSS having no effect. (113500023)
  • Fixed negative SVGTransform scale values to be correctly stringified. (118656892)
  • Fixed the layout of an SVG when it is inside an <iframe> without affecting the size of the <iframe>. (120178866)
  • Removed support for SVGRenderingIntent. (102516681)

URLs

  • Fixed CSS invoked URL parsing to always use UTF-8 as agreed by the W3C CSS WG. (114889625)

Web Animations

  • Fixed style invalidation for animations. (118500247)
  • Fixed a paused animation where currentTime is changed to 0 not restarting when unpaused. (118826588)

Web API

  • Fixed invalid coordinates on wheel and gesturechange events inside an iframe. (105243167)
  • Fixed HTMLAreaElement to align with the HTML Standard. (110028213)
  • Fixed the result of Range.getClientRects() and Range.getBoundingRect() for certain ranges. (112543805)
  • Fixed Scroll To Text Fragment to not scroll after dynamic stylesheet loads and the user has scrolled. (112608578)
  • Fixed SharedWorker referrer policy to default to its context referrer policy if none is provided in its script http response. (114625126)
  • Fixed URL encoding for Request‘s referrer feature and Response.redirect(). They now always use UTF-8. (115219660)
  • Fixed reprocessing <meta name="color-scheme"> when their name or content attribute changes. (115958450)
  • Fixed FetchResponse.formData() to parse headers names as case insensitive. (116742000)
  • Fixed declarative shadow trees to match the latest specifications. (117655691)
  • Fixed jiggling caused by repeated calls to scrollIntoView({ block: 'center' }). (117755250)
  • Fixed fullscreen warning banner to prevent cutting off long domain names. (118078137)
  • Fixed updating resizeBy and resizeTo to use int rather than float to align with specifications. (118872048)
  • Fixed the CookieChangeEvent to not be exposed when the Cookie Store API is disabled. (118902989)
  • Fixed Element.prototype.setAttributeNode() to not treat attribute names case insensitively. (119013600)
  • Fixed toggling the spellcheck attribute not toggling spelling markers on input elements. (119269616)
  • Fixed removing highlights in the Custom Highlights API. (119531671)
  • Fixed getElementsByName() to only return HTML elements, not SVG, MathML, or other types of elements. (120275680)
  • Fixed the button value for a pointerup event not matching the pointerdown event. (120429508)
  • Fixed a wheel event to fire on an element that has been re-inserted after document.open. (120893136)
  • Fixed Scroll To Text Fragment Text Directives to find text with additional unrendered white space in their node data. (120913588)
  • Fixed changing HTMLCanvasElement width or height causing intermediate buffer allocations. (122309325)
  • Fixed canvas captureStream stuttering with WebGL. ((122471664)

Web Inspector

  • Fixed Home Screen Web Apps in Simulator to be listed under a “Home Screen Web Apps” section in the device submenu of the Develop menu. (117742935)
  • Fixed the tan() function to not trigger the color picker. (118724061)

WebGL

  • Fixed Canvas WebGL context capture to WebCodecsVideoFrame not capturing all frames. (108459224)
  • Fixed: Improved performance of MSAA rendering, including antialiased default framebuffer and fixed PBO uploads of PVRTC1 textures. (117461678)
  • Fixed WebGL OffscreenCanvas returning the previously created WebGL1 context when asking for WebGL2. (119028794)
  • Fixed WebGL to be available in nested workers. (120279728)

WebKit

  • Fixed HTML content not displaying in a Simulator, affecting projects using the web extension project template. (121338366)

WebRTC

  • Fixed media tracks obtained with {"width":1920,"height":1080,"frameRate":24}. (61747755)
  • Fixed triggering resolution scaling in the case of WebRTC maintain-framerate degradationPreference. (121041723)
  • Fixed a bug that prevented HTML canvas elements from always being marked dirty on initialization. This could cause some video effects to have choppy animations. (121257960)

Updating to Safari 17.4

Safari 17.4 is available on iOS 17.4, iPadOS 17.4, macOS Sonoma 14.4, macOS Ventura, macOS Monterey and in visionOS 1.1.

If you are running macOS Ventura or macOS Monterey, you can update Safari by itself, without updating macOS. On macOS Ventura, go to  > System Settings > General > Software Update and click “More info…” under Updates Available.

To get the latest version of Safari on iPhone, iPad, or Apple Vision Pro, go to Settings > General > Software Update, and tap to update.

Feedback

We love hearing from you. To share your thoughts on Safari 17.4, find us on Mastodon at @jensimmons@front-end.social and @jondavis@mastodon.social. Or send a reply on X to @webkit. You can also follow WebKit on LinkedIn. If you run into any issues, we welcome your feedback on Safari UI, or your WebKit bug report about web technologies or Web Inspector. Filing issues really does make a difference.

Download the latest Safari Technology Preview on macOS to stay at the forefront of the web platform and to use the latest Web Inspector features.

You can also find this information in the Safari 17.4 release notes.

March 05, 2024 06:30 PM

February 28, 2024

An HTML Switch Control

Surfin’ Safari

We’re very excited to introduce a new HTML form control as part of Safari 17.4: a switch.

Two switches as seen on iOS, with the second switch using a custom CSS accent-color.

The HTML for this example looks roughly like this:

<style> .special { accent-color: papayawhip } </style>
<input type=checkbox switch checked>
<input type=checkbox switch checked class=special>

Switches are a popular control on mobile platforms as well as in a large variety of UI frameworks, but until now they were not built into the web platform. Seeing the widespread need for this control we decided to change that.

The way we approached this is consistent with how other new form controls have been added to HTML:

  • We wanted it to have the look and feel of the OS control by default. It should match the end user’s preferences, and they should be able to manipulate the “thumb” of the switch in various ways.
  • We wanted to ensure it was backward compatible. A browser that does not support <input type=url> will treat it as <input type=text>. Likewise, a browser that does not support <input type=checkbox switch> will treat it as <input type=checkbox>.
  • We wanted the markup and API to be familiar. It mirrors that of checkboxes, except that the :indeterminate pseudo-class never matches.
  • We wanted the control to be accessible. Under the hood, it uses the ARIA switch role, announcing the states “On” and “Off” just like a switch from the OS. On macOS, if you enable “Differentiate Without Color”, or on iOS, if you enable “On/Off Labels”, the switch will show accessibility indicators.
  • We wanted the control to be stylable. As with <input type=checkbox>, if you use appearance: none, you get full control over its appearance as a web developer. It goes even a bit further as we ensured all properties will have their initial values when you use appearance: none. (See also the experimental ::thumb and ::track pseudo-elements explained below for planned enhancements in this area.) Control over styling is more limited with appearance: auto (the default). The accent-color property is supported and will be applied to the switch’s “track” background, as demonstrated in the image above.
  • We wanted the control to support a wide variety of languages, and as such, it has vertical rendering support out of the box. (This is in line with how we recently added vertical rendering support to the existing form controls.)

We’d love to see what you do with it and what else you would like this control to do.

Switch versus checkbox

Generally, we recommend using a switch when the end user understands the user interface element as a setting that is either “on” or “off”. A checkbox is well suited for when the end user would understand the element as something to be selected.

Experimental ::thumb and ::track pseudo-elements

In Safari if you go to Settings → Feature Flags, you can enable “::thumb and ::track pseudo-elements”, which gives web developers more control over the individual parts that make up a switch: its thumb and track. Essentially, this turns a single element into three, one parent with two sibling children.

We decided not to ship these pseudo-elements for now to give the standardization process additional time to finalize how they should work. We also want to make sure that when we ship them, they work for the other HTML controls they are designed for, such as <input type=range>. This way @supports(::thumb) will be all the feature detection you need.

Interactive demos of switch

Feedback appreciated

We’d love to hear what you make of this new HTML form control. Send a tweet to @webkit to share your thoughts on this feature. You can find us on Mastodon at @jensimmons@front-end.social and @jondavis@mastodon.social. You can also follow WebKit on LinkedIn. If you run into any issues, we welcome your WebKit bug reports on WebKit features like this. Reporting issues makes an enormous difference.

You can also download the latest Safari Technology Preview to try out new web platform features like this before they appear in a Safari beta.

And finally, a big thank you to Lily Spiniolas for doing a lot of work on this feature during her internship at Apple. Not just prototyping an implementation in WebKit, but also creating a pull request for the HTML Standard and carefully thinking through many aspects of its design.

February 28, 2024 07:47 PM

February 22, 2024

Release Notes for Safari Technology Preview 189

Surfin’ Safari

Safari Technology Preview Release 189 is now available for download for macOS Sonoma and macOS Ventura. If you already have Safari Technology Preview installed, you can update it in System Settings under General → Software Update.

This release includes WebKit changes between: 273602@main…274066@main.

CSS

New Features

  • Added support for light-dark() function for color values. (273634@main) (117033939)
  • Added support for @starting-style. (273740@main) (121918611)

Resolved Issues

  • Fixed named at-rule container getting skipped when the container is named in a :host selector. (273987@main) (120428386)
  • Fixed the name for a ::slotted pseudo-element in a container query getting resolved against the wrong scope. (274050@main) (122224135)

Deprecations

  • Removed :-webkit-full-screen-ancestor pseudo-class. (273620@main) (100782937)
  • Removed :-webkit-full-screen-controls-hidden pseudo-class. (273619@main) (121323330)
  • Removed :-webkit-full-page-media pseudo-class. (273618@main) (121752962)
  • Removed :-webkit-full-screen-document pseudo-class. (273639@main) (121816310)

HTML

Resolved Issues

  • Fixed parsing a self-closing SVG script element. It now successfully executes. (273697@main) (121887875)
  • Fixed “about:blank” document.referrer initialization. (273830@main) (97689906)

JavaScript

Resolved Issues

  • Fixed throwing a RangeError if Set methods are called on an object with negative size property. (274009@main) (121310940)
  • Fixed eval() function from another realm to not cause a direct eval call. (273782@main) (121546048)
  • Fixed eval() call with ...spread syntaxt to be a direct call. (273788@main) (121547890)
  • Fixed programming style for bitwise and in setExpectionPorts. (273960@main) (122138733)

Rendering

Resolved Issues

  • Fixed Greek uppercase transforms failing for some characters. (274036@main) (90364897)
  • Fixed lingering boxes with content-visibility: hidden. (273602@main) (117916396)
  • Fixed parts of the content disappear when interacting with overflow:scroll, z-index and positioning (including Heroku apps). (273999@main) (120373474)
  • Fixed align-content and justify-content on scroll containers causing overflowing content to become inaccessible. (273737@main) (121366949)
  • Fixed line break at gaps between two inline elements in a container with white-space: nowrap. (121859917)
  • Fixed a floating element causing the latter half of a hyphenated word to disappear. (273836@main) (121889487)
  • Fixed cropped first letter for custom fonts that report negative advance width. (121891210)

SVG

Resolved Issues

  • Fixed SVG title to have display: none as the default UA style rule. (273991@main) (122185838)

Web API

Resolved Issues

  • Fixed history.pushState() and history.replaceState() to ignore the title argument. (273650@main) (75695791)
  • Fixed lang attribute in no namespace to only apply to HTML and SVG elements. (273726@main) (117795695)
  • Fixed spelling of clonable. (274063@main) (121516711)

Deprecations

  • Removed which from KeyboardEvent. (273701@main) (106580687)

Web Inspector

New Features

  • Added a Media details panel to Web Inspector when selecting a <video> or <audio> element in the Elements tab. (273777@main) (118865793)

Resolved Issues

  • Fixed font sizes in the Audits tab. (273615@main) (76162927)

WebAuthn

Resolved Issues

  • Fixed navigator.credentials.create() rejects with “NotAllowedError: Operation Failed” after a conditional UI request is aborted. (273918@main) (109936742)

WebGL

New Features

  • Enabled support for EXT_texture_mirror_clamp_to_edge, WEBGL_render_shared_exponent, and WEBGL_stencil_texturing. (273645@main) (121835897)

WebRTC

New Features

  • Added support for missing WebRTC stats. (273643@main) (121594743)

Resolved Issues

  • Fixed a bug that prevented HTML canvas elements from always being marked dirty on initialization. This could cause some video effects to have choppy animations. (273897@main) (121257960)
  • Fixed VideoTrackGenerator writer to close when its generator track (and all its clones) are stopped. (273778@main) (121835553)

February 22, 2024 01:10 AM

February 20, 2024

Carlos García Campos: A Clarification About WebKit Switching to Skia

Igalia WebKit

In the previous post I talked about the plans of the WebKit ports currently using Cairo to switch to Skia for 2D rendering. Apple ports don’t use Cairo, so they won’t be switching to Skia. I understand the post title was confusing, I’m sorry about that. The original post has been updated for clarity.

By carlos garcia campos at February 20, 2024 06:11 PM

February 19, 2024

Carlos García Campos: WebKitGTK and WPEWebKit Switching to Skia for 2D Graphics Rendering

Igalia WebKit

In recent years we have had an ongoing effort to improve graphics performance of the WebKit GTK and WPE ports. As a result of this we shipped features like threaded rendering, the DMA-BUF renderer, or proper vertical retrace synchronization (VSync). While these improvements have helped keep WebKit competitive, and even perform better than other engines in some scenarios, it has been clear for a while that we were reaching the limits of what can be achieved with a CPU based 2D renderer.

There was an attempt at making Cairo support GPU rendering, which did not work particularly well due to the library being designed around stateful operation based upon the PostScript model—resulting in a convenient and familiar API, great output quality, but hard to retarget and with some particularly slow corner cases. Meanwhile, other web engines have moved more work to the GPU, including 2D rendering, where many operations are considerably faster.

We checked all the available 2D rendering libraries we could find, but none of them met all our requirements, so we decided to try writing our own library. At the beginning it worked really well, with impressive results in performance even compared to other GPU based alternatives. However, it proved challenging to find the right balance between performance and rendering quality, so we decided to try other alternatives before continuing with its development. Our next option had always been Skia. The main reason why we didn’t choose Skia from the beginning was that it didn’t provide a public library with API stability that distros can package and we can use like most of our dependencies. It still wasn’t what we wanted, but now we have more experience in WebKit maintaining third party dependencies inside the source tree like ANGLE and libwebrtc, so it was no longer a blocker either.

In December 2023 we made the decision of giving Skia a try internally and see if it would be worth the effort of maintaining the project as a third party module inside WebKit. In just one month we had implemented enough features to be able to run all MotionMark tests. The results in the desktop were quite impressive, getting double the score of MotionMark global result. We still had to do more tests in embedded devices which are the actual target of WPE, but it was clear that, at least in the desktop, with this very initial implementation that was not even optimized (we kept our current architecture that is optimized for CPU rendering) we got much better results. We decided that Skia was the option, so we continued working on it and doing more tests in embedded devices. In the boards that we tried we also got better results than CPU rendering, but the difference was not so big, which means that with less powerful GPUs and with our current architecture designed for CPU rendering we were not that far from CPU rendering. That’s the reason why we managed to keep WPE competitive in embeeded devices, but Skia will not only bring performance improvements, it will also simplify the code and will allow us to implement new features . So, we had enough data already to make the final decision of going with Skia.

In February 2024 we reached a point in which our Skia internal branch was in an “upstreamable” state, so there was no reason to continue working privately. We met with several teams from Google, Sony, Apple and Red Hat to discuss with them about our intention to switch from Cairo to Skia, upstreaming what we had as soon as possible. We got really positive feedback from all of them, so we sent an email to the WebKit developers mailing list to make it public. And again we only got positive feedback, so we started to prepare the patches to import Skia into WebKit, add the CMake integration and the initial Skia implementation for the WPE port that already landed in main.

We will continue working on the Skia implementation in upstream WebKit, and we also have plans to change our architecture to better support the GPU rendering case in a more efficient way. We don’t have a deadline, it will be ready when we have implemented everything currently supported by Cairo, we don’t plan to switch with regressions. We are focused on the WPE port for now, but at some point we will start working on GTK too and other ports using cairo will eventually start getting Skia support as well.

By carlos garcia campos at February 19, 2024 01:27 PM

February 16, 2024

How to use Media Source Extensions with AirPlay

Surfin’ Safari

Media Source Extensions (MSE) is a popular way to provide streaming video on the web. It gives JavaScript control of the way bytes are sent to the browser for playback. At the 2023 Worldwide Developer conference, Apple announced a new Managed Media Source API that improves on MSE with efficient video streaming and longer battery life for iOS and other devices.

However, MMS and MSE, by nature, are not compatible with AirPlay, which requires a unique playback URL. AirPlay allows you to start playback of your favorite videos on your phone, move them to your TV and then switch off that phone. An AirPlay-compatible URL can be of any format such as an mp4, mpeg-ts, or HTTP Live Streaming (HLS).

This post will guide you through providing both sources and, in the process, build out a demo example.

WebKit MSE + AirPlay Demo

Since MMS/MSE uses binary blobs appended to a SourceBuffer it won’t work with AirPlay. But, if you create an alternative source that can be served as an AirPlay-compatible URL, there is a way to get them to work together.

When it comes to an AirPlay-compatible alternative, HLS is an option that offers a great deal of efficiency for users. There are numerous resources that can guide you in converting your video content to serve it with HLS. Apple offers a toolkit you can use and there are many other options as well.

const airplayURL = './video/demo.m3u8';
const videoURL = './video/demo.mp4';
const mediaType = 'video/mp4; codecs="avc1.640028"';

// Create a video element
const video = document.createElement('video');

// Set video element properties for the demo
video.controls = true;
video.loop = true;
video.muted = true;
video.autoplay = true;

In setting up the MediaSource, it’s easy to use feature detection to use Managed Media Source API to offer power-efficient streaming on browsers that support it and gracefully fallback to MSE where its not available:

// Feature detect MMS and fallback to MSE
const MediaSource = self.ManagedMediaSource || self.MediaSource;
const source = new MediaSource();

We also need a way to offer both our Media Source and AirPlay sources at the same time. The HTML video element’s ability to define multiple sources will do just that. It was originally intended to allow a user-agent to choose the best format of the video to be played and fallback should it not be supported.

<video>
  <source src="format/video.m3u8" type="application/x-mpegURL">
  <source src="format/video.ogg" type="video/ogg">
  <source src="format/video.mp4" type="video/mp4">
  <source src="format/video.webm" type="video/webm">
</video>

The browser will look over the list of available formats from top to bottom. If no matches are found, or if decoding fails along the way, it will select the next element in the list.

We can make use of this behavior, combining the ability for the user-agent to choose the best format and allowing AirPlay to select a playable source. You’ll create a URL from your Media Source and add it to a video element as the first source. This URL is local to the user-agent and can’t be shared, as it has no meaning outside the local context. Then you add the AirPlay-compatible URL as the second source.

// Add MSE/MMS streaming source
const videoSource1 = document.createElement('source');
videoSource1.type = 'video/mp4';
videoSource1.src = URL.createObjectURL(source); // Create URL from MediaSource
video.appendChild(videoSource1);

// Add AirPlay-compatible HLS source
const videoSource2 = document.createElement('source');
videoSource2. type = 'application/x-mpegURL';
videoSource2.src = airplayURL;
video.appendChild(videoSource2);

Now when Safari detects that an alternative source is available in addition to the MediaSource URL object, it will display the familiar AirPlay icon to the video player control. Should the user select AirPlay, it will switch over from MSE to the AirPlay-compatible URL.

The streaming code for this demo is very basic and serves as an example of getting the bytes from the video to the browser.

document.onload = async () => {
  if (!MediaSource.isTypeSupported(mediaType)) {
    return console.log('Media type is not supported.');
  }

  await new Promise((resolve) => {
    source.addEventListener("sourceopen", resolve, { once: true });
    document.body.appendChild(video);
  });

  const sourceBuffer = source.addSourceBuffer(mediaType);

  async function loadSegment() {
    const response = await fetch(videoURL);
    const buffer = await response.arrayBuffer();
    await new Promise((resolve) => {
      sourceBuffer.addEventListener("updateend", resolve, { once: true });
      sourceBuffer.appendBuffer(buffer);
    });
  }

  if (typeof(source.onstartstreaming) !== 'undefined') {
    source.onstartstreaming = async () => {
      loadSegment();
    };
  } else loadSegment();
});

Feedback

Offering support for MMS/MSE and AirPlay is gives users the best of all worlds and the video element makes it easy to offer multiple sources. You can learn more about the Managed Media Source API proposal at the W3C, and learn about HTTP Live Streaming from Apple’s documentation.

We love to hearing from you. Send a tweet to @webkit to share your thoughts on this feature. Find us on Mastodon at @jensimmons@front-end.social and @jondavis@mastodon.social. You can also follow WebKit on LinkedIn. If you run into any issues, we welcome your WebKit bug reports on WebKit-powered features like this. Reporting issues and sharing your feedback makes an enormous difference.

February 16, 2024 06:35 PM

February 07, 2024

Release Notes for Safari Technology Preview 188

Surfin’ Safari

Safari Technology Preview Release 188 is now available for download for macOS Sonoma and macOS Ventura. If you already have Safari Technology Preview installed, you can update it in System Settings under General → Software Update.

This release includes WebKit changes between: 272449@main…273601@main.

Accessibility

New Features

  • Added support for the new CSS content alternative text syntax. (272455@main) (26942023)

Resolved Issues

  • Fixed role assignment for <header> inside <main> and sectioning elements. (273188@main) (48370244)
  • Fixed text missing from accessibility labels for many common shadow DOM scenarios. (272531@main) (120223342)
  • Fixed the mapping for the iOS accessibility framework to speak “switch button”, “on”, and “off” for <input type=checkbox switch>. (273206@main) (121215059)
  • Fixed comboboxes to expose their linked objects correctly. (273542@main) (121242926)

Animations

Resolved Issues

  • Fixed the transition property to produce the shortest serialization. (272513@main) (119822401)
  • Fixed the animation property to produce the shortest serialization. (272629@main) (120439368)

Browser Changes

Resolved Issues

  • Fixed loading a ⌘Click fragment link in a background tab. (272906@main) (119079650)
  • Fixed saving linked subresources when saving web page resources. (272925@main) (120491493)

CSS

New Features

  • Added supports() syntax for @import rules. (273591@main) (109060734)

Resolved Issues

  • Fixed getComputedStyle() for invalid pseudo-elements. (272543@main) (98504661)
  • Fixed oklab and oklch lightness value clamping. (272501@main) (116195533)
  • Fixed poor performance with :has(+ :not(.class)) pseudo-class selector. (272678@main) (119819247)
  • Fixed CSS content computed value serialization. (272476@main) (120061551)
  • Fixed pseudo-element parsing in getComputedStyle() and KeyframeEffect.prototype.pseudoElement so they require them starting with :: (or : for 4 legacy pseudo-elements). (272499@main) (120170550)
  • Fixed CSS linear() easing. (272613@main) (120290721)
  • Fixed: Aliased :-webkit-full-screen pseudo-class to :fullscreen. (272577@main) (120335917)
  • Fixed: Aliased :-webkit-any-link to :any-link and :matches() to :is(). (272559@main) (120337922)
  • Fixed getComputedStyle() pseudo-element parsing to support the full range of CSS syntax. (272649@main) (120471227)
  • Fixed :not(:has(:not(foo))) getting misclassified as scope breaking. (273177@main) (120492012)
  • Fixed @supports to correctly handle support for some -webkit prefixed pseudo-elements that were incorrectly treated as unsupported. (272726@main) (120577690)
  • Fixed updating media-query sensitive meta tags after style changes. (272947@main) (120854167)

Deprecations

  • Removed -webkit-alt and alt properties. (272480@main) (120051066)
  • Removed the non-standard resize: auto property. (273035@main) (120138995)
  • Made -apple- prefixed pseudo-elements no longer valid. (272538@main) (120268884)
  • Removed :-webkit-animating-full-screen-transition pseudo-class. (273529@main) (121302758)
  • Removed :-khtml-drag pseudo-class. (273261@main) (121303391)

Forms

Resolved Issues

  • Fixed text-indent to affect the selected file(s) label for file inputs. (272837@main) (105223868)

Loading

Resolved Issues

  • Fixed navigator.cookieEnabled to return false when cookies are blocked. (273522@main) (121284878)

Lockdown Mode

Resolved Issues

  • Fixed Lockdown Mode disabling on sites with COOP and COEP HTTP headers. (273243@main) (119503109)

Media

Resolved Issues

  • Fixed HLS video captions where there are multiple text tracks available. (272784@main) (119839950)
  • Fixed fullscreen video not scaling to display size when the Safari window is in Full Screen App Mode. (272733@main) (119893556)
  • Fixed handling key renewal requests that cause playback errors for some DRM content. (272592@main) (120230860)
  • Fixed paint-on captions shifting during playback. (272966@main) (120847946)
  • Fixed blocked encrypted sampled not getting enqueued after a CDM is attached to a SourceBuffer. (273340@main) (120879185)

Rendering

Resolved Issues

  • Fixed resizing a <textarea> element with 1rem padding. (273029@main) (90639221)
  • Fixed incorrectly oriented Traditional Mongolian script characters. (272454@main) (93426525)
  • Fixed handling images with color spaces not supported by the backend to fallback to render in sRGB. (273204@main) (118238178)
  • Fixed check boxes and radio buttons to avoid floats. (273047@main) (118660695)
  • Fixed rendering issues when editing text. (273320@main) (119833765)
  • Fixed a floating element causing a list item bullet to be orphaned on constrained lines. (272451@main) (120022893)
  • Fixed fully repainting form controls with visual overflow and writing-mode: vertical-rl. (272799@main) (120066970)
  • Fixed incorrect inline box (hugging) outline painting in vertical writing modes. (272512@main) (120217559)
  • Fixed incorrect ch unit value in vertical-rl and vertical-lr when text-orientation is not upright. (272536@main) (120293590)
  • Fixed the color correctness of the color matrix filter. (272891@main) (120795573)
  • Fixed overflow: hidden preventing CSS Subgrid. (273134@main) (120848131)
  • Fixed the repaint area for underline text decorations. (273126@main) (121082290)
  • Fixed rendering floats and an out-of-flow <br> element with clear. (273407@main) (121444267)

Scrolling

Resolved Issues

  • Fixed unusable horizontal scrollbars with a right-to-left element. (272466@main) (109858866)
  • Fixed wheel overflow behavior with Shadow DOM elements. (273181@main) (118496293)
  • Fixed keyboard scrolling beyond the page getting stuck at a bad scroll offset. (272957@main) (120053910)
  • Fixed a variety of issues by disabling Scroll Anchoring. (273238@main) (121236706)

Storage

Resolved Issues

  • Fixed cases where website data is unexpectedly evicted. (272951@main) (119818267)

Deprecations

  • Removed support for AppCache. (273297@main) (113343269)

SVG

Resolved Issues

  • Fixed negative SVGTransform scale values to be correctly stringified. (272885@main) (118656892)
  • Fixed the layout of an SVG when it is inside an <iframe> without affecting the size of the <iframe>. (272503@main) (120178866)
  • Fixed displaying an SVG element inside a <switch> element. (272831@main) (120732837)

Web API

New Features

  • Added support for CustomStateSet in custom elements and :state() pseudo-class. (272474@main) (120072599)

Resolved Issues

  • Fixed: Added support for AES-GCM in WebCrypto. (273488@main) (101040216)
  • Fixed removing highlights in the Custom Highlights API. (272723@main) (119531671)
  • Fixed unnecessarily unsetting the iframe fullscreen flag. (272462@main) (120052751)
  • Fixed getElementsByName() to only return HTML elements, not SVG, MathML, or other types of elements. (272530@main) (120275680)
  • Fixed the button value for a pointerup event not matching the pointerdown event. (273263@main) (120429508)
  • Fixed a wheel event to fire on an element that has been re-inserted after document.open. (272960@main) (120893136)
  • Fixed Scroll To Text Fragment Text Directives to find text with additional unrendered white space in their node data. (273016@main) (120913588)
  • Fixed Media Capture API to capture the camera. (273258@main) (121256297)

Deprecations

  • Removed support for KeyboardEvent.altGraphKey. (273379@main) (102980723)
  • Removed AES-CFB support from WebCrypto. (272615@main) (120000331)
  • Removed the non-standard KeyboardEvent.keyLocation. (273457@main) (121564228)

Web Extensions

Resolved Issues

  • Fixed browsing.scripting.executeScript to handle all valid argument types. (120727491)

WebAuthn

Resolved Issues

  • Fixed getClientCapabilities to align with WebAuthn standards to use a record type with camelCase values. (272998@main) (120442670)

WebGL

New Features

  • Enabled support for EXT_conservative_depth and NV_shader_noperspective_interpolation. (272979@main) (120907578)

Resolved Issues

  • Fixed WebGL to be available in nested workers. (272765@main) (120279728)

WebRTC

Resolved Issues

  • Fixed triggering resolution scaling in the case of WebRTC maintain-framerate degradationPreference. (273172@main) (121041723)

February 07, 2024 10:57 PM

February 01, 2024

The web just gets better with Interop 2024

Surfin’ Safari

The web is amazing. It makes collaborating, learning, and connecting easy for billions of people, because it’s intentionally designed to run on radically different devices.

It’s your job as a web developer to ensure your project works in every browser and for every user — and that can be hard to do. It’s a far easier undertaking when browsers have identical implementations of the web technology you use.

Identical implementations are accomplished through the web standards process, where people collaborate together to write extremely detailed technical documents that define each new web technology — right down to how website bugs should work.

One way to check and see if browsers follow the web standards is through automated testing. There are several shared repositories of such tests, including Web Platform Tests. WPT contains over 1.8 million tests, of which over 95% pass in all of the major browsers.

The Interop Project

The Interop project aims to improve interoperability by encouraging browser engine teams to look deeper into specific focus areas. Now, for a third year, Apple, Bocoup, Google, Igalia, Microsoft, and Mozilla pooled our collective expertise and selected a specific subset of automated tests for 2024.

Some of the technologies chosen have been around for a long time. Other areas are brand new. By selecting some of the highest priority features that developers have avoided for years because of their bugs, we can get them to a place where they can finally be relied on. And by selecting exciting new technology, we can ensure it’s interoperable from the beginning.

To better understand where interoperability is going in the future, let’s first take a look at the impact of Interop 2023.

Interop 2023

Interop 2023 was even more of an overwhelming success than Interop 2022. In January 2023, 48% of the chosen tests passed in all three of the major browser engines (in those shipped to users: Chrome and Firefox for desktop Linux, and Safari on macOS Monterey). A year later, that pass rate rose to 95% (in Chrome Dev and Firefox Nightly for desktop Linux, and Safari Technology Preview on macOS Ventura).

Screenshot of the graph of results from January to December 2023, available at https://wpt.fyi/interop-2023The success of Interop 2023, seen on the “Experimental” dashboard. The “Interop” line, in dark green, shows the percentage of tests that passed in all three — Chrome Dev, Firefox Nightly, and Safari Technology Preview.

What did Interop 2023 accomplish?

  • It ensured that all browsers have full support for P3 color, seven years after it started shipping.
  • Form controls now support vertical writing modes, for the first time in the web’s history.
  • CSS border-image now works as originally intended.
  • Subgrid, Container Queries, :has(), Motion Path, CSS Math Functions, inert and @property are now supported in every modern browser.
  • Improved Web APIs include Offscreen Canvas, Modules in Web Workers, Import Maps, Import Assertions, and JavaScript Modules.
  • The entire Media Queries 4 specification is now supported everywhere, with easier to use syntax.
  • Web Components got a boost with adoptedStyleSheets, ElementInternals, Form-Associated Custom Elements, and the basic behavior of Shadow DOM and Custom Elements.
  • Useful CSS pseudo-classes can now be relied on, with consistent cross-browser support for :nth-child(), :nth-last-child(), :modal, :user-valid, and :user-invalid.
  • Feature queries now have new support for detecting font features.
  • Font Palettes provide robust support for color fonts.
  • Significant progress made improving CSS Masking, HTML Forms, Pointer and Mouse Events, Scrolling, Transforms, URL, WebCodecs, and a bucket of bugs causing web compat issues.
  • And more.

We hope this work gives you a renewed sense of confidence to use these technologies. If you found any of them hard-to-use in the past, give them another try.

Interop 2023 had twenty-six focus areas, twenty of which are being retired as a success. Work will continue on Custom Properties, Pointer and Mouse Events, URL, and a new grouping called “Layout” — consisting of Flexbox, Grid, and Subgrid.

Interop 2024

Now, we are doing it all again for 2024. Ninety-six focus area proposals were submitted for consideration. Ultimately, sixteen were chosen. Grouping some of the new proposals together, and continuing some of the work from 2023, gives Interop 2024 a total of seventeen focus areas.

The Interop 2024 dashboardThe Interop 2024 dashboard, looking at the “stable” browsers (those currently in the hands of everyday people). Coincidentally, the overall Interop score is once again starting at 48%.

New this year, Microsoft Edge now has its own column on the Interop dashboard. This currently represents Edge and Edge Dev running on Windows 10.

The 2024 Focus Areas

Accessibility

Interop 2023 included an Accessibility Investigation project. Led by Apple’s accessibility team, the group worked diligently to create new accessibility testing infrastructure for WPT, and write over 1300 new accessibility tests. These tests have now been included in Interop 2024 as a focus area, encouraging browsers to increase their support.

The majority of new accessibility tests cover WAI-ARIA, in particular, the Roles Model and the Accessible Name and Description Computation (AccName). Together, these provide a consistent mechanism for conveying the purpose or intent of an element so assistive technology users understand what it is and what they can do with it.

Other new accessibility tests cover how those technologies are incorporated into host languages. For example, the HTML Accessibility API Mappings specification (HTML-AAM) defines the default accessibility semantics of HTML elements, along with related rules for how browsers work with features like the <label> element and image alt text. (See the html-aam/roles WPT tests as an example.)

Another new set of tests cover the accessibility of display: contents. This display mode in CSS provides a useful mechanism for removing the box around content — helpful when wanting to adjust the parent/child/grandchild relationships of content for the purposes of Flexbox or Grid. But it was off limits for use for years, because of the lack of accessibility in early implementations. Removing the box on an item completely removed all the contents of that box from the accessibility tree. Sighted users could still see the child content, but many users of assistive technology experienced it completely disappearing. Most of these problems have been fixed in browsers, but not all, not for every situation. These new tests are the next step toward full accessibility and interoperability.

By including these new Accessibility tests in Interop 2024, the hope is to fix every issue in all browsers. We want it to be easier for developers to create accessible sites and make the web better for everyone, including people with disabilities.

CSS Nesting

CSS Nesting is a focus area for Interop 2024 to ensure any differences are ironed out, and to provide you with the confidence to use it. The ability to nest CSS shipped in all four major browsers in 2023 — first in Chrome, Edge, and Safari in April/May. And then in Firefox in August.

The web standard changed slightly between May and August, relaxing the original requirement that every nested selector start with a symbol. Developers can now simply write article, rather than needing to use & article.

All of the implementations have since been updated, but there are still small bits that could benefit from attention to interoperability, especially as the final complex details of how Nesting works are settled in the CSS Working Group. Most of Safari’s test failures, for example, are about how nested CSS interacts with the Shadow DOM via :host.

Custom Properties

The @property at-rule started shipping in browsers over the last few years. As part of Interop 2023, the Custom Properties focus area rose from 4% of tests passing in all stable browsers to 7.6% passing — with 90.7% passing in all of the preview browsers. Firefox is the last browser to add support, which is currently in progress in Firefox Nightly. Since this work isn’t done yet, the focus area is being continued in 2024.

With @property, developers can declare CSS custom properties in a fashion similar to how browser engines define CSS properties — specifying its syntax, inheritance behavior, and initial value.

@property --size {
  syntax: "<length>";
  inherits: false;
  initial-value: 0px;
}

This allows you to do things in CSS that were impossible before, like animating gradients or certain parts of transforms.

Declarative Shadow DOM

Declarative Shadow DOM is a declarative API that lets you create reusable widgets and components by using only HTML — no JavaScript is necessary. It’s been supported in Safari 16.4 since March 2023, and in Chrome 90 since April 2021. Firefox has an implementation in Firefox Nightly.

Declarative Shadow DOM was one of the often-asked-for features in the State of HTML 2023 survey, so it was chosen to be part of Interop 2024 to ensure it becomes interoperable across all browsers.

Font size adjust

The font-size-adjust property is a great example of the usefulness of placing attention on older technology. Firefox first implemented font size adjust in 2008, but it was rarely used or even discussed by web designers and developers. The early spec evolved over time, adding support for more languages through the two-value syntax, and becoming easier to use with the from-font value.

The WebKit team implemented the basic version in Safari 16.4 and added the updates in September’s Safari 17.0. Mozilla updated their implementation in Firefox 118, also in September 2023. Both Safari and Firefox now pass 100% of all tests. Chrome began an experimental implementation in 2015, but has yet to ship it. Now with Interop 2024, it’s likely every browser will gain complete support.

Font size adjust provides a simple way to conform all the fonts used in a string of text to be the same visual size — so every character in 1.4rem-sized text, for example, has the same x-height — or cap height, ch width, ic width, or ic height. The two value syntax allows you to choose which measurement to conform.

This property is especially useful when you are mixing code with regular text, or mixing multiple languages together, and the different fonts within the same sentence have different sizes. With font size adjust you can avoid weirdly-big letters. No more fussing with font metrics to find a magic number that makes them all look the same size.

a code demo showing the visual consistency of the inked x-height of multiple font familiesThe CSS `font-size-adjust: from font` makes the Courier font adjust its size to match its x-height with that from Iowan Old Style, instead of typesetting the code to be visually larger. The size uniformity holds even when fallback fonts are used instead.

Learn more about font-size-adjust by watching What’s new in CSS from WWDC23.

HTTPS URLs for WebSocket

A quirky aspect of the WebSocket API is that you need to use non-HTTP(S) schemes: ws: (insecure) and wss:. As the URLs function otherwise identically to HTTP(S) URLs, this makes the API a bit frustrating to deal with. Based on web developer feedback the WebKit team decided to address this last year by making the API support HTTP(S) URLs as well. We shipped support in Safari 17.0.

This means that instead of writing:

function webSocketHandle(path) {
  const url = new URL(path, location);
  url.protocol = location.protocol === "https:" ? "wss:" : "ws:";
  return new WebSocket(url);
}
// ...
const ws = webSocketHandle(path);

You can now write the much more ergonomic:

const ws = new WebSocket(path);

By bringing this to Interop 2024, we hope other browsers will adopt it as well, making it universally available for web developers.

IndexedDB

IndexedDB is an API that provides powerful ways to store data client-side, in an object-oriented database. It started shipping in browsers in 2011, and over the years the web standard has kept evolving. Both version 2 and version 3 are supported by all major browsers. Version 2 is fully interoperable, but version 3 needs a bit of attention to bring up the quality of implementations. Being part of Interop 2024 will help ensure implementations are completed and aligned.

Layout

CSS Grid and Flexbox were both included in the original Interop project in 2021. Subgrid was added in Interop 2023. While all three layout methods are now in great shape, they still aren’t quite perfect. The tests for these three areas are now being combined into one Focus Area called Layout. Work will continue to ensure complex edge cases are more interoperable. Meanwhile, developers should absolutely feel confident using these three technologies, since all browsers have solid support for Flexbox, Grid, and now Subgrid.

Pointer and Mouse Events

Pointer events are DOM events that are fired for a pointing device. They create a single DOM event model to handle pointing input devices such as a mouse, pen/stylus, or touch with one or more fingers. This API first started shipping in browsers in 2012, and landed everywhere by 2019, but still had rocky interoperability.

In 2022, the Interop team launched an Investigation Project to look deeper into the current state of Pointer and Mouse Events, in an effort to clarify consensus and write tests that captured the state of that consensus. For Interop 2023, those tests enabled Pointer and Mouse Events to be a Focus Area, where the test pass rate was part of the Interop 2023 dashboard and score. Over the year, Pointer and Mouse Events rose from a test pass rate of 34% to 81% — the most significant progress of any area.

While passing 81% of tests is a significant improvement, there is more work to do, therefore Pointer and Mouse Events will continue to be a Focus Area for 2024.

Popover

The new popover attribute in HTML provides a built-into-the-browser way to have an element pop into view on the top layer of a page. If you are creating an overlay over the entire web page, the dialog element is the best option. But when you want to turn any other element into a popup message, user interface, or other kind of content that appears and disappears, the popover element provides a framework to do it.

Support for popover shipped in Chrome 114 and Safari 17.0 in 2023. Firefox currently has support in progress in Firefox Nightly. Being part of Interop 2024 will ensure this highly desired feature has a fantastic start.

Relative Color Syntax

Relative Color Syntax is a new way to define colors in CSS that allows you do so while referring to another color. You can, for instance, lighten or darken an existing color by a certain amount. You can take a color variable, adjust the saturation, and assign the new color to a second variable. Relative Color Syntax can be especially powerful when creating a design system.

a still from the WWDC session teach how relative color syntax worksLearn more about Relative Color Syntax by watching What’s new in CSS from WWDC23.

Safari 16.4 was the first browser to ship support, in March 2023. Chrome 119 and Edge 119 shipped support in Oct and Nov 2023. Currently, none of the implementations have support for using currentcolor with Relative Color Syntax.

The Relative Color Syntax focus area for Interop 2024 doesn’t test overall support of Relative Color Syntax. It’s narrowly focused on whether or not currentcolor is supported, and includes tests of out-of-gamut behavior — checking to see what happens on displays that don’t have support for P3 color. Inclusion in Interop 2024 will help these final bits get done.

requestVideoFrameCallback

The <video> element provides powerful functionality for putting video on the web. But often, developers want and need to do more. The HTMLVideoElement interface provides special properties and methods for manipulating video objects in JavaScript. And one of those methods is requestVideoFrameCallback(). It lets you perform per-video-frame operations on video in an efficient manner — operations like video processing or analysis, painting to canvas, and synchronization with audio sources.

Supported since Chrome 83 and Safari 15.4, inclusion in Interop 2024 will help browsers complete and polish our implementations.

Scrollbar styling

The scrollbar styling focus area includes two CSS properties that can be used to style scrollbars. The scrollbar-width property provides three values: auto, thin, and none. The auto value is the default width; thin provides a thinner scrollbar; and none hides the scrollbar while still allowing content to scroll. Firefox 64 implemented support in December 2018, and it just shipped in Chrome 121 and Edge 121.

The scrollbar-gutter property lets you reserve space for the scrollbar, so the layout is the same whether or not a scrollbar is present. The scrollbar-gutter: stable rule lets you tell the browser to reserve space for a scrollbar, even when a scrollbar isn’t there. This can prevent layout shifts from happening between states where scrollbars are needed or not needed. It shipped in Chrome 94, Edge 94 and Firefox 97, in 2021–22.

Safari has the most work to do to complete this Focus Area. Chrome and Firefox already pass 100% of the tests. Ironically, it was Safari who first provided the ability to style scrollbars with nine pseudo-elements, ::-webkit-scrollbar-*, back in 2009. However that approach to styling scrollbars never became an official CSS web standard. The CSS Working Group instead opted for a far simpler approach.

@starting-style and transition-behavior

This Focus Area brings attention to two new features for controlling animation. Both shipped in Chrome 117 and Edge 177 in Sept 2023.

The @starting-style rule in CSS lets you define starting values for a particular element. This is needed when the element is about to go through a transition. It also provides a way for transitioning in or out of display:none.

.alert {
  transition: background-color 2s;
  background-color: green;
  @starting-style {
    background-color: transparent;
  }
}

In the above example, the background-color will transition from transparent to green when the element is appended to the document.

Previously, only animations could animate discretely-animatable properties. The transition-behavior property in CSS expands that capability to CSS transitions, paving the way for transitioning the display property when showing or hiding elements.

Text Directionality

The direction in which text flows is a vital aspect of typesetting on the web. Some languages flow from left-to-right, while others flow from right-to-left. One of the many bits of technology supporting text direction is the dir attribute. It lets you specifically mark any HTML element with the direction: left, right, or auto — where auto asks the browser to guess from the first letter. The interaction of directionality and shadow trees was not well-defined until recently. Now that it’s been addressed at a standards level, adding it to Interop 2024 helps us ensure implementations align as well.

text-wrap: balance

Web designers have long wished for a way to prevent very short or one-word lines of text — often known as widows or orphans. Since the advent of responsive web design and the lack of control over the width of columns, this desire has gotten even more challenging. The text-wrap property provides you with multiple options for telling the browser how to wrap text with different methods for calculating line breaks for specific use cases.

The text-wrap: balance rule is a great solution for headlines. It balances a few lines of text so that each line has about the same amount of text as the others. It shipped in Chrome 114 and Firefox 121, and is implemented in Safari Technology Preview.

Interop 2024 also includes tests of how text-wrap-mode, text-wrap-style, and white-space-collapse behave. The CSS Working Group recently changed to how these longhands interact with each other, so support is currently uneven between browsers. Interop 2024 will help ensure all browser engines update to the latest web standards.

URL

URLs are one of the most fundamental parts of the web. Without them, the web would not exist. But like many things invented very early in the history of the web, support has yet to be fully interoperable. To improve this, the WHATWG wrote the URL Living Standard packed with details on exactly how URLs should work. The tests supporting this web standard were a focus area for Interop 2023, improving the pass rate from 77% to 85%. To ensure interoperability, the work in this area will continue in 2024.

Safari is proud to lead the pack, passing 99.7% of the tests. Improvements in other browsers will help ensure websites work correctly everywhere.

The 2024 Investigation projects

Interop 2024 also includes three investigation areas. These are “homework projects” for the Interop team to work on. All three this year are about writing and making it possible to run more tests — Accessibility Testing, Mobile Testing, and WebAssembly Testing. The Mobile Testing investigation project aims to complete the infrastructure needed at WPT to be able to test browsers on mobile operating systems, potentially to include those scores on the Interop project dashboard in the future.

While two of the three investigations are projects continuing from last year, they all are starting 2024 at zero percent done. Each team involved will set new goals for this year, and the dashboard will report progress on those goals.

Track the progress

Keep up with the progress of Interop 2024 throughout the year, on the Interop 2024 dashboard.

Our Commitment

We continue to believe that interoperability is one of the fundamental pillars that makes the web such a success. Our efforts in Interop 2022 and 2023 demonstrate how deeply we care about the web. We are excited to again collaborate with our colleagues in seizing this opportunity help the web work better for everyone.

February 01, 2024 05:00 PM

WPE WebKit Blog: Use Case: Server-side headless rendering

Igalia WebKit

WPE and server-side headless rendering

In many distributed applications, it can be useful to run a light web browser on the server side to render some HTML content or process images, video and/or audio using JavaScript.

Some concrete use-cases can be:

  • Video post-production using HTML overlays.
  • Easy 3D rendering with WebGL that can be broadcasted as a video stream.
  • Reusing the same JavaScript code between a frontend web application and the backend processing.

WPE WebKit is the perfect solution for all those use cases as it offers a lightweight solution which can run on low-end hardware or even within a container. It provides a lot of flexibility at the moment of choosing the backend infrastructure as WPE WebKit can, for instance, run from within a container with a very minimal Linux configuration (no need for any windowing system) and with full hardware acceleration and zero-copy of the video buffers between the GPU and the CPU.

Additionally, the fact that WPE WebKit is optimized for lower-powered devices, makes it also the perfect option for server-side rendering when scaling commercial deployments while keeping cost under control, which is yet another important factor to take into account when considering cloud rendering.

February 01, 2024 12:00 AM

January 29, 2024

WPE WebKit Blog: A New WPE Backend Using EGLStream

Igalia WebKit

What is a WPE Backend?

Depending on the target hardware WPE may need to use different techniques and technologies to ensure correct graphical rendering. To be independent of any user-interface toolkit and windowing system, WPE WebKit delegates the rendering to a third-party API defined in the libwpe library. A concrete implementation of this API is a “WPE backend”.

WPE WebKit is a multiprocess application, the end-user starts and controls the web widgets in the application process (which we often call “the UI process” while the web engine itself uses different subprocesses: WPENetworkProcess is in charge of managing network connections and WPEWebProcess (or “web process”) in charge of the HTML and JavaScript parsing, execution and rendering. The WPE backend is at a crossroads between the UI process and one or more web process instances.

Diagram showing a box for the WPE backend in between the UI process and WPEWebProcess

The WPE backend is a shared library that is loaded at runtime by the web process and by the UI process. It is used to render the visual aspect of a web page and transfer the resulting video buffer from the web process to the application process.

Backend Interfaces

The WPE backend shared library must export at least one symbol called _wpe_loader_interface of type struct wpe_loader_interface as defined in the libwpe API. Presently its only member is load_object, a callback function that receives a string with an interface name and returns concrete implementations of the following interfaces:

The names passed to the .load_object() function are the same as those of the interface types, prefixed with an underscore. For example, a .load_object("_wpe_renderer_host_interface") call must return a pointer to a struct wpe_renderer_host_interface object.

Example C code for a load_object callback.
static struct wpe_renderer_host_interface = { /* ... */ };
static struct wpe_renderer_backend_egl_interface = { /* ... */ };

static void*
my_backend_load_object(const char *name)
{
    if (!strcmp(name, "_wpe_renderer_host_interface"))
        return &my_renderer_host;
    if (!strcmp(name, "_wpe_renderer_backend_egl_interface"))
        return &my_renderer_backend_egl;

    /* ... */

    return NULL;
}

struct wpe_loader_interface _wpe_loader_interface = {
    .load_object = my_backend_load_object,
};

Each of these interfaces follow the same base structure: the struct members are callback functions, all interfaces have create and destroy members which act as instance constructor and destructor, plus any additional “methods”. The pointer returned by the create callback will be passed as the object “instance” of the other methods:

struct wpe_renderer_host_interface {
  void* (*create)(void);
  void  (*destroy)(void *object);
  /* ... */
};

In the UI process side WPE WebKit will create:

  • One “renderer host” instance, using wpe_renderer_host_interface.create().
  • Multiple “renderer host client” instances, using wpe_renderer_host_interface.create_client(). These are mainly used for IPC communication, one instance gets created for each web process launched by WebKit.
  • Multiple “view backend” instances, using wpe_view_backend_interface.create(). One instance is created for each rendering target in the web process.

In each web process—there can be more than one—WPE WebKit will create:

  • One “renderer backend EGL” instance, using wpe_renderer_backend_egl_interface.create().
  • Multiple “renderer backend EGL target” instances, using wpe_renderer_backend_egl_target_interface.create(). An instance is created for each new rendering target needed by the application.
How about wpe_renderer_backend_egl_offscreen_target_interface?

The rendererBackendEGLTarget instances may be created by the wpe_renderer_backend_egl_target_interface, or the wpe_renderer_backend_egl_offscreen_target_interface depending on the interfaces implemented in the backend.

Here we are only focusing on the wpe_renderer_backend_egl_target_interface that is relying on a classical EGL display (defined in the rendererBackendEGL instance). The wpe_renderer_backend_egl_offscreen_target_interface may be used in very specific use-cases that are out of the scope of this post. You can check its usage in the WPE WebKit source code for more information.

These instances typically communicate with each others using Unix sockets for IPC. The IPC layer must be implemented in the WPE backend itself because the libwpe interfaces only pass around the file descriptors to be used as communication endpoints.

From a topological point of view, all those instances are organized as follows:

From an usage point of view:

  • The rendererHost and rendererHostClient instances are only used to manage IPC endpoints on the UI process side that are connected to each running web process. They are not used by the graphical rendering system.
  • The rendererBackendEGL instance (one per web process) is only used to connect to the native display for a specific platform. For example, on a desktop Linux, the platform may be X11 where the native display would be the result of calling XOpenDisplay(); or the platform may be Wayland and in this case the native display would be the result of calling wl_display_connect(); and so on.
  • The rendererBackendEGLTarget (on the web process side) and viewBackend (on the UI process side) instances are the ones truly managing the web page graphical rendering.

Graphics Rendering

As seen above, the interfaces in charge of the rendering are wpe_renderer_backend_egl_target_interface and wpe_view_backend_interface. During their creation, WPE WebKit exchanges the file descriptors used to establish a direct IPC connection between a rendererBackendEGL (in the web process), and a viewBackend (in the UI process).

During the EGL initialization phase, when a new web process is launched, WebKit will use the native display and platform provided by the wpe_renderer_backend_egl_interface.get_native_display() and .get_platform() functions to create a suitable OpenGL ES context.

When WebKit’s ThreadedCompositor is ready to render a new frame (in the web process), it calls the wpe_renderer_backend_egl_target_interface.frame_will_render() function to let the WPE backend know that rendering is about to start. At this moment, the previously created OpenGL ES context is made current to be used as the target for GL drawing commands.

Once the threaded compositor has finished drawing, it will swap the front and back EGL buffers and call the wpe_renderer_backend_egl_target_interface.frame_rendered() function to signal that the frame is ready. The compositor will then wait until the WPE backend calls wpe_renderer_backend_egl_target_dispatch_frame_complete() to indicate that the compositor may produce a new frame.

What happens inside the .frame_will_render() and .frame_rendered() implementations is up to the WPE backend. As en example, it could set up a Frame Buffer Object to have the web content draw offscreen, in a texture that can be passed back to the UI process for further processing, or use extensions like EGLStream, or DMA-BUF exports to transfer the frame to the UI process without copying the pixel data.

Typically the backend sends each new frame to the corresponding view backend in in its .frame_rendered() function. The application can use the frame until it sends back an IPC message to the renderer target (in the web process) to indicate that the frame is not in use anymore and may be be freed or recycled. Although it is not a requirement to do it at this exact point, usually when a renderer backend receives this message it calls the wpe_renderer_backend_egl_target_dispatch_frame_complete() function to trigger the rendering of a new frame. As a side effect, this mechanism also allows controlling the pace at which new frames are produced.

Using EGLStream

EGLStream is an EGL extension that defines a mechanism to transfer hardware video buffers from one process to another efficiently, without getting them out of GPU memory. Although the extension is supported only in Nvidia hardware, it makes for a good example as it transparently handles some complexities involved, like buffers with multiple planes.

This backend uses the EGLStream extension to transfer graphics buffers from the web process, which acts as a producer, to the UI process acting as a consumer. The producer extension EGL_KHR_stream_producer_eglsurface allows creating a surface that may be used as target for rendering, then using eglSwapBuffers() finishes drawing and sends the result to the consumer. Meanwhile, in the consumer side, the EGL_NV_stream_consumer_eglimage extension is used to turn each buffer into an EGLImage.

The reference source code for this WPE backend is available in the WPEBackend-offscreen-nvidia repository, which has been tested with WPE WebKit 2.38.x or 2.40.x, and libwpe version 1.14.x.

Behold, the Future Belongs to DMA-BUF!

With the growing adoption of DMA-BUF for sharing memory buffers on modern Linux platforms, the WPE WebKit architecture will be evolving and, in the future, the need for a WPE Backend should disappear in most cases.

Ongoing work on WPE WebKit removes the need to provide a WPE backend implementation for most hardware platforms, with a generic implementation using DMA-BUF provided as an integral, built-in feature of WebKit. It will still be possible to provide external implementations for platforms that might need to use custom buffer sharing mechanisms.

From the application developer point of view, in most cases writing programs that use the WPE WebKit API will be simpler, with the complexity of the communication among multiple processes handled by WebKit.

Stream Setup

The steps needed to set up EGLStream endpoints need to be done in a particular order:

  1. Create the consumer.
  2. Get the stream file descriptor for the consumer.
  3. Send the stream file descriptor to the producer.
  4. Create the producer.

First, the consumer needs to be created:

EGLStream createConsumerStream(EGLDisplay eglDisplay) {
    static const EGLint s_streamAttribs[] = {
        EGL_STREAM_FIFO_LENGTH_KHR, 1,
        EGL_CONSUMER_ACQUIRE_TIMEOUT_USEC_KHR, 1000 * 1000,
        EGL_NONE
    };
    return eglCreateStreamKHR(eglDisplay, s_streamAttribs);
}

The EGL_STREAM_FIFO_LENGTH_KHR parameter defines the length of the EGLStream queue. If set to zero, the stream will work in “mailbox” mode and each time the producer has a new frame it will empty the stream content and replace the frame by the new one. If non-zero, the stream works work in “FIFO” mode, which means that the stream queue can contain up to EGL_STREAM_FIFO_LENGTH_KHR frames.

Here we configure a queue for one frame because in this case the specification of EGL_KHR_stream_producer_eglsurface guarantees that calling eglSwapBuffers() on the producer the call will block until the consumer retires the previous frame from queue. This is used as implicit synchronization between the UI process side and the web process side without needing to rely on custom IPC, which would add a small delay between frames.

The EGL_CONSUMER_ACQUIRE_TIMEOUT_USEC_KHR parameter defines the maximum timeout in microseconds to wait on the consumer side to acquire a frame when calling eglStreamConsumerAcquireKHR(). It is only used with the EGL_KHR_stream_consumer_gltexture extension because the EGL_NV_stream_consumer_eglimage extension allows setting a timeout on each call to eglQueryStreamConsumerEventNV() function.

Second, to initialize the consumer using the EGL_NV_stream_consumer_eglimage extension it is enough to call the eglStreamImageConsumerConnectNV() function.

Once the consumer has been initialized, you need to send the EGLStream file descriptor to the producer process. The usual way of achieving this would be using IPC between the two processes, sending the file descriptor in a SCM_RIGHTS message through an Unix socket—although with recent kernels using pidfd_getfd() may be an option if both processes are related.

When the file descriptor is finally received, the producer endpoint can be created using the EGL_KHR_stream_producer_eglsurface extension:

const EGLint surfaceAttribs[] = {
    EGL_WIDTH, width,
    EGL_HEIGHT, height,
    EGL_NONE
};
EGLStream eglStream = eglCreateStreamFromFileDescriptorKHR(eglDisplay, consumerFD);
EGLSurface eglSurface = eglCreateStreamProducerSurfaceKHR(eglDisplay, config, eglStream, surfaceAttribs);

As with pbuffer surfaces, the dimensions need to be specified as surface attributes. When picking a frame buffer configuration with eglChooseConfig() the EGL_SURFACE_TYPE attribute must be set to EGL_STREAM_BIT_KHR. From this point onwards, rendering proceeds as usual: the EGL surface and context are made active, and once the painting is done a call to eglSwapBuffers() will “present” the frame, which in this case means sending the buffer with the pixel data down the EGLStream to the consumer.

Consuming Frames

While on the producer side rendering treats the EGLStream surface like any other, on the consumer some more work is needed to manager the lifetime of the data received: frames have to be manually acquired and released once they are not needed anymore.

The producer calls eglQueryStreamConsumerEventNV() repeatedly to retire the next event from the stream:

  • EGL_STREAM_IMAGE_ADD_NV indicates that there is a buffer in the stream that has not yet been bound to an EGLImage, and the application needs to create a new one to which the actual data will be bound later.
  • EGL_STREAM_IMAGE_AVAILABLE_NV indicates that a new frame is available and that it can be bound to the previously created EGLImage.
  • EGL_STREAM_IMAGE_REMOVE_NV indicates that a buffer has been retired from the stream, and that its associated EGLImage may be released once the application has finished using it.

This translates roughly to the following code:

static constexpr EGLTime MAX_TIMEOUT_USEC = 1000 * 1000;
EGLImage eglImage = EGL_NO_IMAGE;

while (true) {
    EGLenum event = 0;
    EGLAttrib data = 0;

    // WARNING: The specification states that the timeout is in nanoseconds
    // (see: https://registry.khronos.org/EGL/extensions/NV/EGL_NV_stream_consumer_eglimage.txt)
    // but in reality it is in microseconds, at least with the version 535.113.01 of the NVidia drivers.
    if (!eglQueryStreamConsumerEventNV(display, eglStream, MAX_TIMEOUT_USEC, &event, &data))
        break;

    switch (event) {
      case EGL_STREAM_IMAGE_ADD_NV: // Bind an incoming buffer to an EGLImage.
          if (eglImage) eglDestroyImage(display, eglImage);
          eglImage = eglCreateImage(display, EGL_NO_CONTEXT, EGL_STREAM_CONSUMER_IMAGE_NV,
                                    static_cast<EGLClientBuffer>(eglStream), nullptr);
          continue; // Handle the next event.

      case EGL_STREAM_IMAGE_REMOVE_NV: // Buffer removed, EGLImage may be disposed.
          if (data) {
              EGLImage image = reinterpret_cast<EGLImage>(data);
              eglDestroyImage(display, image);
              if (image == eglImage)
                  eglImage = EGL_NO_IMAGE;
          }
          continue; // Handle the next event.

      case EGL_STREAM_IMAGE_AVAILABLE_NV: // New frame available.
          if (eglStreamAcquireImageNV(display, eglStream, &eglImage, EGL_NO_SYNC))
              break;

      default:
          continue; // Handle the next event.
    }

    /*** Use the EGLImage here ***/

    eglStreamReleaseImageNV(display, eglStream, eglImage, EGL_NO_SYNC);
}

The application is free to use each EGLImage as it sees fit. An obvious example would be to use it as the contents for a texture, which then gets painted in the “content” area of a web browser; or as the contents of the screen for an in-game computer that the player can interact with, enabling display of real, live web content as part of the gaming experience—now that would be a deeply embedded browser!

One Last Thing

There is a small showstopper to have EGLStream support working: currently when WPE WebKit uses surfaceless EGL contexts it sets the surface type to EGL_WINDOW_BIT attribute, while EGL_STREAM_BIT_KHR would be needed instead. A small patch is enough to apply this tweak:

diff --git a/Source/WebCore/platform/graphics/egl/GLContextEGL.cpp b/Source/WebCore/platform/graphics/egl/GLContextEGL.cpp
index d5efa070..5f200edc 100644
--- a/Source/WebCore/platform/graphics/egl/GLContextEGL.cpp
+++ b/Source/WebCore/platform/graphics/egl/GLContextEGL.cpp
@@ -122,9 +122,11 @@ bool GLContextEGL::getEGLConfig(EGLDisplay display, EGLConfig* config, EGLSurfac
         attributeList[13] = EGL_PIXMAP_BIT;
         break;
     case GLContextEGL::WindowSurface:
-    case GLContextEGL::Surfaceless:
         attributeList[13] = EGL_WINDOW_BIT;
         break;
+    case GLContextEGL::Surfaceless:
+        attributeList[13] = EGL_STREAM_BIT_KHR;
+        break;
     }

     EGLint count;

January 29, 2024 06:00 AM