1 Flutter Rendering Principles

In Flutter, there are three objects related to rendering: Canvas, Layer, and Scene:

• Canvas: Encapsulates various drawing instructions of Flutter Skia, such as drawing lines, circles, rectangles, etc.

• Layer: Divided into container types and drawing types; can be understood as carriers of drawn products. For example, after invoking the Canvas drawing API, the corresponding drawing product is stored in the PictureLayer.picture object.

• Scene: The elements to be displayed on the screen. Before rendering, we need to associate the drawn products stored in the Layer with the Scene.

Flutter Rendering Process:

1. Create a Canvas for drawing. A PictureRecorder is also needed, as the drawing instructions will ultimately be passed to Skia. The Canvas may issue multiple drawing instructions, so the PictureRecorder collects all drawing instructions issued by the Canvas over a period. Thus, the first parameter of the Canvas constructor must be a PictureRecorder instance.

2. After completing the Canvas drawing, retrieve the drawing product via the PictureRecorder and store it in a Layer.

3. Construct a Scene object to associate the drawing products from the Layer with the Scene.

4. Render the Scene; call the window.render API to send the drawing products from the Scene to the GPU.

Below is an example demonstrating the entire rendering process:

Remember the previous example of drawing a chessboard? Whether using CustomPaint or a custom RenderObject, the drawing was done within Flutter's Widget framework. In reality, the underlying Flutter will follow the above process to complete rendering. Therefore, we can also directly call these low-level APIs in the main function to accomplish the task. Here’s how to draw a chessboard directly on the screen in the main function:

void main() {
  // 1. Create a drawing recorder and Canvas
  PictureRecorder recorder = PictureRecorder();
  Canvas canvas = Canvas(recorder);
  // 2. Draw in the specified area.
  var rect = Rect.fromLTWH(30, 200, 300, 300);
  drawChessboard(canvas, rect); // Draw chessboard
  drawPieces(canvas, rect); // Draw pieces
  // 3. Create a layer and save the drawn product in it
  var pictureLayer = PictureLayer(rect);
  // Use recorder.endRecording() to get the drawing product.
  pictureLayer.picture = recorder.endRecording();
  var rootLayer = OffsetLayer();
  rootLayer.append(pictureLayer);
  // 4. Render the drawn content on the screen.
  final SceneBuilder builder = SceneBuilder();
  final Scene scene = rootLayer.buildScene(builder);
  window.render(scene);
}

The rendering effect is shown in Figure :

2 Picture

We mentioned earlier that the drawing product of PictureLayer is a Picture. There are two points to clarify about Picture:

1. A Picture is actually a series of graphic drawing operation instructions, which can be referenced in the comments of the Picture class source code.

2. For a Picture to be displayed on the screen, it must undergo rasterization. After that, Flutter will cache the rasterized bitmap information, meaning that the drawing instructions for the same Picture object will only execute once, and once completed, the drawn bitmap will be cached.

Considering the above two points, we can see that the "drawing product" of PictureLayer initially consists of a series of "drawing instructions." Once the first drawing is completed, the bitmap information will be cached, and the drawing instructions will not be executed again. At this point, the "drawing product" refers to the completed bitmap.

Bitmap to Image from Canvas

Since the Picture stores the drawing product, it should also provide a method to export it. In fact, the Picture has a toImage method that can export an Image based on a specified size.

// Export the image as Uint8List final Image image = await pictureLayer.picture.toImage(); final ByteData? byteData = await image.toByteData(format: ImageByteFormat.png); final Uint8List pngBytes = byteData!.buffer.asUint8List(); print(pngBytes);

3 Layer

Now, let’s consider a question: What is the role of Layer as a holder of the drawing product? The answer is:

• It can reuse drawing products between different frames (if they have not changed).

• It delineates drawing boundaries and reduces the repaint range.

Now, let’s examine how Layers work in Flutter, but first, we need to add some preliminary knowledge.

3-1. Layer Types

In the examples at the beginning of this section, we defined two Layer objects:

• OffsetLayer: The root Layer, inheriting from ContainerLayer, while ContainerLayer inherits from the Layer class. We refer to Layers that inherit directly from ContainerLayer as container Layers, which can contain any number of child Layers.

• PictureLayer: A Layer that stores drawing products, inheriting directly from the Layer class. Layers that can directly hold (or associate) drawing results are called drawing Layers.

3-2. Container Layer

What are the functions and specific use cases of container Layers?

• They form a tree structure of the drawing structure of the component tree.

• Since Widgets in Flutter are tree-structured, the corresponding RenderObjects should also have a tree structure. Flutter generates a Layer tree for the component tree based on certain "specific rules" (to be explained later), and container Layers can form a tree structure (a parent Layer can contain any number of child Layers, and child Layers can contain any number of sub-Layers).

Container Layers can apply transformations to their child Layers as a whole.

Container Layers can apply certain transformation effects to their child Layers, such as clipping effects (ClipRectLayer, ClipRRectLayer, ClipPathLayer), filtering effects (ColorFilterLayer, ImageFilterLayer), matrix transformations (TransformLayer), and opacity transformations (OpacityLayer), etc.

Although ContainerLayer is not an abstract class and developers can create instances of ContainerLayer, this is rarely done. Instead, when a ContainerLayer is needed, its subclasses are used. For instance, in the current Flutter source code, the author found no direct creation of the ContainerLayer class. If no transformation effects are needed, then OffsetLayer can be used without worrying about additional performance overhead, as its underlying (Skia) implementation is very efficient.

Convention: In subsequent discussions about ContainerLayer, unless specified otherwise, it may refer to any container class component. Since we typically do not create instances of ContainerLayer directly, there should be no ambiguity.

3-3. Drawing Layer

Now let’s focus on the PictureLayer class, which is the most commonly used drawing Layer in Flutter.

We know that what is ultimately displayed on the screen is bitmap information, which is generated by the Canvas API. In fact, the drawing product of the Canvas is represented by a Picture object, and in the current version of Flutter, only PictureLayer has a picture object. In other words, the drawing results of components that render themselves and their child nodes through Canvas will ultimately fall into PictureLayer.

Exploration: There are two other Layer classes in Flutter: TextureLayer and PlatformViewLayer. Readers can research their functions and applicable scenarios on their own.

3-4. Choice of Implementation Method for Transformation Effects

As mentioned, ContainerLayer can apply transformations to its child Layers. In fact, most UI systems’ Canvas APIs have transformation-related APIs, which means some transformation effects can be implemented via both ContainerLayer and Canvas. For instance, to achieve a translation transformation, we can use either OffsetLayer or the Canvas.translate API directly. If so, what is the principle behind choosing the implementation method?

First, let’s understand the underlying principles of transformation effects implemented by container Layers. The transformations in container Layers are implemented through Skia at a low level, without needing Canvas to handle them. Specifically, container Layers with transformation functionality correspond to a Layer in the Skia engine. To distinguish them from Flutter framework Layers, Skia’s Layers are called engine layers. When a container Layer with transformation functionality is added to the Scene, an engine layer is constructed beforehand. Taking OffsetLayer as an example, let’s look at its related implementation:

@override
void addToScene(ui.SceneBuilder builder, [Offset layerOffset = Offset.zero]) {
  // Construct the engine layer
  engineLayer = builder.pushOffset(
    layerOffset.dx + offset.dx,
    layerOffset.dy + offset.dy,
    oldLayer: _engineLayer as ui.OffsetEngineLayer?,
  );
  addChildrenToScene(builder);
  builder.pop();
}

The functionality of OffsetLayer to apply an overall offset transformation to its child nodes is supported by Skia. Skia can handle multi-layer rendering, but having more layers is not always better; engine layers do consume certain resources. In Flutter's component library, transformation effects are prioritized to be implemented using Canvas. ContainerLayers are only used when implementing through Canvas becomes very difficult or impossible.

So, in what scenarios would implementing transformation effects through Canvas be very challenging, necessitating the use of ContainerLayer? A typical scenario is when we need to apply a transformation to an entire subtree within the component tree that contains multiple PictureLayer instances. This is because a Canvas often corresponds to a PictureLayer, and different Canvases are isolated from each other. Only when all components in the subtree are drawn using the same Canvas can we apply transformations to all child nodes through that Canvas; otherwise, we have to use ContainerLayer. We will discuss when child nodes will reuse the same PictureLayer and when new PictureLayer instances will be created in the next section.

Note: The Canvas object also has APIs related to layers, such as Canvas.saveLayer, but these have different meanings than the Layers discussed in this section. The layers in the Canvas primarily provide a means to cache intermediate drawing results during the drawing process, designed to facilitate separation of multiple drawing elements when rendering complex objects. Readers can refer to relevant documentation for more information on Canvas layer-related APIs. We can simply understand that regardless of how many layers the Canvas creates, they all reside within the same PictureLayer (of course, the specific underlying implementation of Canvas APIs is ultimately determined by the Flutter team, but as application developers, this level of understanding is sufficient).

With this foundational knowledge, we can now explore the drawing process of the component tree in the Flutter framework in the next section.