Merge branch '2022-08-11-verified-boot-for-embedded-initial-support'

To quote Simon:
This adds the concept of a VBE method to U-Boot, along with an
implementation of the 'VBE simple' method, basically a simple way of
updating firmware in MMC from userspace and monitoring it from U-Boot.

VBE simple is implemented in fwupd. U-Boot's role is to set up the
device tree with the required firmware-update properties and provide the
developer with information about the current VBE state. To that end this
series includes a new 'vbe' command that allows VBE methods to be listed
and examined.

As part of this work, support for doing FDT fixups via the event interface
is provided, along with the ability to write to the device tree via the
ofnode interface.

Another (significant) change is that bootmeths now have a 'global' flag,
to allow the implementation of EFI bootmgr (and VBE) to be cleaned up.
The 'system' bootdev is no-longer needed and these bootmeths are scanned
first.

Further work is needed to pull everything together, but this is a step
along the way.

4 files changed, 146 insertions, 30 deletions

diff --git a/doc/develop/bootstd.rst b/doc/develop/bootstd.rst
index 5e9c0d282bb..b8773f8339d 100644
--- a/doc/develop/bootstd.rst
+++ b/doc/develop/bootstd.rst
@@ -32,6 +32,7 @@ way to boot with U-Boot. The feature is extensible to different Operating
 Systems (such as Chromium OS) and devices (beyond just block and network
 devices). It supports EFI boot and EFI bootmgr too.
 
+Finally, standard boot supports the operation of :doc:`vbe`.
 
 Bootflow
 --------
@@ -89,6 +90,12 @@ bootflows.
 Note: it is possible to have a bootmeth that uses a partition or a whole device
 directly, but it is more common to use a filesystem.
 
+Note that some bootmeths are 'global', meaning that they select the bootdev
+themselves. Examples include VBE and EFI boot manager. In this case, they
+provide a `read_bootflow()` method which checks whatever bootdevs it likes, then
+returns the bootflow, if found. Some of these bootmeths may be very slow, if
+they scan a lot of devices.
+
 
 Boot process
 ------------
@@ -112,6 +119,9 @@ the following command::
 which scans for available bootflows, optionally listing each find it finds (-l)
 and trying to boot it (-b).
 
+When global bootmeths are available, these are typically checked before the
+above bootdev scanning.
+
 
 Controlling ordering
 --------------------
@@ -269,18 +279,8 @@ Standard boot requires a single instance of the bootstd device to make things
 work. This includes global information about the state of standard boot. See
 `struct bootstd_priv` for this structure, accessed with `bootstd_get_priv()`.
 
-Within the devicetree, if you add bootmeth devices or a system bootdev, they
-should be children of the bootstd device. See `arch/sandbox/dts/test.dts` for
-an example of this.
-
-
-The system bootdev
-------------------
-
-Some bootmeths don't operate on individual bootdevs, but on the whole system.
-For example, the EFI boot manager does its own device scanning and does not
-make use of the bootdev devices. Such bootmeths can make use of the system
-bootdev, typically considered last, after everything else has been tried.
+Within the devicetree, if you add bootmeth devices, they should be children of
+the bootstd device. See `arch/sandbox/dts/test.dts` for an example of this.
 
 
 .. _`Automatic Devices`:
@@ -291,12 +291,11 @@ Automatic devices
 It is possible to define all the required devices in the devicetree manually,
 but it is not necessary. The bootstd uclass includes a `dm_scan_other()`
 function which creates the bootstd device if not found. If no bootmeth devices
-are found at all, it creates one for each available bootmeth driver as well as a
-system bootdev.
+are found at all, it creates one for each available bootmeth driver.
 
 If your devicetree has any bootmeth device it must have all of them that you
-want to use, as well as the system bootdev if needed, since no bootmeth devices
-will be created automatically in that case.
+want to use, since no bootmeth devices will be created automatically in that
+case.
 
 
 Using devicetree
@@ -347,6 +346,7 @@ Bootmeth drivers are provided for:
    - distro boot from a disk (syslinux)
    - distro boot from a network (PXE)
    - EFI boot using bootefi
+   - VBE
    - EFI boot using boot manager
 
 
@@ -433,18 +433,23 @@ case, the iterator ends up with a `dev_order` array containing the bootdevs that
 are going to be used, with `num_devs` set to the number of bootdevs and
 `cur_dev` starting at 0.
 
-Next, the ordering of bootdevs is determined, by `bootmeth_setup_iter_order()`.
+Next, the ordering of bootmeths is determined, by `bootmeth_setup_iter_order()`.
 By default the ordering is again by sequence number, i.e. the `/aliases` node,
 or failing that the order in the devicetree. But the `bootmeth order` command
 or `bootmeths` environment variable can be used to set up an ordering. If that
 has been done, the ordering is in `struct bootstd_priv`, so that ordering is
 simply copied into the iterator. Either way, the `method_order` array it set up,
-along with `num_methods`. Then `cur_method` is set to 0.
+along with `num_methods`.
+
+Note that global bootmeths are always put at the end of the ordering. If any are
+present, `cur_method` is set to the first one, so that global bootmeths are done
+first. Once all have been used, these bootmeths are dropped from the iteration.
+When there are no global bootmeths, `cur_method` is set to 0.
 
 At this point the iterator is ready to use, with the first bootdev and bootmeth
-selected. All the other fields are 0. This means that the current partition is
-0, which is taken to mean the whole device, since partition numbers start at 1.
-It also means that `max_part` is 0, i.e. the maximum partition number we know
+selected. Most of the other fields are 0. This means that the current partition
+is 0, which is taken to mean the whole device, since partition numbers start at
+1. It also means that `max_part` is 0, i.e. the maximum partition number we know
 about is 0, meaning that, as far as we know, there is no partition table on this
 bootdev.
 
@@ -455,6 +460,10 @@ If the `BOOTFLOWF_ALL` iterator flag is set, even errors are returned as
 incomplete bootflows, but normally an error results in moving onto the next
 iteration.
 
+Note that `bootflow_check()` handles global bootmeths explicitly, but calling
+`bootmeth_get_bootflow()` on each one. The `doing_global` flag indicates when
+the iterator is in that state.
+
 The `bootflow_scan_next()` function handles moving onto the next iteration and
 checking it. In fact it sits in a loop doing that repeatedly until it finds
 something it wants to return.
@@ -473,9 +482,10 @@ the least-sigificant digit on the right, counting like this:
    0           0          2
    0           1          0
    0           1          1
-   0           1          1
+   0           1          2
    1           0          0
    1           0          1
+   ...
    ========    =======    =======
 
 The maximum value for `method` is `num_methods - 1` so when it exceeds that, it
@@ -487,6 +497,31 @@ exceeds its maximum, then the next bootdev is used. In this way, iter_incr()
 works its way through all possibilities, moving forward one each time it is
 called.
 
+Note that global bootmeths introduce a subtlety into the above description.
+When `doing_global` is true, the iteration takes place only among the bootmeths,
+i.e. the last column above. The global bootmeths are at the end of the list.
+Assuming that they are entries 3 and 4 in the list, the iteration then looks
+like this:
+
+   ========    =======    =======   =======================================
+   bootdev     part       method    notes
+   ========    =======    =======   =======================================
+   .           .          3         doing_global = true, method_count = 5
+   .           .          4
+   0           0          0         doing_global = false, method_count = 3
+   0           0          1
+   0           0          2
+   0           1          0
+   0           1          1
+   0           1          2
+   1           0          0
+   1           0          1
+   ...
+   ========    =======    =======   =======================================
+
+The changeover of the value of `doing_global` from true to false is handled in
+`iter_incr()` as well.
+
 There is no expectation that iteration will actually finish. Quite often a
 valid bootflow is found early on. With `bootflow scan -b`, that causes the
 bootflow to be immediately booted. Assuming it is successful, the iteration never
@@ -516,17 +551,19 @@ method `bootdev_get_bootflow()` to ask the bootdev to return a bootflow. It
 passes the iterator to the bootdev method, so that function knows what we are
 talking about. At first, the bootflow is set up in the state `BOOTFLOWST_BASE`,
 with just the `method` and `dev` intiialised. But the bootdev may fill in more,
-e.g. updating the state, depending on what it finds.
+e.g. updating the state, depending on what it finds. For global bootmeths the
+`bootmeth_get_bootflow()` function is called instead of
+`bootdev_get_bootflow()`.
 
-Based on what the bootdev responds with, `bootflow_check()` either
+Based on what the bootdev or bootmeth responds with, `bootflow_check()` either
 returns a valid bootflow, or a partial one with an error. A partial bootflow
 is one that has some fields set up, but did not reach the `BOOTFLOWST_READY`
 state. As noted before, if the `BOOTFLOWF_ALL` iterator flag is set, then all
 bootflows are returned, even partial ones. This can help with debugging.
 
 So at this point you can see that total control over whether a bootflow can
-be generated from a particular iteration, or not, rests with the bootdev.
-Each one can adopt its own approach.
+be generated from a particular iteration, or not, rests with the bootdev (or
+global bootmeth). Each one can adopt its own approach.
 
 Going down a level, what does the bootdev do in its `get_bootflow()` method?
 Let us consider the MMC bootdev. In that case the call to
diff --git a/doc/develop/driver-model/livetree.rst b/doc/develop/driver-model/livetree.rst
index 9f654f3b894..faf3eb5b5f0 100644
--- a/doc/develop/driver-model/livetree.rst
+++ b/doc/develop/driver-model/livetree.rst
@@ -211,9 +211,61 @@ using it in new code.
 Modifying the livetree
 ----------------------
 
-This is not currently supported. Once implemented it should provide a much
-more efficient implementation for modification of the device tree than using
-the flat tree.
+This is supported in a limited way, with ofnode_write_prop() and related
+functions.
+
+The unflattening algorithm results in a single block of memory being
+allocated for the whole tree. When writing new properties, these are
+allocated new memory outside that block. When the block is freed, the
+allocated properties remain. This can result in a memory leak.
+
+The solution to this leak would be to add a flag for properties (and nodes when
+support is provided for adding those) that indicates that they should be
+freed. Then the tree can be scanned for these 'separately allocated' nodes and
+properties before freeing the memory block.
+
+The ofnode_write\_...() functions also support writing to the flat tree. Care
+should be taken however, since this can change the position of node names and
+properties in the flat tree, thus affecting the live tree. Generally this does
+not matter, since when we fire up the live tree we don't ever use the flat tree
+again. But in the case of tests, this can cause a problem.
+
+The sandbox tests typically run with OF_LIVE enabled but with the actual live
+tree either present or absent. This is to make sure that the flat tree functions
+work correctly even with OF_LIVE is enabled. But if a test modifies the flat
+device tree, then the live tree can become invalid. Any live tree tests that run
+after that point will use a corrupted tree, e.g. with an incorrect property name
+or worse. To deal with this we use a flag UT_TESTF_LIVE_OR_FLAT then ensures
+that tests which write to the flat tree are not run if OF_LIVE is enabled. Only
+the live tree version of the test is run, when OF_LIVE is enabled, with
+sandbox_flattree running the flat tree version.
+
+This is of course a work-around, even if a reasonable one. One solution to this
+problem would be to make a copy of the flat tree before the test and restore it
+afterwards, in the same memory location, so that the live tree pointers work
+again. Another would be to regenerate the live tree if a test modified the flat
+tree.
+
+Neither of these solutions is currently implemented, since the situation that
+causes the problem can only occur in sandbox tests, is somewhat esoteric and
+the UT_TESTF_LIVE_OR_FLAT flag deals with it in a reasonable way.
+
+
+Multiple livetrees
+------------------
+
+The livetree implementation was originally designed for use with the control
+FDT. This means that the FDT fix-ups (ft_board_setup() and the like, must use
+a flat tree.
+
+It would be helpful to use livetree for fixups, since adding a lot of nodes and
+properties would involve less memory copying and be more efficient. As a step
+towards this, an `oftree` type has been introduced. It is normally set to
+oftree_default() but can be set to other values. Eventually this should allow
+the use of FDT fixups using the ofnode interface, instead of the low-level
+libfdt one.
+
+See dm_test_ofnode_root() for some examples.
 
 
 Internal implementation
@@ -281,6 +333,6 @@ Live tree support was introduced in U-Boot 2017.07. There is still quite a bit
 of work to do to flesh this out:
 
 - tests for all access functions
-- support for livetree modification
+- more support for livetree modification
 - addition of more access functions as needed
 - support for livetree in SPL and before relocation (if desired)
diff --git a/doc/develop/index.rst b/doc/develop/index.rst
index 7c41e3f1b6e..c94c7fe0efa 100644
--- a/doc/develop/index.rst
+++ b/doc/develop/index.rst
@@ -39,6 +39,7 @@ Implementation
    smbios
    spl
    uefi/index
+   vbe
    version
 
 Debugging
diff --git a/doc/develop/vbe.rst b/doc/develop/vbe.rst
new file mode 100644
index 00000000000..8f147fd9360
--- /dev/null
+++ b/doc/develop/vbe.rst
@@ -0,0 +1,26 @@
+.. SPDX-License-Identifier: GPL-2.0+
+
+Verified Boot for Embedded (VBE)
+================================
+
+Introduction
+------------
+
+VBE provides a standard boot mechanism for embedded systems. If defines
+how firmware and Operating Systems are located, updated and verified.
+
+Within U-Boot, one or more VBE bootmeths implement the boot logic. For example,
+the vbe-simple bootmeth handles finding the firmware (e.g. in MMC) and starting
+it. Typically the bootmeth is started up in VPL and controls which SPL and
+U-Boot binaries are loaded.
+
+A 'vbe' command provides access to various aspects of VBE's operation, including
+listing methods and getting the status for a method.
+
+For a detailed overview of VBE, see vbe-intro_. A fuller description of
+bootflows is at vbe-bootflows_ and the firmware-update mechanism is described at
+vbe-fwupdate_.
+
+.. _vbe-intro: https://docs.google.com/document/d/e/2PACX-1vQjXLPWMIyVktaTMf8edHZYDrEvMYD_iNzIj1FgPmKF37fpglAC47Tt5cvPBC5fvTdoK-GA5Zv1wifo/pub
+.. _vbe-bootflows: https://docs.google.com/document/d/e/2PACX-1vR0OzhuyRJQ8kdeOibS3xB1rVFy3J4M_QKTM5-3vPIBNcdvR0W8EXu9ymG-yWfqthzWoM4JUNhqwydN/pub
+.. _vbe-fwupdate: https://docs.google.com/document/d/e/2PACX-1vTnlIL17vVbl6TVoTHWYMED0bme7oHHNk-g5VGxblbPiKIdGDALE1HKId8Go5f0g1eziLsv4h9bocbk/pub