Upstream Magisk now xz-compresses files in modifies in the ramdisk. This
commit also implements the same in avbroot's MagiskRootPatcher.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
We'll switch to using `fastboot flashall` in the future once that has
been implemented and tested.
Fixes: #252
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This commit replaces the previous approach of patching real OTAs with
patching mock OTAs. The motivation for this change is to make it
possible to test the system partition otacerts.zip patching without
needing to download huge files. Adding the system image to the stripped
OTAs would increase the file size by an order of magnitude.
The mock OTAs are generated from a set of profiles defined in e2e.toml.
The four included profiles are meant to mimic the OTAs used for testing
before:
* pixel_v4_gki ~= cheetah
* pixel_v4_non_gki ~= bluejay
* pixel_v3 ~= bramble
* pixel_v2 ~= sunfish
There is no equivalent profile for ossi because newer OnePlus devices no
longer support custom signing keys properly.
The mock OTAs are perfectly valid, structure and signature-wise. They
just don't include any real partition data where possible. They are
initially signed with a different set of keys to ensure that the changes
made by the patching process are actually visible.
With how small the mock OTAs are, testing every profile only takes about
two seconds. Thus, the Github Actions workflow was adjusted to just run
e2e in the same job as the build.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
When the original payload extents are all in order and have no gaps, we
can efficiently copy unmodified chunks of the system image from the
original payload into the new payload. Only the chunks containing the
modified regions (`otacerts.zip`, hash tree, FEC data, and AVB metadata)
need to be recompressed. This massively reduces the CPU usage since
usually only <20 MiB need to be recompressed.
If the conditions for the optimized path aren't satisfied (eg. extents
aren't in order or `--replace` is used), then it falls back to splitting
and compressing the entire system image.
This fixes the performance issues introduced in #240. This is probably
the best that we can do given that we now always patch the system image.
Issue: #225
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, overriding otacerts.zip in the system partition required the
user to flash a Magisk/KernelSU module that would bind mount over the
file during boot. While this worked well enough, it's insufficient for
unrooted setups, which has become more important since unrooting is the
only safe way to use the new OEM repair mode feature. With the stock
otacerts.zip, the OEM's default OTA updater app could run and install an
OS upgrade that's not signed by the user's key.
With this commit, the raw otacerts.zip bytes in the system partition are
directly replaced with a new zip that contains the user's certificate.
This method was inspired by @pascallj's comment in #216 suggesting
intentionally corrupting the otacerts.zip data in the filesystem.
Because avbroot does not have filesystem parsers for ext4/f2fs/erofs, we
rely on a heuristic-based search on the raw filesystem image. The file
is always smaller than one block (which is at least 4096 bytes on all
known devices), so the file data is stored contiguously on disk and in
the case of erofs, won't be compressed. None of the three filesystems
are copy-on-write and thus, have no filesystem-level data checksums. For
the dm-verity layer one level up, avbroot already knows how to recompute
the hash tree and FEC data.
To ensure that there are no false positives, any match that the search
finds must correctly parse as a valid zip and every entry within the zip
must have a filename that ends in .x509.pem. This matches what
update_engine expects from a proper otacerts.zip file.
Since the new approach is doing a raw search and replace, the old and
new files must have the same size. When the new zip is smaller, null
bytes are added to the zip archive comment field to pad to the correct
size. When the new zip is larger, avbroot will attempt the following to
try and make the file size smaller:
1. Enable zip deflate compression
2. Strip the X.509 signature from the certificate
3. Clear out the issuer RDN sequence from the certificate
4. Clear out the subject RDN sequence from the certificate
The latter three changes work because Android never performs any PKI
operations with the certificate. There is no CA certificate chain. The
X.509 certificate file is nothing more than a way to transport an RSA
public key.
avbroot requires the user's key to be RSA 4096. If the original zip had
the same key size, then none of these shrinking methods are needed. If
it contained an RSA 2048 key, then the first two modifications are
usually sufficient. The latter two modifications should only be needed
if the user picked a really long subject value when generating the
certificate.
With these new changes, the OTA patching time will approximately double
on a system with an SSD and modern CPU. This is dominated by the time it
takes to XZ-compress the system partition image. The compression is
already parallelized and scales linearly with the number of cores.
There's likely not much more that can be done to further speed this up.
Finally, these new changes are currently excluded from the e2e tests
because including the system partition in the stripped OTAs would
increase the file size by an order of magnitude. This could potentially
be solved in the future by generating our own small OTAs to use for
testing instead of running against real device OTAs.
Fixes: #225
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
During patching, all boot images are now extracted and the individual
patchers can inspect them to determine which ones need modifications.
This replaces the previous mechanism of detecting which boot images to
patch based on the name alone.
With this new method, the --boot-partition and --otacerts-partitions
options are no longer needed. The former option is kept (but ignored
with a warning message) for backwards compatibility, but the latter is
completely removed because it never made it to a stable release.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
* Refactor hash tree computation to work on a preallocated hash tree
buffer. This makes it possible to partially update a hash tree, which
is now supported.
* Add new subcommands for working with hash trees. There's no standard
header format for dm-verity information, so these commands write hash
tree files with a custom header. The commands are not really useful
outside of debugging avbroot's hash tree implementation.
Using AVB was considered, but it has no support for the hash tree data
living in a separate file from the input. If other parties agree on a
standard header in the future, avbroot will switch to that format.
* Add tests for the hash tree implementation.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
FEC data can be updated efficiently with "round" granularity when the
regions where the input file was modified are known.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, when running `avbroot ota verify`, only the payload chunk
hashes were verified.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, all replacement partition images (those that have been
patched or `--replace`d) were compressed as a whole, which would be very
slow for larger images. Instead, we'll split the images into 2 MiB
chunks and compress them in parallel. This more closely matches what
AOSP's payload_generator does and scales linearly with the number of CPU
cores. The compression is less efficient, but the file size generally
only increases by 10s of KiB.
This commit also reworks the implementation so that patched images are
stored in temp files instead of in memory again. In hindsight, doing
everything in memory only made things more complex and causes the memory
usage to blow up when doing things like `--replace system <path>`.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Andrew Gunnerson