This is useful when the OTA contains partition images that are signed
with the AOSP test keys.
Issue: #569
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, some of the boot image patches did not sort the cpio entries
and (re)assign inode numbers. In practice, this makes no difference, but
it is still not the intended behavior.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
It turns out that even though newer versions of delta_generator set this
field for CoW v2 (unused), older versions did not.
Fixes: #493
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
The e2e checksums were updated because the new lzma-rust2 version has
slight differences in compression ratio.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
rawzip is a lower-level zip file library that is much more suited for
avbroot's use case. Its speed improvements aren't too important since
OTAs only have a handful of files, but it is a simpler layer of
abstraction and exposes more about zip file internals. We also no longer
need to maintain a perpetual fork of the zip library.
The only caveat is that rawzip (much like avbroot) is built around
writing zip files in a streaming fashion. To support `--zip-mode
seekable`, the output file is post-processed to copy the relevant data
descriptor fields to the local header. The unused data descriptors
remain in the file to avoid needing to shift file data, but this does
not violate the spec and Android's libziparchive accepts it just fine.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
The performance, both in CPU time and compression ratios, is very
comparable to liblzma. This lets us drop the last remaining
compression-related dependency written in C.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
The zip crate gained support for streaming writes in its master branch,
so we can finally upgrade from our ancient fork of it. The new
implementation is done a bit differently, with seekable writers having
the ZipWriter<W> type and streaming writers having the
ZipWriter<StreamWriter<W>> type. This forces us to add a new wrapper
type since we have to switch between them at runtime.
We still need to maintain a (hopefully temporary) fork of the crate due
to a few issues:
1. There's no way to get the original underlying writer instance back
after finalizing a streaming zip. A fix for this has been submitted
upstream:
https://github.com/zip-rs/zip2/pull/367
2. The streaming writes implementation does not include the magic
signature for data descriptors. While the zip spec says the magic
value is optional and parsers should not require it, older versions
of Android's libziparchive do. A fix for this has been submitted
upstream:
https://github.com/zip-rs/zip2/pull/368
3. There is currently no way to get the data offset of zip entries.
avbroot requires this to fill in the OTA metadata's "property files"
entries, which Android uses to read file data without parsing the zip
file structures.
This new zip update produces files that are slightly different to
before. The "version made by" and "version needed to extract" fields are
now set to their minimum possible values. Previously, the zip crate was
hardcoded to use versions 4.6 and 2.0, respectively.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Our previous limit was 15 bytes for the <offset>:<size> placeholder,
matching AOSP's ota_utils.py. Since the size of metadata.pb is almost
always 4 digits, this leaves 10 digits for the offset, which isn't
enough for large OTAs. AOSP never actually hits the limit because it
puts metadata and metadata.pb at the beginning of the output zip file.
We put the files at the end of the zip since we do streaming writes.
Fixes: #451
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
It was hardcoded to set the CoW version to v2 in the payload manifest.
This commit also updates the pixel_v2 profile to disable VABC so that
scenario gets tested. The pixel_v3 profile now uses CoW v2 with gz.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
No known device uses this, but it's very useful for testing that our
overhead calculation is identical to AOSP's. A few more bugs were found
in our overhead calculation logic:
* The CowHeaderV3 size was missing the original CowHeader (v2) fields
that are supposed to be included due to inheritance in the C++ class.
* The additional 1% overhead was incorrectly calculated against the
initial CoW estimate before static overhead for CoW headers was added.
* The V3 num_ops estimation did not set a minimum of 25 to match
delta_generator.
* The V2 size estimation did not take into account that a cluster of CoW
operations cannot be truncated. It must be a multiple of cluster_ops
(200 for avbroot).
With these fixes, the CoW estimation when compression is disabled
matches AOSP exactly. This means all the overhead calculation is now
correct and the only difference when compression is enabled is in the
compression ratios of the various lz4/gz implementations.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
AOSP has long supported CoW version 3, but it wasn't used by the stock
OS on any Pixel devices until the new Pixel 9a.
CoW version 3 is fundamentally similar to version 2, though with
differences in the main header and how the operation headers are stored.
The compression is no longer done in fixed-size chunks equal to the
block size. Instead, the payload specifies a "compression factor", which
is the maximum chunk size to pass to the compressor. The actual chunk
size is the largest power of 2 <= the compression factor and the
remaining input size. Additionally, for version 3, the payload stores an
additional estimate_op_count_max field containing the number of CoW
operations.
While working on support for version 3, a few bugs in the version 2
estimation logic were found and fixed:
* The cluster_ops * sizeof(CowOperationV2) overhead incorrectly
assumed that cluster_ops was a constant 200 instead of the actual
number of CoW operations.
* The overhead did not account for kCowLabelOp headers, which
delta_generator emits once for every InstallOperation in the payload.
* The overhead did not account for kCowClusterOp headers, which batch
CoW operations into groups of 200.
* The overhead did not account for the CowFooter.
The version 3 overhead is much simpler and easier to calculate compared
to version 2.
Fixes: #441
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Devices that launch with Android <14 generally use gzip as the CoW
compression algorithm. This never changes because future full OTAs
always need to be installable from the version of Android the device
launched with.
However, for users that don't care about the upgrade path from old
versions of Android, a new --vabc-algo option can be used to switch from
gz to lz4 compression. This can cut down the OTA installation time by
more than 2/3rds when installing via a custom OTA updater app. On my
Pixel Tablet, the installation time for the update_engine DOWNLOADING
phase decreased from 32:05 to 9:41. Note that this has absolutely no
effect on the performance when sideloading from recovery mode because
that does not use CoW.
When this new option is used, all dynamic partitions need to be
extracted from the OTA during patching so that the CoW estimates can be
recomputed. This will slow down the patching process and use up more
temporary disk space.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Instead of compressing the 64 MiB input in 2 MiB chunks, each loop
iteration was compressing the full 64 MiB. This massively slowed down
the patching process from seconds to potentially hours and would
temporarily waste a bunch of space during OTA installation.
Fixes: #433
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
* There is now a stable release of bzip2-rs with the fix for both the C
and Rust versions of bzip2 being compiled.
* The zlib-rs deflate implementation is faster than the default
miniz_oxide. Changing this requires updating the checksums in the e2e
tests due to slight differences in compression levels between the two
implementations.
* Temporarily silence RUSTSEC-2025-0007 to avoid blocking CI. The ring
library is no longer maintained.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This commit adds a new `--zip-mode` parameter to `avbroot ota patch` to
control whether the patched OTA zip is written with data descriptors or
not. By default, the `streaming` mode is used, which matches the current
behavior where the zip is hashed for signing as it is being written. The
new `seekable` mode fully writes the zip before rereading it to hash the
contents.
The new mode is useful for devices with broken zip parsers that fail to
properly handle data descriptors.
All of the end-to-end tests have been duplicated to test both modes.
Adding the seekable mode necessitated a couple other changes:
* BufWriter is no longer used. Type erasure is very painful in Rust, so
we need to keep the writer types the same for both the streaming and
seekable modes. BufWriter is unusable in the seekable mode because we
need to be able to read back what was written, which isn't supported.
* HolePunchingWriter has been removed. It was a simple way to produce
sparse files by seeking whenever a write buffer consists fully of
zeros. When combined with BufWriter, there was previously never a
situation where this was undesirable. However, with the new seekable
mode and the zip library's pattern of writing one field at a time, the
final 2 zero bytes (representing an empty archive comment) is never
written and the file size is not increased either.
Removing this is not a big deal since we no longer use stripped OTAs
for the end-to-end tests. Those were really the only OTAs that
benefitted from sparse files. A real OTA has very few zero bytes due
to compression.
Issue: #328
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Older devices, like the Pixel 4a 5G (bramble) use gzip instead of lz4.
This commit also reworks the CoW size estimate calculation to add the
same constant headroom that AOSP's delta_generator adds. Previously,
avbroot was already adding an additional 1% to account for differences
in compression ratios across compression library implementations. This
papered over the issue for large partitions, but small partitions could
still have a CoW size estimate that's too small. Adding the constant
headroom prevents ENOSPC when flashing those partitions.
Fixes: #332
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
lz4_flex appears to compress system images better than the original lz4
implementation used in libsnapshot_cow, so the estimates are too low.
Fixes: #306
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Otherwise, if the partition size increases or the data becomes more
incompressible, update_engine might fail to flash the partition due to
the CoW block device running out of space.
Since all known VABC-enabled OTAs in the wild currently use CoW v2 with
lz4 compression, this is the only configuration we support. CoW v3 also
exists in AOSP's libsnapshot_cow, but is much more complicated to
implement and is not yet used, even in the Android 15 beta OTAs.
Fixes: #306
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This allows the user to boot GSIs signed by the same key. The option is
disabled by default because some Android builds disable DSU support by
removing all keys to reduce the attack surface. We don't want to
reenable DSU support on these builds unless the user asks for it.
Closes: #286
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Upstream Magisk now xz-compresses files in modifies in the ramdisk. This
commit also implements the same in avbroot's MagiskRootPatcher.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This commit replaces the previous approach of patching real OTAs with
patching mock OTAs. The motivation for this change is to make it
possible to test the system partition otacerts.zip patching without
needing to download huge files. Adding the system image to the stripped
OTAs would increase the file size by an order of magnitude.
The mock OTAs are generated from a set of profiles defined in e2e.toml.
The four included profiles are meant to mimic the OTAs used for testing
before:
* pixel_v4_gki ~= cheetah
* pixel_v4_non_gki ~= bluejay
* pixel_v3 ~= bramble
* pixel_v2 ~= sunfish
There is no equivalent profile for ossi because newer OnePlus devices no
longer support custom signing keys properly.
The mock OTAs are perfectly valid, structure and signature-wise. They
just don't include any real partition data where possible. They are
initially signed with a different set of keys to ensure that the changes
made by the patching process are actually visible.
With how small the mock OTAs are, testing every profile only takes about
two seconds. Thus, the Github Actions workflow was adjusted to just run
e2e in the same job as the build.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
During patching, all boot images are now extracted and the individual
patchers can inspect them to determine which ones need modifications.
This replaces the previous mechanism of detecting which boot images to
patch based on the name alone.
With this new method, the --boot-partition and --otacerts-partitions
options are no longer needed. The former option is kept (but ignored
with a warning message) for backwards compatibility, but the latter is
completely removed because it never made it to a stable release.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, all replacement partition images (those that have been
patched or `--replace`d) were compressed as a whole, which would be very
slow for larger images. Instead, we'll split the images into 2 MiB
chunks and compress them in parallel. This more closely matches what
AOSP's payload_generator does and scales linearly with the number of CPU
cores. The compression is less efficient, but the file size generally
only increases by 10s of KiB.
This commit also reworks the implementation so that patched images are
stored in temp files instead of in memory again. In hindsight, doing
everything in memory only made things more complex and causes the memory
usage to blow up when doing things like `--replace system <path>`.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, the AVB `algorithm_type` field was unconditionally being set
to a value that is compatible with the AVB private key. However, for
indirectly-signed boot images, the value should be set to `None`. Pixel
bootloaders accept the incorrect value, but other devices' bootloaders
might not.
Issue: #186
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This way, archives with hard links can pass through `cpio unpack` and
`cpio pack`, even though there's no explicit support for hard links.
This also changes the trailer entry logic to not set an inode number.
AOSP's mkbootfs and magiskboot both start at 300000 and increment by one
for each entry, including the trailer. However, GNU cpio, bsdcpio,
busybox, and toybox all set it to 0, which makes more sense given that
it doesn't represent anything on disk.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
There are several things we had to work around with quick-protobuf, like
forcing no_std mode to use BTreeMaps and avoiding helper functions to
read and write non-size-delimited messages. In addition, the pb-rs code
generator doesn't support adding #[derive]s to enums and the existing
support for adding derives to structs is broken due to incorrect string
concatenation.
Prost doesn't have these limitations and bugs. I originally avoided it
because prost_build required the external `protoc` binary, but now that
the protox library exists, the code generation can be done entirely in
Rust without external tools.
Prost also fully supports adding custom attributes to structs, enums,
and fields. This will be helpful for future payload unpack and pack
commands where the payload manifest would have to be serialized to TOML.
The e2e checksums had to be updated because prost's serialized bytes on
the wire differ from quick-protobuf, despite having the same semantic
meaning. Since all checksums need to be updated anyway, the Magisk apk
and OTA images have all been updated to the latest versions.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This fixes `data_offset` being set for `ZERO` and `DISCARD` operations,
which prevents some images (eg. `ossi`) from being flashed due to
update_engine's strict field validation.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Why?
----
It was always my intention to write avbroot in a compiled language.
Python was a stop-gap solution since it was possible to use the various
tools and parsers from AOSP to make the initial prototyping and
implementation easier. However, doing so required a whole lot of hacks
since nearly all of the Python modules we use were intended to be used
as executables, not libraries, and they were definitely not meant to be
used outside of AOSP's code base.
Although the dependencies on AOSP code have been reduced over time,
working on the Python code is still frustrating. The majority of the
modules we use from both the standard library and external dependencies
are lacking type annotations. All of the Python language servers and
type checker tools I've used choked on them. There have been serveral
avbroot bugs in the past that wouldn't have happened with any
statically typed language.
The catalyst for me working on this recently was dealing with some
python-protobuf versions that wouldn't work with AOSP's pregenerated
protobuf bindings. When parsing protobuf messages, it would fail
with obscure runtime type errors. I need my projects to not feel
frustrating or else I'll just get burnt out.
Hence, the Rust rewrite. With fewer hacks this time! avbroot no longer
has any dependencies on external tools like openssl. I'll be providing
precompiled binaries for the three major desktop OS's, built by GitHub
Actions. avbroot will also be versioned now, starting at 2.0.0.
Whats new?
----------
* A new `avbroot ota verify` subcommand has been added to check that all
OTA and AVB related components have been properly hashed and signed.
This works for all OTA images, including stock ones.
* A couple new `avbroot avb` subcommands have been added for dumping
vbmeta header/footer information and verifying AVB signatures. These
are roughly equivalent to avbtool's `info_image` and `verify_image`
subcommands, though avbroot is about an order of magnitude faster than
the latter.
* A new set of `avbroot boot` subcommands have been added for packing
and unpacking boot images. It supports Android v0-v4 images and vendor
v3-v4 images. Repacking is lossless even when using deprecated fields,
like the boot image v4 VTS signature.
* A new `avbroot ramdisk` subcommand has been added for inspecting
the CPIO structure of ramdisks.
* A new set of `avbroot key` subcommands have been added for generating
signing keys so that it's no longer necessary to install openssl and
avbtool (though of course, keys generated by other tools remain fully
compatible).
* Since avbroot has a ton of CLI options, a new `avbroot completion`
subcommand has been added for generating tab-completion configs for
various shells (eg. bash, zsh, fish, powershell).
What was removed?
-----------------
Nothing :) The `patch` and `extract` subcommands have been moved under
`avbroot ota` and the `magisk-info` subcommand has been moved under
`avbroot boot`, but there are compatibility shims in place to keep all
the old commands working.
The command-line interface will remain backwards compatible for as long
as possible, even with new major releases. The Rust API, however, has no
backwards compatibility guarantees. I currently don't intend for
avbroot's "library" components to be used anywhere outside of Custota
and avbroot itself.
Performance
-----------
Due to having better access to low-level APIs (especially `pread` and
`pwrite`), nearly everything that can be multithreaded in avbroot is now
multithreaded. In addition, during the patching operation, everything
is done entirely in memory without temp files and the maximum memory
usage is still about 100MB lower than with the Python implementation.
The new implementation is bottlenecked by how fast a single CPU core can
calculate 3 SHA256 hashes of overlapping regions spanning the majority
of the OTA file. About 90% of the CPU time is spent calculating SHA256
hashes and another 5% or so performing XZ-compression.
Some numbers:
* Patching should take roughly 40%-70% of the time it took before.
* Extracting with `--all` should take roughly 10%-30% of the time it
took before.
Folks with x86_64 CPUs supporting SHA-NI extensions (eg. Intel 11th gen
and newer) should see even bigger improvements.
Reproducibility
---------------
The new implementation's output files are bit-for-bit identical when the
inputs are the same. However, they do not exactly match what the Python
implementation produced.
* The zip entries, aside from `metadata` and `metadata.pb`, are written
in sorted order.
* All zip entries are stored without compression.
* All zip entries are stored without additional metadata (eg.
modification timestamp).
* The OTA certificate, both in the OTA zip and in the recovery ramdisk's
`otacerts.zip`, goes through deserialization + serialization before
being written. Text in the certificate file before the header and
after the footer will be stripped out.
* The protobuf structures (payload header and OTA metadata) are
serialized differently. Protobuf has more than one way to encode the
same messages "on the wire". The Rust quick_protobuf library
serializes messages a bit differently than python-protobuf, but the
outputs are mutually compatible.
* XZ compression of modified partition images in the payload is now done
at compression level 0 instead of 6. This reduces the patching time by
several seconds at the cost of a couple MiB increase in file size.
* Ramdisks are now compressed with standard LZ4 instead of LZ4HC (high
compression mode). For our use case, the difference is <100 KiB, but
using standard LZ4 allows us to use a pure-Rust LZ4 library and makes
the compression step much faster.
* Older ramdisks compressed with gzip are slightly different due to a
different gzip implementation being used (flate2 vs. zlib). The two
implementations structure the gzip frames slightly differently, but
the output is identical when decompressed.
* Magisk's config file in the ramdisk (`.backup/.magisk`) will have the
`SHA1` field set to all zeros. This allows avbroot to keep track of
less information during patching for better performance. The field is
only used for Magisk's uninstall feature, which can't ever be used in
a locked bootloader setup anyway.
Misc
----
While working on the new `avbroot ota verify` subcommand, I found that
the `ossi` stock image (OnePlus 10 Pro) used in avbroot's tests has an
invalid vbmeta hash for the `odm` partition. I thought it was an avbroot
bug, but AOSP's avbtool reports the same invalid hash too. If that image
actually boots, then I'm not sure AVB can be trusted on those devices...
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Andrew Gunnerson