This is useful when the OTA contains partition images that are signed
with the AOSP test keys.
Issue: #569
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
* flush() or into_inner() is now called for all BufWriters
* flush() is no longer called for Files, which is a no-op operation
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, some of the boot image patches did not sort the cpio entries
and (re)assign inode numbers. In practice, this makes no difference, but
it is still not the intended behavior.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
It turns out that even though newer versions of delta_generator set this
field for CoW v2 (unused), older versions did not.
Fixes: #493
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
The e2e checksums were updated because the new lzma-rust2 version has
slight differences in compression ratio.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
File reopening was conflating ownership of file-like types with the fact
that they support parallel reads and writes at arbitrary offsets.
The Reopen trait has now been replaced with ReadAt and WriteAt traits,
which are implemented for types that support parallel I/O. If a type
compatible with the standard Read/Write/Seek traits is needed, a new
UserPosFile type can act as the bridge by storing its own userspace file
offset. For the opposite bridge, there's MutexFile, which implements
ReadAt/WriteAt by using locks to make the operations sequential. This is
only really used in the tests though.
This eliminates the need for the PSeekFile and SharedCursor types. The
standard File and Cursor types can be used instead, and if shared
ownership is needed, Arc can be used.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
rawzip is a lower-level zip file library that is much more suited for
avbroot's use case. Its speed improvements aren't too important since
OTAs only have a handful of files, but it is a simpler layer of
abstraction and exposes more about zip file internals. We also no longer
need to maintain a perpetual fork of the zip library.
The only caveat is that rawzip (much like avbroot) is built around
writing zip files in a streaming fashion. To support `--zip-mode
seekable`, the output file is post-processed to copy the relevant data
descriptor fields to the local header. The unused data descriptors
remain in the file to avoid needing to shift file data, but this does
not violate the spec and Android's libziparchive accepts it just fine.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
There is not much benefit for our use case to have kernel-level
openat-style sandboxing of paths. We already check all untrusted paths
for safety and the sandboxing prevented the use of symlinks that point
outside of the parent directory of specified paths.
This commit also moves the path safety checks to the util module to
avoid having multiple implementations spread out everywhere.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
The performance, both in CPU time and compression ratios, is very
comparable to liblzma. This lets us drop the last remaining
compression-related dependency written in C.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, we computed the expected property files string (based on
AOSP's rules) and checked if the string in the OTA metadata was a
byte-for-byte match. This would fail for OTAs with strings that differ
from how AOSP generates them. This could be additional files or just
different ordering of the entries.
This commit changes the approach to just verify that the property file
entries is a valid subset of the zip file entries. We no longer try to
compute the expected value.
This does not change what avbroot generates when patching an OTA. Newly
generated property files strings always follow AOSP's rules.
Fixes: #469
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
The zip crate gained support for streaming writes in its master branch,
so we can finally upgrade from our ancient fork of it. The new
implementation is done a bit differently, with seekable writers having
the ZipWriter<W> type and streaming writers having the
ZipWriter<StreamWriter<W>> type. This forces us to add a new wrapper
type since we have to switch between them at runtime.
We still need to maintain a (hopefully temporary) fork of the crate due
to a few issues:
1. There's no way to get the original underlying writer instance back
after finalizing a streaming zip. A fix for this has been submitted
upstream:
https://github.com/zip-rs/zip2/pull/367
2. The streaming writes implementation does not include the magic
signature for data descriptors. While the zip spec says the magic
value is optional and parsers should not require it, older versions
of Android's libziparchive do. A fix for this has been submitted
upstream:
https://github.com/zip-rs/zip2/pull/368
3. There is currently no way to get the data offset of zip entries.
avbroot requires this to fill in the OTA metadata's "property files"
entries, which Android uses to read file data without parsing the zip
file structures.
This new zip update produces files that are slightly different to
before. The "version made by" and "version needed to extract" fields are
now set to their minimum possible values. Previously, the zip crate was
hardcoded to use versions 4.6 and 2.0, respectively.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Our previous limit was 15 bytes for the <offset>:<size> placeholder,
matching AOSP's ota_utils.py. Since the size of metadata.pb is almost
always 4 digits, this leaves 10 digits for the offset, which isn't
enough for large OTAs. AOSP never actually hits the limit because it
puts metadata and metadata.pb at the beginning of the output zip file.
We put the files at the end of the zip since we do streaming writes.
Fixes: #451
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This also fixes a number of disabled-by-default clippy warnings and
updates the Rust edition to 2024.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
It was hardcoded to set the CoW version to v2 in the payload manifest.
This commit also updates the pixel_v2 profile to disable VABC so that
scenario gets tested. The pixel_v3 profile now uses CoW v2 with gz.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
No known device uses this, but it's very useful for testing that our
overhead calculation is identical to AOSP's. A few more bugs were found
in our overhead calculation logic:
* The CowHeaderV3 size was missing the original CowHeader (v2) fields
that are supposed to be included due to inheritance in the C++ class.
* The additional 1% overhead was incorrectly calculated against the
initial CoW estimate before static overhead for CoW headers was added.
* The V3 num_ops estimation did not set a minimum of 25 to match
delta_generator.
* The V2 size estimation did not take into account that a cluster of CoW
operations cannot be truncated. It must be a multiple of cluster_ops
(200 for avbroot).
With these fixes, the CoW estimation when compression is disabled
matches AOSP exactly. This means all the overhead calculation is now
correct and the only difference when compression is enabled is in the
compression ratios of the various lz4/gz implementations.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
AOSP has long supported CoW version 3, but it wasn't used by the stock
OS on any Pixel devices until the new Pixel 9a.
CoW version 3 is fundamentally similar to version 2, though with
differences in the main header and how the operation headers are stored.
The compression is no longer done in fixed-size chunks equal to the
block size. Instead, the payload specifies a "compression factor", which
is the maximum chunk size to pass to the compressor. The actual chunk
size is the largest power of 2 <= the compression factor and the
remaining input size. Additionally, for version 3, the payload stores an
additional estimate_op_count_max field containing the number of CoW
operations.
While working on support for version 3, a few bugs in the version 2
estimation logic were found and fixed:
* The cluster_ops * sizeof(CowOperationV2) overhead incorrectly
assumed that cluster_ops was a constant 200 instead of the actual
number of CoW operations.
* The overhead did not account for kCowLabelOp headers, which
delta_generator emits once for every InstallOperation in the payload.
* The overhead did not account for kCowClusterOp headers, which batch
CoW operations into groups of 200.
* The overhead did not account for the CowFooter.
The version 3 overhead is much simpler and easier to calculate compared
to version 2.
Fixes: #441
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This reverts commit e929ecbe44.
Ring is back to being maintained again, so let's switch back to it since
it has fewer build dependencies and is much faster to compile.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Devices that launch with Android <14 generally use gzip as the CoW
compression algorithm. This never changes because future full OTAs
always need to be installable from the version of Android the device
launched with.
However, for users that don't care about the upgrade path from old
versions of Android, a new --vabc-algo option can be used to switch from
gz to lz4 compression. This can cut down the OTA installation time by
more than 2/3rds when installing via a custom OTA updater app. On my
Pixel Tablet, the installation time for the update_engine DOWNLOADING
phase decreased from 32:05 to 9:41. Note that this has absolutely no
effect on the performance when sideloading from recovery mode because
that does not use CoW.
When this new option is used, all dynamic partitions need to be
extracted from the OTA during patching so that the CoW estimates can be
recomputed. This will slow down the patching process and use up more
temporary disk space.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Instead of compressing the 64 MiB input in 2 MiB chunks, each loop
iteration was compressing the full 64 MiB. This massively slowed down
the patching process from seconds to potentially hours and would
temporarily waste a bunch of space during OTA installation.
Fixes: #433
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
The author of ring recently announced that the library is no longer
being maintained and fixes for security issues may be significantly
delayed. Big thanks to Brian Smith for creating and maintaining the
library for so long!
This commit replaces ring with aws-lc, a cryptography library maintained
by Amazon AWS. It seems to be well-regarded and is used by high-profile
projects like rustls. It is also API-compatible with ring, so it is
effectively a drop-in replacement.
Unfortunately, we still cannot switch back to the RustCrypto SHA1 and
SHA2 implementations because they are still significantly slower than
ring and aws-lc on systems that do not support the SHA-NI extensions.
https://rustsec.org/advisories/RUSTSEC-2025-0007
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
* There is now a stable release of bzip2-rs with the fix for both the C
and Rust versions of bzip2 being compiled.
* The zlib-rs deflate implementation is faster than the default
miniz_oxide. Changing this requires updating the checksums in the e2e
tests due to slight differences in compression levels between the two
implementations.
* Temporarily silence RUSTSEC-2025-0007 to avoid blocking CI. The ring
library is no longer maintained.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Implicit error propagation was originally used because it was convenient
and made it easy to just bubble up errors via the ? operator without
thinking. However, there have been too many situations where this
resulted in error messages that were completely useless in
troubleshooting the problem. "I/O error", even with a specific reason
attached, is useless where there are potentially hundreds of operations
where I/O can fail.
I no longer think implicit error propagation is a good idea, so this
commit removes every single use of #[from] in every custom error type.
There are now many more error variants, allowing more context to be
attached to the underlying errors.
Previously, it was easy to encounter error messages like:
Caused by:
0: Failed to patch payload: payload.bin
1: Failed to patch boot images: boot, init_boot, vendor_boot
2: Boot image error
3: I/O error
4: failed to fill whole buffer
This is a terrible error message because it doesn't mention which of the
3 boot images failed to parse, nor does it mention during which I/O
operation it encountered EOF. With this commit, this sort of information
is now included. For example, if the boot image happened to be truncated
in the middle of the ramdisk, the error message would now be:
Caused by:
0: Failed to patch payload: payload.bin
1: Failed to patch boot images: boot, init_boot, vendor_boot
2: Failed to load boot image: init_boot
3: Failed to read boot image data: Boot::V3::ramdisk
4: failed to fill whole buffer
Changes:
* Remove all uses of #[from] from thiserror-derived error types.
* Errors during parsing and serialization of RSA private keys, RSA
public keys, and X509 certificates now include the file path.
* Removed unnecessary uses of BufReader and BufWriter when reading and
writing RSA private keys, RSA public keys, and X509 certificates,
since they need to be fully read into memory anyway.
* Use ReadFixedSizeExt instead of read_exact() where possible.
* Use &'static str instead of String in error fields where all possible
values are known at compile-time to avoid unnecessary heap allocation.
* Use ok_or() instead of ok_or_else() to construct errors when the error
variant uses known data and does not require heap allocation.
* Using DebugString instead of String in error variants that store a
preformatted debug string.
While working on this commit, an unrelated bug was found and fixed:
* Fix vendor v4 boot images that were truncated within the padding
following the bootconfig section being treated as valid.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Neither of the two new fields need any special handling in avbroot. This
just allows us to preserve the values from the original OTA.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
On older devices, like the Pixel 4a, where `boot` is used for both
Android and recovery mode, the image will be patched by OtaCertPatcher
and PrepatchedImagePatcher. OtaCertPatcher was always set to run first,
so when PrepatchedImagePatcher used the user-supplied image as-is, prior
modifications got wiped out. This made is so users could no longer flash
further patched OTAs.
This is an unfortunate regression that was introduced in avbroot 2.0.0.
The e2e tests never caught this issue because the --prepatched test was
being fed the boot image previously patched by --magisk. That already
had valid certs so the result of OtaCertPatcher's modifications being
lost were not visible. This commit also fixes the e2e tests so that this
type of issue will be caught in the future.
Fixes: #356
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This commit adds a new `--zip-mode` parameter to `avbroot ota patch` to
control whether the patched OTA zip is written with data descriptors or
not. By default, the `streaming` mode is used, which matches the current
behavior where the zip is hashed for signing as it is being written. The
new `seekable` mode fully writes the zip before rereading it to hash the
contents.
The new mode is useful for devices with broken zip parsers that fail to
properly handle data descriptors.
All of the end-to-end tests have been duplicated to test both modes.
Adding the seekable mode necessitated a couple other changes:
* BufWriter is no longer used. Type erasure is very painful in Rust, so
we need to keep the writer types the same for both the streaming and
seekable modes. BufWriter is unusable in the seekable mode because we
need to be able to read back what was written, which isn't supported.
* HolePunchingWriter has been removed. It was a simple way to produce
sparse files by seeking whenever a write buffer consists fully of
zeros. When combined with BufWriter, there was previously never a
situation where this was undesirable. However, with the new seekable
mode and the zip library's pattern of writing one field at a time, the
final 2 zero bytes (representing an empty archive comment) is never
written and the file size is not increased either.
Removing this is not a big deal since we no longer use stripped OTAs
for the end-to-end tests. Those were really the only OTAs that
benefitted from sparse files. A real OTA has very few zero bytes due
to compression.
Issue: #328
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Older devices, like the Pixel 4a 5G (bramble) use gzip instead of lz4.
This commit also reworks the CoW size estimate calculation to add the
same constant headroom that AOSP's delta_generator adds. Previously,
avbroot was already adding an additional 1% to account for differences
in compression ratios across compression library implementations. This
papered over the issue for large partitions, but small partitions could
still have a CoW size estimate that's too small. Adding the constant
headroom prevents ENOSPC when flashing those partitions.
Fixes: #332
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Some devices have "full" OTAs where the payload is missing the recovery
partition. These subcommands make it possible to manually add back the
missing image. Given the strict requirements for how the OTA zip is laid
out and signed, users can't just replace payload.bin in a zip and call
it a day, but it's sufficient for feeding a modified input to
`avbroot ota patch`.
Issue: #328
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
By default, the helper program is invoked in a way that is compatible
with avbtool's --signing_helper. However, the arguments have been
extended slightly to allow passing in the passphrase file or environment
variable for non-interactive use.
Fixes: #310
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
lz4_flex appears to compress system images better than the original lz4
implementation used in libsnapshot_cow, so the estimates are too low.
Fixes: #306
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Otherwise, if the partition size increases or the data becomes more
incompressible, update_engine might fail to flash the partition due to
the CoW block device running out of space.
Since all known VABC-enabled OTAs in the wild currently use CoW v2 with
lz4 compression, this is the only configuration we support. CoW v3 also
exists in AOSP's libsnapshot_cow, but is much more complicated to
implement and is not yet used, even in the Android 15 beta OTAs.
Fixes: #306
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, for resizable images, the user had to guess an appropriate
size for the final image that could fit all the AVB metadata. This
commit adds a new `--recompute-size` option to the `avb pack` subcommand
to generate a minimally sized image.
Fixes: #294
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This allows the user to boot GSIs signed by the same key. The option is
disabled by default because some Android builds disable DSU support by
removing all keys to reduce the attack surface. We don't want to
reenable DSU support on these builds unless the user asks for it.
Closes: #286
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Instead of println'ing everything, this commit switches the code base to
using the tracing library. There are now proper log levels and multiple
logging output formats. A bunch of new debug and trace-level messages
have also been added to help with future troubleshooting.
By default, the output is kept nice and short. Spans won't be shown
unless the log level is set to debug or lower.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Upstream Magisk now xz-compresses files in modifies in the ramdisk. This
commit also implements the same in avbroot's MagiskRootPatcher.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This commit replaces the previous approach of patching real OTAs with
patching mock OTAs. The motivation for this change is to make it
possible to test the system partition otacerts.zip patching without
needing to download huge files. Adding the system image to the stripped
OTAs would increase the file size by an order of magnitude.
The mock OTAs are generated from a set of profiles defined in e2e.toml.
The four included profiles are meant to mimic the OTAs used for testing
before:
* pixel_v4_gki ~= cheetah
* pixel_v4_non_gki ~= bluejay
* pixel_v3 ~= bramble
* pixel_v2 ~= sunfish
There is no equivalent profile for ossi because newer OnePlus devices no
longer support custom signing keys properly.
The mock OTAs are perfectly valid, structure and signature-wise. They
just don't include any real partition data where possible. They are
initially signed with a different set of keys to ensure that the changes
made by the patching process are actually visible.
With how small the mock OTAs are, testing every profile only takes about
two seconds. Thus, the Github Actions workflow was adjusted to just run
e2e in the same job as the build.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, overriding otacerts.zip in the system partition required the
user to flash a Magisk/KernelSU module that would bind mount over the
file during boot. While this worked well enough, it's insufficient for
unrooted setups, which has become more important since unrooting is the
only safe way to use the new OEM repair mode feature. With the stock
otacerts.zip, the OEM's default OTA updater app could run and install an
OS upgrade that's not signed by the user's key.
With this commit, the raw otacerts.zip bytes in the system partition are
directly replaced with a new zip that contains the user's certificate.
This method was inspired by @pascallj's comment in #216 suggesting
intentionally corrupting the otacerts.zip data in the filesystem.
Because avbroot does not have filesystem parsers for ext4/f2fs/erofs, we
rely on a heuristic-based search on the raw filesystem image. The file
is always smaller than one block (which is at least 4096 bytes on all
known devices), so the file data is stored contiguously on disk and in
the case of erofs, won't be compressed. None of the three filesystems
are copy-on-write and thus, have no filesystem-level data checksums. For
the dm-verity layer one level up, avbroot already knows how to recompute
the hash tree and FEC data.
To ensure that there are no false positives, any match that the search
finds must correctly parse as a valid zip and every entry within the zip
must have a filename that ends in .x509.pem. This matches what
update_engine expects from a proper otacerts.zip file.
Since the new approach is doing a raw search and replace, the old and
new files must have the same size. When the new zip is smaller, null
bytes are added to the zip archive comment field to pad to the correct
size. When the new zip is larger, avbroot will attempt the following to
try and make the file size smaller:
1. Enable zip deflate compression
2. Strip the X.509 signature from the certificate
3. Clear out the issuer RDN sequence from the certificate
4. Clear out the subject RDN sequence from the certificate
The latter three changes work because Android never performs any PKI
operations with the certificate. There is no CA certificate chain. The
X.509 certificate file is nothing more than a way to transport an RSA
public key.
avbroot requires the user's key to be RSA 4096. If the original zip had
the same key size, then none of these shrinking methods are needed. If
it contained an RSA 2048 key, then the first two modifications are
usually sufficient. The latter two modifications should only be needed
if the user picked a really long subject value when generating the
certificate.
With these new changes, the OTA patching time will approximately double
on a system with an SSD and modern CPU. This is dominated by the time it
takes to XZ-compress the system partition image. The compression is
already parallelized and scales linearly with the number of cores.
There's likely not much more that can be done to further speed this up.
Finally, these new changes are currently excluded from the e2e tests
because including the system partition in the stripped OTAs would
increase the file size by an order of magnitude. This could potentially
be solved in the future by generating our own small OTAs to use for
testing instead of running against real device OTAs.
Fixes: #225
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
During patching, all boot images are now extracted and the individual
patchers can inspect them to determine which ones need modifications.
This replaces the previous mechanism of detecting which boot images to
patch based on the name alone.
With this new method, the --boot-partition and --otacerts-partitions
options are no longer needed. The former option is kept (but ignored
with a warning message) for backwards compatibility, but the latter is
completely removed because it never made it to a stable release.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
FEC data can be updated efficiently with "round" granularity when the
regions where the input file was modified are known.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Andrew Gunnerson