The zip crate gained support for streaming writes in its master branch,
so we can finally upgrade from our ancient fork of it. The new
implementation is done a bit differently, with seekable writers having
the ZipWriter<W> type and streaming writers having the
ZipWriter<StreamWriter<W>> type. This forces us to add a new wrapper
type since we have to switch between them at runtime.
We still need to maintain a (hopefully temporary) fork of the crate due
to a few issues:
1. There's no way to get the original underlying writer instance back
after finalizing a streaming zip. A fix for this has been submitted
upstream:
https://github.com/zip-rs/zip2/pull/367
2. The streaming writes implementation does not include the magic
signature for data descriptors. While the zip spec says the magic
value is optional and parsers should not require it, older versions
of Android's libziparchive do. A fix for this has been submitted
upstream:
https://github.com/zip-rs/zip2/pull/368
3. There is currently no way to get the data offset of zip entries.
avbroot requires this to fill in the OTA metadata's "property files"
entries, which Android uses to read file data without parsing the zip
file structures.
This new zip update produces files that are slightly different to
before. The "version made by" and "version needed to extract" fields are
now set to their minimum possible values. Previously, the zip crate was
hardcoded to use versions 4.6 and 2.0, respectively.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Our previous limit was 15 bytes for the <offset>:<size> placeholder,
matching AOSP's ota_utils.py. Since the size of metadata.pb is almost
always 4 digits, this leaves 10 digits for the offset, which isn't
enough for large OTAs. AOSP never actually hits the limit because it
puts metadata and metadata.pb at the beginning of the output zip file.
We put the files at the end of the zip since we do streaming writes.
Fixes: #451
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This also fixes a number of disabled-by-default clippy warnings and
updates the Rust edition to 2024.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
It was hardcoded to set the CoW version to v2 in the payload manifest.
This commit also updates the pixel_v2 profile to disable VABC so that
scenario gets tested. The pixel_v3 profile now uses CoW v2 with gz.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
No known device uses this, but it's very useful for testing that our
overhead calculation is identical to AOSP's. A few more bugs were found
in our overhead calculation logic:
* The CowHeaderV3 size was missing the original CowHeader (v2) fields
that are supposed to be included due to inheritance in the C++ class.
* The additional 1% overhead was incorrectly calculated against the
initial CoW estimate before static overhead for CoW headers was added.
* The V3 num_ops estimation did not set a minimum of 25 to match
delta_generator.
* The V2 size estimation did not take into account that a cluster of CoW
operations cannot be truncated. It must be a multiple of cluster_ops
(200 for avbroot).
With these fixes, the CoW estimation when compression is disabled
matches AOSP exactly. This means all the overhead calculation is now
correct and the only difference when compression is enabled is in the
compression ratios of the various lz4/gz implementations.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
AOSP has long supported CoW version 3, but it wasn't used by the stock
OS on any Pixel devices until the new Pixel 9a.
CoW version 3 is fundamentally similar to version 2, though with
differences in the main header and how the operation headers are stored.
The compression is no longer done in fixed-size chunks equal to the
block size. Instead, the payload specifies a "compression factor", which
is the maximum chunk size to pass to the compressor. The actual chunk
size is the largest power of 2 <= the compression factor and the
remaining input size. Additionally, for version 3, the payload stores an
additional estimate_op_count_max field containing the number of CoW
operations.
While working on support for version 3, a few bugs in the version 2
estimation logic were found and fixed:
* The cluster_ops * sizeof(CowOperationV2) overhead incorrectly
assumed that cluster_ops was a constant 200 instead of the actual
number of CoW operations.
* The overhead did not account for kCowLabelOp headers, which
delta_generator emits once for every InstallOperation in the payload.
* The overhead did not account for kCowClusterOp headers, which batch
CoW operations into groups of 200.
* The overhead did not account for the CowFooter.
The version 3 overhead is much simpler and easier to calculate compared
to version 2.
Fixes: #441
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
While avbroot initially started as a way to allow a rooted boot image to
be used with a locked bootloader, it has evolved much since then.
Nowadays, many folks use it to make modifications to their OTAs that
don't involve enabling root access. avbroot also has many subcommands
for packing and unpacking various Android image formats that people use
without ever using avbroot's main OTA patching functionality.
This commit updates the project description to reflect this and
simplifies the wording a bit.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This reverts commit e929ecbe44.
Ring is back to being maintained again, so let's switch back to it since
it has fewer build dependencies and is much faster to compile.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Devices that launch with Android <14 generally use gzip as the CoW
compression algorithm. This never changes because future full OTAs
always need to be installable from the version of Android the device
launched with.
However, for users that don't care about the upgrade path from old
versions of Android, a new --vabc-algo option can be used to switch from
gz to lz4 compression. This can cut down the OTA installation time by
more than 2/3rds when installing via a custom OTA updater app. On my
Pixel Tablet, the installation time for the update_engine DOWNLOADING
phase decreased from 32:05 to 9:41. Note that this has absolutely no
effect on the performance when sideloading from recovery mode because
that does not use CoW.
When this new option is used, all dynamic partitions need to be
extracted from the OTA during patching so that the CoW estimates can be
recomputed. This will slow down the patching process and use up more
temporary disk space.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Instead of compressing the 64 MiB input in 2 MiB chunks, each loop
iteration was compressing the full 64 MiB. This massively slowed down
the patching process from seconds to potentially hours and would
temporarily waste a bunch of space during OTA installation.
Fixes: #433
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This way, we can fail hard for parsing errors, but not for verification
errors in `avbroot ota verify`.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Andrew Gunnerson