This fixes `data_offset` being set for `ZERO` and `DISCARD` operations,
which prevents some images (eg. `ossi`) from being flashed due to
update_engine's strict field validation.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
There's an upstream bug that causes an infinite loop in the
`write::BzDecoder` destructor if the decoder is fed invalid data. While
this never happens during normal operation, it is possible to run into
this by running `ota extract` against a `--stripped` OTA file.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Why?
----
It was always my intention to write avbroot in a compiled language.
Python was a stop-gap solution since it was possible to use the various
tools and parsers from AOSP to make the initial prototyping and
implementation easier. However, doing so required a whole lot of hacks
since nearly all of the Python modules we use were intended to be used
as executables, not libraries, and they were definitely not meant to be
used outside of AOSP's code base.
Although the dependencies on AOSP code have been reduced over time,
working on the Python code is still frustrating. The majority of the
modules we use from both the standard library and external dependencies
are lacking type annotations. All of the Python language servers and
type checker tools I've used choked on them. There have been serveral
avbroot bugs in the past that wouldn't have happened with any
statically typed language.
The catalyst for me working on this recently was dealing with some
python-protobuf versions that wouldn't work with AOSP's pregenerated
protobuf bindings. When parsing protobuf messages, it would fail
with obscure runtime type errors. I need my projects to not feel
frustrating or else I'll just get burnt out.
Hence, the Rust rewrite. With fewer hacks this time! avbroot no longer
has any dependencies on external tools like openssl. I'll be providing
precompiled binaries for the three major desktop OS's, built by GitHub
Actions. avbroot will also be versioned now, starting at 2.0.0.
Whats new?
----------
* A new `avbroot ota verify` subcommand has been added to check that all
OTA and AVB related components have been properly hashed and signed.
This works for all OTA images, including stock ones.
* A couple new `avbroot avb` subcommands have been added for dumping
vbmeta header/footer information and verifying AVB signatures. These
are roughly equivalent to avbtool's `info_image` and `verify_image`
subcommands, though avbroot is about an order of magnitude faster than
the latter.
* A new set of `avbroot boot` subcommands have been added for packing
and unpacking boot images. It supports Android v0-v4 images and vendor
v3-v4 images. Repacking is lossless even when using deprecated fields,
like the boot image v4 VTS signature.
* A new `avbroot ramdisk` subcommand has been added for inspecting
the CPIO structure of ramdisks.
* A new set of `avbroot key` subcommands have been added for generating
signing keys so that it's no longer necessary to install openssl and
avbtool (though of course, keys generated by other tools remain fully
compatible).
* Since avbroot has a ton of CLI options, a new `avbroot completion`
subcommand has been added for generating tab-completion configs for
various shells (eg. bash, zsh, fish, powershell).
What was removed?
-----------------
Nothing :) The `patch` and `extract` subcommands have been moved under
`avbroot ota` and the `magisk-info` subcommand has been moved under
`avbroot boot`, but there are compatibility shims in place to keep all
the old commands working.
The command-line interface will remain backwards compatible for as long
as possible, even with new major releases. The Rust API, however, has no
backwards compatibility guarantees. I currently don't intend for
avbroot's "library" components to be used anywhere outside of Custota
and avbroot itself.
Performance
-----------
Due to having better access to low-level APIs (especially `pread` and
`pwrite`), nearly everything that can be multithreaded in avbroot is now
multithreaded. In addition, during the patching operation, everything
is done entirely in memory without temp files and the maximum memory
usage is still about 100MB lower than with the Python implementation.
The new implementation is bottlenecked by how fast a single CPU core can
calculate 3 SHA256 hashes of overlapping regions spanning the majority
of the OTA file. About 90% of the CPU time is spent calculating SHA256
hashes and another 5% or so performing XZ-compression.
Some numbers:
* Patching should take roughly 40%-70% of the time it took before.
* Extracting with `--all` should take roughly 10%-30% of the time it
took before.
Folks with x86_64 CPUs supporting SHA-NI extensions (eg. Intel 11th gen
and newer) should see even bigger improvements.
Reproducibility
---------------
The new implementation's output files are bit-for-bit identical when the
inputs are the same. However, they do not exactly match what the Python
implementation produced.
* The zip entries, aside from `metadata` and `metadata.pb`, are written
in sorted order.
* All zip entries are stored without compression.
* All zip entries are stored without additional metadata (eg.
modification timestamp).
* The OTA certificate, both in the OTA zip and in the recovery ramdisk's
`otacerts.zip`, goes through deserialization + serialization before
being written. Text in the certificate file before the header and
after the footer will be stripped out.
* The protobuf structures (payload header and OTA metadata) are
serialized differently. Protobuf has more than one way to encode the
same messages "on the wire". The Rust quick_protobuf library
serializes messages a bit differently than python-protobuf, but the
outputs are mutually compatible.
* XZ compression of modified partition images in the payload is now done
at compression level 0 instead of 6. This reduces the patching time by
several seconds at the cost of a couple MiB increase in file size.
* Ramdisks are now compressed with standard LZ4 instead of LZ4HC (high
compression mode). For our use case, the difference is <100 KiB, but
using standard LZ4 allows us to use a pure-Rust LZ4 library and makes
the compression step much faster.
* Older ramdisks compressed with gzip are slightly different due to a
different gzip implementation being used (flate2 vs. zlib). The two
implementations structure the gzip frames slightly differently, but
the output is identical when decompressed.
* Magisk's config file in the ramdisk (`.backup/.magisk`) will have the
`SHA1` field set to all zeros. This allows avbroot to keep track of
less information during patching for better performance. The field is
only used for Magisk's uninstall feature, which can't ever be used in
a locked bootloader setup anyway.
Misc
----
While working on the new `avbroot ota verify` subcommand, I found that
the `ossi` stock image (OnePlus 10 Pro) used in avbroot's tests has an
invalid vbmeta hash for the `odm` partition. I thought it was an avbroot
bug, but AOSP's avbtool reports the same invalid hash too. If that image
actually boots, then I'm not sure AVB can be trusted on those devices...
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
We have to emulate this with fchmod because Python's NamedTemporaryFile
always opens the file descriptor with 600 permissions.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
PR #110 fixed this incorrectly. I misinterpreted "set to zero in the
local header" from §4.4.4 in the spec as referring to the two 32-bit
size fields in the local header. However, the spec considers the zip64
extra record as part of the local header. The correct behavior for
writing zip64 entries to unseekable files is 0xffffffff in the 32-bit
local header size fields (to enable zip64), 0 in the zip64 extra record
(due to streaming writes), and the actual sizes in the data descriptor.
Fixes: #109
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This commit adds support for replacing any partition image within
the payload with a custom image. This is useful, for example, to add a
custom kernel to an OTA, which may involve partitions that wouldn't
normally be touched (eg. `vendor_dlkm`).
Any image specified via `--replace` will have its corresponding
descriptor in the vbmeta image updated. This is handled recursively. For
example, replacing `vendor_dlkm` would update both `vbmeta_vendor` and
`vbmeta`. This requires all vbmeta images to be extracted during the
patching process so that a complete dependency graph can be computed.
The performance hit in doing so is negligible, but does require the
checksums of the stripped images to be updated for the tests.
Fixes: #102
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
There are 3 levels of warnings:
* Level 0: Warnings that don't affect booting
* Mismatched `id` or `os_version` fields
* Level 1: Warnings that may affect booting
* Mismatched `cmdline` or `extra_cmdline` fields
* Unexpected addition of `kernel`, `second`, `recovery_dtbo`, `dtb`,
or `bootconfig`
* Level 2: Warnings that are very likely to affect booting
* All other mismatched, added, or removed fields
By default, any warning of level 1 or higher is treated as a fatal
error. Each time `--ignore-prepatched-compat` is passed in, the
permitted warning level is increased.
Fixes: #108
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This commit fixes two issues:
* Python 3.11.4 has a regression where it sets the local file header's
compressed and uncompressed size fields to 0xffffffff when data
descriptors are used. These should be set to 0 (along with the CRC)
according to §4.4.4 of the spec. We work around this by monkey
patching zipfile's local file header serialization function to correct
the fields.
* avbroot tries to preserve as much metadata from the original zip as
possible, including the `extra` records. This caused zip64 records
(0x0001) to be duplicated. The first instance was from the original
zip (containing invalid size fields) and the second instance was newly
created by zipfile. We fix this by stripping out all 0x0001 records,
similar to what we already do for 0xd935.
Fixes: #109
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Andrew Gunnerson