Andrew Gunnerson
8549fa1dfc
Why? ---- It was always my intention to write avbroot in a compiled language. Python was a stop-gap solution since it was possible to use the various tools and parsers from AOSP to make the initial prototyping and implementation easier. However, doing so required a whole lot of hacks since nearly all of the Python modules we use were intended to be used as executables, not libraries, and they were definitely not meant to be used outside of AOSP's code base. Although the dependencies on AOSP code have been reduced over time, working on the Python code is still frustrating. The majority of the modules we use from both the standard library and external dependencies are lacking type annotations. All of the Python language servers and type checker tools I've used choked on them. There have been serveral avbroot bugs in the past that wouldn't have happened with any statically typed language. The catalyst for me working on this recently was dealing with some python-protobuf versions that wouldn't work with AOSP's pregenerated protobuf bindings. When parsing protobuf messages, it would fail with obscure runtime type errors. I need my projects to not feel frustrating or else I'll just get burnt out. Hence, the Rust rewrite. With fewer hacks this time! avbroot no longer has any dependencies on external tools like openssl. I'll be providing precompiled binaries for the three major desktop OS's, built by GitHub Actions. avbroot will also be versioned now, starting at 2.0.0. Whats new? ---------- * A new `avbroot ota verify` subcommand has been added to check that all OTA and AVB related components have been properly hashed and signed. This works for all OTA images, including stock ones. * A couple new `avbroot avb` subcommands have been added for dumping vbmeta header/footer information and verifying AVB signatures. These are roughly equivalent to avbtool's `info_image` and `verify_image` subcommands, though avbroot is about an order of magnitude faster than the latter. * A new set of `avbroot boot` subcommands have been added for packing and unpacking boot images. It supports Android v0-v4 images and vendor v3-v4 images. Repacking is lossless even when using deprecated fields, like the boot image v4 VTS signature. * A new `avbroot ramdisk` subcommand has been added for inspecting the CPIO structure of ramdisks. * A new set of `avbroot key` subcommands have been added for generating signing keys so that it's no longer necessary to install openssl and avbtool (though of course, keys generated by other tools remain fully compatible). * Since avbroot has a ton of CLI options, a new `avbroot completion` subcommand has been added for generating tab-completion configs for various shells (eg. bash, zsh, fish, powershell). What was removed? ----------------- Nothing :) The `patch` and `extract` subcommands have been moved under `avbroot ota` and the `magisk-info` subcommand has been moved under `avbroot boot`, but there are compatibility shims in place to keep all the old commands working. The command-line interface will remain backwards compatible for as long as possible, even with new major releases. The Rust API, however, has no backwards compatibility guarantees. I currently don't intend for avbroot's "library" components to be used anywhere outside of Custota and avbroot itself. Performance ----------- Due to having better access to low-level APIs (especially `pread` and `pwrite`), nearly everything that can be multithreaded in avbroot is now multithreaded. In addition, during the patching operation, everything is done entirely in memory without temp files and the maximum memory usage is still about 100MB lower than with the Python implementation. The new implementation is bottlenecked by how fast a single CPU core can calculate 3 SHA256 hashes of overlapping regions spanning the majority of the OTA file. About 90% of the CPU time is spent calculating SHA256 hashes and another 5% or so performing XZ-compression. Some numbers: * Patching should take roughly 40%-70% of the time it took before. * Extracting with `--all` should take roughly 10%-30% of the time it took before. Folks with x86_64 CPUs supporting SHA-NI extensions (eg. Intel 11th gen and newer) should see even bigger improvements. Reproducibility --------------- The new implementation's output files are bit-for-bit identical when the inputs are the same. However, they do not exactly match what the Python implementation produced. * The zip entries, aside from `metadata` and `metadata.pb`, are written in sorted order. * All zip entries are stored without compression. * All zip entries are stored without additional metadata (eg. modification timestamp). * The OTA certificate, both in the OTA zip and in the recovery ramdisk's `otacerts.zip`, goes through deserialization + serialization before being written. Text in the certificate file before the header and after the footer will be stripped out. * The protobuf structures (payload header and OTA metadata) are serialized differently. Protobuf has more than one way to encode the same messages "on the wire". The Rust quick_protobuf library serializes messages a bit differently than python-protobuf, but the outputs are mutually compatible. * XZ compression of modified partition images in the payload is now done at compression level 0 instead of 6. This reduces the patching time by several seconds at the cost of a couple MiB increase in file size. * Ramdisks are now compressed with standard LZ4 instead of LZ4HC (high compression mode). For our use case, the difference is <100 KiB, but using standard LZ4 allows us to use a pure-Rust LZ4 library and makes the compression step much faster. * Older ramdisks compressed with gzip are slightly different due to a different gzip implementation being used (flate2 vs. zlib). The two implementations structure the gzip frames slightly differently, but the output is identical when decompressed. * Magisk's config file in the ramdisk (`.backup/.magisk`) will have the `SHA1` field set to all zeros. This allows avbroot to keep track of less information during patching for better performance. The field is only used for Magisk's uninstall feature, which can't ever be used in a locked bootloader setup anyway. Misc ---- While working on the new `avbroot ota verify` subcommand, I found that the `ossi` stock image (OnePlus 10 Pro) used in avbroot's tests has an invalid vbmeta hash for the `odm` partition. I thought it was an avbroot bug, but AOSP's avbtool reports the same invalid hash too. If that image actually boots, then I'm not sure AVB can be trusted on those devices... Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
3.8 KiB
avbroot extra
avbroot includes several feature-complete parsers for various things, like boot images. Some of these are exposed as extra subcommands. They aren't needed for normal OTA patching, but may be useful in other scenarios.
Note that while avbroot maintains a stable command line interface for the patching-related subcommands, these extra subcommands do not have backwards compatibility guarantees.
avbroot avb
Showing vbmeta header and footer information
avbroot avb dump -i <image>
This subcommand shows all of the vbmeta header and footer fields. vbmeta partition images will only have a header, while partitions with actual data (eg. boot images) will have both a header and a footer.
Verifying AVB hashes and signatures
avbroot avb verify -i <root vbmeta image> -p <public key>
This subcommand verifies the vbmeta header signature and the hashes for all vbmeta descriptors (including hashtree descriptors). If the vbmeta image has a chain descriptor for another partition, that partition image will be verified as well (recursively). All partitions are expected to be in the same directory as the vbmeta image being verified.
If -p is omitted, the signatures and hashes are checked only for validity, not that they are trusted.
avbroot boot
Unpacking a boot image
avbroot boot unpack -i <input boot image>
This subcommand unpacks all of the components of the boot image into the current directory by default (see --help). The header fields are saved to header.toml and each blob section is saved to a separate file. Each blob is written to disk as-is, without decompression.
Packing a boot image
avbroot boot pack -o <output boot image>
This subcommand packs a new boot image from the individual components in the current directory by default (see --help). The default input filenames are the same as the output filenames for the unpack subcommand.
Repacking a boot image
avbroot boot repack -i <input boot image> -o <output boot image>
This subcommand repacks a boot image without writing the individual components to disk first. This is useful for roundtrip testing of avbroot's boot image parser. The output should be identical to the input, minus any footers, like the AVB footer.
Showing information about a boot image
avbroot boot info -i <input boot image>
All of the boot subcommands show the boot image information. This specific subcommand just does it without performing any other operation. To show avbroot's internal representation of the information, pass in -d.
avbroot ramdisk
Dumping a cpio archive
avbroot ramdisk dump -i <cpio archive>
This subcommand dumps all information about a cpio archive to stdout. This includes the compression format, all header fields (including the trailer entry), and all data. If an entry's data can be decoded as UTF-8, then it is printed out as text. Otherwise, the binary data is printed out \x##-encoded for non-ASCII bytes. The escape-encoded data is truncated to 512 bytes by default to avoid outputting too much data, but this behavior can be disabled with --no-truncate.
Repacking a cpio archive
avbroot ramdisk repack -i <input cpio archive> -o <output cpio archive>
This subcommand repacks a cpio archive, including recompression if needed. This is useful for roundtrip testing of avbroot's cpio parser and compression handling. The uncompressed output should be identical to the uncompressed input, except:
- files are sorted by name
- inodes are reassigned, starting from 300000
- there is no excess padding at the end of the file
The compressed output may differ from what other tools produce due to differences in compression levels and header metadata. avbroot avoids specifying header information where possible (eg. gzip timestamp) for reproducibility.