The precompiled binaries are compiled for aarch64 API 31, which should
work for every device that avbroot supports.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
OnePlus seems to be getting worse and worse with some devices not being
recoverable.
Issue: #290
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This allows the user to boot GSIs signed by the same key. The option is
disabled by default because some Android builds disable DSU support by
removing all keys to reduce the attack surface. We don't want to
reenable DSU support on these builds unless the user asks for it.
Closes: #286
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
`fastboot flashall` is identical to the `fastboot update` command used
by the Pixel factory images, except it reads from a directory instead of
a zip file. It knows how to flip between the fastboot and fastbootd
modes without user intervention.
Fixes: #252
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Patching the image was never implemented because it's specific to
OnePlus devices where the bootloader does not respect the custom root of
trust.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
We'll switch to using `fastboot flashall` in the future once that has
been implemented and tested.
Fixes: #252
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Previously, overriding otacerts.zip in the system partition required the
user to flash a Magisk/KernelSU module that would bind mount over the
file during boot. While this worked well enough, it's insufficient for
unrooted setups, which has become more important since unrooting is the
only safe way to use the new OEM repair mode feature. With the stock
otacerts.zip, the OEM's default OTA updater app could run and install an
OS upgrade that's not signed by the user's key.
With this commit, the raw otacerts.zip bytes in the system partition are
directly replaced with a new zip that contains the user's certificate.
This method was inspired by @pascallj's comment in #216 suggesting
intentionally corrupting the otacerts.zip data in the filesystem.
Because avbroot does not have filesystem parsers for ext4/f2fs/erofs, we
rely on a heuristic-based search on the raw filesystem image. The file
is always smaller than one block (which is at least 4096 bytes on all
known devices), so the file data is stored contiguously on disk and in
the case of erofs, won't be compressed. None of the three filesystems
are copy-on-write and thus, have no filesystem-level data checksums. For
the dm-verity layer one level up, avbroot already knows how to recompute
the hash tree and FEC data.
To ensure that there are no false positives, any match that the search
finds must correctly parse as a valid zip and every entry within the zip
must have a filename that ends in .x509.pem. This matches what
update_engine expects from a proper otacerts.zip file.
Since the new approach is doing a raw search and replace, the old and
new files must have the same size. When the new zip is smaller, null
bytes are added to the zip archive comment field to pad to the correct
size. When the new zip is larger, avbroot will attempt the following to
try and make the file size smaller:
1. Enable zip deflate compression
2. Strip the X.509 signature from the certificate
3. Clear out the issuer RDN sequence from the certificate
4. Clear out the subject RDN sequence from the certificate
The latter three changes work because Android never performs any PKI
operations with the certificate. There is no CA certificate chain. The
X.509 certificate file is nothing more than a way to transport an RSA
public key.
avbroot requires the user's key to be RSA 4096. If the original zip had
the same key size, then none of these shrinking methods are needed. If
it contained an RSA 2048 key, then the first two modifications are
usually sufficient. The latter two modifications should only be needed
if the user picked a really long subject value when generating the
certificate.
With these new changes, the OTA patching time will approximately double
on a system with an SSD and modern CPU. This is dominated by the time it
takes to XZ-compress the system partition image. The compression is
already parallelized and scales linearly with the number of cores.
There's likely not much more that can be done to further speed this up.
Finally, these new changes are currently excluded from the e2e tests
because including the system partition in the stripped OTAs would
increase the file size by an order of magnitude. This could potentially
be solved in the future by generating our own small OTAs to use for
testing instead of running against real device OTAs.
Fixes: #225
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
During patching, all boot images are now extracted and the individual
patchers can inspect them to determine which ones need modifications.
This replaces the previous mechanism of detecting which boot images to
patch based on the name alone.
With this new method, the --boot-partition and --otacerts-partitions
options are no longer needed. The former option is kept (but ignored
with a warning message) for backwards compatibility, but the latter is
completely removed because it never made it to a stable release.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
* Split out requirements from the warnings/caveats section and move it
to the top. Hopefully this helps new users determine whether they can
even use avbroot quicker.
* Split usage section into separate usage, initial install, and updates
sections. This should hopefully make the steps much easier to follow
without a bunch of steps being prefixed with `[Initial setup only]`.
* Explicitly state what arguments are required for Magisk, KernelSU, and
unrooted setups instead of directing users to the advanced usage
section.
* Don't assume that everyone will use Magisk in the other sections.
* Explicitly state the assumption that the device should already running
the OS build that the user wants to patch during initial install.
* Add step to updates section for installing an updated Magisk or
KernelSU app.
* Move command for building the modules from the modules section to the
building from source section.
* Update sample error message in the clear vbmeta flags section to match
what avbroot will actually print out.
* General rewording to try and make things clearer (especially reducing
parenthesized parts).
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Android 11 OTAs use the same `payload.bin` format, but lack the
`metadata.pb` protobuf representation of the OTA metadata. This commit
adds support for parsing the legacy plain-text `metadata` format. Like
before, the output files will still contain both the legacy and
protobuf representations.
Note that the legacy format allowed OEMs to specify arbitrary key/value
pairs. These will be discarded during patching because they cannot be
represented in the protobuf format, which is used in avbroot's internal
representation.
Issue: #195
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
During the migration to avbroot 2.0, these options have been renamed. Reflect this in the Readme.
Signed-off-by: Pascal Roeleven <dev@pascalroeleven.nl>
Why?
----
It was always my intention to write avbroot in a compiled language.
Python was a stop-gap solution since it was possible to use the various
tools and parsers from AOSP to make the initial prototyping and
implementation easier. However, doing so required a whole lot of hacks
since nearly all of the Python modules we use were intended to be used
as executables, not libraries, and they were definitely not meant to be
used outside of AOSP's code base.
Although the dependencies on AOSP code have been reduced over time,
working on the Python code is still frustrating. The majority of the
modules we use from both the standard library and external dependencies
are lacking type annotations. All of the Python language servers and
type checker tools I've used choked on them. There have been serveral
avbroot bugs in the past that wouldn't have happened with any
statically typed language.
The catalyst for me working on this recently was dealing with some
python-protobuf versions that wouldn't work with AOSP's pregenerated
protobuf bindings. When parsing protobuf messages, it would fail
with obscure runtime type errors. I need my projects to not feel
frustrating or else I'll just get burnt out.
Hence, the Rust rewrite. With fewer hacks this time! avbroot no longer
has any dependencies on external tools like openssl. I'll be providing
precompiled binaries for the three major desktop OS's, built by GitHub
Actions. avbroot will also be versioned now, starting at 2.0.0.
Whats new?
----------
* A new `avbroot ota verify` subcommand has been added to check that all
OTA and AVB related components have been properly hashed and signed.
This works for all OTA images, including stock ones.
* A couple new `avbroot avb` subcommands have been added for dumping
vbmeta header/footer information and verifying AVB signatures. These
are roughly equivalent to avbtool's `info_image` and `verify_image`
subcommands, though avbroot is about an order of magnitude faster than
the latter.
* A new set of `avbroot boot` subcommands have been added for packing
and unpacking boot images. It supports Android v0-v4 images and vendor
v3-v4 images. Repacking is lossless even when using deprecated fields,
like the boot image v4 VTS signature.
* A new `avbroot ramdisk` subcommand has been added for inspecting
the CPIO structure of ramdisks.
* A new set of `avbroot key` subcommands have been added for generating
signing keys so that it's no longer necessary to install openssl and
avbtool (though of course, keys generated by other tools remain fully
compatible).
* Since avbroot has a ton of CLI options, a new `avbroot completion`
subcommand has been added for generating tab-completion configs for
various shells (eg. bash, zsh, fish, powershell).
What was removed?
-----------------
Nothing :) The `patch` and `extract` subcommands have been moved under
`avbroot ota` and the `magisk-info` subcommand has been moved under
`avbroot boot`, but there are compatibility shims in place to keep all
the old commands working.
The command-line interface will remain backwards compatible for as long
as possible, even with new major releases. The Rust API, however, has no
backwards compatibility guarantees. I currently don't intend for
avbroot's "library" components to be used anywhere outside of Custota
and avbroot itself.
Performance
-----------
Due to having better access to low-level APIs (especially `pread` and
`pwrite`), nearly everything that can be multithreaded in avbroot is now
multithreaded. In addition, during the patching operation, everything
is done entirely in memory without temp files and the maximum memory
usage is still about 100MB lower than with the Python implementation.
The new implementation is bottlenecked by how fast a single CPU core can
calculate 3 SHA256 hashes of overlapping regions spanning the majority
of the OTA file. About 90% of the CPU time is spent calculating SHA256
hashes and another 5% or so performing XZ-compression.
Some numbers:
* Patching should take roughly 40%-70% of the time it took before.
* Extracting with `--all` should take roughly 10%-30% of the time it
took before.
Folks with x86_64 CPUs supporting SHA-NI extensions (eg. Intel 11th gen
and newer) should see even bigger improvements.
Reproducibility
---------------
The new implementation's output files are bit-for-bit identical when the
inputs are the same. However, they do not exactly match what the Python
implementation produced.
* The zip entries, aside from `metadata` and `metadata.pb`, are written
in sorted order.
* All zip entries are stored without compression.
* All zip entries are stored without additional metadata (eg.
modification timestamp).
* The OTA certificate, both in the OTA zip and in the recovery ramdisk's
`otacerts.zip`, goes through deserialization + serialization before
being written. Text in the certificate file before the header and
after the footer will be stripped out.
* The protobuf structures (payload header and OTA metadata) are
serialized differently. Protobuf has more than one way to encode the
same messages "on the wire". The Rust quick_protobuf library
serializes messages a bit differently than python-protobuf, but the
outputs are mutually compatible.
* XZ compression of modified partition images in the payload is now done
at compression level 0 instead of 6. This reduces the patching time by
several seconds at the cost of a couple MiB increase in file size.
* Ramdisks are now compressed with standard LZ4 instead of LZ4HC (high
compression mode). For our use case, the difference is <100 KiB, but
using standard LZ4 allows us to use a pure-Rust LZ4 library and makes
the compression step much faster.
* Older ramdisks compressed with gzip are slightly different due to a
different gzip implementation being used (flate2 vs. zlib). The two
implementations structure the gzip frames slightly differently, but
the output is identical when decompressed.
* Magisk's config file in the ramdisk (`.backup/.magisk`) will have the
`SHA1` field set to all zeros. This allows avbroot to keep track of
less information during patching for better performance. The field is
only used for Magisk's uninstall feature, which can't ever be used in
a locked bootloader setup anyway.
Misc
----
While working on the new `avbroot ota verify` subcommand, I found that
the `ossi` stock image (OnePlus 10 Pro) used in avbroot's tests has an
invalid vbmeta hash for the `odm` partition. I thought it was an avbroot
bug, but AOSP's avbtool reports the same invalid hash too. If that image
actually boots, then I'm not sure AVB can be trusted on those devices...
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This commit adds support for replacing any partition image within
the payload with a custom image. This is useful, for example, to add a
custom kernel to an OTA, which may involve partitions that wouldn't
normally be touched (eg. `vendor_dlkm`).
Any image specified via `--replace` will have its corresponding
descriptor in the vbmeta image updated. This is handled recursively. For
example, replacing `vendor_dlkm` would update both `vbmeta_vendor` and
`vbmeta`. This requires all vbmeta images to be extracted during the
patching process so that a complete dependency graph can be computed.
The performance hit in doing so is negligible, but does require the
checksums of the stripped images to be updated for the tests.
Fixes: #102
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
There are 3 levels of warnings:
* Level 0: Warnings that don't affect booting
* Mismatched `id` or `os_version` fields
* Level 1: Warnings that may affect booting
* Mismatched `cmdline` or `extra_cmdline` fields
* Unexpected addition of `kernel`, `second`, `recovery_dtbo`, `dtb`,
or `bootconfig`
* Level 2: Warnings that are very likely to affect booting
* All other mismatched, added, or removed fields
By default, any warning of level 1 or higher is treated as a fatal
error. Each time `--ignore-prepatched-compat` is passed in, the
permitted warning level is increased.
Fixes: #108
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
The module helps reduce the chance of OEM unlocking being disabled,
whether by the user or by some OS's initial setup wizard. It works by
running some Java code at boot, which connects to the `OemLockService`
binder service and calls `setOemUnlockAllowedByUser(true)`, the same as
what the Settings app does.
Fixes: #8
Issue: #84
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Unlike the prior RULESDEVICE, PREINITDEVICE uses a block device name
since the rdev major/minor is not guaranteed to remain the same across
reboots. This commit completely drops support for RULESDEVICE (25207
through 25210) since it never made it to a stable Magisk release.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Newer Magisk versions no longer try to autodetect a writable ext4
partition for storing SELinux rules during boot. Instead, the block
device is detected during boot image patching and is stored in the
ramdisk's `.backup/.magisk` as `makedev(rdev_maj, rdev_min)`.
This unfortunately complicates the patching process for avbroot. Even if
we replicate Magisk's algorithm for finding a suitable partition,
there's no way to find the block device's rdev for it with the
information contained in the OTA package. This is the first change that
adds a hard dependency on information only attainable from a running
device.
For these newer Magisk versions, the user will have to patch the boot
image once in the Magisk app (must be on the target device) and then run
`avbroot magisk-info` to show the computed device ID. Then, avbroot can
use this during patching via `--magisk-rules-device`. If the user's
device is unable to run the Magisk app prior to patching (eg.
unbootable), they'll have to go through the patching process twice, once
with `--ignore-magisk-warnings` and a second time with the proper device
ID specified.
When using `--ignore-magisk-warnings`, root still works, but there will
be subtle issues, like certain modules failing to load.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Some Android builds ship with a root vbmeta image with flags that
disable AVB completely. This commit adds a new check for these flags so
that the patching process will fail and also adds a new
`--clear-vbmeta-flags` option for forcibly setting the flags to 0.
Issue: #60
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This is useful for folks who want to apply the Magisk root patch
themselves via the Magisk app or use KernelSU. When `--prepatched` is
used, avbroot will skip the root patch, but still apply the otacerts
patch.
To help protect against accidental use of the wrong boot image, avbroot
will only accept a prepatched image if all the header fields and section
types match the original image. The only exception is the number of
ramdisks, which is allowed to increase (Magisk may create a ramdisk in
an image that didn't originally have one).
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
openssl does not support reading the passphrase from file descriptors on
Windows. Using an environment variable is the next best option.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
This commit reimplements the magiskboot patching process natively in
Python. By doing so, avbroot gains cross platform support and there's no
longer a need to execute an Android binary on a non-Android Linux system,
which had pitfalls that already had to be worked around.
Boot images and cpio archives are handled by new custom parsers added to
avbroot. Legacy lz4 compression is done by a wrapper around the python
lz4 library (a new dependency). Gzip compression is handled natively by
python. Other compression methods and OEM-specific boot image formats
are not supported because devices that use the modern Android A/B OTA
scheme do not use those.
Output files are still bit-for-bit reproducible across runs, but they
are different from what prior avbroot commits produced:
* avbroot's cpio writer follows GNU cpio and libarchive's behavior of
setting the mode field to 0 in the trailer entry. magiskboot sets the
mode to 0o755.
* When patching a vendor boot v4 image, the ramdisk table entries are
now updated correctly. Prior vendor boot images were only bootable
because:
* the image only contained a single ramdisk
* the single ramdisk shrunk in size, stayed the same, or grew little
enough to not exceed a page boundary
* the bootloader is lenient in validating boot image fields
magiskboot does not handle vendor boot v4 images correctly, but it
doesn't need to just for the regular root patch. avbroot made use of
it in an unsupported way to patch vendor boot ramdisks.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
* Clarify boot image partitions. It is no longer the case that only
`boot` and `vendor_boot` are patched.
* Describe how to check if an OTA file is A/B.
* Remove mentions of signapk now that avbroot signs zips itself.
* Minor grammar changes.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
With this commit, avbroot now performs the entire OTA signing process
itself. This removes the 4 full .zip builds needed for signing before,
which means that avbroot can now do the full patching process in a
single pass.
Since signapk is no longer needed, the java dependency has been dropped.
This also means that avbroot no longer depends on glibc-based Linux
distros, which was the case before due to signapk's conscrypt
dependency.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Andrew Gunnerson