Version 3.6.0

Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>

CHANGELOG.md

@@ -7,6 +7,22 @@
     to update the actual links at the bottom of the file.
 -->
 
+### Version 3.6.0
+
+* Add support for gzip compression when computing CoW size estimates ([Issue #332], [PR #333])
+  * This allows `--replace` to successfully replace dynamic partitions on legacy devices, like the Pixel 4a 5G
+* Minor code cleanup ([PR #334], [PR #335])
+
+### Version 3.5.0
+
+* Update all dependencies ([PR #329])
+* Add new unpack and pack commands for `payload.bin` files ([Issue #328], [PR #331])
+
+### Version 3.4.1
+
+* Update all dependencies ([PR #321])
+* Add support for Magisk 27006 ([PR #323])
+
 ### Version 3.4.0
 
 * Fix (unreachable) minor error handling logic when attempting to use unsupported AVB signing algorithms ([PR #311])
@@ -225,6 +241,8 @@ Behind-the-scenes changes:
 [Issue #301]: https://github.com/chenxiaolong/avbroot/issues/301
 [Issue #306]: https://github.com/chenxiaolong/avbroot/issues/306
 [Issue #310]: https://github.com/chenxiaolong/avbroot/issues/310
+[Issue #328]: https://github.com/chenxiaolong/avbroot/issues/328
+[Issue #332]: https://github.com/chenxiaolong/avbroot/issues/332
 [PR #130]: https://github.com/chenxiaolong/avbroot/pull/130
 [PR #132]: https://github.com/chenxiaolong/avbroot/pull/132
 [PR #133]: https://github.com/chenxiaolong/avbroot/pull/133
@@ -318,3 +336,10 @@ Behind-the-scenes changes:
 [PR #309]: https://github.com/chenxiaolong/avbroot/pull/309
 [PR #311]: https://github.com/chenxiaolong/avbroot/pull/311
 [PR #312]: https://github.com/chenxiaolong/avbroot/pull/312
+[PR #321]: https://github.com/chenxiaolong/avbroot/pull/321
+[PR #323]: https://github.com/chenxiaolong/avbroot/pull/323
+[PR #329]: https://github.com/chenxiaolong/avbroot/pull/329
+[PR #331]: https://github.com/chenxiaolong/avbroot/pull/331
+[PR #333]: https://github.com/chenxiaolong/avbroot/pull/333
+[PR #334]: https://github.com/chenxiaolong/avbroot/pull/334
+[PR #335]: https://github.com/chenxiaolong/avbroot/pull/335

Cargo.toml

@@ -4,7 +4,7 @@ members = ["avbroot", "e2e", "fuzz", "xtask"]
 resolver = "2"
 
 [workspace.package]
-version = "3.4.0"
+version = "3.6.0"
 license = "GPL-3.0-only"
 edition = "2021"
 repository = "https://github.com/chenxiaolong/avbroot"

README.extra.md

@@ -255,6 +255,36 @@ This will check if the input file has any corrupted blocks. Currently, the comma
 
 ## `avbroot payload`
 
+### Unpacking a payload binary
+
+```bash
+avbroot payload unpack -i <input payload>
+```
+
+This subcommand unpacks the payload header information to `payload.toml` and the partition images to the `payload_images` directory.
+
+Only full payload binaries can be unpacked. Delta payload binaries from incremental OTAs are not supported.
+
+### Packing a payload binary
+
+```bash
+avbroot payload pack -o <output payload> --key <OTA private key>
+```
+
+This subcommand packs a new payload binary from the `payload.toml` file and `payload_images` directory. Any `.img` files in the `payload_images` directory that don't have a corresponding entry in `payload.toml` are silently ignored.
+
+Packing a payload binary requires compressing all of the partition images, which is very CPU intensive. If re-signing an existing payload binary without making any other modifications is all that's needed, use the `repack` subcommand instead.
+
+### Repacking a payload binary
+
+```bash
+avbroot payload repack -i <input payload> -o <output payload> --key <OTA private key>
+```
+
+This subcommand is logically equivalent to `avbroot payload unpack` followed by `avbroot payload pack`, except significantly more efficient. Instead of decompressing and recompressing all partition images, the raw data is directly copied from the input payload binary.
+
+This is useful for re-signing a payload binary without making any other changes.
+
 ### Showing payload header information
 
 ```bash

README.md

@@ -35,7 +35,11 @@ avbroot applies the following patches to the partition images:
 
     Repeat: **_ALWAYS leave `OEM unlocking` enabled if rooted._**
 
-* Any operation that causes an improperly-signed boot image to be flashed will result in the device being unbootable and unrecoverable without unlocking the bootloader again (and thus, triggering a data wipe). This includes the `Direct install` method for updating Magisk. Magisk updates **must** be done by repatching the OTA, not via the app.
+* Any operation that causes an improperly-signed boot image to be flashed will result in the device being unbootable and unrecoverable without unlocking the bootloader again (and thus, triggering a data wipe). A couple ways an improperly-signed boot image could be flashed include:
+
+    * The `Direct install` method for updating Magisk. Magisk updates **must** be done by repatching the OTA, not via the app.
+
+    * The `Uninstall Magisk` feature in Magisk. If root access is no longer needed, Magisk **must** be removed by repatching the OTA with the `--rootless` option, not via the app.
 
     If the boot image is ever modified, **do not reboot**. [Open an issue](https://github.com/chenxiaolong/avbroot/issues/new) for support and be very clear about what steps were done that lead to the situation. If Android is still running and root access works, it might be possible to recover without wiping and starting over.
 
@@ -132,15 +136,21 @@ If you lose your AVB or OTA signing key, you will no longer be able to sign new
 
 ## Initial setup
 
-1. Reboot into fastboot mode and unlock the bootloader if it isn't already unlocked. This will trigger a data wipe.
+1. Make sure that the version of fastboot is 34 or newer. Older versions have bugs that prevent the `fastboot flashall` command (required later) from working properly.
+
+    ```bash
+    fastboot --version
+    ```
+
+2. Reboot into fastboot mode and unlock the bootloader if it isn't already unlocked. This will trigger a data wipe.
 
     ```bash
     fastboot flashing unlock
     ```
 
-2. When setting things up for the first time, the device must already be running the correct OS. Flash the original unpatched OTA if needed.
+3. When setting things up for the first time, the device must already be running the correct OS. Flash the original unpatched OTA if needed.
 
-3. Extract the partition images from the patched OTA that are different from the original.
+4. Extract the partition images from the patched OTA that are different from the original.
 
     ```bash
     avbroot ota extract \
@@ -151,7 +161,7 @@ If you lose your AVB or OTA signing key, you will no longer be able to sign new
 
     If you prefer to extract and flash all OS partitions just to be safe, pass in `--all`.
 
-4. Flash the partition images that were extracted.
+5. Flash the partition images that were extracted.
 
     ```bash
     ANDROID_PRODUCT_OUT=extracted fastboot flashall --skip-reboot
@@ -161,7 +171,7 @@ If you lose your AVB or OTA signing key, you will no longer be able to sign new
 
     Alternatively, for Pixel devices, running `flash-base.sh` from the factory image will also update the bootloader and modem.
 
-5. Set up the custom AVB public key in the bootloader after rebooting from fastbootd to bootloader.
+6. Set up the custom AVB public key in the bootloader after rebooting from fastbootd to bootloader.
 
     ```bash
     fastboot reboot-bootloader
@@ -169,7 +179,7 @@ If you lose your AVB or OTA signing key, you will no longer be able to sign new
     fastboot flash avb_custom_key /path/to/avb_pkmd.bin
     ```
 
-6. **[Optional]** Before locking the bootloader, reboot into Android once to confirm that everything is properly signed.
+7. **[Optional]** Before locking the bootloader, reboot into Android once to confirm that everything is properly signed.
 
     Install the Magisk or KernelSU app and run the following command:
 
@@ -183,7 +193,7 @@ If you lose your AVB or OTA signing key, you will no longer be able to sign new
     init: [libfs_avb]Returning avb_handle with status: Success
     ```
 
-7. Reboot back into fastboot and lock the bootloader. This will trigger a data wipe again.
+8. Reboot back into fastboot and lock the bootloader. This will trigger a data wipe again.
 
     ```bash
     fastboot flashing lock
@@ -195,7 +205,7 @@ If you lose your AVB or OTA signing key, you will no longer be able to sign new
 
     **WARNING**: If you are flashing CalyxOS, the setup wizard will [automatically turn off the `OEM unlocking` switch](https://github.com/CalyxOS/platform_packages_apps_SetupWizard/blob/7d2df25cedcbff83ddb608e628f9d97b38259c26/src/org/lineageos/setupwizard/SetupWizardApp.java#L135-L140). Make sure to manually reenable it again from Android's developer settings. Consider using the [`OEMUnlockOnBoot` module](https://github.com/chenxiaolong/OEMUnlockOnBoot) to automatically ensure OEM unlocking is enabled on every boot.
 
-8. That's it! To install future OS, Magisk, or KernelSU updates, see the [next section](#updates).
+9. That's it! To install future OS, Magisk, or KernelSU updates, see the [next section](#updates).
 
 ## Updates
 
@@ -474,23 +484,7 @@ It is possible to run the tests if the host is running Linux, qemu-user-static i
 
 ## Verifying digital signatures
 
-First, save the public key to a file listing the keys to be trusted. This is the same key listed in [the author's profile](https://github.com/chenxiaolong/).
-
-```bash
-echo 'avbroot ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDOe6/tBnO7xZhAWXRj3ApUYgn+XZ0wnQiXM8B7tPgv4' > avbroot_trusted_keys
-```
-
-Then, verify the signature of the zip file using the list of trusted keys.
-
-```bash
-ssh-keygen -Y verify -f avbroot_trusted_keys -I avbroot -n file -s <file>.zip.sig < <file>.zip
-```
-
-If the file is successfully verified, the output will be:
-
-```
-Good "file" signature for avbroot with ED25519 key SHA256:Ct0HoRyrFLrnF9W+A/BKEiJmwx7yWkgaW/JvghKrboA
-```
+To verify the digital signatures of the downloads, follow [the steps here](https://github.com/chenxiaolong/chenxiaolong/blob/master/VERIFY_SSH_SIGNATURES.md).
 
 ## Contributing
 

README.ru.md

@@ -33,7 +33,11 @@ avbroot модифицирует следующие образы:
 
     Повторюсь: **_ВСЕГДА оставляйте `Заводскую разблокировку` включенной при наличии root-прав._**
 
-* Любая операция, приводящая к прошивке некорректно подписанного загрузочного образа, приведет к тому, что устройство больше не сможет загрузиться в систему/режим Recovery, а для его восстановления потребуется повторная разблокировка загрузчика (и, следовательно, стирание всех пользовательских данных). Это же относится и к методу `Прямой установки` для обновления Magisk. Обновление Magisk **должно выполняться только путем обновления OTA,** а не через Magisk Manager.
+* Любая операция, приводящая к прошивке некорректно подписанного загрузочного образа, приведет к тому, что устройство больше не сможет загрузиться в систему/режим Recovery, а для его восстановления потребуется повторная разблокировка загрузчика (и, следовательно, стирание всех пользовательских данных). К подобным операциям в том числе относятся:
+
+    * Метод `Прямой установки` для обновления Magisk. Magisk можно обновлять **только путем репатчинга OTA,** но не через его приложение.
+
+    * Функция `Удаление Magisk` в приложении Magisk. Если вам больше не нужен root-доступ, Magisk **должен быть удален путем репатчинга OTA** с использованием параметра `--rootless`, но не через его приложение.
 
     Если в загрузочный раздел были внесены какие-либо изменения, **не перезагружайтесь**. Обратитесь за помощью, [открыв Issue,](https://github.com/chenxiaolong/avbroot/issues/new) и четко разъясните, какие конкретные действия привели к возникновению такой ситуации. Если Android всё еще работает и доступ к root-правам сохранился – вероятно, получится откатить изменения до исходного состояния, не стирая ваши данные.
 
@@ -130,15 +134,21 @@ avbroot совместим с любым стандартным 4096-битны
 
 ## Первоначальная настройка
 
-1. Перезагрузитесь в режим fastboot и разблокируйте загрузчик, если не сделали этого ранее. Это приведет к стиранию всех пользовательских данных.
+1. Убедитесь, что вы используете утилиту fastboot версии 34 или новее. Предыдущие версии содержат баги, что не позволяют команде `fastboot flashall` (которая понадобится по ходу инструкции) работать правильно.
+
+    ```bash
+    fastboot --version
+    ```
+
+2. Перезагрузитесь в режим fastboot и разблокируйте загрузчик, если не сделали этого ранее. Это приведет к стиранию всех пользовательских данных.
 
     ```bash
     fastboot flashing unlock
     ```
 
-2. Перед первой установкой, на устройстве уже должна быть установлена в оригинальном виде та прошивка, пропатченную версию которой вы собираетесь ставить. Если это не так, сначала установите оригинальную непропатченную OTA.
+3. Перед первой установкой, на устройстве уже должна быть установлена в оригинальном виде та прошивка, пропатченную версию которой вы собираетесь ставить. Если это не так, сначала установите оригинальную непропатченную OTA.
 
-3. Извлекаем из пропатченного OTA модифицированные образы:
+4. Извлекаем из пропатченного OTA модифицированные образы:
 
     ```bash
     avbroot ota extract \
@@ -149,7 +159,7 @@ avbroot совместим с любым стандартным 4096-битны
 
     Если вы на всякий случай хотите прошить вообще все разделы из ОТА, извлечь их можно, указав аргумент `--all`.
 
-4. Прошейте извлеченные образы разделов.
+5. Прошейте извлеченные образы разделов.
 
     ```bash
     ANDROID_PRODUCT_OUT=extracted fastboot flashall --skip-reboot
@@ -159,7 +169,7 @@ avbroot совместим с любым стандартным 4096-битны
 
     Для устройств Pixel есть ещё один вариант: запуск скрипта `flash-base.sh` из папки заводских образов (factory images) обновит загрузчик и модем.
 
-5. После перезагрузки из fastbootd в загрузчик (bootloader), установите пользовательский публичный ключ AVB в загрузчик:
+6. После перезагрузки из fastbootd в загрузчик (bootloader), установите пользовательский публичный ключ AVB в загрузчик:
 
     ```bash
     fastboot reboot-bootloader
@@ -167,7 +177,7 @@ avbroot совместим с любым стандартным 4096-битны
     fastboot flash avb_custom_key /путь/к/avb_pkmd.bin
     ```
 
-6. **[Опционально]** Перед блокировкой загрузчика загрузитесь в систему, дабы убедиться, что все подписано правильно.
+7. **[Опционально]** Перед блокировкой загрузчика загрузитесь в систему, дабы убедиться, что все подписано правильно.
 
     Установите приложение Magisk или KernelSU и выполните следующую команду:
 
@@ -181,7 +191,7 @@ avbroot совместим с любым стандартным 4096-битны
     init: [libfs_avb]Returning avb_handle with status: Success
     ```
 
-7. Перезагрузитесь в fastboot и заблокируйте загрузчик. Это снова приведет к стиранию данных.
+8. Перезагрузитесь в fastboot и заблокируйте загрузчик. Это снова приведет к стиранию данных.
 
     ```bash
     fastboot flashing lock
@@ -193,7 +203,7 @@ avbroot совместим с любым стандартным 4096-битны
 
     **ПРЕДУПРЕЖДЕНИЕ**: Если вы прошили CalyxOS, мастер настройки [автоматически отключит опцию `Заводской разблокировки`.](https://github.com/CalyxOS/platform_packages_apps_SetupWizard/blob/7d2df25cedcbff83ddb608e628f9d97b38259c26/src/org/lineageos/setupwizard/SetupWizardApp.java#L135-L140) Не забудьте снова включить её вручную в настройках для разработчиков. Для перестраховки можете использовать [модуль `OEMUnlockOnBoot`,](https://github.com/chenxiaolong/OEMUnlockOnBoot) который автоматически включает пункт Заводской разблокировки при каждом запуске системы.
 
-8. Готово. Установка последующих обновлений системы, Magisk или KernelSU, описывается в [следующем разделе.](#обновления)
+9. Готово! Установка последующих обновлений системы, Magisk или KernelSU, описывается в [следующем разделе.](#обновления)
 
 ## Обновления
 

avbroot/Cargo.toml

@@ -27,11 +27,12 @@ hex = { version = "0.4.3", features = ["serde"] }
 liblzma = "0.3.0"
 lz4_flex = "0.11.1"
 memchr = "2.6.0"
+miniz_oxide = "0.8.0"
 num-bigint-dig = "0.8.4"
 num-traits = "0.2.16"
 phf = { version = "0.11.2", features = ["macros"] }
 pkcs8 = { version = "0.10.2", features = ["encryption", "pem"] }
-prost = "0.12.1"
+prost = "0.13.1"
 rand = "0.8.5"
 rayon = "1.7.0"
 regex = { version = "1.9.4", default-features = false, features = ["perf", "std"] }
@@ -71,8 +72,9 @@ features = ["deflate"]
 rustix = { version = "0.38.9", default-features = false, features = ["process"] }
 
 [build-dependencies]
-prost-build = "0.12.1"
-protox = "0.6.0"
+constcat = "0.5.0"
+prost-build = "0.13.1"
+protox = "0.7.0"
 
 [dev-dependencies]
 assert_matches = "1.5.0"

avbroot/build.rs

@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: 2023 Andrew Gunnerson
+ * SPDX-FileCopyrightText: 2023-2024 Andrew Gunnerson
  * SPDX-License-Identifier: GPL-3.0-only
  */
 
@@ -30,8 +30,62 @@ fn main() {
 
     let file_descriptors = protox::compile(&protos, [&in_dir]).unwrap();
 
+    const CUE_AI: &str = ".chromeos_update_engine.ApexInfo";
+    const CUE_DAM: &str = ".chromeos_update_engine.DeltaArchiveManifest";
+    const CUE_DPG: &str = ".chromeos_update_engine.DynamicPartitionGroup";
+    const CUE_DPM: &str = ".chromeos_update_engine.DynamicPartitionMetadata";
+    const CUE_PU: &str = ".chromeos_update_engine.PartitionUpdate";
+    const CUE_VABCFS: &str = ".chromeos_update_engine.VABCFeatureSet";
+
+    const DERIVE_SERDE: &str = "#[derive(serde::Deserialize, serde::Serialize)]";
+    const SERDE_DEFAULT: &str = "#[serde(default)]";
+    const SERDE_SKIP: &str = "#[serde(skip)]";
+    const SERDE_SKIP_IF_VEC_EMPTY: &str = "#[serde(skip_serializing_if = \"Vec::is_empty\")]";
+
+    use constcat::concat as c;
+
     prost_build::Config::new()
         .btree_map(["."])
+        // Allow deserializing and serializing the types we care about.
+        .type_attribute(CUE_AI, DERIVE_SERDE)
+        .type_attribute(CUE_DAM, DERIVE_SERDE)
+        .type_attribute(CUE_DPG, DERIVE_SERDE)
+        .type_attribute(CUE_DPM, DERIVE_SERDE)
+        .type_attribute(CUE_PU, DERIVE_SERDE)
+        .type_attribute(CUE_VABCFS, DERIVE_SERDE)
+        // Allow default-initializing all fields.
+        .type_attribute(CUE_AI, SERDE_DEFAULT)
+        .type_attribute(CUE_DAM, SERDE_DEFAULT)
+        .type_attribute(CUE_DPG, SERDE_DEFAULT)
+        .type_attribute(CUE_DPM, SERDE_DEFAULT)
+        .type_attribute(CUE_PU, SERDE_DEFAULT)
+        .type_attribute(CUE_VABCFS, SERDE_DEFAULT)
+        // Don't serialize fields that define the structure of the payload
+        // binary and that we recompute during packing.
+        .field_attribute(c!(CUE_DAM, ".signatures_offset"), SERDE_SKIP)
+        .field_attribute(c!(CUE_DAM, ".signatures_size"), SERDE_SKIP)
+        .field_attribute(c!(CUE_PU, ".operations"), SERDE_SKIP)
+        .field_attribute(c!(CUE_PU, ".estimate_cow_size"), SERDE_SKIP)
+        .field_attribute(c!(CUE_PU, ".old_partition_info"), SERDE_SKIP)
+        .field_attribute(c!(CUE_PU, ".new_partition_info"), SERDE_SKIP)
+        // Don't serialize AVB 1.0 fields.
+        .field_attribute(c!(CUE_PU, ".hash_tree_data_extent"), SERDE_SKIP)
+        .field_attribute(c!(CUE_PU, ".hash_tree_extent"), SERDE_SKIP)
+        .field_attribute(c!(CUE_PU, ".hash_tree_algorithm"), SERDE_SKIP)
+        .field_attribute(c!(CUE_PU, ".hash_tree_salt"), SERDE_SKIP)
+        .field_attribute(c!(CUE_PU, ".fec_data_extent"), SERDE_SKIP)
+        .field_attribute(c!(CUE_PU, ".fec_extent"), SERDE_SKIP)
+        .field_attribute(c!(CUE_PU, ".fec_roots"), SERDE_SKIP)
+        // Don't serialize fields for incremental OTAs.
+        .field_attribute(c!(CUE_PU, ".merge_operations"), SERDE_SKIP)
+        // Don't serialize fields for vendor-signed images, which update_engine
+        // doesn't support anyway.
+        .field_attribute(c!(CUE_PU, ".new_partition_signature"), SERDE_SKIP)
+        // Don't serialize empty lists.
+        .field_attribute(c!(CUE_DAM, ".apex_info"), SERDE_SKIP_IF_VEC_EMPTY)
+        .field_attribute(c!(CUE_DAM, ".partitions"), SERDE_SKIP_IF_VEC_EMPTY)
+        .field_attribute(c!(CUE_DPG, ".partition_names"), SERDE_SKIP_IF_VEC_EMPTY)
+        .field_attribute(c!(CUE_DPM, ".groups"), SERDE_SKIP_IF_VEC_EMPTY)
         .compile_fds(file_descriptors)
         .unwrap();
 }

avbroot/src/cli/args.rs

@@ -37,39 +37,6 @@ pub enum Command {
     MagiskInfo(boot::MagiskInfoCli),
 }
 
-#[derive(Debug, Clone, Copy, ValueEnum)]
-pub enum LogLevel {
-    Trace,
-    Debug,
-    Info,
-    Warn,
-    Error,
-}
-
-impl LogLevel {
-    fn as_level(self) -> Level {
-        match self {
-            Self::Trace => Level::TRACE,
-            Self::Debug => Level::DEBUG,
-            Self::Info => Level::INFO,
-            Self::Warn => Level::WARN,
-            Self::Error => Level::ERROR,
-        }
-    }
-}
-
-impl Default for LogLevel {
-    fn default() -> Self {
-        Self::Info
-    }
-}
-
-impl fmt::Display for LogLevel {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        f.write_str(self.to_possible_value().ok_or(fmt::Error)?.get_name())
-    }
-}
-
 #[derive(Debug, Clone, Copy, ValueEnum)]
 pub enum LogFormat {
     Short,
@@ -96,8 +63,8 @@ pub struct Cli {
     pub command: Command,
 
     /// Lowest log message severity to output.
-    #[arg(long, global = true, value_name = "LEVEL", default_value_t)]
-    pub log_level: LogLevel,
+    #[arg(long, global = true, value_name = "LEVEL", default_value_t = Level::INFO)]
+    pub log_level: Level,
 
     /// Output format for log messages.
     #[arg(long, global = true, value_name = "FORMAT", default_value_t)]
@@ -124,11 +91,11 @@ impl FormatTime for ShortUptime {
     }
 }
 
-pub fn init_logging(log_level: LogLevel, log_format: LogFormat) {
+pub fn init_logging(log_level: Level, log_format: LogFormat) {
     let builder = tracing_subscriber::fmt()
         .with_writer(io::stderr)
         .with_ansi(io::stderr().is_terminal())
-        .with_max_level(log_level.as_level());
+        .with_max_level(log_level);
 
     match log_format {
         LogFormat::Short => {
@@ -164,7 +131,7 @@ pub fn main(logging_initialized: &AtomicBool, cancel_signal: &AtomicBool) -> Res
         Command::HashTree(c) => hashtree::hash_tree_main(&c, cancel_signal),
         Command::Key(c) => key::key_main(&c),
         Command::Ota(c) => ota::ota_main(&c, cancel_signal),
-        Command::Payload(c) => payload::payload_main(&c),
+        Command::Payload(c) => payload::payload_main(&c, cancel_signal),
         // Deprecated aliases.
         Command::Patch(c) => ota::patch_subcommand(&c, cancel_signal),
         Command::Extract(c) => ota::extract_subcommand(&c, cancel_signal),

avbroot/src/cli/cpio.rs

@@ -303,7 +303,7 @@ struct UnpackCli {
     #[arg(short, long, value_name = "FILE", value_parser)]
     input: PathBuf,
 
-    /// Path to output cpio info TOML.
+    /// Path to output info TOML.
     #[arg(long, value_name = "FILE", value_parser, default_value = "cpio.toml")]
     output_info: PathBuf,
 

avbroot/src/cli/ota.rs

@@ -34,7 +34,7 @@ use crate::{
         avb::{self, Descriptor, Header},
         ota::{self, SigningWriter, ZipEntry},
         padding,
-        payload::{self, PayloadHeader, PayloadWriter},
+        payload::{self, PayloadHeader, PayloadWriter, VabcAlgo},
     },
     patch::{
         boot::{
@@ -623,7 +623,7 @@ fn update_vbmeta_headers(
 /// If `ranges` is [`None`], then the entire file is compressed. Otherwise, only
 /// the chunks containing the specified ranges are compressed. In the latter
 /// scenario, unmodified chunks must be copied from the original payload.
-fn compress_image(
+pub fn compress_image(
     name: &str,
     file: &mut PSeekFile,
     header: &mut PayloadHeader,
@@ -676,12 +676,11 @@ fn compress_image(
     // Otherwise, compress the entire image. If VABC is enabled, we need to
     // update the CoW size estimate or else the CoW block device may run out of
     // space during flashing.
-    let need_cow = partition.estimate_cow_size.is_some();
-    if need_cow {
+    let vabc_algo = if partition.estimate_cow_size.is_some() {
         info!("Needs updated CoW size estimate: {name}");
 
-        // Only CoW v2 + lz4 seems to exist in the wild currently, so that is
-        // all we support.
+        // Only CoW v2 seems to exist in the wild currently, so that is all we
+        // support.
         let Some(dpm) = &header.manifest.dynamic_partition_metadata else {
             bail!("Dynamic partition metadata is missing");
         };
@@ -696,13 +695,17 @@ fn compress_image(
         }
 
         let compression = dpm.vabc_compression_param();
-        if compression != "lz4" {
+        let Some(vabc_algo) = VabcAlgo::new(compression) else {
             bail!("Unsupported VABC compression: {compression}");
-        }
-    }
+        };
+
+        Some(vabc_algo)
+    } else {
+        None
+    };
 
     let (partition_info, operations, cow_estimate) =
-        payload::compress_image(&*file, &writer, name, block_size, need_cow, cancel_signal)?;
+        payload::compress_image(&*file, &writer, name, block_size, vabc_algo, cancel_signal)?;
 
     partition.new_partition_info = Some(partition_info);
     partition.operations = operations;
@@ -726,15 +729,13 @@ fn patch_ota_payload(
     cert_ota: &Certificate,
     cancel_signal: &AtomicBool,
 ) -> Result<(String, u64)> {
-    let header = PayloadHeader::from_reader(payload.reopen_boxed()?)
+    let mut header = PayloadHeader::from_reader(payload.reopen_boxed()?)
         .context("Failed to load OTA payload header")?;
     if !header.is_full_ota() {
         bail!("Payload is a delta OTA, not a full OTA");
     }
 
-    let header = Mutex::new(header);
-    let mut header_locked = header.lock().unwrap();
-    let all_partitions = header_locked
+    let all_partitions = header
         .manifest
         .partitions
         .iter()
@@ -752,7 +753,7 @@ fn patch_ota_payload(
     // Determine what images need to be patched. For simplicity, we pre-read all
     // vbmeta images since they're tiny. They're discarded later if the they
     // don't need to be modified.
-    let required_images = RequiredImages::new(&header_locked.manifest);
+    let required_images = RequiredImages::new(&header.manifest);
     let vbmeta_images = required_images.iter_vbmeta().collect::<HashSet<_>>();
 
     // The set of source images to be inserted into the new payload, replacing
@@ -764,7 +765,7 @@ fn patch_ota_payload(
         payload,
         &required_images,
         external_images,
-        &header_locked,
+        &header,
         cancel_signal,
     )?;
 
@@ -806,7 +807,7 @@ fn patch_ota_payload(
         &mut vbmeta_order,
         clear_vbmeta_flags,
         key_avb,
-        header_locked.manifest.block_size().into(),
+        header.manifest.block_size().into(),
     )?;
 
     // Unmodified vbmeta images no longer need to be kept around either.
@@ -818,7 +819,7 @@ fn patch_ota_payload(
             let modified_operations = compress_image(
                 &name,
                 &mut input_file.file,
-                &mut header_locked,
+                &mut header,
                 // We can only perform the optimization of avoiding
                 // recompression if the image came from the original payload.
                 if name == system_target && !external_images.contains_key(&name) {
@@ -836,7 +837,7 @@ fn patch_ota_payload(
 
     info!("Generating new OTA payload");
 
-    let mut payload_writer = PayloadWriter::new(writer, header_locked.clone(), key_ota.clone())
+    let mut payload_writer = PayloadWriter::new(writer, header.clone(), key_ota.clone())
         .context("Failed to write payload header")?;
     let mut orig_payload_reader = payload.reopen_boxed().context("Failed to open payload")?;
 
@@ -854,7 +855,7 @@ fn patch_ota_payload(
 
         let pi = payload_writer.partition_index().unwrap();
         let oi = payload_writer.operation_index().unwrap();
-        let orig_partition = &header_locked.manifest.partitions[pi];
+        let orig_partition = &header.manifest.partitions[pi];
         let orig_operation = &orig_partition.operations[oi];
         let data_offset = orig_operation
             .data_offset
@@ -884,7 +885,7 @@ fn patch_ota_payload(
 
         // Otherwise, copy from the original payload.
         let data_offset = data_offset
-            .checked_add(header_locked.blob_offset)
+            .checked_add(header.blob_offset)
             .ok_or_else(|| anyhow!("data_offset overflow in partition #{pi} operation #{oi}"))?;
 
         orig_payload_reader
@@ -900,7 +901,7 @@ fn patch_ota_payload(
         .with_context(|| format!("Failed to copy from original payload: {name}"))?;
     }
 
-    let (_, properties, metadata_size) = payload_writer
+    let (_, _, properties, metadata_size) = payload_writer
         .finish()
         .context("Failed to finalize payload")?;
 
@@ -1091,7 +1092,7 @@ fn patch_ota_zip(
     Ok((metadata, payload_metadata_size.unwrap()))
 }
 
-fn extract_ota_zip(
+pub fn extract_payload(
     raw_reader: &PSeekFile,
     directory: &Dir,
     payload_offset: u64,
@@ -1442,7 +1443,7 @@ pub fn extract_subcommand(cli: &ExtractCli, cancel_signal: &AtomicBool) -> Resul
     let directory = Dir::open_ambient_dir(&cli.directory, authority)
         .with_context(|| format!("Failed to open directory: {:?}", cli.directory))?;
 
-    extract_ota_zip(
+    extract_payload(
         &raw_reader,
         &directory,
         payload_offset,
@@ -1654,7 +1655,7 @@ pub fn verify_subcommand(cli: &VerifyCli, cancel_signal: &AtomicBool) -> Result<
         .cloned()
         .collect::<BTreeSet<_>>();
 
-    extract_ota_zip(
+    extract_payload(
         &raw_reader,
         &temp_dir,
         pf_payload.offset,
@@ -1864,7 +1865,7 @@ pub struct PatchCli {
         value_names = ["PARTITION", "FILE"],
         value_parser = value_parser!(OsString),
         num_args = 2,
-        help_heading = HEADING_PATH,
+        help_heading = HEADING_PATH
     )]
     pub replace: Vec<OsString>,
 

avbroot/src/cli/payload.rs

@@ -3,31 +3,448 @@
  * SPDX-License-Identifier: GPL-3.0-only
  */
 
-use std::{fs::File, io::BufReader, path::PathBuf};
+use std::{
+    collections::HashMap,
+    ffi::{OsStr, OsString},
+    fs::{self, File},
+    io::{BufReader, BufWriter, Seek, SeekFrom},
+    path::{Path, PathBuf},
+    sync::atomic::AtomicBool,
+};
 
-use anyhow::{Context, Result};
-use clap::{Parser, Subcommand};
+use anyhow::{anyhow, bail, Context, Result};
+use cap_std::{ambient_authority, fs::Dir};
+use clap::{Args, Parser, Subcommand};
+use tracing::info;
 
-use crate::{format::payload::PayloadHeader, stream::FromReader};
+use crate::{
+    cli::ota,
+    crypto::{self, PassphraseSource, RsaSigningKey},
+    format::payload::{PayloadHeader, PayloadWriter},
+    stream::{self, FromReader, PSeekFile},
+};
 
-fn info_subcommand(cli: &InfoCli) -> Result<()> {
-    let mut reader = File::open(&cli.input)
+fn open_reader(path: &Path) -> Result<(BufReader<File>, PayloadHeader)> {
+    let mut reader = File::open(path)
         .map(BufReader::new)
-        .with_context(|| format!("Failed to open payload: {:?}", cli.input))?;
+        .with_context(|| format!("Failed to open payload for reading: {path:?}"))?;
     let header = PayloadHeader::from_reader(&mut reader)
-        .with_context(|| format!("Failed to read payload: {:?}", cli.input))?;
+        .with_context(|| format!("Failed to read payload header: {path:?}"))?;
+    if !header.is_full_ota() {
+        bail!("Payload is a delta OTA, not a full OTA");
+    }
 
-    println!("{header:#?}");
+    Ok((reader, header))
+}
+
+fn open_writer(
+    path: &Path,
+    header: PayloadHeader,
+    key: RsaSigningKey,
+) -> Result<PayloadWriter<BufWriter<File>>> {
+    let writer = File::create(path)
+        .map(BufWriter::new)
+        .with_context(|| format!("Failed to open payload for writing: {path:?}"))?;
+    let payload_writer = PayloadWriter::new(writer, header, key)
+        .with_context(|| format!("Failed to write payload header: {path:?}"))?;
+
+    Ok(payload_writer)
+}
+
+fn read_info(path: &Path) -> Result<PayloadHeader> {
+    let data = fs::read_to_string(path)
+        .with_context(|| format!("Failed to read payload info TOML: {path:?}"))?;
+    let info = toml_edit::de::from_str(&data)
+        .with_context(|| format!("Failed to parse payload info TOML: {path:?}"))?;
+
+    Ok(info)
+}
+
+fn write_info(path: &Path, manifest: &PayloadHeader) -> Result<()> {
+    let data = toml_edit::ser::to_string_pretty(manifest)
+        .with_context(|| format!("Failed to serialize payload info TOML: {path:?}"))?;
+    fs::write(path, data)
+        .with_context(|| format!("Failed to write payload info TOML: {path:?}"))?;
 
     Ok(())
 }
 
-pub fn payload_main(cli: &PayloadCli) -> Result<()> {
-    match &cli.command {
-        PayloadCommand::Info(c) => info_subcommand(c),
+fn display_header(cli: &PayloadCli, header: &PayloadHeader) {
+    if !cli.quiet {
+        println!("{header:#?}");
     }
 }
 
+fn load_key(group: &KeyGroup) -> Result<RsaSigningKey> {
+    let source = PassphraseSource::new(
+        &group.key,
+        group.pass_file.as_deref(),
+        group.pass_env_var.as_deref(),
+    );
+    let signing_key = if let Some(helper) = &group.signing_helper {
+        let public_key = crypto::read_pem_public_key_file(&group.key)
+            .with_context(|| format!("Failed to load key: {:?}", group.key))?;
+
+        RsaSigningKey::External {
+            program: helper.clone(),
+            public_key_file: group.key.clone(),
+            public_key,
+            passphrase_source: source,
+        }
+    } else {
+        let private_key = crypto::read_pem_key_file(&group.key, &source)
+            .with_context(|| format!("Failed to load key: {:?}", group.key))?;
+
+        RsaSigningKey::Internal(private_key)
+    };
+
+    Ok(signing_key)
+}
+
+fn unpack_subcommand(
+    payload_cli: &PayloadCli,
+    cli: &UnpackCli,
+    cancel_signal: &AtomicBool,
+) -> Result<()> {
+    let (mut reader, header) = open_reader(&cli.input)?;
+    let payload_size = reader
+        .seek(SeekFrom::End(0))
+        .with_context(|| format!("Failed to get file size: {:?}", cli.input))?;
+
+    display_header(payload_cli, &header);
+
+    write_info(&cli.output_info, &header)?;
+
+    let authority = ambient_authority();
+    Dir::create_ambient_dir_all(&cli.output_images, authority)
+        .with_context(|| format!("Failed to create directory: {:?}", cli.output_images))?;
+    let directory = Dir::open_ambient_dir(&cli.output_images, authority)
+        .with_context(|| format!("Failed to open directory: {:?}", cli.output_images))?;
+
+    ota::extract_payload(
+        &PSeekFile::new(reader.into_inner()),
+        &directory,
+        0,
+        payload_size,
+        &header,
+        &header
+            .manifest
+            .partitions
+            .iter()
+            .map(|p| &p.partition_name)
+            .cloned()
+            .collect(),
+        cancel_signal,
+    )?;
+
+    Ok(())
+}
+
+fn pack_subcommand(
+    payload_cli: &PayloadCli,
+    cli: &PackCli,
+    cancel_signal: &AtomicBool,
+) -> Result<()> {
+    let signing_key = load_key(&cli.key)?;
+
+    let mut header = read_info(&cli.input_info)?;
+
+    let authority = ambient_authority();
+    let directory = Dir::open_ambient_dir(&cli.input_images, authority)
+        .with_context(|| format!("Failed to open directory: {:?}", cli.input_images))?;
+
+    for p in &header.manifest.partitions {
+        let name = &p.partition_name;
+
+        if Path::new(name).file_name() != Some(OsStr::new(name)) {
+            bail!("Unsafe partition name: {name}");
+        }
+    }
+
+    // Pre-open all of the image files.
+    let input_files = header
+        .manifest
+        .partitions
+        .iter()
+        .map(|p| {
+            let path = format!("{}.img", p.partition_name);
+            let file = directory
+                .open(&path)
+                .map(|f| PSeekFile::new(f.into_std()))
+                .with_context(|| format!("Failed to open file: {path:?}"))?;
+
+            Ok((p.partition_name.clone(), file))
+        })
+        .collect::<Result<HashMap<_, _>>>()?;
+
+    // Compress the images and compute the list of install operations for
+    // insertion into the payload header. The compressed data is stored in new
+    // temp files and the original input files are dropped.
+    let mut compressed_files = input_files
+        .into_iter()
+        .map(|(name, mut input_file)| {
+            ota::compress_image(&name, &mut input_file, &mut header, None, cancel_signal)
+                .with_context(|| format!("Failed to compress image: {name}"))?;
+
+            Ok((name, input_file))
+        })
+        .collect::<Result<HashMap<_, _>>>()?;
+
+    info!("Generating new OTA payload");
+
+    // Now we can write the actual payload. With everything precomputed, this is
+    // mostly just a simple copy.
+    let mut payload_writer = open_writer(&cli.output, header.clone(), signing_key)?;
+
+    while payload_writer
+        .begin_next_operation()
+        .context("Failed to begin next payload blob entry")?
+    {
+        let name = payload_writer.partition().unwrap().partition_name.clone();
+        let operation = payload_writer.operation().unwrap();
+
+        let Some(data_length) = operation.data_length else {
+            // Otherwise, this is a ZERO/DISCARD operation.
+            continue;
+        };
+
+        let pi = payload_writer.partition_index().unwrap();
+        let oi = payload_writer.operation_index().unwrap();
+        let orig_partition = &header.manifest.partitions[pi];
+        let orig_operation = &orig_partition.operations[oi];
+        let data_offset = orig_operation
+            .data_offset
+            .ok_or_else(|| anyhow!("Missing data_offset in partition #{pi} operation #{oi}"))?;
+
+        // The compressed chunks are laid out sequentially and data_offset is
+        // set to the offset within that file.
+        let Some(input_file) = compressed_files.get_mut(&name) else {
+            unreachable!("Compressed data not found for image: {name}");
+        };
+
+        input_file
+            .seek(SeekFrom::Start(data_offset))
+            .with_context(|| format!("Failed to seek image: {name}"))?;
+
+        stream::copy_n(input_file, &mut payload_writer, data_length, cancel_signal)
+            .with_context(|| format!("Failed to copy from replacement image: {name}"))?;
+    }
+
+    let (_, header, properties, _) = payload_writer
+        .finish()
+        .context("Failed to finalize payload")?;
+
+    // Display the header information now that it has been finalized.
+    display_header(payload_cli, &header);
+
+    // Optionally, write payload_properties.txt.
+    if let Some(path) = &cli.output_properties {
+        fs::write(path, properties)
+            .with_context(|| format!("Failed to write payload properties: {path:?}"))?;
+    }
+
+    Ok(())
+}
+
+fn repack_subcommand(
+    payload_cli: &PayloadCli,
+    cli: &RepackCli,
+    cancel_signal: &AtomicBool,
+) -> Result<()> {
+    let signing_key = load_key(&cli.key)?;
+
+    let (mut reader, header) = open_reader(&cli.input)?;
+
+    info!("Generating new OTA payload");
+
+    let mut payload_writer = open_writer(&cli.output, header.clone(), signing_key)?;
+
+    while payload_writer
+        .begin_next_operation()
+        .context("Failed to begin next payload blob entry")?
+    {
+        let name = payload_writer.partition().unwrap().partition_name.clone();
+        let operation = payload_writer.operation().unwrap();
+
+        let Some(data_length) = operation.data_length else {
+            // Otherwise, this is a ZERO/DISCARD operation.
+            continue;
+        };
+
+        let pi = payload_writer.partition_index().unwrap();
+        let oi = payload_writer.operation_index().unwrap();
+        let orig_partition = &header.manifest.partitions[pi];
+        let orig_operation = &orig_partition.operations[oi];
+        let data_offset = orig_operation
+            .data_offset
+            .ok_or_else(|| anyhow!("Missing data_offset in partition #{pi} operation #{oi}"))?;
+
+        // Directly copy blobs from the original payload.
+        let data_offset = data_offset
+            .checked_add(header.blob_offset)
+            .ok_or_else(|| anyhow!("data_offset overflow in partition #{pi} operation #{oi}"))?;
+
+        reader
+            .seek(SeekFrom::Start(data_offset))
+            .with_context(|| format!("Failed to seek original payload to {data_offset}"))?;
+
+        stream::copy_n(&mut reader, &mut payload_writer, data_length, cancel_signal)
+            .with_context(|| format!("Failed to copy from original payload: {name}"))?;
+    }
+
+    let (_, header, properties, _) = payload_writer
+        .finish()
+        .context("Failed to finalize payload")?;
+
+    // Display the header information now that it has been finalized.
+    display_header(payload_cli, &header);
+
+    // Optionally, write payload_properties.txt.
+    if let Some(path) = &cli.output_properties {
+        fs::write(path, properties)
+            .with_context(|| format!("Failed to write payload properties: {path:?}"))?;
+    }
+
+    Ok(())
+}
+
+fn info_subcommand(payload_cli: &PayloadCli, cli: &InfoCli) -> Result<()> {
+    let (_, header) = open_reader(&cli.input)?;
+
+    display_header(payload_cli, &header);
+
+    Ok(())
+}
+
+pub fn payload_main(cli: &PayloadCli, cancel_signal: &AtomicBool) -> Result<()> {
+    match &cli.command {
+        PayloadCommand::Unpack(c) => unpack_subcommand(cli, c, cancel_signal),
+        PayloadCommand::Pack(c) => pack_subcommand(cli, c, cancel_signal),
+        PayloadCommand::Repack(c) => repack_subcommand(cli, c, cancel_signal),
+        PayloadCommand::Info(c) => info_subcommand(cli, c),
+    }
+}
+
+#[derive(Debug, Args)]
+struct KeyGroup {
+    /// Path to signing key.
+    ///
+    /// This should normally be a private key. However, if --signing-helper is
+    /// used, then it should be a public key instead.
+    #[arg(short, long, value_name = "FILE", value_parser)]
+    key: PathBuf,
+
+    /// Environment variable containing private key passphrase.
+    #[arg(long, value_name = "ENV_VAR", value_parser, group = "pass")]
+    pass_env_var: Option<OsString>,
+
+    /// File containing private key passphrase.
+    #[arg(long, value_name = "FILE", value_parser, group = "pass")]
+    pass_file: Option<PathBuf>,
+
+    /// External program for signing.
+    ///
+    /// If this option is specified, then --key must refer to a public key. The
+    /// program will be invoked as:
+    ///
+    /// <program> <algo> <public key> [file <pass file>|env <pass env>]
+    #[arg(long, value_name = "PROGRAM", value_parser)]
+    signing_helper: Option<PathBuf>,
+}
+
+/// Unpack a payload binary.
+///
+/// Each partition is extracted to `<partition name>.img` in the output images
+/// directory. The payload header metadata is written to the info TOML file.
+///
+/// If any partition names are unsafe to use in a path, the extraction process
+/// will fail and exit. Extracted files are never written outside of the tree
+/// directory, even if an external process tries to interfere.
+#[derive(Debug, Parser)]
+struct UnpackCli {
+    /// Path to input payload binary.
+    #[arg(short, long, value_name = "FILE", value_parser)]
+    input: PathBuf,
+
+    /// Path to output info TOML.
+    #[arg(
+        long,
+        value_name = "FILE",
+        value_parser,
+        default_value = "payload.toml"
+    )]
+    output_info: PathBuf,
+
+    /// Path to output images directory.
+    #[arg(
+        long,
+        value_name = "DIR",
+        value_parser,
+        default_value = "payload_images"
+    )]
+    output_images: PathBuf,
+}
+
+/// Pack a payload binary.
+///
+/// The new payload binary will *only* contain images listed in the info TOML
+/// file. Extra images in the input images directory that aren't listed will be
+/// silently ignored. Images are added to the payload in the order that they are
+/// listed in the info TOML file.
+#[derive(Debug, Parser)]
+struct PackCli {
+    /// Path to output payload binary.
+    #[arg(short, long, value_name = "FILE", value_parser)]
+    output: PathBuf,
+
+    /// Path to output payload properties file.
+    #[arg(short, long, value_name = "FILE", value_parser)]
+    output_properties: Option<PathBuf>,
+
+    /// Path to input info TOML.
+    #[arg(
+        long,
+        value_name = "FILE",
+        value_parser,
+        default_value = "payload.toml"
+    )]
+    input_info: PathBuf,
+
+    /// Path to input images directory.
+    #[arg(
+        long,
+        value_name = "DIR",
+        value_parser,
+        default_value = "payload_images"
+    )]
+    input_images: PathBuf,
+
+    #[command(flatten)]
+    key: KeyGroup,
+}
+
+/// Repack a payload binary.
+///
+/// This command is equivalent to running `unpack` and `pack`, except without
+/// storing the unpacked data to disk nor recompressing the partition images.
+#[derive(Debug, Parser)]
+struct RepackCli {
+    /// Path to input payload binary.
+    #[arg(short, long, value_name = "FILE", value_parser)]
+    input: PathBuf,
+
+    /// Path to output payload binary.
+    #[arg(short, long, value_name = "FILE", value_parser)]
+    output: PathBuf,
+
+    /// Path to output payload properties file.
+    #[arg(short, long, value_name = "FILE", value_parser)]
+    output_properties: Option<PathBuf>,
+
+    #[command(flatten)]
+    key: KeyGroup,
+}
+
 /// Display payload information.
 #[derive(Debug, Parser)]
 struct InfoCli {
@@ -38,6 +455,9 @@ struct InfoCli {
 
 #[derive(Debug, Subcommand)]
 enum PayloadCommand {
+    Unpack(UnpackCli),
+    Pack(PackCli),
+    Repack(RepackCli),
     Info(InfoCli),
 }
 
@@ -46,4 +466,8 @@ enum PayloadCommand {
 pub struct PayloadCli {
     #[command(subcommand)]
     command: PayloadCommand,
+
+    /// Don't print payload header information.
+    #[arg(short, long, global = true)]
+    quiet: bool,
 }

avbroot/src/format/payload.rs

@@ -5,6 +5,7 @@
 
 use std::{
     collections::{HashMap, HashSet},
+    fmt,
     io::{self, Cursor, Read, Seek, SeekFrom, Write},
     ops::Range,
     sync::atomic::AtomicBool,
@@ -26,6 +27,7 @@ use rayon::{
     prelude::{IntoParallelIterator, IntoParallelRefIterator, ParallelIterator},
 };
 use ring::digest::{Context, Digest};
+use serde::{Deserialize, Serialize};
 use thiserror::Error;
 use x509_cert::Certificate;
 
@@ -104,11 +106,13 @@ pub enum Error {
 
 type Result<T> = std::result::Result<T, Error>;
 
-#[derive(Clone, Debug)]
+#[derive(Clone, Debug, Deserialize, Serialize)]
 pub struct PayloadHeader {
     pub version: u64,
     pub manifest: DeltaArchiveManifest,
+    #[serde(skip)]
     pub metadata_signature_size: u32,
+    #[serde(skip)]
     pub blob_offset: u64,
 }
 
@@ -384,12 +388,12 @@ impl<W: Write> PayloadWriter<W> {
     }
 
     /// Finalize the payload. If this function is not called, the payload will
-    /// be left in an incomplete state. Returns the original writer, the
-    /// contents that should be written for `payload_properties.txt` and the
-    /// length of the header + manifest + manifest signature sections (for
-    /// constructing the `payload_metadata.bin` OTA metadata property files
+    /// be left in an incomplete state. Returns the original writer, the final
+    /// header, the contents that should be written for `payload_properties.txt`
+    /// and the length of the header + manifest + manifest signature sections
+    /// (for constructing the `payload_metadata.bin` OTA metadata property files
     /// entry).
-    pub fn finish(mut self) -> Result<(W, String, u64)> {
+    pub fn finish(mut self) -> Result<(W, PayloadHeader, String, u64)> {
         // Append payload signature.
         let payload_partial_hash = self.h_partial.clone().finish();
         let payload_sig = sign_digest(payload_partial_hash.as_ref(), &self.key)?;
@@ -413,7 +417,7 @@ impl<W: Write> PayloadWriter<W> {
             self.metadata_size as u64,
         );
 
-        Ok((self.inner, properties, metadata_with_sig_size))
+        Ok((self.inner, self.header, properties, metadata_with_sig_size))
     }
 
     /// Prepare for writing the next source data blob corresponding to an
@@ -884,19 +888,50 @@ fn compress_chunk(raw_data: &[u8], cancel_signal: &AtomicBool) -> Result<(Vec<u8
     Ok((data, digest_compressed))
 }
 
-fn compress_cow_size(mut raw_data: &[u8], block_size: u32) -> u64 {
-    let mut total = 0;
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Deserialize, Serialize)]
+pub enum VabcAlgo {
+    Lz4,
+    Gzip,
+}
 
-    while !raw_data.is_empty() {
-        let n = raw_data.len().min(block_size as usize);
-        let compressed = lz4_flex::block::compress(&raw_data[..n]);
-
-        total += compressed.len().min(n) as u64;
-
-        raw_data = &raw_data[n..];
+impl VabcAlgo {
+    pub fn new(name: &str) -> Option<Self> {
+        match name {
+            "lz4" => Some(Self::Lz4),
+            "gz" => Some(Self::Gzip),
+            _ => None,
+        }
     }
 
-    total
+    fn compressed_size(&self, mut raw_data: &[u8], block_size: u32) -> u64 {
+        let mut total = 0;
+
+        while !raw_data.is_empty() {
+            let n = raw_data.len().min(block_size as usize);
+            let compressed = match self {
+                Self::Lz4 => lz4_flex::block::compress(&raw_data[..n]),
+                // We use the miniz_oxide backend for flate2, but flate2 doesn't
+                // expose a nice function for compressing to a vec, so just use
+                // miniz_oxide directly.
+                Self::Gzip => miniz_oxide::deflate::compress_to_vec_zlib(&raw_data[..n], 9),
+            };
+
+            total += compressed.len().min(n) as u64;
+
+            raw_data = &raw_data[n..];
+        }
+
+        total
+    }
+}
+
+impl fmt::Display for VabcAlgo {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            Self::Lz4 => f.write_str("lz4"),
+            Self::Gzip => f.write_str("gz"),
+        }
+    }
 }
 
 /// Compress the image and return the corresponding information to insert into
@@ -907,18 +942,18 @@ fn compress_cow_size(mut raw_data: &[u8], block_size: u32) -> u64 {
 /// update [`InstallOperation::data_offset`] in each operation manually because
 /// the initial values are relative to 0.
 ///
-/// If `need_cow_estimate` is true, the VABC CoW v2 + lz4 size estimate will be
-/// computed. The caller must update [`PartitionUpdate::estimate_cow_size`] with
-/// this value or else update_engine may fail to flash the partition due to
-/// running out of space on the CoW block device. CoW v2 + other algorithms and
-/// also CoW v3 are currently unsupported because there currently are no known
-/// OTAs that use those configurations.
+/// If `vabc_algo` is set, the VABC CoW v2 size estimate will be computed. The
+/// caller must update [`PartitionUpdate::estimate_cow_size`] with this value or
+/// else update_engine may fail to flash the partition due to running out of
+/// space on the CoW block device. CoW v2 + other algorithms and also CoW v3 are
+/// currently unsupported because there currently are no known OTAs that use
+/// those configurations.
 pub fn compress_image(
     input: &(dyn ReadSeekReopen + Sync),
     output: &(dyn WriteSeekReopen + Sync),
     partition_name: &str,
     block_size: u32,
-    need_cow_estimate: bool,
+    vabc_algo: Option<VabcAlgo>,
     cancel_signal: &AtomicBool,
 ) -> Result<(PartitionInfo, Vec<InstallOperation>, Option<u64>)> {
     const CHUNK_SIZE: u64 = 2 * 1024 * 1024;
@@ -977,8 +1012,8 @@ pub fn compress_image(
             .map(
                 |(raw_offset, raw_data)| -> Result<(Vec<u8>, InstallOperation, u64)> {
                     let (data, digest_compressed) = compress_chunk(&raw_data, cancel_signal)?;
-                    let cow_size = if need_cow_estimate {
-                        compress_cow_size(&raw_data, block_size)
+                    let cow_size = if let Some(algo) = vabc_algo {
+                        algo.compressed_size(&raw_data, block_size)
                     } else {
                         0
                     };
@@ -1025,11 +1060,25 @@ pub fn compress_image(
         hash: Some(digest_uncompressed.as_ref().to_vec()),
     };
 
-    let cow_estimate = if need_cow_estimate {
-        // Because lz4_flex compresses better than official lz4.
-        let fudge = cow_estimate / 100;
+    let cow_estimate = if vabc_algo.is_some() {
+        // lz4_flex and miniz_oxide usually compress better than the lz4 and
+        // zlib implementations used by libsnapshot_cow. Make up for this by
+        // adding percentage-based overhead.
+        cow_estimate += cow_estimate / 100;
 
-        Some(cow_estimate + fudge)
+        // We also need to account for constant overhead, especially with
+        // smaller partitions. We can match what delta_generator normally adds
+        // in CowWriterV2::InitPos() exactly. Since we only ever create full
+        // OTAs, we can assume that all CoW operations are kCowReplaceOp.
+
+        // sizeof(CowHeader).
+        cow_estimate += 38;
+        // header_.buffer_size (equal to BUFFER_REGION_DEFAULT_SIZE).
+        cow_estimate += 2 * 1024 * 1024;
+        // CowOptions::cluster_ops * sizeof(CowOperationV2).
+        cow_estimate += 200 * 20;
+
+        Some(cow_estimate)
     } else {
         None
     };

avbroot/src/patch/boot.rs

@@ -161,6 +161,7 @@ impl MagiskRootPatcher {
     const VER_XZ_BACKUP: Range<u32> =
         26403..Self::VERS_SUPPORTED[Self::VERS_SUPPORTED.len() - 1].end;
 
+    const ZIP_INIT_LD: &'static str = "lib/arm64-v8a/libinit-ld.so";
     const ZIP_LIBMAGISK: &'static str = "lib/arm64-v8a/libmagisk.so";
     const ZIP_LIBMAGISK32: &'static str = "lib/armeabi-v7a/libmagisk32.so";
     const ZIP_LIBMAGISK64: &'static str = "lib/arm64-v8a/libmagisk64.so";
@@ -462,6 +463,13 @@ impl BootImagePatch for MagiskRootPatcher {
             xz_files.insert(Self::ZIP_STUB, b"overlay.d/sbin/stub.xz");
         }
 
+        // Add init-ld, which only exists after Magisk commit
+        // 33aebb59763b6ec27209563035303700e998633d
+        if zip.file_names().any(|n| n == Self::ZIP_INIT_LD) {
+            debug!("Magisk init-ld found");
+            xz_files.insert(Self::ZIP_INIT_LD, b"overlay.d/sbin/init-ld.xz");
+        }
+
         for (source, target) in xz_files {
             let reader = zip
                 .by_name(source)

deny.toml

@@ -62,8 +62,6 @@ include-workspace = true
 bypass = [
     # Copies of unmodified crashwrangler objects for old macOS versions.
     { name = "honggfuzz", allow-globs = ["honggfuzz/third_party/mac/CrashReport_*.o"] },
-    # Test files for liblzma's test suite
-    { name = "liblzma-sys", allow-globs = ["xz/tests/compress_prepared_bcj_*"] },
 ]
 
 [sources]

e2e/e2e.toml

@@ -12,6 +12,9 @@ security_patch_level = "2024-01-01"
 # Google Pixel 7 Pro
 # What's unique: init_boot (boot v4) + vendor_boot (vendor v4)
 
+[profile.pixel_v4_gki]
+vabc_algo = "Lz4"
+
 [profile.pixel_v4_gki.partitions.boot]
 avb.signed = true
 data.type = "boot"
@@ -46,12 +49,15 @@ data.version = "vendor_v4"
 data.ramdisks = [["otacerts", "first_stage", "dsu_key_dir"]]
 
 [profile.pixel_v4_gki.hashes]
-original = "6b140c378d21eae2fa4fc581bce13a689b21bd32f5fba865698d1fd322f2f8c6"
-patched = "f00e9745f90754be28ce8355501d876759cd8336451e4c3633908fbb4217b422"
+original = "c00f891f941f3dddb28966f7b07f3acea773bee104dace82b37c2d1341f09422"
+patched = "ce9d8ee97828d233809742a5d3f23aa27b042675b1935ca9e3df0592c55788fd"
 
 # Google Pixel 6a
 # What's unique: boot (boot v4, no ramdisk) + vendor_boot (vendor v4, 2 ramdisks)
 
+[profile.pixel_v4_non_gki]
+vabc_algo = "Lz4"
+
 [profile.pixel_v4_non_gki.partitions.boot]
 avb.signed = true
 data.type = "boot"
@@ -80,12 +86,15 @@ data.version = "vendor_v4"
 data.ramdisks = [["init", "otacerts", "first_stage", "dsu_key_dir"], ["dlkm"]]
 
 [profile.pixel_v4_non_gki.hashes]
-original = "31963e6f81986c6686111f50e36b89e4d85ee5c02bc8e5ecd560528bc98d6fe7"
-patched = "43959409034dbb9aa0a605d7c5c0e7885012bbcd63510ec87d8e97015b37a746"
+original = "4d692bc777b568b0626d3c08d2e6f83f1b472db5ad903486daaec6a78d0cc26e"
+patched = "e27673e4f30933710c11d51f0e73849068cbe9bc9f54e6076bdd93f9a5c8ea0a"
 
 # Google Pixel 4a 5G
 # What's unique: boot (boot v3) + vendor_boot (vendor v3)
 
+[profile.pixel_v3]
+vabc_algo = "Lz4"
+
 [profile.pixel_v3.partitions.boot]
 avb.signed = true
 data.type = "boot"
@@ -115,12 +124,15 @@ data.version = "vendor_v3"
 data.ramdisks = [["otacerts", "first_stage", "dsu_key_dir"]]
 
 [profile.pixel_v3.hashes]
-original = "e684aacb54464098c1b8e3f499efe35dff10ea792e89d71a83404620d0108b3e"
-patched = "08e03ec327bf5bd841b91ad8d53028c3439a1722b423aaaac6cc0ceee4ef66b1"
+original = "f432dc7931520feb238474aa707dd5299747562ffe6129f3f763b5f11ac473ab"
+patched = "3850a2e73bd783a1ec4a70c59f37d2374e017c20df7ab4b591182b14d187c18e"
 
 # Google Pixel 4a
 # What's unique: boot (boot v2)
 
+[profile.pixel_v2]
+vabc_algo = "Gzip"
+
 [profile.pixel_v2.partitions.boot]
 avb.signed = false
 data.type = "boot"
@@ -144,5 +156,5 @@ data.type = "vbmeta"
 data.deps = ["system"]
 
 [profile.pixel_v2.hashes]
-original = "ee9568797d9195985f14753b89949d8ebb08c8863a32eceeeec6e8d94661b1cf"
-patched = "5e265094d4164cedde8f483911c58860f6008b314dc8e5ed3b44deb53fbb2f96"
+original = "1b45235b58054009cc496f6c3ee11d3dc16ed5c388c861761e26a6fce83103a0"
+patched = "193b2dc70dd465d686f35c7b7f74d2cc1b06a55e48cf5c2e4df0f667e03032fc"

e2e/src/cli.rs

@@ -5,8 +5,9 @@
 
 use std::{ffi::OsString, path::PathBuf};
 
-use avbroot::cli::args::{LogFormat, LogLevel};
+use avbroot::cli::args::LogFormat;
 use clap::{Args, Parser, Subcommand, ValueEnum};
+use tracing::Level;
 
 #[derive(Debug, Args)]
 pub struct ProfileGroup {
@@ -69,8 +70,8 @@ pub struct Cli {
     pub command: Command,
 
     /// Lowest log message severity to output.
-    #[arg(long, global = true, value_name = "LEVEL", default_value_t)]
-    pub log_level: LogLevel,
+    #[arg(long, global = true, value_name = "LEVEL", default_value_t = Level::INFO)]
+    pub log_level: Level,
 
     /// Output format for log messages.
     #[arg(long, global = true, value_name = "FORMAT", default_value_t = LogFormat::Medium)]

e2e/src/config.rs

@@ -6,6 +6,7 @@
 use std::{collections::BTreeMap, fs, path::Path};
 
 use anyhow::{Context, Result};
+use avbroot::format::payload::VabcAlgo;
 use serde::{Deserialize, Serialize};
 use toml_edit::DocumentMut;
 
@@ -109,6 +110,7 @@ pub struct Partition {
 #[derive(Serialize, Deserialize)]
 #[serde(deny_unknown_fields)]
 pub struct Profile {
+    pub vabc_algo: Option<VabcAlgo>,
     pub partitions: BTreeMap<String, Partition>,
     pub hashes: Hashes,
 }

e2e/src/main.rs

@@ -577,6 +577,7 @@ fn create_payload(
     partitions: &BTreeMap<String, Partition>,
     inputs: &BTreeMap<String, PSeekFile>,
     ota_info: &OtaInfo,
+    profile: &Profile,
     key_ota: &RsaSigningKey,
     cancel_signal: &AtomicBool,
 ) -> Result<(String, u64)> {
@@ -594,14 +595,14 @@ fn create_payload(
             .map(PSeekFile::new)
             .with_context(|| format!("Failed to create temp file for: {name}"))?;
 
-        let (partition_info, operations, cow_estimate) = payload::compress_image(
-            file,
-            &writer,
-            name,
-            4096,
-            dynamic_partitions_names.contains(name),
-            cancel_signal,
-        )?;
+        let vabc_algo = if dynamic_partitions_names.contains(name) {
+            profile.vabc_algo
+        } else {
+            None
+        };
+
+        let (partition_info, operations, cow_estimate) =
+            payload::compress_image(file, &writer, name, 4096, vabc_algo, cancel_signal)?;
 
         compressed.insert(name, writer);
 
@@ -645,7 +646,7 @@ fn create_payload(
                 }],
                 snapshot_enabled: Some(true),
                 vabc_enabled: Some(true),
-                vabc_compression_param: Some("lz4".to_owned()),
+                vabc_compression_param: profile.vabc_algo.map(|a| a.to_string()),
                 cow_version: Some(2),
                 vabc_feature_set: None,
             }),
@@ -688,7 +689,7 @@ fn create_payload(
             .with_context(|| format!("Failed to copy from image: {name}"))?;
     }
 
-    let (_, properties, metadata_size) = payload_writer
+    let (_, _, properties, metadata_size) = payload_writer
         .finish()
         .context("Failed to finalize payload")?;
 
@@ -746,6 +747,7 @@ fn create_ota(
                     &profile.partitions,
                     &inputs,
                     ota_info,
+                    profile,
                     key_ota,
                     cancel_signal,
                 )
@@ -830,12 +832,16 @@ fn create_fake_magisk(output: &Path) -> Result<()> {
 
     for path in [
         "assets/stub.apk",
+        "lib/arm64-v8a/libinit-ld.so",
         "lib/arm64-v8a/libmagisk64.so",
         "lib/arm64-v8a/libmagiskinit.so",
+        "lib/armeabi-v7a/libinit-ld.so",
         "lib/armeabi-v7a/libmagisk32.so",
         "lib/armeabi-v7a/libmagiskinit.so",
+        "lib/x86/libinit-ld.so",
         "lib/x86/libmagisk32.so",
         "lib/x86/libmagiskinit.so",
+        "lib/x86_64/libinit-ld.so",
         "lib/x86_64/libmagisk64.so",
         "lib/x86_64/libmagiskinit.so",
     ] {