Deploy Windows Recovery Partition Manually
Check Windows RE status
reagentc /infoDisable current Windows RE
reagentc /disableCheck or create a recovery partition by diskpart
# X/y is the target disk/partition
DISKPART> select disk x
DISKPART> list partition
DISKPART> select partition y
#The recovery partition looks like:
DISKPART> detail partition
Type : de94bba4-06d1-4d40-a16a-bfd50179d6ac
Hidden : Yes
Required: Yes
Attrib : 0X8000000000000001
# If no recovery partition, create one and set below properties. Only "set id=27" is needed for BIOS
DISKPART> set id=de94bba4-06d1-4d40-a16a-bfd50179d6ac
DISKPART> gpt attributes=0x8000000000000001
# Keep the recovery partition/volume active from above steps.
DISKPART> format quick fs=ntfs label=”Windows RE”
DISKPART> assign letter=RPrepare the Windows RE image “Winre.wim” from below location
"C:\Windows\System32\Recovery\Winre.wim"
#If the file doesn't exist, usually because WinRE has been moved to the RE tools partition prior to capturing the image. Get it from the mounted windows installation install.wim
> dism /mount-image /ImageFile:"D:\sources\install.wim" /index:4 /mountdir:"C:\WinInstall_mount" /ReadOnly
"C:\WinInstall_mount\Windows\System32\Recovery\Winre.wim"
# Now copy winre.wim to the recovery partition:
mkdir R:\Recovery\WindowsRE
xcopy /h C:\Windows\System32\Recovery\Winre.wim R:\Recovery\WindowsRE
# Unmount install.wim after done copying
dism /unmount-Image /MountDir:"C:\WinInstall_mount" /discardRegister the winre.wim image
reagentc /setreimage /path R:\Recovery\WindowsRE /target C:\Windows
reagentc /enable
reagentc /infoRemove the volume letter
DISKPART> select disk x
DISKPART> select partition y
DISKPART> remove letter=RConnect Alpine email client to Office365 via OAuth2
Alpine email client already supports XOAUTH2 authentication. Read the guide here:
https://alpine.x10host.com/alpine/alpine-info/misc/xoauth2.html
However it requires your administrator have Alpine client_id registered. Otherwise it will block the authentication from Alpine.
My organization allows ThunderBird. So this guide will show a “workaround” way to fake Alpine as a whitelist client to pass the OAuth2 authentication process.
I’m using Arch linux and install Alpine from AUR. Recent version is 2.24 but suppose all version with XOAUTH2 support should work.
No need to modify User-Agent for HTTP post.
The first step is to modify Alpine User-Agent to ThunderBird: Current UA is “Alpine”: https://repo.or.cz/alpine.git/blob/HEAD:/imap/src/c-client/http.c#l1011 The ThunderBird UA can be checked by “Help”->”Troubleshooting Information” Modify it to UA of ThunderBird. It may require to set ThunderBird UA by enable “general.useragent.compatMode.firefox” for some IMAP server, which make ThunderBird UA in form “Gecko/xxx Firefox/yyy Thunderbird/zzz” instead of “Gecko/xxx Thunderbird/zzz”. Make Alpine with UA changed.
The second step is to configure Alpine use ThunderBird client_id and secret:
Follow this guide for the overall reference: https://alpine.x10host.com/alpine/alpine-info/misc/xoauth2.html
In Alpine main menu, press S U to configure xoauth2:
Change client_id and secret of Outlook provide to ThunderBirds:
And change the “Auth Flow” to Authorize.
Now setup IMAP as usual. Login and Alpine will prompt you an URL for OAuth2 authentication. Open that URL in browser, either you will be prompt a code after proper authentication, either the authentication server will feedback an URL in form “http://localhost/?code=xxxx&session_state=xxxx#”. In the former case, input the code and you will get the authentication token to continue. In the latter case, Alpine will parse the token from the return URL directly.
A new Multi-Touch Test utility
Recently during my work of adding multi-touch feature to virtio-input tablet Windows driver, I was trying to find a simply and elegant test utility which can track position, width and other information of all contacts on screen. The closet I can found is one feature shipped with DesktopOK [1]. However, DesktopOK can’t show the actual contact area, which is one of the feature I’m adding to virto-input if backend input device support it. DesktopOK is freeware but not open-sourced which makes it impossible to add new features if someone is interested.
Thus I decided to make my own, Multi-Touch Tester [2].
Multi-Touch Tester is a native Windows desktop application which use imgui [3] as the rendering backend. The utility can list all present digitizer HID devices on system. When user move contacts with the window, the utility can track and show real-time x/y position of each contact, together with the width/height if supported by the device. As a visualized effect, a ring or a circle will be drawn within window client area at it’s x/y position. Whether to draw ring or circle depends on if contact area is supported. A filled circle is drawn if the contact has area information, and the radius depends on the actual contact area. Otherwise a ring with fixed radius is drawn to indicate the contact has no area support. The color is randomly picked. When moving contacts, the currently used touch device will be highlighted to indicate which touch device are those contacts come from.
Screen shot:
Reference:
[1] http://www.softwareok.com/?seite=Freeware/DesktopOK
Build and Run Fuchsia in QEMU
Abstract:
This document will introduce a BKM to build and run Fuchsia, based on Google’s official guide. The build process has been verified on Linux only. The built image can run on both upstream QEMU on Linux and Windows.
https://fuchsia.googlesource.com/fuchsia/+/master/docs/getting_started.md
1. Prepare build tools:
- Install below build tools or equivalent in your Linux environment:
- build-essential curl git python unzip ccache
2. Use python 2 or virtual env on python3.
3. Prepare Fuchsia source. (The source code takes about 18G disk space.)
- Create a working directory for fuchsia.
- Bootstrap the development environment in the working directory:
- curl -s “https://fuchsia.googlesource.com/fuchsia/+/master/scripts/bootstrap?format=TEXT” | base64 –decode | bash
- Add “.jiri_root/bin” to PATH.
- Run “cipd auth-login” to setup an OAuth token
- [Optional] Run “jiri init -analytics-opt=false “…”” to stop jiri from collecting some anonymized data
- Run above curl command again until all repo fetched.
- Update repo:
- fx clean
- jiri update
4. Build:
- The working directory size is increased to 80-90G after building.
- fx set terminal.x64 –release –ccache –with //bundles:kitchen_sink
- Refer to https://fuchsia.googlesource.com/fuchsia/+/master/products/ for all products.
- Refer to https://fuchsia.googlesource.com/fuchsia/+/master/bundles/BUILD.gn for all bundles.
- fx clean
- fx build
5. Parse QEMU launch parameters:
- Run on Linux and save the “exec” line or get the QEMU launch parameters from “ps -aux” for later use. For example:
- /home/coxu_arch/dev2/fuchsia/buildtools/linux-x64/qemu/bin/qemu-system-x86_64 -kernel /home/coxu_arch/dev2/fuchsia/out/default/../default.zircon/multiboot.bin -initrd /tmp/tmp.6G3EcZtxZZ/fuchsia-ssh.zbi -m 2048 -serial stdio -vga std -drive file=/tmp/tmp.6G3EcZtxZZ/fvm.blk,format=raw,if=none,id=mydisk -device ich9-ahci,id=ahci -device ide-drive,drive=mydisk,bus=ahci.0 -net none -smp 4,threads=2 -machine q35 -device isa-debug-exit,iobase=0xf4,iosize=0x04 -cpu Haswell,+smap,-check,-fsgsbase -append ‘TERM=xterm-256color kernel.serial=legacy kernel.entropy-mixin=7842c24733b30f2f17642b6245d6c034d3e190fe3c2edcf9ff3c20932304c07f kernel.halt-on-panic=true ‘
- To understand how the QEMU launch parameters are created, refer to below files for details:
- ./tools/devshell/run
- ./zircon/scripts/run-zircon
5. Run on Linux using Fuchsia shipped QEMU
- fx run -k
6. Run on Linux using upstream QEMU.
- Copy below images to somewhere else:
- ./out/default.zircon/multiboot.bin
- ./out/default/obj/build/images/fvm.blk
- ./out/default/fuchsia.zbi
- Pre-process above images. Simply double the original size of fvm.blk for the new size.
- ./out/default.zircon/tools/fvm fvm-test.blk –length <new_size_in_bytes>
- ./out/default.zircon/tools/zbi -o fuchsia-test.zbi ./out/default/fuchsia.zbi
- Modify the launch parameters as from step 4, the QEMU executable path, the kernel ramdisk and drive file path, etc.
7. Run on Windows using upstream QEMU.
- The Fuchsia/Zircon still needs to be built on Linux. Make sure you can run on Linux first.
- Build upstream QEMU as described here:
- Copy multiboot.bin, fvm-test.blk and fuchsia-test.zbi to your Windows machine. The 3 images are generated as described in step 6.
- Launch a msys2 shell, as has been used to build QEMU. Modify below commands to match the path on your machine:
- /c/_dev/qemu-build/qemu-system-x86_64.exe -kernel /c/_dev/fuchsia/image/multiboot.bin -initrd /c/_dev/fuchsia/image/fuchsia-test.zbi -m 2048 -serial stdio -vga std -drive file=/c/_dev/fuchsia/image/fvm-test.blk,format=raw,if=none,id=mydisk -device ich9-ahci,id=ahci -device ide-drive,drive=mydisk,bus=ahci.0 -net none -smp 2,threads=2 -machine q35 -device isa-debug-exit,iobase=0xf4,iosize=0x04 -accel hax -append ‘TERM=xterm-256color kernel.serial=legacy kernel.entropy-mixin=6ab4e081e365876ea06d1f276075122a3ffd951b7ab0d9eb06989c6b8587c45a kernel.halt-on-panic=true’
Build Chromium OS for QEMU
Get the Source:
mkdir chromiumos && cd chromiumos
repo init -u https://chromium.googlesource.com/chromiumos/manifest.git
repo sync -j $(nproc)
Basic build:
Enter env:
(outside) cros_sdk
Setup Board:
(inside) export BOARD=amd64-generic
# Setup board with default profile
(inside) setup_board –board=${BOARD}
# Setup board with chosen profile, useful to switch kernel and packages
(inside) setup_board –board=${BOARD} –profile kernel-5_4
Set password:
(inside) set_shared_user_password.sh
Build packages
(inside) build_packages –board=${BOARD}
Build image:
(inside) build_image –board=${BOARD} –noenable_rootfs_verification dev
Convert image to to QEMU raw:
(inside) image_to_vm.sh –from=/mnt/host/source/src/build/images/${BOARD}/latest/
Customize build kernel:
When setup board, use “–profile” to switch kernel and dependent packages. If only want to customize the kernel, see steps below
Change default kernel version
Available kernels are listed here:
~/trunk/src/third_party/chromiumos-overlay/sys-kernel
Modify the kernel version for selected board.
~/trunk/src/overlays/overlay-amd64-generic/profiles/base/make.defaults
i.e.
-USE=”${USE} legacy_keyboard legacy_power_button sse kernel-4_14″
+USE=”${USE} legacy_keyboard legacy_power_button sse kernel-5_4″
Change default kernel config:
~/trunk/src/third_party/kernel/v5.4/chromeos/config/x86_64/*
Modify kernel source:
~/trunk/src/third_party/kernel
Unmerge the old kernel first since only 1 kernel can be merged:
emerge-amd64-generic –unmerge sys-kernel/chromeos-kernel-4_14
Build kernel package:
emerge-amd64-generic sys-kernel/chromeos-kernel-5_4
After done building, check the using config and built kernel and modules:
/build/amd64-generic/var/cache/portage/sys-kernel/chromeos-kernel-5_4/.config
~/trunk/chroot/build/amd64-generic/boot/vmlinuz
~/trunk/src/build/images/amd64-generic/latest/boot_images/
Now, build_image again so that the selected kernel is used.
Or you could replace vmlinuz or other modules if only want to update the kernel image:
sudo mount -o rw,remount /
cp the built vmlinuz here and update grub entry.
Or replace the kernel modules.
Development/debug:
[TODO]
Do below on “first” boot
1.
Change password of default user chronos:
chromeos-setdevpasswd
2.
sudo /usr/share/vboot/bin/make_dev_ssd.sh –force –remove_rootfs_verification
reboot
3. enable debugging features
SSH password:
root:test0000
chronos:n/a
Reference:
https://www.chromium.org/chromium-os/quick-start-guide
https://www.chromium.org/chromium-os/how-tos-and-troubleshooting/using-an-upstream-kernel-on-snow
https://www.chromium.org/chromium-os/how-tos-and-troubleshooting/chromiumos-board-porting-guide
https://www.chromium.org/chromium-os/how-tos-and-troubleshooting/debugging-features
Resolve Flash/Upload Issue of MH-ET LIVE Tiny88
MH-ET LIVE Tiny88 is the cheapest dev board I found for BadUSB, similar to Digispark USB (ATtiny85 > ATtiny88), but much cheaper (less than $2 in Chinese market).
After setup Arduino IDE according to the official guide, it looks like the binary can’t be uploaded to the board. Error message:
Warning: device with unknown new version of Micronucleus detected. This tool doesn't know how to upload to this new device. Updates may be available. Device reports version as: 2.2
Others also met same issue as posted on the forum but no official reply.
Someone posted solutions here:
https://forum.arduino.cc/index.php?topic=629327.0
http://kovo-blog.blogspot.com/2019/01/how-to-upgrade-bootloader-on-digistump.html
The reason is simply: the flash tool shipped with MH-ET json is tool old which can’t support the new device.
The solution is simple, just update the flash tool.
git clone https://github.com/micronucleus/micronucleus cd commandline make cp ./micronucleus ~/.arduino15/packages/mhetlive/tools/micronucleus/2.0a4/micronucleus/
You may want to backup the old micronucleus first.
Now you can flash tiny88 using Arduino IDE or command line:
./micronucleus --run Blink.ino.hex
Bootstrapping macOS installer UFD on Linux
I found this guide describing a very detailed guide about how to bootstrapping on Windows. However I haven’t yet found a Linux version so cook my own as below.
Pre-Prepare Base System:
- Check Apple SW Catalog from here, or here depending on the version. Locate all product ID for package contains BaseSystem.dmg or InstallESDDmg.pkg.
- Read this blog to check the product ID for the macOS version that you’ll install. You can also parse it manually:
For each product ID key, there is ServerMetadataURL subkey points to a *.smd file URL. Open it and you will find the description and version. Choose the desired product ID. - Download below files (recommended by this guide):
BaseSystem.dmg
BaseSystem.chunklist
InstallInfo.plist
InstallESDDmg.pkg
AppleDiagnostics.dmg
AppleDiagnostics.chunklist - Clone BaseSystem to UFD.
(1) Convert BaseSystem.dmg to BaseSystem.img
dmg2img -v -i BaseSystem.dmg
(2) Check the partition info of BaseSystem.img for total size of the HFS+ partition. “Sectors”=”End”-“Start”+1, the default bytes of each sector is 512.
fdisk -l BaseSystem.img
Prepare UFD and HDD:
Use gdisk (GPT) or gdisk/fdisk(MBR) to create the expected partition table and partition. It’s recommend to use GPT unless your system is quite old and can only support BIOS boot from MBR.
e.g. Use gdisk to create a GPT disk:
Open the UFD/HDD:
sudo gdisk /dev/sdX
Create new GPT partition table:
o
Create two partitions for UFD. No need for HDD since the installer will do it.
- Partition 1: type code EF00, size 100–550 MiB
- Partition 2: type code AF00, set the size to exact same as output of previous “fdisk -l BaseSystem.img”.
n
Write change to disk:
w
Quite gdisk.
q
Format EFI partition:
sudo mkfs.vfat -F32 /dev/sdX1
Post-Prepare Base System:
-
Clonse BaseSystem to UFD. (continue)
(3) Clone HFS+ partition to UFD partition 2 (previously created with same size)sudo losetup --find --show --partscan --read-only BaseSystem.img dd if=/dev/loop0p1 of=/dev/sdX2 bs=1M umount /dev/loop0p1 losetup -d /dev/loop0 - Expand the HFS+ partition to cover all unallocated UFD space. (gparted doesn’t work so I have to use Paragon Hard Disk Manager for Windows. I’ll try to find an alternative for Linux)
Prepare macOS installer:
Now partition 2 on UFD should have base system installed. As suggested by this guide, create a folder named “SharedSupport” on this partition.
- Copy the 6 downloaded files into it.
- Rename InstallESDDmg.pkg to InstallESD.dmg.
- Modify InstallInfo.plist as belows:
- In “Payload Image Info”, delete the key chunklistURL and chunklistid.
- In “Payload Image Info”, rename InstallESDDmg.pkg to InstallESD.dmg for key “URL”, and rename com.apple.pkg.InstallESDDmg to com.apple.pkg.InstallESD for “id”.
Install Clover to UFD:
Download latest Clover EFI bootloader. By the time this post updated, the latest version is 2.5k rev 5070.
Extract all files from the ISO version and copy to the EFI partition of UFD /dev/sdX1.
Configure Clover in UFD:
There many articles discussing Clover configuration, add necessary EFI driver to Clover and prepared necessary kexts to boot the installer. You need sometime to adjust the drivers and kexts. An alternative to Clover is OpenCore.
-
Some EFI Drivers:
- AptioMemoryFix You may have to try different versions or try OsxAptioFixnDrv.efi.
- HFSPlus
- EmuVariableUefi or get it from old Clover.
-
Some kexts: Refer to this guide this guide.
-
config.plist: Choose your config.plist, such as here.
The new Clover has changed the way adding Injected GPU ID. To add a FakeID, modify the config.plist as:
-
Add below to
<key>Devices</key>>>><key>FakeID</key>section:<key>IntelGFX</key> <string>0x12345678</string> -
Add below to
<key>DSDT</key>>>><key>Fixes</key>section:<key>FixDisplay</key> <true/> <key>FixIntelGfx</key> <true/> -
Add below to
<key>Graphics</key>>>><key>Inject</key>section:<key>Intel</key> <true/>
Setup Proxy for Installer:
Apple doesn’t trust any proxy for installation, and the networksetup has been removed from /usr/sbin so you can’t use it to setup proxy for Mojave but have to set up manually. Open a terminal in macOS Utilities then do the following:
Show current proxy, you will find not proxy is set.
>scutil –proxy
<dictionary> {
ExceptionsList : `<array>`{
0 : *.local
1 : 169.254/16
}
FTPPassive : 1
}
Now setup the proxy:
scutil
>open>list>get State:/Network/Global/Proxies>d.show>d.add HTTPEnable # 1>d.add HTTPProxy [proxy_url]>d.add HTTPPort # [proxy_port]>d.add HTTPSEnable # 1>d.add HTTPSProxy [proxy_url]>d.add HTTPSPort # [proxy_port]>set State:/Network/Global/Proxies>quit
Check proxy again using “scutil –proxy”, you will find the desired proxy is set. Be noticed that proxy port number is integer type so you need add a pound before port number.
Alternative way to set proxy:
- Open a terminal in macOS Utilities and get the current settings from “/Library/Preferences/SystemConfiguration/preferences.plist” and scp to another host.
- Modify preferences.plist by adding proxy settings.
- Copy the file to UFD path “/Library/Preferences/SystemConfiguration/”. By default, preferences.plist doesn’t exist in the base image on UFD and is created on-the-air by macOS Utilities. Put an existing file in the location will make the installer use it instead of creating a new one, thus the expected proxy settings are loaded.
Boot from UFD and install macOS now.
Native-Build Android Emulator for Windows
Native-Build Android Emulator for Windows
Abstract:
In Android emulator development guide, Google has suggested detailed steps of cross-compiling the Windows binaries from linux. However, if developer would like to debug the emulator using PDB support debugger like Visual Studio Debugger and WinDbg, since mingw-gcc doesn’t support generating PDB format symbol file yet, WinDbg can’t debug the emulator in source-code level. In this article, a native compiling method using Clang is introduced, which can generate PDB symbol file so that source-code level debugging is possible.
Official Guides:
https://developer.android.com/studio
Updated on 20191101:
https://android.googlesource.com/platform/external/qemu/+/refs/heads/emu-master-dev/README.md
Follow Google guide to setup repo tools, fetch code and compile. It takes about 50-60G to download all codes and compiling.
Cross-compile for Windows on Linux Host:
Follow the official guide to compile.
Pros:
No need setup Windows build environment.
No need to manually copy dependent DLLs.
Cons:
mingw cross-compile can’t generate PDB symbol for live debugging using MS tools.
cd <emu-master-dev>/external/qemu/android
Help:
/rebuild.sh –helpfull
Check config:
./rebuild.sh –verbosity=0 –build=config –config=debug –target=mingw –notests
Clean Build:
./rebuild.sh –verbosity=0 –config=debug –target=mingw –notests
OR:
cmake –build <full_path_to_qemu_objs> –target install
Incremental Build:
./rebuild.sh –verbosity=0 –noclean –config=debug –target=mingw –notests
OR:
cmake –build <full_path_to_qemu_objs> –target install
The built binaries can be found in: <emu-master-dev>/external/qemu/objs/distribution
Native-compile on Windows Host:
Pros:
LLVM/Clang 5+ can generate PDB symbol for debugging purpose using MS tools.
Cons:
Need install some additional tools for building.
Need manually modify some files to compile successfully.
May need to manually move some dlls for a successful out-of-tree run.
Required tool-chains:
- Visual Studio tool chain, either one of below:
- Visual Studio 2015/2017/2019 (With IDE)
- Visual Studio Build Tools 2015/2017/2019 (No IDE)
- Latest EWDK
- Git for Windows
https://git-scm.com/download/win
- repo tool
https://gerrit.googlesource.com/git-repo/
- Gpg4win
- Python 2.7 x86_64
- LLVM/Clang x86_64
- CMake x86_64
- Ninja
- JDK 8
Environment Preparation:
- Enable “Long Path” mode of Windows:
https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file
- Install all above toolchains manually, or you can use Chocolatey, the package manager for Windows. You can find most frequent used development tools from choco online package repository.
- If the above tools are not added to %PATH%, add the location to %PATH% so that you can run them anywhere from cmd prompt.
- If *.py is not associated to python.exe, you can execute *.py using “python *.py”.
- Make sure gpg.exe from gpg4win is used instead of from git-for-windows, the former will address a local path in Windows style, while the latter is in msys style. You could see gpg related error when running repo tool. Simply add path of gpg4win ahead of git-for-windows.
Updated on 20191101:
Now VS2019 is officially supported.
The build process now requires JDK8.
Fetch the code:
Once all toolchain is ready, you can simply clone the code as on Linux, please check I.3 in the official guide.
Patch local source:
Obsolete version:
Some local files need to be patched to compile the emulator with Clang. Check the appendix patch list V1 for the details.
Updated on 20191101:
Check V2 for the full list of files to patch.
Compile:
Obsolete version:
Similar commands except append some cmake parameters:
Check config:
rebuild.cmd –verbosity=0 –build=config –config=debug –target=windows –notests –cmake_option DIASDK_INCLUDE_DIR=”<path to visual studio DIA SDK Include>”
Clean Build:
rebuild.cmd –verbosity=0 –config=debug –target=windows –notests –cmake_option DIASDK_INCLUDE_DIR=”<path to visual studio DIA SDK Include>”
Incremental Build:
rebuild.cmd –verbosity=0 –noclean –config=debug –target=windows –notests –cmake_option DIASDK_INCLUDE_DIR=”<path to visual studio DIA SDK Include>”
Updated on 20191101:
cd <emu-master-dev>/external/qemu/
Check config:
android\rebuild.cmd –verbosity=0 –build=config –config=debug –target=windows –notests
Clean Build:
android\rebuild.cmd –verbosity=0 –config=debug –target=windows –notests
OR
cmake –build <full_path_to_emu_objs> –target install
Incremental Build:
android\rebuild.cmd –verbosity=0 –noclean –config=debug –target=windows –notests
OR:
cmake –build <full_path_to_emu_objs> –target install
The built binaries can be found in: <emu-master-dev>/external/qemu/objs/distribution
The built symbol files can be found in:
<emu-master-dev>/external/qemu/objs/build/debug_info
Run:
You can replace Android studio shipped qemu with compiled one:
– Replace <android-sdk-root>/…/emulator/qemu/windows-x86_64/qemu-system-x86_64.exe with <emu-master-dev>/…/objs/build/distribution/emulator/qemu/windows-x86_64/qemu-system-x86_64.exe
– Copy all DLLs from <emu-master-dev>/…/objs/build/distribution/emulator/lib64/ and it’s subdirectories to <android-sdk-root>/…/emulator/qemu/windows-x86_64/. The step will make sure qemu load the copied dll first.
Or you can start the compiled emulator directly with below batch: (Recommended way)
setlocal
SET ANDROID_SDK_ROOT=<path_to_android_sdk>
SET ANDROID_EMULATOR_LAUNCHER_DIR=<emu-master-dev>\external\qemu\objs\distribution\emulator
<emu-master-dev>\external\qemu\objs\distribution\emulator\emulator.exe -avd <avd-name>
Debug:
- Local debug on target machine directly using WinDbg or VisualStudio.
- Remote debug from host machine using WinDbg or VisualStudio.
Appendix:
Troubleshooting:
- repo sync error with error msg like:
From https://android.googlesource.com/platform/external/lz4
! [rejected] android-q-preview-4 -> android-q-preview-4 (would clobber existing tag)
! [rejected] android-q-preview-5 -> android-q-preview-5 (would clobber existing tag)
…
…
error: Exited sync due to fetch errors
It’s due to someone has forced tags to move in a public project.
Solution:
Just delete those offensive tags in the specific project and sync again.
> repo forall external/lz4 -c git tag -d android-q-preview-4 android-q-preview-5
2. When run rebuild.cmd, missing python modules like:
ImportError: No module named absl
Just install the missing py module:
1) Upgrade existing python pip
>python -m pip install –upgrade pip
2) Install the
> python -m pip install absl-py
3) If prompt for “there was a problem confirming the ssl certificate”, use below command:
> python -m pip install –trusted-host pypi.org –trusted-host files.pythonhosted.org absl-py
Modifications required to build on Windows:
V2:
*****************
diff –git a/android/build/cmake/toolchain-windows_msvc-x86_64.cmake b/android/build/cmake/toolchain-windows_msvc-x86_64.cmake
index 1701f7fc7c..252977d2a6 100644
— a/android/build/cmake/toolchain-windows_msvc-x86_64.cmake
+++ b/android/build/cmake/toolchain-windows_msvc-x86_64.cmake
@@ -30,11 +30,15 @@ get_filename_component(ANDROID_QEMU2_TOP_DIR “${CMAKE_CURRENT_LIST_FILE}/../../.
# Cmake goes crazy if we set AR manually.. so let’s not do that.
if(WIN32)
# set(CMAKE_SYSTEM_NAME WinMSVCCrossCompile)
– get_clang_version(CLANG_VER)
– message(STATUS “Configuring native windows build using clang-cl: ${CLANG_VER}”)
– get_filename_component(CLANG_DIR “${ANDROID_QEMU2_TOP_DIR}/../../prebuilts/clang/host/windows-x86/${CLANG_VER}”
– REALPATH)
+ # get_clang_version(CLANG_VER)
+ # message(STATUS “Configuring native windows build using clang-cl: ${CLANG_VER}”)
+ # get_filename_component(CLANG_DIR “${ANDROID_QEMU2_TOP_DIR}/../../prebuilts/clang/host/windows-x86/${CLANG_VER}”
+ # REALPATH)
+ # set(CLANG_CL ${CLANG_DIR}/bin/clang-cl.exe)
+ set(CLANG_DIR “C:/LLVM”)
set(CLANG_CL ${CLANG_DIR}/bin/clang-cl.exe)
+ execute_process(COMMAND ${CLANG_CL} –version OUTPUT_VARIABLE CLANG_VER)
+ message(STATUS “Configuring native windows build using clang-cl: ${CLANG_VER}”)
if(NOT EXISTS “${CLANG_CL}”)
message(
FATAL_ERROR
*****************
V1:
*****************
diff –git a/android/android-emu/android/base/memory/LazyInstance.h b/android/android-emu/android/base/memory/LazyInstance.h
index a54ec66524..5d79f5fee4 100644
— a/android/android-emu/android/base/memory/LazyInstance.h
+++ b/android/android-emu/android/base/memory/LazyInstance.h
@@ -114,8 +114,8 @@ struct LazyInstanceState {
static_assert(std::is_standard_layout<LazyInstanceState>::value,
“LazyInstanceState is not a standard layout type”);
-static_assert(is_trivially_default_constructible<LazyInstanceState>::value,
– “LazyInstanceState can’t be trivially default constructed”);
+//static_assert(is_trivially_default_constructible<LazyInstanceState>::value,
+// “LazyInstanceState can’t be trivially default constructed”);
} // namespace internal
@@ -178,9 +178,9 @@ T* LazyInstance<T>::ptrInternal() const {
// NB: this can’t go in a class scope as class is still incomplete there.
static_assert(std::is_standard_layout<LazyInstance>::value,
“LazyInstance<T> is not a standard layout type”);
– static_assert(
– internal::is_trivially_default_constructible<LazyInstance>::value,
– “LazyInstance<T> can’t be trivially default constructed”);
+ //static_assert(
+ // internal::is_trivially_default_constructible<LazyInstance>::value,
+ // “LazyInstance<T> can’t be trivially default constructed”);
if (mState.needConstruction()) {
new (&mStorage) T();
diff –git a/android/android-emu/android/recording/video/player/VideoPlayer.h b/android/android-emu/android/recording/video/player/VideoPlayer.h
index 12a96a92f6..7da27515a7 100644
— a/android/android-emu/android/recording/video/player/VideoPlayer.h
+++ b/android/android-emu/android/recording/video/player/VideoPlayer.h
@@ -36,6 +36,7 @@
#include “android/utils/compiler.h”
#include <memory>
+#include <string>
namespace android {
namespace videoplayer {
diff –git a/android/android-emu/android/snapshot/Hierarchy.h b/android/android-emu/android/snapshot/Hierarchy.h
index 4ffe05ea16..9bcfda2061 100644
— a/android/android-emu/android/snapshot/Hierarchy.h
+++ b/android/android-emu/android/snapshot/Hierarchy.h
@@ -16,6 +16,7 @@
#include <functional>
#include <map>
#include <vector>
+#include <string>
namespace android {
namespace snapshot {
diff –git a/android/build/cmake/toolchain-windows_msvc-x86_64.cmake b/android/build/cmake/toolchain-windows_msvc-x86_64.cmake
index 62fb5fe9ac..5fc251efee 100644
— a/android/build/cmake/toolchain-windows_msvc-x86_64.cmake
+++ b/android/build/cmake/toolchain-windows_msvc-x86_64.cmake
@@ -30,11 +30,15 @@ get_filename_component(ANDROID_QEMU2_TOP_DIR “${CMAKE_CURRENT_LIST_FILE}/../../.
# Cmake goes crazy if we set AR manually.. so let’s not do that.
if(WIN32)
# set(CMAKE_SYSTEM_NAME WinMSVCCrossCompile)
– get_clang_version(CLANG_VER)
– message(STATUS “Configuring native windows build using clang-cl: ${CLANG_VER}”)
– get_filename_component(CLANG_DIR “${ANDROID_QEMU2_TOP_DIR}/../../prebuilts/clang/host/windows-x86/${CLANG_VER}”
– REALPATH)
+ # get_clang_version(CLANG_VER)
+ # message(STATUS “Configuring native windows build using clang-cl: ${CLANG_VER}”)
+ # get_filename_component(CLANG_DIR “${ANDROID_QEMU2_TOP_DIR}/../../prebuilts/clang/host/windows-x86/${CLANG_VER}”
+ #REALPATH)
+ # set(CLANG_CL ${CLANG_DIR}/bin/clang-cl.exe)
+ set(CLANG_DIR “C:/Program Files/LLVM”)
set(CLANG_CL ${CLANG_DIR}/bin/clang-cl.exe)
+ execute_process(COMMAND ${CLANG_CL} –version OUTPUT_VARIABLE CLANG_VER)
+ message(STATUS “Configuring native windows build using clang-cl: ${CLANG_VER}”)
if(NOT EXISTS “${CLANG_CL}”)
message(
FATAL_ERROR
diff –git a/android/build/python/aemu/definitions.py b/android/build/python/aemu/definitions.py
index 3d74c7e12f..0539ffc4a7 100644
— a/android/build/python/aemu/definitions.py
+++ b/android/build/python/aemu/definitions.py
@@ -67,7 +67,7 @@ def get_visual_studio():
TODO(jansene): Use https://github.com/Microsoft/vswhere vs this here”’
candidates = recursive_iglob(os.path.join(os.sep, “Program Files (x86)”, “Microsoft Visual Studio”,
– “2017”), [lambda f: f.endswith(‘vcvars64.bat’)])
+ “2019”), [lambda f: f.endswith(‘vcvars64.bat’)])
# Create a preference ordering, we prefer not to use build tools..
pref = sorted(candidates, reverse=True)
vs_release = next(iter(pref), None)
diff –git a/android/rebuild.cmd b/android/rebuild.cmd
index 9ba2cc4f87..0c523c8a34 100644
— a/android/rebuild.cmd
+++ b/android/rebuild.cmd
@@ -1,2 +1,2 @@
@echo off
-python android\build\python\cmake.py %*
\ No newline at end of file
+python build\python\cmake.py %*
\ No newline at end of file
—
Create a Universal Bootable USB Drive Using GNU GRUB2
[Last updated on 2023.12.29]
Background:
GNU GRUB2 (not GRUB Legacy) is very powerful to act as the universal bootloader on a USB drive to install OSs from. There’re already many user friendly tools to customize a multi-OS installer (Rufus, WinUSB). There’re also guides around the internet explain the manual steps. The question is: how to make it “universally” and “compatible” with both BIOS and UEFI systems for different OSs. This article introduce a complete manual way to customize a multi-OS installation USB drive based on GRUB, providing multiple Linux & Window installation on a single USB drive.
Analysis:
The universal USB drive should at least meet below criteria:
- Compatible with both BIOS and UEFI.
- Multiple partitions to hold different OS installation source.
- Able to chainloader other bootloader or OS installer.
1. BIOS/UEFI compatible bootloader.
GRUB2 can support both BIOS and UEFI systems (different boot sequence) so there’s no gap in bootloader choosing.
2. Partition Table:
As long as there’s an ESP (EFI system partition) [1][5], both MBR and GPT disk can be used. However GPT disk has more benefits. Multiple primary partition support is “better to have” instead of “must have”. Most Linux Distro ISO can be booted directly via GRUB2, so putting all these ISOs in same partition won’t be a problem. Boot Windows ISO directly via GRUB2 is not always successful in both BIOS and UEFI (details later), so putting all Windows ISOs in same partition can’t be an option. Extracting all contents from different Windows installation ISOs to different folders in same partition is theoretically possible, however extra manipulation on each boot.wim are needed to make sure correct install.wim can always be found (Windows installation ISOs have very much similar file tree hierarchy from DVD_ROM root). Thus, extracting Windows installation ISO content to different partition without modification would be a better solution. Windows Boot Manager can only boot from primary partition (won’t recognize logical partition) [13], so using MBR disk will only have 3 available primary partitions (1 already assigned for ESP) to keep Windows installation files which is less flexible.
Based on above reasons, choose GPT with protective MBR [2][6] for the universal USB drive.
3. Chainloader capability.
Due to below facts, we’ll not consider booting Windows ISO from GRUB2 directly:
- Legacy grub4dos [15] only recognizes MBR but not GPT (nor GPT Protective MBR). Hybrid MBR is recognizable, but can only have 3 GPT partitions added as MBR primary. In addition, Hybrid MBR itself is kind of “messy” [3] so not recommended only for using grub4dos. Chenall’s grub4dos fork [16] for BIOS can recognize all GPT partitions, while it’s UEFI version can only recognize ESP, however with both I fail to boot a Windows ISO, although it’s possible [10].
- With memdisk (Syslinux Project) [9], GRUB 2 can boot harddisk or ISO image directly (via initrd16) on BIOS, but not UEFI (stuck on boot). In addition, memdisk will read all ISO content to memory which could exceed the total installed memory size.
GRUB2 meets below requirements as the chosen bootloader:
- Work on both BIOS and UEFI system. With protective MBR, the boot code won’t interfere each other.
- Recognize both GPT and MBR disks, partitions and different file systems.
- Boot real mode applications via linux16 (i.e. memtest for BIOS).
- Boot UEFI applications via chainloader (i.e. memtest for UEFI)
- Boot ISO via memdisk, loopback device, initrd16/initrd (i.e. Linux Distro, FreeDOS LiveCD, CloneZilla)
- Boot raw disk image.
- Boot other bootloaders via ntldr, chainloader (i.e. grub4dos, NT Loader, Windows Boot Manager)
Boot Flow:
Based on above analysis, boot flow for different purposes are listed below:
BIOS system: Boot code on BIOS Boot Partition -> GRUB2 boot directory (set by grub-install)
UEFI system: Boot code on ESP -> GRUB2 boot directory (set by grub-install)
Linux Distro, live CD or harddisk:
GRUB2 -> loopback if ISO file -> load kernel via linux, boot from ramdisk via initrd
Legacy or UEFI Application:
GRUB2 -> linux16/linux or chainloader
Bootable ISO:
GRUB2 -> memdisk iso -> initrd16/initrd16 ISO file
Windows XP installation ISO or harddisk (NT Loader):
GRUB2 ->chainloader setupldr.bin/bootsect.bin (ISO) or harddisk
Windows installation ISO, WinPE or harddisk (Windows Boot Manager) [11][12]:
Notes: Sorry for my miss, Windows installation will fail with this method “Windows could not prepare the computer to boot into the next phase of installation.” This could due to Winsetup couldn’t figure disk order if executed from a GPT USB: https://superuser.com/questions/1381492/why-does-windows-10-fail-to-install-on-uefi-gpt-laptop. Temporary solution is to deploy Windows using dism: https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/capture-and-apply-windows-using-a-single-wim
BIOS system: GRUB2 -> memdisk harddisk -> initrd16 bootmgr VHD [7][8]
UEFI system: GRUB2 -> chainloader Windows Boot Manager for UEFI
grub4dos:
BIOS system: GRUB2 -> ntldr grldr
UEFI system: GRUB2 -> chainloader grub4dos UEFI
syslinux:
BIOS system: GRUB2 -> memdisk harddisk -> initrd16 syslinux.img
UEFI system: GRUB2 -> chainloader syslinux.efi
Detailed steps for each boot are explained in next section.
Steps:
This section explains all the details about how to make a universal bootable USB manually.
Disk Preparation:
# Start from gdisk
> sudo gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.9.1
# A blank disk with no partition table
Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present
Creating new GPT entries in memory.
# Destroy existing partition table and create new GPT and protective MBR (implicitly)
Command (? for help): o
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): y
# Create 1st partition as BIOS Boot partition to store MBR boot code
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-61341662, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-61341662, default = 61339647) or {+-}size{KMGTP}: +1M
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): EF02
Changed type of partition to 'BIOS boot partition'
# Create 2nd partition as ESP for UEFI boot code
Command (? for help): n
Partition number (2-128, default 2):
First sector (34-61341662, default = 4096) or {+-}size{KMGTP}:
Last sector (4096-61341662, default = 61339647) or {+-}size{KMGTP}: +64M
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): EF00
Changed type of partition to 'EFI system partition'
# Enter expert menu and create protective MBR. This is implicitly done on new GPT creation, so this step is just make the manual step clear
Command (? for help): x
Expert command (? for help): n
# Now let's hide BIOS boot partition from EFI and mark it bootable
Expert command (? for help): a
Partition number (1-2): 1
Known attributes are:
0: system partition
1: hide from EFI
2: legacy BIOS bootable
60: read-only
62: hidden
63: do not automount
Attribute value is 0000000000000000. Set fields are:
No fields set
Toggle which attribute field (0-63, 64 or <Enter> to exit): 1
Have enabled the 'hide from EFI' attribute.
Attribute value is 0000000000000002. Set fields are:
1 (hide from EFI)
Toggle which attribute field (0-63, 64 or <Enter> to exit): 2
Have enabled the 'legacy BIOS bootable' attribute.
Attribute value is 0000000000000006. Set fields are:
1 (hide from EFI)
2 (legacy BIOS bootable)
Toggle which attribute field (0-63, 64 or <Enter> to exit):
# Check the protective MBR in recovery menu. Hybrid MBR in same menu is not recommended.
Expert command (? for help): r
Recovery/transformation command (? for help): o
Disk size is 61341696 sectors (29.3 GiB)
MBR disk identifier: 0x00000000
MBR partitions:
# All range are covered by protective MBR.
Number Boot Start Sector End Sector Status Code
1 1 61341695 primary 0xEE
# Two partitions are ready now as the basic universal USB drive.
Recovery/transformation command (? for help): p
Disk /dev/sdb: 61341696 sectors, 29.3 GiB
Model: Cruzer Blade
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 6690CF32-AF0F-4667-995F-C290207BBF33
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 61341662
Partitions will be aligned on 2048-sector boundaries
Total free space is 61208509 sectors (29.2 GiB)
Number Start (sector) End (sector) Size Code Name
1 2048 4095 1024.0 KiB EF02 BIOS boot partition
2 4096 135167 64.0 MiB EF00 EFI system partition
# Write everything down to the disk.
Recovery/transformation command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
# Format ESP as FAT32
> sudo mkfs.vfat -F32 /dev/sdb2
mkfs.fat 4.2 (2021-01-31)
# Set a label (optional)
> sudo fatlabel /dev/sdb2 ESP
# Mount ESP to installer UEFI boot loader
> sudo mount /dev/sdb2 /mnt/data
# Install grub for BIOS (BIOS boot partition) and i386/amd64 EFI (different binary and mod folder).
> sudo grub-install --target=i386-pc --boot-directory=/mnt/data/boot --removable /dev/sdb
Installing for i386-pc platform.
Installation finished. No error reported.
> sudo grub-install --target=i386-efi --efi-directory=/mnt/data/ --boot-directory=/mnt/data/boot --removable /dev/sdb
Installing for i386-efi platform.
Installation finished. No error reported.
> sudo grub-install --target=x86_64-efi --efi-directory=/mnt/data/ --boot-directory=/mnt/data/boot --removable /dev/sdb
Installing for x86_64-efi platform.
Installation finished. No error reported.
# Unmount.
> sudo umount /dev/sdb2Now you have a GRUB-ready USB drive for both BIOS/UEFI boot. The following sections will introduce how to configure grub.cfg for different boots.
As for partition arrangement, 1 M should be enough for BIOS boot partition [4]; ESP size is based on what content you’d like save in ESP, grub-install only takes ~18 M. I’m using 64 M here as I’ll save Windows Boot Manager VHD/ISO, grub4dos, syslinux and other applications. Other partitions are not created here, but will be described along with grub.cfg explanation.
Basic GRUB Configuration:
Most part of grub.cfg is organized by entry, but still share some common configs.
insmod part_gpt # GPT partition table
insmod part_msdos # MBR partition table
insmod fat # FAT partition
insmod exfat # exFAT partition
insmod ext2 # ext partition
insmod ntfs # NTFS partition
insmod ntfscomp # NTFS compression
insmod allvideo # Additional video and graphics
insmod chain # chainloader
if [ ${grub_platform} == "pc" ]; then
insmod ntldr # chainloader from file without reading boot record
else
insmod efi_uga # UEFI UGA
insmod efi_gop # UEFI GOP
fiGRUB provide some variables to help identify different architecture: ${grub_platform}: pc or efi, ${grub_cpu}: i386 or x86_64, so that you can use same grub.cfg for more flexible configuration. As above, ntldr is only available for BIOS, GOP driver is only available for UEFI.
In order to boot both BIOS and UEFI machine, the USB drive must be initialized as MBR disk (BIOS doesn’t recognize GPT), and should be partitioned compatible for both BIOS and UEFI can boot from. According to UEFI spec [1], UEFI boot requires ESP (EFI System Partition) which “happens” to be compatible with FAT32, and assign GPT partition type of C12A7328-F81F-11D2-BA4B-00A0C93EC93B. The ESP partition type on MBR disk is 0xEF.
Thus, only FAT32/ESP stores EFI binary could be booted for both BIOS and UEFI.
To use GRUB as the universal USB drive bootloader, you have to install BIOS/UEFI grub on FAT32 partition, write the MBR boot sector for BIOS boot. If you have large files above 4G, consider separating the boot partition with data partitions (could be FAT32, ext4, exFAT or NTFS).
System Management:
menuentry "Reboot" {
reboot
}
menuentry "Shutdown" {
halt
}Legacy Application:
GRUB linux16 can load image from 16-bit mode, examples like memtest and FreeDOS ISO. initrd16 can be used if initial ramdisk is required.
menuentry "MemTest86 BIOS" {
linux16 (hd0,gpt2)/memtest86/MEMTEST
boot
}
menuentry "FreeDOS" {
linux16 (hd0,gpt2)/bios/memdisk iso
initrd16 (hd0,gpt3)/freedos/FD13LIVE.iso
}UEFI Application:
Simply chainloader it:
menuentry "MemTest86 UEFI" {
set root='(hd0,gpt2)'
if [ ${grub_cpu} == "i386" ]; then
chainloader /memtest86/EFI/BOOTIA32.EFI
else
chainloader /memtest86/EFI/BOOTX64.EFI
fi
boot
}grub4dos:
Someone may prefer load Windows via grub4dos, so you can chainloader via GRUB. grub4dos fork now has UEFI support [16].
menuentry "grub4dos" {
if [ ${grub_platform} == "pc" ]; then
ntldr (hd0,gpt2)/boot/grub4dos/grldr
else
chainloader (hd0,gpt2)/boot/grub4dos/BOOTX64.EFI
fi
}Syslinux:
Get the latest syslinux from: https://wiki.syslinux.org/wiki/index.php?title=Download, create for BIOS and UEFI as below, then add a GRUB2 entry:
menuentry "syslinux" {
if [ ${grub_platform} == "pc" ]; then
linux16 (hd0,gpt2)/bios/memdisk harddisk
initrd16 (hd0,gpt2)/bios/syslinux.img
else
chainloader (hd0,gpt2)/EFI/syslinux/syslinux.efi
fi
}BIOS:
Create a raw image which contains syslinux only as below, then copy to (ESP)/BIOS/syslinux.img, boot the raw image to syslinux.
# Prepare a empty image for syslinux
> dd if=/dev/zero of=syslinux.img bs=1M count=4
# Init as MBR and create one active primary partition
> fdisk syslinux.img
Welcome to fdisk (util-linux 2.39.3).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS (MBR) disklabel with disk identifier 0xceefcff7.
Command (m for help): o
Created a new DOS (MBR) disklabel with disk identifier 0x94f80733.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
# Start from sector 2048
First sector (1-8191, default 1): 2048
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-8191, default 8191):
Created a new partition 1 of type 'Linux' and of size 3 MiB.
Command (m for help): a
Selected partition 1
The bootable flag on partition 1 is enabled now.
Command (m for help): p
Disk syslinux.img: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x94f80733
Device Boot Start End Sectors Size Id Type
syslinux.img1 * 2048 8191 6144 3M 83 Linux
Command (m for help): w
The partition table has been altered.
Syncing disks.
# Format the primary partition as ext4
> mkfs.ext4 syslinux.img -E offset=$(( 512 * 2048 ))
mke2fs 1.47.0 (5-Feb-2023)
Warning: offset specified without an explicit file system size.
Creating a file system with 3072 blocks but this might
not be what you want.
Discarding device blocks: done
Creating filesystem with 3072 1k blocks and 768 inodes
Allocating group tables: done
Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done
# Mount the partition to install syslinux
> sudo mount -o offset=$(( 512 * 2048 )) syslinux.img /mnt/data
> sudo mkdir /mnt/data/syslinux
# Copy all .c32 for BIOS, create syslinux.cfg as configuration.
> find <extracted_syslinux> -type f -name \*.c32 | xargs -i sudo cp {} /mnt/data/syslinux/
# Copy memdisk as well
> sudo cp <extracted_syslinux>/bios/memdisk/memdisk /mnt/data/syslinux/
# Install syslinux
> sudo extlinux --install /mnt/data/syslinux
/mnt/data/syslinux is device /dev/loop0
Warning: unable to obtain device geometry (defaulting to 64 heads, 32 sectors)
(on hard disks, this is usually harmless.)
> sudo umount /mnt/data
# Set MBR boot sector
> dd if=<extracted_syslinux>/bios/mbr/mbr.bin of=./syslinux.img bs=440 count=1 conv=notrunc
1+0 records in
1+0 records out
440 bytes copied, 5.0004e-05 s, 8.8 MB/sUEFI:
Copy syslinux.efi and *.c32 files for UEFI 32/64 to (ESP)/EFI/syslinux/, create syslinux.cfg in same folder.
Linux LiveCD or Distro.:
Almost all Linux distros share the very similar steps: load the kernel and the initial ramdisk. They differ in parameters which are distro specific. Like Clonezilla, “findiso” and “toram=filesystem.squashfs” are necessary to boot LiveCD, other wise the rootfs can’t be mounted. Below’s the example for booting Clonezilla and Ubuntu from ISO.
menuentry "Clonezilla 3.1.1-27" {
set root='(hd0,gpt3)'
set isofile="/clonezilla/clonezilla-live-3.1.1-27-amd64.iso"
loopback loop $isofile
linux (loop)/live/vmlinuz boot=live union=overlay username=user config components quiet loglevel=0 noswap edd=on nomodeset enforcing=0 locales= keyboard-layouts= ocs_live_run="ocs-live-general" ocs_live_extra_param="" ocs_live_batch="no" vga=788 ip= net.ifnames=0 nosplash i915.blacklist=yes radeonhd.blacklist=yes nouveau.blacklist=yes vmwgfx.enable_fbdev=1 findiso=$isofile toram=filesystem.squashfs
initrd (loop)/live/initrd.img
}
submenu "Ubuntu 22.04.3" {
set root='(hd0,gpt3)'
set gfxpayload=keep
menuentry "Try Ubuntu 22.04.3 Desktop without installing" {
set isofile="/ubuntu-22.04.3-desktop-amd64.iso"
loopback loop $isofile
linux (loop)/casper/vmlinuz iso-scan/filename=$isofile boot=casper quiet splash ---
initrd (loop)/casper/initrd
}
menuentry "Install Ubuntu 22.04.3 Desktop" {
set isofile="/ubuntu-22.04.3-desktop-amd64.iso"
loopback loop $isofile
linux (loop)/casper/vmlinuz iso-scan/filename=$isofile boot=casper only-ubiquity quiet splash ---
initrd (loop)/casper/initrd
}
menuentry "Try or Install Ubuntu 22.04.3 Server" {
set isofile="/ubuntu-22.04.3-live-server-amd64.iso"
loopback loop $isofile
linux (loop)/casper/vmlinuz iso-scan/filename=$isofile boot=casper ---
initrd (loop)/casper/initrd
}
}Notes for Windows:
Maybe there exists perfect solution to boot Windows ISO directly (like via grub4dos), from my experience extracting ISO contents to separate partition root and let Windows Boot Manager handle it is more straightforward and easy to configure, as describe in section 2. All subsequent Windows boot methods are assuming the ISO contents are extracted to different partition (recommend NTFS instead of FAT32 to avoid 4G file size limit)
Windows (NTLDR: NT Loader):
Not sure how many will still install Windows XP nowadays. Simply extract the ISO content to some partition and chainloder it:
menuentry "Windows XP" {
chainloader (hd0,gpt4)/winxp/i386/setupldr.bin
# Or chainloader (hd0,gpt4)/winxp/bootsect.bin
}Windows (BOOTMGR: Windows Boot Manager):
bootmgr is the replacement of NTLDR since Windows Vista, the boot sequence always starts from bootmgr reading BCD file to get the actual boot device. For installation ISO (Windows Vista+), bootmgr->boot.wim (WinPE) -> install.wim (Windows). For installed Windows on disk, bootmgr -> winload. In this recommended guide, the extracted contents from different Windows ISOs need NO modification at all if saved on different partitions. We just need put correct bootmgr and BCD in expected place and chainloader bootmgr, then let bootmgr handle different Windows.
Below is the entry added to grub.cfg.
menuentry "Windows Boot Manager" {
set root='(hd0,gpt2)'
if [ ${grub_platform} == "pc" ]; then
#linux16 /bios/memdisk iso
#initrd16 /bios/bootmgr.iso
linux16 /bios/memdisk harddisk
initrd16 /bios/bootmgr.vhd
else
if [ ${grub_cpu} == "i386" ]; then
chainloader /EFI/BOOT/BOOTIA32W.EFI
else
chainloader /EFI/BOOT/BOOTX64W.EFI
fi
fi
}Let’s explain it with more details. For BIOS, use memdisk to load bootmgr.vhd in memory and boot it as a harddisk. For UEFI, just chainloader bootmgr, then bootmgr and BCD (BIOS: on VHD, UEFI: in ESP) will handle the actual Windows booting. Be noticed that bootmgr and BCD are different for the two platforms so need to be customized separately. GRUB can’t directly chainloder bootmgr or bootmgr.efi, but need its own boot code (MBR boot code or /EFI/Boot/bootx64.efi or /EFI/Boot/bootia32.efi).
UEFI:
Copy below files from ISO to ESP. I was using the latest WinPE, but you could also get any version of bootmgr from other Windows installation ISO.
# Copy from ISO and rename to differentiate with GRUB's boot loader.
cp /mnt/iso/EFI/Boot/bootx64.efi /mnt/SDB_ESP/EFI/BOOT/BOOTX64W.EFI
# The BCD on ISO can't be used directly, copy to somewhere else (ESP isn't accessible from Windows), modify it, and copy back to ESP.
cp /mnt/iso/EFI/Microsoft/Boot/BCD (somewhere)/BCD
[bcdedit /store BCD ...]# Modify BCD, see details later
cp (somewhere)/BCD /mnt/SDB_ESP/EFI/Microsoft/Boot/BCD
# Also copy Windows memtest for diagnostic purpose on Windows Boot Menu.
cp /mnt/iso/EFI/Microsoft/Boot/memtest.efi /mnt/SDB_ESP/EFI/Microsoft/Boot/
BIOS:
Unlike UEFI boot which can directly chainloader bootmgr, BIOS need firstly boot from VHD/ISO MBR, then let bootmgr on VHD/ISO handle the actual Windows boot.
1. Use VHD:
Let’s use VHD method first. Use diskpart in Windows (or WinPE, or shift+F10 during Windows installation) to create a VHD as below:
X:\Windows\System32>diskpart
Microsoft DiskPart version 10.0.25398.1
Copyright (C) Microsoft Corporation.
On computer: MININT-I9PC50R
# Create an empty VHD file. I found NTFS need 8M to format so can't be smaller although the total files will only take less than 2M.
DISKPART> create vdisk file="c:\bootmgr.vhd" type=fixed maximum=8
100 percent completed
DiskPart successfully created the virtual disk file.
# Select the create vhd
DISKPART> select vdisk file="c:\bootmgr.vhd"
DiskPart successfully selected the virtual disk file.
# Attach the selected vhd as disk
DISKPART> attach vdisk
100 percent completed
DiskPart successfully attached the virtual disk file.
# List and select the correct disk to operate
DISKPART> list disk
Disk ### Status Size Free Dyn Gpt
-------- ------------- ------- ------- --- ---
Disk 0 Online 29 GB 16 GB *
* Disk 1 Online 8 MB 8 MB
# Create primary partition at offset 1024 and mark it as active so that it's bootable
DISKPART> create partition primary offset=1024
DiskPart succeeded in creating the specified partition.
DISKPART> list partition
Partition ### Type Size Offset
------------- ---------------- ------- -------
* Partition 1 Primary 6144 KB 1024 KB
DISKPART> select partition 1
Partition 1 is now the selected partition.
DISKPART> active
DiskPart marked the current partition as active.
# Format as NTFS. FAT32 also works but with NTFS the VHD is smaller in size
DISKPART> format fs=ntfs label="BOOTMGR" quick
100 percent completed
DiskPart successfully formatted the volume.
DISKPART> detail partition
Partition 1
Type : 07
Hidden: No
Active: Yes
Offset in Bytes: 1048576
Volume ### Ltr Label Fs Type Size Status Info
---------- --- ----------- ----- ---------- ------- --------- --------
* Volume 3 BOOTMGR NTFS Partition 6144 KB Healthy
# Assign the newly created partition a volume letter
DISKPART> list vol
Volume ### Ltr Label Fs Type Size Status Info
---------- --- ----------- ----- ---------- ------- --------- --------
Volume 0 D DVD_ROM UDF CD-ROM 332 MB Healthy
Volume 1 C WIN10_22H2 NTFS Removable 4600 MB Healthy
* Volume 3 BOOTMGR NTFS Partition 6144 KB Healthy
DISKPART> select vol 3
Volume 3 is the selected volume.
DISKPART> assign letter=V
DiskPart successfully assigned the drive letter or mount point.
DISKPART> exit
Leaving DiskPart...
# Now use bootsect to write the BIOS/MBR boot code. It has to be successful.
X:\Windows\System32>bootsect /nt60 V: /mbr
Target volumes will be updated with BOOTMGR compatible bootcode.
V: (\\?\Volume{7c5f0f54-0000-0000-0000-100000000000})
Successfully updated NTFS filesystem bootcode.
\??\PhysicalDrive1
Successfully updated disk bootcode.
Bootcode was successfully updated on all targeted volumes.
# Now manually copy necessary files to the partition. bcdboot also works but since you only need 3 files, copy them manually is straight-forward to understand. Be noticed that the BCD here won't work directly as it's copied from current system. See later sections for how to customize the BCD.
X:\Windows\System32>mkdir V:\Boot
X:\Windows\System32>copy c:\bootmgr V:\bootmgr
1 file(s) copied.
X:\Windows\System32>copy C:\Boot\BCD V:\Boot\BCD
1 file(s) copied.
X:\Windows\System32>copy c:\Boot\memtest.exe V:\Boot\
1 file(s) copied.
# Safely detach the vdisk
C:\>diskpart
Microsoft DiskPart version 10.0.25398.1
Copyright (C) Microsoft Corporation.
On computer: MININT-I9PC50R
DISKPART> list vol
Volume ### Ltr Label Fs Type Size Status Info
---------- --- ----------- ----- ---------- ------- --------- --------
Volume 0 D DVD_ROM UDF CD-ROM 332 MB Healthy
Volume 1 C WIN10_22H2 NTFS Removable 4600 MB Healthy
Volume 3 V BOOTMGR NTFS Partition 6144 KB Healthy
DISKPART> select vol 3
Volume 3 is the selected volume.
DISKPART> remove letter=V
DiskPart successfully removed the drive letter or mount point.
DISKPART> select vdisk file="c:\bootmgr.vhd"
DiskPart successfully selected the virtual disk file.
DISKPART> detach vdisk
DiskPart successfully detached the virtual disk file.
DISKPART> exit
Leaving DiskPart...2. Use ISO:
The steps of creating a bootable ISO are much easier than VHD. With oscdimg included in Assessment and Deployment Kit, you could create a bootable ISO from the specified folder, and with customized BCD included, you could boot into different Windows partitions with one bootmgr:
# -u2 and -udfver102 give more compatibility. Unlike WinPE, here we only add 1 boot entry: legacy boot using etfsboot.com as the boot code. Check oscdimg help for all the details.
> oscdimg.exe -m -o -u1 -udfver102 -bootdata:1#p0,e,bC:\WinPE_amd64\etfsboot.com "C:\WinPE_amd64\ISO" "C:\WinPE_amd64\bootmgr.iso"
OSCDIMG 2.56 CD-ROM and DVD-ROM Premastering Utility
Copyright (C) Microsoft, 1993-2012. All rights reserved.
Licensed only for producing Microsoft authorized content.
Scanning source tree
Scanning source tree complete (4 files in 2 directories)
Computing directory information complete
Image file is 3702784 bytes (before optimization)
Writing 4 files in 2 directories to C:\WinPE_amd64\bootmgr.iso
100% complete
Storage optimization saved 0 files, 0 bytes (1% of image)
After optimization, image file is 3686400 bytes
Space saved because of embedding, sparseness or optimization = 0
Done.
# Here's all files you'll need. Same as above, BCD needs customization, otherwise bootmgr can't find the correct device to boot from.
>tree C:\WinPE_amd64\ISO\ /F
Folder PATH listing for volume Windows
Volume serial number is 0000020F 589F:DC0E
C:\WINPE_AMD64\ISO
│ bootmgr
│ readme.txt
│
└───Boot
BCD
memtest.exeBCD customization:
BCD entry is device specific, and it seems bcdedit can only modify BCD entry with the pointed device available in current system, as it relies on the device object that being identified by current running system to set the BCD entry, which relies on the USB drive present in the same system. Below is an example how it looks from WinObj and gdisk. With same USB drive present in the system, bcdedit can set correct entry based on the device element definition [14]. If create a BCD another system without the same USB drive attached, bcdedit will not set correct device, \Device\HarddiskVolume4 will point to another partition, and after copy the BCD to the USB drive, those entry will show device points to “unknown”.
umber Start (sector) End (sector) Size Code Name
1 2048 4095 1024.0 KiB EF02 BIOS boot partition
2 4096 135167 64.0 MiB EF00 EFI system partition
3 135168 16314367 7.7 GiB 8300 Linux filesystem
4 16314368 25735167 4.5 GiB 0700 Microsoft basic data
Comparing the dump result of WinObj with gdisk: \Device\HarddiskVolumex can point to same partition index x from gdisk. As long as you make the BCD customization with the target USB drive attached, there’s no need to worry x will be different on other systems: BCD stores the device element by disk/partition signature and offsets.
BCD for BIOS and UEFI sit in different location: \Boot\ vs. \EFI\Microsoft\Boot\, share most part but the executable name and few other settings: winload.exe vs. winload.efi, memtest.exe vs. memtest.efi. It’s suggested to take the original BCD as the base, add customized entry, and save at (ESP)\EFI\Microsoft\Boot and (ESP)\BIOS\BOOTMGR.VHD (or BOOTMGR.ISO). Below is the example of adding Windows 10 from partition 4.
# Display boot menu in legacy style to choose which boot.wim to load
> bcdedit /store BCD /set {bootmgr} displaybootmenu yes
# Change the ramdisk device location to the partition which the specific Windows stores. HarddiskVolume4 indicate it's 4th partition.
> bcdedit /store BCD /set {default} device ramdisk=[\Device\HarddiskVolume4]\sources\boot.wim,{7619dcc8-fafe-11d9-b411-000476eba25f}
# Ditto to osdevice
> bcdedit /store BCD /set {default} osdevice ramdisk=[\Device\HarddiskVolume4]\sources\boot.wim,{7619dcc8-fafe-11d9-b411-000476eba25f}
# Change the entry description to match the actual Windows to be installed
> bcdedit /store BCD /set {default} description "Windows 10 22H2"
# Change the location of SDI
> bcdedit /store BCD /set {7619dcc8-fafe-11d9-b411-000476eba25f} ramdisksdidevice Partition=\Device\HarddiskVolume4Below is the example of the modified BCD save at (ESP)\EFI\Microsoft\Boot\BCD. You could modify the BCD for BIOS in similar way. Be noticed that although \Device\HarddiskVolume4 is used during bcdedit, the result shows as partition C. This is what mentioned before as “bcdedit can only modify BCD entry with the pointed device available in system”. According to [14], the actual data saved as device entry is composed of disk signature and partition offset. As long as you specific the correct \Device\HarddiskVolumex or volume letter when the USB drive is attached, boot manager will parse the correct partition to boot from and won’t literally use “partition C”.
> bcdedit /store BCD_UEFI /enum all
Windows Boot Manager
--------------------
identifier {bootmgr}
description Windows Boot Manager
locale en-US
inherit {globalsettings}
flightsigning Yes
default {default}
displayorder {default}
toolsdisplayorder {memdiag}
timeout 30
displaybootmenu Yes
Windows Boot Loader
-------------------
identifier {default}
device ramdisk=[C:]\sources\boot.wim,{7619dcc8-fafe-11d9-b411-000476eba25f}
path \windows\system32\boot\winload.efi
description Windows 10 22H2
locale en-US
inherit {bootloadersettings}
isolatedcontext Yes
osdevice ramdisk=[C:]\sources\boot.wim,{7619dcc8-fafe-11d9-b411-000476eba25f}
systemroot \windows
bootmenupolicy Standard
detecthal Yes
winpe Yes
ems No
Windows Memory Tester
---------------------
identifier {memdiag}
device boot
path \efi\microsoft\boot\memtest.efi
description Windows Memory Diagnostic
locale en-US
inherit {globalsettings}
EMS Settings
------------
identifier {emssettings}
bootems No
Debugger Settings
-----------------
identifier {dbgsettings}
debugtype Serial
debugport 1
baudrate 115200
Global Settings
---------------
identifier {globalsettings}
inherit {dbgsettings}
{emssettings}
Boot Loader Settings
--------------------
identifier {bootloadersettings}
inherit {globalsettings}
{hypervisorsettings}
Hypervisor Settings
-------------------
identifier {hypervisorsettings}
description Hypervisor Settings
hypervisordebugtype Serial
hypervisordebugport 1
hypervisorbaudrate 115200
Device options
--------------
identifier {7619dcc8-fafe-11d9-b411-000476eba25f}
ramdisksdidevice partition=C:
ramdisksdipath \boot\boot.sdi Reference:
- http://www.uefi.org/sites/default/files/resources/UEFI%202_5.pdf
- https://thestarman.pcministry.com/asm/mbr/GPT.htm
- https://www.rodsbooks.com/gdisk/hybrid.html
- https://www.gnu.org/software/grub/manual/grub/grub.html#BIOS-installation
- https://superuser.com/questions/1025423/uefi-exfat-partition
- https://superuser.com/questions/1165557/how-grub2-works-on-a-mbr-partitioned-disk-and-gpt-partitioned-disk
- https://wzyboy.im/post/1049.html
- http://reboot.pro/index.php?showtopic=19516&page=2
- https://wiki.syslinux.org/wiki/index.php?title=MEMDISK
- https://superuser.com/questions/1722409/boot-windows-7-10-11-installation-in-grub-on-gpt-partittioned-usb
- https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/configure-uefigpt-based-hard-drive-partitions
- https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/configure-biosmbr-based-hard-drive-partitions
- https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/windows-and-gpt-faq
- http://www.mistyprojects.co.uk/documents/BCDEdit/files/bcd.htm
- https://grub4dos.sourceforge.net/
- http://grub4dos.chenall.net/
Sync Chromium src behind proxy
- Install depot_tools
- Open powershell as administrators and set proxy in netsh>winhttp> set proxy. Backup existing settings before set new one.Set http_proxy and https_proxy in environment variable.
- Edit cipd.ps1, force use proxy for System.Net.WebClient.
...
$proxyAddr = "http://proxy_address:proxy_port"
$proxy = new-object System.Net.WebProxy
$proxy.Address = $proxyAddr
...
$wc.proxy = $proxy
...
- Run gclient.bat to update cipd client.
- Restore cipd.ps1
- Follow the official guide to fetch the code.
- Run netsh>winhttp again to restore original proxy settings.
About Me
Software Engineer.
Colin's Blog 
Recent Comments