brho@google.com
2025-06-05

This directory contains a bunch of scripts and working directories to build a
Linux guest kernel and a u-root image plus some extra junk.

Typical usage: make a Localconfig from Localconfig.template and run
full_guest_build.sh.

You'll need to acquire u-root (git@github.com:u-root/u-root.git), go build it,
and set UROOT_REPO in your Localconfig.  

You also must set LINUX_REPO.

I used to have a bunch of extra scripts that did things like install tinycore
packages or let you fine-tune what parts of the build to use.  If you need
that, check out the git history.

TODO: tc-sys.sh's magic IP configuration is busted - no ifconfig in u-root.
Need to convert that stuff to ip commands.


Various files and directories:
--------------------
- Localconfig: you must set this up, using Localconfig.template as an example.
  Customizations in Localconfig:
  - UROOT_EXTRA: commands to u-root, in particular to add files/programs from
    your machine to the initramfs.
  - ssh keys (looks for my key named "db_rsa", provide your own keys)
  - etc.

- full_guest_build.sh: produces vmlinux and initrd from scratch.  Calls the
  other scripts.
  - optionally copies kernel.config to $LINUX_REPO/.config
  - builds u-root initramfs
  - builds the extra "TC" initramfs, including kernel modules
  - slams them together, rebuilds the kernel

- progs: compiled helper programs to run in the guest.
- tc_root: TC's root filesystem.  Extracted and edited by the scripts
- vm-apps: Akaros scripts to run an app as a guest VM, e.g. an ext4 9p server.
- tc-sys.sh: script run during the guest's boot, called from u-root's init-cmd.


Misc helpers:
--------------------
- cat_cpio_gz.sh: Dumps the contents of an initramfs to easily see what takes up
  the most space.

- embed_payload.sh: Embeds a payload with a shell script and makes it
  executable.  Used for Akaros's vm-apps.


Example Commands to run the Virtual Machine:
--------------------
- Akaros:
  	vmrunkernel -k tinycore_cmdline -n vnet_opts -N 8 -i initramfs.cpio.gz vmlinux

	tinycore_cmdline:
		earlyprintk=akaros	# optional
		console=hvc0
		mitigations=off
		nozswap			# for tinycore faster boots
	vnet_opts:			# all optional
		snoop
		nat_timeout = 30
		map_diagnostics
		port:tcp:23:22

	With this setup, you can ssh to localhost:23 and it'll port-foward to
	the guest.

- Tun/Tap:
	A bunch of VMMs on Linux want a tun/tap setup.  I use tun-up.sh, which
	is mostly this:

	ip link add br-vm type bridge
	ip addr add 10.0.2.2/24 dev br-vm 2>/dev/null || true
	ip addr add fd0:1234:4321::2/64 dev br-vm
	ip link set up dev br-vm
	ip tuntap add tap-vm mode tap
	ip link set up dev tap-vm
	ip link set tap-vm master br-vm

	This creates a virtual network similar to the one that Qemu's usermode
	networking will make: guest will get 10.0.2.15, host gets 10.0.2.2.
	Akaros's VMM NAT does the same thing.

	It also adds an IPv6 address, in case your host kernel is IPv6-only.
	The guest will use fd0:1234:4321::15/64

- Qemu:
	Note that qemu takes the bzImage, but not vmlinux.  There might be some
	kernel config issue, but my qemu with the vmlinux from these scripts
	will complain with "qemu: linux kernel too old to load a ram disk".

	If you have a tun/tap setup, with a tap named "tap-vm":

	qemu-system-x86_64 -s -enable-kvm -cpu host -smp 8 -m 1024 -nographic \
	-kernel arch/x86/boot/bzImage -initrd initramfs.cpio.gz \
	-append "console=ttyS0 nozswap" \
	-device virtio-net-pci,netdev=vmnet,mac=00:01:02:03:04:0b \
	-netdev tap,id=vmnet,ifname=tap-vm,script=no,downscript=no

	You can also use the old user-mode networking (forwarding host 5555 to
	guest 22).  Replace the -device and -netdev with:
	-net nic,model=e1000 -net user,hostfwd=tcp::5555-:22

	There are plenty of other options.

	Either way, the guest will have IP 10.0.2.15, gateway 10.0.2.2 (the
	host).  For usermode networking, that comes from DHCP.  For the tun/tap,
	that comes from the magic MAC address 00:01:02:03:04:0b.  The tinycore VM
	has an init script that looks for that address and statically configures
	networking.

	You can add other linux command line args, such as console=hvc or
	"console=ttyS0 earlyprintk=serial".  hvc (virtio-cons) didn't get the
	bootspew that ttyS0 gets, which helps with debugging, so YMMV.

	To stop the guest, try "Ctl-A, X".  "Ctl-A, C" gives you the qemu
	monitor.

	The biggest thing with qemu for me is that it jacks up your terminal,
	such that the console text won't wrap anymore.  Try:

	tput smam

	I put that in all of my qemu scripts and even have a bash alias for it.

- Cloud Hypervisor:

	./target/release/cloud-hypervisor --console off --serial tty \
	--cpus boot=8 --memory size=1024M \
	--kernel vmlinux --initramfs initramfs.cpio.gz \
	--cmdline "console=ttyS0 nozswap" \
	--net tap=tap-vm,mac=00:01:02:03:04:0b,ip=,mask=

	Same as with qemu, hvc0 works, but is a little on the quiet side.
	nozswap is for tinycore.  The MAC address tells the guest to set itself
	up as 10.0.2.15.

	To stop the guest, there's some involved incantation to tell
	cloud-hypervisor to shutdown, but I just:

	killall cloud-hypervisor

- SSHing to the guest under various environments:

	If the guest is listening on host port 5555 (from my example qemu
	usermode networking), set up your ~/.ssh/config like so:  The
	IdentityFile is the one you specified in full_guest_build.sh.

	If you mind the constant MITM warnings, you can suppress them in
	.ssh/config with these:
		StrictHostKeyChecking no
		UserKnownHostsFile /dev/null

	Host qemu
		Hostname 127.0.0.1
		User root
		Port 5555
		IdentitiesOnly yes
		IdentityFile ~/.ssh/db_rsa

	This ought to work for a tap:

	Host qemu-tap
		Hostname 10.0.2.15
		User root
		Port 22
		IdentitiesOnly yes
		IdentityFile ~/.ssh/db_rsa

	But if your workstation has an "ssh-unfriendly" policy, you might have
	issues.

	Instead, you can use port forwarding.  Set up a tunnel from your
	workstation's 5555 to the VM guest's 22 on the tun/tap network of your
	workstation.

	ssh -L 5555:10.0.2.15:22 127.0.0.1

	Now you can ssh to the guest just like with qemu's usermode net port
	forwarding. (ssh qemu, from above).

	Similarly, if you're running the VM on *another* host that you have ssh
	access to, but you can't ssh to any port other than 22, you can forward
	from your workstation's 2345 to the host's port where the guest is
	listening.

	ssh -L 2345:127.0.0.1:23 $HOST

	I'll use that to ssh from my Linux workstation to an Akaros $HOST
	(though it could also be Linux).  The tunnel endpoint is $HOST,
	connecting to it's localhost:23, which I'll often port-forward to the
	guest (e.g. in Akaros's vnet_opts above).

	Host linux-guest
		Hostname 127.0.0.1
		User root
		Port 2345
		IdentitiesOnly yes
		IdentityFile ~/.ssh/db_rsa

	If you don't want to mess around with ssh-config, you can do:

	ssh -o IdentitiesOnly=yes -o IdentityFile=db_rsa root@fd0:1234:4321::15

	or even make a bash alias:

	alias ssh-guest-v6="ssh -o IdentitiesOnly=yes -o IdentityFile=db_rsa root@fd0:1234:4321::15"