Mirrored from https://github.com/shadow/shadow/discussions/3187:
This is part of a series of periodic updates of development in Shadow. This work is sponsored by the NSF. Previous update: 2023-06.
We’ve merged 74 non-dependabot pull requests and closed 8 issues since our previous update.
Latest improvements
Spawning processes
Shadow now supports running programs that spawn additional processes. Technically, this means it now supports fork
, vfork
, execve
, and other related syscalls. Some uses for this:
-
The primary way for
tor
to work with pluggable transports is for thetor
process to dynamically spawn the pluggable transport process. This is currently the only way that pluggable transports are supported inarti
. This now works inshadow
. -
Orchestration of multiple processes on a single host in
shadow
can now be done using e.g.sh
orpython
scripts, instead of having to specify every process directly inshadow
’s configuration file. -
shadow
can now run other software that spawns worker or helper processes.
Another exciting application for this in shadow
development is that we can now more easily run third party test suites, which typically spawn multiple test processes. For example, running tor
’s own self-tests helps validate that shadow
is correctly emulating the platform functionality that tor
uses; currently more than 99% of the tests pass!
New TCP stack
Shadow’s experimental new TCP stack is merged, and can be enabled with the experimental command-line flag --use-new-tcp
. This new implementation is in Rust instead of C, and is developed to be very testable (e.g. in its own crate decoupled from the rest of Shadow, and with pluggable system dependencies). While some functionality is still being finished, we expect it will be much easier to maintain and to validate its correctness than the previous C implementation. Improved support for pcap file output also makes it easier to review Shadow’s simulated network traffic.
Shim stability
We’ve occasionally run into problems due to shadow
’s preloaded shim calling into libc
. This is unsafe because some initialization can run before libc
itself is fully initialized, and much of the code runs in a seccomp signal handler, running afoul of async-signal-safety.
Luckily, Rust has a rich ecosystem of code that doesn’t depend on libc (no_std
code). We’ve made substantial progress in migrating the shim’s C code to no_std
Rust code.
Socket API improvements
Our UDP socket implementation has been migrated from C to Rust, and along the way we’ve made many improvements to the socket API for UDP, TCP, and Unix sockets. We’ve added support for the following:
sendmsg
,recvmsg
, andshutdown
syscall support for UDP socketsMSG_TRUNC
support for UDP and Unix socketsMSG_PEEK
support for UDP socketsSO_DOMAIN
,SO_PROTOCOL
, andSO_ACCEPTCONN
socket options for TCP and UDP socketsSIOCGSTAMP
ioctl support for TCP and UDP sockets
We’ve also made various fixes and improvements to existing socket functionality so that Shadow more accurately follows the behaviour of Linux.
C to Rust migration
We continue to progress our migration from C to Rust. In addition to Rust progress in the shim and in the new TCP stack, other notable migrations since the last update include the shared memory allocator, epoll and timerfd descriptors and syscalls, UDP sockets, and several time-related syscalls. Our current status is 74% Rust code and 21.4% C code (much of this being C tests) according to Github statistics.
Release status
We expect to release Shadow 3.1.0 some time in the next few weeks.
Project status - NSF grant is wrapping up
The last three years of development on Shadow have been sponsored by an NSF grant. That grant ends at the end of this month (September 2023). This isn’t the end of development on Shadow — Rob Jansen (@robgjansen) at the U.S. Naval Research Laboratory continues to head the project. Over the course of this project, The Tor Project has incorporated Shadow into its own research and testing workflows, and is likely to continue contributing as well. We look forward to continue improving and maintaining Shadow, but at a slower pace of development since we no longer expect contributions from programmers singularly dedicated to Shadow development. Jim Newsome (@sporksmith) is continuing to be a Tor Project employee but shifting focus to other Tor projects; Steven Engler (@stevenengler) is open to opportunities.
Happy simulating!
The Shadow team