Mirrored from https://github.com/shadow/shadow/discussions/3187:
Shadow now supports running programs that spawn additional processes. Technically, this means it now supports
execve, and other related syscalls. Some uses for this:
The primary way for
torto work with pluggable transports is for the
torprocess to dynamically spawn the pluggable transport process. This is currently the only way that pluggable transports are supported in
arti. This now works in
Orchestration of multiple processes on a single host in
shadowcan now be done using e.g.
pythonscripts, instead of having to specify every process directly in
shadow’s configuration file.
shadowcan now run other software that spawns worker or helper processes.
Another exciting application for this in
shadow development is that we can now more easily run third party test suites, which typically spawn multiple test processes. For example, running
tor’s own self-tests helps validate that
shadow is correctly emulating the platform functionality that
tor uses; currently more than 99% of the tests pass!
Shadow’s experimental new TCP stack is merged, and can be enabled with the experimental command-line flag
--use-new-tcp. This new implementation is in Rust instead of C, and is developed to be very testable (e.g. in its own crate decoupled from the rest of Shadow, and with pluggable system dependencies). While some functionality is still being finished, we expect it will be much easier to maintain and to validate its correctness than the previous C implementation. Improved support for pcap file output also makes it easier to review Shadow’s simulated network traffic.
We’ve occasionally run into problems due to
shadow’s preloaded shim calling into
libc. This is unsafe because some initialization can run before
libc itself is fully initialized, and much of the code runs in a seccomp signal handler, running afoul of async-signal-safety.
Luckily, Rust has a rich ecosystem of code that doesn’t depend on libc (
no_std code). We’ve made substantial progress in migrating the shim’s C code to
no_std Rust code.
Our UDP socket implementation has been migrated from C to Rust, and along the way we’ve made many improvements to the socket API for UDP, TCP, and Unix sockets. We’ve added support for the following:
shutdownsyscall support for UDP sockets
MSG_TRUNCsupport for UDP and Unix sockets
MSG_PEEKsupport for UDP sockets
SO_ACCEPTCONNsocket options for TCP and UDP sockets
SIOCGSTAMPioctl support for TCP and UDP sockets
We’ve also made various fixes and improvements to existing socket functionality so that Shadow more accurately follows the behaviour of Linux.
We continue to progress our migration from C to Rust. In addition to Rust progress in the shim and in the new TCP stack, other notable migrations since the last update include the shared memory allocator, epoll and timerfd descriptors and syscalls, UDP sockets, and several time-related syscalls. Our current status is 74% Rust code and 21.4% C code (much of this being C tests) according to Github statistics.
We expect to release Shadow 3.1.0 some time in the next few weeks.
The last three years of development on Shadow have been sponsored by an NSF grant. That grant ends at the end of this month (September 2023). This isn’t the end of development on Shadow — Rob Jansen (@robgjansen) at the U.S. Naval Research Laboratory continues to head the project. Over the course of this project, The Tor Project has incorporated Shadow into its own research and testing workflows, and is likely to continue contributing as well. We look forward to continue improving and maintaining Shadow, but at a slower pace of development since we no longer expect contributions from programmers singularly dedicated to Shadow development. Jim Newsome (@sporksmith) is continuing to be a Tor Project employee but shifting focus to other Tor projects; Steven Engler (@stevenengler) is open to opportunities.
The Shadow team