[tor-project] shadow update 2023-09-27

Mailing Lists tor-project
1

Mirrored from https://github.com/shadow/shadow/discussions/3187:

This is part of a series of periodic updates of development in Shadow. This work is sponsored by the NSF. Previous update: 2023-06.

We’ve merged 74 non-dependabot pull requests and closed 8 issues since our previous update.

Latest improvements

Spawning processes

Shadow now supports running programs that spawn additional processes. Technically, this means it now supports fork, vfork, execve, and other related syscalls. Some uses for this:

  • The primary way for tor to work with pluggable transports is for the tor process to dynamically spawn the pluggable transport process. This is currently the only way that pluggable transports are supported in arti. This now works in shadow.

  • Orchestration of multiple processes on a single host in shadow can now be done using e.g. sh or python scripts, instead of having to specify every process directly in shadow’s configuration file.

  • shadow can now run other software that spawns worker or helper processes.

Another exciting application for this in shadow development is that we can now more easily run third party test suites, which typically spawn multiple test processes. For example, running tor’s own self-tests helps validate that shadow is correctly emulating the platform functionality that tor uses; currently more than 99% of the tests pass!

New TCP stack

Shadow’s experimental new TCP stack is merged, and can be enabled with the experimental command-line flag --use-new-tcp. This new implementation is in Rust instead of C, and is developed to be very testable (e.g. in its own crate decoupled from the rest of Shadow, and with pluggable system dependencies). While some functionality is still being finished, we expect it will be much easier to maintain and to validate its correctness than the previous C implementation. Improved support for pcap file output also makes it easier to review Shadow’s simulated network traffic.

Shim stability

We’ve occasionally run into problems due to shadow’s preloaded shim calling into libc. This is unsafe because some initialization can run before libc itself is fully initialized, and much of the code runs in a seccomp signal handler, running afoul of async-signal-safety.

Luckily, Rust has a rich ecosystem of code that doesn’t depend on libc (no_std code). We’ve made substantial progress in migrating the shim’s C code to no_std Rust code.

Socket API improvements

Our UDP socket implementation has been migrated from C to Rust, and along the way we’ve made many improvements to the socket API for UDP, TCP, and Unix sockets. We’ve added support for the following:

  • sendmsg, recvmsg, and shutdown syscall support for UDP sockets
  • MSG_TRUNC support for UDP and Unix sockets
  • MSG_PEEK support for UDP sockets
  • SO_DOMAIN, SO_PROTOCOL, and SO_ACCEPTCONN socket options for TCP and UDP sockets
  • SIOCGSTAMP ioctl support for TCP and UDP sockets

We’ve also made various fixes and improvements to existing socket functionality so that Shadow more accurately follows the behaviour of Linux.

C to Rust migration

We continue to progress our migration from C to Rust. In addition to Rust progress in the shim and in the new TCP stack, other notable migrations since the last update include the shared memory allocator, epoll and timerfd descriptors and syscalls, UDP sockets, and several time-related syscalls. Our current status is 74% Rust code and 21.4% C code (much of this being C tests) according to Github statistics.

Release status

We expect to release Shadow 3.1.0 some time in the next few weeks.

Project status - NSF grant is wrapping up

The last three years of development on Shadow have been sponsored by an NSF grant. That grant ends at the end of this month (September 2023). This isn’t the end of development on Shadow — Rob Jansen (@robgjansen) at the U.S. Naval Research Laboratory continues to head the project. Over the course of this project, The Tor Project has incorporated Shadow into its own research and testing workflows, and is likely to continue contributing as well. We look forward to continue improving and maintaining Shadow, but at a slower pace of development since we no longer expect contributions from programmers singularly dedicated to Shadow development. Jim Newsome (@sporksmith) is continuing to be a Tor Project employee but shifting focus to other Tor projects; Steven Engler (@stevenengler) is open to opportunities.

Happy simulating!
The Shadow team

1 Like