The Surgical Assistant Workstation (SAW) provides a modular, open-source software framework to support rapid prototyping of computer-assisted surgery systems, especially those that benefit from enhanced 3D visualization and user interaction. The framework includes a library of components that can be used to implement master-slave or collaborative robot control systems with support for complex video pipelines and a novel interactive surgical visualization environment. SAW includes standardized interface definitions (e.g., command names and payloads), with the goal of making the framework easily extensible so that developers can add support for their own robotic devices and associated hardware platforms. This paper presents an overview of the component-based architecture, followed by applications (use cases) in the areas of minimally-invasive surgery (MIS), microsurgery, and surgical skill assessment.