Building Federated Learning Solutions: Setting Up PySyft

PySyft is a Python library for secure and private Deep Learning. It uses Federated Learning, Differential Privacy, and Encrypyted Computation to decouple private and sensitive data. It can be used within the major Deep Learning Frameworks such as TensorFlow and PyTorch.

The first part of this guide will show you how to install PySyft using Anaconda, as referenced here: https://blog.openmined.org/install/

Step 1: Install Anaconda (https://www.anaconda.com/)

To check that Anaconda is installed, type:

conda -V

Step 2: Set up a virtual environment:

I recommend using python 3.8 through 3.10

conda create -n env python=3.8 anaconda
conda activate env

Step 3: Install PySyft

pip install --upgrade pip
pip install jupyterlab
pip install syft

Currently, the stable version of PySyft is 0.6.0, so if you want to use the latest version from docker you can instead use:

pip install --pre syft

The second part of this guide will show you how to use Hagrid to do deployments in PySyft, using PyGrid, as referenced here: https://openmined.github.io/PySyft/deployment/index.html

Hagrid is a command-line tool that speeds up the deployment of PyGrid, the software providing a peer-to-peer network of data owners and data scientists who can collectively train AI models using PySyft.

Hagrid is able to orchestrate a collection of PyGrid Domain and Network nodes and scale them in a local development environment (based on a docker-compose file). By stacking multiple copies of this docker, you can simulate multiple entities (e.g countries) that collaborate over data and experiment with more complicated data flows such as SMPC.

Step 4: Install Docker and Docker Compose. Docker Compose comes with Docker Desktop on MacOS, which you can install here: https://docs.docker.com/desktop/mac/install/

If you are on Windows or Linux OS, follow the instructions in the article above. To test this, type the following command.

docker compose version

Step 5: Install tox for testing purposes

pip install tox

Step 6: Install Hagrid

pip install hagrid

Step 7: Launch domain node to certain repository and branch

hagrid launch domain to docker:8081 --tag=latest --repo https://github.com/OpenMined/PySyft --branch dev

If you are on MacOS, you might need to change your Memory to 8 GB for Docker Desktop, under the Preferences — Resources tab. To test out this command more simply, feel free to just run

hagrid launch domain

This can take 5 to 10 minutes to run (or a bit longer on MacOS, as linux is preferred). To view the status of your containers, just run

ctop

If ctop is not installed, to install ctop on MacOS run

brew install ctop

You should be able to see the following containers:

backend-1, backend_stream-1, celeryworker-1, db-1, docker-host-1, frontend-1, proxy-1, queue-1, redis-1, seaweedfs-1, tailscale-1

To do a healthcheck, type:

hagrid check localhost:8081

You should see this output (though ideally with more green checks):
┃ PyGrid ┃ Info ┃ Status┃
│ Host │ localhost:8081 │ ❌ │
│ UI │ http://localhost:8081/login │ ✅ │
│ API │ http://localhost:8081/api/v1 │ ✅ │
│ SSH │ hagrid ssh localhost:8081 │ ❌ │
│ Jupyter │ http://localhost:8081:8888/ │ ❌ │