While writing a p2p communication protocol for my current project, I was faced with the problem of discovery of peers and their authenticity.
The first security measure is to sign every client or peer that is deployed to the network. The server has a RSA keypair. It uses the private key to sign the IP of the client being installed. The public key of the server is also given to the client for verification of the server instructions.
This seems like a good idea because it prevents a third party to infiltrate the network unless the server infects the attackers computer. We also make sure that the instructions from the server are from the server.
To allow clients to replicate and install themselves on other computers, we need a way of asking the server for a certificate for a given IP. For this to work, the server has to verify that the client asking for a new certificate is already part of the network. This is problematic because if an attacker gets their computer infected, they now can create as many peers as they want.
Instead of trying to keep attackers out of the network, I propose the idea of making every peer pay a price for getting into the network. The peers should start with an "untrusted" status and become "trusted" only after they give the server something we want. This could be a certain amount of time mining monero for example.
With this system, the "untrusted" peers cannot get a very limited amount of information from other peers. They are able of relaying instructions to other peers. This way, you allow the attacker to get information only if they help the network function and they potentially make you some money.
How do we know the state of a peer? This could be fixed in a centralized manner, just by asking the server about the peer. This is good if you want to verify peers by making them mine crypto. Peers could send their proof of work to the server, the server verifies that it is valid and lists the peer as trusted on a database.
The decentralized solution is to let every peer decide if a new peer is or not trusted. This way, peers will only give information to already trusted peers. This would only work if some peers are trusted by default. As a solution to this, the only peers that start as untrusted would be the ones not directly created by the server. So if a peer installs the program on another computer, the new peers is not trusted by anyone. As an example, imagine you are a peer from the network. You have in your known peer list an untrusted one. After this untrusted peer has sent you 10 instructions from the server and you have verified they have not been modified, you can set the status of this peer to trusted.
A hybrid model would first ask the server if the peer is trusted, and update the local database accordingly. When a peer changes the status of another peer to trusted, this would be communicated to the server and when the server receives enough of these, it would update the global status of the peer.
So what is enough peers? We should set a minimum number of peers, but this absolute value is not good. We also need to check the number of peers that do not trust it. So we need to get the list of connected peers. This should be fairly easy using DHT and XOR to calculate the distance to peers. The hash of each peer would be the certificate given by the server. And so you would calculate how many of the neighbors of your peer trust it and judge it that way.