2024/01/14 Skyname Incident

On January 14, 2024, Skyname experienced an incident where a malicious actor was able to release and re-register usernames (and subsequently DNS records) for most registered usernames. The attack was detected, mitigated and rolled back within 6 hours, and at most Skyname users had their usernames missing, replaced by “Invalid Handle”, for 5 hours.

For context, Skyname is a service I made for Bluesky that allows users to register a variety of usernames based on domains such as tired.io, bsky.cool and excuseme.wtf. Bluesky usernames are essentially domains, verified via a TXT record, so when you register william.bsky.cool, a DNS record containing your Bluesky account identifier is created and your account is automatically updated.

The exploit was based on how Skyname authenticates users. Since Skyname does not store any tokens or passwords, the browser sends the same authentication token issued by the Bluesky Personal Data Server (PDS) alongside each request. Skyname then goes to the PDS and verifies the token.

Since Bluesky is a decentralised network, there are many servers, so when signing into Skyname you’re able to specify which server to login with. The main server is bsky.social, which is pre-filled.

Unfortunately, due to an oversight / lack of understanding of the Bluesky federation model, after initial authentication took place, Skyname database lookups were not scoped per-user and per-server, which allows a malicious actor to create a PDS that verifies any token with a DID (distributed identifier) inside of it, which Skyname takes as verbatim, and then allows database lookups.

For example, if my DID is did:plc:123 and I am authenticated with bsky.social, an attacker can create a dummy JSON Web Token, attach the DID inside the body attribute, and then attach the url of their dummy PDS to the request. Skyname will reach out to this PDS, which the PDS will then say is a valid authentication token and return a dummy user with the matching DID. When Skyname does a database call, it will not scope the DID to the server, so any record for any user on a different server with the same DID will be returned and/or modified.

Below is an example of this call:

The attacker used this exploit, a copy of the public plc directory, and a Python script to systematically go through every registered username, unregister it, and register it to their own Skyname account.

The fix to this issue was updating each SELECT * WHERE did = did to SELECT * WHERE did = did AND server = server

I apologise for the error here, I take security extremely seriously and this was 100% a mistake on my part. I should have learned more about the potential attacks that can take place trusting a third party server for authentication before releasing a service to the public, especially one so critical for storing peoples online identities. Since the primary Bluesky instance, bsky.social, currently has federation disabled, I only had that PDU in mind when working on Skyname, and didn’t take into consideration how this would change as more people ran their own PDUs.

While I wish the person who did this would have reached out to me privately to responsibly disclose this vulnerability, I’m proud of how quickly I was able to mitigate and fix this attack on my own. The infrastructure I’ve set up in terms of ensuring daily backups of all my services, copious logging and monitoring, and knowing how a system works inside out allowed Skyname to resume operations in record time. I hope Skyname users feel reassured by this, and know that I wouldn’t just tear down shop at the slightest inconvenience. I take the role of being custodian of these services very seriously :~)

Statistics

  • Affected userbase
    • 1,364 out of 1,485 - 91.8% (121 unaffected)
  • Time between first learning about the scope of the attack to shutting down impacted services
    • 3 minutes and 48 seconds
  • Time between the beginning of the attack to restoring all usernames
    • 4 hours, 47 minutes and 20 seconds
  • Time between first learning of the attack to restoring all usernames
    • 60 minutes
  • Time of total site downtime
    • 4 hours, 51 minutes and 53 seconds

Timeline

In the below timeline of events, timestamps are in ETC time, MA stands for Malicious Actor and AU stands for Affected User.

13 January 2024, 9:12:18 pm

  • MA makes first request to Skyname, after being referred via GitHub
IP_ADDRESS - - [13/Jan/2024:21:12:18 -0500] "skyna.me" "GET / HTTP/1.1" 200 925 "https://github.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" "-"

13 January 2024, 10:39:22 pm

  • MA begins to attempt to exploit Skyname, via using a fake PDS called malicious-pds.5u.workers.dev which authenticates any JWT sent to it containing a DID
IP_ADDRESS - - [13/Jan/2024:22:39:22 -0500] "skyna.me" "GET /api/list?server=malicious-pds.5u.workers.dev HTTP/1.1" 500 32 "https://skyna.me/" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" "-"

14 January 2024, 9:36:03 am

  • MA manually releases @drnfsh.tired.io from the @darn.fish account. Subsequent registrations are also made

14 January 2024, 1:47:40 pm

  • MA begins the attack. In total, 3.1k requests would be sent across /api/list, /api/release and /api/register
IP_ADDRESS - - [14/Jan/2024:17:00:32 -0500] "skyna.me" "GET /api/list HTTP/1.1" 200 33076 "-" "python-requests/2.25.1" "-"
IP_ADDRESS - - [14/Jan/2024:17:00:33 -0500] "skyna.me" "POST /api/release HTTP/1.1" 200 0 "-" "python-requests/2.25.1" "-"
IP_ADDRESS - - [14/Jan/2024:17:00:55 -0500] "skyna.me" "POST /api/register HTTP/1.1" 200 0 "-" "python-requests/2.25.1" "-"
IP_ADDRESS - - [14/Jan/2024:17:00:55 -0500] "skyna.me" "GET /api/list HTTP/1.1" 200 179 "-" "python-requests/2.25.1" "-"
IP_ADDRESS - - [14/Jan/2024:17:00:56 -0500] "skyna.me" "POST /api/release HTTP/1.1" 200 0 "-" "python-requests/2.25.1" "-"
IP_ADDRESS - - [14/Jan/2024:17:00:57 -0500] "skyna.me" "POST /api/register HTTP/1.1" 200 0 "-" "python-requests/2.25.1" "-"

14 January 2024, 2:53:13 pm

  • I wake up and share a screenshot of the Skyname log channel with a friend group in Telegram, remarking “Skyname did not blow up or anything”, initially thinking in popularity, while saying a minute later “Actually a lot of it looks like spam lol”

14 January 2024, 2:55 pm

  • I go through the channel and see a lot of activity from MA, who followed me on Bluesky recently. I make a post to them, saying “having fun? 🤪”.
    • At this time I did not realise that the attack was malicious, and believed it was real user activity mixed in with a spammer registering usernames via exploiting the username limit functionality
    • The logs I was basing this assumption on were a mix of: 
      @ma.bsky.social released their username @au.tired.io
@au.tired.io registered a new username!
@au.tired.io released their username @ma.a12b3.tired.io
  • At 15:04, MA replies to my post with “mayhaps” and a screenshot showing 548 usernames registered on their Skyname account. I reply at 15:06 with an emoji

14 January 2024, 3:28:34 pm

  • MA temporarily stops sending automated requests

14 January 2024, 3:31 pm

  • MA shares a post on Bluesky, stating “I now control every skyname handle :)” with a screenshot of their Skyname account having owned 846 handles

14 January 2024, 4:37 pm

  • AU with the handle “Invalid Handle” reaches out to me on the same thread with “Ah, that’s what happened to me. Will this be fixed?”. I reply at 16:49 with “Hey! Could you describe the problem in more detail?”, since I had not yet realised the scope of the attack. They reply later at 5:24 pm with “I mean, look at my handle, it says it’s invalid now. It was .vibes.cool handle registered on @skyna.me”. I reply at 5:36 pm with “Sorry, working on this now, expect to see your username back in 1-2 hours”

14 January 2024, 5:00:32 pm

  • MA resumes the attack via sending automated requests

14 January 2024, 5:25:15 pm

  • MA finally stops the attack

14 January 2024, 5:27:05 pm

  • The production database is dumped and loaded onto my local computer, where I see most, if not all usernames have had their server replaced with malicious-pds.5u.workers.dev

14 January 2024, 5:34:19 pm

  • A message is sent to the original group of Telegram friends saying “Skyname security incident”

14 January 2024, 5:39:48 pm

  • Skyname is taken offline

14 January 2024, 5:43:26 pm

  • MA tries to call /api/list, but gets a 502. Subsequent requests are made using a Firefox browser, but are also 502’d
IP_ADDRESS - - [14/Jan/2024:17:43:26 -0500] "skyna.me" "GET /api/list HTTP/1.1" 502 157 "-" "python-requests/2.25.1" "-"

14 January 2024, 5:50:10 pm

  • A backup of the production database from 14/Jan/2024:00:00:00 is downloaded, the infected production database is backed up to a .tar.gz file, and the backed up database is restored. The database is then turned back on, dumped again, and viewed to ensure data integrity

14 January 2024, 5:59:00 pm

  • Work is begun on a recovery script using the backed up database that downloads all DNS records across all Skyname domains, compares them to what should be in the non-infected production database backup, and then deletes and re-creates the record if necessary

14 January 2024, 6:05:14 pm

  • Messages are sent out to other domain owners who lease their domains on Skyname to alert them as to what happened

14 January 2024, 6:26:15~ pm

  • A script is ran that restores every Skyname users username in DNS

14 January 2024, 6:32~ pm

  • An email is sent to a contact who works at Bluesky, seeing if they’d be able to manually force a handle DNS refresh based on a list of DIDs. This later did not turn out to be required

14 January 2024, 6:35 pm

  • A post is made to the @skyna.me Bluesky account, informing all users of what happened, that it is now fixed, and when the site will be online next

14 January 2024, 7:01 pm

  • I leave to go grocery shopping now that all Skyname usernames have been restored and the exploit can no longer be exploited due to production being taken offline

14 January 2024, 9:18:25 pm

  • MA makes their last request to Skyname, attempting to view the homepage
IP_ADDRESS - - [14/Jan/2024:21:18:25 -0500] "skyna.me" "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" "-"

14 January 2024, 10:11 pm

  • A fix is pushed for the exploit used in the attack

14 January 2024, 10:31:41 pm

  • After five hours of downtime and more auditing, Skyname is bought back online