Self-hosting a Matrix server for 5 years

Yaky's

home |
reads |
notes |
apps

Self-hosting a Matrix server for 5 years
Experiences with the Matrix protocol, Matrix Synapse server, bridges, and Element mobile apps.
I have been hosting a Matrix server for about five years now, mostly for text chats between a few relatives and close friends, and a bridge to WhatsApp for a few more people. These are my experiences.

Matrix protocol
I don't have many thoughts on the protocol itself.
The only thing that I don't really understand is the decision on data replication. If a user on server A joins a room on server B, recent room data is copied from server B to server A and then kept in sync on both servers. I suppose this reduces the load on the original server at the expense of federation overhead and space on other servers. However, this also creates a situation where anything said across federation cannot be unsaid, which is an ironic situation for a protocol/system that often comes up when talking about privacy.
IIRC, fediverse/ActivityPub uses a similar approach.

Synapse server
Synapse is the only choice that supports bridges, which was why I wanted to try Matrix in the first place. And back in 2019-2020 this was the only choice anyway.
As of right now, I run Synapse, PostgreSQL, and coturn directly, without containerization, on a small VPS.
Works well
Works fairly reliably, supports bridges, and is more efficient that it was in 2020.
API is well documented, and allows authenticating and sending (unencrypted) messages via simple HTTP calls. At some point in time, I wanted to write a simple shell client to use with SXMO and such.
Does not have an admin panel
There is no admin page or panel. There was a third-party admin site, but it's an entire site just for making HTTP calls. So I ended up writing my own.
My Simple Synapse Admin page
(Nowadays, the ESS deployment includes developer-made admin, see Future section)
Requires PostgreSQL
While technically, Synapse can work with a sqlite database (and which at first seems like an OK choice for having <10 users on the server), it WILL become corrupted. So PostgreSQL is de-facto mandatory.
(Already a part of new ESS)
Requires federation
Initial setup presumes that the server is going to be federated, and there is no good way to turn it off. The best workaround involves a blank whitelist of federated servers.
GitHub issue: Single config option to disable federation
I don't know the implications of disabling it.
Needs constant cleanup
Message retention policy can be set up server-wide, but also per-room. There are specific lines in the configuration that need to be set to actually enable a service that runs the cleanup.
Synapse keeps the room even after all of the members leave it, including federated rooms. This results in many (sometimes large) rooms without local members orphaned on the server, taking up database space.
Deleting messages (events) with attachments does not delete the attachment (because another message might refer to it?), which means that the sent files continue existing on the server indefinitely. Another privacy implication. A simple "delete all files older than X" script works great until it deletes avatars. So yeah, seems like this is something that should be handled by the Synapse server instead of cobbled-together scripts.
Even after extensive cleanup, PostgreSQL database might need to be vacuumed to reduce the disk space it takes up.
Database grows out of control
Even for my small server with <10 active users, database size reached several gigabytes.
Synapse keeps track of room states in an append-only (!) table named state_groups_state. Deleting a room does not delete the state_groups_state records. So it is never automatically cleaned up, and grows in size infinitely. It is possible to delete many of those records from the database directly, and Element (the company) provides some tool to "compress" those records, but again, something that should be handled by the server.
Good article about state_groups_state
Users cannot be deleted
This is simply not an option in the API. Server admin can perform a "deactivate" (disable login) and "erase" (remove related data, which claims to be GDPR-compliant) on user accounts, but the accounts themselves stay on the server forever.
Wait, what? Why?
How this not considered a GDPR violation is a mystery to me. Even on my tiny server, I have users who use their first name as their ID and bridged WhatsApp users that use phone numbers as IDs.
GitHub issue
Future
While Matrix-Element ecosystem has been catering towards government and corporate entities for some time, there have been multiple recent announcements about its future.
Specifically, Element (the company) is now providing an all-in-one Element Server Suite (ESS) to replace the current setup, including
ESS Community
It is intended for non-professional use, evaluations, and small to mid-sized deployments (1–100 users).
ESS Community includes 7 components/services, now requires a minimum of 2 CPUs, 2GB of RAM, and runs using... Kubernetes? IMO, this is an overkill for dozen users.
For comparison, Snikket, an all-in-one solution with similar functionality using XMPP, requires a single CPU and 128MB (!) RAM for 10 or so users.
Yes, I have seen the ansible setup script setup recommended, but at this point, making setup easier does not address the issue of extra services being required in the first place.
Matrix server setup using Ansible and Docker
Also, the ESS handles account creation and calls in an entirely different way, more on that later.

Matrix-WhatsApp bridge
Pretty great. Easy to install and set up, works really well, and needs only occasional (semi-yearly or so) updates when WhatsApp changes their web API. Does not support calls.

Element Classic
Same on all platforms
Element exists and looks consistent on Android, iOS, and web, making it easier for regular users and for troubleshooting.
No image captions
This is silly, but while (official?) bridges support image captions, official Element app does not. The answer in the FAQ? Get a better app. Well, OK.

No image caption in Element Classic.

Image with a caption in SchildiChat Classic (the better app).
Slow notifications
Sometimes it can take up to a few minutes to get a message, even between two Android clients using Google Cloud Messaging. Sometimes it is nearly instant. Still unsure of the cause.
No offline indication
One unreliable way to tell that the server is unreachable is the endless loading bar. But even then, it eventually goes away without indicating any errors.
Then, when sending a message, the user receives "Unable to send message". Frustration ensues.
But I know the app is trying to call the /sync endpoint. Why doesn't it show any errors when that fails?
Security key and device verification
IIRC the first thing the app does is ask user to back up their signing keys and enter the key password, without a simple explanation. Not a great experience for regular users.
Some people reported issues with Element losing its keys or frequently requesting to be re-verified. Thankfully I have not encountered these.
Third-party services
Even if you connect to a self-hosted server, Element Classic could attempt to connect to vector.im integration server and matrix.org key backup server.

Element X
Element X is now recommended as the new and better client. It is not.
Slower
Somehow, it is slower. Clicking on a conversation takes 0.5-1.0 seconds to load it, compared to almost instant load on Classic.
Perhaps it does work better for accounts with many large rooms, but that is not my case.
Sorting
Conversations are sorted by... who knows. It is not recent nor alphabetical.
No background sync
Element X does not support periodic background sync, so you need to set up ntfy or something similar to use Element X on a de-googled device. Seems like a simple enough fail-safe (even WhatsApp does this), but it was dropped for some reason.

Requires "sliding sync" option on the server
This "sliding sync" option is available only for newer Synapse versions, and only if running with PostgreSQL database (which should already be the case - see above). Probably not an issue unless the user tries to connect Element X to an outdated Synapse.
Calls are not backward compatible
Calling with Element X requires Element Call (part of ESS). This supports group calls, but... only video calls at the moment.

You also might be asked to tell your contact to install the new app:

I don't regularly use calls, but some people I would like to invite to my server would want to use them.

Onboarding is bad
A few years ago, I ended up either temporarily enabling unrestricted registration (a terrible idea), or creating my users' accounts manually, because the "invite" matrix.to link was broken, and registration tokens did not work correctly in mobile apps.
So let's see how it works now. Keep in mind, I am still on standalone Synapse, not ESS.
Element X onboarding
I am a user, and I was to register an account on my friend's server. I see that Element X is now a recommended app, so let's try that.

Click "Create account" (which is a different style that does not look like a button for some reason).

But I want an account on a different server. Click "Change account provider".

Click "Other".

Now I can search for the server my friend is hosting, and it should appear in the list below the search.
As server admin: I do not remember if Synapse server has to enable/keep federation for this to work.

Yes! That is what I want, why is this so verbose?

WTF. So Element X cannot create even the simplest username+password account. That is all I want, I don't want to sign in with Google, Apple, or any other form of third-party authentication.
Element Classic onboarding
I was unable to register an account using Element X, so Element Classic should work better.

Ok, "CREATE ACCOUNT".

What difference does this make? Skip.

The current official app is telling me to use Element X. Just tried that. Click "EDIT" where it says "matrix.org" (which does not say "server", actually) and enter the server name.

Why not? No explanation. Sure, I'll use a web client.

Well, fuck me, I guess. Why can't I just create an account?
As a server admin: Synapse is set to allow registrations via registration tokens, because unrestricted registration is a bad idea. I did not find where the /static/client/register path is set.
IIRC it is possible to register an account by going to a web-hosted Element app, such as app.element.io, which will allow to register an account using a registration token. But then the user has to deal with the headache of cross-verifying their mobile device to the web app (which they might never use).

So now what?
Matrix-Element is growing, building new features, and acquiring large customers (mostly government entities AFAIK). However, the new corporatesque ESS Community is not worth it in my opinion. I don't need fancy auth, third-party IDs, group video conferencing, or even federation for that matter. But it is clear that Synapse and Element X are severely crippled and are not designed to work without these services.
I will probably switch to Snikket, which is more efficient, has timely notifications, and very smooth onboarding.
Snikket

Who cares?
¯\_(ツ)_/¯

home | email me: hi@yaky.dev
Contains no LLM-generated content
Distributed under CC BY-NC 4.0

Here’s a detailed summary of the document, “Self-hosting a Matrix server for 5 years,” designed for a college graduate:

This document details the experiences of a long-term (five-year) self-hosted Matrix server user, primarily focused on a small, private network of relatives and friends, utilizing a WhatsApp bridge for expanded connectivity. The author’s primary focus was on a standalone Synapse server alongside Element Classic and Element X clients. The report highlights the strengths and weaknesses of these components and the broader Matrix ecosystem as of 2023.

**Core Server: Synapse**

The author’s choice of Synapse was driven by its bridge capabilities, which were crucial at the time of its initial adoption (2019-2020). Synapse, coupled with PostgreSQL, proved to be a relatively reliable solution, particularly when running without containerization. It supports bridges and is more efficient than earlier versions. The server includes a well-documented API allowing for authenticating and sending (unencrypted) messages. However, the absence of an official admin panel prompted the author to develop a custom solution.

**PostgreSQL Database Dependency:**

The document stresses the critical requirement of using PostgreSQL over SQLite for Synapse. SQLite becomes corrupted under moderate user load, making PostgreSQL essential for a stable server. The database grows rapidly without active management. Efforts to manage this growth include the use of vacuum commands to reduce disk space utilization.

**State Management and Orphaned Rooms:**

A significant operational challenge is Synapse’s append-only state_groups_state table. Deleting rooms doesn’t remove this data, resulting in “orphaned” rooms – rooms with no active members – that consume valuable database space indefinitely. While the author employs cleanup scripts and the ESS team provides a “compression” tool, proactive management is necessary. The lack of automated cleanup highlights a critical area for improvement within the Synapse server itself.

**User Management Limitations:**

The document identifies several fundamental limitations within the Synapse server’s user management system. Users cannot be deleted—accounts are perpetually retained on the server. Despite employing registration tokens or “deactivating” and “erasing” accounts, these actions do not permanently remove user data, raising serious GDPR compliance issues, especially given users’ reliance on first names and phone numbers for identification. The lack of a straightforward option to create simple username/password accounts is a significant friction point.

**Element X and ESS Community - A Shift**

The emergence of the Element X client and the ESS Community (Element Server Suite) represent a significant shift in the Matrix ecosystem. The ESS Community, intended for non-professional use and small to mid-sized deployments, requires a minimum of 2 CPUs and 2GB of RAM – a considerable resource investment compared to the author’s minimal configuration. The ESS Community’s focus on third-party authentication and group video conferencing is deemed unnecessary and potentially problematic by the author.

**Onboarding Challenges:**

The author encountered difficulties onboarding new users, particularly with Element X. The inflexible account creation process—requiring registration through a web-hosted Element app – and reliance on registration tokens rather than simple username/password authentication created significant operational hurdles. This highlights a fundamental design flaw within the newer clients.

**Component Breakdown**

* **Element Classic:** Remains a reliable client, offering consistent functionality across platforms.
* **Element X:** Slower than Classic, with sorting and background sync issues.
* **Element Call:** Primarily for video conferencing, dependent on ESS.
* **WhatsApp Bridge:** Functional and relatively easy to maintain, updated semi-annually to adapt to WhatsApp's API changes.

**Overall Assessment**

The author’s experience demonstrates the technical challenges of self-hosting a Matrix server, particularly with older clients like Synapse and Element Classic. The ESS Community, while offering advanced features, appears overly complex and resource-intensive for the author's purposes. The core issues of database management, user account limitations, and inconsistent onboarding processes highlight areas where the Matrix ecosystem, particularly its newer components, must improve to provide a more seamless and user-friendly self-hosting experience. The author plans to migrate to Snikket, suggesting dissatisfaction with the complexity and resource demands of Synapse and Element X.