Decentralising the Internet
by Jason Prado (@jasonpjason) on September 23, 2018

In August, Apple became the first company to surpass a trillion dollars in market cap, with other tech giants not far behind. What makes these companies so valuable is the control they wield over different services on the internet, like app stores and communication products, which has allowed them to accumulate wealth at an unprecedented rate. In contrast to the early days of the internet, when vital internet infrastructure embraced decentralisation as technical and organisational feature, the tech giants’ platforms have embraced a more centralised structure that is friendlier to the corporate form and to profit.

In order to challenge this state of affairs, we should first examine the underlying technical traits that have made these platforms so successful in the first place. That requires thoroughly investigating the link between a tendency toward centralisation inherent in technological platforms and the capital accumulation process. Only then can we envision robust alternatives: internet platforms that offer the same value as centralised services without being captured by the drive for capital accumulation. Specifically, we should look at decentralisation as a way to tackle both current companies’ dominance while increasing democratic control and user agency.

The failures of federation

The computing field seems to have a pendulum that swings back and forth between centralisation and decentralisation. The transitions from mainframes to desktop computers to web browsers to mobile apps each demonstrate the swing of this pendulum. We may be used to the internet being highly centralised today, but it wasn’t always this way.

A recent piece in New Socialist responding to Corbyn’s Alternative MacTaggart speech, which proposed state alternatives to digital monopolies, critiques Corbyn’s implicit acceptance of centralisation. Instead of a state-run social networking platform, the authors argue, it’s time to swing back toward a more decentralised model.

To achieve this, the authors suggest that popular internet platforms should be re-architected to use open standards and the ‘federation’ model. The former involves service providers agreeing on a common set of protocols to interoperate between competing services. The latter involves a system where users choose among instances of a service, operated by different entities, and communicate through their chosen instance with users on other instances via standardised protocols. Email is a prime example, as an email user [email protected] can use their email program to communicate with [email protected] despite the fact that Yahoo and Gmail are competitors.

It makes sense to look at federation as a model of decentralisation, as it has a long track record of success in core internet services like email. Unfortunately it also has a long, and more recent, track record of failure, as federated services have either been replaced by closed competitors or become de facto centralised. In the past, internet users often purchased email services through their local internet service provider (ISP). Today, most regions are served by only a few or even one broadband ISP, and most email users use a handful of email services owned by tech giants like Google and Microsoft.

We should also stop to understand exactly what the architects of the internet aimed to achieve through decentralisation. In itself, decentralisation has no value. Rather, it is valuable for how it affects the landscape of power. By embedding decision-making power in the leaves of the network, in the citizenry that make up the internet, decentralisation provides the basis - at least in theory - for more democratic control. In practice, however, federated services have tended towards centralisation under privately-held tech companies who offer users few choices and little control. If the criterion we use for evaluating an architectural scheme of internet platforms is how much it gives people power over their own lives, then it’s clear that federation does not actually achieve substantial empowerment.

Finally, it is worthwhile to pry apart the conflation of platform capitalism with platform centralisation in general, as platforms seem to tend towards centralisation and homogeneity even divorced from the imperatives of capital. Why do people want their preferred open source JavaScript framework or their favorite free software distribution of GNU/Linux to become popular? In the case of software platforms, it’s largely because contributors want to contribute their labour to something valuable; the more people use a piece of software, the more people will contribute, and the network effects of standardisation will yield more value via larger platforms. We shouldn’t underestimate the utility of these network effects; instead, we should aim to liberate platforms from the control of capital while simultaneously capturing the value made possible by modern technology platforms.

Looking back

It was out of technical necessity, not just political ambition, that the internet was designed using the most cutting-edge decentralisation technologies available at the time. Its design was never robust enough to withstand the forces of capital accumulation permanently, and it’s clear that these forces have now fully toppled the bulwarks of decentralisation built into the early internet. New technological innovations will be necessary to build a re-decentralised internet, and new organisational schema must be developed to redistribute power on the internet.

Email (1972), the domain name system (1983), and the world wide web (1991) were designed for a hobbyist internet, a completely different system than the one we use today. Even XMPP, an open standard for messaging software first released in 1999, was built for a different iteration of the fast-moving world of technology. Today, these open technologies have lost ground to centralised, commercially-backed successors.

Early internet services were built on the federation model. In this model, anyone can host an instance of a service and users connect to their preferred instance. Instances connect to other instances and communicate between each other on their users’ behalf, allowing any user in the federation to contact any other user. While email is the one remaining successful example of the federated model, newer experimental communication services like Diaspora or Mastodon also build upon the federation model. These new services have yet to see any remotely significant success, and with the exception of email, every communication product that currently supports a billion users is centralised.

The rise of centralised services

The reason for the success of centralised services is that they are often just better. Centralised services win by delivering a better user experience, by accumulating data which guides development, and by capturing the value of network effects. Centralised services also attract investment because extracting profit from a centralised service is significantly easier than from a federated service built on open standards. To compete with such commercial entities, a new generation of services will need to innovate in these domains both technologically and in organisational structure.

In terms of user experience, federated services are typically harder to use than their centralised competitors. This may have been acceptable for hobbyists and tinkerers, but building a service that scales to billions of users requires the most straightforward possible user experience. Compared to social networks with simple identity models like real names or phone numbers, an email address or Mastodon ID like [email protected] requires information about the particular federation server used. What’s more, federated services built on open protocols have failed to dynamically respond to changing user preferences. When the iPhone launched, push notifications for email were essentially unavailable for years, and messaging protocols like XMPP took a full decade to account for the realities of mobile phones. Solely on the basis of features and user experience, federated services have not kept up.

Why have federated products failed to keep up in terms of user experience? It’s partly the result of inadequate focus on design. Developers of open protocols often misunderstand their goals. A standards-minded developer might view the output of their labour as a set of standards, a platform upon which others may build. However, for a platform to have value, others must actually build upon it. Convincing someone else to build upon a platform that really only exists as a standards proposal or a proof of concept is difficult. On the other hand, a commercial entity makes it their goal to deliver an experience to a user through a product. If a product becomes successful, it can spawn a platform with itself as the first client. If decentralisation is to have a chance, builders of the decentralised services of the future should follow a product-centric development model that incorporates the past several decades of developments in design methodology.

While much of the poor user experience found in decentralised services can be attributed to obsolete development processes, it is increasingly important to incorporate data about a service’s usage into the development process. With users’ usage data siloed across different instances, federated services will always lack the accumulation of data that a centralised service uses to improve itself. Services use data-driven decision-making processes to prioritise feature development and pinpoint flaws in their products. While data collection by private corporations has been revealed to be invasive and problematic, especially when it comes to ad targeting, telemetry about usage of a service does not need to push against norms around privacy to help a service generate value.

Lessons & Prospects

If federation has failed, what can we learn from its history and what should we build going forward? The key realisation is that what decentralisation has given us is user control. Federation offers user control by giving users a choice; if a user is unhappy with one instance of a federated service, they can switch to another or start their own. This choice is not always realistic, though, as few users actually have the time, expertise, or resources to host their own instance. It’s analogous to the false freedom available to workers in a market economy - despite the typical rejoinder to those who complain about their job, switching to a different job won’t always improve things, and few have the resources to start their own business. The other downside of federation’s implementation of user control is that it leads to fragmentation, resulting in complicated, poor user experiences. The goal, then, should be to build services that give their users control with fewer downsides so they can compete with private, centralised alternatives.

On the technological front, there are new models of decentralising control over online services. Blockchains and related technologies offer platforms new ways to distribute control of a platform to its users or to a chosen set of validators without the tendency towards fragmentation found in federation. These technologies are not yet mature and while Bitcoin was designed to be decentralised in theory, in practice it is anything but, as a handful of companies control essentially the entire Bitcoin infrastructure ecosystem. Still, the underlying technology, which is enabled by novel solutions to decades-old theoretical problems in computer science, is promising.

Organisationally, we need to innovate quickly to catch up to the quality of products delivered by today’s centralised service providers. We could learn a lot from the open-source software community. Many of the internet’s most important infrastructure components, from the LLVM compiler suite to the node.js application server, are controlled by non-profit foundations with varying degrees of democratic process around standards and governance. Taking this model to the governance and control of online services gives us essentially the platform cooperative model, where users and stakeholders democratically control a shared platform such as a communication network.

The role of the state

Catching up to privately-controlled monopoly platforms will be difficult given the head start they have already. The state should use its resources and capacity to spur the development and use of more democratic alternatives to the current crop of platforms.

Funding decentralisation research

Most research into blockchain technology is funded by private interests looking for financial applications. The technology has potential beyond finance, however. Governments should start funding research into using blockchain for secure online voting, logistics, and decentralised internet platforms.

Funding cooperative competitors to lean platform companies

Platform cooperatives may be a tool for distributing control of internet services, but cooperatives have to compete against venture-backed capitalist endeavours, where they are at a disadvantage. This is where the state has a role to play: if the services provided by lean platform companies like Uber are valuable, then the state should fund non-extractive cooperatively-owned and community-owned alternatives to them. Given that the state already owns public transit systems and that Labour has already backed plans for encouraging worker-ownership, cooperative ownership of platforms is a logical next step.

Even the playing field with regulations on platforms

Platform companies like Uber and Airbnb have made it clear that they are willing to go to the mat with municipal governments over their operations around the world. Thus far, governments have only extracted minor concessions from these platform providers in the form of taxes. Rather than haggling over tax rates, the state should aim to extract concessions that challenge the structural dominance of private platforms.

Specifically, the state can use regulation to limit the tendency of platforms to form monopolies through platform lock-in, the difficulty experienced by users who wish to switch to another platform. If your friends or photos or reviews are already on one platform, you are disincentivised from switching. Regulation should empower users by giving them more choices or by giving them more control over the choices they do have. Even if federated services have failed in practice, any platform of a certain size should be required to have substantive data portability features so users can extract their data and move to a competitor.

Regulation can also push platforms to publicly furnish data that is socially beneficial. Perhaps a transportation platform should only be allowed to operate in a municipality if it publishes data on traffic patterns and transit density. Keeping such data secret serves only private interests, whereas opening the data to the public would benefit society overall. Such platforms could also be pushed to operate analogously to common carriers in the telecom sector by opening their data stores and ride-matching services to competitors. This should go further than voluntary gestures by companies like Uber, which has begun to publish aggregate data, but only on their terms, and in a limited way that forbids use by competitors. It has already become cliche that data is “the new oil”, but unlike oil, data is nonrivalrous and can be copied and shared with competitors and the public at no additional cost.

Finally, the state should use regulation as a hedge against the negative effects of automation. Platforms that seek to automate away more and more labour could be required to contribute parts of their software to the commons under an open source license. If AI and automation technology were developed in the open, the benefits of automation would be more likely to accrue to the populace at large. Platforms could still form monopolies given the data they possess, but AI platforms could also be required to release their AI models to the commons.

Swinging back

Driven by the needs of capital, technological development in the software industry has been on a decades-long shift towards centralisation. While this trend has coincided with the explosion of new products and services, centralised internet architectures ultimately limit the choice and freedom of internet users, while also enabling dangerous levels of wealth concentration among a few corporations.

While old models of decentralisation have run their course, there are new opportunities for developing decentralised architectures and democratic organisational structures. Such efforts could achieve the twin goals of taking the internet out of the capital accumulation process and giving users meaningful democratic control over their experience.

Of course, this isn’t an inevitability; pendulums may swing back on their own, but reversing the decades-long trend toward centralisation requires reversing the neoliberal paradigm that enabled corporate capture of the internet. This, in turn, requires state action, and a broader shift in the political climate. Questioning the hegemony of today’s tech giants is the start; the next step is proposing concrete alternative structures for a future internet.


Jason Prado (@jasonpjason)

Jason is a software engineer with a decade of experience building communication platforms at large tech companies. He is also a volunteer with Tech Workers Coalition.


Do we really need a StateBook?

Corbyn’s digital speech is a good start, but we should look to redecentralise the web rather than build new social media monopolies.

We need a state-owned platform for the modern internet

It’s time to take the internet back from the American megacorps. But how should a nationalised version of the international internet look?

Do we really need a StateBook?

Corbyn’s digital speech is a good start, but we should look to redecentralise the web rather than build new social media monopolies.

We need a state-owned platform for the modern internet

It’s time to take the internet back from the American megacorps. But how should a nationalised version of the international internet look?