All of us here at Glitch were watching Apple’s WWDC announcements today with excitement. Product launches are always fun, and new operating systems and features are great. But our takeaway from what Apple just announced is so much more. It feels like a wonderful, familiar story of connecting to people, customers and non-customers alike, through the magic of open standards powering new realities. 

When I was a kid Apple computers were the monochromatic way I got to live the Oregon Trail in the computer room at school. I liked the rainbows in the logo, but it wasn’t until I first saw the internet almost 30 years ago that I really started to care about computers. 

In 2007, everything about the internet changed when Apple introduced the first iPhone. Steve Jobs got up and said “We’ve got a very sweet story,” and talked about how developers would be able to build apps for iOS with HTML. Since then the web’s only grown more central to our lives but those of us who want to to build full-featured apps for the iPhone with HTML have always had a wishlist of one more thing we were waiting for — until today. 

The Glitch community has been building installable web apps for years, taking advantage of the web features often called "progressive web apps" (PWAs) or web apps. Our Glitch-in-Bio app was a PWA from day one so maybe you’ve built one already! To celebrate this milestone from Apple we’re releasing a couple new tools the community can use to build installable apps: 

Hello Installable is a basic PWA starter that includes everything you need to build installable apps — including push support,

And what fun would push support be without a way to send notifications? Push Sender is a free and open push server that’s easy to set up and plug into your new app! 

Web apps aren’t the only exciting thing Apple talked about today. Their new VR headset, Vision Pro, looks amazing and has the potential to level-up the augmented reality experience which we are very excited about since WebXR developers are one of the biggest communities on Glitch, building VR and AR for the web. This is so cool because it’s an amazing example of the developer community defining a space, working with the W3C to define open standards for the immersive web, and really showing companies like Apple the potential of what they’re building. 

With iOS 17, Apple is making good on that original vision for web apps as first-class citizens in their app ecosystem. Mobile Safari is gaining increased support for web apps by implementing the web push API — which means developers can write web apps that install directly to your phone and can send notifications like every other app you’ve got. 

Because they’re built on open standards championed by the W3C, web apps already have wide support on other platforms — but Apple embracing PWAs on iOS means your app will just work everywhere. Web developers get access to things like device rotation, offline access, and even push notifications and the whole web wins. Open standards are how we connect the web, letting everyone participate no matter where they are or what kind of device they’re using.

We’re showing off lots of great WebXR apps from the community on glitch.com, but there’s no shortage of authoring tools either. A-Frame is one of the most intuitive ways to get started with WebXR and they maintain community starters on Glitch. You can remix and get started writing VR right in your HTML. And our friends at Needle have built tools that plug in to Blender and publish standards-based VR and AR scenes ready for the web — you can even publish straight to Glitch!

I still miss the rainbow in Apple’s logo, but when I see them embracing the open web and following amazing work from the developer community it reminds me why I was so excited about the web when I first saw it so many years ago: this is a place we all get to build together. And now it’s easier than ever to do that in a browser, on mobile devices, and in new realities. I can’t wait to see what the Glitch community builds with all this good news coming out of Cupertino. 

Celebrating an open app ecosystem with Apple and a new vision for the web

For most people building the internet today, the hardest part of coding is all the non-code barriers that stand in the way — from overly-complex technical requirements to exclusionary social and cultural factors that keep people from being able to create. This stands in contrast to the “no code” or low-code vision, which has often been based upon the idea that actual coding is the hard part, and should be hidden away or tucked behind a different UI.

But coding, done right, is a superpower. It’s not just a way of communicating with a computer or a compiler: it’s a way of communicating with a community, and of connecting to other people. The millions of people who’ve shown up to create on Glitch, and the millions more who’ve used the apps those creators have built, are testimony to the potential of that idea. I summarized some of this a few years ago in one of our earlier posts about Yes Code:

[A] Yes Code tool can create something unexpected, can talk to a legacy system that hasn't been updated in years, lets you try out a brand new framework or technology that you're discovering, and lets you reuse the billions of lines of code that already exist in the world.

And this fundamental, empowering idea of “Yes Code” matters more than ever now that AI’s transformation of the discipline of coding is rapidly accelerating. Now more than ever, the hard part of coding is not the code — it’s the connection to community.

As with all new technologies these days, the attention around AI and LLMs (large language models) and machine learning is a combination of genuinely interesting new capabilities and genuinely ridiculous hype. The exact ratio of value vs. hype depends a lot on your own personal perspective, but at Glitch we have a strong bias towards wanting to make choices that age well over time, and to make technical decisions that endure. 

From that standpoint, we do think AI assistance for coders is going to be an increasingly standard and pervasive part of coding workflows (I’m old enough to remember when we first got smart completion in our IDEs!) but we also think that the widespread accessibility of these tools will make more people into coders. Or put another way, it will make more people who are coders realize that they are coders. We’ve long said that anyone who ever made a really complex spreadsheet formula, or who hacked on a Neopets or MySpace website back in the day, or who installed an advanced Minecraft mod — they’re all coders! And a lot of people who have fallen out of the daily coding world, or who struggle to keep up with the relentless pace of advancement in things like JavaScript frameworks, will be more able to find the time to create when they have LLM tools to help them do it.

But what we’re not doing is presuming to choose your tools for you. We think there’s power, and value, in enabling our ecosystem to bring all of these new tools together, to compare and contrast their strengths and weaknesses, and especially to identify the things that AI tools are not good at, so we can help debug those problems collectively. For example, while a lot of commercial offerings in the coding AI space get a lot of attention, it’s just as important that open source or community-led tools be a vibrant part of the ecosystem — especially since we can see how they work, and know how their models have been trained. It’s really important that one or two companies don’t control the whole ecosystem, and communities of coders can be a powerful countermeasure that ensures that the AI markets stays competitive as it matures. Because we’re still in the early days of AI-assisted coding!

Recently, Clive Thompson (who’s long been augmenting his journalism with apps built on Glitch and other platforms), wrote “Why ChatGPT won’t replace coders just yet”, showing off how he built a todo-list app in 15 minutes using Glitch and ChatGPT. It’s exciting! But Clive reaches the conclusion that the splashy new AI tool ultimately serves to “help me quickly blast through boilerplate code so I can more quickly get to the tricky bits”. And that matches the experience we’ve heard from lots of coders — these new tools will accelerate experimentation by taking care of the drudgery, but they’re not nearly as good at the parts that have to do with innovation or the unexpected.

This is why community and connection and trust matter as much in the new world of AI-assisted coding as training models or prompt engineering. Teaching new coders, or teaching experienced coders about new tools, is going to require learning about intellectual property and permissions and how code can be ethically reused just as much as it’s going to be about the latest and greatest frameworks or programming languages. But these things can be a lot easier to figure out if you’re not doing it alone. And that goes back to the story we’ve been saying since the first day of the Glitch community, which just passed its six year anniversary since our launch: we make the internet better by lowering the barriers to creation, and letting everyone code together.

So I’ll close the same way I did a few years ago, when we first started publicly talking about the “yes code” idea:

There are cutting-edge areas like machine learning and bots and augmented reality that will never quite fit into a template, and there are millions of coders around the world who want to work together to put their unique coding superpowers to use for solving meaningful problems at work, at home, or in society.

We’re excited that Glitch is the place where so many are exploring these new tools together, and more than ever, we want to hear our community’s vision for how we all team up to make sure that every who creates with code is empowered by new AI tools to build a better internet.

“Yes Code” is how coding evolves with AI

We have some huge, exciting news for Glitch and the entire Glitch community: Today, Glitch is becoming part of Fastly! We’re joining forces with the Fastly team to bring the creative power of Glitch to even more people, and to help make the internet better by making it easier for everyone to create.

First, for our loyal community of Glitch users who’ve been with us for years: you’re good, we got you. Nothing changes about your apps or your Glitch account with this news, and the changes you’ll see in the future will be about making Glitch better and even more fun and rewarding to use going forward.

If you’re new to Glitch, you should know what we’re about — Glitch is a friendly community where millions of people come together to create, share and remix web apps for work, play and everything in between. 

So, why are we joining forces with Fastly? There are a couple of great reasons. It starts with having more support and resources to keep improving Glitch. We’re hiring for incredible people to come help with this work, starting today with engineering roles on the Developer Relations team, and watch for Glitch-focused support, community and engineering roles coming soon, too. And they’ll be in good company — every single member of the Glitch team is coming along to our new home at Fastly, to continue making the site and the community with the same spirit and care that they’ve shown over the last 5+ years.

Over time, we’ll be tapping into Fastly’s incredible technologies to make it easier than ever to make fast, secure, expressive apps. Today, Fastly is best known for delivering content on the web incredibly quickly (thus the name!) and securely. But we’re also super excited about their edge platform, which will let people build new kinds of apps. We’ll talk more about that once we’ve got some specific stuff to show you, but if you can imagine the cool stuff you’d build with that power, then you’ve probably got a good sense of where we’re headed.

Most fundamentally, we know that Fastly’s whole team really understands what makes Glitch special, and has believed from day one in empowering more people to make a better internet. Glitch will be one of the most prominent parts of a broader effort to give all developers, regardless of their background or level of experience, the power to build sites and apps that are second to none. A lot of the time, when a little company becomes part of a big company, we worry about losing the things that made the little company (and its not-so-little community) really work. In this case, we’re enthusiastic and optimistic about being able to tap into the resources of a much larger and more global company to help Glitch reach the full potential that we’ve always hoped it would. After teaming up with the Fastly team last year on a partnership to give Glitch users access to Fastly’s new edge platform, we realized that we were thinking in very similar ways about how to help the most people create the most cool things possible, and that set the course for the announcement that we’re excited to make today!

We’ll share more details and updates as we go along, but for now, a great way to get a taste of what’s to come is to try out this starter app and make your own.

An Exciting Leap Forward for Glitch

Over the years, we’ve done a lot of work to make sure apps and sites that are published on Glitch are trustworthy. That’s our responsibility as a platform, but it’s also good for our community, as it means your apps will be part of a healthy, trustworthy ecosystem. It’s been a while since we’ve shared details of this work, so we wanted to share a high-level overview of how we think about the challenge. We've got some wins to celebrate, some lessons learned, and a clear sense of where our future challenges lie.

TL;DR: Glitch intentionally makes app creation as easy and open as possible, which means opening up to more potential misuses of the platform, including phishing sites. By working with a community of security researchers and our own user community, we've been able to significantly improve both our response time and our proactive action on removing potentially harmful contents over the last year. Today, less than one half of one percent of new apps created on Glitch are flagged as being potentially harmful, with that statistic going down over time. And the mean time-to-resolution for reports of potential risk continues to improve, along with faster and more comprehensive suspension and blocking of users that create potentially harmful sites. We expect future challenges to come from phishing attempts that are built across multiple sites, and will be working with the other platforms in the ecosystem to address those threats as they arise.

First, let’s consider what problem we’re trying to solve. Glitch is designed to make it as easy as possible to create full-stack websites without any barriers. In any system, reducing barriers to creation means it’s easier for bad actors to abuse those systems to make malicious creations. In our particular case, Glitch allows for creating web pages or apps without even logging in, so reducing potential abuse is even trickier than platforms that rely on traditional authentication or login.

The next question that arises is what kind of untrustworthy content could be created on Glitch? The primary concern here is the creation of phishing sites that are designed to capture users’ credentials or personal information. We’ve used a series of layered approaches to counter this risk over the years, with each building on the prior efforts. (This is a high-level overview so our community knows our philosophy on handling the problem, not as an exhaustive technical review of all the efforts we’ve taken.)

We began with selective reduction in privileges for anonymous accounts, including limiting the automated creation of projects or content on Glitch. By introducing a CAPTCHA for app creation, we were able to greatly reduce the amount of projects that were created by bots or automated systems, without compromising the ability for users to make apps without logging in. (This is a common use case for students using Glitch in schools, for example.)

Within projects, we limited access to a very small subset of code libraries that were primarily being used in abusive or malicious ways, and added systems for scanning for known malicious content. Together, these efforts have been extremely effective in curtailing potentially harmful projects while having very little impact on legitimate users of the platform.

We’ve worked with outside security organizations to build automated scanning for malicious projects if any get past our internal controls, including partners like Phishlabs and Netcraft, and resources like openphish.com and urlscan.io. The research community that works diligently around these issues has been incredibly supportive of our efforts here, especially over the last year or so as we’ve begun tying their work more closely into our internal tooling for preventing or suspending malicious project creation.

Being a community-based platform, we’ve also made continual improvements to our community reporting functions as well. Every page of the Glitch.com site includes a “report abuse” button, and every report sent through that system receives a response within 24 business hours. For the vast majority of cases, user-submitted abuse reports cover sites that our internal tools have already flagged for review, which means we've often taken action even before letting the reporting user know that we've done so.

Finally, we’ve built tools to improve our response times to be even faster. We’ve always had a straightforward security reporting infrastructure with defined processes for handling security or trust issues that are reported, but thanks to the upgrades we’ve put in place, we’re able to act more quickly to take down any malicious projects, or even to take action on accounts which create malicious content.

These improvements have yielded huge improvements across the board. Having better support tools has actually helped all Glitch users, as our mean time to response for any support ticket has decreased by nearly 40% even as the volume of support requests has increased by almost 75%.

But specific to issues around malicious content on Glitch, we’ve found that, from a sampling of 50,000 projects created during November 2021, fewer than 0.3% were flagged as potentially harmful content by either our systems or the reports generated by our partners. We’re working to improve this further, but consider it a good sign of progress in addressing the issue.

With any security or trust domain, having success in one area always results in the battleground shifting a bit to new areas. For example, while we initially focused on malicious full-stack apps being the biggest area of concern on Glitch, what we’ve found with more recent reports is that bad actors are increasingly using static resources (sometimes static HTML, sometimes other document formats like PDF) to try to capture credentials, often with the actual submission of content pointing to other, larger providers like the giant cloud hosting platforms. These static resources are typically served as assets on Glitch, not as part of the core app hosting platform, so they send different signals from full-stack apps that might be potentially harmful.

We’ve worked with our partners to update our monitoring to accommodate these new vectors of attack, and initial results are promising, but what we take away from this learning is that we’ll increasingly see sophisticated attempts that go across multiple services, with Glitch (or part of Glitch’s infrastructure) just being an intermediary in a larger, networked effort. One recent such set of attempts was documented by DomainTools, which looked at these phishing attempts in depth, revealing both their attempted targets (businesses with employees working in the Middle East) as well as their intended mechanism of capturing data (compromised WordPress sites forwarding to an Outlook email address) that show the complexity of tracking these efforts across multiple services. While these patterns are well-established in the general web hosting industry, they're relatively new to Glitch as a community, and represent novel iterations on the pattern that will take time to fully prevent. We expect to have some advantages in this work over time by relying on the social aspects of our platform to help us better distinguish the small number of malicious actors on the platform from the millions of legitimate users.

Overall, we’ve been grateful for the support of our community, including both our users who report potentially harmful apps, as well as the security researchers who provide infrastructure for proactively stopping bad actors. Over time, we’ve gotten better and better at reducing the amount of potential risk on Glitch, both in the absolute and as a percentage of what gets created on the platform every day. We are firmly committed to keeping creation and remixing on Glitch as open and accessible as possible, which means we’re also committed to being responsible for stopping the inevitable misuses that come along with that openness. And we’re glad to do it, because it means that Glitch gets to keep being a great, trusted place for you to make the most creative, interesting, innovative, and useful stuff on the web.

Over the years, we’ve done a lot of work to make sure apps and sites that are published on Glitch are trustworthy. That’s our responsibility as a platform, but it’s also good for our community, as it means your apps will be part of a healthy, trustworthy ecosystem. It’s been a while since we’ve shared details of this work, so we wanted to share a high-level overview of how we think about the challenge

How we keep Glitch sites secure and trusted

We’re taking a break from our weekly round up of noteworthy apps to share a post written by Mads Hartmann, the Engineering Manager for our platform team. From time to time, we’ll post articles written by Glitch employees that give you a sneak peek into some of the behind-the-scenes work that goes into making Glitch go and grow. 

Over the past year, Mads has written a multi-part series about observability at Glitch and the tools and processes we’ve implemented to make the site more reliable. The most recent post goes into a deep dive on SLO-based alerts - challenges with our previous on-caller system and an excerpt of our internal tech spec. There’s also a getting started guide that’s a good primer if you’re not familiar with SLOs or want to learn more about what goes into their implementation.

“We’re hoping that by implementing SLOs - and alerting on them - we’ll be able to improve communication during incidents, reduce the toil on on-callers, and help improve our reliability in a way that’s meaningful to our users.”

Read the full post here. And if you want to work alongside talented and thoughtful engineers like Mads, we’re hiring. 

Adopting SLO-based alerting on Glitch

This past week has again made visible the centuries-old systems of violence that have been one of the ugliest and most painful parts of America's foundation. It's time to uproot those foundations.

There are pragmatic, specific, achievable actions we can take to end police violence. Police contract reform, demilitarization and non-police alternatives work and save lives. We encourage our community to join us in financially supporting Campaign Zero and their work to advocate for policy solutions to end police violence in America, which are grounded in these meaningful, effective tactics for stopping the bloodshed in our communities.

We support the Black Lives Matter movement, and offer our full support and and strength to our Black workers and community members who are deeply affected by this ongoing trauma. On top of these sentiments, we make specific requests to the leaders who regularly tell us they want to support companies like ours. While our team is remote and distributed, we are a company based in New York City, and so we begin our work in our own community.

Our goal is simple: we demand the end of police violence in our community. We categorically reject NYPD Commissioner Shea's blaming of violence on protestors, and the broader NYPD pattern of attributing violent policing actions to "protecting business interests". It is never valid to commit systemic violence against people, and we reject our work and our workers being used as an excuse to do so. We reject media characterizations and policing policies that prioritize objects or buildings over lives.

We call upon our Mayor Bill de Blasio to unequivocally take action on the reforms listed above before the end of 2020, and to commit to ending police violence. We ask our Public Advocate Jumaane Williams to prioritize advocacy around these reforms. We ask Borough President Gale Brewer to use her powers of fund allocation to defund programs which enable this kind of systemic violence. We ask District Attorney Cy Vance to end carceral abuses like cash bail and over-prosecution.

We ask New York Governor Andrew Cuomo to stop enabling violent policing through funding militarization of police and through enacting criminalizing actions like curfews, which are disproportionately and unfairly enforced against vulnerable communities, especially Black communities. We ask NY Attorney General Letitia James to continue her stated intent to prosecute policing abuses. We ask Senators Schumer and Gillibrand to support the specific actions above to reduce, and quickly end, police violence.

We commit to asking our leaders about their actions on these efforts each time they look to us to support their claims of supporting businesses, entrepreneurship, or innovation. There can be no opportunity without justice. Black Americans deserve accountability and justice from the systems meant to serve them. It should be enough for a single person to stand up and say "this violence is wrong, and these killings are wrong", but if it takes companies adding their voices, then we lend ours to amplifying the call to end police violence.

We remember George Floyd. We remember Breonna Taylor, Tony McDade, and Ahmaud Abery. We honor their lives, their names, their memories, and those of the far too many other people whose voices have been silenced.

Yassin Mohamed, Finan H. Berhe, Sean Reed, Steven Demarco Taylor, Breonna Taylor, Ariane McCree, Terrance Franklin, Miles Hall, Darius Tarver, William Green, Samuel David Mallard, Kwame Jones, De’von Bailey, Christopher Whitfield, Anthony Hill, De’Von Bailey, Eric Logan, Jamarion Robinson, Gregory Hill Jr, JaQuavion Slaton, Ryan Twyman, Brandon Webber, Jimmy Atchison, Willie McCoy, Emantic Fitzgerald Bradford J, D’ettrick Griffin, Jemel Roberson, DeAndre Ballard, Botham Shem Jean, Robert Lawrence White, Anthony Lamar Smith, Ramarley Graham, Manuel Loggins Jr, Trayvon Martin, Wendell Allen, Kendrec McDade, Larry Jackson Jr, Jonathan Ferrell, Jordan Baker, Victor White III, Dontre Hamilton, Eric Garner, John Crawford III, Michael Brown, Ezell Ford, Dante Parker, Kajieme Powell, Laquan McDonald, Akai Gurley, Tamir Rice, Rumain Brisbon, Jerame Reid, Charly Keunang, Tony Robinson, Walter Scott, Freddie Gray, Brendon Glenn, Samuel DuBose, Christian Taylor, Jamar Clark, Mario Woods, Quintonio LeGrier, Gregory Gunn, Akiel Denkins, Alton Sterling, Philando Castile, Terrence Sterling, Terence Crutcher, Keith Lamont Scott, Alfred Olango, Jordan Edwards, Stephon Clark, Danny Ray Thomas, DeJuan Guillory, Patrick Harmon, Jonathan Hart, Maurice Granton, Julius Johnson, Jamee Johnson, Michael Dean

Taking Action to Protect Black Lives

We published our look back at our year at Glitch, and I’m incredibly proud of what our team and community have accomplished! But we also encountered our share of challenges — some serious — over the course of 2019. The end of the year is a time of reflection, so I want to talk about those bugs we found, both to acknowledge the bugs we’ve found and share what we’re learning as we work on solving them.

All of this is in keeping with the spirit that underlies our company. When we picked the name “Glitch”, we wanted to emphasize the very human reality that things/people/processes are often imperfect - they have bugs or mistakes. But if we commit to working on them together, we can make great things.

The most visible challenge we had in serving our community this year is keeping up with scaling the technical infrastructure that makes all these millions of Glitch apps possible. Glitch uses an unusual system design (every Glitch app is its own, unique Docker container with a full Linux environment in it) that may not be obvious when simply browsing the site, but it does take some real heavy lifting to make it go.

A few times this year, that system didn’t run as smoothly as everyone would expect. Most notably, we had a fairly lengthy outage on November 4th that we documented here. During each of the times we faced technical issues like this over the course of this year, we’ve been really heartened to see the community be so supportive (so much #hugops!) and understanding about the challenges of scaling.

But if we are serious about Glitch becoming a part of every developer’s toolkit (see Connecting to Coders, below), we have to be a platform everyone can trust. We have to meet and exceed our community's expectations for stability and reliability. We've made a big investment on that front, while also focusing more on user-facing features that makes the creation experience on Glitch smoother and even more enjoyable.

2019 was also the year that many of our talented people chose to come work at Glitch—most of our team has been with the company less than 12 months! It’s been incredible to see our team grow by leaps and bounds and to find out everyone’s superpowers as they work together.

But as we rapidly grew our team, we failed to commensurately grow the processes and infrastructure necessary to support everyone properly. The result has been a lot of needless stress and tension and frustration. On its own, this is a significant problem, but when we’ve talked a lot about wanting to build a company that does these things better, a failure here is twice as painful for the people on our team who are affected. As the person who most often talks publicly about the positive ambitions we have here at Glitch, I’m also the person ultimately responsible for the times we haven’t delivered on those promises.

Underinvesting in HR and people infrastructure while a startup goes through a big period of growth is one of the most common, and egregious, mistakes. And no surprise, this has had the biggest negative impact on the areas where we were trying hardest to do something different — salary transparency, benefits, and supporting remote work. We are still deeply committed to getting these things right, and I’ve been working with our whole leadership team to make sure we get this fixed as quickly and completely as possible. Our next steps are bringing in experienced help immediately, reviewing where we’ve fallen short of our documented goals, and talking to every member of our team to gather their explicit feedback about expectations going forward. We’re also making hiring a head of people a top priority and are actively seeking out qualified candidates.

The third big challenge we’ve faced this year is making it clear and easy for every creator, from professional programmers to brand new coders, to understand how Glitch will fit into their workflow. We have a complicated product, that can be used in many different ways for many different things, and while people love our community and its creations, it’s not always obvious for a new user where to start and how to use Glitch. On top of this, there are so many stresses and distractions in the tech world right now that many people don’t know how a happy web community fits into the problems they’re trying to solve.

We’ll be addressing this by simplifying and sharpening our focus in 2020. None of our capabilities will go away, but we will make it far more obvious how a working coder can take advantage of Glitch’s power and simplicity. We’ll also strengthen Glitch as a platform and as a business by introducing paid services for the features you’ve told us you need most and find most valuable.

You can get an idea of how Glitch will fit into your workflow already with examples like the starter kits for Magic Leap, MapBox, Datasette, and integration into Visual Studio Code. Expect to see more of this in the new year, and if you run an open source project or offer an API to developers, get in touch and we’ll help you make it easier for any coder to get started.

As we’ve reflected back on both the biggest successes and the biggest challenges we faced in 2019, there’s something interesting that connects them both together. Everyone on our team at Glitch, and everyone in our community, is unified in believing in the vision of a creative community. Everyone is inspired by seeing what regular people all over the world are creating, and seeing a flourishing of a more human internet filled with things that surprise or delight us, or simply solve a meaningful problem. Even at our most stressed moments, we still believe deeply in the community we’re building on Glitch, and in the importance of enabling people to create a better internet for themselves.

There’s something really powerful about a mission that a community, and a team, can believe in even when the going gets tough. One of the underlying themes for all of Glitch is that we can make really amazing things together, especially if we’re able to fix our bugs in collaboration with others. We’re excited to do exactly that with all of you in 2020.

The Year in Glitches

For context, that’s more than Apple’s App Store! We mentioned crossing the 2.5 million mark earlier this year, and it’s clear the community on Glitch is creating the cutting edge of the web. We’re proud to enable creativity with code like no tool has done before.

You asked, and we delivered — Glitch for Visual Studio Code

You can now develop Glitch apps entirely using Visual Studio Code, the most widely used developer environment. Download the extension today on the Visual Studio Marketplace!

You helped us partner with MythBusters Jr.

Glitch teamed up with Adam Savage for the premiere of MythBusters Jr., a show where six of the nation’s most talented kids tackle a variety of scientific myths. We created companion Glitch apps for each episode, which take the experiments to a whole new level.

You helped us develop three new starter kits

Since our starter kits for React and website building were such a big hit, we made three new starter kits this year! Learn more about WebVR using A-Frame, build your first Discord bot, and then have some fun with our Glitch Party starter kit!

Anyone can build the web with Glitch, so we put that to the test at some of the Web’s biggest events this year! We brought Appy Hour to XOXO, got our remix on at Capitol Royale, and took questions with Mindy Kaling at this year’s Twilio Signal conference!

Function, our tech podcast hosted by Glitch CEO Anil Dash, returned for season 2 this year with a big question: can we trust the Internet? Anil talked with authors, advocates, and developers to uncover the answer.

You helped us make history with Revision Path

Revision Path became the first podcast to be included in the permanent collection of the Smithsonian’s National Museum of African American History and Culture -- a tremendously significant milestone!

We've been bootstrapped for almost 20 years, so the decision to raise funding wasn't one we took lightly. With this investment, we’re able to turn our words into policies and practices that are unprecedented for a company at our stage.

You kept us honest as we expanded our board member search to the community

We think we have a chance to make a fundamentally more innovative strategy and a dramatically more inclusive company culture by making our call for board members as open as everything else we do at our organization.

We've more than doubled in size and with that growth, we've learned a lot. There's more to say about what these lessons are, but we're going to be putting those into practice in 2020 with a renewed focus on making Glitch a great place for both users and employees alike.

Glitch Year in Review

Last week, on Monday the 4th of November, 2019, we had a seven hour outage. At 2.14pm UTC we deployed a new release to Glitch. Normally, we do this many times a day, without incident. In this instance the code we released broke our API servers, which resulted in the sitewide outage. During this time you were unable to access the Glitch site (which is a Glitch application) and your own Glitch projects.

As part of our normal process, we reverted the change and re-deployed. This however, did not result in the API servers recovering. After several hours of diagnosis, we identified a combination of issues, principally a crash loop in our API servers and very aggressive retry logic in our API proxies, that prevented the API servers from recovering. We re-deployed a new fleet of API proxies to address the issue, the API servers returned to normal, and the site began serving traffic again at 9.00pm UTC.

We’re deeply sorry this occurred and are grateful for the patience and support shown by the community. To help understand this outage and what we’ve learned from it, we want to share with you what happened and why it took so long to resolve.

At 2.14pm (UTC) a release was deployed to production. The triggering release was a substantial redesign of our logging module, called ng-logger. This is used by all of our backend services to aggregate and send telemetry to a number of different services. We believe the accidental introduction of a “deep clone” led to a depletion of memory, leaving our API servers unable to process requests.

The release led to an immediate drop in requests being served by our API proxies. Automated alerting via Pingdom and DataDog triggered a page at approximately 2.15pm, 1 minute into the deploy. The page triggers were Pingdom being unable to access the Glitch site, the API and a number of other key sites. After we detected the drop in requests, we began tracking the incident. Our Infrastructure team began to diagnose the problem and our Support team notified users of an issue.

We identified the immediate culprit as the recent deploy. We reverted the deploy and this new deployment was completed by 2.41pm. Unfortunately, rather than resolving the issue as we had hoped, the API servers remained locked.

We then began to diagnose the issue further. Our diagnosis was hindered by some gaps in our telemetry. We have begun to instrument all of our services with distributed traces but the process is not yet complete. This means we have limits in what we can observe outside of our API and API proxy servers.

During our diagnosis we identified several problems that we believe either contributed to the outage, or exacerbated the underlying outage. These included:

We have very aggressive retry logic in the API proxies, and some related systems, resulting in repeated attempts to contact the API servers. This kept the API servers overloaded and locked up.

A possible crash loop in the API service due to how we handle Sequelize timeouts.

Once we’d identified these issues, we determined that we needed some way of diverting traffic away from our API proxies to us to spin up a new fleet that wouldn’t become locked down immediately due to the flood of traffic. We made some changes to our load balancing configuration to allow us to create new fleets of API proxies that would start without receiving traffic.

Once these fleets were created, we began to send traffic to them and the API servers began to recover. This recovery began at 8.45pm and full site recovery was achieved at 9.00pm.

We learned a lot during this process. We conducted a blameless retrospective with the involvement of most of the product, design, and engineering teams, as well as a few other folks from across the company. It was our first “at scale” retrospective and we learned a lot about how we will run retrospectives in the future.

We also learned a lot about our incident management process. We’ve grown dramatically in the last year and some people on the team weren’t familiar with the process. Their first exposure to it was during this incident. We’ll be updating and familiarising everyone with the process going forward.

Our existing incident process also didn’t call for a “scribe” role to record the timeline of events concurrently with the incident, meaning we had to reconstruct events and data after the incident. We’ve updated our process to clearly define this role and ensure a timeline is initiated when the incident is declared.

We’re pleased our monitoring identified the issue in a very short period of time and that our (only partially-implemented) observability framework through Honeycomb allowed us to rapidly test and resolve many theories. It’s a priority for us to complete the deployment of this observability framework to additional parts of the platform to remove the current gaps in coverage. This lack of complete coverage resulted in some time as we followed false leads and leading to some frustration as we occasionally ran out of ideas of what to investigate.

We’ve identified we need to work on the retry logic and the timeouts of the API servers and proxies. We’re also going to be looking at our deploy process, which could be more resilient and better designed, especially to allow us to deploy in more staged manner. This might have allowed us to mitigate the issue before it hit the whole fleet.

This outage was upsetting and stressful for our users and for the team. But we learned a great deal and are correcting the issues that allowed the outage to occur and going deeper on our incident management, deploy and other processes to ensure future incidents are handled more efficiently. Thank you for your patience during the outage and we'll see you on Glitch.com!

Details of the Glitch outage on November 4th, 2019

He also serves on the boards of the Electronic Frontier Foundation and the Data & Society Research Institute, and was formerly a contributing editor of WIRED. He’s been blogging since 1999 and won’t stop telling people about that one time he was on the Desus and Mero late-night show.

Incidents

The Workers Who Keep the Web Safe in an Unsafe World

Animal Crossing Helped Me Get Back to Real Life

New Rules of the Classroom: Remote Teaching in the Age of Covid-19

Better Tech: Why We Need Calm Technology Now More Than Ever

Function with Anil Dash

Open World is a fiction anthology podcast about how science and technology serve as a backdrop for imagining a better future, helping us envision the positive influence that science and tech can play in a real future. 

Open World is produced by Keisha "TK" Dutes and hosted by critically-acclaimed podcaster Rose Eveleth, for Glitch. By speaking to the pressing concerns of our times through love, friendship, community, compassion, and sustainability, the host will help listeners imagine new ways forward into a future that is less scary and more positive, and something we can start working on constructively.  

Open World launches weekly episodes in Spring 2020. Want to contact the show? Email youropenworld@glitch.com.

Open World

Shift Shift Forward is a podcast that showcases everything that makes Glitch the best place to create on the web. We’ll talk to some of our top creators, explore cool apps, and even hear from the people who work at Glitch!

Shift Shift Forward

From idea to code in seconds, build and learn alongside the most creative coders on the Web

Company News