Even SCARIER Web Dev Nightmares (Spooky Stories Pt. 2)

Welcome to Syntax.

We've got our second episode of twenty twenty five spooky stories. These are stories of web development and software development in general of just awful things that have happened in your career.

You've taken down websites. We've got some real doozies, coming up. Just just awful stories, and we are going to read them. If you have your own spooky story, please submit it at any point. Go to syntax auto fam forward slash spooky, and, you can you can submit your story to us, and we'll read it on next year's episode. We're always collecting them and always love hearing them.

So we have a century plug in one of these stories, so I'm gonna leave it for that. Is that you think that's alright, Scott? It's actually a good one. I want to read the ad.

I want to read the ad. So if you didn't catch the the last Node, if you're listening on audio, Scott has, like, a very good vampire costume.

Real blood. Why thank you. I was doing this costume, and my wife said, do you have a plan? And I said, no.

I don't. I said, what is this? I'm losing the accent. I don't have a plan.

And so I just started mashing the face paint around, and, what you see here is what you get. And you know what? I did a pretty good job, actually. You look great.

Thank you.

Alright. Let's get in on the first Scott story JS a bug from beyond the grave. By the way, I wrote all the titles for these ones, so please appreciate them.

Once, I built a website for a foundation that did medical research.

People could donate money for special occasions like birthdays, but mostly in memory of somebody who has passed away. That's nice. Everything worked perfectly at launch, and I even tested it myself. The payment showed up as reserved on my card. All good. You know? Like, you you it reserves the money in your card. It it hasn't actually gone through. Sometimes you need to do that. That's a part of credit card processing.

Six months oh my gosh.

Six months later, I'm at my summer cabin with a beer in hand and the grill going Wes the phone rings.

None of the donations have come through.

A donator, was complaining that their donation never came through.

Strange.

After some panicking and digging through, all the test suites were still passing. I realized I'd done everything right except one tiny thing. I never hit capture.

The payment provider, thankfully, out of business today, had a special endpoint for capturing the payments. TypeScript is exact same way, by the way. Auto capture was optional, but not default and, very hurried in the documentation.

So all those donations in memory for dead people have only been authorized but never actually charged. Oh, no.

Oh, so people have been donating for months to honor loved ones, and the money never moved.

Total panic. Imagine your entire business is taking money for people.

You don't do the one thing supposed to do. But luckily, the payment gateway kept pending transactions for six months. It had been five months in twenty days.

No.

I wrote a TypeScript to capture everything and sent out an email to everyone affected. In the end, it all worked out.

Spooky.

Oh.

Oh, this Node, because it involves the the spirit of lost ones, I am going to say this one is a nine out of 10. That is very spooky to me. Oh, man. I'm gonna give that one a nine out of nine and a half. That was

just the fact that it not only is awful and includes money, but also includes, like, something sensitive, which is donating memory of loved ones. Yep.

Yeah. Next one's about the NHL, though. Maybe those make us feel better.

Oh, Scott the NHL? Go grab my hockey stick I got over there. I got a hockey stick in my office, Wes.

Oh.

Yes. The next one here is NHL spookiness.

I built a very nice website for the National Hockey League.

A whole team of content specialists, about 10 people, start filling the site with content.

The day before launch, the whole team even starts working from six in the morning for their last big push of content.

I export a single table from our staging site, not content for a special feature, to SQL and blindly import it in the production database. Oh, blindly import it in the production database. What a sentence. The whole site goes down, and my new table is left.

Only my new table is left.

Oh, man. It turns out the c dot SQL export didn't just contain a create table query, but also a drop or all drop table queries.

That sounds like an AI, style migration if you're asking me. That's, oh, jeez. So they they ran a SQL command that just dropped all the tables and then added their new table. The latest database table was from three days before, so all of the content was gone. Yes. The launch was delayed. Yes. I was ashamed.

Yes. I always check my exported Scott SQL files from now on.

This is now more than ten years ago, and I still get chills thinking about it. Oh, there's no backup strategy here. They what? They everybody had to reenter their stuff or what? It's just gone? Probably.

Yeah. That's awful.

If that was me, I'd just be, like, constantly backing that thing up. You know? Just every every 20 minutes backing it up because that's JS very spooky to have that happen, especially when it's such a big like, the NHL. You know? Like, there's 10 people working on it. This is that's that's a lot of money in in paying people's salaries.

Yeah. I mean, like, brutal.

I mean, I want a seven out of 10 spooky spooky bookies.

Spooky bookies. I'm gonna give it five Jason masks out of seven Jason masks.

He did wear a hockey goalie mask if you don't know about Jason.

I never know with you, Wes. So the next one is entitled white space ghost faced.

I'm a full stack dev that writes mostly TypeScript, of course, of course, like any good full stack dev. I'm a seasoned staff with lots of experience writing back end code, managing infrastructure, writing Helm charts, setting up alerts, yada yada yada. So I'm not afraid to jump into some Python code Bos and make some changes. This sounds like something I would say. I've been writing a lot of Python lately. The Python service I've been working on was originally written by a messy ex Google engineer, and I'm stuck with them Shame. And a broom cleaning up the mess. A typical Google employee writing a bunch of messy Python.

This particularly messy Python service was hitting Google's cloud video generation model. Okay. So this is relatively new. So I was cleaning up some of the messes. Every time video creation fails, multiline logs are barfed up over the standard out in using a print statement instead of our structured logger.

So I updated the main dot py file in our code Bos to use our logger instead of print. So doing so required moving around a few things, etcetera. Makes sense? I opened up a PR. I ping warp trusted front end dev to rubber stamp it, and logs are pretty, and people can create VO videos through the API. Everything is good. Alright.

Three weeks go by.

That's a long time, folks. That's a long time for this the the just wait part. And I see a group of people crowded by a laptop on my way to the office, and the CFO is standing by them looking moderately perturbed.

They are wondering why our VO two Scott last month had shot up five x.

Certainly nothing to do with my innocuously logging change. Right? Well, time to find out.

My little cleanup PR introduced the line that unindented the message acknowledgments outside of the try catch block, resulting in failed video generations.

Example, failed due to safety checks. Yeah. Like, you try to create video with something that is is not appropriate, they'll they'll fail it, to be tried retried five times. Just for reference, a thirty second video generation on Google costs about $10.

Oh, yeah. It's it's an expensive API, so this literally had cost the company hundreds of thousands of dollars. The crazy part is it was just a white space change. It didn't even show up on our code review platform, which is Graphite. So the dev I pnpm rubber stamped the PR, didn't even notice the change. Freaking Python with its indentation. I used to be such an indentation fan. You know? I used Pug and Stylus and CoffeeScript, and everything was indentation based. Same. Yeah.

I liked it. No more. I liked it.

I don't know if I'll still like it, but I liked it at the time. That was the style then. I would blame him. I would blame graphite, but, ultimately, I have to be forthcoming and put my ego aside in such situations JS there is only one person to blame here, the Python programming language and its effing syntactic white space.

He queued it up to blame himself. He just says it's the Python problem. So, yeah, lots of yapping to say Node invisible line change caused JS a hundreds of thousands of dollars to the Google abyss. Thankfully, my colleagues were quite forgiving, and I'm still working hard to redeem myself in the finance department. I've definitely been given a shorter leash on the head of engineering.

Yeah.

Oh, you gotta have some alerts set up or something if things were to to go awry.

Oh, bro.

Yeah. I get that I get that one about four Dracula fangs of which I'm going to be taking out so that way I don't have permanent damage done to my lips here.

That is not the most expensive goof up we've had.

We've had a couple stories in the past that were in the millions of dollars, because they they shipped out physical products Yeah. To people who did not pay for them.

But that is still a set seven oops out of eight, I think.

You think seven oops out of eight? Okay. Next one here, over order nightmare.

I had been working at a premium pet food ecommerce Scott.

To give you an idea, we average about 4,500 orders a day with about 300 of that amount were subscription orders.

Am I subscribed to some dog food? Maybe I'm a customer.

The customer had an option to start a subscription of their pet food orders. This would create an order on the interval the customer set him or herself. However, the person that created the whole subscription system went into a burnout and eventually left the company.

I joined when that person, was already burnt out. Let me tell you, working in reoccurring billing without, like, tools to do it for you can absolutely drive you to burnout. Absolutely. Billing.

What's the worst? The can we can we ask this? What's the worst parts to work in? I'm gonna say Recurring bill. Sign on. Recurring billing. That. Single sign on and maybe auth in general.

Access control?

I don't mind access control either. I think those are the Bos. Anything that involves kind of, like, ping ponging back and forth. But, like, with reoccurring billing, like, testing it, you have to be it has to be done through webhook so you can send a test webhook. And then, like, when an actual subscription, like, webhook comes in, yeah, you could have tested it, whatever, but it's still, like, it's it's still always very scary for for, like, what actually is is doing that. There was a big demand of the customers that they could choose which day of the week they would receive their food JS hybrid remote work was very popular in our country. They wanted to receive their food when they worked from home. This would be for regular orders and subscription orders. For example, for regular orders, you could choose that you would receive it the first Thursday after placing the order. This got really complex as we sold it to about 20 countries. Oh, yeah. Ho ho ho.

We work together with different shipping companies depending on the warehouse that order came in from. These shipping companies all had different lead times for each country.

Eventually, we worked this out and thinking about the implementation of the existing subscriptions.

For running subscriptions, we had to calculate what was the last day of the week they received their last order. Then we calculated the next shipping date so that it would receive on the same day. But with the given subscription interval calculated into it, we had to keep in mind the subscriptions were running so that we could need to calculate the time from their last order until now and subtract that from the interval they had specified.

We then took the closest day calculated from the next shipment date and would then change the shipment date for the existing subscription. Oh my god. Bro, I I hate subscriptions with digital products. With shipping and then shipping dates that that things have to arrive in, take me out back. I'm done. Man. Yeah. Logistics.

I don't like logistics. I'll tell you about that. I'm not good at planning.

Jeez. That's crazy. At the time, I had been there for about a year. The subscription system was a bit of a gray box for everyone, where we could change some things, but nobody on the team had full context on how it all worked exactly, and there was no documentation.

As I said earlier, the original creator had left the company. I had been Node bugs in the system at my time there and felt confident to implement this new feature.

I took some days to get familiar with the system and wrote out exactly how I would change it in the areas that would be touched. From running subscriptions, I wrote a migration strategy that discussed everything with the team and my manager.

Everybody gave the green light. We tested the implementation.

We put this into production. Everything ran smoothly, and we were getting positive feedback from our customers. Oh, man. The hammer's about to drop here. But after ten days, we were getting complaints. People who had subscriptions and had their next order delivered in these days had been paying every day since the feature launch.

That's that's the hang on. I knew I was waiting for it. They've been paying every day.

Turns out, we had been calculating the next order date wrong. And once the new date was calculated, the next order date was placed on the current date as this is a running system and had been going for ten days before we noticed.

Thousands of extra orders created, packed, and shipped.

Oh, no.

And it's it's not even like you're getting, like like, a hard drive or something fun. It's like like dog food.

It's so huge.

I've been I'll take 10 extra orders of dog food because you know they're gonna refund it at this point. Luckily, I created a backup of the database when I launched the feature into production so we could see who had upcoming orders, and we then had to spend the next week with the whole team figuring out which customers had placed how many extra orders to reimburse them all. Oh, fuck. This I give this one, nine demon pups out of 10 demon pnpm.

Brutal.

That is and, you know, like, the poor support people as Wes, they're just gonna be going through it for the next, like, weeks or months and, like, probably lost a bunch of money on the dog food. And I I would be mad if I got all that extra dog food because, like, where do you put that? You know? I guess you can give it to a, like, donate it to a shelter or something, but, oof. Hopefully, that's the silver lining here. Some shelter got a bunch of free dog food.

By the way, we keep all of these anonymous.

This one includes the company name, so I'm just going to change the company name for the sake of anon anonymity.

Anomanimity.

Anomanimity.

The year was 2019. I was on the experimentation platform team at the time, but was given responsibility over everything analytics after being told it should be in the keep the lights on mode, which is minimal development. Just keep it running. The service API was based on ESLint IO in order to leverage Alaskian specific wrappers.

I changed the name.

Wrappers around the clients.

Alaskian developers had been given a halt order for feature development after being plagued with reliability issues in the front end that devs were flying blind to. It was decided that the best way to give front end quick visibility into performance and reliability was to leverage the analytics JS in order to capture as many events as possible. This is pretty common when you're sending events.

You'll throw them in, like, a variable or you'll throw them in local storage so that if there is if they're blocked by, like, a a thing, if if the person goes offline for whatever reason, at least they're in local storage and you can retry over and over again. This uses multiple local storage entries to manage the queues for processing, and to be able to recover queues from tabs that have been closed. That's another good reason for it. We started getting reports from internal users that local storage was filling up, and it was due to the analytics client.

After digging around, I found that the client would detect that local storage JS full and switch the entire storage solution to memory, which prevented any further cleanup inside of local storage. Once we got that fixed out, we pushed it to all products only to to start causing more instability to our back end analytics service.

So it took us a while to realize that we would occasionally be seeing single analytics event, be able to be duplicated tens of thousands of times, which was also causing additional slowdowns on the other clients. It was a hard case for us to get budget to add more fixes for a service that should have been in maintenance Node, but the volume it was needed for processing grew from 1,000,000,000 today to 3,000,000,000 with the influx of additional monitoring front end teams that were trying to be shoehorned in. Yeah. This is this is monstrous. Like, we we had folks on the past from Century when it's just like like, you're essentially getting DDoS. Like, being able to just receive the events from everybody is a massive, massive thing.

Soon, it became apparent that the fix was causing more slowdowns on client browsers and also sometimes prevented garbage collection from running due to the frequency of local storage checks. This is really interesting because local storage is a a synchronous API. So every time you're touching it, you're you're actually running a synchronous function, which if you do it often enough, can start blocking the main event loop. In testing, I even managed to crash my entire laptop.

I was thinking Wes should do a video of who can crash the browser tab first.

Who can write the worst code to crash a browser tab? I think that would be good. Crash your browser tab in

seconds, milliseconds. I can crash Really? Faster. Oh. You know how to do it? You just do it. I mean, just use use effect. Just use use effect. Use effect.

We tried a few hacks to get on top of this, but at the end of the day, we warp trying to treat local storage like a transactional database and became very clear that only one thing we could do is move the functionality to IndexedDB, which removed a lot of complexity and overhead, and almost all of the issues went away after that, except for the fact that many devs kept throwing more and more load at it because the analytics pipeline turned into a hammer for many problems, but that's a different story.

Moral of the story, don't treat local storage like a transactional database. Yes. Yeah. As much as especially, like, you can you can run out. You can run out of space in local storage. And every time you write to it, it only holds strings, so you gotta there's, like, a JSON parse, you gotta pay for there.

And a second story here is the analytics is not designed for platform monitoring. I think there's another tool for that, like ...century.io.

Why, thank you for the the ad there, dear friend. I don't know what your name is, but, yes, you guys should be using Century at century.i0 forward slash syntax and get two months for free using the coupon code Sanity treat.

Yeah. Take a bite out of your errors.

That's good. PS, the main problem, was the Vercel alliance on the analytics problem kept growing despite being in maintenance mode, and it hit a peak of 13,000,000,000 events a day before dedicated team was assigned to rein in the cost.

Oof. I can't imagine the bill for that, man.

B with billion.

Billion.

PPS.

I'm no longer Tolinski.

To this day, there is still some code that will attempt to reclaim events from local storage despite no new analytics being added to local storage since the start of 2021.

Oh, man.

Man, man. I I gave it a nine and a half out of 11. That was good.

Yeah.

Yeah. I would I would say that is I would say that is a nine out of, 10 great white sharks that are

infected with some kind of zombie disease. So they're Zeke in the chat just said that's a 150,000 events per second.

Oh. What? It's a big that's a big company, Alaskian.

I think they are have every person in Alaska as a customer.

They a lot of them do go skiing for sure. Yes. Alright. The next one, Rack Spaced Out.

I'm gonna get a little Arnold there.

Ivan, the dreaded r m r f forward slash star when I meant to run the rm-rf./star.

Never.

Never do this.

On the production server of a large commercial real estate company.

I didn't use Git or any other version control.

Backups had been offline for years.

The server was completely trashed, but the Rackspace tech was entertained. Can you imagine getting this email? They they'll be like, hey, brother. I just, ran r m r f on my entire, server.

23 year old me was thoroughly impressed that the dude stayed on the phone with me for hours, ended up spinning up a whole new dedicated server and moving over what pieces he could to the new box. I lost a few days of data in the end, but got most of it back up. Well, that's that's a lucky, end result.

One time I ran git clean on my root directory.

I did that one time. I think I talked about it in one of these spooky episodes. And then, I I deleted a bunch of, video production files I had going and kinda borked my system, but it didn't I I noticed what I was doing, so I I was, really quick on the command c, but I still it's still,

chomped through some stuff either way. Very smooth. Everybody Deno right now and check that your backups are working because, man, they often are not. And when you need them, it's not the time to find out they Yarn not working. And, also, like, shout out to, like, support engineers or, like, the technical support staff that actually know what they're doing.

Yeah. Because they're able to Century has a really good support team. I've been to the Toronto office several times, and you just, like, listen in on them trying to help everybody with all their problems, and they know about everything. It's crazy.

Yeah. Next one's called fired.

Before I worked as a web dev, I worked in IT doing desktop network and Vercel support. I had a job working for a financial company in San Francisco.

My job was to prepare laptops and other hardware for new fire new fires new hires. The process involved wiping the ORM images and loading custom images on my colleague and I maintained depending on the laptop or in rare case, desktops. My job was very routine and robotic. I loved it. One day, however, the CEO's laptop was brought in for servicing and definitely not a wipe.

Sadly Mhmm.

In the routine of my dry job, I formatted a CEO's hard drive. So, Drew, you do something every single day.

Oh. We have a backup of his laptop before, but it was not one of the CEO's laptop that was backed up. I was looking for work the next day.

Oh, that sucks.

That sucks so much.

Oh, man. CEO's hard drive.

I can't think of, like, a Wes thing to make a impression with. Ugh.

Man. But, like, I don't know if these big companies can do this, but, like, I have rolling backups running in multiple places all the time because this is bound to happen at one point.

Yeah. Yeah. I mean, you gotta have a backup. The IT people should have been making sure that that CEO had backups or whatever, but still, very spooky. Yeah. I'm getting fired for something. Very spooky. I would say that this is a, you know, a six out of eight killer snowman for sure. I would give the same as well.

Killer snowman.

Killer snowman. Jack Frost's, I believe. I don't that's a movie, but I haven't seen it. WordPress woes. I was doing some freelance work for WordPress when I was on a hot streak with quite a few jobs done at all five star reviews until this job.

It was actually pretty simple. It was just a WooCommerce site that needed some minor changes, so I did what I always did and created a backup, a staging site, etcetera, and got to work.

I completed everything, uploaded onto staging, and then got the client to overview it with a green check mark.

Awesome. They were using SiteGround, so I was able to just hit the sync button, merge the staging sites to production, forgetting that this was an ecommerce site.

I just overrode the database entries for all of the last few days of purchases.

There was no way to tell who had placed the orders or when or for what.

Suffice to say, the client was very upset, and I still got lucky. I still got away with a three star review even though it should have been a lot worse. I would imagine what they would have to do is, like, go through amounts and, like, Wes and then contact the customers and be like, our dumbass web developer

screwed the pooch here. What did you order? Man. Yeah. That there's no way to fix that but manually. We've had this this many times, the story of someone who did it, and they've had to, like, rebuild everything from, like, emails sent out. Yes. Hopefully, you can go into your your sent emails or logs or something like that. But just that's always a tricky one, especially, like it's not a WordPress specific one, but a lot of people hit this issue where I have config that needs to change in the database. You know? Yeah. But I also have content that lives in the remote database.

And sometimes you bring the remote one locally and sometimes vice versa, and you gotta be very careful

doing these migrations. I always found that to be very scary when you have that much can like, with WordPress or Drupal, like, doing migrations in that way where the database is such an essential part of the whole thing. I yeah. There was, like, a Drupal features and ways that you would, commit things and whatever to get things into Git, and it was just, like, always very scary when you did any kind of deployment process. Another WordPress woah. One time, I was working on converting an old shop site.

This guy submitted, like, ten ten WordPress horror stories. Like, he never learned from his thing because the guy wanted his own site. I was doing everything in WP ESLint to speed things up because there was tons of data. Well, I didn't realize that WP CLI was going to trigger the WooCommerce email system as I was migrating the data over. It sent out emails to people about their purchases.

There weren't actual charges made it. Thanks goodness. Just retriggering of email notifications.

And the guy I was doing the job woke up to over 18,000 emails from angry and confused customers.

Luckily, I was able to salvage our relationship and get the job done with a five star review.

This guy's got five stars.

And the site is still running to this day five years later. It made me think a little more carefully. Well, apparently Scott because the next one is from the same guy.

But, like, if you worked with somebody and they really screwed the pooch, but then they fixed it, would that give you confidence in working with them in the future?

Well, you know what they say, you know, fool me once, shame on you. Fool me twice, you you can't be fooled again.

But I don't think they say that.

Did you ever hear that? That's a George Bush quote. He, like, he forgets what he's supposed to say, so he's like, you can't get fooled again.

So this is like it's a it's a train wreck.

But, yeah, I give it one time. One time, you give him a a a break, and then two times

yeah. And Jay Node said something about that. Well, the third time, you got the next time.

Yes. I was working for a small WordPress based company when we had a client who was thinking of hiring us on retainer.

So we said we would spend a lot of time to look at their site and their project to get up in up to speed and discuss what we could offer them. One of the big things they needed was help with Wes they approached us was dealing with Stripe.

The way it was set up was not correctly doing something. I don't remember why exactly.

So we asked if they had any custom code within the site dealing with Stripe, and they told us no.

When I pull up the site to my local machine and spun up in my dev environment at the time, and I start to look through some things, well, it turns out that they were wrong about custom Stripe stuff, and there was some code that automatically ran a bunch of webhook stuff when loaded locally.

All of their customers, around 5,000, got a second charge to their credit card for the month Node matter how recently it was paid.

All in all, they lost about 500 customers by the time we talked to them the next afternoon. We were not hired on a retainer, and we learned some very valuable lessons that day. Didn't even get hired, and they lost 500 customers for that's bad day at the office.

You know what? They probably had the Stripe API keys in the database, and he pulled the production database locally.

Yeah. Holy don't put API keys in your I guess, in some cases, you have to. But, man, that sucks. A 10% loss of all their customers.

And on top of that, how many of those 5,000 people that got the second charge, like, filed a, like, a chargeback? Because that's $15 out of your pocket every like, regardless. If if if you the way that Stripe works is if you can refund it before they do a chargeback Chargeback. Then you're in good shape.

But if you don't if they file a chargeback and then you immediately refund it, you still pay that $15 even if you you win or regardless. So, like, I don't know, 5,000 people, maybe of 1% of them charged back, that's still $750

out of your pocket. If you wanna get pissed off people your customers, Scott, messing with their money. Yeah.

What does the p in VPS stand for? This is a really quick one. I switched my VPS provider and forgot to point my portfolio to the new IP address.

How do you do that? A month later, a friend texted me saying that my portfolio was turned into an adult website.

Ah. How can you not check it? Dude.

Oh, I guess that IP address was reused, and it just never changed where it was pointing to. That's great. Yeah. One of the first agencies I worked for, we were doing WordPress sites. And the only site I'd ever worked on, it wasn't even my my client, but it had gotten hacked somehow. I don't know how.

And that's that's what had happened to it. They just replaced every URL with a bunch of redirects to adult sites. And, it was a crazy day at the office. I'll tell you that.

Never never do

that. Beyond the Grave two.

When I first joined my company as a junior developer, I was testing out an issue with one of our alert systems. By tagging a very popular artist to my test event, I managed to send everyone a push notification stating that artist was going to be performing in their area. Oh, no.

The only issue being that that artist had died a few months prior. Oh, no.

That's a much bigger oh, no than I had initially given that.

Oh my god.

Luckily, I work for a nice company. No blame was placed on me, and the new guardrails were put in place before tagging deceased artists and test events dispatching real alerts.

Bro, that's two now that have dealt with people who have passed away. That is very spooky.

That I give that one ten ghosts out of 10. People talk about, like, vibe coding and building an app in an afternoon.

This is the stuff that you're not be told about, how these companies have, unfortunately, sent push notifications about a dead artist to those people. And now there's some code in their code Bos that checks if an artist is deceased or not before it sends events. The hottest Scott fix. I once worked on new features that introduce API calls to a somewhat complex web component.

From everything my team tested, it worked great. We thought we had accounted for everything, so we push it up to our repo, get it approved, publish the version, and get it installed in our front end applications.

Fast forward a smidge. The reports start to roll in that tests in the front end application are failing and holding up CI and CD.

The common thread amongst the reports was the test suite can't find any elements. I don't pay much mind because we ran a colossal Cypress test suite for a monolithic repro, so that could be anything.

And that kind of thing has happened before. It's so true. Your Cypress tests are failing.

Almost nobody goes, oh, must be actual error. Your initial thoughts are just like, probably Cypress is acting up.

Then I see the error.

Crap. This is our API call failing in their local dev environment because we didn't add a proxy or dummy data. We forgot to add a failure Scott. And because we didn't catch the error, it's throwing at Webpack's red screen of death, which blocks Cypress from querying anything on the page. We rushed out a failure state, which solves a problem in our main development branches.

Nope. The hotfix branch is running into the same issue. Let's get that fixed out. Oh, crap. They were weren't supposed to get the apps, so they did a hotfix on, not on their production, but on their, I guess, their main branch, which is not yet released. So they pushed out a bunch of code that was not meant to be pushed out yet. Suffice to say, it was a comedy of errors that took days to fix. And if we had Century ESLint our build system, we likely wouldn't have run into this this issue. Thank you, century.century.i0/ Syntaxi's coupon code tasty treat. Hope you enjoyed the spooky story.

Oh, that sucks. That sucks. I'm gonna give that one a a one out of no. A two out of three.

Yeah.

Yeah. I'll give it I'll give it a a a five out of a 14 Legos on the floor.

Bad redirect. In a company I worked where everything was basically in shambles and barely making ends meet, a coworker thought it was funny to somehow magically redirect hackers to an adult site when he thought something was of malicious intent. The boss made a demo for the company that was playing during the Deno. He was redirected to the adult site several times. It was a miracle we didn't lose the contract and that he didn't get fired.

You sent your you sent your boss to an adult site.

Don't put adult sites in your work. Don't do that.

Ever

Never.

Ever put a even put a swear in. I was having some frustrations with my recording the other day. I was sent, like, a screen recording to some someone, and I sent it to Randy, our producer JS well. And the name of the screen recording was f this, and guess what they saw? Never never put squares or any adult stuff or or even, like, think that you understand. Because, also, a lot of this, like, trying to catch people doing bad stuff,

you're going to eventually catch somebody who's not doing bad stuff for whatever reason. Don't do that. Yeah. That's not a good idea. You can you can just send them to a blank page. That's just as easy. And they're like, no harm, no foul. You can trap the you can do honeypots. You can do all that stuff, and you can still annoy them by I had a band system on LevelUp that would, like, act as if they weren't logged ESLint the site. So then they would go to try to log in again, but then the login would fail because they were already logged ESLint the site. Yeah. Like and that way, like, if, if for some reason, someone ever got accidentally stuck into the that banned system, it would just be like, hey. I'm trying to log in. The login says this. Oh, let me fix it for you. You must have tried, five different credit cards 40 different times, and the system failed. You Node? Like, you must have hit these specific criteria, but, like, don't send them to an adult site. That's bad bad idea.

Next Node. Instead of making money, you spend money. So back when I started web development, I was working for one of those sketchy cell phone game ringtone wallpaper companies, and we're billing the victim I mean, customer via premium SMS messages. That that's how it used to be. You want a ringtone, you you send pnpm SMS.

For development and testing purposes, we had regular SMS numbers that we use that didn't bill our cell phone. Okay. So you had, like, a you have a premium one, which you make money, and you have a regular SMS, which you actually you pay to send. One morning, our CEO called us into a meeting to give us a bad news that there wouldn't be a bonus that month as the SMS provider had screwed us over. Instead of paying out the bumper sale, they were billing us for excessive use of the free SMS.

I'm sure you had already guessed what happened. So at a moment, I had to confess, I expected to be written up for having left the dev mode enabled in production, but I had an amazing CEO that just said, make sure it doesn't happen again and left it at that.

Like, you cost not just the company, but your coworkers real money by leaving a a dev flag in production.

That is three bones out of three for me. That sucks.

Three bones out of three? I'll do eight cobwebs out of 10 cobwebs.

Certbot's certain death.

How to lose hundreds of hours of meeting recordings in one easy step.

Just let certbot fail silently and wait for the SSL to expire.

We record our Zoom meetings through their RTMP streams.

All of these streams land on our loyal ingest server, which is tied to a domain name and secured with a Let's Encrypt SSL certificate.

Like responsible grown up adults, we even let certbot to handle renewals automatically.

Yeah. That's that's fine.

Then one day during the mysterious deployment, probably a Friday, certbot stopped working. No alerts, no warnings, just quiet little failure sitting there waiting.

Days later, when the SSL certificate finally expired, boom.

Every single meeting stream started failing.

And how did we find out? Not through fancy monitoring or alerts.

An employee casually said, hey. That little streaming icon isn't showing up anymore. Translation, congratulations.

You've lost recordings.

A round of chaotic debugging later, we discovered the issue. The server was alive, but the SSL was not. We renewed the cert. The streams returned, and we immediately begged the leadership to approve buying proper SSL certificate with a longer validity because apparently on certbot alone is a bit like relying on Goldfish to guard your house. Moral of the story, if certbot fails quietly, it's not the failure that hurts. It's the moment SSL expires that breaks your soul. What a wonderfully written, spooky story, Val. Thank you for crafting that.

That's why I just use a system, a host or something that uses certbot behind the scenes or whatever to manage the SSLs. I don't wanna do cert Bos myself, but I certainly don't wanna pay for, SSL certificate in 2025.

I'm just gonna use whatever my host has got baked in because every host in the world's got that now. Who's not giving you SSL renewals for free? I actually just looked it up. Let's Encrypt

used to email you when your certs expired.

But JS of January 2025, they've stopped that. So they won't even you you'd need, like, a a downtime, like a like a ping like a pinging service would Ping service for sure. Easy fix that for you. Alright. Next one. It's always DNS.

Is this about AWS?

Probably.

Was that DNS? Actually, it's about Bluehost.

Oh, classic.

It's always about DNS. I had been debating about staying with Bluehost or moving over to Netlify for my small business website. I was tired of paying for hosting and figured a free Netlify plan was sufficient for what I needed. I mean, my site was pretty simple, just for portfolio, some case studies, and why was they paying a $120 a year to Bluehost when Netlify could host it for free? WordPress JS overkill for what I needed. Now I should mention, I'm a designer who dabbles in front end development.

I can build and style it, make it look gorgeous, and but deployments and server side stuff, that's not really my wheelhouse, and I'll usually work with other developers that would handle those things. Still, I watched a few YouTube tutorials in Netlify, and everybody made it look so easy. How hard could it be? Right? So one Saturday morning in the air of confidence and iced coffee, I decided to cancel my hosting and begin migrating to Netlify.

I signed up for Netlify, connected my GitHub repo, and watched my site deploy in minutes. It felt like magic. Then came the DNS settings. Now I know DNS was important JS we all do, but when I logged into my domain registrar and I saw that wall of cryptic records, a records, MX records, c name, TXT SPF, my brain just went, yeah. I got this. Shouldn't be too hard.

I updated my name servers to point to Netlify as their documentation walked through and watched, the little propagation, may take twenty four to forty eight hours pop up, and I felt like a tech genius. Is this really what back end devs deal with? Super easy. JS a self certificate check, working. I gave myself a little victory fist pump.

What I didn't realize is I had removed my email, which was running through Google Workspace, and those MX records I casually deleted, those were the only things telling the Internet where my email lived.

For two weeks, it felt like months. I'm sitting there thinking, wow. Business is slow today. Nobody's reached out. Guess it's a quiet week. Meanwhile, my email service is completely nonfunctional, and every email sent to me is bouncing into the void. Clients trying to send me project files, potential leads, and time sensitive Wes, all gone, never received. I get that way sometimes where I turn on do not disturb for the entire day, and then I go, wow. Nice nice quiet day today.

Like, oh, shit.

Do not disturb on all day. Yeah. Hey. I've been trying to email you all week. Are you getting my messages? They keep bouncing back. The Sanity, like, that set in. Oh, no. What? But that's where it gets worse. I had no idea how to fix it. Remember, I didn't know back end network DNS stuff, so I did what any panic designer does. I started googling.

I spent the next two days brother, you could've you could've DM'd me, and I would've fixed it for you in, like, ten seconds.

Oh, going down the most frustrating documentation rabbit hole of my life. I've read Netlify's DNS documentation, which would tell me to check my email provider settings. So I go to Google Workspace documentation, which would tell me to configure my DNS records with my hosting provider. So I go back to Netlify docs. I was stuck in an infinite loop. I even Wes chat g p t and gave it to me back and forth in the same flow.

I watched YouTube tutorials, read hundreds of Reddit thread, all with different variations of how to fix DNS records, Netlify Google Workspace.

I finally pieced together enough enough information to understand the need to manually add Google's MX records back into the Netlify DNS settings. I copied those magic strings of text from Google support page. I pasted them into Netlify. I waited for what seemed like another agonizing twenty four hours for propagation, and then I finally emailed myself.

It started working again. You don't even need to wait twenty four hours there. You Node me. Yeah. There's, like, two things with DNS JS Wes you're changing the name servers, you're literally changing who manages your DNS. If you can change it from provider to provider or you can just use your registrar to change the DNS. You Node, that's what I do. I put them all in Cloudflare, and then I just point them at whichever servers you actually need because that way you don't have downtime. But, also, like, when you import a domain name into Cloudflare, it scans the domain name for as many records as it can because you could accidentally be bringing down so much stuff more than email if you don't know about it. That's the fun part. I craft the most embarrassing sorry for the technical difficulty email I've ever written and sending it to every client I could think of. I had to do damage control. I apologize perversely and basically admit that, yes, the UX guy broke his own website user experience.

Talk about updating my services to we offer development, but only front end development.

Oh, I actually did I did that once where I switched over somebody's name servers, and I thought I had taken their email down, but it turns out that I didn't. But I was panicking for, like, probably an hour. Yeah. It sucks.

It does suck. Why has DNS propagation gotten faster? We used to have to wait twenty four hours. Like, what's changed there? I don't know. How does the Internet work? You know? There's there are name servers around the world, and Right. When one of them has changed, they have to let the other ones know.

And I've had it in the past where updates instantly for me, but then you there's there's, like, a website you can visit where it will ping your website from, like like, hundreds of servers around the world and tell you where it's pointing to, and you can see if it's updated. So, like, just such a common thing. And the the infrastructure between all of these peers, as I understand it, has gotten much better, so things things update much more quickly.

Cash ruins everything around me. Cream get the money. Dollar dollar bill, y'all.

First day on the job was for a small startup serving hundreds of thousands of users daily.

I picked up an easy ticket, updated some dependencies, ran fine locally, ran the Wes. CI was fine. PR was approved and merged. I headed out for lunch with the CEO and founders who were in the office that day.

Oh, nice little day at the office.

Mid conversation, his phone starts buzzing. He checks it and goes pale.

Site's down. White screen.

My stomach drops. That two line PR? No way.

Turns out, our monorepo had the same package at two slightly different minor versions locally and in CI.

Yarn's cache handled it fine, but Vercel's build didn't use the cache.

White screen for a thousand users.

The CEO stepped away to make some calls. I pulled out my phone, jumped into the dashboard, and rolled back the deployment myself.

Twelve minutes of downtime.

My second PR, implementing sync pack lint to the CI to catch version mismatches.

Oh, yeah.

Build cache, bro.

People always get annoyed by, like, lock files in pull Wes, or or sometimes they'll just be like yeah. Yeah. I'll do it myself. I'll just delete the thing and reinstall it if there's, like, a, like, a thing. But what's happening is that, like, there's a possibility that eight levels down your package tree, there's some package that got bumped from one point zero point one to one point zero point two, and and it introduces a major bug in your system. I've I've had it myself once in the past, and that's where lock files are are very, very important.

Node. Yep. Next one JS called a fiber f up. I ripped out and destroyed the ventilation system at a US mall along with a $100,000 in equipment from a single retry.

A number of years ago, I was working as an embedded engineer for Internet of Things products.

This specific product is designed to guide sunlight into building. It consists of a large array of approximately eight inch lenses and a big stack of fiber wire, led into the building as special fixtures, which is lamps that scatter the sunlight in the room. That's crazy.

So it's sunlight coming from outside being piped through fiber and then into lamps. So, like, they're lighting the mall without any electricity.

It's that easy. Sun hits a lens, travels through the fiber wire, and lights up the room. The entire okay. He just explained how it works. Yes. The entire lens sorry. I didn't read this whole thing because I was like, this is gonna be good. The entire lens array is programmed to follow the sun with fractions of a degrees of accuracy.

Everything in these projects is expensive, because of the insane tolerances required for good light, but the fiber wire especially needs to be custom made. Normal fiber for Internet audio, etcetera, do not work for visible light. They are made to transmit signal frequency data, not broad spectrum light. Interesting.

Node of the first of these devices was installed in a mall in The US. The fiber wire was simply fed down into the mall through the air ducts. The code had validation checks for everything, including valid encoder entries. That's the position of the motor. Right? For the motor that spun the entire array of lenses. If any value was wrong, the application would just stop.

Last minute, we make these are always the best. Half of these stories Yarn, last minute, we made a separate decision. The operating system should just restart any application that crashed.

Well, unfortunately, one of the encoders turned out to be faulty. The application would start up, validate everything, start to spin the device, realize the motor wasn't turning, and crash. Then it would start up, spin for a bit, and crash. For an additional context, due to the insane precision needed, the motor was mounted on a one to 65,000 gear. It took thirty minutes to rotate a 180 degrees in full speed. It had crazy torque.

It took a few days before the thick it took a few days before the thick fiber wires finally ripped out large chunks of the ventilation system in multiple rooms and then finally snapping in half.

Somehow, I made those two decisions all by myself, and I did not make the connection.

This guy caused physical damage in the magnitude of hundreds of thousands of dollars.

Physical damage.

One little decision to reboot.

Man, that's like, a fail safe. Fail safes, man, are crazy when especially when they're real world where they could like, I often think about, like, self driving fail safes and all of the code that needs to be written for that.

I think, unfortunately, for humanity, I I don't know how many of these companies are gonna put fail safes over, you know, the dollar. So we'll see how that works out once that all because there there was, like, some something where, like, a electric car got pulled over, and it's like the car Yarn can't be held at fault. So, like, what are they gonna do? Like, what what are they gonna do when something crazy happens with electric cars? I better not hear a dev horror story about a No. I know. Or right. That's true. Self driving car ride. Wes decided it's okay to hit dogs as long as you're not killing a human.

As long as the dog is 15 years old, we don't mind. The dog was on its way out anyways.

We didn't account for the fact that the API could be down and the gas would never let up.

I did one indentation, and the thing drove itself into the ocean. Uh-huh. Please don't We gave our AI

access to the, self driving code and had it rewrite it.

Wow. Well, we had more stories than than we actually needed, but please keep them coming. Syntax.fm4/spooky.

We'll throw the ones that we didn't read today onto the onto the schedule for next year because there are some brutal ones that have have yet to come.

Yes.

Brutal.

Yes.

What a wonderful and spooky spooky stories episode, Wes. Thank you so much, for making this happen live. I thought that was a really neat little titch you did. People Yarn watching on podcast. We ended up streaming this live kind of randomly on Twitter. Scott of fun, folks. I hope you enjoyed yourself if you're watching live, if you're listening to the episode in your car, if you're watching on YouTube, we do this every single year. So if you like these dev horror stories, there are some millions and millions of dollar, cost horror stories in the previous spooky stories episodes.

So, folks, if if you enjoyed these and you're looking for more bone chilling tales, check out the spooky stories episodes on Syntax.

There are a ton of them. I don't even know what we've done. One, two, three, four, five, six, seven. We've done, like, seven years of them, and we've done at least two or three every single year. So there's there's at least Like, 16 or 17. Yeah.

Yeah. Including these two, probably 20. So check them out. Tons and tons of spooky stories. If you have a spooky story, submit it on our web our website and text Scott of him, and, we might read it next year. So shout out.

Peace.

Peace.