Episode 13 | One Hotspot Away From Disaster

Paul: Cool as cucumber. Let's just do it from my hotspot. And the whole keynote was streamed through a, uh, from a phone hotspot.

Jen: Uh, oh. Hey everyone, I'm Jen Santos and this is Mayhem and Method, where event pros tell the real story of what happened and what it reveals about how events actually work. Hi there, this is Jen Santos, and welcome back to another episode of Mayhem and Method. I am here today with a former colleague of mine, Paul Singh. We've worked together on some events in the past, and he is here to share his mayhem stories with us today. Paul, thank you so much for joining me.

Paul: Of course, thank you for having me today.

Jen: Yeah, are you an airline loyalty person, a hotel loyalty person, both, or neither?

Paul: Uh, I would say I'm an airline loyalty person. I'm a Delta Alaska only sort of person. Yeah, I get really, like, really into getting my miles and sometimes it annoys my colleagues, but I'm just that person. I like my miles. I do too. Hotel-wise, I feel like Marriott is where I have some connection, gets me good rates. But other than that, when I'm traveling on personal, I'm avoiding the typical hotels because in events you get to so many hotels.

Jen: Yeah.

Paul: It's gone so bad that after 12 years in events, when I go to stay in Vegas, sometimes I just pack more formal clothes than I should have because it's my personal vacation. So I wore those.

Jen: Oh, I like that. Yeah, I'm definitely an airline person. I'm actually starting to look at like my miles for the year and see how close I am. I'm on United, how close I am to platinum status. because I need to retain Platinum status because of my international travels, right? And so I'm like, okay, like, am I going to go to IMAX or not? Do I need the miles? Like, literally, like, that's where I am in my, like, airline planning cycle for the year.

Paul: I love that. So you're actively selecting events based on how many miles I get?

Jen: I do.

Paul: I do indeed. You know what people should do? Like, I've never seen events pairing up with, like, large conferences They should say that if you register, you get extra miles as a perk.

Jen: Would they be credited flight miles, right? Because I don't need miles, miles per se, right? Because I have more miles than I can spend. But what I need is credited flight miles for status. Like, could an event provide that? I don't know if they could.

Paul: Why not? Why not? Like, I feel like you could— when you— a lot of times, right, Uber provides that for Delta. Like, there's some promotion going, but I've never seen that promotion with a conference, but they're huge brands. I can totally do it. But yeah, I don't know.

Jen: No, that would be awesome. I mean, imagine if you put on an event for event planners, like so many of these industry events are, and then you said yes, and you're gonna earn extra 5,000 qualified flight miles. Sign me up. Yeah, I'm there.

Paul: I mean, if it's for event planners, like, who knows, even like Delta, United might just even want to sponsor that. Yeah.

Jen: Interesting. I like it. I would love to hear a mayhem story that you have for today. Take a couple of minutes and tell our listeners what was it that happened on site. And then after we hear about that, we'll take a little bit of time and we'll kind of unpack what happened before what happened happened. Because none of these things happen in a vacuum.

Paul: That's so true. None of these things happen in a vacuum. I have a couple of stories. I think one of my favorite ones happens to be Moscone. Um, we had a 10,000-people conference and suddenly the Wi-Fi goes down, which is pretty common for venues, but we didn't have a backup and the keynote had to be streamed.

Jen: Oh geez.

Paul: So now the executive team— and it's like a huge company, it's like a Fortune 20 company, and keynote is getting streamed. There's millions of people going to watch it. The clock is ticking. There was no way out. And of course the badges can't be printed. So there's line queuing up here. So I'm running here, running down. And, uh, the thing that worked was someone just— event planner, I just have so much respect for them. They're cool as cucumber. Let's just do it from my hotspot. And the whole keynote was streamed through a, uh, from a phone hotspot.

Jen: Uh, oh, I'm gonna need a minute on this one. Okay. Um, how did that— I like— well, sorry, let me process this for a minute.

Paul: Um, yeah, yeah.

Jen: What was the streaming quality like? Was it okay?

Paul: Yeah, it was 4K. It was great.

Jen: Okay, fabulous. Somebody had a really good cell plan.

Paul: Yeah, yeah. And the funny bit is we had Verizon in the room. Their execs were getting yelled at because how can you do this? We like, because we were speaking to them like a week prior, like, hey, ensure Wi-Fi is dialed in, ensure Wi-Fi is dialed in. And the funny bit is the hotspot was from Verizon itself. So it's just like, such a weird angle.

Jen: Yeah. Yeah. Okay. So let's talk about badge printing though, because if the Wi-Fi— so you're talking about the venue, venue Wi-Fi as a whole was down, right?

Paul: Yes.

Jen: And so what did you do? Because you have a keynote starting, you conceptually have thousands of people backed up there waiting to get their badges. So did you like send out digital badges via email? Did you just say, okay, everybody just come in and we'll sort it out later? Like what happened?

Paul: No, I mean, we had to make people wait a little bit, and I think once the hotspot started working, then our reg desk staff just started connecting hotspots and started printing badges, and we were able to clear the queue. I think the only other— also the problem was Moscone is not dialed in for a lot of traffic the right way.

Jen: Yeah.

Paul: Because it's just right in the middle of that street, so That's why the crowd was even getting more and more. So we just, we just waited, then didn't take any action on it, to be honest.

Jen: Yeah. So did the keynote end up starting late as a result?

Paul: No, because a lot of people were checked in. So, but we still had about a couple of hundred people waiting outside. So keynote had to start.

Jen: Right.

Paul: And when the Wi-Fi was not there, and now execs are talking to execs at Verizon, and it's just nothing is working out, this event planner just like pulls her phone and goes, why don't we just hotspot? And everybody's like, it's not gonna work. It's like, did you try it? Let's just try it.

Jen: Yeah.

Paul: And they checked the bandwidth and everything, and it's just like, cool. So that's why, like, event planners— I have so much respect for event planners. Like, no matter what situation you give, they just handle it.

Jen: Wow. So what did the AV team say? Like, yep, we'll take your phone, let's just— I mean, how? Because usually there's like 8,000 miles of wire, right? Cable?

Paul: Yeah. AV team was like, no, uh, this wouldn't work, this won't work. But once they checked the bandwidth, it was the bandwidth— it was pulling the bandwidth that they needed. Uh, they had to shut down some other streams. For example, if the stream didn't go on some other social media, so they prioritized 2 platforms versus 6 platforms.

Jen: Right.

Paul: So it had to take a little hit, but that was more aspirational, the other 4. There's not a lot of traffic there anyway, but these 2 were pulling the main traffic for millions of people, so they had to push it there.

Jen: Wow, that's just wild. And then, so then to get the badges, obviously you're not gonna put, or maybe not obviously, you're not gonna put the badges on the same Wi-Fi as the keynote was streaming. So did like a bunch of other people just pull up their phones and start hotspotting, or?

Paul: Yeah, because they were different, uh, how should I say it? Now this is where it gets a little technical and above my head, but, uh, there were different versions of Wi-Fi, but the main line was the same.

Jen: Okay.

Paul: So they were bandwidth restricted, but the whole thing went down, right? So nothing worked. Uh, but on the, on the flip side, when people saw, like, the moment the keynote story happened We were in the war room. The whole war room was clapping behind the scenes. And then we just dashed out to the reg desk outside. We were like, guys, just start putting it on your hotspot. Right.

Jen: Right.

Paul: And that's all we did at that point. Wow.

Jen: So tell me then about the postmortem that happened. I mean, somebody— there had to be words other than executives yelling at each other at the keynote. Like, what happened after that?

Paul: There were some organizational changes at the company. and at some of the other companies too, which I cannot disclose, but there were some org changes that have happened.

Jen: Okay, yeah, that would make sense. I mean, I don't think, I take it back, I have been at a tier 1 where the Wi-Fi didn't work. A whole bunch of brand new Cisco routers that didn't decide they wanted to play that time. Brand new, they'd never been tested and they did not work. But yeah, usually, I'm used to though these tier 1 events, right? They go, they bring their team in and they like lay their own Wi-Fi, Like 10,000 people using Moscone's actual Wi-Fi was really surprising to me when you said that.

Paul: And that, and that was like a little bit in the postmortem. Like we were like, well, one, you should, someone should have figured out for the backup. So venue Wi-Fi, we all know keynote should get its own dedicated Wi-Fi. And also like AV team should have had all these dialed in.

Jen: Right.

Paul: So a lot of touchpoints got missed there.

Jen: Yeah, yeah. And even if the Wi-Fi had gone down, there should have been a backup, right? Particularly if you're going to rely on house Wi-Fi, you have to have a plan B. You need to have a satellite, you need to have— right? I mean, there's so many ways that this could have gone differently.

Paul: Especially now, I feel like a lot more event planners are just relying on Starlink a lot.

Jen: Yeah, yeah, I hear that too. I think the, the tier 1 I did most recently, I think that that was our backup. We ran everything, it was at Moscone as well. We ran everything and then the backup was definitely Starlink. A little bit slower, sure. But does it go out? Also sure. So speaking about the Wi-Fi going down, and I'd mentioned my client with the Cisco router, so same client. And I remember one time we were sitting on site and I was like, in our, into the website, right? And I go to log in and I'm like, I click the button and I'm like, that should have just worked and it didn't. And I click it again and I'm like, it's got like this weird hiccup to it. And I'd been working with this client for quite a while and I'm like, something's not right here. And so I message over to the architect. I'm like, dude, can you check this out? 'Cause something's not right. He's like, Jen, you're imagining it. And I'm like, and I keep going back and I'm testing it over and over. And I'm like, you know how you like, I call it this, our event sense. You know something is wrong, but you don't know what's wrong quite yet. And so I'm trying it like another 15, 30 minutes, and I'm like, no, I'm like, I really need you to look because I really think something is wrong. And I got lucky in that I somehow managed to get like an error. Oh, I managed to get a URL to pop up for me that was not one of our normal URLs, and I got— or something like that. Oh no, it was Network Tools. I had the Network Tools open. I'm trying this over and over and over again, right? I'm like— So I grabbed the screenshot of the Network Tools and I sent it over to him. He's like, oh, Yeah, there's something here. And so he gets on with the security team, and it turns out that we were undergoing a denial of service attack. Like somebody was trying to take the site down during our keynote, right? And then they were going to be like, hey, look who we made look bad. So yeah, like this internet security and Wi-Fi stuff, I mean, it's not, it's not a joke.

Paul: Yeah, I mean, when I was in on the Ventec side, like we had a lot of 'Cause the problem is when you're working with these big brands, a lot of hackers don't realize that their websites are not necessarily their website. They're using an event tech platform.

Jen: Yep.

Paul: So we get so much, like we have all the protocols every event tech tool does today.

Jen: Totally.

Paul: But the attacks just keep coming wild. And like when you are a multi-billion dollar company, you have the infrastructure to handle it. handle it better. But when it comes down to smaller event tech companies, we have to manage it very differently. And it happens way more, I think, in the last 3 years than I've seen before.

Jen: Yeah.

Paul: Interestingly, one of the customers was also— this was a couple of years ago— one of the customers, this was in Vegas. I landed, I went to my hotel, And then I'm collecting my team like, okay guys, let's prepare for the show tomorrow. Everything is going to be great. Everything was dialed in. I suddenly get a call that, Paul, we think we might get attacked. I'm like, okay, what does that mean? How can I help?

Jen: Right.

Paul: Well, we need a full team of DevOps. We need a full— I'm like, okay, well, what changed? Well, Yeah. There were some brand-to-brand competition piece that was going on in the background between those couple of companies. So now since they have their flagship conference, some executive in an executive meeting thought, what if we get attacked? We need extra security. Even though, like, what I couldn't tell the customer is that the other customer is also on a platform. It's like they can't hack themselves, but—

Jen: Right.

Paul: So I had to fly a team of 4 DevOps people at 1:00 AM and it had to be done 'cause next day they wanted in the war room people with their laptops checking the servers. But it doesn't really matter 'cause they can check that from the work from home. It's the same thing. Like everybody's on Slack. It's the same thing. But I guess sometimes what execs want is what execs want.

Jen: Right. Yeah. And there's something really reassuring, right? As an exec, you're like, ooh, I can see the team here, they're keeping an eye on things, you know? And it's just, it's that I kind of understand it. Like there really is a difference when you're allowed to be on site somewhere and you can kind of feel the vibe for what's going on. And then I imagine just like the safety, the sense of safety you get from knowing that people really are looking and it's not just people, yeah, we're on top of it. As they're sitting on the by the poolside or something, right?

Paul: Yeah. No, it definitely helps, and and that actually helped a little bit more because same executive group at the same time the next day wanted to change the agenda and add sessions into everybody's schedule at the lunch hour.

Jen: Of course they did.

Paul: Yeah. Like okay, how? Well, we can't change it on the backend right now. So, but the session had to be added to 7,000 attendees' agenda in like 10 minutes.

Jen: Oh man.

Paul: But having that DevOps person was like, he was able to run some local script and figure that piece out, which was great. But yeah, I guess what execs want, execs want, I guess.

Jen: Yeah. I went to add an exec and he was like, he was like, He was in the, like, the show was actually happening and he was like making like game time decisions on like reorganizing sessions and everything. And so without telling the event tech team. And so like, and we're like, guys, I'm like, we don't have the right people at the right places. We're not streaming the right sessions now. And he's like, well, it's just, it's just bits and bytes. Just move them around. And we're like, dude, this, this isn't how that works.

Paul: Oh my God. Oh my God.

Jen: That, that, that was a really, really, really bad event for me. I am serious trauma after that. I refused to work on that show after that. I'm like, I'm not doing that again. I will not.

Paul: So what did you guys do? Did you make the changes last minute?

Jen: Well, he made the changes and then we kept finding out after the session was starting because it wasn't the session that we thought was starting. And then we had attendee uproar, as you can imagine. And we actually ended up having to like literally, 'cause this was, this was, um, a virtual event. We ended up having to replay the second day of the event the following day again so people could attend the right stuff.

Paul: Oh my God. Way to lose your audience. That's crazy.

Jen: That's crazy.

Paul: I mean, I'm glad it was not an in-person event, 'cause could you imagine like the session check-in would have been disaster.

Jen: It totally would have been totally disastrous. Yeah. No, no, no, no, no. So yours was an in-person story though, wasn't it?

Paul: Yeah. Yeah. I mean, I guess that's where like sometimes having extra tech is helpful in that regard.

Jen: Yeah. Yeah. No. And I think that's why like when our clients don't want onsite tech support, right. Or they don't account to have anybody on call because they don't want, they don't have the budget for it. They don't want to pay for it. Right. And then it just, it makes it really hard for us as event tech professionals to provide them with what we want to provide them because we aren't being paid, even if we're like salaried or something, right? You're an employee and you're working for an event tech provider. If you're not being paid to be dedicated to that event the week that it's happening, and if you're not staffed that way, you might be off in another meeting, you might be doing something else. And it's like, I'm sorry your stuff blew up. I really, like, I truly am. I feel really terrible that this is what happened. And decisions have consequences. And I don't say that to be cruel. It's just the reality that if you want this, if you want this level of support, then you need to be willing to pay for this level of support or understand that things may happen. Yeah.

Paul: I mean, like, if you, if you flip the script, like for their own business, it's the same thing, right? They're not, they have their account managers and customer service reps and no one is dedicated to an account unless you pay for it.

Jen: Right.

Paul: It's interesting how the urgency— I also feel like there is a— I don't want to say disconnect, but there is a level of understanding that you get from the event team and marketing team. But when the exec is on the top, yeah, then it goes left and right. And that sometimes also allows to unlock budgets. Like the example I gave you where I had the team at 1:00 AM.

Jen: Right.

Paul: If they would've told me that they need the team, the DevOps team a week prior, I would've probably built them something different 'cause I didn't have to uproot people from actively working on a product and now actively flying to Vegas. But then when at that point when they just need it, then they don't think about the budget sometimes, which is also very interesting 'cause if I would've asked, If I would have recommended the same thing a couple of weeks ago, it would have been denied from a budget perspective.

Jen: Yeah, exactly. No, we don't have, we don't have $20,000, but we'll happily pay $50,000 at the last minute, right? Yeah. I mean, yeah, when we think about our corporate clients, right, we think about like the people that are living the day to day, and then we go and we stop and we kind of think about our executives and how they tend to be, even yet today in like 2026, there is still kind of this disconnect. between what the exact level expectations are for an event and what the costs are and the planning that goes into it. And of course we can just move stuff around, it's not that big a deal. And like what it actually takes to deliver, there seems to be a fairly significant disconnect on that.

Paul: Yeah, I mean, that's the other reason why I feel like event jobs are like still top 5, the most stressful jobs in America.

Jen: Yeah, yeah.

Paul: Like anyone who runs events, it's just crazy how much pressure they are consistently in.

Jen: Totally. And the fact that we love it, right? And the fact that we haven't yet run away screaming, I think says something about us. I'm not quite sure what, but it definitely says something. So Paul, this has been a really good conversation with you. I have enjoyed riffing on cybersecurity and Wi-Fi and all things event tech. And I'd love to have you take a few minutes and tell my listeners where they can learn more about you and connect with you and anything you'd like to share.

Paul: Of course. Thanks, Jen, and thank you for having me today. So after 12 years in event tech, leading and building teams for Cvent, then Swugo, then Zettle, I think what's interesting is I was able to sit on both sides of the aisle. Being in event industry and also being a go-to-market leader. And I started seeing a lot of pain where event attendees don't get follow-up. So based on that, I built an AI conversational layer recently. It's called Helen. What it does is it's an AI concierge for conference attendees. And at the same time, when you have field events, it follows up to ensure the room is full because everybody hates when you're spending $30,000, $50,000 for a dinner and attendees don't show up. So it covers that. Or if they walk up to your booth, the AI automatically can follow up with them. This is one of those jobs where sales reps and marketing, they don't— both of them don't want to do it, but the consequences are instrumental if the follow-ups are not done because then you lose those prospects. Right.

Jen: Right. Which has a direct impact on event ROI. 100%.

Paul: So, so that's what I've been working with. We are in beta right now and you can find more information on helina.ai or follow me on LinkedIn.

Jen: That's awesome. That is super, super cool. Real quick question for you. Like, what is your ideal kind of event size? Do you have one in mind for your client size? For Helina?

Paul: Yeah, for Helina. I think for Helina, anyone that has a conference that's over 2,000 people, I think that's a good size or where you need to give a premium experience to your attendees. And if it's a field event, I think anyone who's spending money on prospect dinners, games, and needs to ensure the room is full or has an exhibition booth, I think this directly applies to them. They're paying like $5,000 a month to do this and still nobody wants to do the job. So this is one AI conversational layer that covers both sides.

Jen: That's super cool. I really, really like that. Well, thank you for sharing. Thank you for being here today talking a little bit about Helena, and I look forward to chatting with you again soon. Thanks, Paul.

Paul: Likewise. Thanks, Jen. Thank you for having me.

Jen: Thanks for listening in to another roller coaster someone else rode so you don't have to. If you enjoyed this conversation, please take a moment to rate, review, subscribe, and share so other event pros can find the show. Until next time, tame the mayhem. AudioTocracy.

‍