Palworld server costs near $500K per month as network engineer is ordered to 'never let the service go down no matter what'

nanoUFO@sh.itjust.worksmod to Games@sh.itjust.works – 193 points –
Palworld server costs near $500K per month as network engineer is ordered to 'never let the service go down no matter what'
pcgamer.com
43

well this is probably PR as there is no such system nor it can be made that can have 100% uptime. not talking about the fact that network engineers rarely work with servers :)

Not 100% but 99.9%... IIRC Guild Wars 2 servers had like 1 actual outage in 11 years. They have pretty amazing structure.

Fun fact, uptime goals are measured in nines -- for example, 99.9% is three nines of uptime. If that one outage lasted an entire day, and they were never down at any other time, that would indeed be three nines of uptime.

Yeah, my net admin colleagues explained that one to me a while back because the bosses were making similar uninformed demands ("this needs to never go down!" "Sure, here is how much that costs"). It was very enlightening :)

And by "never," a lot of bosses really mean, "you're working overtime if it goes down." It always comes down to cost.

Those colleagues are pretty much all quitting now, after their over time calculation has been modified because it was costing the company too much, while for other teams like mine it stayed the same. It's hilarious seeing the complete panic from the higher ups realizing how badly they fucked up. And I don't think they actually really realize...

Once I got a serious response to that from a manager saying that he could go on eBay, buy his own servers and do it himself. My response was to quit.

a lot of things are possible if you are lucky enough ;)

well this is probably PR as there is no such system nor it can be made that can have 100% uptime.

Five-nines is entirely possible with enough resources and competent outage-minded engineers.

Agree, but five nines are not 100% ;) Anyway - this discussion reminds me of Technical Report 85.7 - Jim Gray, which might be of the interest to some of you.

Distinguishing between 5 nines and 100% is just semantics in any discussion outside of contractual ones.

If you just threaten your employees enough they river go down /s

This is a software development business, which is a positively bananas trade no matter what’s getting written. And the smaller the business, the more hats network guys wear. We work with everything from the server app down to the coffee machine fueling the devs. And 100% uptime isn’t the most crazy demand I’ve heard. I’m sure Chujo is busier than a one-armed paper hanger with jock itch.

At least he’s got money to throw at his hosting company. Scaling up would have been much slower in the old days.

I'm not versed in videogame network infrastructures, but wouldn't be enough just having a load balancer and a couple of instances to ensure "100% uptime"? At least before all instances and the load balancer itself decide to join a suicidal pact, but more instances mean less chance of a critical event happening, no?

At a press level, sure, and the same for the average user. Legally speaking these numbers do have significance, though. Amazon Web Services (at least at one time) offer a guarantee of 99.99% uptime for their infrastructure. That 0.001% covers things like once a year outages that make the news. A 10000th of a year is actually a tangible amount of time and not even Amazon is confident enough to ignore it.

Depends on the cloud provider. AWS, as an example, have up to three “availability zones” within a single data center. If the customer needs HA, they are encouraged to run their applications in separate availability zones. It means different subnets within the VPC, redundant LBs spread across those zones, and more.

There is also probably DNS-based global load balancing across different data centers.

That’s just the hosting infrastructure. I’m sure Chujo works on the office LAN as well. He might wear the infosec hat also, which means he’s up to his eyeballs in firewall policy.

I don’t envy my brethren in software development orgs. Been there, done that, got that t-shirt long ago.

Math seems fine if 2 million people payed $30 usd.

Pocketpair says it's sold about 12 million copies of Palworld on Steam. At $30 each, that puts Palworld's gross revenue at $360 million so far, and that's ignoring its Xbox sales (it's on Game Pass, too).

Looks like a little more than 2M.

Yeah but even at $360m, $500k every month is operating costs for the server only, and doesnt include the business' other expenses. That's a big chunk of profit going to operating costs.

If the business had no other expenses then yeah, they could keep the servers up for 60 years. But more expenses lowers that and the look to get a subscription model or new game becomes more attractive to a company.

If they were only up for 1/10 of that, they'd be doing better than some games I paid $60 for

Ive seen articles that also say they've paid $500k on monthly server fees (ie total not per month) a couple times, so it could also just be a game of telephone messing up the info

They can optimize over time to get that number down. I assume they’re being a little sloppy right now in just over provisioning to keep up with peak demand.

But, yes, not particularly sustainable long term on just an initial purchase.

That's in the first month of release, when users are at their highest, the code is at its buggiest, and everyone is getting their first impression of the game.

Eventually they'll have to be more reasonable, but I can see this making sense for the first few months.

They need to make 17,000 sales every month perpetually to cover the costs, and then those sales will cause the server costs to rise as more people start playing.

It's a one time income to cover perpetual costs. They will probably either need to start raising prices, reduce server costs, or maybe start a subscription service eventually, or start doing micro transactions.

As the hype dies down the cost will go down a lot.

And I'm guessing their focusing on server improvements in software to further reduce the costs. $500k is a lot of money for this kind of thing, I still balk at the $50k or so my company pays and we're nowhere near the scale of Palworld.

12,000,000 sales so far on steam alone, so that covers the next 60 years or so

Why did you randomly choose that number? It sold more than that on its first day.

I vaguely recalled hearing a number of 1-2 million - but in hindsight it was a number related to peak consecutive users.

Honestly, other than the questionable Pokemon-esque models, Pocketpair seem like they have really solid heads on their shoulders

Like 90% of them are distinct enough. And besides there are no pals that are outright copies. Not Anubis, Cremis, Lovander or whatever other pals.

Imo it's like getting upset/weirded out by someone's fan art of heavily-Pokemon-inspired OC.

They could literally be Pokemon and I wouldn't care. I'm not going to defend the largest franchise with one of the most litigious copyright teams that primarily got popular having kids gamble on card packs.

That is an extremely valid point, and one I support and agree with.

However I'm more saying that I think it's weird that people are even thinking it's "cutting it close" that they even look like they could be Pokemon. It's like the weird fake controversy behind that game Mouse that's supposed to come out this year. Everyone was saying shit like "Disney is gonna come after them!" but again it's just like: why? It's an art style, you can't put a copyright on an entire style of art that was used by almost every single cartoon company in the 20s-40s, just like you can't copyright a design style like Pokemon's.

Azurobe's hair is very likely a model rip, but aside from that every comparison I've seen is either "someone who doesn't know anything about Pokemon would think these are Pokemon" or "the fluffy fox pal is obviously a ripoff of the fluffy fox pokemon, because only one company is allowed to make a fluffy fox!"

There's also that one pal that isn't actually in the game who looks like a trace of Luxray, but I give Pocket Pair the benefit of the doubt and say that it was just practice. Plenty of artists get their start with tracing.

No one going to point out users self-host their servers?

I think it's like rust where lots of people play official servers and I think more play community and modded servers. But there are a lot of people on those official servers. But I also think Rusts player based is more tech competent and it's a more mature game which lends itself to more people setting up their own servers.

Thanks, the cloud?

I mean yeah, pretty much, that and good software / network engineers. But otherwise hosting a global game like Palworld and having the player base it does would've been absolutely impossible if they were self hosting servers or calling up individual hosting providers around the world to work with. Being able to manage your entire network as software and be able to deploy anywhere around the globe nearly instantly does have huge benefits, not the least of which is that anyone can do it, even a small Indy dev, and there's little no upfront infrastructure costs, the costs only really scale with your users, so if your game flops you don't pay much, and if it's massive you should have the revenue to pay your bills.