Rendered at 22:28:05 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jubilanti 6 hours ago [-]
Almost 3 hours in and still "We are still working to identify the root cause"
Can't even view diffs in the web interface!
I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issues, GH actions CI, and pages host to deploy static sites. I don't use more than 100 mins of CI a month. Any recommendations?
edit: GitLab free tier has all I need, but I kind of want to pay and have some kind of reliability (not necessarily an SLA, but). But their next tier is $29/mo, which is too much.
adamddev1 5 hours ago [-]
I was in a very similar situation and this is what I did:
- put Forgejo on a little $5 Vultr VPS (I share the VPS with other things, Forgejo doesn't need much resources. I also added some block storage to host my Docker containers in the Forgejo Docker registry)
- got a 8GB ram cheaper Hetzner VPS (you have to wait for their availability) for around 8 EUR and used that to install the CI runner
- Use Netlify for hosting static sites and PWA
It takes a bit to set up but everything works snappy, clean, and flawlessly. I have unlimited CI minutes. As a bonus I get free Docker image and registry hosting because it's included with Foregejo. I am so, so happy with it all.
Also with Forgejo you just feel like you're using something clean and solid. GitHub started to feel more and more bloated, slow, and unstable.
epistasis 5 hours ago [-]
I've been very happy with forgejo for a personal, private server.
I don't care much about the social aspects of GitHub, but I do care a ton about hosting code for long periods of time, long after I've completely changed my personal homelab.
GitHub's most essential service for me is it being the equivalent of the Internet Archive, but for code.
cursuve 1 hours ago [-]
Similar! I was tired of the outages and crazy runner bills so I just finished moving everything to my own server hosting Forgejo. I've had to recreate a couple things (like a dependabot replacement) but it's been a week so far and it's going well - I haven't had any outages nor have I been finding ways to lessen my Runner billed minutes.
Forgejo has some quirks (like no ability to share workflows between private repos) but I'm happy with my choice so far.
cortesoft 3 hours ago [-]
This sounds great for personal projects, but what about open source work? Or if you want to collaborate with more than a few people?
skydhash 2 hours ago [-]
Unless you want those people to have merging rights, collaboration can be easily done with exporting a read-only version of the git repo via http (even a browsable one using cgit). Then anyone can work on their copy, generate patches (git format-patch) and send the files via any means of communication (email, slack,...).
Great setup, two questions, do you have 2 VPNs now or all on one Hetzner VPS? And how do you setup security?
Just recommend swapping vercel/netlify with something else, we had quite a few issues and high costs with them. Not opiniated here, currently using Cloudflare and BunnyCDN (which is great)
adamddev1 4 hours ago [-]
I just SSH into the servers when I need to. I don't have any VPNs with these servers. I just try to follow normal server security setup practices, disabling password authentication etc.
chasd00 4 minutes ago [-]
That’s what I do. Sensible firewall rules with ufw, turn off services I don’t use, follow the docs for the services I do. For a handful of servers the maintenance overhead is fine.
tom1337 6 hours ago [-]
Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues.
just shy of 3 hours to find the issue
unknownfuture 5 hours ago [-]
Weird, I've been told you can just throw AI at all your logs and metrics and it can magically solve all the things...
rurban 5 hours ago [-]
My blackest Monday today.
Deepseek is off/too expensive, Claude tokens out for 3 days, kimi refuses, and Github down for the whole afternoon. Maybe I'll take the week off
ajmurmann 4 hours ago [-]
It's crazy that it wasn't that long ago that all we worked with was a compiler on the local machine and maybe a SVN server you'd interact with every few hours
intrasight 2 hours ago [-]
Not that long ago that I used a local CVS server for SCCS. OK, almost 40 years - but still.
sillysaurusx 2 hours ago [-]
CVS was alive and well as of 16 years ago. It was our primary source control system at S2 Games. I was shocked but it really wasn’t all that bad.
eterm 1 hours ago [-]
Disadvantage: You have to coordinate with teammates to make sure you don't need the same file at the same time.
Advantage: You have to coordinate with teammates to make sure you don't need the same file at the same time.
Sometimes, not being able to work independently can be an advantage. Two people discovering they're trying to work on the same file can lead to useful conversations.
ALLTaken 4 hours ago [-]
hmm very interesting combination of services being down. I've been using DeepSeek v4 Pro and Qwen3.8 now daily for close to 2 weeks. Damn good and reliable. Anytime I need to verify math I check with Kimi k3 tho, it still feels like being superior to these both, not sure if actually true.
timcobb 3 hours ago [-]
Nvidia build might have DeepSeek capacity
rambojohnson 4 hours ago [-]
[dead]
hombre_fatal 5 hours ago [-]
Well, it would be a mistake on your part to assume that everything in an organization, especially a large one like github, is a reflection of the capabilities of the state of the art in anything (AI, architecture, programming language, organizational management, operations, etc.).
It's clear that updating the status page as quickly as possible isn't even something github cares about.
unknownfuture 1 hours ago [-]
It would be a mistake to judge AI based on the performance of one of its strongest pushers and promoters? Interesting take.
therealdrag0 60 minutes ago [-]
My company is heavy on AI promotion, but many many many people still do things sub-optimally, and many systems are still not well integrated/able with AI.
“The future is here it’s just not evenly distributed”, and that applies inside companies too.
Hard to actually judge without being in the room and knowing the team that owns the incident. And as other said, it’s pretty common IME to have a root cause but slow roll communication of it until remediation has more confidence.
esseph 4 hours ago [-]
> It's clear that updating the status page as quickly as possible isn't even something github cares about.
Status pages for large companies are social, political and legal tools, and not technical indicators. It has always been this way, and likely always will unless some laws were passed.
"Strong signs of recovery" is a crazy way to describe the status of a computer system. Wow.
bargainbin 2 minutes ago [-]
It’s all just ones and zeroes, and maybe a two. You never can tell with these computers!
throw-the-towel 25 minutes ago [-]
IDK, when a system is very complex — which GH certainly is — this phrasing sounds kinda fine to me? It's recovering, but not yet operating fully as intended.
I mean to be fair it’s a huge production system, there are many moving parts. Having worked on outages before in complex systems it is very difficult to identify root causes under pressure
ofjcihen 1 hours ago [-]
Especially when you’re worn down because it happens weekly right?
rurban 5 hours ago [-]
So just 3 more hours to fix and restart and process the queues
throwaway2037 21 minutes ago [-]
I'm not here to shill, but would SourceHut[1] work for you?
Are you paying for GitHub? Are you complaining about an outage from a product you aren't even paying for? You know the whole point of git is you don't _need_ a central server.
I don't understand the entitlement in your post here.
jubilanti 1 minutes ago [-]
> I don't understand the entitlement in your post here.
Excuse me, I'm literally asking where I can pay to get the specific cloud/web-based features I listed. I'm trying hard to NOT rely on a free tier.
sellmesoap 2 minutes ago [-]
Are you the user or the product? You can be both with GitHub!
GitHub provide a compelling product, enough to become the defacto forge for many. But their reputation has taken a lot of hits, as a grey beard I'm salty since the Microsoft takeover, the downtime is just the leaky diaper icing.
To diagnose at a distance it feels like they need a stronger QA process, it's a product that scales horizontally, but somehow they find a way to break it for everyone on the regular! They need less hub and more node!
lowercased 6 minutes ago [-]
Not OP, but I pay. And it was an annoyance today for several hours that did impact some work.
I don't need one, but I do have that because it's doing a bit more work than just git hosting (administratable access for others on the team, build actions, etc) and yes... I'm paying for it.
throw-the-towel 22 minutes ago [-]
And I don't understand why some people feel they need to be the Criticism Constable.
ChoGGi 6 hours ago [-]
No idea about some of that stuff, as I have more basic needs; nearlyfreespeech.net is cheap and stays out of my way.
memonkey 5 hours ago [-]
wow totally forgot about nearlyfreespeech. so glad they're still around. wonderful host.
bad_username 3 hours ago [-]
> GitLab free tier has all I need ... their next tier is $29/mo, which is too much.
Pay every Nth month, or M months per year :-) Use the free tier features throughout.
stephenway 4 hours ago [-]
Forgejo is a great fit for what you're describing. I run Fjord, which is managed Forgejo. Our dedicated instances are above your $5-10 budget, but we also run a free public Forgejo instance if you want to try it without operating anything yourself.
bryanmikaelian 4 hours ago [-]
Disclaimer: I work at exe.dev
Forgejo has been great for me too. I liked it so much, I added a feature to make it relatively easy to get up and running on exe.dev. Support for Gitea and Tangle.org is also available.
hinkley 5 hours ago [-]
Sounds like they don’t have a fine enough lens on some particular class of problems. Been there before. The source of the meta problem is so old few people will learn the lesson that hadn’t known it already.
Aeolun 5 hours ago [-]
I’m working on it. Unfortunately ‘high reliability’ also means ‘tons of testing’, and ‘finding all the performance holes before your customers’.
It’s kind of a shame, because Github keeps giving me fantastic ways to toot my own horn, but I can hardly do that if I can’t improve on their numbers.
__float 4 hours ago [-]
Architecturally, how are you storing Git data? If it isn't fundamentally different from bare-repo-on-disk with libgit2 or wrapping the git binary, you're not going to scale any better.
imglorp 2 hours ago [-]
$work just moved a few dozen repos to GitLab using their import tool, including issues, PRs, comments, etc. It got everything over but stumbled on a few comments.
Not a prop to GitLab, it's kindof table stakes now, just saying whichever place you go, you can find or write a migrator that will capture most of what's in GitHub, assuming their API is up long enough...
juliend2 2 hours ago [-]
Check out `git init --bare`.
You can basically self-host your repos on a cheap VPS. I did that some time ago. You'll have to create a ssh user that has only 'git' as a shell, iirc, on that box. Works okay. But doesn't provide a web ui.
I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.
noahbp 8 hours ago [-]
Because any price at all will immediately cause users to shift to another platform, and GitHub's value is that it is _the_ place to put your code on the internet.
stusmall 3 hours ago [-]
Being the _the_ place for open source is what let them dominate the enterprise. Now that they dominate the enterprise, it'd make financial sense to put hard limits on the free tier. It's cynical but it would be foolish to lose that cash machine. It'd open them up for a competitor to slowly build up good will in open source then take them on in enterprise, but right now they are risking losing it all.
dbmikus 2 hours ago [-]
Big enterprises are on a different github deployment, so I doubt they are affected by this
rtpg 15 minutes ago [-]
I don't know how true that is nowadays, I think if you sign up to Enterprise Cloud you're in the same spot as everyone else
(maybe F500 people have their colocated box, but I have the impression that those are the "first class" seats (very expensive but _very few_) whereas enterprise cloud is the "business class" (loooooots of seats making a lot of money)
throw-the-towel 19 minutes ago [-]
In my experience, we were typically affected just like everyone else. Maybe a public company with its market cap in the billions isn't big enough.
snazz 1 hours ago [-]
Enterprise Cloud was indeed affected by this (as of this morning)
milkshakes 48 minutes ago [-]
lol
floro 1 hours ago [-]
Increasing prices are a sure way to enshittification. It will push people away from github. Many such previous examples
dannyw 6 hours ago [-]
The network effects isn’t that much, and like what we see with Steam, you want to be where the audience is.
More realistically, MAU or something is a metric/OKR. It’s easy to imagine why a business wouldn’t want to cap that.
hinkley 5 hours ago [-]
Someone did a study that suggested the network effect is logarithmic in nature, but if that’s true it means something with 95% market share is 3 times “better” for having everyone there. Not 20 or two orders of magnitude better but not nothing.
graceful6800 5 hours ago [-]
And the audience you want are users who will never, ever pay to get access. If GitHub starts throttling free users, they throttle the audience you're talking about far, far harder than you.
The audience will go away to somewhere you aren't. Then what?
esseph 4 hours ago [-]
> The network effects isn’t that much
There are people in this thread that have said their biggest reason for not moving away is the network effect.
sailfast 5 hours ago [-]
Weird. For me it's in Actions, Issues, and other integrations / workflows / add-ons that aren't that easily replaced.
ljm 5 hours ago [-]
And they can keep training their own models off of the code, and on private repos too.
AI makes investors/shareholders go brrr.
2 hours ago [-]
MisterPea 6 hours ago [-]
I wonder how much network effects help here.
There's a lot of consumer surplus with the features GitHub offers. I'm sure they can have better pricing tiers to capture the surplus without losing too many users.
aurareturn 8 hours ago [-]
And another platform will suffer the same thing and charge probably even more?
gib444 3 hours ago [-]
Haha people have put up with a LOT of price rises the past 5 years in many industries, validating many company's pricing strategy, over and over
Quarrelsome 1 hours ago [-]
uh, isn't its value that some of us pay them money to have a service?
frollogaston 2 hours ago [-]
It's easy from an armchair to say things like they should just get more capacity, or they should segregate free repos onto other servers. But maybe they are doing what you're thinking, and with that churn they get mistakes that cause outages, esp dealing with legacy systems. I've been there (not at GitHub or MSFT).
sqren 1 hours ago [-]
They do not need to make any technical changes. They could make the change overnight: $1 per repo per month (or whatever).
This would stop the aggressive growth that they clearly cannot handle anyway.
The question is: are they willing to compromise growth for QoS. So far the answer is a resounding: "no". They are deliberately sacrificing the experience of existing users to gain new ones.
frollogaston 1 hours ago [-]
There are tons of existing free users, they'd be sacrificing those too. And yeah they don't want to sacrifice growth either, so the remaining is an engineering question.
ksbd-pls-finish 58 minutes ago [-]
No way I pay $1/repo/month. No way I pay $1/repo/year. I still host and maintain my old code on GitHub, but any free tier pricing update and I jump ship immediately.
rob-lag 3 hours ago [-]
Why not just add a rule such as: "Non paying customers can only upload X commits per day"
X doesn't even need to be that low. It could be 100 or 200. Just something that slows down the agents that spam them with commits/pushes.
Waterluvian 8 hours ago [-]
The people with the spreadsheets have determined the value of this loss leader to be too enormous to shut it off.
adverbly 7 hours ago [-]
> My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude
Seems presumptuous to assume a root case while the incident is still ongoing, no?
agos 6 hours ago [-]
they've been complaining about LLM generated code and the related traffic for months now, this specific incident might not be caused by it but the general trend is
They also sell the machinery to produce the slop (through OpenAI at least).
If they make it harder to push slop on github, people might treat it as an incentive to generate less slop. Less tokens sold. Bad. Very bad.
The github paying customers are not going to move away.
Plus, it's probably genuinely hard to scale. And they can't throw compute at the problem - all compute is already going to the LLMs.
4 hours ago [-]
blitzar 4 hours ago [-]
I don't understand why Github hasn't solved this problem with vibe coded updates.
duped 8 hours ago [-]
Microsoft executives have an incentive to pretend that AI is great and scaling it has no problems
aurareturn 6 hours ago [-]
I recently analyzed my commit history.
Last month, I made more commits than my entire 2025 combined.
frollogaston 2 hours ago [-]
And even if my loc were equal vs before, my personal projects used to have massive commits with the text "asdfasd"
duped 6 hours ago [-]
How much more value did those commits create than in 2025?
But that discards my point, which is that you should probably have been prevented from pushing all those to github unless you were paying them for it, but you weren't because the executives running github can't admit that AI is bad for their platform.
aurareturn 6 hours ago [-]
How much more value did those commits create than in 2025?
We have 2x fewer developers now than in 2025. Yet, the amount of features and bug fixes we're pushing is something like 5x.
So probably 10x more value is my guess for my entire team.
If you mean translating that into revenue, I'm not sure. Our product is better than ever with features a much bigger team couldn't dream of building before. Stability has also been better with fewer bugs, issues, downtime. When there are bugs, it gets fixed far quicker than before.
justinbaker84 7 hours ago [-]
I have wondered the same thing. Even a small fee like $50 a year would get rid of a lot of AI slop that justs eats up resources.
I would gladly pay that fee if it means Github is reliable for paying customers.
dbbk 5 hours ago [-]
Private personal repos used to be paid. Maybe they should bring that back.
dogleash 2 hours ago [-]
>So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over?
I think it behooves them for the public presentation of their issues to too much load. If they implemented load shedding and still had issues the follow-up questions would be less comfortable.
BryanBeshore 8 hours ago [-]
They would benefit from a HackerNews-esque karma system.
internetter 4 hours ago [-]
That is essentially your number of followers. Pretty hard to have a system that cannot be gamed. I think followers are probably loosely correlated with tech ecosystem impact in some capacity, but the second there is an incentive to accumulate them, well...
guluarte 7 hours ago [-]
they want that code (from private repos) to train the models
khvn26 9 hours ago [-]
I had a lot of goodwill for GitHub but I think today is the tipping point.
Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again.
The hope is dead.
stingraycharles 9 hours ago [-]
I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding.
But maybe I’m wrong?
pera 7 hours ago [-]
Save that for your local mom-and-pop store: Microsoft is a multi-billion dollar corporation with enough resources to, at the very least, provide a reliable service for enterprise clients.
The "AI is using a lot of resources" excuse was maybe acceptable last year but not in Q3 2026.
unknownfuture 6 hours ago [-]
Doubly so given this is hardly a surprise. We've been on this trajectory for at least a couple of years now. They don't get to shrug, point at 10x volume, and act like they've been blindsided.
mhitza 5 hours ago [-]
They had large increase in volume, and trying to move to Azure at the same time. I don't envy them for either work they need to do. But also don't feel pithy because it's Microslop, at the end of the day.
pdimitar 5 hours ago [-]
All being said, I have most sympathy towards engineers of all people. They don't get to call the shots, they do as they are told. 99% of the time by an extremely out-of-touch managers.
Managers and executives though? Those I do blame. Surely at this point it should be blindingly obvious their current strategy is not working.
m0llusk 3 hours ago [-]
The end of the day is now.
inquirerGeneral 5 hours ago [-]
[dead]
frollogaston 2 hours ago [-]
Ok but if the same problem will happen with the mom-and-pop store, switching there doesn't help. This isn't a moral dilemma, people just want their stuff to be up.
grumple 3 hours ago [-]
If you use Azure you’d know they are definitely not able to scale up quickly. Tons of quotas and lack of availability.
worble 8 hours ago [-]
I have little sympathy that they're buckling over a problem they helped create.
kokanee 5 hours ago [-]
Yeah let's not forget that in those brief few days when Sam Altman was fired, Satya Nadella swooped in and hired him. Despite the failure of that move, Microsoft has invested $13 billion directly into OpenAI, and has spent $100 billion on the partnership. No sympathy from me watching their software crumble as a result.
bamboozled 8 hours ago [-]
Exactly.
mrighele 8 hours ago [-]
> scaling problems due to the influx of LLM assisted coding
You mean people pushing LLM assisted coding, or github's own coders pushing LLM assisted code to production ? :-).
I am more inclined to think that it is the second, but if the reason is the first, what is particular hard to scale ? It's not like everybody is vide-coding into a single repository, so I would expect that most of the load would naturally be scalable by increasing the resources. If for some reason that is not possible, I would expect a corporation as big as Github to keep the paying customers working by gracefully denying service to the free tier, not to collapse like an house of cards.
krs_ 6 hours ago [-]
> You mean people pushing LLM assisted coding, or github's own coders pushing LLM assisted code to production ? :-).
Why not both?
Groxx 5 hours ago [-]
All three: their own quite aggressive pushing of LLM coding as a whole.
nxc18 7 hours ago [-]
The quality and performance of GitHub enterprise has degraded horribly this year. Their problems are self-inflicted and influx is just an excuse.
Despite minimal new features, and drastically increasing hardware resources to keep up, self-hosted GitHub enterprise went from things being instant to most requests taking many seconds. And activity on issues in one repository tends to mysteriously re-open issues with the same number in other repositories.
I think they are vibe coding and facing the consequences.
ALLTaken 4 hours ago [-]
Please let this not be true :( I mean nothing against vibe coding for hobby and assisting you when you get stuck, but at something important like Github they should have hooks to stop low-quality commits and repeated CI-breaking or issue causing should automatically restrict accounts at such companies I think.
Do you think they have such measures in place? I remember when my indention wasn't spotless our university rejected my commits or if my function complexity was too high.. Tears in my eyes.. I think it was called Gerrit Code Review.
ToucanLoucan 2 hours ago [-]
It’s become clear to me that any company acquired by Microsoft should be avoided. All software dev is hard. Software dev within a huge organization is damned hard. And Microsoft is SO HUGE and has so much infighting and finger pointing going on that they’re now, as far as I’m concerned, incapable (let alone unwilling) to ship good software.
No faith left. None. They turn everything into a quagmire.
BowBun 8 hours ago [-]
How many weeks/months can they use this excuse? They are literally are at the forefront of this emerging industry and are capturing untold value. To let their product suffer and potentially lose market share because of it is extremely foolish
kypro 8 hours ago [-]
Scaling can take time but they've had at least a year to prepare. It's not like any demand increase they've seen was overnight...
At most normal companies you monitor your systems and address potential bottlenecks before they reach a tipping point. And generally you want enough headroom that a sudden 2-3x increase in demand wouldn't take out the service.
Either Github's technical leadership/talent is completely out of their depth (seems unlikely), or they're not hiring enough engineers to work on these capacity issues.
unknownfuture 6 hours ago [-]
> Either Github's technical leadership/talent is completely out of their depth (seems unlikely), or they're not hiring enough engineers to work on these capacity issues.
Regarding their leadership, I'd argue the latter implies the former.
So yeah I'd say they're out of their depth.
dannyw 6 hours ago [-]
More likely, as seen in a lot of companies, perhaps infra teams got whittled down (or frozen HC, less than BAU, etc) with resources reallocated to AI org units.
lkjdsklf 5 hours ago [-]
Hardware resources are also allocated to AI uses so you can't scale out that way anymore.
peezd 5 hours ago [-]
I got the impression from a previous thread that the continual issues were largely driven by the required move to Azure, and underlying Azure stability
mooreds 6 hours ago [-]
Found this reddit thread from a month ago that talks about the scale:
Commit volume went from 1 billion total in all of 2025 to 275 million per week by early 2026.
Do you know how many thousands/millions of dollars that would cost to serve!?
Microsoft has to spend more money on acquiring games studios for billions of dollars then shut them down.
mooreds 3 hours ago [-]
No idea about the cost.
I do know that very few systems are designed to handle a 14x increase in traffic painlessly (275M*52 = 14.3B), especially at scale.
rfgplk 2 hours ago [-]
Tracks with some of my internal metrics. There are certain projects I'm tracking (solo dev or a small team of a few people) doing 500k LoC changes _per week_. There's one project that caught my attention in particular because they had something like 800k commits and around 7 million lines of code... and they created the repo in _July_. Also, yes I checked, the repo gradually grew over time they didn't just splat it in the first commit.
dragon-hn 11 minutes ago [-]
Bonkers. Almost a million commits in less than 2 months.
Scubabear68 6 hours ago [-]
The thing is Github’s core is git’s core plus extras, and thet core is fundamentally text wrangling.
Youtube and Insta and Facebook and many other social media platforms handle very resource intensive media for billions of people plus bots.
Increased influx from AI is not plausible when viewed from that lens.
edgyquant 6 hours ago [-]
It is though. Maybe you think they should be doing a better job but 12-24xing scale when you were already handling billions of commits is definitely understandable to me. It just does feel like they should have some way of ensuring customers, paying ones, don’t have to suffer so much.
Scubabear68 34 minutes ago [-]
First off, there aren’t that many developers in the world compared to consumers.
Taking every developer in the world and multiplying by 12x-24x is not a-lot compared to many consumer oriented sites.
Second, you can’t ignore the payloads, even an AI mostly text push is nothing compared to, say, 10 iPhone pictures or one moderate video.
spockz 5 hours ago [-]
Yes. It seems that at the very least repositories owned by paying customers should be processed on priority threads. And perhaps throttle commits/pushes to repos where there are a lot of pushes from the same person all the time. This can be effectively throttled at the push level.
ajasmin 4 hours ago [-]
Lots of paying customers also depend on the open source part of GitHub to run their business.
verve_rat 2 hours ago [-]
That sounds like an opportunity for GitHub to clip the ticket for providing a way for those businesses to support those projects more directly.
slowin 8 hours ago [-]
Services like Facebook have orders of magnitude more activity than GitHub and manage much better uptime.
dannyw 6 hours ago [-]
Services like Facebook aggressively block automated activity, including AI agents.
GitHub sells this as a service.
slowin 5 hours ago [-]
Bots or not, Facebook has orders of magnitude more traffic.
ljm 5 hours ago [-]
Github has had a pretty spotty track record for uptime for a very, very long time. It didn't get any better when MS bought it out.
It has really just coasted on GH's network effect because everybody uses GH for git by default. It doesn't really have much direct competition and anything that does offer something like GH just doesn't have the same mindshare.
In that way, GH became more like MS.
cautiouscat 7 hours ago [-]
That’s the reason they gave, and I believe them. But I also think there has to be more to it. If your paying users are seeing degradation caused by (likely) free tier users (for years by the way), restrict the free users.
theappsecguy 8 hours ago [-]
They've had an insane influx. It seems that they should more aggressively limit free tier activity or isolate paid users, this has become unbearable...
dannyw 6 hours ago [-]
Or maybe prioritise humans / human-shaped activity? Agents can tolerate a 503+Retry-After is a lot less frustrating to an agent than a human.
ALLTaken 4 hours ago [-]
No, that's almost exactly right, except that scaling problem is inverse!
MSFT is giving away most of their Azure Datacenters resources to AI, to make money and doesn't care if Github suffers along the way and drains Github's available resources to bare minimum. Leading to outages like this, but you're 100% spot on right too. All those Claude/OpenClaw instances produce so much Code, people setup more and more repositories.
benrutter 9 hours ago [-]
I'm really curious how other codeforges are faring. I only use codeberg and github - codeberg seems to have less outages, especially for a much smaller organisation, but they've recently removed support for LLM projects which could be linked.
Anyone here have data on whether the likes of gitlab have been impacted in the same was as github?
koalp 8 hours ago [-]
I love codeberg but it also have outages and is sometimes quite slow.
But I’m more lenient with codeberg because they are a non-profit and as you say, they are banning vibe-coded project because of the cost to infrastructure. In the meantime, GitHub is encouraging vibe-coding, multiplication of the source code, and genAI discussions.
sznio 8 hours ago [-]
my personal forgejo instance has had 100% availability for the last two years.
spockz 4 hours ago [-]
So it has run on a server that hasn’t had updates in 2 years? And it itself hasn’t been updated in 2 years?
forgejojoj 4 hours ago [-]
The key is that maintenance is done on your own terms, and thus it's available when you need it.
cyberax 4 hours ago [-]
You can run it in an HA configuration with multiple servers and live updates.
spockz 2 hours ago [-]
Yes. And the same for the database and then you get it into “professional” territories with more moving parts that can also fail and take 3 hours to figure it out and recover.
cyberax 1 hours ago [-]
Yup. It's kinda binary: either almost no downtime for years with a crazy number of nines, or a total mess for several days while people figure out the issues.
That being said, a Git forge for a small/mid-sized company is not something that requires rocket surgery. Forgejo with an SQLite database and a daily backup to S3 would be more than enough for the majority of users.
bamboozled 8 hours ago [-]
Isn't all the LLMs and AI supposed to be "solving these problems" because all the engineers are now 10x ? Not being sarcastic but I mean, this is what we were told and sold...by the likes of...Microsoft.
PrayagS 4 hours ago [-]
That excuse was valid a few months ago or back in 2025. Not anymore.
nicoburns 9 hours ago [-]
I saw an HN comment a few weeks back claiming their traffic has increased ~10x.
Even Claude could have caught and addressed most of these scalability issues long ago. This can’t be due to load or capacity. This is something else.
dannyw 6 hours ago [-]
I’d love to see the org chart sizes
How many hires went into infra and platform? How many went to Gen AI org units?
From what I see, and that is a constant wave of very meh features, it seems like the latter.
lijok 7 hours ago [-]
If they did, they would’ve paused new feature development a long time ago and redirected that resource towards scaling the platform. And yet they’ve been releasing tonedeaf features more than ever.
StilesCrisis 5 hours ago [-]
Feature work is frankly much easier than scaling and bullet-proofing.
kedihacker 8 hours ago [-]
It should be embarrassingly paralel I don't get how they can't figure out
blanched 8 hours ago [-]
What should? Presumably if it were easy and solved their most visible problem then they would have done it already?
esafak 8 hours ago [-]
Maybe they just suck.
dboreham 6 hours ago [-]
The decentralized part (git itself) kept working.
jimbob45 5 hours ago [-]
HN is heavily anti-US and will always insist that you move away from anything US-owned in favor of something European or Chinese instead. The fact is that there are semi-legitimate reasons to take this long. FB had DNS issues last year (?) that were difficult to spot and out of their control (somewhat).
mnewme 1 hours ago [-]
The vast majority of HN users are from the US.
esseph 4 hours ago [-]
> HN is heavily anti-US
This is a very distorted lens.
Many on the outside might say the US is very pro-self and anti-everyone-else right now! America first, right?
Well, take a random country. We'll make a made-up one called Tristonia. Should Tristonia do things in Tristonia's best interests? Absolutely! If Tristonia had some kind of military or economic threat from another country, should they take steps to protect their country, resources, and people from the threat? Absolutely! The same thing the US would do.
Most of these people aren't necessarily anti-US, they're against being beholden to another country, especially one that has been incredibly... inconsistent, as of late.
iAMkenough 6 hours ago [-]
Considering their leadership ushered that in themselves, that’s just poor planning.
catigula 7 hours ago [-]
Enterprise software isn't about giving "breaks"; it's about does the product work for you or not.
rvz 9 hours ago [-]
Well, it has been 8 years since the acquisition and they have had access to Azure infrastructure to keep up with the scale and yet, it is still struggling due to free compute thanks to Actions.
They need to start getting rid of the free tiers for GitHub Actions and actually charge appropriately or increase prices for it.
hansmayer 4 hours ago [-]
[dead]
Maxion 9 hours ago [-]
They've done the typical sillicon valley sillyness of having 10 000 engineers over-engineer a product so that there's way too many features built on top of what should be a simple tool and are now paying the price. Literally every larger US company makes this mistake.
bjackman 9 hours ago [-]
I deployed Forgejo on my homelab yesterday. Now GitHub is just a free backup service.
I guess they're quite happy with that TBF, it doesn't seem like they're that interested in being a git forge anyway.
jbs789 5 hours ago [-]
The Unicorn page rubbed me the wrong way. Thankfully I was able to revert changes to my prod instance using another approach. But I count on GH for reliability.
thanh76 5 hours ago [-]
I think the same. But the GitHub CLI is the biggest block for me. It is a killer feature. No other tool is close. To self-host the forge is easy. To replace gh in my scripts and agent workflows is not.
regexorcist 4 hours ago [-]
Any good LLM should be able to come up with the equivalent git and curl commands fairly easily.
brahma-dev 8 hours ago [-]
Atleast you're getting the unicorn. I am getting 404 on valid links. For a moment I thought the repo is gone.
They should replace the unicorn image with a horse or at least a zebra during ongoing incidents.
chrisjj 8 hours ago [-]
Donkey, surely.
tom1337 4 hours ago [-]
Wow they just went into "another" outage
> Update 16:59 UTC - The degradation affecting API Requests, Actions, Git Operations, Issues, Pages, Pull Requests and Webhooks has been mitigated. We are monitoring to ensure stability.
> Update 17:30 UTC - Git Operations is experiencing degraded performance. We are continuing to investigate.
> Update 17:36 UTC - Issues is experiencing degraded performance. We are continuing to investigate.
> Update 18:48 UTC - API Requests is experiencing degraded availability. We are continuing to investigate.
s_dev 8 hours ago [-]
I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat.
Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn't someone recently calculate that GitHub is running at 95%? For comparison the Irish Rail service which is not reliable has 80% of it's trains run on time.
This seems absurd and really challenges a lot of ideas I had about big tech and cloud infrastructure. GitHub seems to have remained the dominant player relative to GitLab etc.
tokioyoyo 7 hours ago [-]
People overestimate how much they care about stuff. Moving off GitHub would be more costly for us than having 5% downtime. Obviously there’s a tipping point, but it shows people are tolerant given the price tags.
dboreham 6 hours ago [-]
For a reasonably common class of deployment and organization (E.g. an online service where the cost of a short outage is significant) it becomes a pain point. For instance if there's a sudden security flap and you need to re-spin your system and redeploy, but that process is gated on GitHub working, now you're screwed if that security flap happens when GitHub is in its 5% down time. Basically you've coupled your uptime to GitHub's uptime in some measure.
tokioyoyo 5 hours ago [-]
I agree with you in spirit, as we have this problem from time to time as well, but number of businesses migrating off-GH shows this isn't the trigger point yet.
acedTrex 5 hours ago [-]
More specifically you've coupled the likelihood of your downtime coinciding with githubs downtime. Which as its a product is likely a very small % chance.
advisedwang 6 hours ago [-]
If I'm running, say, an e-commerce business, an hour of downtime on my serving-path might cost me $100k of revenue. An hour of downtime for my development stack might cost me 30 person-hours = a few grand. As a result, I'm much more willing to have unreliable github than unreliable CDN/Compute/etc.
We don't care. We don't have to. We're the phone company.
Galanwe 8 hours ago [-]
That canot happen if the market becomes monopolistic, with every bigtech out there buying every startup worth a penny. The erosion of anti trust in the US created this monstrosity.
willchis 8 hours ago [-]
A lot of it is inertia I think. I hear people saying that they "just use so-and-so for version control" but really we all have tons of CI/CD build and test pipelines, config, business processes etc in Github (rightly or wrongly).
frollogaston 2 hours ago [-]
That number of 9s might be a thing for the lower layer cloud services like on AWS or Azure or GCP. Even then, that us-east-1 outage was an accidental power move. It established that many are best off putting all their stuff on us-east-1 cause then their outages only happen at the best possible times, whereas multi-region complexity might cause its own kind of outage when nobody else is down.
2 hours ago [-]
akmarinov 8 hours ago [-]
Competing services can't afford the data storage to compete with Big tech
threetonesun 6 hours ago [-]
This is the important part, we've moved away from monopolies on software to creating a vertical stack that allows a few companies to corner all cloud hardware, which let's them prevent anyone from competing on both price AND scale.
efficax 7 hours ago [-]
I think AI is going to change this. Once you've built around github you feel locked in, since moving all your CI and other actions out of there, and your workflows out of there, is labor intensive. Now you could migrate from github to another service in a week, maybe less depending on how many workflows you need to move.
JackuB 4 hours ago [-]
I feel you are correct. Throwing AI at this problem works reasonably well. “Rewrite this Actions workflow to run on $PROVIDER” is a sensible task.
It takes a while by hand, but I recently saw a migration of old Jenkins projects to Actions with AI (yea), and it was smooth.
I was there before 2018, their reliability was never that good.
I still remember that they launched GitHub Actions in 2018 and back then I was thinking: They can't even make the website reliable, surely this will only cause them more problems.
rotis 7 hours ago [-]
This site's trustworthiness when it goes for pre-aquisition data was disputed here already:
seems to be correct. there's no record of the outage on 8/13 :/
let_rec 7 hours ago [-]
HN is likely a better source than GH status...
copper-float 3 hours ago [-]
Why would they do that?
pyaamb 5 hours ago [-]
Wow.
8 hours ago [-]
winrid 4 hours ago [-]
Pretty sure this is caused by trying to move to Azure (and I'd bet like $1000 partially due to trying to move DBs to network backed disk instances)
liendolucas 17 minutes ago [-]
There is no single product Microsoft hasn't touched and ruined. People should leave all Microsoft products en masse. I'm glad I'm no longer on Github.
What I find a bit weird is why so many people rely on it. Today we have tons of VPS providers where anyone can host their on git repo and CI/CD pipeline.
What is preventing so many people to effectively have a custom setup?
adamtaylor_13 12 minutes ago [-]
Momentum + path of least resistance.
My company is still on GitHub... for now. But incidents like this are becoming more common and I'm becoming less tolerant of it. I suspect many feel the same, but it takes time to persuade the masses.
Arubis 9 hours ago [-]
This is starting to feel like Twitter. For a variety of reasons, we had a centralized place where everyone of a particular set of persuasions could connect, and this had outsize benefits for the community as a whole. That place is becoming untenable, and with the loss of goodwill and stability, the community’s started to dissipate. But there isn’t one obvious transition candidate, so the diaspora is finding itself spread across a bunch of disparate places and services. The stated need they satisfy (source control with a web interface and some technical features, like pull requests with reviews) will be fulfilled. But those emergent features like a core community and default expectation of where you can find someone will fade. And that is a very real loss for all of us.
IncandescentGas 8 hours ago [-]
Good. That's a gain, not a loss. We're better off without social media psyop manipulations and subscription services bolted onto our code repositories. We're better off without a giant evilcorp getting wholesale access to a large majority of the world's source code.
fallingbananna 7 hours ago [-]
Personally I never engaged in the community aspect of GitHub.
In my eyes it would be a ginormous gain to our colective mental health to not have the expectation to contribute to open source projects for free, or to have "green squares" every day on a public GitHub profile.
Treegarden 6 hours ago [-]
This seems to be such a non issue to me, I always ignored this and its strange to hear others feel compelled.
graceful6800 5 hours ago [-]
That's the point of it though. It's designed to be gamified and addictive to coerce users into spending more time Engaging with the site.
Great that it doesn't work on you, but it works really well on a depressingly large number of people.
dwedge 4 hours ago [-]
Especially now with LLMs. My hand coded github activity is through the roof the last two years just because of a job change, but I assume everyone's activity is up in the last 2 years for other reasons and if it meant anything in the past it definitely doesn't now
dkdbejwi383 8 hours ago [-]
I just use GitHub for the basics: repo hosting, reviews and actions - what are the "social media psyop manipulations"? I've never really paid much attention to the other features.
MisterTea 7 hours ago [-]
All the metrics based around how many commits you have made, how many stars your repo has, etc. Why is that info gamified? Its a dopamine addiction channel.
tokioyoyo 7 hours ago [-]
It’s for subset of people who cares about that. You can just blanket-ignore everyone who cares about that, and move on.
mirekrusin 7 hours ago [-]
Seriously?
chicken1991 8 hours ago [-]
But now the users who only use core functions (re: repo hosting, reviews and actions) suffer when the service is down due to overreach with feature implementation.
artyom 7 hours ago [-]
It's good you can isolate yourself from it. But there's plenty of cases.
Open source maintainers being harassed or being demanded work through Github's issues. Or being feed AI slop to review endlessly. Or recruiters filtering resumes via Github activity graphs or repository stars.
And obviously the side shady market of all of the above (e.g. buying/selling popularity).
Like the parent comment, I can't wait for social media to be completely obliterated from code repositories.
fn-mote 7 hours ago [-]
> Open source maintainers being harassed or being demanded work through Github's issues
This isn't "psyops". Sure, it's a bad side of social interactions, but when you interact with other people, there's always some of this - in person as well as online.
The lead-off argument here seems more like "interacting with society can be bad".
I have no argument that selling popularity is bad and filtering by stars is stupid, but I don't blame Github for any of that.
You could equally well assign the success of certain coders to YouTube because they came to attention by showing off their work in livestreams. Which I do not. They found a way to advertise their own success, but it's theirs.
artyom 6 hours ago [-]
> The lead-off argument here seems more like "interacting with society can be bad".
The argument is that there's no need for that much social interaction to produce quality code, quite the contrary, the more social interaction, the more focus is lost.
I know it's an unpopular opinion, but there's a reason why the Linux Kernel communication is still done through acerbic email threads.
creato 7 hours ago [-]
None of this is specific to GitHub. The exact same problems will sometimes happen any time users and developers communicate, which is a basic requirement of developing useful software. I doubt GitHub issues are worse than mailing lists or any other mechanism.
nemomarx 6 hours ago [-]
They almost have to be - which users will be able to get to your issues page and which will be able to find a mailing list? You'll have a much more selective audience in the latter
ascii0eks84 6 hours ago [-]
Demanding work isn't really an issue. Anyone can do that, and also reject the work. Dishonest actors on the other hand would take money for someone else's work.
ascii0eks84 6 hours ago [-]
You outlined what it (and all git hosting other services) are for: repository hosting, optionally issue tracker, mandatorily pull requests (for syncing with master or other branches), Actions for automated steps on merge to a tracked branch.
zerox7felf 7 hours ago [-]
Could be argued to be both a gain and a loss...
bookofjoe 6 hours ago [-]
"Replicants are either a benefit or a hazard."
andrewmutz 8 hours ago [-]
If there are no subscription services, what will fund the free open source repositories?
mbonnet 3 hours ago [-]
Who the hell uses GitHub socially?
hliyan 8 hours ago [-]
"Protocols, not platforms" should have remained the philosophy of the Internet. Alas, we are here.
Even if the platform itself doesn't take off, the protocol and the data ownwership (you host your repo artifacts, issues, prs, etc on your ATproto server) mean that a competitor could come in easily if you were no longer satisfied with performance, reliability, or monetization.
zoobab 6 hours ago [-]
"Protocols" now that everything has been moved to a master-slave protocol (HTTP), we are all slaves of the masters.
It's a social network with github-like social features and git hosting, but you can still self host your git and have all the features like issues, PRs and so on just like GitHub.
And it's built on ATProto so you can also have your issues PRs and so on hosted on your infra, but everything interconnects nicely with your single account, just like using GitHub.
They even added a vouching system for users, to know who to trust based on who you already trusted.
sp4ke 8 hours ago [-]
At least this will wake up more people to the existing alternative efforts such as Forgejo and its progress with federation protocol.
I suppose people can use git too if they ever figure out how to send emails from a dedicated client.
MoreQARespect 8 hours ago [-]
There are plenty of alternatives for git hosting. The last thing which needs to be migrated before we get an exodus off this vibe coded monstrosity is reputation (i.e. github stars) which are inherently sticky.
dewey 8 hours ago [-]
I don't think GitHub stars are what makes GitHub sticky. Having a single UI / account and being able to jump between work projects, private projects and random open source projects where I want to open a quick issue or submit a PR is much more sticky.
mbreese 7 hours ago [-]
The reason why GitHub is sticky was exactly this. It’s easy to fork, make a change, and submit a PR to another project. It’s the same interface for your code vs another person’s. It’s a simple, consistent interface to submit an issue.
You can argue that the lack of diversity in infrastructure is bad, and I wouldn’t disagree. I think we should have more projects to explore better ways of working. But, the tradeoff is that when you have a centralized system, like GitHub, you also have less friction. The learning curve is small when everything uses the same tools.
ryandrake 8 hours ago [-]
Git doesn't even need to be "hosted" anywhere. That's the point of it being distributed. What needs to be hosted somewhere are all the barnacles (like issue tracking, code reviewing, CI, and so on) that have grown out around git, that people and businesses have become dependent on.
thayne 7 hours ago [-]
It doesn't have to be. But without a hosted "source of truth", it's kinda hard to actually distribute it, and have a way to accept pull/merge requests/patches.
jolaflow 2 hours ago [-]
In regards to issue tracking there are alternatives. Epiq is a git-native (vendor agnostic) issue tracker you self-host: https://ljtn.github.io/epiq/
seanc 8 hours ago [-]
Which is why they cleverly moved the central story of a software change from the commit to the PR. Commits migrate automatically with the repo. But migrating PR descriptions and comments takes engineering effort.
kortex 8 hours ago [-]
Rhetorically, why can't folks add the feature of PRs to Git? Git is fundamentally just a graph manipulation engine and data structure. It would just be another object type, in a layer which sits above the normal commit graph. Obviously there is no "just make X" in software, but this feels reasonably achievable, and would have tremendous benefit for the cost.
Where would you put comments then? Sure, there’s always mailing lists, but even so, someone still has to host Mailman.
I do think GitHub has positively contributed to the software industry and open source. It’s been enshittified for sure, but it has had a golden era.
jayd16 7 hours ago [-]
LFS files sorta do need to be hosted. At least, it exists because full clones are untenable for many projects.
NetOpWibby 1 hours ago [-]
My main reason for using Github is stars (bookmarking)
hosh 8 hours ago [-]
I think as we see more AI participation in code, github stars are going to matter less.
zer00eyz 8 hours ago [-]
Github stars are a metric of what?
If I build something that satisfies me and say 100 users (who all star me) does that really say anything. Are those 100 stars even "real" any more?
All the old problems of the web are new again, and still remain unsolved.
hugs 7 hours ago [-]
github stars are a metric of investability.
some investors use stars (and star growth velocity) to determine which projects they invest or don't invest in. (not saying it's good or bad - just saying it's a real thing.)
graemep 8 hours ago [-]
I like Fossil, but there are good reasons to use git too.
There are many git hosting options, including some for self hosting. its a lot easier to switch to a different version control host than to a different version control system.
AlexandrB 8 hours ago [-]
> Luckily we already have a distributed version control system that can take over github
Surprised that this wasn't just a link to Git. The biggest benefit of Git over predecessors was that it enabled decentralization. Github re-centralized the common case for the sake of convenience.
jeremyjh 8 hours ago [-]
Yes but on Earth every project needs a canonical main branch. Most also need a way to manage and review changes to that branch that all project members have direct access to.
Sanzig 8 hours ago [-]
Gitea (and its popular fork, Forgejo) are very lightweight. You can easily stand up an instance on a cheap VPS.
jeremyjh 3 hours ago [-]
Agreed. Just moved some of my projects to codefloe.com (Forgejo) this weekend. FYI - it provides a lot more than a link to git.
skydhash 8 hours ago [-]
It's on the project to decide which is canonical and export a read only accessible version. Change review only requires a communication medium (the patches are mostly plain text) and that can be done in a lot of way. As for merging those changes, it's an organizational issue.
jeremyjh 3 hours ago [-]
If it’s on the project to determine their requirements, why are you doing it?
poisonborz 8 hours ago [-]
Github and other providers before proved that git itself is just a small part of managing code, there's a lot more needed for a full workflow, and it's meaningful for it all to be tightly coupled.
Xirdus 7 hours ago [-]
The biggest benefits of Git over predecessors were branching, 3-way merge, and history rewrites (in particular, rebase and squash). As much as Git fans wanted decentralization to be a major selling point, decentralization has never been a major selling point.
prerok 6 hours ago [-]
Not exactly decentralization but distribution was the main selling point.
In comparison to SVN and other non-DVCs git did not need to continuously talk to the central server for history, "branch" creations, unlocking/locking files before changes, etc.
Linus wanted a DVCS, but they wanted centralization and licenses were expensive, so he decided to launch the git initiative where he provided scaffolding and the rest came from contributions.
AlexandrB 4 hours ago [-]
I think you understate what a big deal the decentralization was when Git came out. Back in the early 2000s, getting internet while out of the office was hard. 3G was in its infancy and expensive so you were at the mercy of finding a WiFi hotspot. Git's predecessors, such as SVN, were unusable without an internet connection to the server. Git (and Mercurial) allowed doing version control without an internet connection for the first time, which is a massive benefit for some types of development or doing work on the road. This aspect of Git definitely became less important as always-on internet connections became ubiquitous, but it's still super helpful sometimes. I can't imagine going back.
Arubis 8 hours ago [-]
> "Crypto had attracted grifters from the beginning, even though its most respected figures were more interested in changing the world. As Silicon Valley investors piled in after them, they wanted to fund companies that would become essential to the crypto economy, allowing them to make vast amounts of money. Entrepreneur after entrepreneur discovered that the best way to turn a profit in a decentralized economy was to figure out ways to centralize parts of it again." (Henry Farrell, Underground Empire)
rbanffy 6 hours ago [-]
We had Launchpad before Github (IIRC).
duxup 8 hours ago [-]
And like Twitter most people will keep using it anyway?
OtomotO 8 hours ago [-]
Just in the US. I have ONE European client... ONE of dozens who use GitHub.
EVERY other client I've ever worked with used their own gitforge or at least bitbucket...
nixgeek 8 hours ago [-]
Selection bias then as the majority of European companies are for sure not hosting their own Forgejo or GitForge.
mechazawa 8 hours ago [-]
Where I work we're hosting our own gitlab instance and the previous company I worked for hosted their own Bitbucket. For most developers I know it's either a self hosted git platform instance or Github enterprise.
7 hours ago [-]
OtomotO 8 hours ago [-]
I've asked around in my circles, because I am genuinely interested: Everyone not working in startup space IS hosting their own instances.
Hell, even I do it.
8 hours ago [-]
xigoi 8 hours ago [-]
Even my faculty, which otherwise licks Microsoft’s boots at all opportunities, uses self-hosted GitLab.
pelorat 7 hours ago [-]
Github pricing is aimed at US companies. Small EU companies that can only afford to pay their 5-6 developers 60K euro per year rather selfhost than pay GitHub.
lxgr 8 hours ago [-]
It is, but a single point of failure is itself also an unstable equilibrium, given that a lot of things have to continuously go right for its continued existence in terms of governance of all kinds (technical, moderation policies, commercial/pricing etc.)
8 hours ago [-]
dimgl 8 hours ago [-]
Twitter is unstable? I can't remember the last time it had an outage.
sunaurus 8 hours ago [-]
Are you potentially in the US? I think the experience is very different globally.
Ever since the Twitter -> X rebranding, every time I open Twitter links, I have a very high chance of hitting a "Something went wrong" type page, with a retry button that just does nothing. It's been like this for ages. I barely open Twitter links anymore as a result.
obeattie 8 hours ago [-]
I think they may mean Twitter's 'Fail Whale' era – late 2000s/early 2010s, when it regularly fell over under load.
BryanBeshore 8 hours ago [-]
~2007-12?!
8 hours ago [-]
logicchains 6 hours ago [-]
It's becoming the Microsoft Teams of Git hosting providers.
watwut 7 hours ago [-]
I think that especially when it comes to hosting, competition is good. Everything centered on github was the bad state previously. I remember people on HN claiming they wont hire someone without github account.
This splintering should happen more often.
CivBase 7 hours ago [-]
Good. Git is a decentralized version control solution. It's built for this. The centralization around GitHub was beneficial for the initial proliferation of Git, but has long since become a net negative. The "community" you speak of was never bound to GitHub. Better alternatives have emerged and for many it's time to move to greener pastures.
8 hours ago [-]
icepush 8 hours ago [-]
I think that, exactly how 95% of the people that left Twitter eventually returned, we will see a similar outflux of people from Github and the vast majority of them will quietly be back in a few months.
mort96 8 hours ago [-]
Do you have numbers to back that statement up? I don't necessarily doubt it, but I haven't seen anything to support it.
Note that raw user numbers won't really cut it; if 1 million people left Twitter, it's possible that 500 000 of those people joined and 500 000 new people joined; the user numbers would look like everyone returned, but really only 50% did in this hypothetical.
TazeTSchnitzel 7 hours ago [-]
Twitter is a shadow of what it once was. The design of the site has shifted very very hard in favour of being an algorithmic content soup rather than a social network in the traditional sense. This has juiced the engagement numbers but there's a lot less meaning to them now. (I can say all this because I am a bitter holdout who still tries to use the site to have conversations with people and form personal connections. Tragically it still offers something that Bluesky and the Fediverse do not, even if that thing is slowly but surely being smothered out.)
xpct 8 hours ago [-]
I'd say about a third of software engineers whose blogs I stumble upon and end up liking aren't active on twitter. Their accounts typically have nothing posted for 3-4 years, but they're active on bluesky or mastodon.
giaour 8 hours ago [-]
Was this reported and quantified somewhere, or does 95% reflect your experience? My experience was the opposite -- everybody I cared to follow left and never went back -- but I'd be willing to believe I'm in the 5%.
kakwa_ 8 hours ago [-]
Both are not comparable as the stickiness dynamics are completely different.
On Twitter, it's mostly the networking effect, you want to be where peoples are, and unless a large chunk move at once, you are stuck there.
On Github, it's the technical cost of migrating, the git part is trivial, but tickets, PR history and CI/Integration are another can of worms.
Projects which moved away, moved away for good, at most only leaving a synchronized mirror.
CSMastermind 8 hours ago [-]
There was nothing wrong with twitter, it was a social fad to performatively leave.
Github is actually down and I can't push code right now, it's blocking my business which is unacceptable. It's a broken tool.
jurassic 7 hours ago [-]
X’s revenue is down by 2/3 from the acquisition peak. That’s all you need to know about whether the decline was real or not.
xnorswap 8 hours ago [-]
That was a political decision, this is an operational one. A lot of companies have become dependent on a working github and effectively have to down tools when it is down.
icepush 8 hours ago [-]
Being down for a few hours 2-3 times a month is obviously garbage from a reliability standpoint, but I think they will keep it going at just the level necessary to barely hold on to these customers.
insane_dreamer 6 hours ago [-]
> how 95% of the people that left Twitter eventually returned
and this assertion is based on ... ?
tstactplsignore 7 hours ago [-]
this varies by field, most scientists have left twitter for good
bewareofscams 8 hours ago [-]
Fuck the community, nobody needs that, especially in a version source control system. Thinking otherwise is just dweebs' eternal longing for belonging, or someone has some bullshit to sell to the unsuspecting dirty masses.
athoneycutt 11 minutes ago [-]
I'm at the point of having my own stuff on GitLab and mirrored to my own Forgejo or on Codeberg and mirrored to my own Forgejo. I really like GitLabs CI better than GitHub/Forgejo tho.
I only really use GitHub if a project that I want to contribute to is already on there at this point.
Nice - even their own action runners are getting rate limited atm
dboreham 3 hours ago [-]
No fair! You got a runner.
beardedetim 9 hours ago [-]
We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?"
No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it.
> We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?"
Did you have the conversation "do we need to move from $whatever CI/CD to GitHub Actions?" beforehand? What were the arguments in favour?
Macha 9 hours ago [-]
Not the same company but similar discussions here:
- It was believed that GitHub would be more reliable than our self hosted Gitlab CI runners. This has proven to be very false.
- It was believed it would be easier to write tooling. I’m not on the dev tooling team, so can’t confirm if they found it true or not, but from the outside I see a lot more “Github can’t do that” than I did “Gitlab can’t do it”
Normal_gaussian 8 hours ago [-]
Those are both hilarious to me. In my last FTE role we had the exact opposite beliefs - we wanted to leave GitLabs increases in pricing and move to annual sub, but the reliability of self-hosted runners and the flexibility (and clarity) of the CI system was fantastic.
dataengineer56 8 hours ago [-]
> Those are both hilarious to me.
Why is it hilarious to believe that the industry leader might be better at something than doing it yourself?
Normal_gaussian 8 hours ago [-]
The hilarious part is more the absurdity and cognitive disconnect experienced when reading about serious evaluations that went in a completely opposite direction to those directly participated in.
8 hours ago [-]
beardedetim 9 hours ago [-]
It's a ~year old product that started on GH. So this is the first time this team has had to think through "okay, so this isn't as reliable as it was at $oldjob."
layer8 9 hours ago [-]
Self-hosted GitLab at work, self-hosted Gitea at home.
lsuresh 9 hours ago [-]
We use self-hosted Github runners, but it's a moot point if all of Github is mostly down as is the case right now. :\
mitchjj 8 hours ago [-]
Worth being upfront with, I'm an employee, but; Buildkite.
Frontier labs and global saas heavyweights run CI through us, plus a crop of open source tooling like Bazel, vLLM, Firecracker, Rails, Ray, Sorbet and Rust.
Shopify, Uber, Airbnb, Canva stresstested the platform ten years ago for scaling issues.
You need to take a little time to design upfront and take advantage of dynamically programmable primitives. It isn't the same ease/convenience as Actions. Bit more available though.
maccard 2 hours ago [-]
I want to second build kite - it’s an excellent piece of tech and is best described as “sane Jenkins” as a compliment. My only gripe is the pricing - it’s not designed for “small teams”. But I’d use it again in a heartbeat.
tom1337 7 hours ago [-]
How would I be able to run my CI via Buildkite when I still host in GitHub and their API is down? We are currently already using self-hosted runners and they also don't work because either they aren't started by GitHub or can't download the code.
maccard 2 hours ago [-]
We ran a mirror of our git repo on codecommit when things started to get really hairy, and pointed out CI to that instead of GitHub. It’s still a moving target though as we were using GitHub releases to publish to. We replaced that with an s3 bucket and mirrored _those_ to GH via actions, which had the added benefit of only failing when GitHub was down!
nerdypepper 7 hours ago [-]
ultimately, folks need to stop writing massive YAML files to define their builds. CI should schedule jobs, your build system should define them. my workflow files look like this:
steps:
name: Do the thing
run: nix build .#
at any point, if my CI is down or i need to switch or i need to run builds locally, i can do so trivially.
maccard 2 hours ago [-]
This is all well and good until you need a cache, or credentials, or to pass data between steps, or to parallelize across runners.
jon_adler 3 hours ago [-]
My CI runner just calls make targets, that can easily be run locally if needed.
If you're seriously looking at Gitea/Forgejo, I run Fjord, dedicated managed Forgejo, so you don't have to operate it yourself. We can help with migration and runners too. We also run a free public Forgejo instance if you just want to kick the tires first.
pojntfx 6 hours ago [-]
Thank you for your work on Fjord! We're currently evaluating it vs. self-hosting, the comparison at https://fjord.sh/compare has been very useful
stephenway 5 hours ago [-]
That's really good to hear, thank you. If you run into anything that's unclear while comparing the two, I'd genuinely like to hear it. Happy to answer questions here too.
jscd 4 hours ago [-]
It's not the most elegant thing ever, but I've been using (self-hosted) WoodpeckerCI and it's been dead simple. It doesn't have many bells and whistles, but honestly I think that's good. Forced me to put complex behavior in better systems as opposed to doing bash scripts in YAML.
qweqwe14 9 hours ago [-]
Self-hosted GitLab and self-hosted GitLab Runners (Omnibus Docker image)
What's your experience with tangled? Would love to hear the experience from other users as well
MarkBennett 7 hours ago [-]
It's worked great for hosting my smaller personal projects so far, and I've already received a few helpful pull requests from the community.
Tangled.org really been working to ensure all their features work from data hosted on your personal data server (it's based on ATproto) which means should their performance, reliability, or monetization ever get you down it's easy to switch to a competitor since you're not locked in.
classified 9 hours ago [-]
I'm completely untangled. From GitHub.
speleding 8 hours ago [-]
If your app is not too large you can do CI/CD perfectly fine from your laptop. Put it in a docker container if you want to ensure it is perfectly reproducible. You still need Github as the origin repo to pull from, but the "git part" of GitHub seems to be (almost) always up and running. (We use "capistrano" for this)
cesarb 8 hours ago [-]
> If your app is not too large you can do CI/CD perfectly fine from your laptop.
Not if you want to make sure it compiles and works on Windows (which you don't have) or macOS (which you don't have and can't emulate). That's something github gives for free, and for me, it's the only reason to stay on github for small/tiny free software projects (at least as a secondary host).
drno123 9 hours ago [-]
Self hosted Gitea
newswasboring 9 hours ago [-]
When you self host, what happens with your compute costs? CI can be super expensive depending upon your test pipeline. Do you end up spending more than you would with GitHub actions for example?
JoshStrobl 9 hours ago [-]
Just get some x86 and ARM runners off netcup, manage Woodpecker CI + Woodpecker Agent. Generally pretty hands-off once you get it set up. That plus a cheap VPS running Forgejo with postgres off-site backups, you are looking at a small fraction of the cost compared to per-seat GitHub pricing for orgs + Action Minutes.
Maxion 9 hours ago [-]
Huh? The VPS boxes have a fixed monthly fee
applidhumanco 9 hours ago [-]
He is likely referring to the CI runners. Gitea has their own version of Actions runner. It is very much usable, though currently inferior to GitHub Actions (not always a 1:1 replacement, even though they aim for it to be):
https://docs.gitea.com/usage/actions/
Cost as always depends on your usage, but it will likely be less than your current cost of GitHub Actions.
newswasboring 6 hours ago [-]
Thank you. This is exactly what I meant. I work for a large company where our testing budget is massive and most of it goes towards these CI runners. Provisioning enough and consistent compute for testing pipelines sometimes becomes the bane of my existence.
Maxion 2 hours ago [-]
I can recommend that you self-host Jenkins on another box.
dcan 8 hours ago [-]
Self hosted Gitlab at work with local runners
latchkey 8 hours ago [-]
blacksmith.sh just raised tens of millions on the core premise that self-hosting CI/CD is hard [0].
they've had multiple outages in the last week, so i guess they are proving themselves correct.
in practice, giving ssh/sudo to AI (codex) and telling it to setup ephemeral self-hosted runners on your own host VM (qemu), takes about 30 minutes and works extremely well. it bangs out all the shell scripts. moved my own business over to it. zero problems.
you might even find it is kind of neat too. you can customize your VM with software pre-installed. speeds up your builds quite a bit.
I remember in 2024 passing over codeberg as the host for some of my personal projects because of the uptime. I don’t think Codeberg’s uptime has gotten any better but despite that it’s getting a lead over GitHub now.
peri-cl 9 hours ago [-]
Ah, so that's what the agent meant by summarizing a documentation page as "Unicorn!" I'd thought it was a very colorful hallucination (it's just GitHub's (unicorn-themed) failure page).
Gathered details on embedding request syntax from
Unicorn! · GitHub
I was querying docs to figure out how to set up RAG to query docs locally. Alas.
abluecloud 8 hours ago [-]
i'm surprised the agent didn't take into consideration the http status code
gentlewater 8 hours ago [-]
What’s the code for http unicorn? /s
arccy 6 hours ago [-]
There's one for a teapot, surely there can be one for a unicorn
klaussilveira 7 hours ago [-]
I encourage everyone to switch to Forgejo. We ditched GitHub and could not be happier. The experience is smoother, faster and distraction-free.
Our runners are faster, our CI is smoother than ever. Transitioning was a matter of a few hours, which we took as a fun hackaton.
For which team would be a plain git server enough to manage a codebase?
fatal94 3 hours ago [-]
The Linux kernel team
ivanjermakov 3 hours ago [-]
GitHub is a git server as a service.
wolttam 3 hours ago [-]
+ issues + pull requests. There’s some nice to haves in there, that you’d need some other service or convention to manage
sebmellen 7 hours ago [-]
How complex is the CI migration? We’re heavily dependent on GitHub Actions and the full GitHub ecosystem by extension, but by the same token these frequent outages are destroying us.
nrabulinski 7 hours ago [-]
Forgejo actions are by design compatible with GHA. Obviously if you still depend on github to get reusable actions you’re just as dependent on GitHub being up as ever, but you can very easily migrate or mirror GitHub repositories to your forgejo instance and in most cases migrating the CI requires close to no changes. Not all features are supported, but I haven’t had the need to make any changes to my actions, and they’re continuously making improvements so compatibility only goes up
sebmellen 7 hours ago [-]
How about ticketing? We use Linear and their integration with GitHub is pretty buttery. Willing to give it up for better service stability, but wondering if Forgejo offers any kind of native ticketing integrations. I know GitLab does, but I'm not massively impressed with GitLab either.
nrabulinski 6 hours ago [-]
I know at least part of the API is fully compatible with GitHub so chances are as long as you can specify custom URL base and auth token, there’s a chance linear and other integrations will “just work” too. E.g. I had migrated a custom action which interacted with the PR it was run inside of using the GitHub JS client, and I didn’t have to change a line.
https://fly.io/ - it runs the machine and serves the traffic
rglullis 8 hours ago [-]
I'm self-hosting Gitea and Woodpecker CI for least 6 years, alongside my own docker and pypi registries. Adding x86 and ARM runners, it costs me circa $30/month to run the whole thing, and I am reasonably sure that I could make the whole thing scale at a lower cost per user than what Github charges.
I am seriously, 100%, contemplating the idea of offering consulting for migrating companies and projects out of Github. Given that I am fully employed, I'd only be able to do this on the weekends. Could I get a show of hands to get an idea of potential customers?
BowBun 8 hours ago [-]
I work at a startup with ~20-30 devs. We're talking about a project like this. I'm not sure weekends would work for an actual company, though maybe it depends. I think there would be an appetite for this. From my POV - the main areas of interest are how you can roll this out to a team gracefully without breaking everyone's workflows for multiple days. Good luck
rglullis 7 hours ago [-]
The most natural way to roll this out would be to have a "blue/green" system, where both systems are running and the weekends would be the time to take the steps towards the migration. First set up the servers, integrate Auth/SSO (depending on the current company set up), etc. That could be done probably in a single weekend.
Once this is done, the team would have two possible git remotes to work with. Given that gitea also allows mirroring with Github, the team would be able to even switch to Gitea as the "main" git remote and for PR reviews even if the CI part is still on Github.
The harder part would be migrating the CI jobs. But this also can be done in parallel and no one needs to cut out their Github actions immediately. Start with the simplest of the stages, e.g, the automated test suite, and once you get the whole pipeline from Woodpecker on an staging environment, you'd could take another weekend to test the whole workflow. Once the first project is successfully migrated, it becomes a matter of repeating the process.
Woodpecker is still a bit finicky and debugging it can be a bit annoying, but given that it's all based on docker it is a lot easier to debug a pipeline locally.
stephenway 5 hours ago [-]
This is exactly the migration problem we're trying to make boring with Fjord. It shouldn't require a flag day, bring up Forgejo alongside GitHub, mirror while the team moves over, then migrate CI incrementally.
For a 20–30 dev team I'd be happy to help work through what that rollout would actually look like
ramon156 8 hours ago [-]
Doing the same thing, and I'd join you in this consulting haha. It's a breath of fresh air to know what your CI is actually doing
herpdyderp 8 hours ago [-]
Sounds interesting but I'm just going to do the same thing as you (self host).
ieie3366 9 hours ago [-]
To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code
Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside.
If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health
askonomm 8 hours ago [-]
That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.
datsci_est_2015 8 hours ago [-]
It is interesting, how much money is being lost during this outage?
My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced out by end-of-year due to incompetence. The new executive has taken to firing employees much more eagerly than their predecessor, like some sort of Jack Welch acolyte.
Their firing has put a lot of things into perspective for me. Mostly, fuck "at-will" employment and its negative effect on the American social contract.
But also this "angry ranting" online that the original poster was referencing. Not everyone has the privilege to calmly respond to things that directly impact their livelihood.
theappsecguy 7 hours ago [-]
Any engineered system where an individual can accidentally cause a 7 figure outage is poorly designed. And engineering leadership that decider to terminate an individual due to such failure (as long as there was no malicious actions) is completely clueless.
starkshift 7 hours ago [-]
Captain obvious over here
TwelveEyes 4 hours ago [-]
Is responding calmly a privilege to have, or a kindness to yourself?
mplewis 4 hours ago [-]
don't do this
rkapsoro 5 hours ago [-]
A story as old as time.
pkilgore 5 hours ago [-]
Name names.
datsci_est_2015 4 hours ago [-]
Local (to us) firm. Two main reasons I didn't:
- Have never seen it mentioned on this forum, in any context.
- We're potentially pursuing legal action.
red-iron-pine 2 hours ago [-]
names or it didn't happen, mate
50% of social media traffic is bots and i want to believe
Waterluvian 8 hours ago [-]
I'd be curious what the statistics might actually be for people who are directly affected because their business is suffering vs. people affected because their employer's business is suffering.
6LLvveMx2koXfwn 8 hours ago [-]
Most people's employment prospects are directly correlated with their employers ability to make money.
john_strinlai 8 hours ago [-]
most people's employers arent firing people over a few hours of github outage.
not to mention that any business which could potentially lose enough money that they would need to let go of developers from a github outage should probably already have some business continuity plans in place.
polycaster 8 hours ago [-]
For Germany it's roughly 1:11 if you go by the self-employment rate (~8% of the workforce).
vaylian 4 hours ago [-]
> For a lot of us GitHub is critical infrastructure
GitHub Enterprise Cloud has been chugging along with no issues. I hope your critical infrastructure isn't dependent on a free tier / service. And that you have a business continuity process in place.
946789987649 7 hours ago [-]
Are you sure? We have enterprise and I couldn't even access our repo briefly.
nullify88 6 hours ago [-]
We have data residency in the EU and didn't notice anything.
roastedfunction 5 hours ago [-]
GHE.com has only been around since November 2024. Most GHEC customers I imagine are still stuck on the shared global infrastructure on github.com.
antaviana 8 hours ago [-]
"It's just money. It's made up. Pieces of paper with pictures on it so we don't have to kill each other just to get something to eat". Jeremy Irons in movie Margin Call
faeyanpiraat 8 hours ago [-]
You cant eat a quote
red-iron-pine 2 hours ago [-]
but you can download a car
parthdesai 8 hours ago [-]
How is github being down losing you guys money?
dewey 8 hours ago [-]
If you pay developers x money / day and one of their core tools is down for n hours during the day and they spend their money on HN instead that's pretty straight forward to calculate.
parthdesai 8 hours ago [-]
I meant more for their customers specifically
red-iron-pine 2 hours ago [-]
we push customer code on mondays. it is monday. code is not being pushed.
we do not bill.
the outages they promised their customers are now different.
this has costs for everyone downstream.
aurareturn 8 hours ago [-]
I was in the middle of a hot fix. Our pipeline goes through Github.
parthdesai 8 hours ago [-]
That's understandable, but surely there's a way to bypass it? You need to have a breakglass procedures lol
penultimatename 7 hours ago [-]
“lol just use a workaround” doesn’t work in an environment with hundreds or thousands of employees coupled with audit, security, and other legal requirements to ship software.
parthdesai 7 hours ago [-]
If you're actually bleeding money, you better believe you'll get permissions for a workaround, if you know what you're doing.
It all ties back to the OP, where the issue you've might not be as bad as you think. I have been in situations where we have dropped all procedures to push a hot fix because we were actively bleeding money, and in situations where you know there is an issue, and you let it be.
penultimatename 5 hours ago [-]
What an odd hill to die on. Maybe I just lost a $30M client because they wanted an issue fixed now. Maybe my company would get sued for violating their contractual obligations if someone went cowboy to bypass procedures.
Maybe I pay for a service and I want that service to work consistently during core business hours.
bmitc 5 hours ago [-]
"Just recreate GitHub internally."
red-iron-pine 2 hours ago [-]
"just redo the whole CI/CD pipeline bro, it's easy"
phoronixrly 8 hours ago [-]
Imagine Github being critical infrastructure for you... The ineptitude...
parthdesai 8 hours ago [-]
We use github quite a bit as well. Github being down currently is not actively losing us money, i.e. having customer impact.
kakacik 8 hours ago [-]
If its so critical why relying on it, and not having ie some mirror or some other way to handle any sort of outage like this. its not like Microsoft is your friend or good business partner, ever.
With every single of these enterprise 'cloud' offerings you are giving (almost) complete power over your business/project to somebody else who couldn't care less about your success or failure, you are simply irrelevant for them. I see it at work too, every time critical external systems go down whole bank stops still, just because few bucks were saved yearly on some on-prem servers.
Look at it this way, you are learning some important lesson today and finding great area of improvement for resiliency from now on.
AlexandrB 8 hours ago [-]
Boomer opinion but trusting third parties to be critical infrastructure, especially with no SLA in sight, will always end in tears. "The cloud" is very convenient, but its providers will never care about your infrastructure or your customers as much as you will.
Maxion 8 hours ago [-]
Older millennial here and I agree. If you rely on Github you should at least be have your processes such that you can work around it.
whynotmaybe 8 hours ago [-]
That's the reason why my build agent is self-hosted
otterley 8 hours ago [-]
3P-maintained infrastructure is what makes civilizations work efficiently. We're not all digging our own wells, generating our own electricity, and burning or burying our own garbage.
kalaksi 8 hours ago [-]
You are still supposed to be prepared for outages.
otterley 7 hours ago [-]
Agreed. So is the issue really then that people were inadequately prepared with backup plans and now they're suffering the consequences?
It's not all that different from, say, an AWS region having a service impact. People would rather complain about AWS than prepare and utilize a well-tested recovery plan to shift to a standby region. Oftentimes there's no fallback plan because the business already considered it and decided it was too costly relative to the benefit, but when the incident happens, they still can't help but complain. Humans being humans.
advisedwang 3 hours ago [-]
Eeh, you always rely on someone else's infrastructure. Even if you are off cloud entirely and own your own datacenter, you still need peering/transit. You still need power (or at least fuel). You can't avoid depending on a DNS infrastructure.
Boomer here as well, but I'd add that trusting your own org for critical infra usually also ends in tears. Most everything in IT involves failure, including in well designed systems designed by great engineers.
I worked for a few years in an exceedingly well capitalised place which ran everything in their own data centers, money no object, with a truck parked somewhere, ready to go, with a smaller version of our critical infra. We had a serious business-stopping outage once every 18 months or so, every time for fringe reasons one only learns about when trying to run a large data center. Its convenient to blame the cloud and pretend that self-hosting in private sector was so, so great with six nines.
6031769 8 hours ago [-]
At this point github is barely managing one nine.
aenis 8 hours ago [-]
True. I'd be very annoyed if I had anything mission critical on github. While we have everything in the cloud, we do actually self-host that part.
swills 7 hours ago [-]
Agree. But also, it's affecting everyone equally, whether they have a free personal account or are part of an enterprise account with SLA. Understanding the practical value of an SLA is an interesting problem.
raffael_de 8 hours ago [-]
+1 (as a millenial) ... especially given that setting up a git server for non-OSS company code isn't too much of a challenge really. also, no need to self-denigrate this reasonable opinion in preemptive obedience.
otterley 7 hours ago [-]
GitHub is way more than just a git repo host. It manages code reviews, merge (pull) requests, and has an entire CI/CD workflow engine in it. Replicating all that is a challenge that most orgs are not up to.
ethagnawl 8 hours ago [-]
You're right.
However, I'm also of the boomer opinion that you should get what you pay for. "Ranting online" about a service (you pay for) being unavailable is a reasonable reaction. It's not like they have a call center you can dial into for support ...
poisonborz 8 hours ago [-]
How did this become a boomer opinion? It is proved truth thousand times a day. Not that you shouldn't use third parties - but in this industry you can shrink this exposure to the minimum, and have plan B for anything else.
SideburnsOfDoom 8 hours ago [-]
People are already saying things like "We need a plan B in case we urgently need to deploy a fix to production, and GitHub Actions is unavailable again".
But in general, it's not feasible to do everything in house.
The issue is that they're not achieving two nines uptime in practice.
TZubiri 8 hours ago [-]
Boomer who has never had a business? Depending on third party suppliers is business as usual.
pigeons 7 hours ago [-]
Well it's not business at all today!
TZubiri 5 hours ago [-]
Arguably, git is decentralized, so contingency plans, like working with local and server repos, should still work.
BryanBeshore 8 hours ago [-]
Truly a boomer opinion. Respect!
TZubiri 8 hours ago [-]
And it's not just the user's fault, GH spent a considerable effort in marketing to position themselves as such, see Github Actions and similar junk.
jjice 8 hours ago [-]
> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code
I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory).
I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain.
You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org.
They _are_ the LLM coding agent vendor, and are _owned_ by MS, the ~majority~ biggest shareholder of OpenAI. How can you NOT consider that 20x+ scaling in your capacity roadmap projections, where you are trying to get everyone to use these agents as part of your core OKRs?
Or, maybe your #1 IT priority was moving everything to Azure instead ;)
rafram 8 hours ago [-]
> the majority owner of OpenAI
Largest shareholder (27%), not majority.
chilipepperhott 7 hours ago [-]
Plurality should have been the operative word.
debarshri 8 hours ago [-]
It looks like right after Microsoft acquisition this started going off.
This is the outcome of violating single responsibility principle in business.
Topfi 8 hours ago [-]
Github being under the CoreAI division probably also doesn't help the engineers prioritize addressing infrastructure issues and makes using LLM load as an excuse feel self-inflicted. Akin to feeling sorry when a pyromaniacs house burns down...
chrisjj 8 hours ago [-]
They've had a couple of months and still have not brought in sufficient extra capacity?? What's their excuse?
eddieroger 8 hours ago [-]
For the amount of money enterprises are paying to GitHub, there is a reasonable (and contractual) expectation of uptime. I don't think my boss would find it to fun if I bailed work to hit the gym just because GitHub was down.
Edit: it seems GHE is also affected by the outage and isn't much better.
cyansmoker 8 hours ago [-]
As an enterprise user I can tell you that:
- we are getting no benefit beyond getting excuses replies to our emails
- right or wrong, but we depend heavily on GitHub actions so this has a very real impact on us
twistedpair 8 hours ago [-]
Since upgrading to Cloud Enterprise... why didn't my GH uptime get any better?
Guess on-prem GHE server is the only way to do that.
joeyhage 8 hours ago [-]
My understanding is the us.githubstatus.com is only if you are using GitHub enterprise with a custom subdomain and data residency in the US because then *some* of the infrastructure is separate from GitHub.com.
mh- 8 hours ago [-]
[dead]
eddieroger 7 hours ago [-]
Yeah, I know, and it's down.
tedivm 8 hours ago [-]
This, like all of their dashboards, is complete bullshit. As a GitHub enterprise user i am absolutely affected by the outage today, and have been affected by all of the GitHub Cloud outages.
There is no separate enterprise platform unless you buy their self hosted server product.
8 hours ago [-]
leishman 8 hours ago [-]
> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code
Which means they have a pricing problem. Rate limit non-paying users.
xnorswap 8 hours ago [-]
Just take away free Actions and it would surely solve a lot of their scale issues.
It's absurd that I have dozens of repos, many with GH Actions that run CI, test and then package and push to prod/package managers, and haven't paid GH anything.
anoplus 8 hours ago [-]
Right. It is not an excuse and they were notorious for outages before LLM era. Also, LLM can serve Github engineers as well, so we are all playing in the same field.
cluckindan 8 hours ago [-]
They already do
rzzzt 8 hours ago [-]
I can't check the commit history of some OSS projects without hitting a rate limit if I'm not signed in. My request rate is one request per (arbitrary time interval) at that point.
Actually, I do have an important fix for a deal we are trying to close that does need to go out. And I am paying GitHub to host this. Once or twice, I can see, but GitHub's SLA is getting worse than just hosting it myself, and that's the entire reason I pay GitHub.
sarjann 8 hours ago [-]
This has to be rage bait, this is a critical piece of infrastructure for many people. Stuff going down can lead to deployments failing and as you mentioned in many cases emergency hotfix's.
The idea that we should be fine with this unreliability is just amazing. It's not a mental health issue to have problems when important infra fails.
john_strinlai 8 hours ago [-]
>The idea that we should be fine with this unreliability is just amazing.
the comment doesnt say you should be "fine" with the unreliability.
they are saying people shouldnt get so emotionally worked up over it. which, while i wouldn't phrase it in the way the parent did, i agree with the direction of their point.
sarjann 8 hours ago [-]
Except he tries justifying it.
"Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code"
"Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside."
with assumptions like above.
If it was, "hey they f*ed up but there's no point having an overly emotional reaction" that would be fine. But he seems to be justifying this. Especially with Github's record up to now of unreliability I think it's completely fair to be annoyed.
world2vec 8 hours ago [-]
Geez, sorry that the paid service my company paid for is down and I can't do my work.
constant_flux 8 hours ago [-]
GH shouldn't have promoted LLM usage the way they have if they didn't have the infrastructure to support it. Regardless of how you feel about the code quality (which you have zero evidence of), if GH has made commitments to supporting broad LLM usage, they need to back that up with the proper hardware and without whatever fragile SDLC processes they have.
pamcake 8 hours ago [-]
> GH shouldn't have promoted LLM usage the way they have
No qualifiers necessary. But try arguing with a lawnmower...
madeforhnyo 8 hours ago [-]
> due to LLM sloppers pushing large amounts of trash code
Which they encouraged by pushing Copilot down everyone's throat
bhouston 8 hours ago [-]
> To everyone who is angry: calm down. Github’s servers are constantly on fire
Is this rage bait? Isn't Google, Amazon, and all the other services in the world similarly impacted by LLM's? Isn't Claude, OpenAI, etc.? Why can they handle the load but not Github?
anonymousab 8 hours ago [-]
Allowing and accepting the LLM load is a willful choice they made and are making at the expense of their users, including their paying enterprise users.
They could easily tighten things up in that regard, and make a choice that is right for their main users at the expense of The MS corpo mandate/mission. It is a choice to do otherwise.
pamcake 8 hours ago [-]
Allowing and accepting? More like encouraging and instigating.
s_dev 8 hours ago [-]
What if we're not angry and just perplexed? Do we calm down further or can we still ask questions?
ethagnawl 8 hours ago [-]
No questions. Only bench press.
bluerooibos 3 hours ago [-]
> calm down
If people and companies are paying for critical infra they expect it to be available - it's not MY problem that their servers are on fire - last time I checked, having too much demand was a good problem to have and they've had over a year to fix it and come up with a plan.
I'm so glad I migrated to GitLab months ago.
malfist 8 hours ago [-]
Why do you get to tell me how I should feel?
AgentK20 8 hours ago [-]
what if I do have an emergency hotfix for a production outage though ;-;
armSixtyFour 7 hours ago [-]
Hard disagree, people are paying GitHub a lot of money.
If they are facing an influx of commits from bots maybe they need to start metering commits from those accounts and charge them money.
This is a self imposed problem.
_joel 8 hours ago [-]
It's every other day now. We pay for a service, we expect service.
DharmaPolice 8 hours ago [-]
This might be naïve but wouldn't the appropriate response be to reduce access/rate limit free/new accounts in order for service to be maintained for everyone else?
kilroy123 8 hours ago [-]
As a _paying_ customer, I disagree.
stetrain 8 hours ago [-]
> as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code
Github and their parent company are active participants in pushing LLM-driven coding.
phtrivier 8 hours ago [-]
I understand the need to level-head the discussion and calming everyone.
However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow.
The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available.
(I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)
fn-mote 6 hours ago [-]
It's past time to move your critical infrastructure off Github then, no?
phtrivier 6 hours ago [-]
It's not exactly done in a pinch, but yeah, I guess someone must be working on disrupting CIs by the audacious value proposition of "we'll work when you need to".
AustinDev 8 hours ago [-]
Alternatively, don't use GH for critical infrastructure they are no longer reliable, simple as.
slowin 8 hours ago [-]
Well, they’re a critical piece of infrastructure with terrible stability. I’m in the process of migrating us off GitHub now.
I’m not a Meta fan but it’s interesting that they manage to keep their systems up with an order of magnitude more traffic. GitHub’s uptime is inexcusable.
raincole 8 hours ago [-]
Or, you know, it's perfectly reasonable and natural to feel angry when a service you paid for gets worse over time.
> Unless you have a emergency hotfix (you don’t)
Oh, we do. Given the sheer number of users, it's almost guaranteed someone is on fire ever time GitHub is down. Statistics is a very charming branch of reality.
robofanatic 8 hours ago [-]
> constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code
I find it hard to believe that github APIs don't have rate limits to handle traffic spikes.
javier2 8 hours ago [-]
While I agree in principle, Github with their CI is very critical to many parties.
Topfi 8 hours ago [-]
No one ever has to commit anything time sensitive.
delduca 8 hours ago [-]
> increased something like 50x due to LLM
This should not be an excuse since they probably can use AI to fix it /partially sarcasm
semiquaver 8 hours ago [-]
Why would you take what they say at face value? I believe that this is basically a lie covering up deeper rot.
bradhe 8 hours ago [-]
> due to LLM sloppers pushing large amounts of trash code
hey bud your bias is showing
bmitc 5 hours ago [-]
If we all go to the gym every time GitHub is down, we're all gonna get ripped. GitHub was down all day just last week.
They get paid millions of dollars by my company. They need to get it together.
thejosh 8 hours ago [-]
it's been trash for years, their status page just doesn't reflect that.
OtomotO 8 hours ago [-]
You know, I am not employed.
I have little time in my day to spend with my daughter. But I also have responsibilities. And one client has decided to bet on github. It's the only client I've ever worked with who has their code on github. EVERY other client has hosted their own gitforge or used bitbucket.
So now I am not angry because some critical piece of US-american infrastructure is down all the time.
I am angry because instead of spending time with my daughter I have to work on this later, because there are due dates and "Well, fucking GitHub was down" ain't gonna cut it.
8 hours ago [-]
c2403 8 hours ago [-]
In this case, perhaps they shouldn't allow AI slop or promote usage of AI agents...
nerdix 8 hours ago [-]
Wouldn't be surprised if we lost free private repos because of the slopocalypse. Though most don't seem to be shy about sharing their slop with the world so I'm not sure if that will actually increase stability.
First, that source is unreliable, particularly for before the acquisition. (We don’t know whether the data is accurate, it’s based on GitHub’s own self reporting.)
Secondly, the scale is WAY off and makes it look far worse than it is. It makes it look like 99.5% availability is practically zero availability.
Finally, GitHub’s availability is not a binary all-or-nothing proposition. They report incidents on a granular level, for example, webhook firing can be impaired while Git hosting may be working fine.
dimgl 8 hours ago [-]
What if you, you know, have a job?
TZubiri 8 hours ago [-]
is the infrastructure of the free tier shared with that of paid users? That might be the issue. Otherwise if usage of paid users scales, then it would be fine.
john_strinlai 8 hours ago [-]
>is the infrastructure of the free tier shared with that of paid users?
github enterprise is fully operational at the moment
We cannot pull any actions images (or how it is called) for example.
8 hours ago [-]
neo_doom 8 hours ago [-]
Agreed. This impatience culture is so toxic. Been down less than 30 mins.
anonymars 8 hours ago [-]
That's cool, I mean if you'd gotten here earlier you could have said "been down less than 5 minutes"
So with the "elapsed time" out of the way (now over an hour, incidentally), how about "how long will it be down?"
cb33 6 hours ago [-]
2 hours later - still down. Is this "impatience culture" or incompetence culture at this point?
imnes 8 hours ago [-]
GitHub, Charles Schwab, Microsoft Teams, Copilot, New York State, AWS, Progressive, Azure, Microsoft Store, Cloudflare, ADP, Google Sheets, E-Trade, Fidelity... Did Mythos break out of it's container?
The most shocking thing to me about the entire GitHub fiasco is: No matter how bad the uptime is, no matter how bad the UI is. We will keep using the software / service we depend on.
Why do we as software engineers have set fault tolerance and high availability as the primary goal of any software / infrastructure architecture? This shows it's completely irrelevant.
5 hours ago [-]
winrid 4 hours ago [-]
Personally, Fastcomments and all my projects can easily be switched to a self hosted provider, keeping GitHub as a read only mirror. It's probably a couple days work with today's tools.
esafak 5 hours ago [-]
It is not. I am just not happy with the alternatives; Gitlab, and Codeberg, for well-discussed reasons. I will pounce as soon as the right one comes along, and I have a few on my radar.
decodebytes 8 hours ago [-]
Cannot even merge code, this is the limit for me. We are prioritising migration to a different platform and this time going to decouple CI - right now we have too many points of failure on one vendor.
arionhardison 7 hours ago [-]
Apologies if you already tried this, but I have been able to merge and get some things done via command line.
merges work fine, the git experience as well, this is a web infra issue
bhouston 9 hours ago [-]
It is not working at all, I can not load any repo pages.
Github.com's promise is that it can be the central broker of open source code because it is reliable.
That promise hasn't been kept recently.
That said, Github is hard to displace and it is similar to the era of the Twitter Fail Whales. It was a sign of growth that couldn't be properly managed but there was not viable alternative.
jillesvangurp 8 hours ago [-]
> Github is hard to displace and it is similar to the era of the Twitter Fail Whales
Hard but not impossible. Being unreliable might nudge a few people. Also, Twitter had their fail whale in their early phase (mostly), not twenty years or so into their existence.
The issue with replacing Github is that it takes more than version control and issue tracking to replace the network that is Github. It's effectively the largest social network for software developers on this planet.
Tangled (tangled.org) is built on the at protocol (bluesky) and seems like an interesting take on this. Most other solutions seem to just move the problem to another much smaller silo (gitlab, codeberg, etc) or throw out the baby with the bathwater by focusing on self hosting without attempt at networking with other instances.
A federated approach seems like it is directionally right. Tangled is far from feature complete but I like the direction and it probably is a bit more resilient against global outages like this. I'm not a user yet but I'm keeping an eye on it.
palata 7 hours ago [-]
> throw out the baby with the bathwater by focusing on self hosting without attempt at networking with other instances
May I ask what networking with other instances brings? Git is decentralised, so it works with self-hosted instances, right?
jillesvangurp 5 hours ago [-]
Git is decentralized in the sense that you can clone your repository wherever. But there is no real networked community on top. The first versions actually piggy backed on the social network that is email. Linux still is managed via email. It's social network was mailing lists. By decentralizing, they could avoid the pesky topic of having to give each other read write access to each other's git servers.
But if you want more than what email can offer, social networking is a thing with Github and it's a big reason for its success. It added a friendly UI and a way for users to engage with each other via issue trackers, pull requests, and all the rest. It has tens of millions of users that have accounts that each can each create issues, pull requests, fork each other's projects, etc. These people collaborate with each other on many millions of projects. The network effect of all these people being in one place where they can collaborate is vital to a lot of open source projects.
It's effectively the largest developer centric social network around. If the open source community fragments over a gazillion self hosted servers that each require a signup that don't really federate any signins and that don't allow aggregating notifications from all the projects you contributed to, etc. it would be a big loss for the open source world.
Tangled by making use of the atproto replicates much of this functionality. While also allowing people to run their own nodes and while allowing users and their projects to be moved between those nodes. That makes it the best of both worlds. You can self host it but without completely disconnecting from the community.
palata 2 hours ago [-]
> That makes it the best of both worlds
Assuming that everybody moves to your community (in this case the atproto one). Then the ActivityPub people ask you to join their community, and so does every new community that wants to become the new standard.
That's the thing with every new social network: "You don't want to be locked into the current community that is not my community, instead you want to move to my community, which is the best! And if you do, and convince everyone around you to join as well, then you will only ever need one account for everything!".
And now, assuming that most people leave GitHub and join Tangled, it will be to the detriment of the minority that is not using Tangled. If I host a repo on SourceHut, nobody will contribute because they want GitHub. If Tangled replaces GitHub, will that change? I don't think so, people will just refuse to use anything other than Tangled.
What I find interesting is that you advocate for a federated system that everyone should use, because "it would be better for everybody", but the fact is that everybody has an email already. No need to create an account, we can send each other patches, it just works.
Why don't you promote email? Because it's not your preference.
PS: I do realise that email has a lot of issues. My point is just that I avoid the entire issue by just being open to sharing my code however it works. And I feel like patches are easier to share "universally" than pull requests. Maybe I would be in favour of a new, "richer" patch format that would make it even nicer to share over whatever system I want (email, atproto, activitypub, signal, ...)?
jillesvangurp 32 minutes ago [-]
Email indeed still works. That is in fact why I brought it up. But it still has the same deficiencies that it had before Github even existed. Which I would predict is not going to convince a lot of open source projects to suddenly start processing git patches from mailing lists now. Which is why I don't advocate that. I'm not sure I'd want to do that myself.
Tangled is kind of there as an alternative that has most of the comforts of Github but is federated and can be self hosted. And there are a lot of Bluesky users that have access to it even if they might not be aware of that yet. If enough projects switch over, their contributors might follow along. If something else shows up, more power to them. But I'm not really aware of (m)any other properly federated alternatives.
The beauty of federated protocols of course is that they can be bridged and connected in all sorts of ways. Some hybrid at proto / email based bridge could be pretty doable. And if somebody wants to expose at proto feeds as activity feeds, that's probably not too technically hard either. These things aren't actually mutually exclusive.
pkulak 6 hours ago [-]
I'll keep my open source stuff there forever, because I like the eyeballs. But this has got to be making corporate customers flee in droves. Of course, that's exactly what GitHub _doesn't_ want.
hedgehog 8 hours ago [-]
It's a bit of a different situation though, in many cases the unit of adoption is a team or organization that is free to switch platforms without consideration of what anyone else is doing. At this point I'm going to move my personal projects, it's not bad enough to be a priority at work yet but it's trending that way.
beached_whale 7 hours ago [-]
github's value is that it has a lot of eyes and visibility for smaller projects(thought many get lost too). More and more this value is going down and that stability is getting more important. More so when the story of github these days is copilot(good or bad) and their downtime. AI is giving a lot of project owners second thoughts and being so open with an AI company too.
dv_dt 8 hours ago [-]
github replaced sourceforge, so something or things can at least offset github
smcameron 7 hours ago [-]
I wonder if you could replace the whole github issues feature with just a directory within each git repo named "Issues/". Within that, have another couple directories, "open/", and "closed/". Within "open", have a directory per issue, e.g. "such-and-such-does-not-work/". Within each issue directory, have "issue.txt", or "issue.md", which takes the place of the page for each issue on github. You could store other things in the issue directory which "issue.md" could reference, e.g. screen shots, logs, whatever. When an issue is closed, it gets git mv'ed to "closed/". You want to see a list of open issues sorted by time? Just "ls -ltr Issues/open". Want to see open issues related to the frakfrobber? "grep -i frakfrobber Issues/open/*". Add a comment to an issue? "vim Issues/open/my-issue/issue.md; git add !$; git commit -m 'Updated my-issue';" It would move the entire issues subsystem back under control of bare "git" and make it portable. Undoubtedly, I am not the first to think of such a thing.
jasongi 7 hours ago [-]
The whole point of issues is that users can create them when they don't have repo write access.
nafey 7 hours ago [-]
Check out Haxxy, it has a lot of fresh ideas related to issues/pr management
Help encourage alternatives by trying Gitlab.com or Codeberg.org or Tangled.org or others, and please star some projects there to help motivate more options.
Self-promo links below if you would like to star a few of mine good for developers.
Codeberg isn't a viable infrastructure alternative. Its an opinionated community not a neutral host as they will delete repos based off subjective opinions on LLM usage and resource consumption. If continued hosting is dependent on moderators deciding if a project fits whatever the current philosophy is, and the whims of the moderation team, then I cannot consider it a serious choice for infrastructure.
Infrastructure should be neutral. Codeberg is a social club
usrbinbash 6 hours ago [-]
> Infrastructure should be neutral.
Infrastructure can never be neutral, because infrastructure is owned and costs resources to run. All infrastructure comes with rules, be they the voting consensus of some e.V. or state-run entity, the whims of an individual on their private server, or the corporate interests of something owned by a, well, corporation.
The problem isn't "neutral infrastructure". The problem is an overreliance on centralized services even though most such services are relatively easy to host yourself.
I run my own DevOps server based entirely on OSS. Is it github? No. Does it what I need it to do? Absolutely. Do I get 99.999% uptime? No, but when it's down or slow, I can fix it. When the corporate platform is down, I can only wait.
And "network effects" are not a good argument. Open Source was wildly successful long before centraliced DevOps platforms.
And in hindsight, the added friction of getting in touch with projects before being able to send them any code, was a benefit not a disadvantage. Because it acted as a barrier of entry that disincentivized alot of subpar code from hitting these projects in the first place.
rbanffy 6 hours ago [-]
> are relatively easy to host yourself.
The value these centralized services provide is the centralization itself - that makes things easy to find. Remember the web before Altavista? Finding anything was detective work, guessing domains, following links through multiple pages, digging in newsgroups, asking around, and so on.
Open source was successful, but not nearly at the same level it is now thanks to precisely platforms such as GitHub (and Launchpad, and Bitbucket, and Gitlab, etc) that added useful functionality such as bug trackers and project management on top of a version control system.
Now, if we could do something like GitHub on top of a network such as Mastodon, then we would have something that's centralized enough to be useful and distributed enough to be indestructible.
usrbinbash 6 hours ago [-]
> Finding anything was detective work
And in recent years, how is the centralized world of websearch doing?
> Open source was successful, but not nearly at the same level it is now thanks to precisely platforms such as GitHub (and Launchpad, and Bitbucket, and Gitlab, etc) that added useful functionality such as bug trackers and project management on top of a version control system.
None of that depends on a centralized provider. I have all the features you listed on my self-hosted platform as well.
ascii0eks84 6 hours ago [-]
Infrastructure can at least be content-politically neutral.
usrbinbash 6 hours ago [-]
Not even that is really the case.
Even if whoever owns and runs the infra would like that to be the case, the law of wherever they are can put restrictions on this as well.
jph 7 hours ago [-]
Can you suggest a viable infrastructure alternative?
My wishlist: free for open source and paid for closed source, able to do simple projects that are handwritten and also complex projects including AI LLM code that needs CI/CD/auditing, and hosting in the UK or EU.
Aside, I donate money to Codeberg because I think Codeberg is a wonderful service with strong commitment to free open source, and because I host some larger projects and AI projects that are for public welfare social good.
icy 7 hours ago [-]
https://tangled.org would love to have you :) We're hosted in the EU, and are free for open source, forever. Tangled is also designed to be federated from ground up, so you get to own your infra & data while still participating in an open network.
Tangled is a VC-funded Oy (Finnish limited liability corporation, €3.8M seed a few months ago). I searched and couldn't find a published donation route, which makes sense given it's VC-funded.
To Tangled -- it's great that the core code is MIT licensed and anyone could stand up a competing appview (the credible-exit property of atproto).
Do Tangled Labs Oy's articles include a purpose clause instead of the default profit-only purpose[0]? Or is something similar on the roadmap?
[0] OYL 1:5 reads The purpose of a company is to generate profits for the shareholders, unless otherwise provided in the Articles of Association.
crote 6 hours ago [-]
What is the long-term sustainability story here?
You're deliberately avoiding a network effect lock-in by being federated, you make competition trivial by giving away not only the protocol but also making your code available, and you don't even restrict commercial competition by using the MIT license rather than, say, BSL.
To me that sounds like you are deliberately starting a race-to-the-bottom competition on price with, well, everyone capable of hosting a SaaS? Heck, if you don't end up with some solid competition the whole "federated code" USP doesn't play out. To me that doesn't exactly scream "good return on my €3,8M seed round". Am I missing a brilliant plan here, or should we expect a rug pull a few years down the line?
chilmers 7 hours ago [-]
I feel like any service that is free for open source is going to struggle if it gets significant adoption and becomes a default target for hosting LLM generated code. If even MS/GitHub cannot scale to the current demand, what hope do smaller and less well-funded alternatives have?
voidnullvalue 6 hours ago [-]
Rate limiting the infrastructure-hammering actions (mainly CI stuff) would be a reasonable method to handle this.
wyclif 6 hours ago [-]
Doesn't GitHub already do this? (I don't know if they do or not; just wondering).
swedishuser 2 hours ago [-]
I'm often surprised to find very questionable repos hosted on GitHub. Such as plug 'n play malware that has been public for years. Not against it, but surprised as most parts of the internet has gone in the kindergarten direction.
pojntfx 7 hours ago [-]
Codeberg is one public flagship instance. They are not neutral and don't claim to be. Decisions on what to host are made via elections by e.V. members. Other platforms (GitHub, GitLab, Tangled etc.) will eventually run into this same thing as well, e.g. how GitHub responds to DMCA requests, the difference with Codeberg is that there are elections which decide on what gets hosted on the main instance. For hosting slop or running a ton of actions, the recommendation is to self-host Forgejo directly instead, meaning that you, not the e.V. pays for the consequences, and you'd use the future AP federation support for cross-instance support. Forgejo is particularly simple to self-host as well, and lots of people do it (ffmpeg, Fedora and more). That way, you get to set your own rules and e.V. election results have no effect on you.
ascii0eks84 6 hours ago [-]
Infra should be immutable piece that is certainly not influenced by someone power tripping or being butt-hurt by conflicting alignment on worldview issues. These things should be completely detached from each other, and infra should be neutral with technical guarantees it cannot be cut off aside of warfare acts.
8bb6ce5638bdc 7 hours ago [-]
[flagged]
superze 7 hours ago [-]
Sounds more like a cult rather than a social club.
tcfhgj 7 hours ago [-]
actually it's a democracy
aliasxneo 6 hours ago [-]
This gives the impression that there's a collective group of differing views that come together and make decisions. It's quite the opposite from my experience and they seem to actively chase out anyone who dissents from the collective hive mind's view.
tcfhgj 2 hours ago [-]
while I don't know if your allegations are true, democracy means there is a group which decides on rules based on a majority vote. It doesn't mean that they have the same or different views at all, although some overlap is probably certain, because otherwise you probably wouldn't decide to join the same association.
Anyone who joins the association (small fee) can vote.
aliasxneo 42 minutes ago [-]
I know what the technical definition of democracy is. However, a lot of people throw it around as if it insinuates a free and open system where every idea can be voiced. That's certainly not the case at Codeberg, and they make that fairly clear. They heavily moderate content and view points that challenge the hive mind are generally moderated out of existence (or the topic/thread is simply closed).
So, yes, it's technically a democracy, but it deserves extra nuance in my mind because it might not be what everyone immediately thinks a democracy should feel like.
giancarlostoro 7 hours ago [-]
Rather use BitBucket or GitLab really, both GitLab and BitBucket have been around nearly as long as GitHub? Apparently BitBucket and GitHub are both 2008, and GitLab is 2011. The others you mentioned could go on a whim.
tcfhgj 6 hours ago [-]
I don't know BitBucket, but GitLab is felt extremely bloated and slow - Codeberg is much leaner and faster
aic0d3r 6 hours ago [-]
gitlab has my vote
crabbone 6 hours ago [-]
I started my career in programming with Subversion was the most popular tool. I also had to work a little with VSS, Perforce, Mercurial and, obviously, Git. I was one of the first users of Google Code...
So, relatively, not so long ago it was pretty common to host all this stuff yourself. I don't think (or maybe don't remember) Subversion having any Web interface of its own, but, since forever Git came with GitWeb. I remember configuring it for the first time: any extra bit of configuration, especially those that resulted in some added interactive functionality seemed exciting. For a week, I'd push something to my branch and immediately refresh the browser page to see my changes reflected in the remote repository :D
The infatuation wore off very quickly: after a month I have almost forgotten the Web interface even existed.
The cycle repeated with hosted services, the addition of PR interface, the addition of "pages", the bug reports, the built-in Wiki. After discovering the new gadget and associated hype, invariably, came the realization that the predating existing functionality was... better. The PR discussion in the Web page detached from the code is a lot worse than the PR discussion in the PR itself. The built-in issue tracker doesn't come close to the stand-alone product functionality. The built-in Wiki is ridiculously underwhelming when compared to an actual Wiki.
And so was the CI. For me, it started with Gitlab. There are oh so many reasons to hate Jenkins, so, whenever anyone announced an alternative platform, I'd absolutely check it out, and so did a lot of others, I presume. I was super hyped up about the possibility of running CI from the command line, completely locally on my laptop. And then it turned out to be a bait-and-switch kind of thing. I've created monumental dozen-steps and branching pipelines connecting multiple projects only to realize that the best CI is the one that uses as little as possible of the platform tools.
In the end of the day, I came to realization, that beside the social aspect of the platform like GitHub, everything that it offers on top of the essential Git repository storage is too low quality. And so do competitors. If I was ever again tasked with setting up a company code repository storage, I'd go with a physical server attached to the local office network, unless there's no office, in which case, probably, some EC2 VM would do the job...
Places like GitHub or Gitlab lure you in with their free tier offering, where you get the storage space and some other goodies for free, but once you need to upgrade to the paid tier, you inevitably realize that it's not worth the money they charge for it.
1matin 7 hours ago [-]
GitHub being ruined after the AI hype is a shallow take imo.
The root problem isn't AI, Microsoft, or Azure; It's why people keep _trusting_ centralized infra.
GitLab and Codeberg can absolutely become trash, each in their own way.
The solution isn't just "migrating off of GitHub"; it's "migrating to a platform that is unownable by design, not only hope".
And there are no good enough decentralized alternatives to GitHub yet.
Tangled.org's decentralization has serious holes.
Radicle made an alien universe; a niche within a niche. Also, it's already having problems with ISPs:
https://radicle.dev/2026/04/23/domain-move
varun_ch 6 hours ago [-]
What are the serious holes?
1matin 5 hours ago [-]
The main hole is that ATProto's data model is a bit awkward for a code forge.
Issues/PRs/comments are records owned by their authors, not by the repo. So an issue can span N users' PDSes. That's a neat model for social media, but a forge wants the repo to be the durable owner/context of its issues and history because that's where the comment/issue/etc. logically belongs.
The solution? tangled.org acts as a super advanced cache/aggregator for that. What can possibly go wrong?
brw 2 hours ago [-]
There's a good chance this will change in the future and the repo on the knot will also contain the social aspects of repos (issues, PRs, etc.)
> As time goes on we are re-assessing the idea of users owning what is "collaborative data" (issues, PRs, etc.) on their PDSes - soon may come the day that an issue also lives on the knot as a source of truth, with an accompanying pointer record on user PDS to attest that it's theirs.
Azure was arguably as garbage four years ago as it is today, the only thing that's changed in the past few years is MS mandated vibe coding, and suddenly Github has an outage every week? A ridiculous coincidence if it's not causal.
trebaud 33 minutes ago [-]
Shameless plug, AI built me this cli tool to visualize gh like diffs from the terminal https://github.com/trebaud/diffcat , could prove handy in times like these lol
Maxion 9 hours ago [-]
> We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become available
Well it's 100% down for me, so does this mean 20% of all users are fully down?
lionkor 9 hours ago [-]
It means it doesn't count towards their uptime at all, I guess, because all their uptime stats are very high for a service that is down more than anything else I use
Maxion 8 hours ago [-]
> Pull Requests is experiencing degraded performance. We are continuing to investigate.
> Posted 1 minute ago. Aug 17, 2026 - 13:58 UTC
It also took them 20 minutes to realize that peoples requests to /pull/xx are not working :D
xnorswap 7 hours ago [-]
If you load an issue page, you'll see 1 failed request to:
/project/product/issues/<number>
And sure, that's what you care about, but consider the working requests to:
Funny enough, I received a marketing email from GitHub during the outage with the following subject: "What if every pull request you started actually shipped"
Aachen 8 hours ago [-]
Huh, they usually do? I'm not a full-time developer though so I'm surely biased towards doing the obvious quick fixes or already-discussed things, but discussing how to solve a problem before throwing something at the wall seems like the normal thing to do for me
Or is their point about abandoned projects where the dev never clicks release after merging, or even never merges?
BowBun 8 hours ago [-]
The joke was that we have open pull requests that we cannot merge because of this downtime.
Aachen 4 hours ago [-]
Yes, see sibling comment posted slightly ahead of yours (dead/invisible at first, to be fair)
Laurel1234 8 hours ago [-]
The point is you can't ship shit during an outage.
Aachen 8 hours ago [-]
Yeah I saw the humor of that :) Just curious about the thing they're trying to even sell, it doesn't sound like something they can offer even if they had a good SLA
stefangordon 7 hours ago [-]
The exponential growth they are having right now is absolutely insane. I would caution against assuming its an easy problem, or that anyone else has demonstrated an ability to handle this level of scale/growth any better.
In May they had to solve for 30x growth overnight, and I would guess its closer to 100x by now. The reality is a centralized solution like this may no longer be a viable option with the scale of code generation.
tryauuum 7 hours ago [-]
Have your seen safe_sleep.sh?
crote 5 hours ago [-]
[dead]
mirekrusin 7 hours ago [-]
Why are people complaining? It’s not a bug, it’s a feature.
You have a free bank holiday, go look at green trees, listen to birds, find somebody who's wrong on HN etc.
The only poor souls here are some github devops team (team of copilots?) and those using alternative services.
ashvardanian 46 minutes ago [-]
If only we had a myriad of heavily funded, founder-led AI-for-coding startups that could afford to store a few source files and need all that code for training anyway :)
PS: Saw Cursor’s Origin announcement a second later.
larsonian 8 hours ago [-]
Whatever it is, it seems to be getting worse: started with pull requests and Actions being down, now I can't even list my repos. Might be the worst one yet.
forcedtolinux 8 hours ago [-]
copilot is also down
aaronmdjones 6 hours ago [-]
Oh nooo! Anyway...
niffydroid 2 hours ago [-]
If the LLM usage is the cause then they need to start placing limits or add costs, otherwise normal fee paying customers will move. I work in a financial institution, while moving away from GitHub is hard our parent group uses gitlab, so really it's just our resourcing to do that is the issue, but if we asked group for help it quickly becomes possible. I feel another 2 of these instances we'll make the decision
anticensor 8 hours ago [-]
Anyone else getting "Cannot retrieve latest commit at this time?"
hariso 8 hours ago [-]
Yup.
ozarkerD 9 hours ago [-]
Is github down?
Does the day end in y?
Hammerhead12321 9 hours ago [-]
No, es lunes.
ozarkerD 7 hours ago [-]
Damn. Wonder why it's not working then
aleksandrm 9 hours ago [-]
Can anyone shed some light why GitHub is having so many problems lately?
consumer451 9 hours ago [-]
Copy/paste of a comment I posted a couple months ago:
I am not trying to make any excuses for GitHub, however I recall one tweet that made my eyes bug-out:
> Yup, platform activity is surging. There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
> GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week.
That's fine, but rate limiting is a solved problem at this point. They need to rate-limit free accounts and be realistic about their limits instead of letting the system fail.
chilmers 8 hours ago [-]
Free (and non-free) accounts are already rate limited. The likely problem is that individual accounts/repos rarely hit those limits, instead the explosive demand is due to a massive growth in the number of small, individual projects being creatd.
materielle 8 hours ago [-]
The person you were applying to said “rate-limiting” but I think they meant “load-shedding”.
consumer451 8 hours ago [-]
[dead]
8 hours ago [-]
Maxion 8 hours ago [-]
Probably can't do that easily due to shitty engineering decisions in the past
4lx87 8 hours ago [-]
Or inept managers.
Manager: Why is the site down!?
Engineer: Because it can’t handle this level of traffic. We can rate limit to the level we can handle to keep the service running.
Manager: Okay, let’s do that!
Engineer: Okay. Rate limiting at N req/min.
Manager: I’m getting complaints we’re denying service to customers!? We can’t rate limit our users if they want to use it. Increase the rate limits.
I’ve had this conversation more than once in my career.
zzbzq 8 hours ago [-]
They were about to push a big change to actions pricing a little wihle ago and people revolted
cedws 7 hours ago [-]
And most of that growth is just garbage. I'm not even anti-AI, but just look at the quality of projects on GitHub these days. The site is full of spam, it's not a place for professionals anymore.
MontagFTB 9 hours ago [-]
Their infrastructure is buckling under increased demand from AI generated code.
Imagine the software development lifecycle is a river blocked by several dams- necessary processes that also slow the water’s volume and speed. Writing the code was a very big dam upstream, and successive dams didn’t need to handle a lot of water. With that dam busted wide open, the torrent of code is inundating downstream processes. GitHub happens to host many of them.
Earlier this year the GitHub COO shared a post on Twitter with explosive numbers they were facing. I cannot seem to find it at the moment. (Edit: shared in other replies to this thread.)
slowin 8 hours ago [-]
I mean, that’s what they say but it doesn’t pass the sniff test. Which other major service struggles so much with basic scaling? They’re not serving streaming video, they’re not a real-time system. Any problems scaling should have been fixed years ago. Especially with all of Microsoft’s resources behind them.
hbn 8 hours ago [-]
> they’re not a real-time system
I would argue they are. You can't cache a git repository and send people what it looked like a minute ago, because someone might have pushed 2 seconds ago.
Retr0id 9 hours ago [-]
I think the most charitable theory is that they're seeing rapidly growing usage (more users churning out higher volumes of code, via LLMs), and are struggling to scale to meet demand.
Alternative theories include "azure sucks" and "they're vibecoding everything now".
dtech 9 hours ago [-]
They say it's the skyrocketing load because of AI tooling
Many people suspect it's the year-long migrating to Azure and all their products instead of the things they have been using.
sunaookami 8 hours ago [-]
People will say LLMs but the issues with GitHub long predate LLMs. It all started when Microsoft acquired them.
perrygeo 9 hours ago [-]
Github's, COO Kyle Daigle, claimed that the number of commits has increased 14x since last year. So shear scale has something to do with it. I don't know of too much software already operating a planetary scale that could easily absorb a 14x bump in a year.
slowin 8 hours ago [-]
GitHub is pretty tiny compared to something like YouTube and Instagram. Both of which have much better uptime.
evilfred 8 hours ago [-]
14x is barely an order of magnitude
whateveracct 4 hours ago [-]
seriously. if you are ready to scale, you are ready for that much. if that kills you, you had architecture issues that a company as big and established as GH shouldn't have.
Whatever the actual increase in productivity may be, the commits, clones, pushes, action runs, etc. are skyrocketing.
everfrustrated 9 hours ago [-]
Lately? GitHub has always been bad with downtime. It hasnt got worse just more visible as people do more CI now than before.
allanmacgregor 9 hours ago [-]
Its all vibes man
thinkingtoilet 9 hours ago [-]
Microsoft simply can not make software anymore. I worked at S&P Global and when you work at a global-mega corp, you quickly see why they lose the ability to create anything and just acquire companies to grow.
brookst 9 hours ago [-]
It's getting to where we should just post when it's working fine, to reduce noise.
standwportugul 8 hours ago [-]
Funny thing about the status page is that you can't find any record of last Thursday's outage. It's all green across every service.
blinky88 9 hours ago [-]
This is getting ridiculous. Issues, PRs, actions... nothing's working
biennvops 9 hours ago [-]
Interestingly enough if you access the PR number directly it still works.
imstil3earning 9 hours ago [-]
But the `Files changed` tab still 500's so the direct PR link is not worth much
majewsky 9 hours ago [-]
Not for me.
Maxion 9 hours ago [-]
Probably cached data and the listing search is not?
duxup 8 hours ago [-]
Not for me. Unicorn error.
orangeisthe 9 hours ago [-]
gh CLI works for me
swills 7 hours ago [-]
Here's an article from Oct 2025:
"Microsoft’s data-center shortages to persist longer than expected - Bloomberg"
Perhaps relevant tho. I would imagine if they could simply throw more hardware at it they would. And I wouldn't imagine inability to throw hardware at it would be due to software scaling issues. So perhaps it is simply lack of available hardware?
toastal 7 hours ago [-]
The PSA is that you should always have a mirror—even if just a bare repo folks can pull from during downtime.
…& this is ignoring all the moral reason for avoiding US-based, publicly-traded, proprietary software for host largely free software. They also deny access to users due to US sanctions, privacy-conscious users won’t want to create accounts/agree to ToS, & rubbing shoulders with other corpos your project can be taken down if it upsets the corpos don’t like it (see youtube-dl, Switch emulators, etc.).
_joel 7 hours ago [-]
Sure, but it's not just the reoo. It's PR's, code review and you've got a bunch of actions then it's difficult (not impossible) to deal with those being down.
BIGFOOT_EXISTS 8 hours ago [-]
Alright! Let's go make some coffee and get fresh air
lol768 9 hours ago [-]
Issues seems to be completely broken, just getting unicorns.
Interestingly the API still works for creating an issue (but webhooks weren't fired).
bhouston 8 hours ago [-]
What is the best Github alternative that has something like Github Actions? What is the easiest to switch to? I do not want to switch but it is having too much of an impact on me.
It is time to give up on Github. This is just too much.
vinnymac 8 hours ago [-]
I’m hosting my own https://tangled.org knot and spindle, and also hosting and building my own Frontend browser for https://forgejo.org looking forward to open sourcing it in the future for others to enjoy.
For people who want a similar experience today, I’d recommend trying Codeberg (An official instance of Forgejo) though. However, keep in mind they voted in July to ban mostly AI generated repositories. You can read about that here: https://blog.codeberg.org/protecting-our-floss-commons-from-...
bhouston 8 hours ago [-]
Yeah, Codeberg banning AI generated code is weird. AI generated code is the future. It is like during the days of assembly banning high level languages.
mariocesar 8 hours ago [-]
GitLab is probably the most reasonable alternative. I migrated and it took me some weeks to adapt to the new CI, but for my use case it's basically equivalent in features.
I started with the cloud version, then moved to self-hosted because it wasn't a big effort for our team. We also reduced our monthly invoice by about 50% after the move.
lionkor 8 hours ago [-]
* GitLab
* Forgejo
Those are my favorites. Both you can self-host for your company, including CI/CD.
stephenway 7 hours ago [-]
Forgejo is probably worth a look if Actions compatibility and an easy migration are priorities. I run Fjord, a dedicated managed Forgejo, so you don't have to operate it yourself. We can help with the migration and runners too.
We also have a free public Forgejo instance if you want to try it first.
nlightcho 8 hours ago [-]
I am building an MIT-licensed compatible alternative to GitHub Actions at github.com/automata-ci/automata. Very early in development but it already runs some of our internal workflows.
TrueSlacker0 8 hours ago [-]
[dead]
pavo-etc 9 hours ago [-]
> Update - We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become available
brutal
gottagocode 9 hours ago [-]
20%, do we just make numbers up?
anonymousab 8 hours ago [-]
It seems vastly higher on every region and VPN I've tested, so I think this is a case of the "technicalies".
It is technically 20%, because a bunch of the requests that happen on page load do succeed. Just not the few crucial ones that are required for the page to load correctly - those return a 500 the vast supermajority of the time.
brookst 8 hours ago [-]
I can't get a single PR or issue to work. Maybe it's "20% globally, 100% anywhere people are awake".
malfist 8 hours ago [-]
Feel free to disregard this because ultimately its a tangent that doesn't matter; but supermajority is specifically a political term referring to the required threshold to advance legislation when simple majority isn't procedurally enough. It doesn't mean simply a greater share of.a majority
zmgsabst 8 hours ago [-]
This is how language evolves.
For supermajority: the construction means “beyond majority” or “above majority”, it is implemented as a higher percentage in most systems, and there is a utility in being able to refer to high ratio majority.
So people have repurposed a technical word to a general word, of roughly the same meaning.
albatross79 6 hours ago [-]
No, some usage is just wrong, even if many people are doing it.
bhelkey 6 hours ago [-]
English is descriptive not prescriptive. The meaning of words change over time.
"Disappoint" was originally the inverse of "appoint" and meant to remove from office.
albatross79 4 hours ago [-]
It's neither. If you accept and approve of people wearing their pants around their knees, its a fashion evolution. If you think pants should cover your ass, it's just poor behavior. Analogous to the use of language.
normie3000 8 hours ago [-]
20% of requests on a GUI page also makes the whole thing unusable.
ozarkerD 9 hours ago [-]
Depends if this is coming from engineering or management
9 hours ago [-]
dboreham 8 hours ago [-]
The error rate is 100% on the service that allows you to create PRs in the web site. Actual remote git service is working just fine. I might try the API to make a PR, see if that's working.
Update: yes that worked. gh CLI using the API was able to make a PR while the web interface was inoperable.
Second update: defeat from the jaws of victory -- the page that displays open PRs is also down.
Third update: CI isn't working anyway, so the CLI PR creation workaround doesn't achieve much.
woeirua 7 hours ago [-]
Is there any scenario in which Github doesn't start yanking features out of the free tier to protect the core service for paying users?
Channel9877 6 hours ago [-]
IMO this is what they should be doing first to stop the bleeding. I believe a huge portion of their traffic is leech traffic.
frou_dh 3 hours ago [-]
Ideally they'd grandfather in accounts that were created long before the spammer days :)
zeroq 8 hours ago [-]
It's been a downward spiral since covid.
Enterprise works ok-ish, but the standard version has daily issue ever since.
esskay 8 hours ago [-]
Not sure covid was the sole driving force. A combination of MS taking over and deciding to move it to Azure, a massive increase in traffic due to ai usage (not that this should be an excuse for a platform that should be scaleable), and Github going all in on AI usage to write its own code.
They're making it very easy for a viable alternative to pop up and take their lunch - it just wont be Gitlab.
csvance 8 hours ago [-]
While it annoys me how things have gone downhill, I think you missing one key detail with regards to competition. It's only "very easy" if you feel like throwing enough money at CI compute to power the majority of the open source ecosystems needs. That would be the bare minimum in order to even have a shot of achieving similar levels of network effects as Github.
macNchz 8 hours ago [-]
I don't have data for it, but I have been a Github user since 2012 and have found it to be down often for pretty much that entire time. I always figured it's cultural to a degree–new features always seem pretty buggy/underbaked, and often continue to long term but remain unloved and incomplete.
zeroq 8 hours ago [-]
I has never been so bad as in the couple last years. And it's getting worse.
I never had to wonder if Steam is going to be working today, so I could play my game after work, but it's been an issue with GitHub since covid. At least for me.
Fortunately, due the nature of the service, I can sit out most downtimes. Most of the time at least.
bluedino 7 hours ago [-]
Enterprise here and it's down all the time. Right now and it was down 4 days ago as well.
jjice 9 hours ago [-]
I'm seeing "Merge status cannot be loaded" when attempting to merge a PR.
It's a shame I had to go to HN to check if I was the only one instead of relying on their status page that they link.
abelanger 7 hours ago [-]
If you really need to merge and deploy, I've been having some luck getting things moving with the gh cli:
Might not help your deploy process if webhooks are degrading, but at least it can keep your remote in sync with local changes, if you need to deploy from local.
chilipepperhott 7 hours ago [-]
What I want:
A self hosted version of GitHub that is a precise mirror of an actual GitHub repository. That is, PRs from GitHub appear on the selfhosted instance and my reviews from the self hosted instance appear on GitHub. That way I can work on my private instance when GitHub is down (or slow) and I can be assured my work still appears to everyone online.
Does this exist?
rsanheim 4 hours ago [-]
AFAIK there is nothing like this out there, because there is not one standard for a "pull reauest" in the sense of how a web based forge does it. and it gets complicated quickly due to complexity of git and forks.
Git gets you 90% of the way there. Sounds like you just want a decentralized review solution to go with it.
mlhpdx 4 hours ago [-]
I can’t help but think of how AWS didn’t go through with killing CodeCommit and how happy that user base is to not be on GitHub.
nine_k 7 hours ago [-]
(Umpteenth time:) GitHub is not about just publishing source code; with git, it's trivial out of the box.
GitHub's value is in the PR process, in CI/CD, releases, the issue tracker, and an identity system propping all of that. This is the thing to replace, and it's quite a bit harder.
georgehotz 8 hours ago [-]
We're moving to self hosted Gitea. It's never been easier to do devops with the help of LLMs. I cannot believe how flaky GitHub is.
Funny because I spent a couple of years on self-hosted Gitea, including doing some work on Gitea itself to add features we wanted. Then I changed projects and thought "well that was a lot of work, let's just do what everyone does and use GitHub". Now twiddling my thumbs this morning...wondering if I need to resurrect the Gitea deployment.
stephenway 7 hours ago [-]
This is pretty much why I started Fjord. I like Forgejo/Gitea, but didn't think “run your own forge forever” should be the only alternative to GitHub. We run dedicated Forgejo instances for people who want that middle ground.
wren6991 8 hours ago [-]
At this point, they should notify us on days that it's up
DrScientist 8 hours ago [-]
Never quite understood why large enterprises use github as core infrastructure.
Sure use it to collab externally, use it if you can't justify running your own infrastructure, but why are large companies choosing to place code externally - which is one secret leak away from compromise?
Nemo_bis 6 hours ago [-]
The larger the company, the more likely that it's enmeshed with Microsoft in some of its parts. Microsoft is relentless in expanding its footprint with vendor lock-in and cross-sells until it replaces all competitors.
hbn 7 hours ago [-]
Pass the buck. There's no one to blame but yourself if you're self-hosting.
"Yeah we're unable to offer services, but so is [X BIG CORP] and [Y BIG CORP]. It's an industry outage, nothing we can do!"
stetrain 9 hours ago [-]
githubstatus.com has been updated with an incident now:
> Update - We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become available
Aug 17, 2026 - 13:45 UTC
arccy 8 hours ago [-]
experiences, as if we're in disneyland
c-hendricks 8 hours ago [-]
It sounds like what I was seeing when trying to view PRs last night, so it started almost 12 hours ago.
How did Github become so famous for outages? It is almost like a routine. I haven't used other code repo hosting / CI/CD pipelines platform for last decade, so maybe it's just the nature of this business, I don't know.
vanillax 8 hours ago [-]
By switching to Azure....
forestry 8 hours ago [-]
Do you have some information that it’s Azure related that isn’t public?
zipy124 7 hours ago [-]
it seems to be coinciding with teams and other microsoft services having issues would be some indication, but some other non microsoft products are also having a bit of a spike too.
NB: I am not OP but just providing some context as to why they might think this.
vanillax 7 hours ago [-]
[flagged]
LeicaLatte 4 hours ago [-]
I got out of GitHub last year and self hosted a vps that runs gitea.
My work llm provider is GitHub copilot and we are completely blocked since morning. My question is who’s comparable to copilot vscode and cli. Multi model is table stakes.
So glad I moved my OSS org mostly over to a self-hosted Forgejo + Woodpecker CI. I get that we are operating on a completely different scale but I swear the GitHub outages are far too regular than they ought to be.
mosura 7 hours ago [-]
At least we have moved on from the tedious hug-ops response.
The question is which features do we want of Github but functioning in a distributed way?
Identity? Stars? Issues?
thelastgallon 8 hours ago [-]
Did they switch to Azure?
datadrivenangel 7 hours ago [-]
They've been moving to it, which is likely part of the problem.
I think for Github to redeem itself in the community eyes, it should come out with a very detailed technical analysis (short of proprietary details) what went wrong technically and what went wrong from a management perspective. I think it is hard to believe the line about the LLM influx at this point. I think nothing short of a full postmortem break down will restore trust in Github at this point. Even then it Github has probably lost nearly all trust and ability to turn around its reputation in the short to medium term.
chilipepperhott 3 hours ago [-]
I absolutely agree. I don't feel that I have sufficient evidence that they even really care about these reliability problems.
kwanbix 7 hours ago [-]
Well, seems that AI vibecoding is working it's magic!
twistedpair 6 hours ago [-]
My repos and PRs have gone from throwing 500 errors, to now all being 404.
Not trending in the right direction here...
hohoangphu 6 hours ago [-]
Unable to log in to GitHub at this time. "No server is currently available to service your request..."
5 hours ago [-]
herpdyderp 8 hours ago [-]
I've actually started loving GitHub down days because it means I get to spend time migrating away from them! (moving to self hosted Gitea)
1saadcodes 6 hours ago [-]
This has been happening a lot now huh
__vivek 8 hours ago [-]
Someone tried loading the Bun Zig-to-Rust PR again.
cebert 5 hours ago [-]
GitHub can’t focus on the basics like a stable platform. However, when they release stack commits a few weeks ago somebody had time to make the hamburger menu an unprofessional stack of pancakes…cute.
cvburgess 5 hours ago [-]
While i get the sentiment and the frustration, I highly doubt an SRE worked on the hamburger menu.
In my experience some teams move a lot faster than others and that leads to asymmetric quality. If all devs could pick up a new language and skillset overnigh then companies would have a lot more flexibility in their roadmaps, but humans don't scale that way - with or without AI.
ivanjermakov 6 hours ago [-]
I have a habit of reading the source code via github web interface instead of cloning and reading it in a much more comfy text editor.
Today I was forced to clone one because of "the incident". Which is a good thing.
jemiluv8 9 hours ago [-]
Awwww maaaan. I was just reading an upstream patch on the erpnext framework. I was going to do their copy the pr to patch my erpnext v15
tempaccount420 9 hours ago [-]
Why haven't we all moved off GitHub yet?
Or let me ask a more useful question: if we were to move, then to where? Ideally somewhere new (not Gitlab)
ndriscoll 9 hours ago [-]
Ideally something like IPNS to have a mutable ref based on (repo, pubkey, branch) to resolve a commit hash, a DHT for who has what commit hashes, and then just p2p transfers. You could still have a github or gitlab for issues and CI, but code (and maybe issues) could all be p2p like git is already designed for. Your forge could also act as a peer for code cloning, of course.
bhouston 8 hours ago [-]
I do not need it to be meaningfully different than Github, it sounds like that introduces additional complexity. I just want Github that works.
ndriscoll 8 hours ago [-]
The complexity is in locating what is already a distributed database item (git is fundamentally already designed that way), just using a protocol meant for that. Basically just support `git clone` with e.g. IPFS URIs.
ShinyLeftPad 8 hours ago [-]
Who's we? Your company?
Otherwise there's no "we", Git is decentralized. Everyone can use whatever they want.
esafak 8 hours ago [-]
What do you use for code review and CI/CD?
duxup 8 hours ago [-]
For us the uptime issues have been annoying but not consequential enough to justify the work / risk of moving.
I suspect even poor uptime for many services is a cosmetic problem for most users.
mrkaye97 9 hours ago [-]
I think the clear reason is even with their unreliability, the cost of migrating off of GitHub for _most_ places is not worth it, and so companies don't / won't (yet)
ArcHound 8 hours ago [-]
This bugs me so much. Management decides what is reasonable cost of service and the employees have to deal with all bullshit coming from that product. And you can't really escape it. Maybe the real moat was enterprise deals we made along the way.
mrkaye97 8 hours ago [-]
Could be - I’d also argue the time cost to the engineers to switch is pretty significant too, regardless of the monetary cost
datakan 8 hours ago [-]
Its hard to get lemmings to change course.
gortok 7 hours ago [-]
So how’s using Azure to power GitHub going?
Seriously. It seems like there’s an issue every other week. It would historically be normal to provide post mortems, and maybe I’ve missed them but seems like there are almost weekly issues and no follow up into the core reasons why.
wejick 7 hours ago [-]
I can imagine the massive growth they're in right now, old assumptions are breaking down, many things need to be rethought.
That's why personally i don't hate them with all of this saga. But they need to work faster in terms of product decisions and technology.
mhovd 7 hours ago [-]
I really wish I could move to a different provider, but GitHub is just so damn practical. Downtimes such as this should at least be compensated, providing an actual financial incentive to improve their services, instead of slapping AI on yet another product.
rickreynoldssf 4 hours ago [-]
Does anyone get the feeling that Microsoft is trying to kill GitHub in favor of ADO?
junto 2 hours ago [-]
That’s not the story I’m hearing. Large numbers of developers were moved away from ADO when they bought GitHub. ADO has more or less been on feature freeze for years.
rossdavidh 3 hours ago [-]
Less conspiratorially; they may care a lot less about it than ADO, and not give it the priority/resources they would otherwise. Neglect can often do the work of malign intention.
xet7 9 hours ago [-]
1) Issues showed unicorn. Reloading page.
2) Issues showed Not found.
Anyway, GitHub Actions for building new release has progress. Good enough.
bfrog 8 hours ago [-]
GitHub is down days of that week that end with day these days. If I were a paying customer I'd be cancelling soon. GitHub is sort of foundational at this point and when its down nearly every day of the week it cannot be foundational.
samuell 7 hours ago [-]
A self-hosted git-based issue tracker like Epiq [1] seems more relevant by each day going forward.
The GitHub failure unicorn image always brings me back to the good old days of trying out new Android ROMs. That unicorn used to be the logo of the Android Open Kang Project, which used to be my favorite ROM.
I wonder how it made it onto GitHub.
z3ugma 8 hours ago [-]
You know, I think Fossil with its integrated issue management right in the repo is actually a perfect match for the LLM era. Maybe today is the day I migrate to using a Fossil forge with GitHub becoming increasing untenable.
Anyone else using Fossil?
robin_reala 8 hours ago [-]
Biggest problem with Fossil is immutable history. That’s a design choice, and I respect them for that, but it’s not the right choice for me.
melezhik 9 hours ago [-]
If someone needs lightweight self hosted git server with ci pipelines via general programming languages - consider DSCI - http://deadsimpleci.sparrowhub.io
ferguess_k 8 hours ago [-]
What is really interesting is that no insider engineer has leaked anything. What we got was essentially "LLM" hitting GitHub really hard. But there is no retro or maybe i missed it.
nightski 7 hours ago [-]
All of my code repos are on gitea (personal and professional). The instance is not exposed to the internet (wireguard access). It's fantastic honestly, simple and just works.
ramoz 7 hours ago [-]
Nobody's replacing the social and community aspects of GitHub.
I think it's fine if we begin to move back towards distributed software supply chain infrastructure.
Ithildin 6 hours ago [-]
On the plus side, Fable is sooo snappy right now with everyone blocked by GitHub. I've never seen such fast response times.
b800h 8 hours ago [-]
Well I was dumped out of Github Copilot as this issue started, and haven't been able to get back in. Seems like a slightly strange thing to be connected, but feasible.
prdonahue 7 hours ago [-]
What are some good SaaS alternatives to GitHub for CI (that aren't GitLab)? Is anyone doing anything particularly novel in this space?
stephenway 4 hours ago [-]
I build Fjord, managed Forgejo with runners. The idea is to keep the GitHub-like workflow without having to operate the forge and CI infrastructure yourself.
bagol 7 hours ago [-]
Why excluding GitLab?
whalesalad 7 hours ago [-]
gitlab is a pig to self host.
Dezvous 7 hours ago [-]
Their omnibus docker image isn't bad at all in my experience
If you are required to use a Microsoft tool is Azure DevOps now a better choice? Our Azure hosted instance rarely (if ever) has had an outage that effects our ability to work.
lionkor 8 hours ago [-]
As someone who maintained a product that was, until recently, on Azure DevOps, stay away. Like, really really far away. There's nothing positive I can say about it.
chilmers 9 hours ago [-]
The worry I would have is, is anybody at MS still working on it? It is barely updated anymore:
That's not necessarily a bad thing for stability, but long-term I would not be surprised if it gets EOL-ed, or radically shrunk to just a basic Git mirroring service.
Discussion about it going ELO has been around since 2020-ish. But that is a worry i agree.
I get the feeling there is a long tail of old-school corps (migrated from the TFS days) that will keep the lights on. And for some orgs you just need "good enough" CI/CD pipelines that are reliable.
Maybe I'm a weirdo, but i find the kanban/sprint board pretty usable as well.
hinkley 5 hours ago [-]
This weekend I managed to get a test into some sort of infinite loop that didn’t overflow or time out, so I had to single-step in to figure out where the last line it broke on. Something to do with events propagating. Oh and someone misunderstanding an unfamiliar mocking framework and not preserving someone else’s cleanup code. I won’t name and shame but his name stars with an h and ends with inkley. Mangled someone else’s abandoned PR thinking one cleanup function was a superset of another. It was not, dumdum.
Event systems have interesting ways to fail that resist throwing good engineering at them to fix. For a long time I refused to have anything to do with them, but now we have whole companies built on them and also things like React and what can you do? There are harmonic failure modes where the system doesn’t fail but an event keeps echoing forever and using up 50% of the available compute before anyone notices it’s been like that for months and slowly getting worse. And then there’s the Poison Pill where a message crashes a process, then gets picked up by the next and kills that one too until they’re all dead.
I remember a few years back when there was 2 major outages per month.
Thankfully this is encouraging me to stop working overtime for now and enjoy the rest of my night
sarjann 8 hours ago [-]
You would think for a company that has close to a monopoly of developer mind share wrt git that they'd maybe put in the bair minimum to get several 9s.
nickstaggs 7 hours ago [-]
Has Github Enterprise Server been isolated from these issues? Presumably it has because it's self hosted
allanmacgregor 9 hours ago [-]
So whats the alternative? I'm thinking I'll migrate my company to something self hosted, but what about personal projects?
warmwaffles 8 hours ago [-]
For your personal projects, do they _need_ to be on a remote git server with a full GUI?
Is this GitHub's way of testing how many of you are managing your PRs via agents?
biennvops 9 hours ago [-]
Seems to affect web as well. Currently working on a PR and now it says "Merge status cannot be loaded"
grilledchickenw 9 hours ago [-]
Yep, started with PR changes not being visible, but now can't load entire repos.
I see the pink unicorn.
esskay 9 hours ago [-]
Githubs a great advert as to why you shouldn't use Azure. What an utter mess its become since they took over. Their argument that its scaling issues due to increased traffic just makes them and Azure look incompetent.
deadbunny 7 hours ago [-]
Is it a day that ends in Y?
DataDive 8 hours ago [-]
Maybe they should charge a little more per action, and the onslaught would stop.
Seems like a simple solution.
dhruv3006 9 hours ago [-]
how long can they survive like this?
yndoendo 3 hours ago [-]
As the saying goes, 'No one gets fired for choosing Microsoft solutions'.
I've talked to a number people in IT and commonly found they use Linux or Mac at home, as their daily, and Windows only at work.
Talking one of the IT works in the company I work for, they were forced to move from GitLab to GitHub because the CTO wants Microsoft everywhere. When all you know are screwdrivers, a personal will gladly use them as a hammer, punch, ....
Back in my IT days, we were pushing for marbled solutions. Example, 50% of the computers have antivirus brand Y and 50% have antivirus brand Z. Or the Windows servers used a different antivirus solution than the desktop. I don't see this anymore in the industry. One basket and all eggs go inside.
Irony, I'm updating code so our main products no longer require Windows as the host OS.
You have to pay me to use Microsoft products and solutions.
datakan 8 hours ago [-]
Microsoft has thrived like this for 40 years
Einenlum 7 hours ago [-]
Well, at least their status page is up.
I slowly but surely lowered my expectations.
AdyanshG 9 hours ago [-]
Which AI model is it this time?
spwa4 8 hours ago [-]
[flagged]
tacanae 7 hours ago [-]
Do better. What a gross comment.
felooboolooomba 8 hours ago [-]
Anyone remember the end of the .com era? This feels early similar.
augustohp 9 hours ago [-]
On the bright side, they were pretty fast on the Incident Declaration this time.
hohoangphu 6 hours ago [-]
Cannot retrieve latest commit at this time. :<
r0b05 6 hours ago [-]
There is no reason to be on github anymore.
Your code will used for training and the platform enshitification has been accelerated by Microsoft. It would be wise to move to another platform or even better, host your own Gitlab/Forgejo/Gitea on a vps or at home.
It's all green which means it's working. Ignore the unicorn error on every page.
edit: After 5-10 minutes they got around to opening an incident and (apparently manually?) marking the API as having 'degraded performance'.
cyphar 8 hours ago [-]
I find it quite amusing what they seem to consider "degraded performance" to mean -- I guess if you really squint, your service not working at all is actually just very large latency spikes (until the service recovers) and thus can be labelled as "services experiencing degraded performance for a small portion of our customers".
malfist 8 hours ago [-]
Even better "degraded performance" doesn't seem to count against their uptime stats. Issues is labeled as "degraded" with 100% uptime.
coolThingsFirst 5 hours ago [-]
Does anyone know what could be the cause of this?
LLM code or just terrible management?
cryo32 7 hours ago [-]
"AI will destroy SaaS"
"No not like that!"
leumon 9 hours ago [-]
Not sure if related, but claude seems to be down for me too.
ape4 9 hours ago [-]
Will our hug of death now make githubstatus.com crash
I thought by the near-unanimous distaste for IIS we would have learned our lesson about making Microsoft products a central part of infrastructure, but evidently I was wrong.
I shit on Microsoft a lot here, but it’s not because the workers are dumb or that all their products are “bad”, because they’re not. It’s because the company very clearly is only ever focused on “the next marketable product”, and will let their existing products rot.
You can see this with a lot of Microsoft products. Windows NT was a a pretty cool OS/kernel at its time. NTFS was a pretty neat filesystem when it was new. They employ smart people for enough time to make a big headline, then make no effort to keep those products any good.
kerv 7 hours ago [-]
Dear Mr AI. Please code a new GitHub for me and my friends. Make no mistakes.
greatgib 7 hours ago [-]
"degraded performance" is the new 2026 term to say that your platform is down unusable.
On my side it is funny,I'm encountering the following errors: "no server available to process your request".
Maybe azure is out of capacity in its datacenters... Or they didn't pay the server bill.
bemcho 9 hours ago [-]
yes in Bulgaria
saliei 7 hours ago [-]
Damn it, this is pissing me off actually.
qsera 9 hours ago [-]
Yes, in India.
amir734jj 7 hours ago [-]
GitHub is too important to fail, and they obviously can't fix their up-time problem. Shouldn't they open-source core and/or problematic components and ask for public to fix it? this is not sustainable. It's literally blocking development/productivity worldwide.
logicallee 8 hours ago [-]
Does either Google or Amazon (the two other large cloud providers besides Microsoft) operate its own alternative?
ebfe1 8 hours ago [-]
It's DNS issue, i'm telling ya!
catigula 8 hours ago [-]
Okay, enoughs enough, time to migrate our org.
porridgeraisin 8 hours ago [-]
Clones, repo browsing etc are all working.
/pull/:id/changes is not working, but /pull/:id.diff _is_. It seems parts of the API is also ok? some gh commands I tried worked.
Files changed works on the mobile app somehow. I guess a particular api version or something is down.
Anyway if someone needs files changed and merge status, check the mobile app. Past actions are also loading on that.
As I said 4 days ago GitHub is in chronic operational decline which is killing trust in hosting repositories [0]
You can use as many AI agents as you want (it won’t solve the problem), but with that many outages ever since getting bought by Microsoft, it is more than enough time to give them to change. They don’t care and there is no CEO of GitHub to help either.
A self hosted instance would have much better uptime than GitHub and GitHub Actions at this point as I predicted 6 years ago. [1]
Do y'all have any evidence that github's recent outages are caused by slopcode rather than the significant increase in load, which is exactly what Github & MS have been claiming as the root cause?
I'm tired of these BS stories being pushed by green accounts on HN like there's some damn agenda by someone who thinks their job will be saved "if only the CEOs realized that Slopcode Bad".
123ahsG 7 hours ago [-]
> I’m based in Brussels, working in tech, AI, and startup funding.
Orange account? I'm tired of the financially invested pushing their agenda. My job is secure, thanks. I don't work in the bullshit industry.
scrollaway 7 hours ago [-]
I have zero investments in agentic coding; in fact, my list of investments is pretty public, so I don't really know what point you're trying to make. But creating a new account to try to defend yourself against a threat you claim to be non-existent doesn't scream "I feel secure" to me.
1ashG16 5 hours ago [-]
You are rambling like Trump. Please leave the EU.
128e3123 7 hours ago [-]
[flagged]
dabbz 6 hours ago [-]
I mean one of those makes them look better than the other. That makes it tricky to be able to tell what is truth vs half truth vs a lie. You're right though, based off of what they claim, their great success is the cause of their failure.
How hard is it to up scale their servers though...
linhns 7 hours ago [-]
“Can’t get latest commit”
Ffs GitHub, get your act together.
CafeRacer 7 hours ago [-]
"oh shit, here we go again"
hmokiguess 8 hours ago [-]
I'm not mad, I'm just disappointed.
szew404 7 hours ago [-]
yes in germany
OtomotO 8 hours ago [-]
I have one client that relies on github. Just one, out of dozens I've worked with over the years.
It's just a constant pain in the ass.
Wherever I have anything to say I prefer ANY other gitforge over this piece of shit.
MrFurious 9 hours ago [-]
Oh, shit, here we go again...
karel-3d 8 hours ago [-]
Is that even news at this point?
Yes, github now has issues. Github is always having issues.
calvinmorrison 8 hours ago [-]
reminder: git is distributed
__vivek 8 hours ago [-]
Go touch grass
Bridged7756 6 hours ago [-]
I'll never get lack of sex used as an insult, unless you are a teenager. Is your self worth that tied to sexual activity so to use it as an insult? Is it that a rare thing to "achieve" for you to put it on a pedestal and demean others based on the lack of it?
coolThingsFirst 5 hours ago [-]
Go touch grass isn't meant metaphorically. It literally means go outside. Has nothing to do with sex.
garo-pro 6 hours ago [-]
[dead]
ArcHound 9 hours ago [-]
[dead]
r_blake 9 hours ago [-]
[dead]
white_dragon88 8 hours ago [-]
[dead]
Brian_K_White 8 hours ago [-]
[flagged]
sashank_1509 7 hours ago [-]
[flagged]
amrit_mirch 8 hours ago [-]
[flagged]
mikert89 9 hours ago [-]
[flagged]
webel0 9 hours ago [-]
I object to the notion that having "american born engineers" is the cause of their issues.
Nextgrid 8 hours ago [-]
It ain't the cause, but is definitely an enabler, because foreign talent on work visas effectively depends on their job much more so than their domestic counterparts and are thus very unlikely to push back against unreasonable demands or workload.
geraneum 8 hours ago [-]
I don’t watch sitcoms anymore. I come read HN for these gems.
noosphr 8 hours ago [-]
We may as well call them working hostages.
righthand 8 hours ago [-]
No but it demonstrates a disinterest in building software products and a bigger interest in playing business cost economics and securing their moat. Part of their strategy to force as much money into AI-hype as possible.
TremendousJudge 9 hours ago [-]
I have worked with some "american born engineers" that were some of the least competent people I've ever seen, while getting paid 2x or 3x of some "foreign contractors" on the same team
O5Council 8 hours ago [-]
[flagged]
mikert89 8 hours ago [-]
^
eddieroger 9 hours ago [-]
Yeah, that was unnecessary. Incompetent are born and work everywhere. In this case, Microsoft's problem with GitHub is trying to shove the square peg that is GitHub in to the round hole that is Azure.
aurareturn 8 hours ago [-]
Unfortunately, my company uses Outlook and Teams. Both are the worst piece of software I've ever used. They're just trash.
The UX is so bad. Things don't work as expected. Even basic things like composing an email, copy and pasting, formatting text are always a chore.
Try setting up a Teams integration vs a Slack integration. Gmail is absolutely light years ahead in UX. I really miss working at a company that uses Slack + Google Suites.
bbkane 8 hours ago [-]
I'm not a huge fan of Team, but I do have to give them credit for making the experience better over the years- a few years ago, search didn't even work!
aurareturn 8 hours ago [-]
- Terrible experience setting up bots/integrations with Teams
- Super poor formatting when trying to do more complex posts such as headings and bullet points in replies
- Copy and pasting images often don't work
- When you upload a video, sometimes the last 1-2 seconds get cut for no reason at all
- Keyboard shortcuts are suboptimal. For example, cmd+ shift + v which should clear formatting does simply does not work on Teams/Outlook.
- Grouping chats is far less intuitive than Teams channels
Nextgrid 9 hours ago [-]
Don't worry, they are also working hard at destroying those products too.
9 hours ago [-]
SoMomentary 8 hours ago [-]
I'm not sure I understand the significance of them having almost no American born engineers. Do we think it would be meaningfully different if they were? I tend to believe that Americans can be just as incompetent as the next guy!
mikert89 8 hours ago [-]
outsourced cost center
bogdan 8 hours ago [-]
> Their employee base in Washington has almost no american born engineers.
First of all, I highly doubt it. Second, how would you even know that?
knorker 9 hours ago [-]
> Their employee base in Washington has almost no american born engineers.
Was that a sharp turn, or was it related in some way?
8 hours ago [-]
mikert89 9 hours ago [-]
the whole software lifecycle is an outsourced cost center
monkey_monkey 9 hours ago [-]
Your comment history is full of barely-hidden racism. Get some help.
dpoloncsak 8 hours ago [-]
Off-topic clearly. but is there like an "NA/AA" for racism? Or like a therapy program or something akin to? Does 'usual' therapy touch on racism?
Like..where does one 'get help' for being racist? Honest question, I don't mean to sound snarky...
cyanydeez 9 hours ago [-]
what the hell does having american born engineers have anything to do with software quality.
Man. This site is filled with weird racism.
8 hours ago [-]
mikert89 8 hours ago [-]
outsourced cost center
cyanydeez 8 hours ago [-]
yeah, that's called capitalism. What does it have to do with "foreign".
If Capitalists could, and are doing, they'd make alabama be engineers if they could afford it.
It's so weird watching the tech work force go ham into the "tey terk ow jobz" rather than look at capitalism.
lucky-rathore 8 hours ago [-]
+1
bob1029 6 hours ago [-]
I am noticing a fairly mundane trend in that the GitHub outages tend to coincide with periods of (likely) high demand. This feels to me like a capacity issue more than a software bug crisis. At 4am on Saturdays GitHub never has any problems.
Centralization combined with massive AI demand is almost certainly the problem. If you move to GitHub Enterprise (on your own servers), you can probably exceed the availability of the current public cloud environment.
It is surprising that GHES rarely comes up as an option. It is a really good answer to this particular issue and eliminates the need to retrain the team on new tools. I understand the inclination to get salty about a vendor and cast the proverbial baby out with the bath water, but we must resist our impulses if we wish to make decisions that the business will find agreeable. The incremental additional licensing cost of moving from public cloud to on-prem is roughly $15-20 per user seat. If you only have 10-20 people using GH today and are deeply struggling with the consequences of these outages, this seems like a pretty easy decision to make.
Can't even view diffs in the web interface!
I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issues, GH actions CI, and pages host to deploy static sites. I don't use more than 100 mins of CI a month. Any recommendations?
edit: GitLab free tier has all I need, but I kind of want to pay and have some kind of reliability (not necessarily an SLA, but). But their next tier is $29/mo, which is too much.
- put Forgejo on a little $5 Vultr VPS (I share the VPS with other things, Forgejo doesn't need much resources. I also added some block storage to host my Docker containers in the Forgejo Docker registry)
- got a 8GB ram cheaper Hetzner VPS (you have to wait for their availability) for around 8 EUR and used that to install the CI runner
- Use Netlify for hosting static sites and PWA
It takes a bit to set up but everything works snappy, clean, and flawlessly. I have unlimited CI minutes. As a bonus I get free Docker image and registry hosting because it's included with Foregejo. I am so, so happy with it all.
Also with Forgejo you just feel like you're using something clean and solid. GitHub started to feel more and more bloated, slow, and unstable.
I don't care much about the social aspects of GitHub, but I do care a ton about hosting code for long periods of time, long after I've completely changed my personal homelab.
GitHub's most essential service for me is it being the equivalent of the Internet Archive, but for code.
Forgejo has some quirks (like no ability to share workflows between private repos) but I'm happy with my choice so far.
Just recommend swapping vercel/netlify with something else, we had quite a few issues and high costs with them. Not opiniated here, currently using Cloudflare and BunnyCDN (which is great)
just shy of 3 hours to find the issue
Advantage: You have to coordinate with teammates to make sure you don't need the same file at the same time.
Sometimes, not being able to work independently can be an advantage. Two people discovering they're trying to work on the same file can lead to useful conversations.
It's clear that updating the status page as quickly as possible isn't even something github cares about.
“The future is here it’s just not evenly distributed”, and that applies inside companies too.
Hard to actually judge without being in the room and knowing the team that owns the incident. And as other said, it’s pretty common IME to have a root cause but slow roll communication of it until remediation has more confidence.
Status pages for large companies are social, political and legal tools, and not technical indicators. It has always been this way, and likely always will unless some laws were passed.
So a "component" failed? That sounds like a software component. We just saw in https://www.wiz.io/blog/red-agent-snowflake-copilot-cicd-bug how sloppy bot coding is.
[1] https://sourcehut.org/pricing/
I don't understand the entitlement in your post here.
Excuse me, I'm literally asking where I can pay to get the specific cloud/web-based features I listed. I'm trying hard to NOT rely on a free tier.
To diagnose at a distance it feels like they need a stronger QA process, it's a product that scales horizontally, but somehow they find a way to break it for everyone on the regular! They need less hub and more node!
I don't need one, but I do have that because it's doing a bit more work than just git hosting (administratable access for others on the team, build actions, etc) and yes... I'm paying for it.
Pay every Nth month, or M months per year :-) Use the free tier features throughout.
Forgejo has been great for me too. I liked it so much, I added a feature to make it relatively easy to get up and running on exe.dev. Support for Gitea and Tangle.org is also available.
It’s kind of a shame, because Github keeps giving me fantastic ways to toot my own horn, but I can hardly do that if I can’t improve on their numbers.
Not a prop to GitLab, it's kindof table stakes now, just saying whichever place you go, you can find or write a migrator that will capture most of what's in GitHub, assuming their API is up long enough...
You can basically self-host your repos on a cheap VPS. I did that some time ago. You'll have to create a ssh user that has only 'git' as a shell, iirc, on that box. Works okay. But doesn't provide a web ui.
I think that's where https://git-scm.com/docs/gitweb becomes handy.
(maybe F500 people have their colocated box, but I have the impression that those are the "first class" seats (very expensive but _very few_) whereas enterprise cloud is the "business class" (loooooots of seats making a lot of money)
More realistically, MAU or something is a metric/OKR. It’s easy to imagine why a business wouldn’t want to cap that.
The audience will go away to somewhere you aren't. Then what?
There are people in this thread that have said their biggest reason for not moving away is the network effect.
AI makes investors/shareholders go brrr.
There's a lot of consumer surplus with the features GitHub offers. I'm sure they can have better pricing tiers to capture the surplus without losing too many users.
The question is: are they willing to compromise growth for QoS. So far the answer is a resounding: "no". They are deliberately sacrificing the experience of existing users to gain new ones.
X doesn't even need to be that low. It could be 100 or 200. Just something that slows down the agents that spam them with commits/pushes.
Seems presumptuous to assume a root case while the incident is still ongoing, no?
They’ve been having issues for a while.
If they make it harder to push slop on github, people might treat it as an incentive to generate less slop. Less tokens sold. Bad. Very bad.
The github paying customers are not going to move away.
Plus, it's probably genuinely hard to scale. And they can't throw compute at the problem - all compute is already going to the LLMs.
Last month, I made more commits than my entire 2025 combined.
But that discards my point, which is that you should probably have been prevented from pushing all those to github unless you were paying them for it, but you weren't because the executives running github can't admit that AI is bad for their platform.
So probably 10x more value is my guess for my entire team.
If you mean translating that into revenue, I'm not sure. Our product is better than ever with features a much bigger team couldn't dream of building before. Stability has also been better with fewer bugs, issues, downtime. When there are bugs, it gets fixed far quicker than before.
I would gladly pay that fee if it means Github is reliable for paying customers.
I think it behooves them for the public presentation of their issues to too much load. If they implemented load shedding and still had issues the follow-up questions would be less comfortable.
Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again.
The hope is dead.
But maybe I’m wrong?
The "AI is using a lot of resources" excuse was maybe acceptable last year but not in Q3 2026.
Managers and executives though? Those I do blame. Surely at this point it should be blindingly obvious their current strategy is not working.
You mean people pushing LLM assisted coding, or github's own coders pushing LLM assisted code to production ? :-).
I am more inclined to think that it is the second, but if the reason is the first, what is particular hard to scale ? It's not like everybody is vide-coding into a single repository, so I would expect that most of the load would naturally be scalable by increasing the resources. If for some reason that is not possible, I would expect a corporation as big as Github to keep the paying customers working by gracefully denying service to the free tier, not to collapse like an house of cards.
Why not both?
Despite minimal new features, and drastically increasing hardware resources to keep up, self-hosted GitHub enterprise went from things being instant to most requests taking many seconds. And activity on issues in one repository tends to mysteriously re-open issues with the same number in other repositories.
I think they are vibe coding and facing the consequences.
Do you think they have such measures in place? I remember when my indention wasn't spotless our university rejected my commits or if my function complexity was too high.. Tears in my eyes.. I think it was called Gerrit Code Review.
No faith left. None. They turn everything into a quagmire.
At most normal companies you monitor your systems and address potential bottlenecks before they reach a tipping point. And generally you want enough headroom that a sudden 2-3x increase in demand wouldn't take out the service.
Either Github's technical leadership/talent is completely out of their depth (seems unlikely), or they're not hiring enough engineers to work on these capacity issues.
Regarding their leadership, I'd argue the latter implies the former.
So yeah I'd say they're out of their depth.
Microsoft has to spend more money on acquiring games studios for billions of dollars then shut them down.
I do know that very few systems are designed to handle a 14x increase in traffic painlessly (275M*52 = 14.3B), especially at scale.
Youtube and Insta and Facebook and many other social media platforms handle very resource intensive media for billions of people plus bots.
Increased influx from AI is not plausible when viewed from that lens.
Taking every developer in the world and multiplying by 12x-24x is not a-lot compared to many consumer oriented sites.
Second, you can’t ignore the payloads, even an AI mostly text push is nothing compared to, say, 10 iPhone pictures or one moderate video.
GitHub sells this as a service.
It has really just coasted on GH's network effect because everybody uses GH for git by default. It doesn't really have much direct competition and anything that does offer something like GH just doesn't have the same mindshare.
In that way, GH became more like MS.
MSFT is giving away most of their Azure Datacenters resources to AI, to make money and doesn't care if Github suffers along the way and drains Github's available resources to bare minimum. Leading to outages like this, but you're 100% spot on right too. All those Claude/OpenClaw instances produce so much Code, people setup more and more repositories.
Anyone here have data on whether the likes of gitlab have been impacted in the same was as github?
But I’m more lenient with codeberg because they are a non-profit and as you say, they are banning vibe-coded project because of the cost to infrastructure. In the meantime, GitHub is encouraging vibe-coding, multiplication of the source code, and genAI discussions.
That being said, a Git forge for a small/mid-sized company is not something that requires rocket surgery. Forgejo with an SQLite database and a daily backup to S3 would be more than enough for the majority of users.
The original source was Kyle Daigle, GH COO.
- https://x.com/kdaigle/status/2040164759836778878
- https://xcancel.com/kdaigle/status/2040164759836778878
How many hires went into infra and platform? How many went to Gen AI org units?
From what I see, and that is a constant wave of very meh features, it seems like the latter.
This is a very distorted lens.
Many on the outside might say the US is very pro-self and anti-everyone-else right now! America first, right?
Well, take a random country. We'll make a made-up one called Tristonia. Should Tristonia do things in Tristonia's best interests? Absolutely! If Tristonia had some kind of military or economic threat from another country, should they take steps to protect their country, resources, and people from the threat? Absolutely! The same thing the US would do.
Most of these people aren't necessarily anti-US, they're against being beholden to another country, especially one that has been incredibly... inconsistent, as of late.
They need to start getting rid of the free tiers for GitHub Actions and actually charge appropriately or increase prices for it.
I guess they're quite happy with that TBF, it doesn't seem like they're that interested in being a git forge anyway.
https://imgur.com/b0Xj3Z6
> Update 16:59 UTC - The degradation affecting API Requests, Actions, Git Operations, Issues, Pages, Pull Requests and Webhooks has been mitigated. We are monitoring to ensure stability.
> Update 17:30 UTC - Git Operations is experiencing degraded performance. We are continuing to investigate.
> Update 17:36 UTC - Issues is experiencing degraded performance. We are continuing to investigate.
> Update 18:48 UTC - API Requests is experiencing degraded availability. We are continuing to investigate.
Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn't someone recently calculate that GitHub is running at 95%? For comparison the Irish Rail service which is not reliable has 80% of it's trains run on time.
This seems absurd and really challenges a lot of ideas I had about big tech and cloud infrastructure. GitHub seems to have remained the dominant player relative to GitLab etc.
We don't care. We don't have to. We're the phone company.
It takes a while by hand, but I recently saw a migration of old Jenkins projects to Actions with AI (yea), and it was smooth.
(only 5% /s)
I still remember that they launched GitHub Actions in 2018 and back then I was thinking: They can't even make the website reliable, surely this will only cause them more problems.
https://news.ycombinator.com/item?id=48086786
The earliest recorded downtime for Git Ops I could find is in December 2018. https://www.githubstatus.com/uptime/8l4ygp009s5s?page=32
What I find a bit weird is why so many people rely on it. Today we have tons of VPS providers where anyone can host their on git repo and CI/CD pipeline.
What is preventing so many people to effectively have a custom setup?
My company is still on GitHub... for now. But incidents like this are becoming more common and I'm becoming less tolerant of it. I suspect many feel the same, but it takes time to persuade the masses.
In my eyes it would be a ginormous gain to our colective mental health to not have the expectation to contribute to open source projects for free, or to have "green squares" every day on a public GitHub profile.
Great that it doesn't work on you, but it works really well on a depressingly large number of people.
Open source maintainers being harassed or being demanded work through Github's issues. Or being feed AI slop to review endlessly. Or recruiters filtering resumes via Github activity graphs or repository stars.
And obviously the side shady market of all of the above (e.g. buying/selling popularity).
Like the parent comment, I can't wait for social media to be completely obliterated from code repositories.
This isn't "psyops". Sure, it's a bad side of social interactions, but when you interact with other people, there's always some of this - in person as well as online.
The lead-off argument here seems more like "interacting with society can be bad".
I have no argument that selling popularity is bad and filtering by stars is stupid, but I don't blame Github for any of that.
You could equally well assign the success of certain coders to YouTube because they came to attention by showing off their work in livestreams. Which I do not. They found a way to advertise their own success, but it's theirs.
The argument is that there's no need for that much social interaction to produce quality code, quite the contrary, the more social interaction, the more focus is lost.
I know it's an unpopular opinion, but there's a reason why the Linux Kernel communication is still done through acerbic email threads.
Even if the platform itself doesn't take off, the protocol and the data ownwership (you host your repo artifacts, issues, prs, etc on your ATproto server) mean that a competitor could come in easily if you were no longer satisfied with performance, reliability, or monetization.
It's a social network with github-like social features and git hosting, but you can still self host your git and have all the features like issues, PRs and so on just like GitHub.
And it's built on ATProto so you can also have your issues PRs and so on hosted on your infra, but everything interconnects nicely with your single account, just like using GitHub.
They even added a vouching system for users, to know who to trust based on who you already trusted.
https://forgejo.org/2023-01-10-answering-forgejo-federation-...
Seems like it's far from production-ready and there hasn't been much activity this year?
This is the way. Stop centralizing everything.
On the contrary, I believe it's been detrimental to the open internet and only serves to control us by the powerful.
I never trusted Github and never will.
I suppose people can use git too if they ever figure out how to send emails from a dedicated client.
You can argue that the lack of diversity in infrastructure is bad, and I wouldn’t disagree. I think we should have more projects to explore better ways of working. But, the tradeoff is that when you have a centralized system, like GitHub, you also have less friction. The learning curve is small when everything uses the same tools.
I do think GitHub has positively contributed to the software industry and open source. It’s been enshittified for sure, but it has had a golden era.
If I build something that satisfies me and say 100 users (who all star me) does that really say anything. Are those 100 stars even "real" any more?
All the old problems of the web are new again, and still remain unsolved.
some investors use stars (and star growth velocity) to determine which projects they invest or don't invest in. (not saying it's good or bad - just saying it's a real thing.)
There are many git hosting options, including some for self hosting. its a lot easier to switch to a different version control host than to a different version control system.
Surprised that this wasn't just a link to Git. The biggest benefit of Git over predecessors was that it enabled decentralization. Github re-centralized the common case for the sake of convenience.
In comparison to SVN and other non-DVCs git did not need to continuously talk to the central server for history, "branch" creations, unlocking/locking files before changes, etc.
Linus wanted a DVCS, but they wanted centralization and licenses were expensive, so he decided to launch the git initiative where he provided scaffolding and the rest came from contributions.
EVERY other client I've ever worked with used their own gitforge or at least bitbucket...
Hell, even I do it.
Ever since the Twitter -> X rebranding, every time I open Twitter links, I have a very high chance of hitting a "Something went wrong" type page, with a retry button that just does nothing. It's been like this for ages. I barely open Twitter links anymore as a result.
This splintering should happen more often.
Note that raw user numbers won't really cut it; if 1 million people left Twitter, it's possible that 500 000 of those people joined and 500 000 new people joined; the user numbers would look like everyone returned, but really only 50% did in this hypothetical.
On Twitter, it's mostly the networking effect, you want to be where peoples are, and unless a large chunk move at once, you are stuck there.
On Github, it's the technical cost of migrating, the git part is trivial, but tickets, PR history and CI/Integration are another can of worms.
Projects which moved away, moved away for good, at most only leaving a synchronized mirror.
Github is actually down and I can't push code right now, it's blocking my business which is unacceptable. It's a broken tool.
and this assertion is based on ... ?
I only really use GitHub if a project that I want to contribute to is already on there at this point.
> Error: Response status code does not indicate success: 429 (Too Many Requests).
> Warning: Back off 19.714 seconds before retry.
> Warning: Failed to download action 'https://codeload.github.com/'.
> Error: Response status code does not indicate success: 502 (Bad Gateway).
> Warning: Back off 22.228 seconds before retry.
> Error: Response status code does not indicate success: 429 (Too Many Requests).
> Error: Failed to download archive 'https://codeload.github.com/' after 3 attempts.
Nice - even their own action runners are getting rate limited atm
No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it.
What's everyone else using? Self hosted Gitlab? Gitea?
Did you have the conversation "do we need to move from $whatever CI/CD to GitHub Actions?" beforehand? What were the arguments in favour?
- It was believed that GitHub would be more reliable than our self hosted Gitlab CI runners. This has proven to be very false.
- It was believed it would be easier to write tooling. I’m not on the dev tooling team, so can’t confirm if they found it true or not, but from the outside I see a lot more “Github can’t do that” than I did “Gitlab can’t do it”
Why is it hilarious to believe that the industry leader might be better at something than doing it yourself?
Frontier labs and global saas heavyweights run CI through us, plus a crop of open source tooling like Bazel, vLLM, Firecracker, Rails, Ray, Sorbet and Rust.
Shopify, Uber, Airbnb, Canva stresstested the platform ten years ago for scaling issues.
You need to take a little time to design upfront and take advantage of dynamically programmable primitives. It isn't the same ease/convenience as Actions. Bit more available though.
tangled.org!
Tangled.org really been working to ensure all their features work from data hosted on your personal data server (it's based on ATproto) which means should their performance, reliability, or monetization ever get you down it's easy to switch to a competitor since you're not locked in.
Not if you want to make sure it compiles and works on Windows (which you don't have) or macOS (which you don't have and can't emulate). That's something github gives for free, and for me, it's the only reason to stay on github for small/tiny free software projects (at least as a secondary host).
Cost as always depends on your usage, but it will likely be less than your current cost of GitHub Actions.
they've had multiple outages in the last week, so i guess they are proving themselves correct.
in practice, giving ssh/sudo to AI (codex) and telling it to setup ephemeral self-hosted runners on your own host VM (qemu), takes about 30 minutes and works extremely well. it bangs out all the shell scripts. moved my own business over to it. zero problems.
you might even find it is kind of neat too. you can customize your VM with software pre-installed. speeds up your builds quite a bit.
[0] https://www.blacksmith.sh/blog/blacksmith-seed-announcement
https://status.blacksmith.sh/cmsrg61yg09vd0lo9s1eagibm
I mean, that plus better uptime.
See: https://blog.codeberg.org/protecting-our-floss-commons-from-...
Our runners are faster, our CI is smoother than ever. Transitioning was a matter of a few hours, which we took as a fun hackaton.
https://www.youtube.com/watch?v=iuIdBfjL62s
I am seriously, 100%, contemplating the idea of offering consulting for migrating companies and projects out of Github. Given that I am fully employed, I'd only be able to do this on the weekends. Could I get a show of hands to get an idea of potential customers?
Once this is done, the team would have two possible git remotes to work with. Given that gitea also allows mirroring with Github, the team would be able to even switch to Gitea as the "main" git remote and for PR reviews even if the CI part is still on Github.
The harder part would be migrating the CI jobs. But this also can be done in parallel and no one needs to cut out their Github actions immediately. Start with the simplest of the stages, e.g, the automated test suite, and once you get the whole pipeline from Woodpecker on an staging environment, you'd could take another weekend to test the whole workflow. Once the first project is successfully migrated, it becomes a matter of repeating the process.
Woodpecker is still a bit finicky and debugging it can be a bit annoying, but given that it's all based on docker it is a lot easier to debug a pipeline locally.
For a 20–30 dev team I'd be happy to help work through what that rollout would actually look like
Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside.
If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health
My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced out by end-of-year due to incompetence. The new executive has taken to firing employees much more eagerly than their predecessor, like some sort of Jack Welch acolyte.
Their firing has put a lot of things into perspective for me. Mostly, fuck "at-will" employment and its negative effect on the American social contract.
But also this "angry ranting" online that the original poster was referencing. Not everyone has the privilege to calmly respond to things that directly impact their livelihood.
50% of social media traffic is bots and i want to believe
not to mention that any business which could potentially lose enough money that they would need to let go of developers from a github outage should probably already have some business continuity plans in place.
Please read https://berthub.eu/articles/posts/cyber-security-pre-war-rea...
we do not bill.
the outages they promised their customers are now different.
this has costs for everyone downstream.
It all ties back to the OP, where the issue you've might not be as bad as you think. I have been in situations where we have dropped all procedures to push a hot fix because we were actively bleeding money, and in situations where you know there is an issue, and you let it be.
Maybe I pay for a service and I want that service to work consistently during core business hours.
With every single of these enterprise 'cloud' offerings you are giving (almost) complete power over your business/project to somebody else who couldn't care less about your success or failure, you are simply irrelevant for them. I see it at work too, every time critical external systems go down whole bank stops still, just because few bucks were saved yearly on some on-prem servers.
Look at it this way, you are learning some important lesson today and finding great area of improvement for resiliency from now on.
It's not all that different from, say, an AWS region having a service impact. People would rather complain about AWS than prepare and utilize a well-tested recovery plan to shift to a standby region. Oftentimes there's no fallback plan because the business already considered it and decided it was too costly relative to the benefit, but when the incident happens, they still can't help but complain. Humans being humans.
Github does have an SLA: https://github.com/customer-terms/github-online-services-sla. But like virtually all SLAs, it's really just a token gesture. I have never seen an SLA that pays the losses you suffer due to the outage.
I worked for a few years in an exceedingly well capitalised place which ran everything in their own data centers, money no object, with a truck parked somewhere, ready to go, with a smaller version of our critical infra. We had a serious business-stopping outage once every 18 months or so, every time for fringe reasons one only learns about when trying to run a large data center. Its convenient to blame the cloud and pretend that self-hosting in private sector was so, so great with six nines.
However, I'm also of the boomer opinion that you should get what you pay for. "Ranting online" about a service (you pay for) being unavailable is a reasonable reaction. It's not like they have a call center you can dial into for support ...
But in general, it's not feasible to do everything in house.
And I don't think that GitHub is devoid of SLA: https://github.com/customer-terms/github-online-services-sla
The issue is that they're not achieving two nines uptime in practice.
I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory).
I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain.
You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org.
[0] https://damrnelson.github.io/github-historical-uptime/
Or, maybe your #1 IT priority was moving everything to Azure instead ;)
Largest shareholder (27%), not majority.
This is the outcome of violating single responsibility principle in business.
Edit: it seems GHE is also affected by the outage and isn't much better.
- we are getting no benefit beyond getting excuses replies to our emails - right or wrong, but we depend heavily on GitHub actions so this has a very real impact on us
There is no separate enterprise platform unless you buy their self hosted server product.
Which means they have a pricing problem. Rate limit non-paying users.
It's absurd that I have dozens of repos, many with GH Actions that run CI, test and then package and push to prod/package managers, and haven't paid GH anything.
The idea that we should be fine with this unreliability is just amazing. It's not a mental health issue to have problems when important infra fails.
the comment doesnt say you should be "fine" with the unreliability.
they are saying people shouldnt get so emotionally worked up over it. which, while i wouldn't phrase it in the way the parent did, i agree with the direction of their point.
"Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code"
"Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside." with assumptions like above.
If it was, "hey they f*ed up but there's no point having an overly emotional reaction" that would be fine. But he seems to be justifying this. Especially with Github's record up to now of unreliability I think it's completely fair to be annoyed.
No qualifiers necessary. But try arguing with a lawnmower...
Which they encouraged by pushing Copilot down everyone's throat
Is this rage bait? Isn't Google, Amazon, and all the other services in the world similarly impacted by LLM's? Isn't Claude, OpenAI, etc.? Why can they handle the load but not Github?
They could easily tighten things up in that regard, and make a choice that is right for their main users at the expense of The MS corpo mandate/mission. It is a choice to do otherwise.
If people and companies are paying for critical infra they expect it to be available - it's not MY problem that their servers are on fire - last time I checked, having too much demand was a good problem to have and they've had over a year to fix it and come up with a plan.
I'm so glad I migrated to GitLab months ago.
If they are facing an influx of commits from bots maybe they need to start metering commits from those accounts and charge them money.
This is a self imposed problem.
Github and their parent company are active participants in pushing LLM-driven coding.
However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow.
The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available.
(I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)
I’m not a Meta fan but it’s interesting that they manage to keep their systems up with an order of magnitude more traffic. GitHub’s uptime is inexcusable.
> Unless you have a emergency hotfix (you don’t)
Oh, we do. Given the sheer number of users, it's almost guaranteed someone is on fire ever time GitHub is down. Statistics is a very charming branch of reality.
I find it hard to believe that github APIs don't have rate limits to handle traffic spikes.
This should not be an excuse since they probably can use AI to fix it /partially sarcasm
hey bud your bias is showing
They get paid millions of dollars by my company. They need to get it together.
I have little time in my day to spend with my daughter. But I also have responsibilities. And one client has decided to bet on github. It's the only client I've ever worked with who has their code on github. EVERY other client has hosted their own gitforge or used bitbucket.
So now I am not angry because some critical piece of US-american infrastructure is down all the time.
I am angry because instead of spending time with my daughter I have to work on this later, because there are due dates and "Well, fucking GitHub was down" ain't gonna cut it.
My mind immediately went to this classic xkcd: https://xkcd.com/303/
Secondly, the scale is WAY off and makes it look far worse than it is. It makes it look like 99.5% availability is practically zero availability.
Finally, GitHub’s availability is not a binary all-or-nothing proposition. They report incidents on a granular level, for example, webhook firing can be impaired while Git hosting may be working fine.
github enterprise is fully operational at the moment
https://eu.githubstatus.com/posts/dashboard
We cannot pull any actions images (or how it is called) for example.
So with the "elapsed time" out of the way (now over an hour, incidentally), how about "how long will it be down?"
https://downdetector.com/
Why do we as software engineers have set fault tolerance and high availability as the primary goal of any software / infrastructure architecture? This shows it's completely irrelevant.
e.g.: gh pr merge 5062 -R <org/repo> --squash --delete-branch
I'm going to leave them alone for the rest of the day.
Github.com's promise is that it can be the central broker of open source code because it is reliable.
That promise hasn't been kept recently.
That said, Github is hard to displace and it is similar to the era of the Twitter Fail Whales. It was a sign of growth that couldn't be properly managed but there was not viable alternative.
Hard but not impossible. Being unreliable might nudge a few people. Also, Twitter had their fail whale in their early phase (mostly), not twenty years or so into their existence.
The issue with replacing Github is that it takes more than version control and issue tracking to replace the network that is Github. It's effectively the largest social network for software developers on this planet.
Tangled (tangled.org) is built on the at protocol (bluesky) and seems like an interesting take on this. Most other solutions seem to just move the problem to another much smaller silo (gitlab, codeberg, etc) or throw out the baby with the bathwater by focusing on self hosting without attempt at networking with other instances.
A federated approach seems like it is directionally right. Tangled is far from feature complete but I like the direction and it probably is a bit more resilient against global outages like this. I'm not a user yet but I'm keeping an eye on it.
May I ask what networking with other instances brings? Git is decentralised, so it works with self-hosted instances, right?
But if you want more than what email can offer, social networking is a thing with Github and it's a big reason for its success. It added a friendly UI and a way for users to engage with each other via issue trackers, pull requests, and all the rest. It has tens of millions of users that have accounts that each can each create issues, pull requests, fork each other's projects, etc. These people collaborate with each other on many millions of projects. The network effect of all these people being in one place where they can collaborate is vital to a lot of open source projects.
It's effectively the largest developer centric social network around. If the open source community fragments over a gazillion self hosted servers that each require a signup that don't really federate any signins and that don't allow aggregating notifications from all the projects you contributed to, etc. it would be a big loss for the open source world.
Tangled by making use of the atproto replicates much of this functionality. While also allowing people to run their own nodes and while allowing users and their projects to be moved between those nodes. That makes it the best of both worlds. You can self host it but without completely disconnecting from the community.
Assuming that everybody moves to your community (in this case the atproto one). Then the ActivityPub people ask you to join their community, and so does every new community that wants to become the new standard.
That's the thing with every new social network: "You don't want to be locked into the current community that is not my community, instead you want to move to my community, which is the best! And if you do, and convince everyone around you to join as well, then you will only ever need one account for everything!".
And now, assuming that most people leave GitHub and join Tangled, it will be to the detriment of the minority that is not using Tangled. If I host a repo on SourceHut, nobody will contribute because they want GitHub. If Tangled replaces GitHub, will that change? I don't think so, people will just refuse to use anything other than Tangled.
What I find interesting is that you advocate for a federated system that everyone should use, because "it would be better for everybody", but the fact is that everybody has an email already. No need to create an account, we can send each other patches, it just works.
Why don't you promote email? Because it's not your preference.
PS: I do realise that email has a lot of issues. My point is just that I avoid the entire issue by just being open to sharing my code however it works. And I feel like patches are easier to share "universally" than pull requests. Maybe I would be in favour of a new, "richer" patch format that would make it even nicer to share over whatever system I want (email, atproto, activitypub, signal, ...)?
Tangled is kind of there as an alternative that has most of the comforts of Github but is federated and can be self hosted. And there are a lot of Bluesky users that have access to it even if they might not be aware of that yet. If enough projects switch over, their contributors might follow along. If something else shows up, more power to them. But I'm not really aware of (m)any other properly federated alternatives.
The beauty of federated protocols of course is that they can be bridged and connected in all sorts of ways. Some hybrid at proto / email based bridge could be pretty doable. And if somebody wants to expose at proto feeds as activity feeds, that's probably not too technically hard either. These things aren't actually mutually exclusive.
https://youtu.be/iBDUTaKWVdU
They're struggling to maintain a single nine these days.
Edit: I just realised it's hosted on gh pages and therefore probably can't be updated
https://www.githubstatus.com/ is today
Self-promo links below if you would like to star a few of mine good for developers.
https://gitlab.com/architecture-decision-record
https://gitlab.com/ways-of-working
https://gitlab.com/coordinated-vulnerability-disclosure
Infrastructure should be neutral. Codeberg is a social club
Infrastructure can never be neutral, because infrastructure is owned and costs resources to run. All infrastructure comes with rules, be they the voting consensus of some e.V. or state-run entity, the whims of an individual on their private server, or the corporate interests of something owned by a, well, corporation.
The problem isn't "neutral infrastructure". The problem is an overreliance on centralized services even though most such services are relatively easy to host yourself.
I run my own DevOps server based entirely on OSS. Is it github? No. Does it what I need it to do? Absolutely. Do I get 99.999% uptime? No, but when it's down or slow, I can fix it. When the corporate platform is down, I can only wait.
And "network effects" are not a good argument. Open Source was wildly successful long before centraliced DevOps platforms.
And in hindsight, the added friction of getting in touch with projects before being able to send them any code, was a benefit not a disadvantage. Because it acted as a barrier of entry that disincentivized alot of subpar code from hitting these projects in the first place.
The value these centralized services provide is the centralization itself - that makes things easy to find. Remember the web before Altavista? Finding anything was detective work, guessing domains, following links through multiple pages, digging in newsgroups, asking around, and so on.
Open source was successful, but not nearly at the same level it is now thanks to precisely platforms such as GitHub (and Launchpad, and Bitbucket, and Gitlab, etc) that added useful functionality such as bug trackers and project management on top of a version control system.
Now, if we could do something like GitHub on top of a network such as Mastodon, then we would have something that's centralized enough to be useful and distributed enough to be indestructible.
And in recent years, how is the centralized world of websearch doing?
https://www.techspot.com/news/101560-study-confirms-search-r...
> Open source was successful, but not nearly at the same level it is now thanks to precisely platforms such as GitHub (and Launchpad, and Bitbucket, and Gitlab, etc) that added useful functionality such as bug trackers and project management on top of a version control system.
None of that depends on a centralized provider. I have all the features you listed on my self-hosted platform as well.
Even if whoever owns and runs the infra would like that to be the case, the law of wherever they are can put restrictions on this as well.
My wishlist: free for open source and paid for closed source, able to do simple projects that are handwritten and also complex projects including AI LLM code that needs CI/CD/auditing, and hosting in the UK or EU.
Aside, I donate money to Codeberg because I think Codeberg is a wonderful service with strong commitment to free open source, and because I host some larger projects and AI projects that are for public welfare social good.
https://tangled.org/joelparkerhenderson.com/architecture-dec...
https://tangled.org/joelparkerhenderson.com/ways-of-working
https://tangled.org/joelparkerhenderson.com/coordinated-vuln...
To Tangled -- it's great that the core code is MIT licensed and anyone could stand up a competing appview (the credible-exit property of atproto).
Do Tangled Labs Oy's articles include a purpose clause instead of the default profit-only purpose[0]? Or is something similar on the roadmap?
[0] OYL 1:5 reads The purpose of a company is to generate profits for the shareholders, unless otherwise provided in the Articles of Association.
You're deliberately avoiding a network effect lock-in by being federated, you make competition trivial by giving away not only the protocol but also making your code available, and you don't even restrict commercial competition by using the MIT license rather than, say, BSL.
To me that sounds like you are deliberately starting a race-to-the-bottom competition on price with, well, everyone capable of hosting a SaaS? Heck, if you don't end up with some solid competition the whole "federated code" USP doesn't play out. To me that doesn't exactly scream "good return on my €3,8M seed round". Am I missing a brilliant plan here, or should we expect a rug pull a few years down the line?
Anyone who joins the association (small fee) can vote.
So, yes, it's technically a democracy, but it deserves extra nuance in my mind because it might not be what everyone immediately thinks a democracy should feel like.
So, relatively, not so long ago it was pretty common to host all this stuff yourself. I don't think (or maybe don't remember) Subversion having any Web interface of its own, but, since forever Git came with GitWeb. I remember configuring it for the first time: any extra bit of configuration, especially those that resulted in some added interactive functionality seemed exciting. For a week, I'd push something to my branch and immediately refresh the browser page to see my changes reflected in the remote repository :D
The infatuation wore off very quickly: after a month I have almost forgotten the Web interface even existed.
The cycle repeated with hosted services, the addition of PR interface, the addition of "pages", the bug reports, the built-in Wiki. After discovering the new gadget and associated hype, invariably, came the realization that the predating existing functionality was... better. The PR discussion in the Web page detached from the code is a lot worse than the PR discussion in the PR itself. The built-in issue tracker doesn't come close to the stand-alone product functionality. The built-in Wiki is ridiculously underwhelming when compared to an actual Wiki.
And so was the CI. For me, it started with Gitlab. There are oh so many reasons to hate Jenkins, so, whenever anyone announced an alternative platform, I'd absolutely check it out, and so did a lot of others, I presume. I was super hyped up about the possibility of running CI from the command line, completely locally on my laptop. And then it turned out to be a bait-and-switch kind of thing. I've created monumental dozen-steps and branching pipelines connecting multiple projects only to realize that the best CI is the one that uses as little as possible of the platform tools.
In the end of the day, I came to realization, that beside the social aspect of the platform like GitHub, everything that it offers on top of the essential Git repository storage is too low quality. And so do competitors. If I was ever again tasked with setting up a company code repository storage, I'd go with a physical server attached to the local office network, unless there's no office, in which case, probably, some EC2 VM would do the job...
Places like GitHub or Gitlab lure you in with their free tier offering, where you get the storage space and some other goodies for free, but once you need to upgrade to the paid tier, you inevitably realize that it's not worth the money they charge for it.
GitLab and Codeberg can absolutely become trash, each in their own way.
The solution isn't just "migrating off of GitHub"; it's "migrating to a platform that is unownable by design, not only hope".
And there are no good enough decentralized alternatives to GitHub yet.
Tangled.org's decentralization has serious holes. Radicle made an alien universe; a niche within a niche. Also, it's already having problems with ISPs: https://radicle.dev/2026/04/23/domain-move
Issues/PRs/comments are records owned by their authors, not by the repo. So an issue can span N users' PDSes. That's a neat model for social media, but a forge wants the repo to be the durable owner/context of its issues and history because that's where the comment/issue/etc. logically belongs.
The solution? tangled.org acts as a super advanced cache/aggregator for that. What can possibly go wrong?
> As time goes on we are re-assessing the idea of users owning what is "collaborative data" (issues, PRs, etc.) on their PDSes - soon may come the day that an issue also lives on the knot as a source of truth, with an accompanying pointer record on user PDS to attest that it's theirs.
from https://tangled.org/tangled.org/core/tree/master/knot2
Well it's 100% down for me, so does this mean 20% of all users are fully down?
> Posted 1 minute ago. Aug 17, 2026 - 13:58 UTC
It also took them 20 minutes to realize that peoples requests to /pull/xx are not working :D
Or is their point about abandoned projects where the dev never clicks release after merging, or even never merges?
In May they had to solve for 30x growth overnight, and I would guess its closer to 100x by now. The reality is a centralized solution like this may no longer be a viable option with the scale of code generation.
You have a free bank holiday, go look at green trees, listen to birds, find somebody who's wrong on HN etc.
The only poor souls here are some github devops team (team of copilots?) and those using alternative services.
PS: Saw Cursor’s Origin announcement a second later.
I am not trying to make any excuses for GitHub, however I recall one tweet that made my eyes bug-out:
> Yup, platform activity is surging. There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
> GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week.
https://x.com/kdaigle/status/2040164759836778878
https://xcancel.com/kdaigle/status/2040164759836778878
---
edit: see a counterpoint I posted down-thread.
Manager: Why is the site down!?
Engineer: Because it can’t handle this level of traffic. We can rate limit to the level we can handle to keep the service running.
Manager: Okay, let’s do that!
Engineer: Okay. Rate limiting at N req/min.
Manager: I’m getting complaints we’re denying service to customers!? We can’t rate limit our users if they want to use it. Increase the rate limits.
I’ve had this conversation more than once in my career.
Imagine the software development lifecycle is a river blocked by several dams- necessary processes that also slow the water’s volume and speed. Writing the code was a very big dam upstream, and successive dams didn’t need to handle a lot of water. With that dam busted wide open, the torrent of code is inundating downstream processes. GitHub happens to host many of them.
Earlier this year the GitHub COO shared a post on Twitter with explosive numbers they were facing. I cannot seem to find it at the moment. (Edit: shared in other replies to this thread.)
I would argue they are. You can't cache a git repository and send people what it looked like a minute ago, because someone might have pushed 2 seconds ago.
Alternative theories include "azure sucks" and "they're vibecoding everything now".
Many people suspect it's the year-long migrating to Azure and all their products instead of the things they have been using.
Whatever the actual increase in productivity may be, the commits, clones, pushes, action runs, etc. are skyrocketing.
"Microsoft’s data-center shortages to persist longer than expected - Bloomberg"
https://ca.finance.yahoo.com/news/microsoft-data-center-shor...
Perhaps relevant tho. I would imagine if they could simply throw more hardware at it they would. And I wouldn't imagine inability to throw hardware at it would be due to software scaling issues. So perhaps it is simply lack of available hardware?
…& this is ignoring all the moral reason for avoiding US-based, publicly-traded, proprietary software for host largely free software. They also deny access to users due to US sanctions, privacy-conscious users won’t want to create accounts/agree to ToS, & rubbing shoulders with other corpos your project can be taken down if it upsets the corpos don’t like it (see youtube-dl, Switch emulators, etc.).
Interestingly the API still works for creating an issue (but webhooks weren't fired).
It is time to give up on Github. This is just too much.
For people who want a similar experience today, I’d recommend trying Codeberg (An official instance of Forgejo) though. However, keep in mind they voted in July to ban mostly AI generated repositories. You can read about that here: https://blog.codeberg.org/protecting-our-floss-commons-from-...
I started with the cloud version, then moved to self-hosted because it wasn't a big effort for our team. We also reduced our monthly invoice by about 50% after the move.
* Forgejo
Those are my favorites. Both you can self-host for your company, including CI/CD.
We also have a free public Forgejo instance if you want to try it first.
brutal
It is technically 20%, because a bunch of the requests that happen on page load do succeed. Just not the few crucial ones that are required for the page to load correctly - those return a 500 the vast supermajority of the time.
For supermajority: the construction means “beyond majority” or “above majority”, it is implemented as a higher percentage in most systems, and there is a utility in being able to refer to high ratio majority.
So people have repurposed a technical word to a general word, of roughly the same meaning.
"Disappoint" was originally the inverse of "appoint" and meant to remove from office.
Update: yes that worked. gh CLI using the API was able to make a PR while the web interface was inoperable.
Second update: defeat from the jaws of victory -- the page that displays open PRs is also down.
Third update: CI isn't working anyway, so the CLI PR creation workaround doesn't achieve much.
Enterprise works ok-ish, but the standard version has daily issue ever since.
They're making it very easy for a viable alternative to pop up and take their lunch - it just wont be Gitlab.
I never had to wonder if Steam is going to be working today, so I could play my game after work, but it's been an issue with GitHub since covid. At least for me.
Fortunately, due the nature of the service, I can sit out most downtimes. Most of the time at least.
It's a shame I had to go to HN to check if I was the only one instead of relying on their status page that they link.
A self hosted version of GitHub that is a precise mirror of an actual GitHub repository. That is, PRs from GitHub appear on the selfhosted instance and my reviews from the self hosted instance appear on GitHub. That way I can work on my private instance when GitHub is down (or slow) and I can be assured my work still appears to everyone online.
Does this exist?
the closest thing i've found are tooks like https://github.com/raylabshq/gitea-mirror
GitHub's value is in the PR process, in CI/CD, releases, the issue tracker, and an identity system propping all of that. This is the thing to replace, and it's quite a bit harder.
https://git.tinygrad.win/tinygrad/tinygrad
Sure use it to collab externally, use it if you can't justify running your own infrastructure, but why are large companies choosing to place code externally - which is one secret leak away from compromise?
"Yeah we're unable to offer services, but so is [X BIG CORP] and [Y BIG CORP]. It's an industry outage, nothing we can do!"
> Update - We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become available Aug 17, 2026 - 13:45 UTC
NB: I am not OP but just providing some context as to why they might think this.
My work llm provider is GitHub copilot and we are completely blocked since morning. My question is who’s comparable to copilot vscode and cli. Multi model is table stakes.
https://github.blog/news-insights/company-news/github-availa...
https://github.blog/news-insights/company-news/an-update-on-...
The question is which features do we want of Github but functioning in a distributed way?
Identity? Stars? Issues?
In my experience some teams move a lot faster than others and that leads to asymmetric quality. If all devs could pick up a new language and skillset overnigh then companies would have a lot more flexibility in their roadmaps, but humans don't scale that way - with or without AI.
Today I was forced to clone one because of "the incident". Which is a good thing.
Or let me ask a more useful question: if we were to move, then to where? Ideally somewhere new (not Gitlab)
Otherwise there's no "we", Git is decentralized. Everyone can use whatever they want.
I suspect even poor uptime for many services is a cosmetic problem for most users.
Seriously. It seems like there’s an issue every other week. It would historically be normal to provide post mortems, and maybe I’ve missed them but seems like there are almost weekly issues and no follow up into the core reasons why.
2) Issues showed Not found.
Anyway, GitHub Actions for building new release has progress. Good enough.
[1] https://github.com/ljtn/epiq
Did Anthropic or OpenAI fail to sandbox their models again?
https://downdetector.com/
I wonder how it made it onto GitHub.
Anyone else using Fossil?
I think it's fine if we begin to move back towards distributed software supply chain infrastructure.
https://docs.gitlab.com/install/docker/installation/#install...
https://azure.microsoft.com/en-us/updates?filters=%5B%22Azur...
That's not necessarily a bad thing for stability, but long-term I would not be surprised if it gets EOL-ed, or radically shrunk to just a basic Git mirroring service.
Discussion about it going ELO has been around since 2020-ish. But that is a worry i agree.
I get the feeling there is a long tail of old-school corps (migrated from the TFS days) that will keep the lights on. And for some orgs you just need "good enough" CI/CD pipelines that are reliable.
Maybe I'm a weirdo, but i find the kanban/sprint board pretty usable as well.
Event systems have interesting ways to fail that resist throwing good engineering at them to fix. For a long time I refused to have anything to do with them, but now we have whole companies built on them and also things like React and what can you do? There are harmonic failure modes where the system doesn’t fail but an event keeps echoing forever and using up 50% of the available compute before anyone notices it’s been like that for months and slowly getting worse. And then there’s the Poison Pill where a message crashes a process, then gets picked up by the next and kills that one too until they’re all dead.
Thankfully this is encouraging me to stop working overtime for now and enjoy the rest of my night
Seems like a simple solution.
I've talked to a number people in IT and commonly found they use Linux or Mac at home, as their daily, and Windows only at work.
Talking one of the IT works in the company I work for, they were forced to move from GitLab to GitHub because the CTO wants Microsoft everywhere. When all you know are screwdrivers, a personal will gladly use them as a hammer, punch, ....
Back in my IT days, we were pushing for marbled solutions. Example, 50% of the computers have antivirus brand Y and 50% have antivirus brand Z. Or the Windows servers used a different antivirus solution than the desktop. I don't see this anymore in the industry. One basket and all eggs go inside.
Irony, I'm updating code so our main products no longer require Windows as the host OS.
You have to pay me to use Microsoft products and solutions.
I slowly but surely lowered my expectations.
Your code will used for training and the platform enshitification has been accelerated by Microsoft. It would be wise to move to another platform or even better, host your own Gitlab/Forgejo/Gitea on a vps or at home.
It's all green which means it's working. Ignore the unicorn error on every page.
edit: After 5-10 minutes they got around to opening an incident and (apparently manually?) marking the API as having 'degraded performance'.
LLM code or just terrible management?
"No not like that!"
I shit on Microsoft a lot here, but it’s not because the workers are dumb or that all their products are “bad”, because they’re not. It’s because the company very clearly is only ever focused on “the next marketable product”, and will let their existing products rot.
You can see this with a lot of Microsoft products. Windows NT was a a pretty cool OS/kernel at its time. NTFS was a pretty neat filesystem when it was new. They employ smart people for enough time to make a big headline, then make no effort to keep those products any good.
On my side it is funny,I'm encountering the following errors: "no server available to process your request".
Maybe azure is out of capacity in its datacenters... Or they didn't pay the server bill.
/pull/:id/changes is not working, but /pull/:id.diff _is_. It seems parts of the API is also ok? some gh commands I tried worked.
Files changed works on the mobile app somehow. I guess a particular api version or something is down.
Anyway if someone needs files changed and merge status, check the mobile app. Past actions are also loading on that.
[Android, India]
You can use as many AI agents as you want (it won’t solve the problem), but with that many outages ever since getting bought by Microsoft, it is more than enough time to give them to change. They don’t care and there is no CEO of GitHub to help either.
A self hosted instance would have much better uptime than GitHub and GitHub Actions at this point as I predicted 6 years ago. [1]
[0] https://news.ycombinator.com/item?id=49276571
[1] https://news.ycombinator.com/item?id=22867803
https://www.freethink.com/robots-ai/github-copilot
Slop code is written and pushed ...
Aug 11, 2025: Dohmke leaves the sinking ship:
https://www.entrepreneur.com/business-news/github-ceo-thomas...
This is a true AI victory.
I'm tired of these BS stories being pushed by green accounts on HN like there's some damn agenda by someone who thinks their job will be saved "if only the CEOs realized that Slopcode Bad".
Orange account? I'm tired of the financially invested pushing their agenda. My job is secure, thanks. I don't work in the bullshit industry.
How hard is it to up scale their servers though...
Ffs GitHub, get your act together.
It's just a constant pain in the ass.
Wherever I have anything to say I prefer ANY other gitforge over this piece of shit.
Yes, github now has issues. Github is always having issues.
The UX is so bad. Things don't work as expected. Even basic things like composing an email, copy and pasting, formatting text are always a chore.
Try setting up a Teams integration vs a Slack integration. Gmail is absolutely light years ahead in UX. I really miss working at a company that uses Slack + Google Suites.
- Super poor formatting when trying to do more complex posts such as headings and bullet points in replies
- Copy and pasting images often don't work
- When you upload a video, sometimes the last 1-2 seconds get cut for no reason at all
- Keyboard shortcuts are suboptimal. For example, cmd+ shift + v which should clear formatting does simply does not work on Teams/Outlook.
- Grouping chats is far less intuitive than Teams channels
First of all, I highly doubt it. Second, how would you even know that?
Was that a sharp turn, or was it related in some way?
Like..where does one 'get help' for being racist? Honest question, I don't mean to sound snarky...
Man. This site is filled with weird racism.
If Capitalists could, and are doing, they'd make alabama be engineers if they could afford it.
It's so weird watching the tech work force go ham into the "tey terk ow jobz" rather than look at capitalism.
Centralization combined with massive AI demand is almost certainly the problem. If you move to GitHub Enterprise (on your own servers), you can probably exceed the availability of the current public cloud environment.
It is surprising that GHES rarely comes up as an option. It is a really good answer to this particular issue and eliminates the need to retrain the team on new tools. I understand the inclination to get salty about a vendor and cast the proverbial baby out with the bath water, but we must resist our impulses if we wish to make decisions that the business will find agreeable. The incremental additional licensing cost of moving from public cloud to on-prem is roughly $15-20 per user seat. If you only have 10-20 people using GH today and are deeply struggling with the consequences of these outages, this seems like a pretty easy decision to make.