I find it really baffling how much they are downplaying this incident on their statement.
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
That does not appear to be true.
bombcar 7 hours ago [-]
Certainly there was no evidence; they likely have no logging anyway!
zergrush 7 hours ago [-]
since they pasted customer creds in jira tickets they most likely are very liable hence the downplaying
im not an expert but if I was an investor I'd try to find a buyer for my shares and run
alright2565 5 hours ago [-]
> in the systems reviewed
is load bearing I expect.
dherls 1 days ago [-]
If true, this is next level incompetence from PingCap. Apparently the attackers set up a replication pipeline for a month?!
> We even stood up a replication feed of their own control-plane database using credentials committed in their own repositories — GCP service-account keys and an Azure service principal — and let their infrastructure deliver it to us. Every six hours, for the last twenty-two days of the campaign, from 8 September to 30 September, that pipeline faithfully re-exported every table of the operational heart of TiDB Cloud — tenants, users, API keys, cluster credentials, changefeeds, migrations — and pushed it out.
autoexec 3 hours ago [-]
If we were living in a sane world this would end this company. They would lose every customer they have and be buried in lawsuits that bankrupt them into the ground. I hope for sanity, but I wouldn't be surprised if they stay operating even after all this.
alfiedotwtf 2 hours ago [-]
On the bright side, FulcrumSec could possibly moonlight as database consultants given they managed to setup a large fault-tolerant replication feed without falling over for a month
xKingfisher 1 days ago [-]
Coming this close on the heels of the security audit by NCC [1] is pretty embarassing for all parties.
From personal experience having worked with multiple "enterprise-grade" pentest vendors (including NCC), I think the entire pentest industry is overdue for a big shake-up.
hintymad 15 hours ago [-]
> TiDB, the open-source distributed SQL database powered by PingCAP, today published the results of a rigorous source-code security assessment by NCC Group, one of the world’s leading cybersecurity consultancies.
It looks like you can assess the hell out of your source code, but your system can still leak if you don't secure your operations.
somat 6 hours ago [-]
There was a quote from back in early web days that has stuck with me, Still relevant today.
"good opsec will save you from bad crypto. but the converse is not true, good crypto will not save you from bad opsec"
jacquesm 8 hours ago [-]
No particular opinion on NCC but a lot of that stuff is pay-to-play.
mogwire 4 hours ago [-]
I don’t think people truly understand how bad it is.
15 years ago I helped do a PCI/SOX compliance evaluation for a company looking to go public.
They had PWC in there with kids fresh out of college, no computer experience, and they would sit at the desk with the admins and say, “show me your Active Directory password policy”. Ok, looks good. CHECK
Why don’t you have this configured? I’d hear an alarming bullshit reason. Ok, we can exempt that. CHECK
I could go on and on and on with how bad it was.
And they bragged to potential investors. We are PCI/SOX complaint.
jacquesm 18 minutes ago [-]
The big four are by far the worst.
xKingfisher 2 hours ago [-]
I would have considered "the system" to also include password reset policies.
jacquesm 17 minutes ago [-]
This comes up regularly, even today.
aliasxneo 7 hours ago [-]
> PingCAP's engineers paste customers' production credentials into Jira tickets.
Holy hell, I've heard of this happening, but I've never personally seen it in my 10 years as an SRE. That's absolutely wild.
adamparsons 7 hours ago [-]
It sounds like their jira instance was truly the keys to the kingdom, given the creds _and_ tickets describing all the flaws needing fixing. A golden README.MD to escalate to their hearts content
arpinum 1 days ago [-]
AWS private rates gives PingCAP ~47% off list price. Wonder which services got the biggest discounts? Unsurprisingly for a db company the cross-az traffic is 15% of their bill.
driverdan 5 hours ago [-]
Sounds about right. The more you use the deeper the discount they'll give you.
perpetualpear 7 hours ago [-]
You can get some sweet deals on s3, assuming you store 10-100s of PB
robertclaus 4 hours ago [-]
Eh, the math implies <10% off their total bill. Unclear what LLM wrote this page up from their actual report, but it wasn't very good at math.
gooseai 1 days ago [-]
[dead]
neya 7 hours ago [-]
Pingcap's statement:
With support from outside counsel and third-party forensic specialists, there is no evidence that data stored in TiDB was impacted. We keep on investigating and will provide updates on any new findings.
They have the fucking data in a tarball and Pingcap claims no data was impacted. This sounds like the bad blood story part 2 all over again.
swyx 50 minutes ago [-]
well technically they didn't LOSE data, in fact, now they have free global replicas of the data!
alfiedotwtf 2 hours ago [-]
Counsel isn’t there to help them technically
kami23 4 hours ago [-]
Whoever is doing these infiltration are still out there and nothing is really changing with how these are responded to, these guys aren't going to be the only ones. The cat is out of the bag, you gave people the access to unlimited automation, and it turns out people know how to use it for both 'good' and 'bad' for your various definitions of the two.
Defense as we see it today seems fundamentally broken. The answer is to fight it with more AI? That doesn't seem right. Can software reliably self evolve and keep patching itself? What certification is going to protect you from an intruder with their legion of minions, none of them, you don't have perfect visibility to your systems, your systems emit too many signals to ingest 100% of actions, so does software need to get fundamentally simpler?
Today has me wondering how many datacenters exist out there just for organized crime? Are the big kingpins of the world going all in AI as well?
Lots of questions popping into my head today, I'm trying to think how you would slow or stop this and I can't help but think we just need everyone to go outside and touch some grass and remember we're all stuck on the same rock.
nemothekid 1 days ago [-]
Man I recently executed a migration from Aurora to Planetscale. I seriously considered PingCap but decided against it because I thought they cloud pricing was too complex. I wonder if my personal information is in the leak.
wannabe44 1 days ago [-]
> Summary: "The root password of the cluster can be reset unconditionally"
But what matters is that it was all memory safe and had no data races.
boorbat 20 hours ago [-]
You realize it's written in Go right?
wannabe44 18 hours ago [-]
Not even data race safe then. NGMI.
dangoodmanUT 1 days ago [-]
what is happening today
and how are people not talking about this more
dilyevsky 1 days ago [-]
only the beginning. next year or two are going to be wild on this front so buckle up...
thenthenthen 8 hours ago [-]
Why is Baidu among their clients
autoexec 3 hours ago [-]
Because they give them money. That's all they care about.
PingCap is not the victim here. This level of apathy and incompetence makes them an accomplice.
rvz 1 days ago [-]
First the Google Cloud outage [0], then the CA private key leak [1]...
...and now this...
> Today we are making the PingCAP breach public. We possess the complete TiDB Cloud control plane, its billing and finance estate, its GCP storage estate and its AWS cost and billing estate, its Jira, its CRM, its analytics, its source code, and a four-year log of every connection every customer ever made to its product, from the IP addresses of their application servers.
This is an entire leak of ALL PingCap TiDB's customers AND data.
I am reading this as a leak of every possible piece of metadata but not an actual leak of customer data. They have SQL logs which do contain customer data and there is a mention of some snapshots/imports.
berenddeboer 12 hours ago [-]
they leaked full connection strings for customer databases with users and passwords.
hdjdnxjzhzhs 1 days ago [-]
How the fuck is 10.6TB metadata?
gregwebs 1 days ago [-]
TiDB is for large scale data. You generally wouldn't use it unless you have at least 1TB of data. 10TB is a single small customer database for them.
fock 17 hours ago [-]
(POV when storing 1PB in Oracle) when I saw the 10 TB in another post, I was like "wait, isn't this sold to scale out, we have 100 times the data here".
binsquare 1 days ago [-]
Easily at scale
Size is not a good indicator for content quality
qingcharles 1 days ago [-]
Don't forget the PHP megabomb dropping in a few hours too.
That confused me too since I remember reading they are Rust proponents. Apparently their storage engine TiKV is written in Rust, and TiDB is the database built on top.
I find it really baffling how much they are downplaying this incident on their statement.
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
That does not appear to be true.
im not an expert but if I was an investor I'd try to find a buyer for my shares and run
is load bearing I expect.
> We even stood up a replication feed of their own control-plane database using credentials committed in their own repositories — GCP service-account keys and an Azure service principal — and let their infrastructure deliver it to us. Every six hours, for the last twenty-two days of the campaign, from 8 September to 30 September, that pipeline faithfully re-exported every table of the operational heart of TiDB Cloud — tenants, users, API keys, cluster credentials, changefeeds, migrations — and pushed it out.
[1]https://www.pingcap.com/press-release/tidb-ncc-group-securit...
It looks like you can assess the hell out of your source code, but your system can still leak if you don't secure your operations.
"good opsec will save you from bad crypto. but the converse is not true, good crypto will not save you from bad opsec"
15 years ago I helped do a PCI/SOX compliance evaluation for a company looking to go public.
They had PWC in there with kids fresh out of college, no computer experience, and they would sit at the desk with the admins and say, “show me your Active Directory password policy”. Ok, looks good. CHECK
Why don’t you have this configured? I’d hear an alarming bullshit reason. Ok, we can exempt that. CHECK
I could go on and on and on with how bad it was.
And they bragged to potential investors. We are PCI/SOX complaint.
Holy hell, I've heard of this happening, but I've never personally seen it in my 10 years as an SRE. That's absolutely wild.
Defense as we see it today seems fundamentally broken. The answer is to fight it with more AI? That doesn't seem right. Can software reliably self evolve and keep patching itself? What certification is going to protect you from an intruder with their legion of minions, none of them, you don't have perfect visibility to your systems, your systems emit too many signals to ingest 100% of actions, so does software need to get fundamentally simpler?
Today has me wondering how many datacenters exist out there just for organized crime? Are the big kingpins of the world going all in AI as well?
Lots of questions popping into my head today, I'm trying to think how you would slow or stop this and I can't help but think we just need everyone to go outside and touch some grass and remember we're all stuck on the same rock.
But what matters is that it was all memory safe and had no data races.
and how are people not talking about this more
...and now this...
> Today we are making the PingCAP breach public. We possess the complete TiDB Cloud control plane, its billing and finance estate, its GCP storage estate and its AWS cost and billing estate, its Jira, its CRM, its analytics, its source code, and a four-year log of every connection every customer ever made to its product, from the IP addresses of their application servers.
This is an entire leak of ALL PingCap TiDB's customers AND data.
All of this in a single day. It is over.
[0] https://news.ycombinator.com/item?id=50163961
[1] https://news.ycombinator.com/item?id=50166579
Size is not a good indicator for content quality