Speaking out loud in case anyone sees this, but regarding jumbo frames:
why aren't they the default since, I don't know, fiber ISP rollouts? Outside of a slight increase in latency and maybe loss of value with small webpages/fragments, surely ISPs can handle it and you could upgrade fiber ONTs to easily handle 2.5, 5, 10 GigE or fiber directly to your internal network. There's almost no excuse. Other than I guess resource starvation of the end device processing data. Am I not understanding something?
Giefo6ah 7 hours ago [-]
Path MTU discovery is broken on the Internet because of (like always) overzealous firewalls that block all ICMP.
nextaccountic 5 hours ago [-]
Then maybe there should be another MTU discovery algorithm on top of whatever firewalls don't block?
zamadatix 4 hours ago [-]
There are algorithms but it's a difficult problem (e.g. it's not always just something you get away checking at the start and then it means the session works til the end) all for a minor benefit in some use cases theoretically. Practically, it can often end up worse or just buggy even after a long time https://bugzilla.mozilla.org/show_bug.cgi?id=2048618
zorgmonkey 5 hours ago [-]
QUIC does exactly this using Datagram Packetization Layer Path MTU Discovery (DPLPMTUD), one of the many things I like about QUIC.
topranks 6 hours ago [-]
The problem is ICMP is often dropped on the internet, making path MTU discovery unreliable.
If the whole world switched to 9000 MTU in the morning everything would be fine.
As things stand if you increase your network to it and allow hosts send packets that size you’ll hit sufficient edge cases and icmp black holes to give you issues.
jon128 4 hours ago [-]
Most people who work at ISPs, the eyeball networks in particular and much of the lower tier networks, just don't care. They do enough to get by in their jobs and "good enough" is the rule rather than the exception. Most of the execs are this way too.
As far as making the internet better, they're generally not interested unless they're really up against a wall that hits their bottom line in a way they can comprehend ; the ones they don't comprehend they tend to ignore too.
caminanteblanco 9 hours ago [-]
To summarize for the uninitiated, I learned today that IPv6 supports packets of up to 4GB! Why you would ever need a packet within even an order of magnitude of that size is beyond my ken. (Also, I don't know what transit media even support packet sizes anywhere close to that size).
The only place I could see the layman ever getting close to using this would be loopback. But at that point, you have IPC.
tosti 8 hours ago [-]
IPv6 is designed to last. Just because we can't fathom a use case today doesn't mean we'll never have one.
nomel 4 hours ago [-]
In the future, when everyone has 100TB internet speeds, this might be one frame of a realtime 3D video call!
Pet_Ant 9 hours ago [-]
> loopback
I mean inter-container communication on a host?
Also I assume it’s not a design goal to support 4GB but it is just a result of a power of 2. They are basically just doubling the size from 16 to 32 bits. Trimming it to 30 bit to cap it at something "reasonable" 1GB or something is more hassle then it’s worth.
I just cannot imagine the error correction on those packets. I mean if anything goes wrong that's a lot to toss away.
wahern 7 hours ago [-]
> I just cannot imagine the error correction on those packets. I mean if anything goes wrong that's a lot to toss away.
I think the original jumbogram work occurred back when alot of people were still convinced end-to-end IP alone wasn't suitable for many uses. This was the era when, e.g., people were still pitching ATM or exploring bespoke solutions for linking research centers. So basically jumbograms were presumed to be delivered atop another long-haul, multi-link protocol that would provide reliability. Jumbograms were a way to improve the throughput and latency between IP peers with minimal but high-impact changes to application and kernel network stacks (presuming IPv6 adoption), placing most of the complexity in the network transit layers. What we ended up with, ironically, was the reverse, everything IP and below largely ossified from software perspective, some minor evolution at the TCP layer, but most progress happening above TCP/UDP, and alot of work invested in local hardware link layers that remains opaque to IP or even the ethernet software stack, effectively placing a ceiling on dividends.
caminanteblanco 9 hours ago [-]
I hadn't thought of that, that would be a potential use case for sure. And I agree that literal 4GB packets probably aren't what's in mind here, but I think it is an oddity that a confluence of other factors has it allowed by the RFC.
riffic 9 hours ago [-]
Transit media was addressed by RFC 6214 - avian carrier! Section 3.1 states that "an adult carrier can carry jumbograms [RFC2675]".
A 4 GB microSD card is well within the carrying capacity of a pigeon.
https://www.rfc-editor.org/info/rfc1149/ has more specific information in regard to the benefits of avian carriers. Particularly "because of the 3D ether space available to the carriers, in contrast to the 1D ether used by IEEE802.3."
Thank you riffic for sharing this, this reminds me of the SIGBOVIK papers :)
SkiFire13 7 hours ago [-]
This is the comment I was looking for.
finnthehuman 4 hours ago [-]
The person who shocked me with that fact put it this way: we already do 9000 regularly. What are the chances uint16 max is big enough for the rest of our lives?
4 byte length suddenly perfectly reasonable.
Palomides 9 hours ago [-]
I dunno, we've got 800Gb/s NICs with TCP offload now, you could send dozens of gigabyte jumbograms a second
andrewstuart2 9 hours ago [-]
Is there really that much room for improvement by changing packet size? You're avoiding a little bit of overhead for the 40 byte packet header and you're also increasing the risk of that packet failing CRC and getting discarded which is a lot more expensive.
Veserv 7 hours ago [-]
There is much improvement to be gained by changing packet size from the current de-facto limit of 1500 byte MTU, but requiring 32-bits is likely unnecessary as just increasing it to the current 16-bit limit of 65535 bytes would be adequate for any reasonably designed protocol.
Given current compute, you would be hard-pressed to control plane more than ~100 million packets per second per core which is ~1 Tb/core-s at a 1500 byte MTU. Increasing this to just 65535 is a ~40x increase to ~40 Tb/core-s (~6.5 TB/core-s). Clocking at 5 Ghz and issuing 2 cache-line sized stores per clock you can still only issue ~640 GB/core-s (~4 Tb/core-s) of memory traffic from a core (assuming you do not get memory bandwidth limited earlier than that). So, at a 1500 byte MTU you need a 1:4 data to control core ratio, but at 65535 you flip that to a 10:1 core ratio. This should be reasonable for basically anything for the foreseeable future.
topranks 5 hours ago [-]
The real problem is the packet arrival rate on today’s high-speed interfaces.
At 1.6Tb/sec, with 1500 byte packets, that leaves very little time for a router to make a forwarding decision and route each packet before the next arrives. It’s kind of wild we have the same max MTU on interfaces that speed as we had with 10Mbit Ethernet.
So far we’ve kept up with some very clever engineering, but I do wonder if it can continue indefinitely.
pixl97 9 hours ago [-]
I mean over the standard 1500 byte packet, for sure. I'd assume that you'd probably find an optimal spot at 1 to 8Mb on super fast networks. Also networks at these speeds tend to have very low retry rates. It's the buffer sizes that start to get crazy though.
jon128 4 hours ago [-]
Some ML/AI networks I have seen would benefit from packet sizes in this ballpark.
warkdarrior 9 hours ago [-]
My prompts to Claude will finally fit in one packet.
lwarfield 8 hours ago [-]
I crashed my college campus's routing infrastructure once by sending a 4GB packet!
bflesch 8 hours ago [-]
(1999) is missing. Crazy how far ahead these folks were planning, I wonder what they work on today.
cyberax 7 hours ago [-]
Sigh. IPv6 again, the tale of bad design and stupidity. IPv6 is forever stuck with 1462 MTU now because they neglected the PMTU discovery.
For larger packets to work, the MTU size signal has to be baked into _each_ packet. ICMP on truncation is NOT a good design.
Of course, there's a belated RFC to fix it: https://www.rfc-editor.org/info/rfc9268/ But it'll never get implemented and deployed, as we're deep in the "ossification" stage for IPv6.
Veserv 4 hours ago [-]
While that RFC is in the right direction, it is still not the right solution. The right solution is to just truncate packets on the path. The destination then compares the length stored in the packet (as literally every single protocol does) against the length of the packet actually received to determine the PMTU. This only fails if you truncate the in-band length, but the lengths are usually stored up-front, so this should only occur if you have a PMTU of less than ~100 bytes which is less than 1/10th of the PMTU guaranteed by IPv6 and less than 1/5th of the PMTU guaranteed by IPv4.
As all protocols must already check that the lengths encoded in the packet are actually in the bounds of the packet to avoid out-of-bounds accesses when handling incorrect or malicious packets, any existing destination that does not add support for PMTU determination via truncated packets must already safely drop those packets otherwise it is currently egregiously insecure. So, such a design is backwards-compatible with any destination that is not a gaping security hole.
The destination, now having learned the PMTU from source to destination, can then feed that back to the source in whatever way it chooses including, and preferably, in-band in whatever higher layer protocol it is using. Flowing it in-band of the higher layer protocol avoids ICMP or stateful firewall blackholes since it flows back inside of the allowed flow from the trusted endpoint and can even have its integrity and confidentiality maintained instead of you getting random ICMP packets from unknown and unverified entitys that may or may not even be on your path.
cyberax 3 hours ago [-]
Truncation is good, but it is still reactive. You first need to send a datagram that is larger than the window to get back the size signal, this is wasted bandwidth and wasted time.
Proactive MTU communication allows you to reliably discover it during the 3-way handshake with essentially no overhead.
topranks 5 hours ago [-]
Path MTU discovery doesn’t work on the internet for either protocol reliably so this is kind of a moot point.
immibis2 7 hours ago [-]
what you you mean they neglected discovery? Everyone knows that IPv6 stops working if you block ICMP because it blocks MTU discovery, ergo everyone knows it relies on MTU discovery.
cyberax 7 hours ago [-]
And yet plenty of networks bungle the ICMPv6. Especially when stateful firewalls are used.
It's also a fundamentally bad approach. ICMP packets travel outside the connection's packet flow, so you have to add a new global entity for it to work.
Truncation also is reactive, you can't proactively discover the MTU during connection establishment.
immibis2 6 hours ago [-]
There's no such thing as a connection flow. The packets are sent back to the source address.
cyberax 6 hours ago [-]
Yes, there is. Both as a kernel-level abstraction and as a logical entity.
Arguably, TCP should also have been redesigned. IPv6 assumes that hosts have multiple IPv6 addresses, yet only one address at a time can be used within a given TCP connection.
kbolino 5 hours ago [-]
> IPv6 assumes that hosts have multiple IPv6 addresses, yet only one address at a time can be used within a given TCP connection.
What IPv6 does differently is that each interface is expected to have multiple addresses, but each of those addresses serves a different purpose. Local network traffic uses the stable link-local address; incoming Internet traffic uses the stable global address; outgoing Internet traffic uses the global privacy address. And, even though privacy addresses get rotated for new connections, the old ones remain valid until all connections using them are closed.
So, there really isn't any reason to use more than one address for the same connection.
What it sounds like you're asking for is something like MPTCP, which was designed to solve a problem that didn't really exist when IPv6 was created.
cyberax 4 hours ago [-]
> What IPv6 does differently is that each interface is expected to have multiple addresses, but each of those addresses serves a different purpose.
Nope. IPv6 was designed from the beginning to support multiple Internet-connected routers. Even the earliest RFCs that defined protocols for router advertisement allowed for multiple routers and addresses.
Yet at the same time, TCP (and DNS) ended up stuck with the "one address, one host" model.
> So, there really isn't any reason to use more than one address for the same connection.
> What it sounds like you're asking for is something like MPTCP, which was designed to solve a problem that didn't really exist when IPv6 was created.
There is. To support Happy Eyeballs natively in the OS.
Happy Eyeballs should have been standardized along with IPv6 in the 90-s, but it took 15 years for it to be deployed!
In the counter-factual world, the IPv6 TCP would have been standardized on something like MPTCP from the beginning. There would have been one call to establish a connection: "connect(const char *hostname, uint16_t port)".
It then would have raced IPv4 TCP and IPv6 TCP connections, the TCPv6 then would have treated the V4 connection as a subflow. With perhaps multiple TCPv6 flows going over different source-dest address pairs.
kbolino 2 hours ago [-]
It feels like you're anachronistically saying IPv6 was always meant to work the way (you think and I agree) that it ought to work today.
As you say, Happy Eyeballs deployment is relatively recent. So is RFC 8028. The original multiple router support did not mean multiple, redundant addresses for the same purpose, it meant failover to the same upstream network using the same addresses, and even that didn't always work that well.
cyberax 26 minutes ago [-]
I did not say that IPv6 was focusing on multi-homing (IETF didn't even want to add PI addresses for IPv6 initially), but on multiple paths to the same uplink. This was very much an intention from the beginning, and the privacy addresses were a fast follow.
And the late addition of Happy Eyeballs is _the_ reason IPv6 is still mostly vaporware.
immibis2 5 hours ago [-]
Fairly sure the kernel is expected to just keep the address alive as long as it's in use, and applications are expected to be robust against brief connection interruptions anyway. (Looking at you, IRC)
There are some idiosyncrasies in the network stack. Nobody's invented a better one anyway.
topranks 5 hours ago [-]
v4 also relies on ICMP for pmtud. And fragmentation also doesn’t work, so I don’t really get your point.
cyberax 4 hours ago [-]
V6 had a chance to fix the MTU issues, and they screwed it up, like literally everything else about the IPv6 design.
theamk 9 hours ago [-]
maybe not everything needs to be IPv6?
No standard equipment can do 64K+ frames, so you are not going to be connecting to global internet soon.
Plus, why are you doing this at all?
If you are super-concerned with the overhead, you don't want IP stack anyway, with all its layers and tunables - you want something like RDMA which can work very close to metal.
And if you are OK with overhead, then why go with huge frames? Go with small ones and reassemble, your life would be much easier. (Normally reassembling is a pain because of possibility of lost or reordered packets, but if you are willing to go with such huge packet size, it must not be a concern with you)
(this reminds of the opposite direction: people stuffing IPv6 into sensor networks.. your frames are 32/64 _bytes_ and you want IPv6 headers in there???)
Dagger2 7 hours ago [-]
It's more convenient if there's a standard layer available. It means you can introduce new things at L1/L2 and they'll still work with all existing software, and also you can introduce new things at L4+ and they'll work with any L1/2 tech, so long as you make your new thing map to L3 in the middle.
> (this reminds of the opposite direction: people stuffing IPv6 into sensor networks.. your frames are 32/64 _bytes_ and you want IPv6 headers in there???)
You mean like in 6LoWPAN, which uses header compression to get them down to 2 bytes? While still working with regular TCP- or UDP-using software?
caminanteblanco 9 hours ago [-]
That was my read on it. I had no idea this was part of the protocol (I saw it while reading the IPv6 Wikipedia article). Like I said in my sibling comment, maybe, maybe, you could use this for loopback, but I can't think of a single use case that actually justifies its existence.
Rendered at 02:33:48 GMT+0000 (Coordinated Universal Time) with Vercel.
why aren't they the default since, I don't know, fiber ISP rollouts? Outside of a slight increase in latency and maybe loss of value with small webpages/fragments, surely ISPs can handle it and you could upgrade fiber ONTs to easily handle 2.5, 5, 10 GigE or fiber directly to your internal network. There's almost no excuse. Other than I guess resource starvation of the end device processing data. Am I not understanding something?
If the whole world switched to 9000 MTU in the morning everything would be fine.
As things stand if you increase your network to it and allow hosts send packets that size you’ll hit sufficient edge cases and icmp black holes to give you issues.
As far as making the internet better, they're generally not interested unless they're really up against a wall that hits their bottom line in a way they can comprehend ; the ones they don't comprehend they tend to ignore too.
The only place I could see the layman ever getting close to using this would be loopback. But at that point, you have IPC.
I mean inter-container communication on a host?
Also I assume it’s not a design goal to support 4GB but it is just a result of a power of 2. They are basically just doubling the size from 16 to 32 bits. Trimming it to 30 bit to cap it at something "reasonable" 1GB or something is more hassle then it’s worth.
I just cannot imagine the error correction on those packets. I mean if anything goes wrong that's a lot to toss away.
I think the original jumbogram work occurred back when alot of people were still convinced end-to-end IP alone wasn't suitable for many uses. This was the era when, e.g., people were still pitching ATM or exploring bespoke solutions for linking research centers. So basically jumbograms were presumed to be delivered atop another long-haul, multi-link protocol that would provide reliability. Jumbograms were a way to improve the throughput and latency between IP peers with minimal but high-impact changes to application and kernel network stacks (presuming IPv6 adoption), placing most of the complexity in the network transit layers. What we ended up with, ironically, was the reverse, everything IP and below largely ossified from software perspective, some minor evolution at the TCP layer, but most progress happening above TCP/UDP, and alot of work invested in local hardware link layers that remains opaque to IP or even the ethernet software stack, effectively placing a ceiling on dividends.
A 4 GB microSD card is well within the carrying capacity of a pigeon.
https://www.rfc-editor.org/rfc/rfc6214.html
Thank you riffic for sharing this, this reminds me of the SIGBOVIK papers :)
4 byte length suddenly perfectly reasonable.
Given current compute, you would be hard-pressed to control plane more than ~100 million packets per second per core which is ~1 Tb/core-s at a 1500 byte MTU. Increasing this to just 65535 is a ~40x increase to ~40 Tb/core-s (~6.5 TB/core-s). Clocking at 5 Ghz and issuing 2 cache-line sized stores per clock you can still only issue ~640 GB/core-s (~4 Tb/core-s) of memory traffic from a core (assuming you do not get memory bandwidth limited earlier than that). So, at a 1500 byte MTU you need a 1:4 data to control core ratio, but at 65535 you flip that to a 10:1 core ratio. This should be reasonable for basically anything for the foreseeable future.
At 1.6Tb/sec, with 1500 byte packets, that leaves very little time for a router to make a forwarding decision and route each packet before the next arrives. It’s kind of wild we have the same max MTU on interfaces that speed as we had with 10Mbit Ethernet.
So far we’ve kept up with some very clever engineering, but I do wonder if it can continue indefinitely.
For larger packets to work, the MTU size signal has to be baked into _each_ packet. ICMP on truncation is NOT a good design.
Of course, there's a belated RFC to fix it: https://www.rfc-editor.org/info/rfc9268/ But it'll never get implemented and deployed, as we're deep in the "ossification" stage for IPv6.
As all protocols must already check that the lengths encoded in the packet are actually in the bounds of the packet to avoid out-of-bounds accesses when handling incorrect or malicious packets, any existing destination that does not add support for PMTU determination via truncated packets must already safely drop those packets otherwise it is currently egregiously insecure. So, such a design is backwards-compatible with any destination that is not a gaping security hole.
The destination, now having learned the PMTU from source to destination, can then feed that back to the source in whatever way it chooses including, and preferably, in-band in whatever higher layer protocol it is using. Flowing it in-band of the higher layer protocol avoids ICMP or stateful firewall blackholes since it flows back inside of the allowed flow from the trusted endpoint and can even have its integrity and confidentiality maintained instead of you getting random ICMP packets from unknown and unverified entitys that may or may not even be on your path.
Proactive MTU communication allows you to reliably discover it during the 3-way handshake with essentially no overhead.
It's also a fundamentally bad approach. ICMP packets travel outside the connection's packet flow, so you have to add a new global entity for it to work.
Truncation also is reactive, you can't proactively discover the MTU during connection establishment.
Moreover, stateful connection tracking is a de-facto requirement for end-user firewalls: https://datatracker.ietf.org/doc/rfc6204/
Arguably, TCP should also have been redesigned. IPv6 assumes that hosts have multiple IPv6 addresses, yet only one address at a time can be used within a given TCP connection.
What IPv6 does differently is that each interface is expected to have multiple addresses, but each of those addresses serves a different purpose. Local network traffic uses the stable link-local address; incoming Internet traffic uses the stable global address; outgoing Internet traffic uses the global privacy address. And, even though privacy addresses get rotated for new connections, the old ones remain valid until all connections using them are closed.
So, there really isn't any reason to use more than one address for the same connection.
What it sounds like you're asking for is something like MPTCP, which was designed to solve a problem that didn't really exist when IPv6 was created.
Nope. IPv6 was designed from the beginning to support multiple Internet-connected routers. Even the earliest RFCs that defined protocols for router advertisement allowed for multiple routers and addresses.
Yet at the same time, TCP (and DNS) ended up stuck with the "one address, one host" model.
> So, there really isn't any reason to use more than one address for the same connection. > What it sounds like you're asking for is something like MPTCP, which was designed to solve a problem that didn't really exist when IPv6 was created.
There is. To support Happy Eyeballs natively in the OS.
Happy Eyeballs should have been standardized along with IPv6 in the 90-s, but it took 15 years for it to be deployed!
In the counter-factual world, the IPv6 TCP would have been standardized on something like MPTCP from the beginning. There would have been one call to establish a connection: "connect(const char *hostname, uint16_t port)".
It then would have raced IPv4 TCP and IPv6 TCP connections, the TCPv6 then would have treated the V4 connection as a subflow. With perhaps multiple TCPv6 flows going over different source-dest address pairs.
As you say, Happy Eyeballs deployment is relatively recent. So is RFC 8028. The original multiple router support did not mean multiple, redundant addresses for the same purpose, it meant failover to the same upstream network using the same addresses, and even that didn't always work that well.
And the late addition of Happy Eyeballs is _the_ reason IPv6 is still mostly vaporware.
There are some idiosyncrasies in the network stack. Nobody's invented a better one anyway.
No standard equipment can do 64K+ frames, so you are not going to be connecting to global internet soon.
Plus, why are you doing this at all?
If you are super-concerned with the overhead, you don't want IP stack anyway, with all its layers and tunables - you want something like RDMA which can work very close to metal.
And if you are OK with overhead, then why go with huge frames? Go with small ones and reassemble, your life would be much easier. (Normally reassembling is a pain because of possibility of lost or reordered packets, but if you are willing to go with such huge packet size, it must not be a concern with you)
(this reminds of the opposite direction: people stuffing IPv6 into sensor networks.. your frames are 32/64 _bytes_ and you want IPv6 headers in there???)
AKA: https://en.wikipedia.org/wiki/Hourglass_model
> (this reminds of the opposite direction: people stuffing IPv6 into sensor networks.. your frames are 32/64 _bytes_ and you want IPv6 headers in there???)
You mean like in 6LoWPAN, which uses header compression to get them down to 2 bytes? While still working with regular TCP- or UDP-using software?