Repository navigation
Limit for the maximum buffer size #1371
Description
Metal has a limit, the error looks like this (see forum):
/Library/Caches/com.apple.xbs/Sources/Metal/Metal-56.6.1/ToolsLayers/Debug/MTLDebugDevice.mm:89: failed assertion `newBufferWith*:length 0x20000000 must not exceed 256 MB.'
The limit appears to be exposed on MTLDevice as maxBufferLength property.
Here are some iOS numbers from SO:
- iPad Pro (12.9", 2nd generation) iOS 13.3 - 993 MB (1041235968 bytes)
- iPhone 11 iOS 13.3 - 960 MB (1007468544 bytes)
- iPhone 8 iOS 12.4 - 747 MB (783597568 bytes)
- iPhone 6 iOS 12.1 - 256 MB (268435456 bytes)
The closest thing in other APIs that I could find is Vulkan's VkPhysicalDeviceMaintenance3Properties::maxMemoryAllocationSize
Proposal: introduce a limit for the maximum buffer size, with baseline value of 256Mb.
Activity
kainino0x commentedon Jan 26, 2021
I really thought we'd be able to hide max buffer sizes behind
"out-of-memory"errors. I think that would work if max buffer sizes were in the same ballpark with system memory sizes. With values as low as 256MiB, that might not be so great. Though do we really target iPhone 6? If we're just talking about numbers close to 1GiB, I think it might be okay to continue hiding the limits behind"out-of-memory".kvark commentedon Jan 26, 2021
Yeah, I guess it could be fine. I wish we had some sort of guarantee about the values returned from
maxBufferLength.I know it was 1Gb on some macs before, and now it reports 2Gb on the 4 year old MBP.
potree commentedon Jan 26, 2021
256MB is very limiting and would block some of my plans to port progressive rendering to WebGPU and also block our future plans on processing and rendering large point clouds directly in browsers using WebGPU. Essentially, it would mean I won't be able to use WebGPU for our recently started 2 year research project. Is it not possible to just report the max buffer length and/or throw an exception if one tries to allocate more than that? My systems are able to handle 2GB buffers, and for the paper above I'd need to generate 8 x 2GB buffers and bind all of them to a single compute shader. With 256 I'd need 64x256MB buffers which, as far as I've seen, isn't even possible to bind to a shader, and would also make it potentially expensive to chose the right one out of 64. With 8, I can just do some ifs.
kainino0x commentedon Jan 26, 2021
Per #1069, the iPhone 6 is out of scope, as it is A8 and its last major OS release was iOS 12.
The next iPhone is probably in scope: the iPhone 6S is A9 and currently has iOS 14.
kainino0x commentedon Jan 27, 2021
If there are limits that low then I think we are likely to expose the limit. But if the limits are in the ballpark of 1GiB then I'd like to try to get away without an explicit limit.
litherum commentedon Dec 4, 2021
This comes from https://developer.apple.com/documentation/metal/mtldevice/2966563-maxbufferlength?language=objc.
litherum commentedon Dec 4, 2021
I guess one result of this should be that it's acceptable for an implementation to fail to create a single buffer of size X but able to create 2 buffers of size X / 2 (or even X - 1). This is already true in the general case because of fragmentation.
litherum commentedon Dec 4, 2021
Apple A9 (the oldest iPhone that supports the newest iOS) reports half a gigabyte for this value. Relatedly, apps can't actually allocate much more than half a gigabyte (even unrelated to the GPU) without hitting the jetsam limit.
litherum commentedon Dec 4, 2021
It's very rare for Metal's allocations-which-are-smaller-than-maxBufferLength to actually fail - they will only fail when you've exhausted all of the GPU's virtual (!!) memory. So, if you go looking for allocation failures, you'll almost never see them.
This field is intended to be a much smaller than the amount of virtual memory on the device, and is intended to be closer to be what is actually usable in any specific draw/dispatch call.
The design makes sense because it's totally reasonable for an app to allocate way more than can physically fit on the GPU at a time, throughout the entire lifetime of the app, as long as each individual allocation fits and the app doesn't use all the allocations at the same time. The VM system will page resources to/from the GPU as necessary.
litherum commentedon Dec 5, 2021
Thinking about this more, I think the relevant question is what the failure mode for buffer allocation looks like in WebGPU. The description in the spec of
createBuffer()doesn't indicate what should happen here.If a buffer allocation fails (for whatever reason, maybe the size was >
maxBufferLengthon Metal or maybe the device is out of local memory and you're running on Vulkan which doesn't do automatic eviction (and the browser hasn't implemented their own automatic eviction)), how should a WebGPU application detect this and react to it?Are browsers going to be required to post a
GPUOutOfMemoryErrorto the current scope for every allocation failure?If the answer is "yes," and applications are expected to be listening for OOM errors during resource creation to detect if their allocation fails, then that is a bit disingenuous on Metal. On Metal, the allocation can fail even if the device isn't actually out of memory. E.g. my Radeon Pro Vega 56 has 8GB of memory but
maxBufferLengthis 3.75GB. If they want to allocate 4GB on this card, that's totally fine, they just have to use 2 buffers to do it. If they try to allocate one 4GB buffer, and it fails, it isn't because the device is out of memory - the device has plenty of memory available. The reaction the app should have in this situation isn't "use less memory" but instead should be "use the same amount of memory but broken up into more allocations."On the other hand, if the answer is "no," then how can an application deterministically know whether or not their buffer failed to allocate? Surely they shouldn't have to submit a dummy command buffer that just uses a single resource just to determine if the resource was allocated successfully.
(Aside: Metal doesn't have buffer binding regions, but instead just has pointers. When encoding a pass, you don't attach a region of a buffer to a shader, but just attach a buffer and an offset. This is a surfaced in the shading language as just a pointer, so the pointer has to be able to index all the way to the end of the resource. So, limiting the "binding size" is meaningful in other APIs, but isn't meaningful in Metal. Instead, Metal has to expose this concept in the form of limiting the size of the resource itself.)
So, we probably should either have a limit for this value (ugh fingerprinting booo) or we should have a new
GPUAllocationTooBigErrorsibling toGPUOutOfMemoryError.litherum commentedon Dec 5, 2021
I guess another option would be to have a single
GPUBufferbe backed by multipleMTLBuffers. I think we shouldn't do this (now), however, because:GPUBuffercorresponding to multiple platform buffers in our discussions about uploads/downloads. (The idea back then was to have the implementation automatically copy between one backingstorageModeSharedbuffer and another backingstorageModePrivatebuffer under the hood if it thought doing so would improve performance - doing so would allow us to relax the "you can't have both STORAGE and MAP_WRITE usages on the same buffer" requirement.)would be translated to this:
but it's actually worse than this, because indexing into a resource array needs to have a dynamically uniform index, so this would actually end up being executed like this on many devices:
And this kind of blowup seems unacceptable from such an innocuous line of source code - people read/write to buffers all the time.
I don't want to close the door on this forever, though - if an application really needs a buffer that's >
maxBufferLength, they can emulate it themselves (accepting the potential performance problems), and if people actually do end up emulating it themselves, that would be a reasonable signal that we should make their lives easier and just do it for them with an extension. But it shouldn't be in the unextended core API.Kangz commentedon Dec 6, 2021
+1, it would also have massive consequences for the binding model because all of a sudden each storage buffer bindings takes more than 1 slot so we would quickly run out of slots on Metal.
I agree that
GPUOutOfMemoryErrordoesn't faithfully represent Metal's behavior (and other drivers too, I'd expect some Vulkan driver to fail to allocate more than 4GB even if there is enough memory because of pointer width), and thinkGPUAllocationTooBigErrorwould work. However how many application do we expect will be able to react in a useful way to this signal to switch to a code path that uses multiple buffers? It seems that it would be a massive change to their shaders for a situation that would happen extremely rarely?19 remaining items
kainino0x commentedon Aug 17, 2022
back to candidate until we know what the number is going to be
litherum commentedon Aug 24, 2022
The number should be 256MB.
Add maxBufferSize limit
Add maxBufferSize limit
Add maxBufferSize limit
kainino0x commentedon Aug 24, 2022
swapping this issue with the PR in the tacit resolution queue
Add maxBufferSize limit (#3366)
[wgsl] reserve binding_array. (gpuweb#1371)
kainino0x commentedon Sep 27, 2022
For implementers' information: the maxMemoryAllocationSize on Vulkan, IIUC, requires Vulkan 1.1 or VK_KHR_maintenance3. On remaining Vulkan platforms, we'll have to make up a number, like 2GiB or 4GiB perhaps. Here's the result of running gpuinfo-vulkan-query with that requirement, showing the devices where we'd have to conjure a number.
In SOME reports (199 deviceNames):wangkuiyi commentedon Mar 10, 2023
When I am running a Metal program in the iOS Simulator (iPhone 14 Pro), I got a 256MB limit.