Cannot upload/download to UMA storage buffer without an unnecessary copy and unnecessary memory use

Background

WebGPU currently has 2 buffer upload facilities: GPUBuffer.mapAsync() and GPUBuffer.writeBuffer().

For GPUBuffer.mapAsync(), there currently is the restriction that no mappable buffer can be used as anything else, other than COPY. This means that, in order to be useful, an application has to allocate 2 buffers - one for mapping and one for using. And, if an application wants to round-trip data through a shader, they have to allocate 3 buffers - one for the upload, one for the download, and one for the shader. Therefore, in order to use mapAsync(), an application needs to double (or triple) their memory use and add one or two extra copy operation. On a UMA system, neither the extra allocation nor the copy is necessary, which means there's both a perf and memory cost to using mapAsync() on those systems. What's more, because the application is explicitly writing this code, there's not really anything we can do to optimize out the extra buffer allocation / copy operation.

On the other hand, GPUQueue.writeBuffer() is associated with a particular point in the queue's timeline, and therefore can be called even when the destination buffer is in-use by the GPU. This means that the implementation of writeBuffer() is required to copy the data to an intermediate invisible buffer under-the-hood, even on UMA systems, and then schedule a copy operation on the queue to copy the data from the intermediate buffer to the final destination. This extra allocation and extra copy operation don't necessarily need to exist on UMA systems. (GPUBuffer.writeBuffer() is a good API in general because of its simple semantics and ease of use, but it does have this drawback.)

It would be valuable if we could combine the best parts of GPUBuffer.mapAsync() and GPUQueue.writeBuffer() into something which doesn't require an extra allocation or copy on UMA systems. This kind of combination would have to be something that isn't UMA-specific, but would work on both UMA and non-UMA, and UMA systems would be able to avoid extra allocations/copies under-the-hood.

Goals

The "async" part of GPUBuffer.mapAsync() would be valuable, because that allows the implementation to not have to stash any data due to the destination buffer being busy.
The "map" part of GPUBuffer.mapAsync() would be valuable because it allows the array buffer to be backed directly by GPU memory, thereby potentially avoiding another copy on UMA systems.
The "queue" part of GPUQueue.writeBuffer() would be valuable, because non-UMA systems would need to schedule an internal copy to the destination, and specifying the queue gives them a place to do that.

Proposal

I think the most natural solution to this would be:

Give mapAsync() an extra GPUQueue argument. (getMappedRange() and unmap() will implicitly use this queue). We could also say that the queue is optional, and if it's unspecified, the device's default queue will be used instead.
Relax the requirement that the only other usage a mappable buffer can have is COPY

That's it!

On a UMA system, you'd be able to map the destination (storage) buffer directly - No copies, no extra allocations, it's living the UMA dream.
- For reading, mapAsync() would just ignore its GPUQueue argument.
- For writing, mapAsync() would use its GPUQueue argument to schedule a clearBuffer() command of the relevant region of the buffer. After the clear operation is complete, the map promise would be resolved.
On a non-UMA system:
- For reading, mapAsync() would schedule a copy from the source (storage) buffer to a temporary buffer using the specified GPUQueue (which is what an application would have had to do themself), and the map operation would proceed just like normal on the temporary buffer. This is exactly what an author would have had to do themself.
- For writing, mapAsync() would just stash the queue, map a temporary buffer, and wait for unmap() to be called. When unmap() is called, it would schedule a copy on the stashed queue from the temporary buffer to the destination buffer. This is exactly what an author would have had to do themself.

It's important to note is that this proposal doesn't restrict the amount of control a WebGPU author has. If an author wants to allocate their own map/copy buffer and explicitly copy the data to/from it on its way to its final destination (as they would do today), they can still do that, and no invisible-under-the-hood temporary buffers would be allocated.

This proposal also has a natural path forward for read/write mapping.

Overall I'm worried about modifying the buffer mapping mechanisms this late, when it was perhaps the single most difficult part of the API to find a good design for and reach consensus on. I think a UMA optional feature would make a

We discussed the slight inefficiencies that UMA has with the current mapping mechanism multiple times in F2F, and #605 suggests a UMA feature could be done to optimize this later (later could be now). The proposal is interesting but has some issues.

Side note: there is also mappedAtCreation which was added so that the initial upload of data in buffers can be made perfectly efficient on UMA systems. (up to the copies necessary for the process separation).

Multiple buffers per `GPUBuffer`

It breaks down the model that one WebGPU buffer == one underlying API buffer. This is a pretty useful thing to keep because it makes it very clear to the developer what the memory cost of things is. That you can have temporary staging for mappedAtCreation, and potentially shmem wrapped in the ArrayBuffer given to JS is already very difficult for developers to reason about in terms of cost.

Cost of consistency for `MAP_WRITE`

Currently MAP_WRITE buffers give you an ArrayBuffer that contains the current content of the buffer. Since the buffer can only be written by Javascript, no copies are ever needed to update the content of the buffer, it's just the ArrayBuffer wrapping shmem shared between the GPU and Web processes. If the GPU can write to the buffer, then we need to copy data from the UMA buffer to the shmem (or even worse, from VRAM to readback to shmem).

On the other side, if we say that mapAsync(MAP_WRITE) always zeroes the buffer, then in most cases the CPU has to zero the buffer, since the UMA/readback buffer isn't shared with the Web process. Either there's a memset(0) or a memcpy from the UMA buffer to the shmem.

Consistency for `MAP_READ`

What happens when Javascript writes into the buffer mapped for reading? Assuming you are able to create an MTLBuffer from a shmem FD to reduce the number of copies to the maximum, the writes that Javascript did all of a sudden become visible to the GPU, while an all other configurations, Javascript writing to the buffer doesn't have any visible effect for the GPU?

Relatively little gains and a feature proposal

The gains you get with the proposal you suggested seem small: if you have a large amount of data to initialize buffers with, then you can use mappedAtCreation that's the optimal path. If you need to modify part of a buffer while it's in use, then you have to schedule a copy because mapping is an ownership transfer of the full buffer (I tried to figure out how to do sub-range mapping efficiently but gave up).

So the cases this help are when you need to upload data to a buffer that's not being created, but also not currently in use by the GPU. This should be a fraction of the actual buffer transfers. Still might be worth speccing an optional feature, but not modifying the core buffer mapping spec.

The optional "UMA" feature could:

Lift the restriction for MAP_WRITE to allow any other read-only usages.
MAP_READ already allows all the write-only usages. But maybe more are added in the future so the extension would also lift that? Or it allows it with any other usages, but assumes there is always a UMA -> shmem copy happening in the GPU process (so that JS writes are never made visible to the GPU).

@Kangz

I was going to have comments but @Kangz covered everything I was going to say and more.

It breaks down the model that one WebGPU buffer == one underlying API buffer.

This isn't true. This proposal requires scratch space, certainly, but so does writeBuffer(). It's no worse.

What happens when Javascript writes into the buffer mapped for reading?

This is a good point! I suppose this proposal only makes sense for read/write buffers (which we don't have today, but I think has a natural path forward).

It's relatively rare for an application to actually need read/write mapping. Sure we could add them for this use-case, but applications would still need to know which one to use and explicitly switch between them based on whether the adapter is UMA or not.

This isn't true. This proposal requires scratch space, certainly, but so does writeBuffer(). It's no worse.

WriteBuffer is quite explicitly an implementation-managed ringbuffer. But it is not tied to a GPUBuffer, it's only GPUDevice extra memory. Plus it doesn't need to be persistent data. The implementation can destroy the ringbuffers when there is memory pressure while extra backings for GPUBuffer would have to stay, otherwise you could get an OOM trying to map the buffer.

I'm not saying I'm for or against any proposal here, only voicing that I agree what's in the API today does not fully satisfy workflows with dynamic data moving across host/device and that this will be a performance issue in real-world usage. In compute workloads getting data back from the device is a major part of the upload -> compute -> download flow and until we have a GPUQueue.readBuffer (🙏 please!) this results in non-trivial complexity in user code and bloat.

But it is not tied to a GPUBuffer, it's only GPUDevice extra memory.

👍 IMO having the implementation manage the ringbuffer with writeBuffer/readBuffer and incurring a copy is acceptable if the alternative is managing exclusively upload or download buffers in user code (as I found the spec detail that indicates a buffer cannot be both, resulting in user staging pools needing double the memory for bidi transfer). This way multiple libraries trying to perform upload/download are not each keeping around large GPUBuffers for this purpose.

you could get an OOM trying to map the buffer.

This is no worse than the possibility of an OOM when trying to writeBuffer(), though...

Sure that's a possibility as well, although implementation could stall to free staging space if they really want to.

The point here wasn't that writeBuffer can or cannot OOM, it is that you want to give developers a way to do transfers without the possibility to trigger unfixable OOM. Buffer mapping can do that since OOM only happens at buffer creation. If you choose to make staging / readback buffer transient, then you lose this control in the application because you can OOM on mapAsync as well. It is possible to decide to do that, but need to be cognizant of all the tradeoffs we are making. In this whole comment thread I suggest it is a bad idea for many reasons, including that OOM issue.

@Kangz

I agree with the concerns about managing temporary buffers for mapping, expressed by @Kangz . Their lifetime is attached to mapping, and it's worse than the ring buffer we currently have for the writes.

I also agree with @litherum that it would be good to be able to avoid copies on systems that can do that.
An optional feature for UMA architectures seems like the right way to proceed. It would basically lift the restriction on usages for buffers, allowing MAP_READ+MAP_WRITE+anything else.

As for the queue argument for mapping, this correlates with #1977 (comment). It's probably needed.

I'm happy to help by writing an optional feature for UMA that allows MAP_READ + WRITE.

It would be pretty unfortunate if authors had to opt-in to avoid using 2x memory on UMA machines.

I don't think anyone is disagreeing about that, but if we're going to avoid it we're going to need a proposal that works. I don't think we're getting any closer to one.

Yes, the minuses are that you need to understand performance implications for write-combined memory. But my argument is that all low level gfx programmers already understand it, since it's the same on all other APIs, including older ones that didn't support persistently mapped buffers. If WebGPU requires CPU cached + coherent, then that feature is pretty much unusable on Android, since there's not enough coverage.

At least provide a flag for professional graphics programmers to enable support for non-cached upload heap. I have no problems with that. And most of the professional users will write WASM with C/C++ or Rust anyways, so JS ArrayBuffer is not even visible for us.

@Chainsawkitten

This thread form is not super great to discuss detailed feedback on a proposal like this that has a lot of details.

For `CACHED` and `COHERENT`

@Chainsawkitten explained in the meeting that requiring CACHED | COHERENT is not reasonable. The big concern from the Chromium side of things is if we are able to triple-map (or even quad-map: JS, WASM, GPU process and the GPU) uncacheable memory, then for the first time, JS and WASM devs will have to deal with uncached memory. Mapped GPUBuffers will result in ArrayBuffers that may have orders of magnitudes of perf difference between cached and uncachend memory. IMHO it is a portability concern and we should try to avoid it. I understand that in native it isn't a concern though and that devs rely on write combining to do the correct thing.

My suggestion would be that for triply-mapped buffers we only require CACHED and add the necessary invalidation/flush in the implementation. For buffers that require an intermediate copy, the implementation would be free to use COHERENT only.

@sebbbi, I didn't find good resources about how to make sure write-combining avoids any uncached reads (though I remember one from years ago). Do you have one? Is it necessary to fully write each cache line touched by a write (to avoid having to load the cache line to combine partial writes)?

Care about GPU-based validation

We need to make sure that this extension doesn't add any TOCTOU attacks where JS or the Web process is able to modify data in buffers after it has been validated, and also that it doesn't prevent future ways we might want to validate data. At the moment only validation of indirect draw/dispatch requires GPU-side validation that could be subject to TOCTOU, so my suggestion would be to not allow INDIRECT with any mappable usage. WebKit also had an idea to validate index buffers on the GPU but I'm not sure about that anymore. @mwyrzykowski are TOCTOU on INDEX buffers a concern?

Multi-process behavior and extra copies

The behaviors listed for the various GPUMapMode may not minimize the number of copies depending on whether triply-mapping is happening vs. not. Note that at this point we expect all browsers to ship WebGPU as a multi-process implementation. @mwyrzykowski this is related to your comment on the PR.

With triply mapping:

MAP_READ needs to make a copy of the contents of the buffer, so as to discard writes.
MAP_WRITE (GPU read only) no extra copies
MAP_WRITE (GPU read write) an extra memset(0) of the whole mapped range.
MAP_READ | MAP_WRITE no extra copies

Without triple mapping (one copy always happen between the GPU and JS):

MAP_READ no extra copies
MAP_WRITE (GPU read only) one extra copy if the application doesn't care about the previous contents and overwrites all of it.
MAP_WRITE (GPU read write) no extra copies if the implementation keeps the previous CPU-side contents.
MAP_READ | MAP_WRITE no extra copies

Also it is a bit weird how the MAP_WRITE has different behavior depending on the buffer usage flag. IMHO that's action at a distance and a footgun. Likewise for the behavior of MAP_WRITE read-write being one of two possibilities, that's a portability concern.

Some ideas of how we could maybe improve this:

Have an explicit ZERO / DISCARD flag that can be used by the application to have more defined behavior.
Consider zeroing memory with GPU commands on mapAsync instead of a memset.
Having a hint for the application on whether a buffer is triply mapped. (can we even know this synchronously in Chromium? not sure)

IMHO the ideal solution, if it exists, would have strong portability guarantees across browsers and types of buffers (triply mapped vs not), have clear optimal paths that applications can use and no additional copy overhead than necessary for triply-mapped buffers (while still being very good for non-triply mapped).

How to use this feature

I'll let @Jiawei-Shao detail. Do you have sample code showing how an application would take advantage of this feature, and write code that can be most efficient with and without it? For @JMS55's question, I believe that this feature would be similar (albeit slightly more limited) than wgpu's MAPPABLE_PRIMARY_BUFFERS.

. @mwyrzykowski are TOCTOU on INDEX buffers a concern?

Yes, it's still a concern. We perform index buffer validation as needed via VS:VS memory barriers on apple graphics chips, falling back to pass splitting on Intel and AMD devices. As long as the copy of memory from JS / WASM is distinct from the UA's memory, then I think we can identify all the places where we must re-validate.

If the GPU memory was shared with JS, which WebKit also doesn't current support, then it would be much harder to perform such validation via that mechanism.

Now there are two issues about write-only buffer mapping:

The performance of CPU read on the array buffer returned from getMappedRange()
The CPU read from non-CPU-cached memory is very slow with triply mapping on non-CPU-cached architectures.
The initial data in the array buffer returned from getMappedRange()
With triply mapping the array buffer exactly contains the data on the GPU, while with non-triply mapping the initial data may not always be the data on the GPU.

So I think the main problem here is that although we are doing write-only buffer mapping we still have to upload data from an array buffer returned from buffer.getMappedRange() which can be read but actually we don't care about the initial data of it.

Maybe we can add another API (e.g. buffer.mapWriteData(arrayBuffer)) to allow directly uploading data into the write-only mapped buffer without returning the current data on the CPU? The arrayBuffer here is managed by the developer and can be reused for other buffers or used for other purposes. buffer.getMappedRange() can still be preserved but we can add a new restriction that the data should be cleared to 0 when the buffer has GPU-write usage.

. buffer.mapWriteData(arrayBuffer))

Would that be different from device.queue.writeBuffer(buffer, 0, arrayBuffer)?

buffer.mapWriteData(arrayBuffer) can do data uploading at once, while device.queue.writeBuffer(buffer, 0, arrayBuffer) can't when buffer is in use, in this time a staging buffer, an extra copy and a barrier are still needed.

A buffer.mapWriteData(arrayBuffer) would be interesting but ultimately that's one more copy as well because the application needs to stage the data somewhere and then ask the browser to do the copy.

A buffer.mapWriteData(arrayBuffer) would be interesting but ultimately that's one more copy as well because the application needs to stage the data somewhere and then ask the browser to do the copy.

I think one good thing for buffer.mapWriteData(arrayBuffer) is that the arrayBuffer can be reused just after the call of this API, while with current WebGPU API the mappable staging buffer cannot be reused until the copy is completed on GPU and the mapAsync() is resolved.

I think one good thing for buffer.mapWriteData(arrayBuffer) is that the arrayBuffer can be reused just after the call of this API, while with current WebGPU API the mappable staging buffer cannot be reused until the copy is completed on GPU and the mapAsync() is resolved.

For what it's worth, wgpu has a similar (native-only) function called write_buffer_with that is meant to temporarily lend a view into a write-only buffer, which is similar to part of this discussion. I was wondering if we could do something similar in WebGPU to avoid a copy for common cases where people wouldn't necessarily need to use mapAsync/copyBufferToBuffer otherwise.

// Get an ArrayBuffer backed by memory that will be used for the encoded writeBuffer command sent to the GPU process
const tmp = queue.writeBufferWith(/* ... */);
// Create views/write to them directly without the application needing to copy from somewhere else
(new Float32Array(tmp.buffer))[0] = 123;
// Somehow detach the ArrayBuffer explicitly so the implementation knows the application is done writing to it
tmp.finish();

A buffer.mapWriteData(arrayBuffer) would be interesting but ultimately that's one more copy as well because the application needs to stage the data somewhere and then ask the browser to do the copy.

So buffer.mapWriteData() will introduce one more copy in JavaScript, while always setting the return arraybuffer of buffer.getMappedRange() to 0 will do one more clear. Not sure which one has better performance.

FYI webgpu.h has already added wgpuBufferReadMappedRange and wgpuBufferWriteMappedRange, which are are just like getMappedRange+memcpy. We added these because Wasm-JS bindings have similar issues with needing to copy data and clear memory. They allow copying directly from the Wasm heap into JS. The bindings implementation currently uses getMappedRange and then memcpy's between the Wasm heap and the resulting ArrayBuffer. If we had this in JS then we would use it directly, instead.

GPU Web WG 2025-04-09 Atlantic-time

CW: close to a solution. This issue is difficult. Problem - no way to get the optimal perf/# copies for both impls that can triply map buffers, and impls that can't, for when you do map_write. We don't want to copy data from GPU when we can't triply map, so we zero out - or in triply mapped, we give you the GPU's data because doing so is free and everything else is work. My hunch - we are going to be able to triply map buffers in all impls eventually, so we should optimize for that, but don't want to predict that. Think only 70% of the cases in Chromium can do this.
GT: can you make those two separate cases? You ask for triply mapped, but if we can't give that to you we don't give you that option?
CW: might not be possible to know in the JavaScript process whether triple mapping will work.
GT: more that, it's a device/adapter feature. Triply mapped is something you can ask for at adapter/device.
CW: not sure this works consistently even on a single device. Could gather that data point. If works, makes things much easier.
CW: shouldn't expose rebar to JS because of caching issues.
JB: so you expect 2 classes of GPUs. When they can share memory efficiently, can also get that memory into the content process efficiently.
CW: yes.
CW: if we're willing to make that guess, then the UMA buffer mapping spec gets easier. When you map-write, you always get the previous contents of the buffer because it's triply mapped.
KN: is there a perf penalty for getting that data? Cache flushes, etc.?
CW: don't think so. On Vulkan, the one that has the most exposed knobs and which runs on GPUs with the most constraints - might need to call invalidate(). Invalidates parts of the CPU cache. It's fine. (Might be wrong.) Just need to do this when GPU execution's finished - not that often.
KN: would probably just impact the time it takes to map.
CW: I'm kind of guessing at this point.
CW: everyone OK if we try going in that direction? Try asking for impl experience before landing in the spec?
KN: this extension would be gated on triple mapping?
CW: yes.
CW: ok if we move forward? Say this'll be the way this works, but we'll get more impl confidence before landing it?
JB: so people that have a separate shmem, they'll have an overhead of copying, until they implement triple mapping? Or, don't expose the extension unless triple mapping is available.
CW: yes. I would advocate we not expose the extension unless you have triple mapping.
KG: earlier we were thinking UMA shouldn't be an extension (?)
CW: first meeting we discussed this, thought we were comfortable with this being an extension.

Navigation Menu

Repository navigation

Cannot upload/download to UMA storage buffer without an unnecessary copy and unnecessary memory use #2388

Background

Goals

Proposal

Multiple buffers per `GPUBuffer`

Cost of consistency for `MAP_WRITE`

Consistency for `MAP_READ`

Relatively little gains and a feature proposal

29 remaining items

For `CACHED` and `COHERENT`

Care about GPU-based validation

Multi-process behavior and extra copies

How to use this feature

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cannot upload/download to UMA storage buffer without an unnecessary copy and unnecessary memory use #2388

Description

Background

Goals

Proposal

Activity

Kangz commented on Dec 7, 2021

Multiple buffers per GPUBuffer

Cost of consistency for MAP_WRITE

Consistency for MAP_READ

Relatively little gains and a feature proposal

kainino0x commented on Dec 8, 2021

litherum commented on Dec 8, 2021

litherum commented on Dec 8, 2021

kainino0x commented on Dec 9, 2021

Kangz commented on Dec 9, 2021

benvanik commented on Dec 10, 2021

litherum commented on Dec 11, 2021

Kangz commented on Dec 13, 2021

kvark commented on Dec 14, 2021

Kangz commented on Dec 14, 2021

litherum commented on Dec 15, 2021

kainino0x commented on Dec 15, 2021

29 remaining items

sebbbi commented on Mar 22, 2025

sebbbi commented on Mar 22, 2025

Kangz commented on Mar 24, 2025

For CACHED and COHERENT

Care about GPU-based validation

Multi-process behavior and extra copies

How to use this feature

mwyrzykowski commented on Mar 25, 2025

Jiawei-Shao commented on Mar 31, 2025

mwyrzykowski commented on Mar 31, 2025

Jiawei-Shao commented on Apr 1, 2025

Kangz commented on Apr 1, 2025

Jiawei-Shao commented on Apr 2, 2025

grovesNL commented on Apr 2, 2025

Jiawei-Shao commented on Apr 3, 2025

kainino0x commented on Apr 4, 2025

Kangz commented on Apr 22, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Multiple buffers per `GPUBuffer`

Cost of consistency for `MAP_WRITE`

Consistency for `MAP_READ`

For `CACHED` and `COHERENT`