Proposal for a simple streaming API

Intro

The reasoning implies vague familiarity with #138 and surrounding discussion.

Here are the base points for this proposal:

Persistent mapping is the most efficient path on native APIs for streaming resources in from CPU. The user sub-allocates within a persistently mapped buffer (or multiple ones) and some parts of it are being updated from CPU while the others are used by GPU for copying data into device-local memory.
WebGPU (currently) tracks usage of a buffer object as a whole, we don't allow parts of the same buffer to be in a different usage state. It also aims to have WebGPU buffers (and textures) to correspond 1:1 to the native objects in the respective APIs.
We can't allow CPU-GPU data races for the upload/download scenarios.

I think those points start conflicting with each other as soon as we are talking about CPU-visible buffers. If we treat the buffer as a whole for GPU side (i.e. when considering it to be used as a vertex buffer, or in a resource group), we can't have parts of it being accessed by CPU. At some level we'd need to track buffer sub-ranges and manage persistent mapping. Here I propose to resolve the conflict by stating that CPU-visible resources are only going to be used for copy operations, and thus we don't need full-featured mapping semantics for a general WebGPUBuffer.

This proposal is introducing new commands for data uploads/downloads. They can be implemented by the browser with the most efficient method of uploads/downloads (via persistent mapping). The regular buffers then keep the 1:1 relation to native objects and the constraints of a single (mutable) usage.

Code

dictionary WebGPUTextureDataLayout {
    u32 rowPitch;
    u32 slicePitch;
};

partial interface WebGPUCommandBuffer {
  Promise<ArrayBuffer> uploadBuffer(
    WebGPUBuffer dst,
    u32 dstOffset,
    u32 size);

  Promise<ArrayBuffer> downloadBuffer(
    WebGPUBuffer src,
    u32 srcOffset,
    u32 size);

  Promise<ArrayBuffer> uploadTexture(
    WebGPUTextureDataLayout layout,
    WebGPUTextureCopyView destination,
    WebGPUExtent3D size);

  Promise<ArrayBuffer> downloadTexture(
    WebGPUTextureCopyView source,
    WebGPUTextureDataLayout layout,
    WebGPUExtent3D size);
}

Semantics

When recording a command buffer, the user may request to upload or download data from some of the resources. Those operations return Promise<ArrayBuffer>.

For upload operations, the promise is resolved once the WebGPU implementation has prepared the staging area for an upload. The initial contents of the ArrayBuffer are guaranteed to be zeroed out. The implementation would also hold the submission of the command buffer until all the upload callbacks (from promises) are done. This guarantees that the data for an upload is ready, and the implementation can proceed with the transfer into device-local memory, where all of the resources (buffers/textures) are allocated.

For download operations, the promise is resolved once the GPU is done executing this command buffer, and the ArrayBuffer would contain the data read from device-local memory of corresponding resources.

Analysis

One clear downside is inability to have resources accessible from both CPU and GPU directly. On some architectures, the shared memory can be effectively used by GPUs and could technically allow the graphics programmer to avoid a copy (from CPU-visible to device-local), but this implementation doesn't allow that.

A follow-up upside, however, is that developers don't need to consider those architectures and implement diverging code paths for them. This results in simpler code and more portable behavior of the programs for our API.

Simplicity, in general, is the main feature of this proposal. The complexity of recycling buffer ranges and synchronizing the mapping access is put on the browser implementation. I believe this goes in line with the current direction WebGPU is heading (implicit synchronization, no memory objects).

There is another downside about the fact that a single ArrayBuffer corresponds to a single copy operation. I don't think this limitation is essential to the core of the proposal, and we can adjust to this need, if it's too constraining.

Thanks for the proposal, it indeed looks very simple. Could we have clarifications on the following points? (feel free to edit to integrate them)

What does it mean for implementations to hold the submission of the WebGPUCommandBuffer that have upload/download operations? Is it an error to submit the command buffer before the promises are fulfilled or does the "GPU wait for the CPU"?
Are the ArrayBuffers detached at any point?
Can you give an example that highlights how many copies of the data are needed before it is usable on the GPU?

What does it mean for implementations to hold the submission of the WebGPUCommandBuffer that have upload/download operations? Is it an error to submit the command buffer before the promises are fulfilled or does the "GPU wait for the CPU"?

The original idea was that the backend on GPU process side would hold onto submission (and any further submissions by the user) until the promise callback is done. I'm open to the idea of erroring out instead, if it's more convenient.

Are the ArrayBuffers detached at any point?

The upload ArrayBuffer is detached after the callback is done. The download ArrayBuffer doesn't need to be detached.

Can you give an example that highlights how many copies of the data are needed before it is usable on the GPU?

Sure! Supposing we are in multi-process environment, i.e. the GPU process is separate from the content. Typically, the ArrayBuffers provided to the user would point to the shared memory between those processes. So the first copy/step would be user filling this shared space from whatever source they have (i.e. streamed in file).

If the backend API supports sharing resources over IPC, e.g. with VK_KHR_external_memory, then this memory can already be readable by the driver, and there is only one copy from this CPU-visible shared memory to the actual device-local memory of the target resource (that the user specifies in the respective copyXxx operation).

Without sharing on the API backend side, the ArrayBuffer is just a regular shared memory. When the callback is done, WebGPU implementation can copy from this memory into CPU-visible buffer of the API.

So, in total, we'd have:

initial fill
a CPU-side memcopy, only if the API doesn't have sharing and the browser is multi-process
an API transfer between CPU-visible buffer and the target resource

@kvark

Thanks @kvark, I’m really glad we got this approach written down.

I’m confused about uploadBuffer. The proposal says:

The implementation would also hold the submission of the command buffer until all the upload callbacks (from promises) are done.

How does the web developer indicate that they are finished filling in the contents of the ArrayBuffer? The end of the microtask? Will the buffer be neutered if it’s retained and lives longer than the microtask?

For upload operations, the promise is resolved once the WebGPU implementation has prepared the staging area for an upload.

Do you anticipate the “preparation” is literally zero-ing out a buffer? Or do you expect the preparation might call map() under the hood sometimes?

IIRC the biggest complaint in the past to this style of API is the performance loss by applications that want to unpack data (random writes) directly into the GPU’s memory. Would it be possible to make an implementation where those ArrayBuffers are backed by the literal GPU memory? If the implementation knows that the destination piece of the buffer is definitely not in use by the GPU right now, can it use persistently-mapped buffers under the hood to implement this API super fast?

Or, could an implementation implement this by keeping a bunch of GPU scratch buffers available, uploading into them, and then dispatching a compute pass to copy the data from the scratch buffers into the real buffer, thereby never having the buffer ownership ping-ponging between the CPU and the GPU?

What are the anticipated performance trade offs between this and
#156?

I really like the simplicity of this design, and I’m very interested to see if it will work.

Thanks for the clarifications! I do have some concerns with the mechanism you suggested.

The first concern is that the promise mechanism makes upload difficult to use. The ArrayBuffer being detached at the end of the promise callback breaks a lot of promise combinators such as Promise.all and doing await staging = cmdBuf.uploadBuffer() in the middle of an async function. Detaching quickly like this forces the data to be ready exactly when the promise fires, which forces applications to first get the data in a staging buffer, then await uploadBuffer then copy the data in the staging region, which adds another copy. A fix could be to detach the ArrayBuffers when the command buffer is submitted.

Another concern is that there is no guarantee that the WebGPUCommandBuffer will be submitted soon (like this frame). This means that allocations of staging memory and shared memory cannot be ringbuffers but have to be general allocators. The WebGPUCommandBuffer will also become a container for GPU memory so it will need an explicit destroy method.

Conceptually uploadBuffer is complicated because it mixes three timelines in a single call: the GPU execution timeline, the JS execution timeline and some new uploadBuffer promise resolution timeline (which is not specified in the current proposal).

These were the concerns about the proposal seen on its own. I'm biased but compared to #147 / #138 I don't think the simplicity gains outweighs the losses.

It doesn't allow minimizing the number of copies, in particular on UMA systems like phones where memory accesses are expensive power-wise.
It is more alien to native application looking to port to WebGPU.
More magic happens on the browser side which makes it more difficult for applications to reason about their performance.
It is also trivial to polyfill with Add explainer document for BufferOperations #147 with the same number of copies.
It requires a lot more client-side state tracking to be efficient (for example the allocator in staging memory would have to live there vs. a state machine per buffer).
It can't skip the zeoing of memory while Add explainer document for BufferOperations #147 might end being modified to avoid the zeroing.

This proposal also doesn't have an immediate data upload but I assume we could have setSubData easily.

@litherum

@litherum and @Kangz thanks for great feedback! I'm ready to evolve the proposal based on it, given the simplicity in the core to persist.

How does the web developer indicate that they are finished filling in the contents of the ArrayBuffer? The end of the microtask? Will the buffer be neutered if it’s retained and lives longer than the microtask?
The first concern is that the promise mechanism makes upload difficult to use.

The concern sound legit, and it just shows how little I'm familiar with JS workflows. I suppose it is resolved by drawing a line at submit():

making sure that the promises are resolved
detaching the upload buffers

Do you anticipate the “preparation” is literally zero-ing out a buffer? Or do you expect the preparation might call map() under the hood sometimes?

The implementation will need to manage multiple native memory objects, so "preparation" could be either just zero-out or allocate + map + zero-out. The likelihood of the latter case can be reduced by adding a hint method like WebGPUDevice::reserveUploadMemory(u32 size) that the user can call before doing the uploads.

IIRC the biggest complaint in the past to this style of API is the performance loss by applications that want to unpack data (random writes) directly into the GPU’s memory.

In this proposal, the application would upload the required-to-write data into a buffer and then copy/scatter it into the GPU memory. You can't do it without this extra copy on non-UMA devices anyway.

What are the anticipated performance trade offs between this and #156?
It doesn't allow minimizing the number of copies, in particular on UMA systems like phones where memory accesses are expensive power-wise.

About this and UMA, I'm genuinely curious how #147 / #156 would work for this use-case. So supposing we do succeed providing ArrayBuffer backed by GPU memory, and we make this use-case a zero-copy on UMA. My question is - how would this work on platforms where we can't have that direct GPU backing of ArrayBuffer? Given the "random writes" case, it seems like the implementation would have to copy much larger chunks of memory between JS - browser - driver - GPU, so I'm concerned about the efficiency of this approach. In comparison, with this proposal (#154) the same code would have exactly the same performance characteristics on UMA and not, with the amount of data transferred being explicit and clear.

Or, could an implementation implement this by keeping a bunch of GPU scratch buffers available, uploading into them, and then dispatching a compute pass to copy the data from the scratch buffers into the real buffer, thereby never having the buffer ownership ping-ponging between the CPU and the GPU?

To clarify, in this proposal the user is expected to do this scatter themselves. We'd only provide means of uploading the data to GPU buffers/textures in linear chunks (which can potentially be small, since the only overhead is Promise creation).

@Kangz

Another concern is that there is no guarantee that the WebGPUCommandBuffer will be submitted soon (like this frame). This means that allocations of staging memory and shared memory cannot be ringbuffers but have to be general allocators. The WebGPUCommandBuffer will also become a container for GPU memory so it will need an explicit destroy method.

I agree.

Conceptually uploadBuffer is complicated because it mixes three timelines in a single call: the GPU execution timeline, the JS execution timeline and some new uploadBuffer promise resolution timeline (which is not specified in the current proposal).

I don't think uploadBuffer is concerned with the GPU execution timeline. At least, not more than any other command buffer call, and that makes uploadBuffer only complicated because of Promise, which happens to be present in other proposals as well ;)

It is more alien to native application looking to port to WebGPU.

I don't think this is a big concern, given the simplicity of the API and the similarity to OpenGL's glTexSubImage/glBufferSubData. As a developer, I'd be more confused by the distinction between buffers and views in #156 and where one can be used but not the other.

More magic happens on the browser side which makes it more difficult for applications to reason about their performance.

I do agree this to be a concern. Although as I described above, it's not clear to me that the proposals without an extra copy make it easy to reason about performance either. And the surprises can be remedied to an extent by the aforementioned hint call to reserve upload/download space.

It is also trivial to polyfill with #147 with the same number of copies.

I don't see it being as trivial. #147 wouldn't let you upload to one part of the native buffer while the other part gets copied from/to on GPU or DMA.

It requires a lot more client-side state tracking to be efficient (for example the allocator in staging memory would have to live there vs. a state machine per buffer).

This is an interesting idea. I didn't consider having this logic on a client side. What's the problem with it being on the server side?

It can't skip the zeoing of memory while #147 might end being modified to avoid the zeroing.

True, although we still need to see what that modification looks like ;)

I don't think uploadBuffer is concerned with the GPU execution timeline. At least, not more than any other command buffer call, and that makes uploadBuffer only complicated because of Promise, which happens to be present in other proposals as well ;)

However in other proposals the promise for buffer mapping is orthogonal to GPU command buffering so you don't need to think about both at the same time.

It is also trivial to polyfill with #147 with the same number of copies.

I don't see it being as trivial. #147 wouldn't let you upload to one part of the native buffer while the other part gets copied from/to on GPU or DMA.

Neither does this proposal, this is why uploadBuffer is a buffered operation and results in a buffer-to-buffer copy from some staging memory to the actual buffer. Steps with this proposal are (forgetting about zeroing):

Write into shared memory
Copy from the shared memory to some staging memory
Record a buffer to buffer copy

The exact same sequence of operations happens with #147 and code like the following:

// Get some staging memory for example with this Gist
// https://gist.github.com/Kangz/cae07358a8fdd5ce38772a26524cd47a
let (mapping, offset) = requestUploadSpace(size);

// This writes into shared memory
putDataIn(mapping.pointer);

// This records a buffer to buffer copy
myPendingCmdBuf.copyBufferToBuffer(mapping.buffer, offset, myBuffer, myOffset, size);

// At the end of the frame the mapping.buffer is unmapped, before commands are
// submitted, which results in the shmem to staging memory copy.

About this and UMA, I'm genuinely curious how #147 / #156 would work for this use-case.

If an application truly needs to do random writes then they use a buffer with the MAP_WRITE | TRANSFER_SRC usage then schedule copies from this buffer. This is equivalent to this proposal as shown in the snippet of code above. What #147 and #156 allow that #154 are mapping for things other than TRANSFER_SRC such as loading a static model with zero-copy, staging uniforms used for a single frame, etc.

It is true that some developers might map buffers too eagerly but buffer mapping should be seen as an operation linear in the size of the mapping. It would be unfortunate to penalize system where zero copy (single write) is possible by forcing at least one extra copy (two writes, 100% worse!)

It requires a lot more client-side state tracking to be efficient (for example the allocator in staging memory would have to live there vs. a state machine per buffer).

This is an interesting idea. I didn't consider having this logic on a client side. What's the problem with it being on the server side?

I guess this would be doable, with the server side answering uploadBuffer requests with a (buffer, offset) pair.

It can't skip the zeoing of memory while #147 might end being modified to avoid the zeroing.

True, although we still need to see what that modification looks like ;)

Only allow MAP_WRITE in combination with read-only usages. The buffers is zeroed when it is created and then MAP_WRITE returns and ArrayBuffer with the data in the buffer, that has only been written by the CPU so it is deterministic.

@Kangz

@Kangz

However in other proposals the promise for buffer mapping is orthogonal to GPU command buffering so you don't need to think about both at the same time.

Sorry, I still don't understand this concern in full. Your original claim was about "GPU execution timeline" being involved, which I refuted. Now you are talking about GPU command buffering. The only difference (in the context of which timelines are involved) with the other proposals is that we get the promise when recording commands. From there on, the timelines appear very similar: user just needs to make sure that the data is fully written by submit() time.

It is also trivial to polyfill with #147 with the same number of copies.

I don't see it being as trivial. #147 wouldn't let you upload to one part of the native buffer while the other part gets copied from/to on GPU or DMA.

Neither does this proposal, this is why uploadBuffer is a buffered operation and results in a buffer-to-buffer copy from some staging memory to the actual buffer.

In this proposal, users don't see the actual native staging buffer. The browser implementation manages those and of course it uses "one part of the native buffer while the other part gets copied from/to on GPU or DMA". Perhaps, you read the "native buffer" part as being WebGPUBuffer? This is not what I meant here, I was talking about the staging buffer.

The exact same sequence of operations happens with #147 and code like the following:

First, if it is meant to include a multi-page long gist file, which is hardly a straight polyfill :)
Also, with this code you still can't partially use the staging buffers, so I feel like a lot of ~~devils in the~~ details are missing, e.g. here

// This writes into shared memory
putDataIn(mapping.pointer);

Where is the mapping happening? How do you ensure that the mapped buffer isn't used for some in-flight copy operation? Is there any blocking expected? And so forth...

loading a static model with zero-copy

This does appear to be an important case that this proposal doesn't handle well.

It would be unfortunate to penalize system where zero copy (single write) is possible by forcing at least one extra copy (two writes, 100% worse!)

Note that it's only possible with other proposals in the following range of scenarios:

if (GPU.isSameProcess() || API.supportsObjectsSharing()) &&
  GPU.isUnifiedMemory() &&
  targetResource.isBuffer() &&
  targetResource.supportedMemoryTypes().contains(|memoryType| memoryType.isHostVisible())
{...}

I wonder if we can adjust this proposal to also handle this case... I mean, unless the user does something really awkward, this could be handled by implementation by just giving a promise into GPU memory and not doing any actual copying, but the trick would be to handle all the possible weird things the user does :)

Only allow MAP_WRITE in combination with read-only usages.

Yeah, that sounds reasonable (and simple) 👍

@devshgraphicsprogramming

However in other proposals the promise for buffer mapping is orthogonal to GPU command buffering so you don't need to think about both at the same time.

Sorry, I still don't understand this concern in full. Your original claim was about "GPU execution timeline" being involved, which I refuted. Now you are talking about GPU command buffering. The only difference (in the context of which timelines are involved) with the other proposals is that we get the promise when recording commands. From there on, the timelines appear very similar: user just needs to make sure that the data is fully written by submit() time.

The "GPU execution timeline" is the timeline in which buffered GPU commands are executed. My concern is exactly that getting a promise when recording commands is complicated conceptually. Right now you are encoding a command that will be executed later by the GPU and returns a promise that will resolve later as well but before the later from the GPU execution because it is an argument to that GPU execution.

This is cognitive load concern and very subjective so it is less important than the other concerns.

It is also trivial to polyfill with #147 with the same number of copies.

I don't see it being as trivial. #147 wouldn't let you upload to one part of the native buffer while the other part gets copied from/to on GPU or DMA.

Neither does this proposal, this is why uploadBuffer is a buffered operation and results in a buffer-to-buffer copy from some staging memory to the actual buffer.

In this proposal, users don't see the actual native staging buffer. The browser implementation manages those and of course it uses "one part of the native buffer while the other part gets copied from/to on GPU or DMA". Perhaps, you read the "native buffer" part as being WebGPUBuffer? This is not what I meant here, I was talking about the staging buffer.

It's understood that the "native buffer" is an implementation detail and not exposed to the application.

The exact same sequence of operations happens with #147 and code like the following:

First, if it is meant to include a multi-page long gist file, which is hardly a straight polyfill :)
Also, with this code you still can't partially use the staging buffers, so I feel like a lot of devils in the details are missing, e.g. here

// This writes into shared memory
putDataIn(mapping.pointer);

Where is the mapping happening? How do you ensure that the mapped buffer isn't used for some in-flight copy operation? Is there any blocking expected? And so forth...

I guess polyfill was a bit strong, and an actual polyfill would require augmenting the webgpu object itself. The point is that this proposal could be implemented with the same performance using #147, and that #147 provides additional benefits (option to have it not zero except on resource creation, zero copy on UMA etc.)

loading a static model with zero-copy

This does appear to be an important case that this proposal doesn't handle well.

It would be unfortunate to penalize system where zero copy (single write) is possible by forcing at least one extra copy (two writes, 100% worse!)

Note that it's only possible with other proposals in the following range of scenarios:
if (GPU.isSameProcess() || API.supportsObjectsSharing()) &&
  GPU.isUnifiedMemory() &&
  targetResource.isBuffer() &&
  targetResource.supportedMemoryTypes().contains(|memoryType| memoryType.isHostVisible())
{...}
I wonder if we can adjust this proposal to also handle this case... I mean, unless the user does something really awkward, this could be handled by implementation by just giving a promise into GPU memory and not doing any actual copying, but the trick would be to handle all the possible weird things the user does :)

The condition above will be 100% of Safari (AFAIK the GPU driver is in-process), most Android devices (UMA + the right extensions) and Intel GPUs when using Vulkan with the right extensions (should be widely available).

Even on discrete GPUs it can be useful to put data in CPU and GPU visible memory: @devshgraphicsprogramming mentioned doing this for dynamic vertex and index data.

(on a side note I'll have to get used to that Rust lambda syntax).

@Kangz

@Kangz

The "GPU execution timeline" is the timeline in which buffered GPU commands are executed. My concern is exactly that getting a promise when recording commands is complicated conceptually. Right now you are encoding a command that will be executed later by the GPU and returns a promise that will resolve later as well but before the later from the GPU execution because it is an argument to that GPU execution.

I respectfully disagree. When uploading data the user doesn't need to be concerned with the GPU execution timeline. Sequence is:

user starts recording
schedules and upload, gets a promise
continues recording, fulfills the promise
submit() is called - this is where the user stops being concerned
GPU actually executes the commands - this is outside of the concern zone

The point is that this proposal could be implemented with the same performance using #147, and that #147 provides additional benefits (option to have it not zero except on resource creation, zero copy on UMA etc.)

Right. With this being the simplest proposal it doesn't aim to cover more ground than the other ones, it aims to reduce complexity of the API. I take the point about sparing the clears.

The condition above will be 100% of Safari (AFAIK the GPU driver is in-process), most Android devices (UMA + the right extensions) and Intel GPUs when using Vulkan with the right extensions (should be widely available).

That's a pretty good list, although the extension, if you are talking about VK_KHR_external_memory_fd, is currently only supported by a third of devices. Also, we can add AMD Vega to the list for it having 256Mb of device local and CPU visible memory.

I'd like to discuss in more detail what use cases are being lost with this PR. The cases are placed in an intersection of these filters:

on some of the platforms... (portability caveat)
only buffer uploads/downloads (not textures)
only if GPU isn't using them (any part of them!) at the moment of transfer
we can exclude the static buffers for now, since it's not ruled out that this proposal can't have a fast path for those

So within what's left of that filtering, it's pretty difficult for me to imagine that use case you are trying to defend/expose. If there is a uniform buffer, you'd likely want to dynamically modify parts of it while other parts are used by GPU, so this is not a target case. If there is a dynamic vertex buffer, and you want to change some parts of it, the transfer/memory cost of this approach is proportional to the total number of changed bytes. How often do you see people dynamically changing large parts of vertex buffers?.. Sufficiently large for us to be concerned about that transfer/staging memory cost.

we can exclude the static buffers for now, since it's not ruled out that this proposal can't have a fast path for those

We would need a description of what setting initial data looks like:

Is it a synchronous or asynchronous process ?
Does it happen somehow as part of createBuffer or is it a method on WebGPUBuffer?
If a method, what are the necessary validation rules, is it a one-time thing before any use of the resource or can it be done at any time?
If asynchronous, what are the guarantees on when the promise will be fulfilled? What happens if the buffer is used before the promise is fulfilled?
Is the ArrayBuffer used for setting the initial data detached at some point, when?
What are the number of copies in UMA vs. non-UMA?

I think such a mechanism will end up extremely close to mapWriteAsync in #147.

I'd like to discuss in more detail what use cases are being lost with this PR.

Efficiency: Allowing zero-copy uploads of static data (Google Maps model and texture streaming falls into this as well). This is important to save power on mobile devices. Likewise, allowing zero-copy uploads of dynamic data (vertex + index, + uniform with Proposal: BufferSubRanges to solve buffer-tracking, streaming and descriptor set explosion. #156).
Usability: Uploading data in the same JS task that's recording the commands. Right now this proposal forces the JS task to end but also doesn't give guarantees about when the promises will be fulfilled so we don't know if it will even be in the same frame. (an application could use setSubData but that's the lest efficient path).
Explicitness / predictability: Letting the application control its CPU and GPU memory consumption. It isn't clear when the ArrayBuffers are detached (on submit?) and it ties lifetime of GPU memory to the garbage collector for unsubmitted commands. The browser also has to implement a general-purpose allocator that can suffer from fragmentation (and no relocations are possible if you want the ArrayBuffers to point to the GPU-visible memory).

While it is true that #147 is bit more difficult to use for very beginner graphics programmers, I think most of the apparent complexity is because the proposal is fleshed out and spec-like. This proposal would have a similar level of complexity if it were written the same way.

The advantage of this proposal is that it looks more "Web-y" and is simpler to use if you've never done graphics programming before. But I'd argue that these developers can use setSubData to get started. Trading the advantages listed above for (unnecessary) beginner friendliness seems like the wrong choice.

@Kangz

@Kangz

We would need a description of what setting initial data looks like:
I think such a mechanism will end up extremely close to mapWriteAsync in #147.

Could be similar, although if it's only on createBuffer and only about writing, it does shove off a fair share of complexity. We could go as simple as:

Promise<WebGpuBufferInitializer> WebGPUDevice::createBufferInitialized(...);

interface WebGpuBufferInitializer: ArrayBuffer {
   void finish(); // unmaps the buffer
}

Then the only requirement we put is that finish() is called before any submissions. This appears much simpler than mapWriteAsync.

I'd like to discuss in more detail what use cases are being lost with this PR.
Efficiency, Usability, Explicitness / predictability

I don't feel like you are addressing my point/question here. It's about finding the actual use cases, while you are trying to analyze the cases.

Google Maps model and texture streaming falls into this as well

This is the case I've been asking for. So let's talk about this.

First, texture streaming. Are you proposing to introduce a similar mapping mechanism for textures? We haven't talked about that yet.

For model updates, if Google Maps wants to dynamically update geometry that is directly accessible and used by GPU, wouldn't you stall the GPU while you write the data? It seems to me that, if you want the case to be handled by #147, you'd either stall the GPU or run into the descriptor set issue because given a ring buffer of buffers, each would need to be in a separate descriptor set in order to avoid CPU-GPU collisions.

Right now this proposal forces the JS task to end but also doesn't give guarantees about when the promises will be fulfilled so we don't know if it will even be in the same frame.

I'm really out of touch on JS task frames and need to read more about it. What is the consequence for the user?

Letting the application control its CPU and GPU memory consumption
The browser also has to implement a general-purpose allocator that can suffer from fragmentation

Hasn't this train already left the station? We are not exposing memory allocation anyway as it stands now, and the implementation has to sub-allocate internally (with a caveat that it might be able to relocate pieces, but this is details).

It ties lifetime of GPU memory to the garbage collector for unsubmitted commands.

Ties to a destroy() call on the command buffer, alternatively. This isn't much different to any other objects that bear associated GPU resources.

@Kangz

@Kangz

We would need a description of what setting initial data looks like:
I think such a mechanism will end up extremely close to mapWriteAsync in #147.

Could be similar, although if it's only on createBuffer and only about writing, it does shove off a fair share of complexity. We could go as simple as:
Promise<WebGpuBufferInitializer> WebGPUDevice::createBufferInitialized(...);

interface WebGpuBufferInitializer: ArrayBuffer {
   void finish(); // unmaps the buffer
}
Then the only requirement we put is that finish() is called before any submissions. This appears much simpler than mapWriteAsync.

I think the above is missing a WebGPUBuffer somewhere ;) I still think that if you specified all of the behavior, it would be very similar to mapWriteAsync except that it happens only once and maybe doesn't need to wait for GPU execution of previous commands.

I'd like to discuss in more detail what use cases are being lost with this PR.
Efficiency, Usability, Explicitness / predictability

I don't feel like you are addressing my point/question here. It's about finding the actual use cases, while you are trying to analyze the cases.

Google Maps model and texture streaming falls into this as well

This is the case I've been asking for. So let's talk about this.

First, texture streaming. Are you proposing to introduce a similar mapping mechanism for textures? We haven't talked about that yet.

Linear textures are out of scope for the MVP and only D3D12 has a standard swizzle pattern.

For model updates, if Google Maps wants to dynamically update geometry that is directly accessible and used by GPU, wouldn't you stall the GPU while you write the data? It seems to me that, if you want the case to be handled by #147, you'd either stall the GPU or run into the descriptor set issue because given a ring buffer of buffers, each would need to be in a separate descriptor set in order to avoid CPU-GPU collisions.

Maps uses a lot of geometry but not that many uniforms. They could use more complicated schemes for caching geometry that would bundle several models up to for example 2MBs bundles. Then have something like an LRU cache except that it takes into account all the models in the bundle.

Or #156.

Right now this proposal forces the JS task to end but also doesn't give guarantees about when the promises will be fulfilled so we don't know if it will even be in the same frame.

I'm really out of touch on JS task frames and need to read more about it. What is the consequence for the user?

Basically the promise cannot be resolved until the JS stack frame is empty (basically until control is given back to the browser). In JS you'd work around this by using "async functions" which are continuations where async yields back to the browser:

async function updateUbo(cmdbuf) {
    const staging = await cmdbuf.uploadBuffer(ubo, uboSize, uboOffset);
    // put data in staging
    // add draws to cmdbuf that use the UBO
    queue.submit([cmdbuf]);
}

The problem is that it isn't clear how long it takes between the call and uploadBuffer and submit. It could be more than 1ms or even one frame. Worse, it is unpredictable.

For WASM applications, things are even worse because it would look like the following (depending on the interface provided by emscriptem):

static void* stagingPtr = nullptr;

void OnUploadCallback(void* ptr) {
    stagingPtr = ptr;
}

// Part 1: Somewhere in the frame we request upload
cmdbuf.UploadBuffer(ubo, uboSize, uboOffset, OnUploadCallback);
cmdbuf.DoSomethingWithTheUBO();

// Part 2: Somewhere else in the frame once stagingPtr != nullptr, we fill the data
decompressModelIn(stagingPtr);
cmdbuf.submit();

The problem is that somehow the engine has to yield control back to the browser between part 1 and part 2, and no native engine is set up to do that currently.

Letting the application control its CPU and GPU memory consumption
The browser also has to implement a general-purpose allocator that can suffer from fragmentation

Hasn't this train already left the station? We are not exposing memory allocation anyway as it stands now, and the implementation has to sub-allocate internally (with a caveat that it might be able to relocate pieces, but this is details).

Relocation cannot happen for mapped resources with mapWriteAsync so the only advantage #147 has is that for allocation will likely be more granular (for example with the RingBufferOfBuffer.js approach) than here where every tiny update would be an allocation.

It ties lifetime of GPU memory to the garbage collector for unsubmitted commands.

Ties to a destroy() call on the command buffer, alternatively. This isn't much different to any other objects that bear associated GPU resources.

Ah yes, we mentioned that earlier. It adds a tiny bit more complexity to the application but seems ok.

@Kangz

@Kangz small nit: I think you meant const staging = await cmdbuf.uploadBuffer(ubo, uboSize, uboOffset);

Thanks, edited!

I think the above is missing a WebGPUBuffer somewhere ;)

Haha, nicely spotted! Of course, a method like this would have to return both the buffer and a promise.

Linear textures are out of scope for the MVP and only D3D12 has a standard swizzle pattern.

Right, that's what I thought, so we shouldn't put up as a use case to support for this proposal (or any other buffer transfers proposal).

Maps uses a lot of geometry but not that many uniforms. They could use more complicated schemes for caching geometry that would bundle several models up to for example 2MBs bundles. Then have something like an LRU cache except that it takes into account all the models in the bundle.

Correct me if that's wrong, but wouldn't this be equivalent to just doing this in the simplified proposal:

if an entry in LRU cache isn't used by the GPU or recorded commands, and we need a new block of 2Mb data, then just create a new WebGPUBuffer initialized with the new data?

I.e. instead of having LRU cache, just destroy the buffers you don't need any more and create new ones. IIRC, it was acknowledged before by the group that resource creation/destruction needs to be fast given that the browser implementation abstracts actual memory backing for it.

The problem is that it isn't clear how long it takes between the call and uploadBuffer and submit. It could be more than 1ms or even one frame. Worse, it is unpredictable.

Discussed this a bit on hangouts. So the real problem here is that by the time the user requests an upload, they need to know what is it uploaded into. So it's more difficult to organize the uploads happening beforehand, and it's not convenient for the user to have to know more information up-front. TL;DR: this is one of the downsides of the proposal.

only advantage #147 has is that for allocation will likely be more granular (for example with the RingBufferOfBuffer.js approach) than here where every tiny update would be an allocation

User still don't control the actual allocation of VRAM. So even if you are managing 2Mb staging buffers in a ring-buffer, it's not guaranteed to do any less native memory allocations by the backend. I do agree that this memory is going to be less fragmented though, given that the user takes responsibility of managing those staging buffers.

This was discussed at the 14 Jan 2019 Teleconference

This proposal requires web developers to decide how much data to upload by passing an offset and size to the uploadBuffer. The API further requires developers to wait 1-2 frames before the ArrayBuffer for the upload becomes available for use.

The amount of data developers upload will vary from frame to frame depending on how many items are in view and other criteria. Forcing developers to wait 1-2 frames to upload data for a given frame seems onerous from an ergonomics perspective and introduce unwanted latency.

With native APIs, developers can calculate how much data they need and retrieve available space in a buffer right away without putting their frame on hold. Is there a way we can make this happen efficiently in WebGPU?

@RafaelCintron

@RafaelCintron what makes you think there is a delay of 1-2 frames? That would be unacceptable.
The ArrayBuffer is provided as soon as the implementation has the staging space ready, which is supposed to be near-instant in most cases, and can only be delayed if we ran out of the staging space, and an allocation is needed.

@Kangz

I believe @Kangz told me Chrome's implementation will wait 1-2 frames before the promise would get resolved, but I could be misremembering.

At any rate, by introducing a promise, you are guarenteeing that, at a minimum, the developer will have to wait until control returns back to the browser before the promise callback will get executed. Once the developer has the count of items to draw, they'll going to want to get the information into the buffer right away, without introducing additional delay.

In upcoming specs such as WebXR, frame based objects such as XRFrame are only valid during a RAF callback. So waiting until the promise callback to figure out what you're going to put into your buffer is not going to work.

If "near-instant" is the (highly scientific) amount of time you're targeting and the browser will be servicing requests out of a ring buffer, perhaps we should have uploadBuffer return a buffer that gets auto-unmapped at the end of Javascript execution.

I think downloadBuffer should remain promise based. I expect the bulk of the time here will be spent on actual readback, not callback servicing.

@RafaelCintron

@RafaelCintron thanks for information! The case of XRFrame is a good one to consider, and I see how this isn't working well with Promise in general. The other proposals also heavily rely on Promise, so it would be good to see how they are planning on handling the case as well.

One straightforward way to address it would be to have the following type returned by upload commands:

interface WebGPUUploadData {
  Promise<ArrayBuffer> promise();
  void fill(ArrayBuffer data);
}

With this the user always has a choice of providing the data immediately instead of scheduling a promise. The trade-off is that the data has to be plain and ready (as opposed to being in some other form that can be processed to fill an ArrayBuffer in place).

Another side-effect of this interface is that in a simple case like this (pseudo-code):

var buffer = device.createBuffer(...);
var commandBuffer = device.createCommandBuffer(...);
commandBuffer.uploadBuffer(buffer, <whole buffer range>).fill(initiaData);
device.getQueue().submit(commandBuffer);

The implementation can see that the provided data covers the whole buffer, and that no other command buffers (using our new buffers) are submitted in the meantime, so we can just use the initialData to initialize the buffer in place if the device local memory is CPU visible. Also, unlike anything going through mapping API, this doesn't need to zero out the contents.

@Kangz

We talked about this proposal with @Kangz , and here is the update.

Proposal V2 API

partial interface WebGPUDevice {
  (WebGPUBuffer, WebGPUUserData) createBufferWithData(...);
  (WebGPUBuffer, Promise<WebGPUUserData>) createBufferWithDataAsync(...);
};

dictionary WebGPUTextureDataLayout {
    u32 rowPitch;
    u32 slicePitch;
};

interface WebGPUUserData: ArrayBuffer {
  /// Detaches the ArrayBuffer from the backing storage.
  /// Needs to be done before the `submit` call using the relevant command buffer.
  void finish();
};

partial interface WebGPUCommandBuffer {
  WebGPUUserData uploadBuffer(
    WebGPUBuffer dst,
    u32 dstOffset,
    u32 size);

  Promise<WebGPUUserData> downloadBuffer(
    WebGPUBuffer src,
    u32 srcOffset,
    u32 size);

  WebGPUUserData uploadTexture(
    WebGPUTextureDataLayout layout,
    WebGPUTextureCopyView destination,
    WebGPUExtent3D size);

  Promise<WebGPUUserData> downloadTexture(
    WebGPUTextureCopyView source,
    WebGPUTextureDataLayout layout,
    WebGPUExtent3D size);
}

Explanation

createBufferWithData*

These are helpful methods for initializing the buffer contents, which allow the implementation to perform the most efficient code path for a platform. This means zero copy initialization on UMA architectures where there either no IPC boundary between content and GPU, or extra capabilities (e.g. Vulkan extensions) are available to share the data between GPU and the content process.

With createBufferWithData if the content process doesn't have enough memory available for the buffer, and it can't allocate it on GPU, it will allocate shared process memory for this, which is then provided to the GPU process. Thus, in some cases there is going to be an extra copy. That is why we also expose createBufferWithDataAsync, which let's the implementation take a bit of preparation time and guarantee zero copy if it's possible.

WebGPUUserData

When uploading the data, instead of returning promises we return WebGPUUserData right away. This allows uploading without losing the stack frame, which is useful for XRFrame scenario. Similarly to createBufferWithData, the implementation may decide to allocate a piece of CPU shared memory if it doesn't have enough staging space available by hand.

Note the finish() call that now has to be explicit by the user before any submit is invoked that (conservatively) depends on the upload results. This allows an application to detach the mapping earlier, giving more time for the implementation to prepare before dependent commands are executed.

When downloading, the Promise is still required, because the content process never has the data by hand.

FAQ

Why do we need this proposal?

It's the only proposal that tries to abstract away the concept of "staging" memory/resources from the user. Using staging memory properly and efficiently is always a challenge for developers, which is magnified by the variety of architectures that expose memory types with different capabilities and constraints. Abstracting the staging away provides us with better portability and wider pit of success for developers.

What is the main drawback of the proposal?

The complexity isn't going away, it's just being abstracted away from the user, but is still contained in the implementation. We'll need a general allocator for staging resources in it, which tracks and re-cycles the arbitrary sized chunks of memory.

The API is constructed in such a way that those allocations are short-lived. The user can still make the life harder for the implementation by recording command buffers that upload resources but not using them, but this is not expected.

How is the general staging allocator complexity different from the one we have for resource creation?

WebGPU doesn't currently expose memory allocations to the user, so resource creation and destruction also relies on general allocation facilities in the implementation. There are differences between this and the staging allocator:

Resource allocator can technically move data around, if it needs to occasionally fight the fragmentation
Resource allocations are long-living, while staging ones are short-living. This aspect puts different requirements and calls for different trade-offs in the implementation.

@Kangz

We talked about this proposal with @Kangz , and here is the update.

Clarification: @Kangz helped evolving this proposal, but it doesn't reflect their position on how we should proceed. It just provides a fair chance for the proposal to be considered.

Navigation Menu

Repository navigation

Proposal for a simple streaming API #154

Intro

Code

Semantics

Analysis

Proposal V2 API

Explanation

createBufferWithData*

WebGPUUserData

FAQ

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal for a simple streaming API #154

Description

Intro

Code

Semantics

Analysis

Activity

Kangz commented on Dec 21, 2018

kvark commented on Dec 21, 2018

litherum commented on Dec 26, 2018

Kangz commented on Dec 26, 2018

kvark commented on Dec 27, 2018

Kangz commented on Dec 30, 2018

kvark commented on Jan 7, 2019

Kangz commented on Jan 9, 2019

kvark commented on Jan 9, 2019

Kangz commented on Jan 10, 2019

kvark commented on Jan 10, 2019

Kangz commented on Jan 10, 2019

grovesNL commented on Jan 10, 2019

Kangz commented on Jan 10, 2019

kvark commented on Jan 10, 2019

grorg commented on Jan 14, 2019

RafaelCintron commented on Jan 15, 2019

kvark commented on Jan 15, 2019

RafaelCintron commented on Jan 17, 2019

kvark commented on Jan 17, 2019

kvark commented on Jan 18, 2019

Proposal V2 API

Explanation

createBufferWithData*

WebGPUUserData

FAQ

kvark commented on Jan 22, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions