Proposal: BufferSubRanges to solve buffer-tracking, streaming and descriptor set explosion.

@Kangz

Motivation and Overview

TL;DR Basically TextureViews, but for Buffers.

As you may have noticed from #138 I'm of the opinion that whole-buffer tracking will be a great hindrance to porting and general development, and will require WebGPU specific work-arounds to make normal resource streaming possible.

However the WebGPU group's focus on portability throws the usual performant implementations off the table.

In general, Mapping Proposal 1 from #138 is quite good, as it allows for:

Zero-copy write or readback (if heap sharing, external memory are available or WebGPU is a singleprocess implementation)
Using the mappable buffer for any purpose
However it fails w.r.t. the following:
Having to create multiple copies of the buffer due to whole-buffer single mutable usage tracking

Extending that proposal by the introduction of the new BufferSubRange (name T.B.C.) object will allow finer-grain usage tracking of buffers by declaring the sub-ranges to be tracked up-front, while an alignment requirement ensures that the total number of sub-ranges and the difficulty of tracking them is reduced.
This allows us to partition a WebGPUBuffer (which we hope is represented by a single native-API buffer) into multiple WebGPUBufferSubRanges on which the actual "single mutable state" rule gets enforced.
This will prevent "Bind Group Explosion" and other side-effects in streaming scenarios, such as with this example streaming allocator made by @Kangz

// A "mapping" is a WebGPUBuffer and its ArrayBuffer mapping.
// {"buffer": WebGPUBuffer, "pointer": ArrayBuffer}

let availableMappings = [];
let mappingsUsedThisFrame = [];

let currentMapping = null;
let currentOffset = 0;

function getFreshMapping() {
    if (availableMappings.length == 0) {
        let buffer = device.createBuffer({"size": 2*1024*1024, "usage": ...});
        let pointer = buffer.mapWriteSync(0, buffer.size);
        availableMappings.push({"buffer": buffer, "pointer":pointer});
    }

    currentMapping = availableMappings.pop();
    mappingsUsedThisFrame.push(currentMapping);

    return currentMapping;
}

// Returns a (mapping, offset) pair.
function requestUploadSpace(size) {
    if (currentMapping == null || currentOffset + size > currentMapping.buffer.size) {
        getFreshMapping();
    }

    let offset = currentOffset;
    currentOffset += size;
    return (currentMapping, offset)
}

function unmapUploadBuffers() {
    mappingsUsedThisFrame.forEach(mapping => mapping.buffer.unmap());
}

function remapAsyncUploadBuffers() {
    mappingsUsedThisFrame.forEach(mapping => {
        mapping.buffer.mapWriteAsync(0, mapping.buffer.size).then(pointer => {
            availableMappings.push({
                "buffer": mapping.buffer,
                "pointer": pointer,
            });
        });
    });
}

function endOfFrame() {
    mappingsUsedThisFrame = [];
    currentMapping = null;
    currentOffset = 0;
}


function frame() {
    let commands = device.createCommandBufferBuilder();

    frameObjects.forEach(object => {
        // uploadUniformData uses requestUploadSpace to know in which ArrayBuffer to put data
        // it also knows which WebGPUBuffer this corresponds to and return it so it can be used
        // to build the command buffer.
        const (buffer, offset) = uploadUniformData(object)
        renderObjectUsingUniform(object, commands, buffer, offset)
    });

    unmapUploadBuffers();

    queue.submit(commands);

    remapAsyncUploadBuffers();
    endOfFrame();
}

In such a scenario the WebGPU app would have to make a WebGPUBindGroup for each possible combination of actual buffer objects being used.
If however we could use dynamic buffer offsets #116 and track per-WebGPUBufferSubRange instead, then there would be no need for creating and using different bind-groups.

Mapping Proposal

We introduce a new object, its creation parameters and mapping flags

interface WebGPUMappingType {
    const u32 MAP_READ = 1;
    const u32 MAP_WRITE = 2;
};

dictionary WebGPUBufferSubRangeDescriptor {
    u32 offset;
    u32 size;
    WebGPUBufferUsageFlags usage;
};

interface WebGPUBufferSubRange {    
    Promise then(WebGPUUsageFinishedCallback success);

    WebGPUMappedMemory mapAsync(WebGPUMappingType accessFlags, u32 offset, u32 size);
    void unmap();

    void destroy();
};

interface WebGPUBuffer {
    WebGPUBufferSubRange createSubRange(WebGPUBufferSubRangeDescriptor descriptor);
};

The actual mapping is a slight spin on Proposal 1 from #138 , with a few changes:

offset is an offset into the WebGPUBufferSubRange, not the parent WebGPUBuffer
WebGPUBuffer cannot be mapped, only a subrange represented by WebGPUBufferSubRange can.
The promise is generalized and is used to queue up a callback which will return after the BufferSubRange's last use by any of the pending and already submitted command buffers. Any command buffer that uses a BufferSubRange that has any promise pending at the time of WebGPU submission will wait with actual native API submission until these promises reject or fullfill and the callbacks return.
Calling mapAsync will throw an error if the BufferSubRange is "in use" by the GPU, which we define as being used in a currently executing command on any queue or by any command which has not yet executed but in a command buffer that has already been submitted to any queue.
SubRange must not be destroyed while mapped or "used" by the GPU
WebGPUBuffer cannot be destroyed while any of its "children" WebGPUBufferSubRange cannot be destroyed.
The memory can be mapped for both read and write (not necessary, but useful)
For non-read mappings the memory is either zero-ed or memory recycled from previous WebGPUBufferSubRange mappings made by the same application (not necessary, but doubles streaming performance)
I'm obviously open to modifications to points 3-onwards.

Key advantages of this proposal:

Persistently Mapped Buffers can be used by all implementations internally
Zero-copy is possible on singleprocess implementations and any implementations which can use Vulkan external memory or D3D12 shared heaps
It's easy to use WebGPUBufferSubRanges in-place of memory-blocks in existing mappable gpu buffer allocators, also using WebGPUBuffer.createSubRange and WebGPUBufferSubRange.destroy as a fast and easy way to coalesce free memory blocks
Two WebGPUBufferSubRanges created from the same WebGPUBuffer are actually sub-ranges of the same native API buffer object, hence internally they are the same object which optimizes descriptor set changes and creation.
Like in native Persistently Mapped Buffer applications, we can have one range of the buffer mapped while having the GPU use another
WebGPUBufferSubRange creation and destruction are really light-weight operations that require no API calls except for event polling
Granularity of range-splitting is reduced by alignment requirement

WebGPUBufferSubRange Creation

The expectation is that WebGPUBufferSubRange creation and destruction will be really high performance compared to WebGPUBuffer creation and destruction.

WebGPUBuffer.createSubRange will return an error if any of the following are not true:

descriptor.offset and descriptor.size are both multiples of 256 (can be higher, even 4096) [1]
The [descriptor.offset,descriptor.offset+descriptor.size) range is not contained within the creating buffer
The [descriptor.offset,descriptor.offset+descriptor.size) range includes any part of a range assigned to a WebGPUBufferSubRange previously created from the same WebGPUBuffer which has not been destroyed yet
Each bit in the descriptor.usage field was also present in the WebGPUBufferDescriptor.usage field that the creating WebGPUBuffer was created with
If descriptor.offset is not equal to 0 or descriptor.size does not equal the creating WebGPUBuffer's size, then descriptor.usage field CANNOT contain the WebGPUBufferUsage flags INDEX nor VERTEX
The creating WebGPUBuffer has been destroyed

[1] This places the same requirement on WebGPUBuffer size alignment.
[2] This only applies is the native API provided robustness guarantees are relied on.

Word about Robustness

Because the WebGPUBufferSubRange does not map 1:1 with the native APIs' buffer objects (only WebGPUBuffer does), we cannot rely on out-of-bounds accesses of index buffers or vertex buffers to be handled gracefully (because they are not bound as ranges, unlike UBOs and SSBOs).

This is why is the WebGPUBufferSubRange does not contain the entire parent WebGPUBuffer, then it could not be used as vertex (fetched by bad indices) or index (fetched by bad draw indirect) input data.

This obviously does not apply if #117 was to be implemented, or if the robustness gurantees of the native APIs were deemed not sufficient and WebGPU implementations handle out-of-bounds and invalid accesses entirely by themselves.

Changes to the rest of WebGPU

Naturally WebGPUBufferSubRange would take the place of WebGPUBuffer in most APIs except for bind-groups.

Changes to Bind Groups

These would be the new Bind Group and Binding definitions:

dictionary WebGPUStaticBufferBinding {
    WebGPUBufferSubRange bufferRange;
    u32 offset;
    u32 size;
};

dictionary WebGPUDynamicBufferBinding {
    WebGPUBuffer buffer;
    u32 size;
};

typedef (WebGPUSampler or WebGPUTextureView or WebGPUStaticBufferBinding or WebGPUDynamicBufferBinding) WebGPUBindingResource;

Assuming #149 being merged, and modifying it

dictionary WebGPUBufferSubRangeAndOffsetPair {
    WebGPUBufferSubRange subBuffer;
    u32 offset; // offset is an offset into subBuffer specified above
};

partial interface WebGPUProgrammablePassEncoder {
    void setBindGroup(u32 index, WebGPUBindGroup bindGroup, optional sequence<WebGPUBufferSubRangeAndOffsetPair> dynamicOffsets);
}

Here we add constraints that:

The WebGPUBuffer given by WebGPUDynamicBufferBinding.buffer of the WebGPUDynamicBufferBinding associated with a sequence element WebGPUBufferSubRangeAndOffsetPair, must be the parent creating buffer of the WebGPUBufferSubRangeAndOffsetPair.subBuffer
The range given by [WebGPUBufferSubRangeAndOffsetPair.offset,WebGPUBufferSubRangeAndOffsetPair.offset+WebGPUDynamicBufferBinding.size) must not overflow WebGPUBufferSubRange.subBuffer

And finally, changing the rest:

partial interface WebGPURenderPassEncoder {
    void setIndexBuffer(WebGPUBufferSubRange buffer, u32 offset);
    void setVertexBuffers(u32 startSlot, sequence<WebGPUBufferSubRange > buffers, sequence<u32> offsets);
};

dictionary WebGPUBufferCopyView {
    WebGPUBufferSubRange subBuffer;
    u32 offset;
    u32 rowPitch;
    u32 imageHeight;
};

partial interface WebGPUCommandBuffer {
    void copyBufferToBuffer(
        WebGPUBufferSubRange src,
        u32 srcOffset,
        WebGPUBufferSubRange dst,
        u32 dstOffset,
        u32 size);
};

Thank you for writing down the proposal!

Calling mapAsync will throw an error if the BufferSubRange is "in use" by the GPU

Unfortunately, this makes the GPU timeline to become a part of valid usage rules. This would be a big foot-gun for the users as well as a potential portability hazard due to timing dependency.

SubRange must not be destroyed while mapped or "used" by the GPU

So the implementation would have to track what ranges of a buffer are used by GPU and compare them against the mapped ranges. To clarify, does that mean with the proposal we'll also be able ot use different parts of the buffer for different usages on GPU?

For non-read mappings the memory is either zero-ed or memory recycled from previous WebGPUBufferSubRange mappings made by the same application (not necessary, but doubles streaming performance)
I'm obviously open to modifications to points 3-onwards.

Right, this concern is common between all 3 proposals.

offset is an offset into the WebGPUBufferSubRange, not the parent WebGPUBuffer

Could you clarify what use cases would require the sub-offsets within views? Is that because of the large size/offset granularity (256) while the user may need to fit multiple smaller entries (of, say, constant data) into a single view?

Unfortunately, this makes the GPU timeline to become a part of valid usage rules. This would be a big foot-gun for the users as well as a potential portability hazard due to timing dependency.

I'm not hell-bent on that behaviour, but it would be unified and make sense if the standard paractice advised to users is to do the actual mapAsync in a callback passed to the aforementioned promise.

So the implementation would have to track what ranges of a buffer are used by GPU and compare them against the mapped ranges. To clarify, does that mean with the proposal we'll also be able ot use different parts of the buffer for different usages on GPU?

Only what ranges have been created and their use by the GPU, the mapped ranges must be wholly contained within the WebGPUBufferSubRange object. In-fact if you didn't care about memory pressure in multi-process Metal implementations, then you could always map the entire SubRange.

To clarify, does that mean with the proposal we'll also be able ot use different parts of the buffer for different usages on GPU?

I don't know which meaning of "different parts of the buffer" and "different usages on GPU" you are using, so you'd need to elaborate before I can answer.

Could you clarify what use cases would require the sub-offsets within views? Is that because of the large size/offset granularity (256) while the user may need to fit multiple smaller entries (of, say, constant data) into a single view?

Its because we don't really want to create a whole lot of WebGPUBufferSubRanges, and in most cases you should really only have 1 SubRange per Buffer.
We still want to retain the usage model as we'd have with native API Buffers, such as being able to copy to and from almost arbitrary offsets, binding vertex data at very low granularity offsets, etc.

Is that because of the large size/offset granularity (256) while the user may need to fit multiple smaller entries (of, say, constant data) into a single view?

That would indeed be one of the reasons (however constant data offset alignments are between 64 and 256 anyway, but BufferSubRange alignments could me made much higher than 256).

@Kangz

I have a bunch of thoughts about this, but before I comment, I’d like to see a streaming allocator using the new facilities, to compare against the first code example that @Kangz wrote above.

@Kangz

I have a bunch of thoughts about this, but before I comment, I’d like to see a streaming allocator using the new facilities, to compare against the first code example that @Kangz wrote above.

It would be basically identical, except that it would be feasible to merge/split the reserved blocks without any native API calls (you would not be tied to 2MB blocks).

And you would not have to create a new WebGPU BindGroup for every block separately (important), and that was not visible in the original example as the bind-group code was omitted.

@devshgraphicsprogramming

@devshgraphicsprogramming thanks for writing this down. It doesn't sound as insane as it first did but it is certainly complicated. Should we decide to go ahead with #147 for the MVP/first snapshot, I think the mechanism you suggest would be a good candidate for 1.0.

A couple high-level points:

It should be based off Add explainer document for BufferOperations #147 which simplifies buffer mapping and removes the need for WebGPUMappedMemory. The semantics for mapping there don't return errors if the buffer is in use but instead return a promise, which solves @kvark's concerns.
I suggest making WebGPUBufferSubRange the same as WebGPUBuffer and make the distinction between root WebGPUBuffer and child WebGPUBuffer.
Mapping could then be simplified to not have a size and offset. If only a small range is needed, the buffer can be partitioned.
We need to know what happens with the parent WebGPUBuffer while it has child buffers. I suggest it puts it in a special "partitioned" state that prevents any usage until all child buffers are deleted.

I suggest making WebGPUBufferSubRange the same as WebGPUBuffer and make the distinction between root WebGPUBuffer and child WebGPUBuffer

That confuses the whole thing in my opinion, its nice to know that you cannot use the WebGPUBuffer for anything except for bind-group creation and that the SubRanges cannot be aliased.

Mapping could then be simplified to not have a size and offset. If only a small range is needed, the buffer can be partitioned.

That's why suggested mapping the entire sub-range, if there was a reasonable assumption of most implementations being able to do zero-copy.

We need to know what happens with the parent WebGPUBuffer while it has child buffers. I suggest it puts it in a special "partitioned" state that prevents any usage until all child buffers are deleted.

In case it wasn't clear, the proposal changes the API so that WebGPUBuffer cannot be used in anything at all (except for dynamic bind group binding). Even the dynamic buffer binding requires a child-buffer to fill-in the role of the offset and size. Only the child buffers are actually usable for anything.

If you want the simplest buffer usage possible (no mapping, etc.) then you only create 1 child buffer encompassing the entire parent.

@devshgraphicsprogramming

@devshgraphicsprogramming

I don't know which meaning of "different parts of the buffer" and "different usages on GPU" you are using, so you'd need to elaborate before I can answer.

Basically, I was trying to confirm if the user can create view A used as a vertex buffer and view B used as a resource from the same buffer. So views A and (referring to "different parts of the buffer" would have "different usages on GPU").

That's why suggested mapping the entire sub-range, if there was a reasonable assumption of most implementations being able to do zero-copy.

That sounds good.

That confuses the whole thing in my opinion, its nice to know that you cannot use the WebGPUBuffer for anything except for bind-group creation and that the SubRanges cannot be aliased.

So you suggest the actual WebGPUBuffer to only be used for:

view creation
bind group creation
copy operations (not mentioned, but I don't see how views are helpful there)

While the WebGPUBufferView is used for vertex/index/indirect buffers and mapping?

Also, going back to Dota's use case (or any other complex real-world-ish use-case), how do you see the implementation based on the proposed API? Would it create the views in chunks of a few MB and map them to fill the uniform data?

Finally, can you describe more what you think the implementation would be doing? I.e. when recording a command buffer, it only checks that each single buffer view has the same usage during a render pass. It also supposedly collects all the ranges used by the buffer resource bindings, at the synchronization scope granularity (e.g. per operation or a render pass), so that, at submission time, it can determine the dependencies between GPU and CPU.

@Kangz

I think the mechanism you suggest would be a good candidate for 1.0

I'm curious to see this proposal rebased on #147.

@Kangz

Basically, I was trying to confirm if the user can create view A used as a vertex buffer and view B used as a resource from the same buffer. So views A and (referring to "different parts of the buffer" would have "different usages on GPU").

Well if there are no limitations on ranges created with vertex buffer usage (robustness).

Then yes, you could create both A and B, if the parent buffer was created with both usage flags.
And obivously A and B cannot overlap.

The only reason I proposed this sub-buffer sub-specialization because of the restriction on the vertex and index buffer usages (if robustness will be used then only full buffer subrange can be used).
In case of invalid usage, this would turn a nasty vertex/index input bind command-buffer-recording-time error into a subrange creation time error.

So you suggest the actual WebGPUBuffer to only be used for:

Only SubRange creation, bind-group creation, and only when declaring the dynamic binding.

If you go back and look at my changes to copyBufferToBuffer you will see that only the subranges get used there.

As you see my reply to your last point, it will become clear that these limits on WebGPUBuffer make a lot of sense and remove a lot of headaches.

While the WebGPUBufferView is used for vertex/index/indirect buffers and mapping?

What is WebGPUBufferView , you mean the proposed subrange object?
(P.S. assuming so)

Also, going back to Dota's use case (or any other complex real-world-ish use-case), how do you see the implementation based on the proposed API? Would it create the views in chunks of a few MB and map them to fill the uniform data?

It could do what @Kangz's allocator did, but thanks to fast creation/destruction of the views (and being able to specify the exact range on which to create), it could work with variable sized blocks and recycle them without hitches. This will mean:

All of the frame's uniforms can sit in one Buffer View
Blocks can fit the required memory size better
Even if the frame data is split across multiple blocks then there is only need for one bind-group

The last point means that even if for some strange reason you just take @Kangz's allocator without algorithmic changes and require 100 blocks (block associated with 1:1 with view) for the frame UBO or SSBO data, then you don't need 100 bind-groups, only 1.

View creation and destruction will be much faster than buffer creation and destruction, because you explicitly state which memory location you want (by providing an offset) and there is never a possibility of:

Running out of memory and migrating resources
Native API memory creation
Suballocation from Native API memory object (Vk Memeory Allocator)
Any Native API call

Performance of view creation and destruction should be linear or logarithmic in the number of already created views from the same buffer (of which you'd rarely need double digit numbers).

Compare that to WebGPUBuffer creation/destruction which will be hopefully logarithmic in the number of all memory allocations (incl. textures) currently held by the App, but more likely all memory allocations currently held by all Apps being serviced by the WebGPU implementation.

Finally, can you describe more what you think the implementation would be doing? I.e. when recording a command buffer, it only checks that each single buffer view has the same usage during a render pass. It also supposedly collects all the ranges used by the buffer resource bindings, at the synchronization scope granularity (e.g. per operation or a render pass), so that, at submission time, it can determine the dependencies between GPU and CPU.

It would do exactly the same to a Buffer View as it would have to do with a WebGPUBuffer prior to this proposal.

Basically WebGPUBufferSubRange takes the place of WebGPUBuffer in all validation, use and usage tracking.

The only time the implementation or the App need to concern themselves with a WebGPUBuffer is when:

creating and destroying the Buffer Views (make sure usage matches, no overlap with already existing, no overflow, etc.)
Creating a Bind Group with a dynamic buffer binding (set up native API binding correctly and hold a binding size and a reference to the WebGPUBuffer for validation providing dynamic BufferView and offset)
Validating the Buffer View and Offset provided for a dynamic binding

I hope that clears it up.

I suggest making WebGPUBufferSubRange the same as WebGPUBuffer and make the distinction between root WebGPUBuffer and child WebGPUBuffer

That confuses the whole thing in my opinion, its nice to know that you cannot use the WebGPUBuffer for anything except for bind-group creation and that the SubRanges cannot be aliased.

Sorry I wasn't clear. I suggest making WebGPUBufferSubRange the same as a WebGPUBuffer and allowing non-partitioned root WebGPUBuffers to work like they do without this proposal. This way the proposal would introduce only the following changes:

WebGPUBuffer partitioning and constraints
Mapping whole buffers only since you can partition to map sub ranges.
A WebGPUBuffer can be used instead of another one for dynamic bindings as long as they have the same root WebGPUBuffer.

I think that would convolute the wording of the spec and make it harder
to write programs that run without errors.

In my opinion the semantics of my proposal are more intuitive than

A |WebGPUBuffer| can be used instead of another one for dynamic
bindings as long as they have the same root |WebGPUBuffer|.

The main problem I foresee is that if you make BufferSubRange and
Buffer the same, you may experience the following problem:

Bound the entire buffer into a bind-group or somewhere else
Partitioned the buffer
Now whole buffer is invalid to use
Previous bindings are now wrong
How to notify this to the user?

With distinction between the objects, the user is never in a situation
where they could experience this sort of "state pollution".

Essentially the buffer behaves as-if you always had an implicit view on
the entire range from the beginning, but it gets implicitly destroyed as
soon as you make an explicit view.

The problem is with middlewares or external libraries which could
suddenly throw buffers into "partitioned mode" unbeknownst to the user.
Subsequently producing errors in the user's own code.

With my proposal, creating a child view is necessary so the errors would
happen in middleware code if the middleware was trying to create a view
over an already assigned range.

But whatever works for you, as long as you keep the "mechanism of
action" that prevents bind-group (descriptor-set) explosion and allows
for explicit specification of ranges, I'm not really strongly opposed.

This was discussed at the 14 Jan 2019 Teleconference

I'm just going to mention that I've had the reason and cause (and doubtful pleasure) to implement a c++ stateful allocator for std::vector this week, for the purpose of in-place procedural resource streaming (could be used in future asset streaming from disk).

Reason: Codebase too big to switch away to anything not API-compatible with std::vector.

Closing since buffer mapping was added to the spec after discussing dozens of proposals. Further discussion can go in new issues (or we could reopen this one)

Update pickExtraBindingTypes for comparison-sampler (gpuweb#156)

Navigation Menu

Repository navigation

Proposal: BufferSubRanges to solve buffer-tracking, streaming and descriptor set explosion. #156

Motivation and Overview

Mapping Proposal

WebGPUBufferSubRange Creation

Word about Robustness

Changes to the rest of WebGPU

Changes to Bind Groups

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal: BufferSubRanges to solve buffer-tracking, streaming and descriptor set explosion. #156

Description

Motivation and Overview

Mapping Proposal

WebGPUBufferSubRange Creation

Word about Robustness

Changes to the rest of WebGPU

Changes to Bind Groups

Activity

kvark commented on Dec 25, 2018

devshgraphicsprogramming commented on Dec 25, 2018

litherum commented on Dec 26, 2018

devshgraphicsprogramming commented on Dec 26, 2018

Kangz commented on Dec 26, 2018

devshgraphicsprogramming commented on Dec 26, 2018

kvark commented on Dec 27, 2018

devshgraphicsprogramming commented on Dec 27, 2018

Kangz commented on Dec 30, 2018

devshgraphicsprogramming commented on Dec 31, 2018

grorg commented on Jan 14, 2019

devshgraphicsprogramming commented on Jan 30, 2019

Kangz commented on Sep 2, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions