Repository navigation
Image uploading is insufficiently expressive to be optimized #4185
Description
Currently, if you have a PNG or JPG for example, and want to upload it into WebGPU, you must upload through an ImageBitmap (... or a canvas, technically). The basic set of ways to upload are:
- HTMLImageElement -> createImageBitmap -> copyExternalImageToTexture
(Noteimage.decodeis not useful here, because createImageBitmap is already asynchronous, and already expresses the intent that the image must be decoded.) - fetch -> blob -> createImageBitmap -> copyExternalImageToTexture
The core problem is, it's impossible to express the full intent of "I have a URL for an image and I want to copy it into a GPUTexture". The createImageBitmap step in particular cannot know that the goal is to copy the image into a GPUTexture and then throw it away and never use it again, so there's no real way to avoid an extra copy in the middle.
EDIT: Note WebGPU no longer requires going through createImageBitmap() to upload images. But the inability to optimize and the inability to stream are still open issues.
Reacted by Kelsey Gilbert
Activity
kainino0x commentedon Jun 14, 2023
IMO, uploading HTMLImageElement (#3430) is not a great solution to this, though it is a better stopgap than what we have now. My reasoning:
image.decodecould theoretically express intent to put it in the DOM, but it's already often used for WebGL as well. And even so, not callingimage.decodeis a very poor signal for what you want to do with the image.kainino0x commentedon Jun 14, 2023
I think my ideal path would look something like:
That is, it's good for the network part to be separate from decode-and-upload, but decode-and-upload should be one step. However, that lacks a place for image decode to occur asynchronously. To spitball:
shaoboyan commentedon Jun 15, 2023
IIUC, this means we want to config the (image) decoder output resource to GPUTexture. It's a bit like
ImportExternalTexturewhich we possible to hold "undecoded" content and "decode" when user use it (in draw or dispatch).So I think I'm tend on option 1 (do that but put decoding "on" the queue), which the decode states is checked only when the resource is in use (and then block for decoding, or decode task is already finished).
But I don't think
copyExternalImageToTextureAPI is a good place for it. Instead, I prefer something likeimportExternalTexture.greggman commentedon Jun 16, 2023
I'm not sure where this discussion is leading
fetch -> blob -> copyExternalImageToTexture
Seems scary to me (maybe it shouldn't?). In my mind I have an image loading library (libpng, libjpg, libwebp) and ideally I'd just like to use them for whatever purpose I want (WebGL, WebGPU, Canvas2D, custom image processing, ML, etc...) Those libraries happen to be built into the browser and are made to run async so I just need well designed API to access them.
ImageBitmapor something like it, seems like the direction there, just add some more flags? .... or not. I can see the conflict. It would be strange to pass in{usage: "webgpu"}to tellcreateImageBitmapthat the ultimate destination for this image is "webgpu". On the other hand, it seems really strange for `copyExternalImageToTexture" to be the place that needs to handle load/decode every possible "external image" including hi-color images, etc for a blobIn other words it would be strange to me if WebGPU has a image loader in it's API. The current model seems to be WebGPU mostly does raw textures and buffers. The loading/decoding all happens elsewhere, outside of WebGPU, only after it's loaded/decoded, is the data passed to WebGPU.
That seems like normal model. I don't think OpenGL, Metal, DirectX, Vulkan have image loaders.
kainino0x commentedon Jun 16, 2023
I think the distinction is that it's common for native applications to use their own decoders, whereas on the Web we strongly prefer apps to use the browser's builtin (async) decoders. Where a native application could easily decode directly into mapped memory, on the Web AFAIK we don't have a way to ask the built-in decoders to do that. Maybe it could be added to WebCodecs' ImageDecoder or something, but I think it would still have to be synchronous. Providing a higher-level API would let the browser implement this however it wants, and asynchronously.
This doesn't strike me as a problem. The browser already does this for createImageBitmap, we're just removing the intermediate ImageBitmap object from the equation.
greggman commentedon Jun 17, 2023
The difference to me is
ImageBitmapis a generic image container, I can pass anImageBitmaptocanvas2DCtx.drawImage, towebgl.texImage2D, to webgpu'scopyExternalImageToTexture, where asblob->copyExternalImageToTextureis specifically WebGPU only. That means the problem being solved (getting data ready to use for a particular API) is only being solved for a single APIAll kinds of new features and issues will be appearing soon. For example, if you decode a jpeg with a gainmap for hi-color, do you get 8bits or 16bits or 10bits, 16 floats or 16bit uint? etc.. Ideally I want to be able to load
.jpgwith gainmap and draw it in canvas2d, load it in webgl, and load it webgpu.If I load a PNG with a display-p3 color profile what data should I get from it? I want to be able to draw it into a display-p3 canvas with canvas2d, webgl, and webgpu. Does that have any implications for how to design a solution?
kainino0x commentedon Jun 27, 2023
This seems OK to me. If we keep the goal that we want all the info up front before we start decoding, then trying to have one API for this (e.g. more ImageBitmap parameters) only makes things more complicated. The list of things we need to know keeps growing:
So I think it's both easiest to standardize, and fine for developers, to solve this separately for each API in a way where all the final usage is known implicitly, and the browser can just use whatever info it wants. The only benefit I see to extended the ImageBitmap API is maybe we could make all of the new parameters backward compatible such that new code still works in older browsers. But ultimately I think this will be more painful for developers.
Implementations (and specs) should share code (prose) between blob-to-GPUTexture, blob-to-ImageBitmap, ImageBitmap-to-whatever, etc. in such a way that they get these implicitly, and we don't have to respecify/reimplement all of the corner cases just because we add a new entry point. Of course, it means we have to test these cases, but that was true anyway.
It seems the same as, if we add gainmapped jpegs, then we have to add gainmapped ImageBitmaps, now we have to handle that in copyExternalImageToTexture.
kainino0x commentedon Jun 28, 2023
To distill a particular point that came up while talking to Gregg: there are AFAIK 4ish representations of encoded images in the web platform:
(await fetch()).body(ImageDecoder can turn this into a VideoFrame)In all cases the object can point to invalid data, so the error cases are roughly similar for all 4, but in the case of both HTMLImageElement and streams, we either have to require the image to be fully downloaded (iirc WebGL does this for HTMLImageElement), or deal with the fact that the download may not be finished yet. There is some value to pipelining decoding with downloading (see ImageDecoder), and we should keep that in mind.
Simply accepting one or more of these (fully downloaded) allows a browser to either do a blocking decode like you get in WebGL today, or implement what I called option 1 above. But if e.g. Chrome goes and makes the effort to implement option 1, we'll create pressure for other browsers to do it too, so I'd want to have sign-off on the idea before we do - or a decision that we should not do that, and prioritize something else instead, like a pipelining design.
Offhand, a pipelining design could be for example option 2/3, but taking a stream (or HTMLImageElement) instead.
kainino0x commentedon Jun 29, 2023
I've revived #3430 as the possible stopgap (with or without ImageData).