From a308f24e8ace0a57dec6dff719493f127a86c81a Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Tue, 6 Oct 2026 10:49:29 -0500 Subject: [PATCH 01/37] Provide code completion Complete code in appose-python scripts in two layers: * tree-sitter, in process: names in scope, script parameters, task, builtins and keywords, instantly and without any Python. * Jedi, in a small tooling environment of its own (built in the background on first use), analyzing scripts against their own environment's interpreter: members, module contents, signatures and documentation. Answers come without blocking typing, as updates; documentation is fetched on demand. Completion never builds a script's environment or disturbs its worker: environments built earlier (in any session) are found read-only. Dependencies a script declares but its environment lacks are installed for completion only, into a separate overlay directory, once they stop changing. Signature help comes from Jedi. Also extract the engine's environment resolution into ScriptEnvironment, shared with completion, and add ResidentWorkerService#find, a lookup that never replaces a running worker. Co-Authored-By: Claude Opus 5.5 --- README.md | 31 + pom.xml | 20 + .../python/ApposePythonCodeCompleter.java | 876 ++++++++++++++++++ .../python/ApposePythonScriptEngine.java | 115 +-- .../appose/python/ScriptEnvironment.java | 203 ++++ .../python/_internal/InlineMetadata.java | 28 + .../_internal/ResidentWorkerService.java | 17 + .../python/completion/DependencyOverlays.java | 178 ++++ .../appose/python/completion/JediClient.java | 288 ++++++ .../python/completion/PythonSyntax.java | 443 +++++++++ .../appose/python/completion/jedi_complete.py | 172 ++++ .../appose/python/completion/overlay.py | 127 +++ .../python/ApposePythonCodeCompleterTest.java | 583 ++++++++++++ ...ApposePythonCompletionIntegrationTest.java | 227 +++++ .../python/ApposePythonScriptEngineTest.java | 18 +- .../python/_internal/InlineMetadataTest.java | 10 + .../completion/DependencyOverlaysTest.java | 109 +++ .../python/completion/PythonSyntaxTest.java | 145 +++ 18 files changed, 3483 insertions(+), 107 deletions(-) create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediClient.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java create mode 100644 src/main/resources/org/scijava/plugins/scripting/appose/python/completion/jedi_complete.py create mode 100644 src/main/resources/org/scijava/plugins/scripting/appose/python/completion/overlay.py create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxTest.java diff --git a/README.md b/README.md index a10393a..2421f0f 100644 --- a/README.md +++ b/README.md @@ -134,3 +134,34 @@ memory) it holds, is otherwise released when the application exits. Text printed to stdout and stderr appears in the Script Editor's output panes. Errors are reported with a Python traceback that refers to the script's own line numbers. + +## Code completion + +The Script Editor completes code in these scripts, in two layers: + +* **Instantly, with no Python needed:** names in scope (variables, + functions, classes, imports, parameters of the enclosing function), the + script's `#@` parameters, `task`, builtins and keywords. This comes from + parsing the script with [tree-sitter](https://tree-sitter.github.io/). +* **With [Jedi](https://jedi.readthedocs.io/):** members of objects and + modules, function signatures (with parameter names and types) and + documentation. Jedi runs in a small environment of its own + (`appose-python-completion`: Python, Jedi and Appose), which is built in + the background the first time you complete code, and takes a while only + then. + +Once a script's environment is built, by running the script (in this session +or an earlier one), Jedi analyzes the script against it, so that the packages +installed there are known too. Packages the script declares (in its +`# /// script` block) but its environment lacks, e.g. a dependency you just +added, are installed for completion only, into a separate directory, a moment +after you stop editing the dependencies; so they are known before you run the +script. This covers PyPI packages; conda packages (`[tool.pixi.dependencies]`) +and environments declared in files are known once the script has run. +Completion never builds a script's environment itself, and never disturbs +its worker. Script parameters are known by the Python types they have at run +time, e.g. images as NumPy arrays. + +Typing never waits for Jedi: if its answer for a word is not ready within a +moment (the first answer for a package can take a few seconds), the instant +completions show, and Jedi's are used once ready. diff --git a/pom.xml b/pom.xml index 9977f03..132480f 100644 --- a/pom.xml +++ b/pom.xml @@ -94,6 +94,10 @@ 2.101.0 + + 0.1.0-SNAPSHOT + 0.26.6 + 0.25.0 @@ -106,8 +110,24 @@ org.scijava script-editor + + org.scijava + scijava-code-api + ${scijava-code-api.version} + + + + io.github.bonede + tree-sitter + ${tree-sitter.version} + + + io.github.bonede + tree-sitter-python + ${tree-sitter-python.version} + com.fasterxml.jackson.core jackson-databind diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java new file mode 100644 index 0000000..2aac6ae --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java @@ -0,0 +1,876 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CancellationException; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.function.Function; +import java.util.function.Supplier; + +import net.imglib2.RandomAccessibleInterval; + +import org.apposed.appose.Appose; +import org.apposed.appose.Environment; +import org.apposed.appose.NDArray; +import org.apposed.appose.Service; +import org.apposed.appose.util.Environments; +import org.scijava.Context; +import org.scijava.code.api.AbstractCodeCompleterPlugin; +import org.scijava.code.api.CodeCompleterPlugin; +import org.scijava.code.api.CodeCompletionService; +import org.scijava.code.api.Completion; +import org.scijava.code.api.CompletionRequest; +import org.scijava.code.api.CompletionResult; +import org.scijava.code.api.SignatureHelp; +import org.scijava.code.api.SignatureHelp.Fit; +import org.scijava.code.api.Signatures; +import org.scijava.plugin.Plugin; +import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; +import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; +import org.scijava.plugins.scripting.appose.python.completion.JediClient; +import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax; +import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Kind; +import org.scijava.script.ScriptLanguage; + +/** + * Code completion for {@link ApposePythonScriptLanguage} scripts, in two + * layers: + *
    + *
  1. {@link PythonSyntax} (tree-sitter, in process): instantly, with no + * Python needed, names in scope, the script's parameters, builtins and + * keywords.
    2. + *
  2. Jedi, in a small tooling environment of its own, built in the + * background on first use: members of objects and modules, signatures and + * documentation. Once the script's own environment is built (by running the + * script), Jedi analyzes the script against it, so its packages are known. + * Jedi's answers are used if they come within a short time budget; typing + * never waits for them.
    3. + *
+ *

+ * Completion never builds a script's environment, and never disturbs its + * worker. + *

+ * + * @author Gabriel Selzer + */ +@Plugin(type = CodeCompleterPlugin.class, name = "appose-python") +public class ApposePythonCodeCompleter extends AbstractCodeCompleterPlugin { + + /** Name of the tooling environment in which Jedi runs. */ + static final String TOOLING_ENV = "appose-python-completion"; + + /** + * How long to wait for Jedi before answering without it (and updating the + * answer once Jedi's arrives), in milliseconds. + */ + private static final long BUDGET = Long.getLong( + "scijava.appose.completion.budget", 25); + + /** How long to wait for a completion's documentation, in milliseconds. */ + private static final long DESCRIBE_BUDGET = 1000; + + /** How many completions get signatures, once a word is begun. */ + private static final int SIGNATURES = 30; + + /** How many signatures (overloads) each gets, at most. */ + private static final int OVERLOADS = 8; + + /** Python executables of script environments, by script and metadata. */ + private final Map pythons = new LinkedHashMap( + 16, 0.75f, true) + { + + @Override + protected boolean removeEldestEntry( + final Map.Entry eldest) + { + return size() > 16; + } + }; + + /** Task scripts (see jedi_complete.py, overlay.py), by resource name. */ + private static final Map SCRIPTS = new HashMap<>(); + + private JediClient jedi; + + private DependencyOverlays overlays; + + /** + * How long a script's dependencies must stay unchanged before packages + * missing from its environment are installed for completion, in ms. + */ + private static final long OVERLAY_SETTLE = 1500; + + @Override + public boolean supports(final ScriptLanguage language) { + return language instanceof ApposePythonScriptLanguage; + } + + @Override + public CompletionResult complete(final CompletionRequest request) { + final String text = request.text(); + final PythonSyntax.Context ctx = PythonSyntax.analyze(text, request + .offset()); + if (ctx.kind == Kind.NONE) return CompletionResult.EMPTY; + if (ctx.kind == Kind.NAME && ctx.prefix.isEmpty()) { + return CompletionResult.EMPTY; + } + + final Map params = parameterTypes(text); + final List local = localCompletions(ctx, params); + final CompletableFuture> answer = jediCompletions(request, + ctx, params); + if (answer == null) return merge(ctx, null, local); + + // Use Jedi's answer if it is (almost) ready; otherwise answer without it + // at once, and announce the better result to follow. + try { + return merge(ctx, answer.get(BUDGET, TimeUnit.MILLISECONDS), local); + } + catch (final TimeoutException exc) { + return merge(ctx, null, local).withUpdate(answer.handle((jedi, + error) -> jedi == null ? null : merge(ctx, jedi, local))); + } + catch (final InterruptedException exc) { + Thread.currentThread().interrupt(); + } + catch (final ExecutionException | CancellationException exc) { + // NB: Jedi failed; the next request asks again. + } + return merge(ctx, null, local); + } + + @Override + public void prepare(final CompletionRequest request) { + // Warm up: start Jedi (building its environment, if needed), and have it + // analyze the script against its environment, in the background. + final JediClient client = jediClient(); + if (client == null) return; + final String text = request.text(); + final String python = scriptPython(text, request.path()); + final String overlay = overlay(text, request.path(), python, true); + final String source = preamble(parameterTypes(text)) + text; + final int line = count(source, '\n') + 1; + final int column = source.length() - source.lastIndexOf('\n') - 1; + client.answer("prepare\n" + request.path() + "\n" + text, + new JediClient.Query(source, line, column, request.path(), python, 0, + overlay)); + } + + @Override + public SignatureHelp signatureHelp(final CompletionRequest request) { + final String text = request.text(); + final int caret = request.offset(); + if (PythonSyntax.analyze(text, caret).kind == Kind.NONE) { + return SignatureHelp.NONE; // NB: In a comment or string. + } + final int open = Signatures.callStart(text, caret); + if (open < 0) return SignatureHelp.NONE; + final List args = Signatures.arguments(text, open, caret); + if (args == null) return SignatureHelp.NONE; + final JediClient client = jediClient(); + if (client == null) return SignatureHelp.NONE; + + final String python = scriptPython(text, request.path()); + final String overlay = overlay(text, request.path(), python, false); + final String preamble = preamble(parameterTypes(text)); + final String before = preamble + text.substring(0, caret); + final int line = count(before, '\n') + 1; + final int column = before.length() - before.lastIndexOf('\n') - 1; + final JediClient.Query query = new JediClient.Query(preamble + text, line, + column, request.path(), python, 0, overlay); + final int active = args.size() - 1; + // NB: Where Jedi cannot tell the argument, count the commas. + final CompletableFuture help = client.signatures( + "signatures\n" + python + "\n" + overlay + "\n" + request.path() + + "\n" + before, query).handle((h, error) -> h == null ? null + : new SignatureHelp(h.signatures(), h.activeParameter() >= 0 ? h + .activeParameter() : active, open)); + try { + final SignatureHelp h = help.get(BUDGET, TimeUnit.MILLISECONDS); + return h == null ? SignatureHelp.NONE : h; + } + catch (final TimeoutException exc) { + return SignatureHelp.NONE.withUpdate(help); + } + catch (final InterruptedException exc) { + Thread.currentThread().interrupt(); + } + catch (final ExecutionException | CancellationException exc) { + // NB: Jedi failed. + } + return SignatureHelp.NONE; + } + + /** + * Adapts the "signatures" task output to the shape of "completions" items + * (one item per signature), for {@link #toCompletions}. + */ + @SuppressWarnings("unchecked") + private static List signatureItems(final Object output) { + final List items = new ArrayList<>(); + if (!(output instanceof List)) return items; + for (final Object o : (List) output) { + if (!(o instanceof Map)) continue; + final Map sig = (Map) o; + final Map item = new HashMap<>(); + item.put("name", sig.get("name")); + item.put("type", "function"); + item.put("signatures", Collections.singletonList(sig)); + items.add(item); + } + return items; + } + + /** + * Merges Jedi's completions (if any) with the local ones: Jedi's first, + * since they carry types, signatures and documentation; then whatever only + * the local analysis knows. + */ + private static CompletionResult merge(final PythonSyntax.Context ctx, + final List jedi, final List local) + { + final List out = new ArrayList<>(); + final Set seen = new HashSet<>(); + if (jedi != null) { + for (final Completion c : jedi) { + if (!matches(c.insertionText(), ctx.prefix)) continue; + if (ctx.kind == Kind.IMPORT) { + // NB: An import names things rather than calling them: one plain + // name per function or class, without its parameters. + if (seen.add(c.insertionText())) out.add(name(c)); + continue; + } + out.add(c); + seen.add(c.insertionText()); + } + } + for (final Completion c : local) { + if (seen.add(c.insertionText())) out.add(c); + } + return new CompletionResult(out, ctx.replaceStart); + } + + /** Sets the dependency overlays, e.g. to use fake ones in tests. */ + synchronized void setOverlays(final DependencyOverlays overlays) { + this.overlays = overlays; + } + + /** Sets the Jedi client, e.g. to use a fake one in tests. */ + synchronized void setJediClient(final JediClient jedi) { + this.jedi = jedi; + } + + // -- Helper methods -- + + /** + * A completion inserting just the name of {@code c}: not callable, so the + * editor inserts no parameter list for it. + */ + static Completion name(final Completion c) { + final Completion.Kind kind = c.kind() == Completion.Kind.METHOD + ? Completion.Kind.OTHER : c.kind(); + return Completion.builder(c.insertionText()).kind(kind).summary(c + .summary()).lazyDescription(c::description).build(); + } + + /** Completions from the local (syntactic) analysis alone. */ + private static List localCompletions( + final PythonSyntax.Context ctx, final Map params) + { + final List out = new ArrayList<>(); + final String prefix = ctx.prefix; + if (ctx.kind == Kind.ATTRIBUTE) { + for (final String name : ctx.attributesUsed) { + if (matches(name, prefix)) out.add(Completion.builder(name).kind( + Completion.Kind.OTHER).summary("Used elsewhere in this script") + .build()); + } + return out; + } + if (ctx.kind != Kind.NAME) return out; + for (final String name : ctx.names) { + if (matches(name, prefix)) out.add(Completion.builder(name).kind( + Completion.Kind.VARIABLE).build()); + } + for (final Map.Entry p : params.entrySet()) { + if (matches(p.getKey(), prefix)) out.add(Completion.builder(p.getKey()) + .kind(Completion.Kind.VARIABLE).summary(p.getValue() == null ? "Script parameter" + : p.getValue() + " (script parameter)").build()); + } + if (matches("task", prefix)) out.add(Completion.builder("task").kind( + Completion.Kind.VARIABLE).summary("appose.python_worker.Task").build()); + for (final String name : PythonSyntax.BUILTINS) { + if (matches(name, prefix)) out.add(Completion.builder(name).kind( + Completion.Kind.OTHER).summary("builtin").build()); + } + for (final String keyword : PythonSyntax.KEYWORDS) { + if (matches(keyword, prefix)) out.add(Completion.builder(keyword).kind( + Completion.Kind.KEYWORD).build()); + } + return out; + } + + /** + * Asks Jedi for completions. + * + * @return Jedi's answer, or pending answer; or null if Jedi is unavailable. + */ + private CompletableFuture> jediCompletions( + final CompletionRequest request, final PythonSyntax.Context ctx, + final Map params) + { + final JediClient client = jediClient(); + if (client == null) return null; + final String text = request.text(); + final String python = scriptPython(text, request.path()); + final String overlay = overlay(text, request.path(), python, false); + + // Ask about the first character typed, if any: later keystrokes of the + // same word then reuse the answer (filtered), while it stays short. + final int caret = ctx.replaceStart + Math.min(1, ctx.prefix.length()); + final String preamble = preamble(params); + final String before = preamble + text.substring(0, caret); + final int line = count(before, '\n') + 1; + final int column = before.length() - before.lastIndexOf('\n') - 1; + // NB: Signatures are expensive, so ask for them only once a word is + // begun, which narrows the completions down. Before that, functions are + // offered by name. + final JediClient.Query query = new JediClient.Query(preamble + text, line, + column, request.path(), python, ctx.prefix.isEmpty() ? 0 : SIGNATURES, + overlay); + final String key = python + "\n" + overlay + "\n" + request.path() + + "\n" + before; + return client.answer(key, query); + } + + /** + * Gets the overlay of packages the script declares but its environment + * lacks (see {@link DependencyOverlays}), if ready; starts preparing it + * otherwise. Only for environments declared inline, for now. + * + * @param settled Whether the dependencies are known not to be changing. + * @return The overlay directory, or null if none is needed or ready. + */ + private String overlay(final String text, final String path, + final String python, final boolean settled) + { + final DependencyOverlays overlays = overlays(); + if (overlays == null) return null; + final List dependencies; + final String name; + try { + final String metadata = InlineMetadata.extract(text); + if (metadata == null) return null; + dependencies = InlineMetadata.dependencies(metadata); + // NB: Unsaved scripts' environments are named after their metadata, + // which changes as dependencies are edited: share one overlay. + name = path == null ? "unsaved" : ScriptEnvironment.inlineName(metadata, + path); + } + catch (final RuntimeException exc) { + return null; // NB: E.g. half-typed metadata. + } + return overlays.overlay(name, python, dependencies, settled); + } + + /** Gets the dependency overlays, creating them on first use. */ + private synchronized DependencyOverlays overlays() { + if (overlays != null) return overlays; + final ResidentWorkerService workers = service( + ResidentWorkerService.class); + if (workers == null) return null; + overlays = new DependencyOverlays((python, dependencies, + dir) -> installOverlay(workers, python, dependencies, dir), new File( + Environments.apposeEnvsDir(), TOOLING_ENV + "-overlays"), + OVERLAY_SETTLE, System::currentTimeMillis); + return overlays; + } + + /** + * Installs an overlay by running overlay.py on the tooling environment's + * worker. Called on the overlays' background thread. + */ + private static String installOverlay(final ResidentWorkerService workers, + final String python, final List dependencies, final File dir) + { + final Map inputs = new HashMap<>(); + inputs.put("python", python); + inputs.put("dependencies", dependencies); + inputs.put("overlay", dir.getAbsolutePath()); + try { + final Service.Task task = toolingWorker(workers).task(resource( + "completion/overlay.py"), inputs); + task.waitFor(); + final Object overlay = task.outputs.get("path"); + return overlay instanceof String ? (String) overlay : null; + } + catch (final Exception exc) { + throw new IllegalStateException("Overlay installation failed", exc); + } + } + + /** + * A line declaring the types of the script's parameters, for Jedi, e.g. + * {@code import numpy; image: numpy.ndarray; sigma: float}. + */ + static String preamble(final Map params) { + final StringBuilder sb = new StringBuilder( + "import appose.python_worker; task: appose.python_worker.Task"); + if (params.containsValue("numpy.ndarray")) sb.append("; import numpy"); + for (final Map.Entry p : params.entrySet()) { + if (p.getValue() == null) continue; + sb.append("; ").append(p.getKey()).append(": ").append(p.getValue()); + } + return sb.append('\n').toString(); + } + + /** + * Gets the script's parameters, with the Python types they have when the + * script runs (see {@link ApposePythonScriptEngine#marshal}), or null types + * if unknown. + */ + private Map parameterTypes(final String text) { + final Map params = new LinkedHashMap<>(); + final Context context = getContext(); + final CodeCompletionService service = context == null ? null : context + .getService(CodeCompletionService.class); + if (service == null) return params; + service.scriptParameters(text).forEach((name, type) -> params.put(name, + pythonType(type))); + return params; + } + + /** The Python type a Java value of the given type becomes in a script. */ + static String pythonType(final Class type) { + if (type == null) return null; + if (RandomAccessibleInterval.class.isAssignableFrom(type) || NDArray.class + .isAssignableFrom(type)) return "numpy.ndarray"; + if (type == boolean.class || type == Boolean.class) return "bool"; + if (type == int.class || type == long.class || type == short.class || + type == byte.class || type == Integer.class || type == Long.class || + type == Short.class || type == Byte.class || type == BigInteger.class) + { + return "int"; + } + if (type == float.class || type == double.class || type == Float.class || + type == Double.class || type == BigDecimal.class) return "float"; + if (CharSequence.class.isAssignableFrom(type) || type == char.class || + type == Character.class || type.isEnum() || type == File.class) + { + return "str"; + } + if (type.isArray() || Collection.class.isAssignableFrom(type)) { + return "list"; + } + if (Map.class.isAssignableFrom(type)) return "dict"; + return null; + } + + /** + * Gets the Python executable of the script's environment, if it has been + * built already: in this session (e.g. by running the script), or in an + * earlier one. Completion never builds it. Returns null if not built. + */ + private String scriptPython(final String text, final String path) { + // NB: Called on every keystroke, so remember what was found (but not + // what was not: the environment may be built at any moment). + final String key; + try { + key = path + "\n" + ScriptEnvironment.directive(text) + "\n" + + InlineMetadata.extract(text); + } + catch (final RuntimeException exc) { + return null; // NB: E.g. half-typed metadata. + } + synchronized (pythons) { + final String known = pythons.get(key); + if (known != null && new File(known).isFile()) return known; + } + final String python = findScriptPython(text, path); + if (python != null) { + synchronized (pythons) { + pythons.put(key, python); + } + } + return python; + } + + /** As {@link #scriptPython}, without remembering. */ + private String findScriptPython(final String text, final String path) { + final ScriptEnvironment env; + try { + final Map directive = ScriptEnvironment.directive(text); + env = ScriptEnvironment.of(text, path, directive.get("env"), directive + .get("scheme")); + } + catch (final IOException | RuntimeException exc) { + return null; // NB: E.g. half-typed metadata. + } + if (env == null) return null; + // NB: A read-only lookup: worker(...) would replace the worker whenever + // the configuration changed, i.e. as the user edits it. + final ResidentWorkerService workers = service( + ResidentWorkerService.class); + final ResidentWorker worker = workers == null ? null : workers.find( + env.name, env.config()); + if (worker != null && worker.environment().isBuilt()) { + try { + final String python = pythonExecutable(worker.environment().get()); + if (python != null) return python; + } + catch (final Exception exc) { + // NB: Fall back to looking on disk. + } + } + // Built in an earlier session? Its files are where Appose keeps them. + return pythonOnDisk(new File(Environments.apposeEnvsDir(), env.name)); + } + + /** Finds the Python executable of a built environment, or null. */ + static String pythonExecutable(final Environment env) { + // NB: Look in the bin paths, and in their parents: on Windows, conda-style + // (e.g. pixi) environments keep python.exe in the environment's root, + // above the bin directory that Appose reports. + final List dirs = new ArrayList<>(); + for (final String dir : env.binPaths()) { + dirs.add(new File(dir)); + final File parent = new File(dir).getParentFile(); + if (parent != null) dirs.add(parent); + } + if (env.base() != null) { + final File base = new File(env.base()); + dirs.add(base); + dirs.add(new File(base, "Scripts")); + dirs.add(new File(base, "bin")); + } + return findPython(dirs); + } + + /** + * Finds the Python executable of an environment in the given directory, as + * Appose builds them (pixi, uv or conda), without building or starting + * anything; or null if there is none. + */ + static String pythonOnDisk(final File envDir) { + if (!envDir.isDirectory()) return null; + final List dirs = new ArrayList<>(); + // pixi: .pixi/envs/, with "default" first. + final File pixiEnvs = new File(envDir, ".pixi" + File.separator + "envs"); + final List pixi = new ArrayList<>(); + pixi.add(new File(pixiEnvs, "default")); + final File[] others = pixiEnvs.listFiles(File::isDirectory); + if (others != null) pixi.addAll(Arrays.asList(others)); + for (final File p : pixi) { + dirs.add(p); + dirs.add(new File(p, "bin")); + } + // uv (a virtual environment) and conda/mamba. + for (final File base : new File[] { envDir, new File(envDir, ".venv") }) { + dirs.add(new File(base, "Scripts")); + dirs.add(new File(base, "bin")); + dirs.add(base); + } + return findPython(dirs); + } + + private static String findPython(final List dirs) { + for (final File dir : dirs) { + for (final String exe : new String[] { "python.exe", "python", + "python3" }) + { + final File f = new File(dir, exe); + if (f.isFile()) return f.getAbsolutePath(); + } + } + return null; + } + + /** Gets the Jedi client, creating it on first use. */ + private synchronized JediClient jediClient() { + if (jedi != null) return jedi; + final ResidentWorkerService workers = service( + ResidentWorkerService.class); + if (workers == null) return null; + final JediClient[] client = { null }; + client[0] = new JediClient(new JediClient.Backend() { + + @Override + public List complete(final String key, + final JediClient.Query query) + { + final Object output = runJedi(workers, "complete", key, query, null) + .get("completions"); + // NB: Documentation is fetched only for completions looked at. + return toCompletions(output, name -> () -> client[0].describe(key, + query, name, DESCRIBE_BUDGET)); + } + + @Override + public SignatureHelp signatures(final String key, + final JediClient.Query query) + { + final Map outputs = runJedi(workers, "signatures", key, + query, null); + final List signatures = new ArrayList<>(); + for (final Completion c : toCompletions(signatureItems(outputs.get( + "signatures")))) + { + // NB: Python's conversions are not rated yet: Jedi's order. + signatures.add(new SignatureHelp.Signature(c, Fit.UNKNOWN)); + } + final Object index = outputs.get("index"); + return new SignatureHelp(signatures, index instanceof Number + ? ((Number) index).intValue() : -1, -1); + } + + @Override + public String describe(final String key, final JediClient.Query query, + final String name) + { + final Object doc = runJedi(workers, "describe", key, query, name).get( + "docstring"); + return doc instanceof String && !((String) doc).isEmpty() ? "
" +
+					escapeHTML((String) doc) + "
" : null; + } + }); + jedi = client[0]; + return jedi; + } + + /** + * Runs a Jedi task (see jedi_complete.py) on the tooling environment's + * worker, building the environment first, if needed. Called on the Jedi + * client's background thread. + * + * @return The task's outputs. + */ + private static Map runJedi( + final ResidentWorkerService workers, final String op, final String key, + final JediClient.Query query, final String name) + { + final ResidentWorker worker = toolingWorker(workers); + final Map inputs = new HashMap<>(); + inputs.put("op", op); + inputs.put("key", key); + inputs.put("source", query.source); + inputs.put("line", query.line); + inputs.put("column", query.column); + inputs.put("path", query.path); + inputs.put("python", query.python); + inputs.put("max_signatures", query.signatures); + inputs.put("max_overloads", OVERLOADS); + inputs.put("name", name); + inputs.put("overlay", query.overlay); + try { + final Service.Task task = worker.task(resource( + "completion/jedi_complete.py"), inputs); + task.waitFor(); + return task.outputs; + } + catch (final Exception exc) { + throw new IllegalStateException("Jedi failed", exc); + } + } + + /** Gets the tooling environment's worker, building it first if needed. */ + private static ResidentWorker toolingWorker( + final ResidentWorkerService workers) + { + final String config = toolingConfig(); + return workers.worker(TOOLING_ENV, config, () -> Appose.content(config) + .scheme("pixi.toml"), Environment::python); + } + + /** + * The tooling environment: Python, Jedi and Appose; and uv, to install + * packages for {@link DependencyOverlays}. + */ + static String toolingConfig() { + return "[workspace]\n" + // + "name = \"" + TOOLING_ENV + "\"\n" + // + "channels = [\"conda-forge\"]\n" + // + "platforms = [\"" + InlineMetadata.condaPlatform() + "\"]\n" + // + "\n" + // + "[dependencies]\n" + // + "python = \"3.12.*\"\n" + // + "jedi = \">=0.19.2\"\n" + // + "\n" + // + "[pypi-dependencies]\n" + // + "appose = \"*\"\n" + // + "uv = \"*\"\n"; + } + + /** Converts the task's output (see jedi_complete.py) to completions. */ + static List toCompletions(final Object output) { + return toCompletions(output, null); + } + + /** + * Converts the task's output (see jedi_complete.py) to completions. + * + * @param docs Supplies documentation, by completion name, on demand; or + * null if there is none. + */ + @SuppressWarnings("unchecked") + static List toCompletions(final Object output, + final Function> docs) + { + if (!(output instanceof List)) return Collections.emptyList(); + final List out = new ArrayList<>(); + for (final Object o : (List) output) { + if (!(o instanceof Map)) continue; + final Map item = (Map) o; + final String name = (String) item.get("name"); + if (name == null) continue; + final String type = (String) item.get("type"); + final String doc = (String) item.get("docstring"); + final Supplier description = doc != null ? () -> "
" +
+				escapeHTML(doc) + "
" : docs != null ? docs.apply(name) : null; + final List signatures = item.get( + "signatures") instanceof List ? (List) item.get("signatures") + : Collections.emptyList(); + if (signatures.isEmpty()) { + // NB: Without a signature, even a function is offered by name only. + final Completion.Kind kind = kind(type); + out.add(Completion.builder(name).kind(kind == Completion.Kind.METHOD + ? Completion.Kind.OTHER : kind).summary((String) item.get( + "description")).lazyDescription(description).build()); + continue; + } + // One completion per signature, e.g. for each typing.overload. + for (final Object s : signatures) { + final Map sig = (Map) s; + final List params = new ArrayList<>(); + for (final Object p : (List) sig.get("params")) { + final List param = (List) p; + params.add(new Completion.Parameter((String) param.get(0), + (String) param.get(1))); + } + out.add(Completion.builder(name).kind(Completion.Kind.METHOD) + .parameters(params).returnType((String) sig.get("returns")).summary( + (String) item.get("description")).lazyDescription(description) + .build()); + } + } + return out; + } + + private static Completion.Kind kind(final String jediType) { + if (jediType == null) return Completion.Kind.OTHER; + switch (jediType) { + case "function": + return Completion.Kind.METHOD; + case "class": + return Completion.Kind.CLASS; + case "module": + return Completion.Kind.PACKAGE; + case "property": + return Completion.Kind.FIELD; + case "keyword": + return Completion.Kind.KEYWORD; + case "instance": + case "statement": + case "param": + return Completion.Kind.VARIABLE; + default: + return Completion.Kind.OTHER; + } + } + + /** Gets a task script (e.g. jedi_complete.py), by resource name. */ + private static String resource(final String name) throws IOException { + synchronized (SCRIPTS) { + final String known = SCRIPTS.get(name); + if (known != null) return known; + try (final InputStream in = ApposePythonCodeCompleter.class + .getResourceAsStream(name)) + { + if (in == null) throw new IOException(name + " not found"); + final java.io.ByteArrayOutputStream out = + new java.io.ByteArrayOutputStream(); + final byte[] buf = new byte[8192]; + int n; + while ((n = in.read(buf)) > 0) out.write(buf, 0, n); + final String script = new String(out.toByteArray(), + StandardCharsets.UTF_8); + SCRIPTS.put(name, script); + return script; + } + } + } + + private S service( + final Class type) + { + final Context context = getContext(); + return context == null ? null : context.getService(type); + } + + private static boolean matches(final String name, final String prefix) { + if (name.startsWith("__") && !prefix.startsWith("_")) return false; + return name.toLowerCase(Locale.ROOT).startsWith(prefix.toLowerCase( + Locale.ROOT)) && !name.equals(prefix); + } + + private static int count(final String s, final char c) { + int n = 0; + for (int i = 0; i < s.length(); i++) { + if (s.charAt(i) == c) n++; + } + return n; + } + + private static String escapeHTML(final String s) { + return s.replace("&", "&").replace("<", "<").replace(">", ">"); + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngine.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngine.java index afc02f2..e5fb180 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngine.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngine.java @@ -37,7 +37,6 @@ import java.io.UncheckedIOException; import java.io.Writer; import java.nio.charset.StandardCharsets; -import java.nio.file.Files; import java.util.ArrayList; import java.util.Collection; import java.util.HashMap; @@ -58,7 +57,7 @@ import org.apposed.appose.Appose; import org.apposed.appose.BuildException; -import org.apposed.appose.Builder; +import org.apposed.appose.Environment; import org.apposed.appose.NDArray; import org.apposed.appose.Service; import org.apposed.appose.Service.TaskStatus; @@ -70,7 +69,6 @@ import org.scijava.log.Logger; import org.scijava.module.ModuleItem; import org.scijava.plugin.Parameter; -import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; import org.scijava.plugins.scripting.appose.python._internal.SciJavaTasks; @@ -440,112 +438,33 @@ static boolean ownsNDArray(final Object value, final NDArray nd) { } /** - * Gets the resident worker for the script's Appose environment, which the - * script declares either inline, via a PEP 723 {@code # /// script} - * metadata block, or in a file named by the {@code env} (and optionally - * {@code scheme}) attributes of its {@code #@script} directive. The - * environment itself is built lazily, by the worker's first task. + * Gets the resident worker for the script's Appose environment (see + * {@link ScriptEnvironment}). The environment itself is built lazily, by + * the worker's first task. */ private ResidentWorker worker(final ScriptInfo info, final String script) throws ScriptException { if (info == null) throw noEnvironment(); - final String envRef = info.get("env"); - final String metadata; + final ScriptEnvironment env; try { - metadata = InlineMetadata.extract(script); + env = ScriptEnvironment.of(script, info.getPath(), info.get("env"), info + .get("scheme")); } - catch (final IllegalArgumentException e) { + catch (final IllegalArgumentException | IOException e) { throw scriptException(e.getMessage(), e); } - if (metadata != null) { - if (envRef != null) { - throw new ScriptException("The script declares its environment " + - "twice: inline, and via #@script(env=\"" + envRef + "\"). " + - "Remove one of them."); - } - return inlineWorker(metadata, info.getPath()); - } - if (envRef == null) throw noEnvironment(); - - final File envFile = resolveEnvFile(envRef, info.getPath()); - if (!envFile.isFile()) { - throw new ScriptException("Appose environment file not found: " + - envFile.getAbsolutePath()); - } - final String content; - try { - content = new String(Files.readAllBytes(envFile.toPath()), - StandardCharsets.UTF_8); - } - catch (final IOException e) { - throw scriptException("Cannot read Appose environment file: " + - envFile.getAbsolutePath(), e); - } - return worker(envName(envFile), content, info.get("scheme")); + if (env == null) throw noEnvironment(); + return workerService.worker(env.name, env.config(), env::builder, + ApposePythonScriptEngine::launch); } - /** Gets the resident worker for an environment declared inline. */ - private ResidentWorker inlineWorker(final String metadata, - final String scriptPath) throws ScriptException - { - // Note: Name the environment after the script, as for environment - // files, so that editing the metadata updates the environment and - // replaces its worker, rather than leaving the old one running. - // Unsaved scripts, e.g. new from a template, have no path to go by. - final String envName = scriptPath == null ? "script-" + String.format( - "%08x", metadata.hashCode()) : envName(new File(scriptPath)); - final String content; - try { - content = InlineMetadata.toPyProject(metadata, envName); - } - catch (final IllegalArgumentException e) { - throw scriptException(e.getMessage(), e); - } - return worker(envName, content, InlineMetadata.SCHEME); - } - - /** - * Gets the resident worker for the named environment, built from the given - * configuration. - */ - private ResidentWorker worker(final String envName, final String content, - final String scheme) - { - return workerService.worker(envName, scheme + "\n" + content, () -> { - final Builder builder = Appose.content(content); - return scheme == null ? builder : builder.scheme(scheme); - }, - // Note: On Windows, importing numpy from a task hangs unless numpy - // was imported during worker initialization. - env -> env.python().init( - "try:\n import numpy\nexcept ImportError:\n pass\n")); - } - - /** Resolves an (optionally relative) env file path against the script path. */ - static File resolveEnvFile(final String envRef, final String scriptPath) { - final File envFile = new File(envRef); - if (envFile.isAbsolute()) return envFile; - if (scriptPath != null) { - final File parent = new File(scriptPath).getAbsoluteFile().getParentFile(); - if (parent != null) return new File(parent, envRef); - } - return envFile; - } - - /** - * Derives a valid Appose environment name from an environment file: its - * base name, plus a hash of its absolute path to avoid collisions between - * like-named files in different directories. E.g.: - * {@code /path/to/cellcast.toml} becomes {@code cellcast-1a2b3c4d}. - */ - static String envName(final File envFile) { - final String fileName = envFile.getName(); - final int dot = fileName.lastIndexOf('.'); - final String baseName = dot > 0 ? fileName.substring(0, dot) : fileName; - final String path = envFile.getAbsoluteFile().toPath().normalize().toString(); - return baseName.replaceAll("[^a-zA-Z0-9_-]", "_") + "-" + - String.format("%08x", path.hashCode()); + /** Creates the service of a worker in the given (built) environment. */ + static Service launch(final Environment env) { + // Note: On Windows, importing numpy from a task hangs unless numpy + // was imported during worker initialization. + return env.python().init( + "try:\n import numpy\nexcept ImportError:\n pass\n"); } private static ScriptException noEnvironment() { diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java new file mode 100644 index 0000000..5c13d83 --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java @@ -0,0 +1,203 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.HashMap; +import java.util.Map; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.apposed.appose.Appose; +import org.apposed.appose.Builder; +import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; + +/** + * The Appose environment a script declares: either inline, via a PEP 723 + * {@code # /// script} metadata block, or in a file named by the {@code env} + * (and optionally {@code scheme}) attributes of its {@code #@script} + * directive. + *

+ * Both running a script ({@link ApposePythonScriptEngine}) and completing code + * in it ({@link ApposePythonCodeCompleter}) resolve its environment here, so + * that they agree on the environment's name and configuration. + *

+ */ +final class ScriptEnvironment { + + /** The {@code #@script(...)} directive. */ + private static final Pattern SCRIPT_DIRECTIVE = Pattern.compile( + "^#@\\s*script\\s*\\((.*)\\)\\s*$", Pattern.MULTILINE); + + /** An attribute of a directive: key="value", key='value' or key=value. */ + private static final Pattern ATTRIBUTE = Pattern.compile( + "(\\w+)\\s*=\\s*(?:\"([^\"]*)\"|'([^']*)'|([^,\\s)]+))"); + + /** Name of the environment. */ + final String name; + + /** The environment's configuration, e.g. the contents of its file. */ + final String content; + + /** The configuration's scheme (e.g. {@code pixi.toml}), or null if implied. */ + final String scheme; + + private ScriptEnvironment(final String name, final String content, + final String scheme) + { + this.name = name; + this.content = content; + this.scheme = scheme; + } + + /** + * Resolves the environment declared by a script. + * + * @param script The script's source code. + * @param scriptPath The script's file path, or null if unsaved. + * @param envRef The {@code env} attribute of its {@code #@script} directive, + * or null. + * @param scheme The {@code scheme} attribute of its {@code #@script} + * directive, or null. + * @return The environment, or null if the script declares none. + * @throws IllegalArgumentException If the declaration is invalid, e.g. + * malformed, given twice, or naming a missing file. + * @throws IOException If the environment file cannot be read. + */ + static ScriptEnvironment of(final String script, final String scriptPath, + final String envRef, final String scheme) throws IOException + { + final String metadata = InlineMetadata.extract(script); + if (metadata != null) { + if (envRef != null) { + throw new IllegalArgumentException("The script declares its " + + "environment twice: inline, and via #@script(env=\"" + envRef + + "\"). Remove one of them."); + } + final String name = inlineName(metadata, scriptPath); + return new ScriptEnvironment(name, InlineMetadata.toPyProject(metadata, + name), InlineMetadata.SCHEME); + } + if (envRef == null) return null; + + final File envFile = resolveEnvFile(envRef, scriptPath); + if (!envFile.isFile()) { + throw new IllegalArgumentException("Appose environment file not " + + "found: " + envFile.getAbsolutePath()); + } + final String content; + try { + content = new String(Files.readAllBytes(envFile.toPath()), + StandardCharsets.UTF_8); + } + catch (final IOException e) { + throw new IOException("Cannot read Appose environment file: " + envFile + .getAbsolutePath(), e); + } + return new ScriptEnvironment(envName(envFile), content, scheme); + } + + /** + * Gets the name of the environment a script declares inline. + * + * @param metadata The script's metadata block (see + * {@link InlineMetadata#extract}). + * @param scriptPath The script's file path, or null if unsaved. + */ + static String inlineName(final String metadata, final String scriptPath) { + // Note: Name the environment after the script, as for environment + // files, so that editing the metadata updates the environment and + // replaces its worker, rather than leaving the old one running. + // Unsaved scripts, e.g. new from a template, have no path to go by. + return scriptPath == null ? "script-" + String.format("%08x", metadata + .hashCode()) : envName(new File(scriptPath)); + } + + /** + * Gets the attributes of a script's {@code #@script(...)} directive, e.g. + * {@code env} and {@code scheme}, without fully parsing its parameters. + */ + static Map directive(final String script) { + final Map attrs = new HashMap<>(); + final Matcher m = SCRIPT_DIRECTIVE.matcher(script); + if (!m.find()) return attrs; + final Matcher a = ATTRIBUTE.matcher(m.group(1)); + while (a.find()) { + final String value = a.group(2) != null ? a.group(2) : a.group(3) != null + ? a.group(3) : a.group(4); + attrs.put(a.group(1), value); + } + return attrs; + } + + /** + * The key under which {@link + * org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService} + * keeps the environment: changes to it replace the environment's worker. + */ + String config() { + return scheme + "\n" + content; + } + + /** Creates a builder for the environment. */ + Builder builder() { + final Builder builder = Appose.content(content); + return scheme == null ? builder : builder.scheme(scheme); + } + + /** Resolves an (optionally relative) env file path against the script path. */ + static File resolveEnvFile(final String envRef, final String scriptPath) { + final File envFile = new File(envRef); + if (envFile.isAbsolute()) return envFile; + if (scriptPath != null) { + final File parent = new File(scriptPath).getAbsoluteFile().getParentFile(); + if (parent != null) return new File(parent, envRef); + } + return envFile; + } + + /** + * Derives a valid Appose environment name from an environment file: its + * base name, plus a hash of its absolute path to avoid collisions between + * like-named files in different directories. E.g.: + * {@code /path/to/cellcast.toml} becomes {@code cellcast-1a2b3c4d}. + */ + static String envName(final File envFile) { + final String fileName = envFile.getName(); + final int dot = fileName.lastIndexOf('.'); + final String baseName = dot > 0 ? fileName.substring(0, dot) : fileName; + final String path = envFile.getAbsoluteFile().toPath().normalize().toString(); + return baseName.replaceAll("[^a-zA-Z0-9_-]", "_") + "-" + + String.format("%08x", path.hashCode()); + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/InlineMetadata.java b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/InlineMetadata.java index 51dc690..38e9b0f 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/InlineMetadata.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/InlineMetadata.java @@ -110,6 +110,34 @@ public static String extract(final String script) { return content; } + /** + * Gets the {@code dependencies} (PEP 508 requirement strings, e.g. + * {@code "numpy > 2"}) of a {@code script} metadata block. + * + * @param metadata TOML content of the metadata block. + * @return The dependencies, in declaration order; empty if none. + * @throws IllegalArgumentException If the metadata is not valid. + */ + public static List dependencies(final String metadata) { + final JsonNode parsed; + try { + parsed = new TomlMapper().readTree(metadata); + } + catch (final IOException e) { + throw new IllegalArgumentException( + "Invalid TOML in script metadata: " + e.getMessage(), e); + } + final JsonNode deps = parsed == null ? null : parsed.get("dependencies"); + if (deps == null) return Collections.emptyList(); + if (!deps.isArray()) throw invalid("dependencies must be an array"); + final List out = new ArrayList<>(); + for (final JsonNode dep : deps) { + if (!dep.isTextual()) throw invalid("dependencies must be strings"); + out.add(dep.asText()); + } + return out; + } + /** * Translates the TOML content of a {@code script} metadata block into an * equivalent {@code pyproject.toml}: diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java index fd85316..d3572ca 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java @@ -109,6 +109,23 @@ public ResidentWorker worker(final String name, final String config, return worker; } + /** + * Gets the resident worker for the named environment, if there is one with + * the given configuration. Unlike {@link #worker}, this never creates, + * replaces or releases anything, so it suits callers that must not disturb + * running workers, such as code completion while the user edits the + * environment's configuration. + * + * @return The worker, or null if there is none with that configuration. + */ + public ResidentWorker find(final String name, final String config) { + synchronized (entries) { + final Entry entry = entries.get(name); + return entry != null && entry.config.equals(config) ? entry.worker + : null; + } + } + /** Gets the current resident workers. */ public List workers() { final List list = new ArrayList<>(); diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java new file mode 100644 index 0000000..b5c6ecc --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java @@ -0,0 +1,178 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import java.io.File; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.function.LongSupplier; + +/** + * Makes packages a script declares, but its environment lacks, known to code + * completion before the environment is rebuilt (by running the script): e.g. + * a dependency just added to the script's {@code # /// script} block. + *

+ * The missing packages are installed, in the background, into a separate + * directory per script environment (an "overlay"), which Jedi adds to its + * search path. The environment itself is never touched, so this never + * disturbs its resident worker. Installs start only once the dependencies + * have stopped changing for a moment, so that half-typed names do not cause + * installs; and only if the missing packages differ from the overlay's. + *

+ *

+ * Alternatives considered, which may be worth revisiting: + *

+ *
    + *
  • Rebuilding the script's environment as its dependencies + * change. Jedi would then see exactly what will run, but builds can take + * minutes and gigabytes, half-finished edits would trigger builds, and Appose + * rebuilds environments in place, which can interfere with (or, by changing + * its configuration, release) a resident worker using it.
    • + *
  • Installing nothing, but hinting that new dependencies are + * known only once the script has run. Cheap and honest, but the user must + * run the script first. It may still be worthwhile for dependencies an + * overlay cannot provide: conda packages (e.g. + * {@code [tool.pixi.dependencies]}), and environments declared in files via + * {@code #@script(env=...)}, which overlays do not handle yet.
    • + *
+ * + * @author Gabriel Selzer + */ +public final class DependencyOverlays { + + /** Installs the missing packages of a script's environment. */ + @FunctionalInterface + public interface Installer { + + /** + * Installs the packages among {@code dependencies} that the environment + * with the given Python lacks into the given overlay directory. + * + * @param python The environment's Python executable, or null if the + * environment has not been built (all packages are missing). + * @param dependencies The script's dependencies, as PEP 508 strings. + * @param overlay The directory to install into. + * @return The overlay directory, or null if nothing is missing. + */ + String install(String python, List dependencies, File overlay); + } + + private final Installer installer; + private final File root; + private final long settleMillis; + private final LongSupplier clock; + private final ExecutorService executor = Executors.newSingleThreadExecutor( + r -> { + final Thread t = new Thread(r, "appose-python-overlays"); + t.setDaemon(true); + return t; + }); + + /** Installs, by environment name, Python and dependencies. */ + private final Map> installs = + new HashMap<>(); + + /** The dependencies last seen for each environment, and since when. */ + private final Map seen = new HashMap<>(); + + /** + * @param installer Does the installing, on a background thread. + * @param root Directory holding the overlays (one per environment). + * @param settleMillis How long dependencies must stay unchanged before + * their missing packages are installed. + * @param clock The current time, in milliseconds. + */ + public DependencyOverlays(final Installer installer, final File root, + final long settleMillis, final LongSupplier clock) + { + this.installer = installer; + this.root = root; + this.settleMillis = settleMillis; + this.clock = clock; + } + + /** + * Gets the overlay providing the packages the environment lacks, if it is + * ready; starts preparing it once the dependencies have settled. Never + * blocks. + * + * @param envName The environment's name, which names its overlay. + * @param python The environment's Python executable, or null if not built. + * @param dependencies The script's dependencies (PEP 508 strings). + * @param settled Whether the dependencies are known not to be changing + * (e.g. the script was just opened), so need no time to settle. + * @return The overlay directory, or null if none is needed or ready. + */ + public String overlay(final String envName, final String python, + final List dependencies, final boolean settled) + { + if (dependencies == null || dependencies.isEmpty()) return null; + final String key = envName + "\n" + python + "\n" + dependencies; + synchronized (installs) { + final CompletableFuture install = installs.get(key); + if (install != null) { + return install.isDone() && !install.isCompletedExceptionally() + ? install.join() : null; + } + // Wait for the dependencies to settle, e.g. a name to be typed fully. + final long now = clock.getAsLong(); + final Seen last = seen.get(envName); + if (last == null || !last.key.equals(key)) { + seen.put(envName, new Seen(key, now)); + if (!settled) return null; + } + else if (!settled && now - last.since < settleMillis) return null; + final File overlay = new File(root, envName); + installs.put(key, CompletableFuture.supplyAsync(() -> installer.install( + python, dependencies, overlay), executor)); + return null; + } + } + + /** Stops the background thread. */ + public void dispose() { + executor.shutdownNow(); + } + + private static final class Seen { + + private final String key; + private final long since; + + private Seen(final String key, final long since) { + this.key = key; + this.since = since; + } + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediClient.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediClient.java new file mode 100644 index 0000000..6098a1b --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediClient.java @@ -0,0 +1,288 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.function.Function; + +import org.scijava.code.api.Completion; +import org.scijava.code.api.SignatureHelp; + +/** + * Asks Jedi for completions without ever making the caller wait long: the + * editor calls completers on its UI thread, on every keystroke, while Jedi's + * answers can take a while (its first for a script's environment, seconds; + * its tooling environment, minutes to build). + *

+ * Requests run one at a time, in the background, and their answers come as + * futures. Answers are cached by a key, which callers choose so that + * keystrokes within a word share one answer. A new query supersedes queued + * ones that have not started. + *

+ * + * @author Gabriel Selzer + */ +public final class JediClient { + + /** What to ask Jedi. */ + public static final class Query { + + /** The source to analyze. */ + public final String source; + + /** The caret's line (1-based) in {@link #source}. */ + public final int line; + + /** The caret's column (0-based) in {@link #source}. */ + public final int column; + + /** The script's path, or null if unsaved. */ + public final String path; + + /** + * The Python executable of the script's environment, or null to analyze + * with Jedi's own interpreter (standard library only). + */ + public final String python; + + /** + * How many completions to give signatures, which are expensive (e.g. + * those of NumPy's many overloaded functions). + */ + public final int signatures; + + /** + * A directory of packages the environment lacks (see + * {@link DependencyOverlays}), to search too; or null. + */ + public final String overlay; + + public Query(final String source, final int line, final int column, + final String path, final String python) + { + this(source, line, column, path, python, 0, null); + } + + public Query(final String source, final int line, final int column, + final String path, final String python, final int signatures, + final String overlay) + { + this.source = source; + this.line = line; + this.column = column; + this.path = path; + this.python = python; + this.signatures = signatures; + this.overlay = overlay; + } + } + + /** Does the actual work: asks Jedi, e.g. by running a task on a worker. */ + public interface Backend { + + /** Gets completions for a query, remembering them by its key. */ + List complete(String key, Query query); + + /** Gets the documentation of a completion of the query with this key. */ + String describe(String key, Query query, String name); + + /** Gets the signatures of the call around the query's caret. */ + default SignatureHelp signatures(final String key, final Query query) { + return SignatureHelp.NONE; + } + } + + /** Number of answers to remember. */ + private static final int CACHE_SIZE = 32; + + /** + * How long an answer stays good, in milliseconds: long enough for typing + * a word, short enough to notice an environment that changed meanwhile + * (e.g. was rebuilt by running the script). + */ + private static final long ANSWER_LIFETIME = 20000; + + private final Backend backend; + private final ExecutorService executor = Executors.newSingleThreadExecutor( + r -> { + final Thread t = new Thread(r, "appose-python-completion"); + t.setDaemon(true); + return t; + }); + + /** Answers (or pending answers), by key, least recently used first. */ + private final Map>> answers = + new LinkedHashMap>>(16, 0.75f, + true) + { + + @Override + protected boolean removeEldestEntry( + final Map.Entry>> eldest) + { + return size() > CACHE_SIZE; + } + }; + + private CompletableFuture> latest; + + /** When each cached answer was asked for, by key. */ + private final Map askedAt = new HashMap<>(); + + /** + * @param backend Asks Jedi; called on a background thread, one request at + * a time. + */ + public JediClient(final Backend backend) { + this.backend = backend; + } + + /** + * @param completer Computes completions for a query; there is no + * documentation. + */ + public JediClient(final Function> completer) { + this(new Backend() { + + @Override + public List complete(final String key, final Query query) { + return completer.apply(query); + } + + @Override + public String describe(final String key, final Query query, + final String name) + { + return null; + } + }); + } + + /** + * Gets Jedi's completions for a query: the answer, or a pending one. + * + * @param key Identifies the answer: queries with the same key share one. + * @param query What to ask, should the answer not be known yet. + * @return The completions (or null, if Jedi has none), once known. Fails if + * they could not be computed; the next request with the same key + * then asks again. + */ + public CompletableFuture> answer(final String key, + final Query query) + { + synchronized (answers) { + final CompletableFuture> known = answers.get(key); + final Long asked = askedAt.get(key); + if (known != null && !known.isCancelled() && !known + .isCompletedExceptionally() && asked != null && System + .currentTimeMillis() - asked < ANSWER_LIFETIME) return known; + // Supersede the previous query, unless it has started already: a + // cancelled future's task is skipped when its turn comes. + if (latest != null) latest.cancel(false); + final CompletableFuture> f = CompletableFuture + .supplyAsync(() -> backend.complete(key, query), executor); + answers.put(key, f); + askedAt.keySet().retainAll(answers.keySet()); + askedAt.put(key, System.currentTimeMillis()); + latest = f; + return f; + } + } + + /** + * Gets Jedi's completions for a query, if available within the time budget. + * + * @return The completions, or null if they are not available in time (or + * could not be computed). + */ + public List complete(final String key, final Query query, + final long budgetMillis) + { + return await(answer(key, query), budgetMillis); + } + + /** + * Gets the documentation of a completion, waiting at most the given time. + * + * @param key The key of the query that yielded the completion. + * @param query That query. + * @param name The completion's name. + * @return Its documentation, or null if none (or not available in time). + */ + public String describe(final String key, final Query query, + final String name, final long budgetMillis) + { + return await(CompletableFuture.supplyAsync(() -> backend.describe(key, + query, name), executor), budgetMillis); + } + + /** + * Gets the signatures of the call around the query's caret (see + * {@link org.scijava.code.api.CodeCompleter#signatureHelp}), once known. + */ + public CompletableFuture signatures(final String key, + final Query query) + { + return CompletableFuture.supplyAsync(() -> backend.signatures(key, query), + executor); + } + + /** Stops the background thread. */ + public void dispose() { + executor.shutdownNow(); + } + + private static T await(final CompletableFuture f, + final long budgetMillis) + { + try { + return f.get(budgetMillis, TimeUnit.MILLISECONDS); + } + catch (final TimeoutException | ExecutionException exc) { + return null; // NB: Not (yet) available. + } + catch (final java.util.concurrent.CancellationException exc) { + return null; + } + catch (final InterruptedException exc) { + Thread.currentThread().interrupt(); + return null; + } + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java new file mode 100644 index 0000000..bb71141 --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java @@ -0,0 +1,443 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Deque; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.treesitter.TSNode; +import org.treesitter.TSParser; +import org.treesitter.TSTree; +import org.treesitter.TreeSitterPython; + +/** + * Syntactic analysis of a Python script at the caret, using tree-sitter: what + * kind of completion the caret calls for, and which names are in scope there. + *

+ * This needs no Python, so it works instantly, before (or without) any Appose + * environment. It knows nothing of types, though: members of objects come + * from Jedi (see {@link JediClient}). + *

+ * + * @author Gabriel Selzer + */ +public final class PythonSyntax { + + /** Identifier marking the caret position in the parsed source. */ + static final String CARET = "__scijava_caret__"; + + /** What the caret calls for. */ + public enum Kind { + /** Nothing: the caret is in a comment, string or directive. */ + NONE, + /** A name, e.g. a variable, function or keyword. */ + NAME, + /** An attribute of the expression before a dot. */ + ATTRIBUTE, + /** A module or name in an import statement. */ + IMPORT + } + + /** The completion context at the caret. */ + public static final class Context { + + /** What the caret calls for. */ + public final Kind kind; + + /** The partial identifier being typed (possibly empty). */ + public final String prefix; + + /** Offset where the partial identifier starts. */ + public final int replaceStart; + + /** For {@link Kind#ATTRIBUTE}: the source of the expression, or null. */ + public final String receiver; + + /** Names in scope at the caret, innermost first. */ + public final Set names; + + /** For {@link Kind#ATTRIBUTE}: attributes used on the same receiver. */ + public final Set attributesUsed; + + private Context(final Kind kind, final String prefix, + final int replaceStart, final String receiver, final Set names, + final Set attributesUsed) + { + this.kind = kind; + this.prefix = prefix; + this.replaceStart = replaceStart; + this.receiver = receiver; + this.names = names; + this.attributesUsed = attributesUsed; + } + } + + /** Python keywords, including soft keywords. */ + public static final List KEYWORDS = Arrays.asList("False", "None", + "True", "and", "as", "assert", "async", "await", "break", "case", "class", + "continue", "def", "del", "elif", "else", "except", "finally", "for", + "from", "global", "if", "import", "in", "is", "lambda", "match", + "nonlocal", "not", "or", "pass", "raise", "return", "try", "while", + "with", "yield"); + + /** Python's built-in functions, types and constants. */ + public static final List BUILTINS = Arrays.asList("abs", "aiter", + "all", "anext", "any", "ascii", "bin", "bool", "breakpoint", "bytearray", + "bytes", "callable", "chr", "classmethod", "compile", "complex", + "delattr", "dict", "dir", "divmod", "enumerate", "eval", "exec", + "filter", "float", "format", "frozenset", "getattr", "globals", + "hasattr", "hash", "help", "hex", "id", "input", "int", "isinstance", + "issubclass", "iter", "len", "list", "locals", "map", "max", + "memoryview", "min", "next", "object", "oct", "open", "ord", "pow", + "print", "property", "range", "repr", "reversed", "round", "set", + "setattr", "slice", "sorted", "staticmethod", "str", "sum", "super", + "tuple", "type", "vars", "zip", "__name__", "__file__", "Ellipsis", + "NotImplemented", "Exception", "BaseException", "ValueError", + "TypeError", "KeyError", "IndexError", "RuntimeError", "AttributeError", + "ImportError", "OSError", "FileNotFoundError", "StopIteration", + "NotImplementedError", "ZeroDivisionError"); + + private static final Pattern IMPORT_LINE = Pattern.compile( + "^\\s*(from\\s+\\S*(\\s+import\\b.*)?|import\\s.*)$"); + + /** Node types that open a new scope. */ + private static final Set SCOPES = new LinkedHashSet<>(Arrays.asList( + "function_definition", "class_definition", "lambda", "list_comprehension", + "set_comprehension", "dictionary_comprehension", "generator_expression")); + + private PythonSyntax() { + // NB: Prevent instantiation of utility class. + } + + /** Analyzes the given script at the given caret offset. */ + public static Context analyze(final String text, final int caret) { + final int lineStart = text.lastIndexOf('\n', caret - 1) + 1; + final String line = text.substring(lineStart, caret); + int start = caret; + while (start > lineStart && isIdentifierPart(text.charAt(start - 1))) { + start--; + } + final String prefix = text.substring(start, caret); + final String before = text.substring(lineStart, start); + final String textBefore = text.substring(0, start); + final String closers = closers(textBefore); + if (closers == null || line.trim().startsWith("#")) return none(prefix, + start); + if (!prefix.isEmpty() && Character.isDigit(prefix.charAt(0))) { + return none(prefix, start); + } + + final Kind kind; + String receiver = null; + if (IMPORT_LINE.matcher(before + prefix).matches()) kind = Kind.IMPORT; + else if (before.trim().endsWith(".") && !before.trim().endsWith("..")) { + receiver = receiver(before.substring(0, before.lastIndexOf('.'))); + if (receiver != null && receiver.matches("\\d+")) { + return none(prefix, start); // a number, e.g. "1." + } + kind = Kind.ATTRIBUTE; + } + else kind = Kind.NAME; + + // Parse the script up to the caret, with the partial identifier replaced + // by a placeholder, and any open brackets closed: a clean tree there. + final String source = textBefore + CARET + closers; + final Set names = kind == Kind.NAME ? namesInScope(source, + textBefore.length()) : new LinkedHashSet<>(); + final Set used = kind == Kind.ATTRIBUTE && receiver != null + ? attributesUsed(text, receiver, start) : new LinkedHashSet<>(); + return new Context(kind, prefix, start, receiver, names, used); + } + + // -- Helper methods -- + + private static Context none(final String prefix, final int start) { + return new Context(Kind.NONE, prefix, start, null, new LinkedHashSet<>(), + new LinkedHashSet<>()); + } + + /** + * Gets the closing brackets for those left open at the end of + * {@code text}, or null if the text ends inside a string or comment. + */ + static String closers(final String text) { + final Deque open = new ArrayDeque<>(); + int i = 0; + final int n = text.length(); + while (i < n) { + final char c = text.charAt(i); + if (c == '#') { + final int end = text.indexOf('\n', i); + if (end < 0) return null; + i = end + 1; + } + else if (c == '\'' || c == '"') { + final boolean triple = text.startsWith(String.valueOf(c) + c + c, i); + final String quote = triple ? String.valueOf(c) + c + c : String + .valueOf(c); + int j = i + quote.length(); + while (true) { + if (j >= n) return null; + final char d = text.charAt(j); + if (d == '\\') j += 2; + else if (!triple && d == '\n') return null; // unterminated line + else if (text.startsWith(quote, j)) break; + else j++; + } + i = j + quote.length(); + } + else { + if (c == '(') open.push(')'); + else if (c == '[') open.push(']'); + else if (c == '{') open.push('}'); + else if ((c == ')' || c == ']' || c == '}') && !open.isEmpty() && open + .peek() == c) open.pop(); + i++; + } + } + final StringBuilder sb = new StringBuilder(); + for (final char c : open) sb.append(c); + return sb.toString(); + } + + /** + * Gets the source of the expression ending the given text, e.g. + * {@code np.random} in {@code x = np.random}, or null if there is none. + */ + static String receiver(final String text) { + int i = text.length(); + while (i > 0 && Character.isWhitespace(text.charAt(i - 1))) i--; + final int end = i; + int depth = 0; + while (i > 0) { + final char c = text.charAt(i - 1); + if (c == ')' || c == ']' || c == '}') depth++; + else if (c == '(' || c == '[' || c == '{') { + if (depth == 0) break; + depth--; + } + else if (depth == 0 && !isIdentifierPart(c) && c != '.') break; + i--; + } + final String receiver = text.substring(i, end).trim(); + return receiver.isEmpty() ? null : receiver; + } + + /** Attributes used on {@code receiver} anywhere in the script. */ + private static Set attributesUsed(final String text, + final String receiver, final int exclude) + { + final Set used = new LinkedHashSet<>(); + // NB: Only spaces around the dot: "a.\nprint" is not an attribute. + final Matcher m = Pattern.compile("(? namesInScope(final String source, + final int caret) + { + final byte[] bytes = source.getBytes(StandardCharsets.UTF_8); + final int caretByte = source.substring(0, caret).getBytes( + StandardCharsets.UTF_8).length; + final Set names = new LinkedHashSet<>(); + final TSParser parser = new TSParser(); + try { + parser.setLanguage(new TreeSitterPython()); + final TSTree tree = parser.parseString(null, source); + final TSNode root = tree.getRootNode(); + TSNode node = root.getNamedDescendantForByteRange(caretByte, caretByte + + CARET.length()); + // Walk outward through the enclosing scopes, innermost first. + boolean inFunction = false; + for (; node != null && !node.isNull(); node = node.getParent()) { + final String type = node.getType(); + if (!SCOPES.contains(type)) continue; + // NB: Class bodies are not visible from their methods. + if (type.equals("class_definition") && inFunction) continue; + if (type.equals("function_definition") || type.equals("lambda")) { + inFunction = true; + } + final TSNode params = node.getChildByFieldName("parameters"); + if (params != null && !params.isNull()) { + addTargets(params, bytes, names); + } + final TSNode body = node.getChildByFieldName("body"); + if (type.endsWith("comprehension") || type.equals( + "generator_expression")) + { + collectBindings(node, bytes, names, true); + } + else if (body != null && !body.isNull() && !type.equals("lambda")) { + collectBindings(body, bytes, names, false); + } + } + collectBindings(root, bytes, names, false); + } + catch (final RuntimeException | LinkageError exc) { + // NB: Native parser unavailable or failed: no names, then. + } + finally { + parser.close(); + } + names.remove(CARET); + return names; + } + + /** + * Collects the names bound by statements within {@code node}, without + * descending into nested scopes (whose own names are bound here, though). + */ + private static void collectBindings(final TSNode node, final byte[] bytes, + final Set names, final boolean comprehension) + { + final int n = node.getNamedChildCount(); + for (int i = 0; i < n; i++) { + final TSNode child = node.getNamedChild(i); + final String type = child.getType(); + switch (type) { + case "function_definition": + case "class_definition": + addIdentifier(child.getChildByFieldName("name"), bytes, names); + continue; // NB: A nested scope. + case "lambda": + case "list_comprehension": + case "set_comprehension": + case "dictionary_comprehension": + case "generator_expression": + if (!comprehension) continue; // NB: A nested scope. + break; + case "assignment": + case "augmented_assignment": + case "for_statement": + case "for_in_clause": + addTargets(child.getChildByFieldName("left"), bytes, names); + break; + case "named_expression": + addIdentifier(child.getChildByFieldName("name"), bytes, names); + break; + case "as_pattern_target": + addTargets(child, bytes, names); + break; + case "import_statement": + addImported(child, bytes, names, true); + continue; + case "import_from_statement": + addImported(child, bytes, names, false); + continue; + default: + } + collectBindings(child, bytes, names, comprehension); + } + } + + /** Adds the names an import statement binds. */ + private static void addImported(final TSNode statement, final byte[] bytes, + final Set names, final boolean plainImport) + { + final int n = statement.getNamedChildCount(); + for (int i = 0; i < n; i++) { + if (!"name".equals(statement.getFieldNameForNamedChild(i))) continue; + final TSNode child = statement.getNamedChild(i); + if (child.getType().equals("aliased_import")) { + addIdentifier(child.getChildByFieldName("alias"), bytes, names); + } + else if (child.getType().equals("dotted_name")) { + // "import a.b" binds a; "from x import y" binds y. + final int count = child.getNamedChildCount(); + if (count > 0) addIdentifier(child.getNamedChild(plainImport ? 0 + : count - 1), bytes, names); + } + } + } + + /** + * Adds the identifiers bound by an assignment target or parameter list, + * e.g. {@code a, (b, c)} or {@code x, y: int = 1, *args, **kwargs}, but not + * attributes or subscripts. + */ + private static void addTargets(final TSNode node, final byte[] bytes, + final Set names) + { + if (node == null || node.isNull()) return; + switch (node.getType()) { + case "identifier": + addIdentifier(node, bytes, names); + return; + case "attribute": + case "subscript": + case "type": + return; + case "default_parameter": + case "typed_default_parameter": + addIdentifier(node.getChildByFieldName("name"), bytes, names); + return; + case "typed_parameter": + // NB: Its name is its first child (e.g. x in "x: int"). + if (node.getNamedChildCount() > 0) addTargets(node.getNamedChild(0), + bytes, names); + return; + default: + for (int i = 0; i < node.getNamedChildCount(); i++) { + addTargets(node.getNamedChild(i), bytes, names); + } + } + } + + private static void addIdentifier(final TSNode node, final byte[] bytes, + final Set names) + { + if (node == null || node.isNull() || !node.getType().equals( + "identifier")) return; + names.add(new String(bytes, node.getStartByte(), node.getEndByte() - node + .getStartByte(), StandardCharsets.UTF_8)); + } + + private static boolean isIdentifierPart(final char c) { + return Character.isLetterOrDigit(c) || c == '_'; + } + + /** For tests: the names found by the given parse, as a list. */ + static List names(final String text, final int caret) { + return new ArrayList<>(analyze(text, caret).names); + } +} diff --git a/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/jedi_complete.py b/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/jedi_complete.py new file mode 100644 index 0000000..6ca716b --- /dev/null +++ b/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/jedi_complete.py @@ -0,0 +1,172 @@ +### +# #%L +# Python scripting language plugin backed by Appose. +# %% +# Copyright (C) 2026 SciJava developers. +# %% +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE +# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. +# #L% +### +# Appose task script that asks Jedi for code completions in a script, or for +# the documentation of one of them. +# +# Runs in the completion tooling environment (Python + Jedi), and analyzes +# the script against its own environment's interpreter, if given, so that +# the packages installed there are known. +# +# Task inputs: +# +# op -- "complete" (the default), "describe" or "signatures" +# key -- identifies the query: "describe" looks up the +# completions of a "complete" with the same key +# source -- the script's source code (with a preamble typing its +# SciJava parameters) +# line, column -- the caret position (line 1-based, column 0-based) +# path -- the script's file path, or None if unsaved +# python -- the Python executable of the script's environment, or +# None to use Jedi's own interpreter +# max_signatures -- for "complete": how many completions to give +# signatures (they are expensive: give few, or none) +# max_overloads -- for "complete": how many signatures to give each +# name -- for "describe": which completion to describe +# overlay -- a directory of packages the environment lacks, which +# Jedi should search too; or None +# +# Task outputs: +# +# completions -- for "complete": list of dicts: name, type, description, +# and signatures: a list of dicts with params (list of +# [name, annotation or None]) and returns (or None) +# docstring -- for "describe": the completion's documentation, or None +# signatures -- for "signatures": the signatures of the call around the +# caret: list of dicts with name, params and returns +# index -- for "signatures": index of the argument the caret is in, +# or None if Jedi cannot tell + +import os as _sj_os +import sys as _sj_sys + + +def _sj_cache(): + # Note: Each task gets fresh globals, so keep state that is slow to + # recompute in a module that persists in the worker: Jedi's environments, + # and the latest completions (to describe them on request). + cache = _sj_sys.modules.get("_scijava_jedi_cache") + if cache is None: + import types + + cache = types.ModuleType("_scijava_jedi_cache") + cache.environments = {} + cache.completions = {} # key -> {name: jedi Completion} + _sj_sys.modules["_scijava_jedi_cache"] = cache + return cache + + +def _sj_script(cache): + import jedi + + environment = None + if python: + environment = cache.environments.get(python) + if environment is None: + try: + environment = jedi.create_environment(python, safe=False) + except Exception: # noqa: BLE001 -- e.g. a Python Jedi cannot use + environment = jedi.InterpreterEnvironment() + cache.environments[python] = environment + else: + environment = jedi.InterpreterEnvironment() + + extra = [overlay] if globals().get("overlay") else [] + project = None + if path or extra: + root = _sj_os.path.dirname(_sj_os.path.abspath(path)) if path else None + project = jedi.Project(root or _sj_os.getcwd(), added_sys_path=extra) + return jedi.Script( + source, path=path, environment=environment, project=project + ) + + +def _sj_completions(cache): + completions = _sj_script(cache).complete(line, column) + # Remember them, to describe them on request; only the latest few. + cache.completions[key] = {c.name: c for c in completions} + while len(cache.completions) > 8: + cache.completions.pop(next(iter(cache.completions))) + return completions + + +def _sj_signature(sig): + params = [] + for p in sig.params: + text = p.to_string() + annotation = None + if ":" in text: + annotation = text.split(":", 1)[1].split("=", 1)[0].strip() + params.append([p.name, annotation]) + sig_text = sig.to_string() + returns = sig_text.rsplit("->", 1)[1].strip() if "->" in sig_text else None + return {"name": sig.name, "params": params, "returns": returns} + + +def _sj_signatures(c): + return [_sj_signature(sig) for sig in c.get_signatures()[:max_overloads]] + + +def _sj_main(): + cache = _sj_cache() + if globals().get("op") == "signatures": + signatures = _sj_script(cache).get_signatures(line, column) + task.outputs["index"] = signatures[0].index if signatures else None + task.outputs["signatures"] = [ + _sj_signature(s) for s in signatures[:max_overloads] + ] + return + if globals().get("op") == "describe": + known = cache.completions.get(key) + if known is None: + # Note: E.g. the worker restarted since; ask Jedi again. + known = {c.name: c for c in _sj_completions(cache)} + c = known.get(name) + task.outputs["docstring"] = (c.docstring(raw=True) or None) if c else None + return + + results = [] + for i, c in enumerate(_sj_completions(cache)): + signatures = [] + if i < max_signatures and c.type in ("function", "class"): + try: + signatures = _sj_signatures(c) + except Exception: # noqa: BLE001,S110 -- best effort + pass + results.append( + { + "name": c.name, + "type": c.type, + "description": c.description, + "signatures": signatures, + } + ) + task.outputs["completions"] = results + + +_sj_main() diff --git a/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/overlay.py b/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/overlay.py new file mode 100644 index 0000000..133f37e --- /dev/null +++ b/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/overlay.py @@ -0,0 +1,127 @@ +### +# #%L +# Python scripting language plugin backed by Appose. +# %% +# Copyright (C) 2026 SciJava developers. +# %% +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE +# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. +# #L% +### +# Appose task script that installs, for code completion only, the packages a +# script declares but its environment lacks (e.g. ones just added to its +# dependencies), so that Jedi knows them before the environment is rebuilt. +# +# Runs in the completion tooling environment, which includes uv. Installs +# into a separate directory (an "overlay", which Jedi adds to its search +# path), never into the script's environment itself; and only the missing +# packages themselves (--no-deps): their dependencies usually are installed. +# +# Task inputs: +# +# python -- the Python executable of the script's environment, or +# None if it has not been built (then all dependencies are +# missing, installed for the tooling environment's Python) +# dependencies -- the script's dependencies (PEP 508 requirement strings) +# overlay -- the directory to install into +# +# Task outputs: +# +# path -- the overlay directory, or None if nothing is missing +# installed -- the requirements installed into it + +import json as _sj_json +import os as _sj_os +import re as _sj_re +import shutil as _sj_shutil +import subprocess as _sj_subprocess +import sys as _sj_sys + +_SJ_INSTALLED = ( + "import importlib.metadata as m, json, re\n" + "print(json.dumps(sorted({re.sub(r'[-_.]+', '-', (d.metadata['Name'] or '')" + ".lower()) for d in m.distributions()})))" +) + + +def _sj_name(requirement): + """The normalized distribution name of a PEP 508 requirement string.""" + m = _sj_re.match(r"\s*([A-Za-z0-9][A-Za-z0-9._-]*)", requirement) + return _sj_re.sub(r"[-_.]+", "-", m.group(1).lower()) if m else None + + +def _sj_overlay(): + target = python or _sj_sys.executable + installed = set() + if python: + result = _sj_subprocess.run( + [python, "-c", _SJ_INSTALLED], + capture_output=True, + text=True, + timeout=120, + ) + if result.returncode == 0: + installed = set(_sj_json.loads(result.stdout)) + missing = [ + r for r in dependencies if _sj_name(r) and _sj_name(r) not in installed + ] + + task.outputs["installed"] = missing + if not missing: + _sj_shutil.rmtree(overlay, ignore_errors=True) + task.outputs["path"] = None + return + + marker = _sj_os.path.join(overlay, ".scijava-overlay.json") + wanted = {"python": target, "requirements": sorted(missing)} + try: + with open(marker, encoding="utf-8") as f: + if _sj_json.load(f) == wanted: + task.outputs["path"] = overlay # Already up to date. + return + except (OSError, ValueError): + pass + + from uv import find_uv_bin + + # Install next to the overlay, then swap it in, so that Jedi never sees a + # half-installed overlay. + staging = overlay + ".new" + _sj_shutil.rmtree(staging, ignore_errors=True) + result = _sj_subprocess.run( + [find_uv_bin(), "pip", "install", "--target", staging, "--no-deps", + "--python", target, *missing], + capture_output=True, + text=True, + timeout=600, + ) + if result.returncode != 0: + _sj_shutil.rmtree(staging, ignore_errors=True) + raise RuntimeError("uv pip install failed:\n" + result.stderr) + with open(_sj_os.path.join(staging, ".scijava-overlay.json"), "w", + encoding="utf-8") as f: + _sj_json.dump(wanted, f) + _sj_shutil.rmtree(overlay, ignore_errors=True) + _sj_os.replace(staging, overlay) + task.outputs["path"] = overlay + + +_sj_overlay() diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java new file mode 100644 index 0000000..56f244b --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java @@ -0,0 +1,583 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.stream.Collectors; + +import net.imglib2.img.Img; + +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.scijava.Context; +import org.scijava.code.api.CodeCompletionService; +import org.scijava.code.api.Completion; +import org.scijava.code.api.CompletionRequest; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; +import org.scijava.plugins.scripting.appose.python.completion.JediClient; +import org.scijava.script.ScriptLanguage; +import org.scijava.script.ScriptService; + +/** + * Tests {@link ApposePythonCodeCompleter}, with a fake Jedi. + * + * @author Gabriel Selzer + */ +public class ApposePythonCodeCompleterTest { + + private Context context; + private ScriptLanguage language; + private CodeCompletionService service; + private ApposePythonCodeCompleter completer; + + @Before + public void setUp() { + context = new Context(ScriptService.class, CodeCompletionService.class, + ResidentWorkerService.class); + language = context.service(ScriptService.class).getLanguageByName( + "appose-python"); + service = context.service(CodeCompletionService.class); + completer = (ApposePythonCodeCompleter) service.getCompleterPlugin( + language); + } + + @After + public void tearDown() { + context.dispose(); + } + + @Test + public void testCompleterIsDiscovered() { + assertTrue(completer != null); + } + + @Test + public void testLocalCompletions() { + noJedi(); + final String header = "#@ net.imglib2.img.Img image\n#@ double sigma\n"; + final List names = complete(header + + "def blur(img, s):\n result = img\n re"); + assertTrue(names.toString(), names.contains("result")); + assertTrue(names.contains("return")); + assertTrue(complete(header + "x = si").contains("sigma")); + assertTrue(complete(header + "pri").contains("print")); + assertTrue(complete(header + "ta").contains("task")); + // Nothing on parameter lines, in strings or comments. + assertTrue(complete("#@ String na").isEmpty()); + assertTrue(complete("s = 'pri").isEmpty()); + } + + @Test + public void testJediResultsMergedFirst() { + final Completion zeros = Completion.builder("zeros").kind( + Completion.Kind.METHOD).parameters(Collections.singletonList( + new Completion.Parameter("shape", "_ShapeLike"))).build(); + final Completion zeta = Completion.of("zeta"); + completer.setJediClient(new JediClient(q -> Arrays.asList(zeros, zeta))); + final List completions = completions( + "import numpy as np\nnp.zeros(3)\nnp.ze"); + assertEquals("zeros", completions.get(0).insertionText()); + assertEquals(1, completions.get(0).parameters().size()); + assertEquals(Arrays.asList("zeros", "zeta"), completions.stream().map( + Completion::insertionText).collect(Collectors.toList())); + } + + @Test + public void testImportsAreNames() { + // Jedi reports imshow as a function, once per overload. + final Completion imshow = Completion.builder("imshow").kind( + Completion.Kind.METHOD).parameters(Collections.singletonList( + new Completion.Parameter("data", "Any"))).summary("def imshow") + .build(); + completer.setJediClient(new JediClient(q -> Arrays.asList(imshow, + imshow))); + // In an import, it is a name to import, not a call: listed once, without + // a parameter list for the editor to insert. + final List imported = completions("from ndv import i"); + assertEquals(1, imported.size()); + assertEquals("imshow", imported.get(0).insertionText()); + assertFalse(imported.get(0).isCallable()); + assertEquals("def imshow", imported.get(0).summary()); + // Elsewhere, it is still a call. + assertTrue(completions("import ndv\nndv.i").get(0).isCallable()); + } + + @Test + public void testTimeBudgetAndReuse() throws InterruptedException { + final AtomicInteger calls = new AtomicInteger(); + final CountDownLatch release = new CountDownLatch(1); + final CountDownLatch done = new CountDownLatch(1); + completer.setJediClient(new JediClient(q -> { + calls.incrementAndGet(); + try { + release.await(); + } + catch (final InterruptedException exc) { + Thread.currentThread().interrupt(); + } + done.countDown(); + return Collections.singletonList(Completion.of("zeros")); + })); + // Jedi is slow: the local results come back, in time. + final String code = "import numpy as np\nnp.zeros(3)\nnp.z"; + final long start = System.nanoTime(); + final List first = complete(code); + assertTrue(System.nanoTime() - start < TimeUnit.SECONDS.toNanos(2)); + assertEquals(Collections.singletonList("zeros"), first); // used before + // Once Jedi answers, later keystrokes of the word reuse its answer. + release.countDown(); + assertTrue(done.await(5, TimeUnit.SECONDS)); + Thread.sleep(50); + assertTrue(complete(code + "e").contains("zeros")); + assertTrue(complete(code + "er").contains("zeros")); + assertEquals(1, calls.get()); + } + + @Test + public void testUpdateWhenJediIsSlow() throws Exception { + final CountDownLatch release = new CountDownLatch(1); + completer.setJediClient(new JediClient(q -> { + try { + release.await(); + } + catch (final InterruptedException exc) { + Thread.currentThread().interrupt(); + } + return Collections.singletonList(Completion.of("zeros")); + })); + // At once: what is known locally, and the announcement of more. + final String code = "import numpy as np\nnp.z"; + final org.scijava.code.api.CompletionResult first = service.complete( + new CompletionRequest(code, language, null)); + assertTrue(first.completions().isEmpty()); + assertTrue(first.update() != null); + // Then Jedi's answer, merged. + release.countDown(); + final org.scijava.code.api.CompletionResult better = first.update().get(5, + TimeUnit.SECONDS); + assertEquals("zeros", better.completions().get(0).insertionText()); + assertEquals(first.replaceStart(), better.replaceStart()); + } + + @Test + public void testLazyDocumentation() { + final List described = new java.util.ArrayList<>(); + final String[] keys = { null, null }; + completer.setJediClient(new JediClient(new JediClient.Backend() { + + @Override + public List complete(final String key, + final JediClient.Query query) + { + keys[0] = key; + final Map zeros = new HashMap<>(); + zeros.put("name", "zeros"); + zeros.put("type", "function"); + return ApposePythonCodeCompleter.toCompletions(Collections + .singletonList(zeros), name -> () -> { + described.add(name); + keys[1] = key; + return "Return a new array of zeros."; + }); + } + + @Override + public String describe(final String key, final JediClient.Query query, + final String name) + { + return null; + } + })); + final Completion zeros = completions("import numpy\nnumpy.z").get(0); + // Not described until asked, then by the same query. + assertTrue(described.isEmpty()); + assertEquals("Return a new array of zeros.", zeros.description()); + assertEquals(Collections.singletonList("zeros"), described); + assertEquals(keys[0], keys[1]); + } + + @Test + public void testSignaturesOnlyOnceWordBegun() { + final List signatures = new java.util.ArrayList<>(); + completer.setJediClient(new JediClient(q -> { + signatures.add(q.signatures); + return Collections.emptyList(); + })); + complete("import numpy\nnumpy."); + complete("import numpy\nnumpy.z"); + assertEquals(Arrays.asList(0, 30), signatures); + // A function without signature is offered by name, not as a call. + final Map zeros = new HashMap<>(); + zeros.put("name", "zeros"); + zeros.put("type", "function"); + final Completion plain = ApposePythonCodeCompleter.toCompletions( + Collections.singletonList(zeros)).get(0); + assertFalse(plain.isCallable()); + } + + @Test + public void testOverlayInQuery() throws Exception { + // A script declaring a dependency its environment lacks. + final List> installed = new java.util.ArrayList<>(); + completer.setOverlays(new org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays( + (python, dependencies, dir) -> { + installed.add(dependencies); + return "/overlays/blur"; + }, new java.io.File("overlays"), 0, System::currentTimeMillis)); + final List queries = new java.util.ArrayList<>(); + completer.setJediClient(new JediClient(q -> { + queries.add(q); + return Collections.emptyList(); + })); + final String code = "# /// script\n# dependencies = [\"tifffile\"]\n" + + "# ///\nimport tifffile\ntifffile.im"; + // The first request notes the dependencies; once they have stayed the + // same (here, for 0 ms), the next one starts installing the overlay. + complete(code); + complete(code); + for (int i = 0; i < 100 && installed.isEmpty(); i++) Thread.sleep(20); + Thread.sleep(50); + complete(code + "r"); + // Once the overlay is ready, Jedi searches it too (anew: a new key). + assertEquals(Collections.singletonList(Collections.singletonList( + "tifffile")), installed); + assertEquals("/overlays/blur", queries.get(queries.size() - 1).overlay); + } + + @Test + public void testSignatureHelp() throws Exception { + final JediClient.Query[] asked = { null }; + final Completion dumps = Completion.builder("dumps").kind( + Completion.Kind.METHOD).parameters(Arrays.asList( + new Completion.Parameter("obj", null), new Completion.Parameter( + "indent", "int"))).build(); + completer.setJediClient(new JediClient(new JediClient.Backend() { + + @Override + public List complete(final String key, + final JediClient.Query query) + { + return Collections.emptyList(); + } + + @Override + public String describe(final String key, final JediClient.Query query, + final String name) + { + return null; + } + + @Override + public org.scijava.code.api.SignatureHelp signatures(final String key, + final JediClient.Query query) + { + asked[0] = query; + // NB: Jedi cannot always tell the argument (-1). + return new org.scijava.code.api.SignatureHelp(Collections + .singletonList(new org.scijava.code.api.SignatureHelp.Signature( + dumps, org.scijava.code.api.SignatureHelp.Fit.UNKNOWN)), -1, -1); + } + })); + final String code = "import json\njson.dumps(obj, "; + org.scijava.code.api.SignatureHelp help = completer.signatureHelp( + new CompletionRequest(code, language, null)); + if (help.update() != null) help = help.update().get(5, TimeUnit.SECONDS); + assertEquals("dumps", help.signatures().get(0).callable().insertionText()); + // The argument the caret is in, counted where Jedi cannot tell; and the + // call's opening parenthesis. + assertEquals(1, help.activeParameter()); + assertEquals(code.indexOf('('), help.callStart()); + // Jedi was asked about the caret (shifted by the preamble). + assertEquals(3, asked[0].line); + assertEquals("json.dumps(obj, ".length(), asked[0].column); + // No help outside calls, or in strings. + assertTrue(completer.signatureHelp(new CompletionRequest("x = 1", + language, null)).isEmpty()); + assertTrue(completer.signatureHelp(new CompletionRequest("s = 'f(1, ", + language, null)).isEmpty()); + } + + @Test + public void testPrepareWarmsUp() throws Exception { + final CountDownLatch asked = new CountDownLatch(1); + completer.setJediClient(new JediClient(q -> { + asked.countDown(); + return Collections.emptyList(); + })); + completer.prepare(new CompletionRequest("import numpy\n", language, + null)); + assertTrue(asked.await(5, TimeUnit.SECONDS)); + } + + @Test + public void testJediQuery() { + final JediClient.Query[] query = { null }; + completer.setJediClient(new JediClient(q -> { + query[0] = q; + return Collections.emptyList(); + })); + final String code = "#@ net.imglib2.img.Img image\n#@ double sigma\n" + + "image.sha"; + complete(code); + // The script's parameters are typed in a preamble, so Jedi knows them. + assertTrue(query[0].source.startsWith( + "import appose.python_worker; task: appose.python_worker.Task; " + + "import numpy; image: numpy.ndarray; sigma: float\n")); + // The caret is after the word's first character, shifted by the preamble. + assertEquals(4, query[0].line); + assertEquals("image.s".length(), query[0].column); + // No script environment has been built: Jedi uses its own interpreter. + assertNull(query[0].python); + } + + @Test + public void testScriptWorkersUntouched() { + noJedi(); + // Completing in a script declaring an environment must not create (or + // replace) a worker for it, even as its metadata is edited. + final String script = "# /// script\n# dependencies = [\"appose\"]\n" + + "# ///\nimport os\nos.pa"; + complete(script); + complete(script.replace("appose", "appose, numpy")); + assertTrue(context.service(ResidentWorkerService.class).workers() + .isEmpty()); + } + + @Test + public void testPythonExecutable() throws Exception { + final java.nio.file.Path tmp = java.nio.file.Files.createTempDirectory( + "appose-envs"); + try { + // pixi on Windows: python.exe is in the environment's root, above the + // bin directory Appose reports (which may not even exist). + final java.nio.file.Path pixiEnv = tmp.resolve( + "pixi/.pixi/envs/default"); + java.nio.file.Files.createDirectories(pixiEnv); + final java.nio.file.Path pixiPython = java.nio.file.Files.createFile( + pixiEnv.resolve("python.exe")); + assertEquals(pixiPython.toFile().getAbsolutePath(), + ApposePythonCodeCompleter.pythonExecutable(env(tmp.resolve("pixi"), + pixiEnv.resolve("bin")))); + // uv on Windows: python.exe is in the reported Scripts directory. + final java.nio.file.Path scripts = tmp.resolve("uv/Scripts"); + java.nio.file.Files.createDirectories(scripts); + final java.nio.file.Path uvPython = java.nio.file.Files.createFile( + scripts.resolve("python.exe")); + assertEquals(uvPython.toFile().getAbsolutePath(), + ApposePythonCodeCompleter.pythonExecutable(env(tmp.resolve("uv"), + scripts))); + // No Python at all. + assertNull(ApposePythonCodeCompleter.pythonExecutable(env(tmp.resolve( + "none"), tmp.resolve("none/bin")))); + } + finally { + delete(tmp); + } + } + + @Test + public void testPythonOnDisk() throws Exception { + final java.nio.file.Path tmp = java.nio.file.Files.createTempDirectory( + "appose-envs"); + try { + for (final String layout : new String[] { + ".pixi/envs/default/python.exe", // pixi, Windows + ".pixi/envs/default/bin/python", // pixi, Linux & macOS + "Scripts/python.exe", // uv, Windows + "bin/python" }) // uv or conda, Linux & macOS + { + final java.nio.file.Path env = tmp.resolve("env" + layout.hashCode()); + final java.nio.file.Path python = env.resolve(layout); + java.nio.file.Files.createDirectories(python.getParent()); + java.nio.file.Files.createFile(python); + assertEquals(layout, python.toFile().getAbsolutePath(), + ApposePythonCodeCompleter.pythonOnDisk(env.toFile())); + } + assertNull(ApposePythonCodeCompleter.pythonOnDisk(tmp.resolve("missing") + .toFile())); + } + finally { + delete(tmp); + } + } + + @Test + public void testEnvironmentFromEarlierSession() throws Exception { + // A script whose environment was built in an earlier session: it is on + // disk, but no worker for it exists in this one. + final java.nio.file.Path tmp = java.nio.file.Files.createTempDirectory( + "appose-envs"); + final String old = System.getProperty("appose.envs-dir"); + System.setProperty("appose.envs-dir", tmp.toString()); + try { + final java.io.File script = tmp.resolve("blur.py").toFile(); + final String header = "# /// script\n# dependencies = [\"numpy\"]\n" + + "# ///\nimport numpy\n"; + final String envName = ScriptEnvironment.of(header, script.getPath(), + null, null).name; + final java.nio.file.Path python = tmp.resolve(envName).resolve( + ".pixi/envs/default/python.exe"); + java.nio.file.Files.createDirectories(python.getParent()); + java.nio.file.Files.createFile(python); + + final JediClient.Query[] query = { null }; + completer.setJediClient(new JediClient(q -> { + query[0] = q; + return Collections.emptyList(); + })); + final String code = header + "numpy.ze"; + service.complete(new CompletionRequest(code, code.length(), language, + null, null, script.getPath())); + // Jedi analyzes against the environment's Python... + assertEquals(python.toFile().getAbsolutePath(), query[0].python); + // ...without anything built or started. + assertTrue(context.service(ResidentWorkerService.class).workers() + .isEmpty()); + } + finally { + if (old == null) System.clearProperty("appose.envs-dir"); + else System.setProperty("appose.envs-dir", old); + delete(tmp); + } + } + + @Test + public void testPythonTypes() { + assertEquals("numpy.ndarray", ApposePythonCodeCompleter.pythonType( + Img.class)); + assertEquals("float", ApposePythonCodeCompleter.pythonType(double.class)); + assertEquals("int", ApposePythonCodeCompleter.pythonType(Long.class)); + assertEquals("str", ApposePythonCodeCompleter.pythonType(String.class)); + assertEquals("bool", ApposePythonCodeCompleter.pythonType(boolean.class)); + assertEquals("list", ApposePythonCodeCompleter.pythonType(int[].class)); + assertNull(ApposePythonCodeCompleter.pythonType(Context.class)); + } + + @Test + public void testJediOutputConversion() { + final Map sig1 = new HashMap<>(); + sig1.put("params", Arrays.asList(Arrays.asList("shape", "_ShapeLike"), + Arrays.asList("dtype", null))); + sig1.put("returns", "NDArray[float64]"); + final Map sig2 = new HashMap<>(); + sig2.put("params", Collections.singletonList(Arrays.asList("shape", + null))); + sig2.put("returns", null); + final Map zeros = new LinkedHashMap<>(); + zeros.put("name", "zeros"); + zeros.put("type", "function"); + zeros.put("description", "def zeros"); + zeros.put("docstring", "zeros(shape) -> a < b"); + zeros.put("signatures", Arrays.asList(sig1, sig2)); + final Map shape = new LinkedHashMap<>(); + shape.put("name", "shape"); + shape.put("type", "property"); + shape.put("signatures", Collections.emptyList()); + final List out = ApposePythonCodeCompleter.toCompletions( + Arrays.asList(zeros, shape)); + // One completion per overload. + assertEquals(3, out.size()); + assertEquals(Completion.Kind.METHOD, out.get(0).kind()); + assertEquals("_ShapeLike", out.get(0).parameters().get(0).type()); + assertEquals("NDArray[float64]", out.get(0).returnType()); + assertEquals(1, out.get(1).parameters().size()); + assertTrue(out.get(0).description().contains("a < b")); + assertEquals(Completion.Kind.FIELD, out.get(2).kind()); + assertFalse(out.get(2).isCallable()); + } + + // -- Helper methods -- + + private static void delete(final java.nio.file.Path dir) throws Exception { + try (java.util.stream.Stream paths = + java.nio.file.Files.walk(dir)) + { + paths.sorted(java.util.Comparator.reverseOrder()).forEach(p -> p + .toFile().delete()); + } + } + + /** A built environment with the given base and bin directory. */ + private static org.apposed.appose.Environment env( + final java.nio.file.Path base, final java.nio.file.Path bin) + { + return new org.apposed.appose.Environment() { + + @Override + public String base() { + return base.toString(); + } + + @Override + public List binPaths() { + return Collections.singletonList(bin.toString()); + } + + @Override + public List launchArgs() { + return Collections.emptyList(); + } + + @Override + public org.apposed.appose.Builder builder() { + return null; + } + }; + } + + /** Uses a Jedi that never answers in time. */ + private void noJedi() { + completer.setJediClient(new JediClient(q -> null)); + } + + private List completions(final String code) { + return service.complete(new CompletionRequest(code, language, null)) + .completions(); + } + + private List complete(final String code) { + return completions(code).stream().map(Completion::insertionText).collect( + Collectors.toList()); + } +} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java new file mode 100644 index 0000000..4b0707b --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -0,0 +1,227 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.File; +import java.net.URISyntaxException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.scijava.Context; +import org.scijava.code.api.CodeCompletionService; +import org.scijava.code.api.Completion; +import org.scijava.code.api.CompletionRequest; +import org.scijava.code.api.CompletionResult; +import org.scijava.code.api.SignatureHelp; +import org.scijava.module.ModuleService; +import org.scijava.script.ScriptInfo; +import org.scijava.script.ScriptLanguage; +import org.scijava.script.ScriptModule; +import org.scijava.script.ScriptService; + +/** + * End-to-end tests of {@link ApposePythonCodeCompleter} with a real Jedi. + *

+ * The first completion builds the completion tooling environment (Python and + * Jedi, via pixi), and a script run builds the small uv environment of + * {@code appose-test/requirements.txt}; both require network access. + *

+ * + * @author Gabriel Selzer + */ +public class ApposePythonCompletionIntegrationTest { + + private static final String HEADER = "#!appose-python\n" + + "#@script(env=\"requirements.txt\", scheme=\"requirements.txt\")\n"; + + /** How long to wait for environments to build. */ + private static final long BUILD_TIMEOUT = TimeUnit.MINUTES.toMillis(15); + + private static Context context; + private static ScriptLanguage language; + private static CodeCompletionService service; + private static File scriptDir; + + @BeforeClass + public static void setUp() throws URISyntaxException { + context = new Context(); + language = context.service(ScriptService.class).getLanguageByName( + "appose-python"); + service = context.service(CodeCompletionService.class); + scriptDir = new File(ApposePythonCompletionIntegrationTest.class + .getResource("/appose-test/requirements.txt").toURI()).getParentFile(); + } + + @AfterClass + public static void tearDown() { + if (context != null) context.dispose(); + } + + @Test + public void testStandardLibrary() throws Exception { + // Before any script environment is built, Jedi knows the standard + // library, with signatures and documentation. + final List completions = completeUntil( + "import json\njson.du", null, "dumps"); + final Completion dumps = completions.stream().filter(c -> c + .insertionText().equals("dumps")).findFirst().get(); + assertTrue(dumps.isCallable()); + assertFalse(dumps.parameters().isEmpty()); + assertTrue(dumps.description() != null); + report("import json\njson.l", null, "loads", "o"); + } + + @Test + public void testScriptEnvironment() throws Exception { + // Run a script, building its environment (which includes numpy). + final File script = new File(scriptDir, "completion.py"); + final String header = HEADER + "#@ net.imglib2.img.Img image\n" + + "import numpy as np\n"; + Files.write(script.toPath(), (header + "x = 1\n").getBytes( + StandardCharsets.UTF_8)); + final ModuleService moduleService = context.service(ModuleService.class); + final ScriptModule module = (ScriptModule) moduleService.createModule( + new ScriptInfo(context, script)); + moduleService.run(module, false, Collections.singletonMap("image", + null)).get(); + + // Now Jedi analyzes the script against its environment: numpy is known, + // and so is the type of the image parameter. + completeUntil(header + "np.ze", script.getPath(), "zeros"); + completeUntil(header + "image.sha", script.getPath(), "shape"); + report(header + "np.random.r", script.getPath(), "randn", "a"); + } + + @Test + public void testSignatureHelp() throws Exception { + final String code = "import json\njson.dumps(obj, "; + final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; + SignatureHelp help = SignatureHelp.NONE; + while (help.isEmpty() && System.currentTimeMillis() < deadline) { + help = completer().signatureHelp(new CompletionRequest(code, language, + null)); + if (help.update() != null) { + final SignatureHelp better = help.update().get(1, TimeUnit.MINUTES); + if (better != null) help = better; + } + } + assertEquals("dumps", help.signatures().get(0).callable().insertionText()); + assertFalse(help.signatures().get(0).callable().parameters().isEmpty()); + assertEquals(1, help.activeParameter()); + assertEquals(code.indexOf('('), help.callStart()); + } + + @Test + public void testNewDependency() throws Exception { + // A script declaring a dependency that is installed nowhere: its + // environment has not even been built. + final File script = new File(scriptDir, "overlaid.py"); + final String header = "#!appose-python\n# /// script\n" + + "# dependencies = [\"tifffile\"]\n# ///\nimport tifffile\n"; + Files.write(script.toPath(), (header + "x = 1\n").getBytes( + StandardCharsets.UTF_8)); + // Opening it prepares completion, including the dependency's overlay. + final String code = header + "tifffile.imr"; + completer().prepare(new CompletionRequest(code, code.length(), language, + null, null, script.getPath())); + completeUntil(code, script.getPath(), "imread"); + } + + // -- Helper methods -- + + private static ApposePythonCodeCompleter completer() { + return (ApposePythonCodeCompleter) service.getCompleterPlugin(language); + } + + /** Completes until the expected completion appears (environments build). */ + private static List completeUntil(final String code, + final String path, final String expected) + { + final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; + while (System.currentTimeMillis() < deadline) { + final List completions = complete(code, path); + if (completions.stream().anyMatch(c -> c.insertionText().equals( + expected))) return completions; + try { + Thread.sleep(1000); + } + catch (final InterruptedException exc) { + throw new RuntimeException(exc); + } + } + fail("No '" + expected + "' completion for: " + code); + return null; + } + + private static List complete(final String code, + final String path) + { + return service.complete(new CompletionRequest(code, code.length(), + language, null, null, path)).completions(); + } + + /** + * Prints how long completions take, once warm: for another keystroke of a + * word Jedi has answered (cached); and for a new word, how long the editor + * waits (barely: it answers at once), and how long until Jedi's completions + * arrive as an update. + */ + private static void report(final String code, final String path, + final String expected, final String more) throws Exception + { + completeUntil(code, path, expected); + long t0 = System.nanoTime(); + final int n = complete(code + more, path).size(); + final long cached = System.nanoTime() - t0; + final String other = code.substring(0, code.lastIndexOf('.') + 1) + "s"; + t0 = System.nanoTime(); + final CompletionResult result = service.complete(new CompletionRequest( + other, other.length(), language, null, null, path)); + final long wait = System.nanoTime() - t0; + final CompletionResult update = result.update() == null ? result : result + .update().get(1, TimeUnit.MINUTES); + final long arrived = System.nanoTime() - t0; + System.out.println("[completion timing] same word (cached): " + cached / + 1000000 + " ms, " + n + " completions; new word: editor waits " + wait / + 1000000 + " ms, Jedi's " + update.completions().size() + + " completions after " + arrived / 1000000 + " ms"); + } +} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngineTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngineTest.java index fc1799b..b1b443d 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngineTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngineTest.java @@ -146,39 +146,39 @@ public void testDoesNotOwnWrappedNDArray() { @Test public void testEnvName() { final String name = - ApposePythonScriptEngine.envName(new File("/a/b/cellcast.toml")); + ScriptEnvironment.envName(new File("/a/b/cellcast.toml")); assertTrue(name, name.matches("cellcast-[0-9a-f]{8}")); // Same file via a different path spelling. assertEquals(name, - ApposePythonScriptEngine.envName(new File("/a/c/../b/cellcast.toml"))); + ScriptEnvironment.envName(new File("/a/c/../b/cellcast.toml"))); // Same file name in a different directory. assertNotEquals(name, - ApposePythonScriptEngine.envName(new File("/a/cellcast.toml"))); + ScriptEnvironment.envName(new File("/a/cellcast.toml"))); - assertTrue(ApposePythonScriptEngine.envName(new File("/x/my env.v2.toml")) + assertTrue(ScriptEnvironment.envName(new File("/x/my env.v2.toml")) .matches("my_env_v2-[0-9a-f]{8}")); - assertTrue(ApposePythonScriptEngine.envName(new File("/x/requirements")) + assertTrue(ScriptEnvironment.envName(new File("/x/requirements")) .matches("requirements-[0-9a-f]{8}")); } @Test public void testResolveEnvFile() { final File abs = new File("/some/where/env.toml").getAbsoluteFile(); - assertEquals(abs, ApposePythonScriptEngine.resolveEnvFile(abs.getPath(), + assertEquals(abs, ScriptEnvironment.resolveEnvFile(abs.getPath(), "/other/script.py")); final File scriptDir = new File("/scripts").getAbsoluteFile(); - assertEquals(new File(scriptDir, "env.toml"), ApposePythonScriptEngine + assertEquals(new File(scriptDir, "env.toml"), ScriptEnvironment .resolveEnvFile("env.toml", new File(scriptDir, "s.py").getPath())); // Relative script path resolves relative to the working directory. assertEquals(new File("sub", "env.toml").getAbsoluteFile(), - ApposePythonScriptEngine.resolveEnvFile("env.toml", "sub/s.py") + ScriptEnvironment.resolveEnvFile("env.toml", "sub/s.py") .getAbsoluteFile()); - assertEquals(new File("env.toml"), ApposePythonScriptEngine + assertEquals(new File("env.toml"), ScriptEnvironment .resolveEnvFile("env.toml", null)); } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/_internal/InlineMetadataTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/_internal/InlineMetadataTest.java index 24eb893..dbc7884 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/_internal/InlineMetadataTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/_internal/InlineMetadataTest.java @@ -38,6 +38,7 @@ import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.List; import com.fasterxml.jackson.databind.JsonNode; @@ -82,6 +83,15 @@ public void testExtract() { "pytorch-gpu = \"*\""), metadata.trim()); } + @Test + public void testDependencies() { + assertEquals(Arrays.asList("appose>=0.12.0,<0.13", "numpy > 2"), + InlineMetadata.dependencies("requires-python = \">=3.12\"\n" + + "dependencies = [\"appose>=0.12.0,<0.13\", \"numpy > 2\"]\n")); + assertEquals(Collections.emptyList(), InlineMetadata.dependencies( + "requires-python = \">=3.12\"\n")); + } + @Test public void testExtractCRLF() { final String metadata = diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java new file mode 100644 index 0000000..3cc49fd --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java @@ -0,0 +1,109 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; + +import java.io.File; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.atomic.AtomicLong; + +import org.junit.Test; + +/** + * Tests {@link DependencyOverlays}. + * + * @author Gabriel Selzer + */ +public class DependencyOverlaysTest { + + private final AtomicLong now = new AtomicLong(); + private final List> installs = new ArrayList<>(); + + private final DependencyOverlays overlays = new DependencyOverlays((python, + dependencies, dir) -> { + synchronized (installs) { + installs.add(dependencies); + } + return dir.getPath(); + }, new File("overlays"), 1500, now::get); + + @Test + public void testWaitsForDependenciesToSettle() throws Exception { + final List deps = Arrays.asList("numpy", "scikit-ima"); + // Still being typed: nothing is installed... + assertNull(overlays.overlay("blur", null, deps, false)); + now.addAndGet(500); + assertNull(overlays.overlay("blur", null, deps, false)); + final List typed = Arrays.asList("numpy", "scikit-image"); + assertNull(overlays.overlay("blur", null, typed, false)); + now.addAndGet(1000); + assertNull(overlays.overlay("blur", null, typed, false)); + Thread.sleep(100); + assertEquals(Collections.emptyList(), installs); + // ...until the dependencies have stayed the same for a while. + now.addAndGet(1000); + assertNull(overlays.overlay("blur", null, typed, false)); + assertEquals(new File("overlays", "blur").getPath(), awaitOverlay("blur", + null, typed)); + assertEquals(Collections.singletonList(typed), installs); + } + + @Test + public void testSettledAndReused() throws Exception { + final List deps = Collections.singletonList("tifffile"); + // Known not to be changing (e.g. just opened): installed at once. + assertNull(overlays.overlay("read", "python", deps, true)); + final String overlay = awaitOverlay("read", "python", deps); + assertEquals(new File("overlays", "read").getPath(), overlay); + // Asked again: the same overlay, not installed again. + assertEquals(overlay, overlays.overlay("read", "python", deps, false)); + assertEquals(1, installs.size()); + // Nothing declared: no overlay. + assertNull(overlays.overlay("read", "python", Collections.emptyList(), + true)); + } + + /** Waits for the overlay to be ready. */ + private String awaitOverlay(final String env, final String python, + final List deps) throws InterruptedException + { + for (int i = 0; i < 100; i++) { + final String overlay = overlays.overlay(env, python, deps, false); + if (overlay != null) return overlay; + Thread.sleep(20); + } + return null; + } +} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxTest.java new file mode 100644 index 0000000..d368538 --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxTest.java @@ -0,0 +1,145 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.util.List; + +import org.junit.Test; +import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Context; +import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Kind; + +/** + * Tests {@link PythonSyntax}. + * + * @author Gabriel Selzer + */ +public class PythonSyntaxTest { + + @Test + public void testNamesInScope() { + final String code = "import numpy as np, os.path\n" + // + "from os import path as p, sep\n" + // + "x = 1\n" + // + "def f(a, b=2, *args, c: int = 3, **kw):\n" + // + " loc = 3\n" + // + " lo"; + final List names = PythonSyntax.names(code, code.length()); + for (final String name : new String[] { "loc", "a", "b", "args", "c", "kw", + "f", "x", "np", "os", "p", "sep" }) + { + assertTrue(name + " in " + names, names.contains(name)); + } + // Innermost scope first. + assertTrue(names.indexOf("loc") < names.indexOf("x")); + assertFalse(names.contains(PythonSyntax.CARET)); + } + + @Test + public void testScopeRules() { + // Class bodies are not visible from their methods. + final String cls = "class C:\n attr = 1\n def m(self):\n at"; + final List inMethod = PythonSyntax.names(cls, cls.length()); + assertTrue(inMethod.contains("self")); + assertTrue(inMethod.contains("C")); + assertFalse(inMethod.contains("attr")); + // Functions' locals are not visible outside them. + final String fn = "def g():\n inner = 1\nin"; + assertFalse(PythonSyntax.names(fn, fn.length()).contains("inner")); + // Comprehension, with, except and walrus targets. + final String comp = "[val for val in range(3) if va"; + assertTrue(PythonSyntax.names(comp, comp.length()).contains("val")); + final String blocks = "with open(f) as fh:\n try:\n pass\n" + + " except E as err:\n if (n := 2):\n e"; + final List names = PythonSyntax.names(blocks, blocks.length()); + assertTrue(names.toString(), names.contains("fh") && names.contains( + "err") && names.contains("n")); + // Unclosed brackets before the caret. + final String open = "def h(alpha):\n print(1, [2, al"; + assertTrue(PythonSyntax.names(open, open.length()).contains("alpha")); + } + + @Test + public void testAttributeContext() { + final String code = "import numpy as np\nnp.random.ra"; + final Context ctx = PythonSyntax.analyze(code, code.length()); + assertEquals(Kind.ATTRIBUTE, ctx.kind); + assertEquals("np.random", ctx.receiver); + assertEquals("ra", ctx.prefix); + assertEquals(code.length() - 2, ctx.replaceStart); + assertEquals("foo(x)[0]", PythonSyntax.analyze("y = foo(x)[0].", 14) + .receiver); + // Attributes used elsewhere on the same receiver. + final String img = "img.shape\nimg.dtype\nmyimg.size\nimg.sh"; + final Context used = PythonSyntax.analyze(img, img.length()); + assertTrue(used.attributesUsed.contains("shape")); + assertTrue(used.attributesUsed.contains("dtype")); + assertFalse(used.attributesUsed.contains("size")); + // A dot ending a line is not followed by an attribute on the next one. + final String nextLine = "a = numpy.ndarray(3)\na.\nprint(a.shape)\n"; + final Context dot = PythonSyntax.analyze(nextLine, nextLine.indexOf( + "a.\n") + 2); + assertTrue(dot.attributesUsed.contains("shape")); + assertFalse(dot.attributesUsed.toString(), dot.attributesUsed.contains( + "print")); + } + + @Test + public void testNoCompletion() { + for (final String code : new String[] { "s = 'np.ra", "# np.ra", + "x = 1 # np.", "d = \"\"\"doc\nmore np.", "#@ String na", "x = 1.", + "f\"{x}.y" }) + { + assertEquals(code, Kind.NONE, PythonSyntax.analyze(code, code.length()) + .kind); + } + // But after a closed string, completion works again. + final String closed = "s = 'a.b'\nx = s.up"; + assertEquals(Kind.ATTRIBUTE, PythonSyntax.analyze(closed, closed.length()) + .kind); + } + + @Test + public void testImports() { + assertEquals(Kind.IMPORT, PythonSyntax.analyze("import nu", 9).kind); + assertEquals(Kind.IMPORT, PythonSyntax.analyze("from scipy.nd", 13).kind); + assertEquals(Kind.IMPORT, PythonSyntax.analyze( + "from scipy import ndim", 22).kind); + } + + @Test + public void testUnicode() { + final String code = "s = '\u00e9t\u00e9'\nvaleur = 1\nval"; + assertTrue(PythonSyntax.names(code, code.length()).contains("valeur")); + } +} From 526729e49ab25e0e7ef8617c944fe3408a31876d Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Tue, 6 Oct 2026 12:51:50 -0500 Subject: [PATCH 02/37] Build Jedi's source from a ScriptDocument The parameter preamble becomes ApposePythonDialect, and where the script runs (its environment's Python, its overlay) an EnvironmentContext. The preamble now lists imports first and task last. Co-Authored-By: Claude Opus 5.5 --- .../python/ApposePythonCodeCompleter.java | 187 ++++++++---------- .../completion/ApposePythonDialect.java | 101 ++++++++++ .../python/ApposePythonCodeCompleterTest.java | 21 +- 3 files changed, 194 insertions(+), 115 deletions(-) create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java index 2aac6ae..56140cd 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java @@ -32,12 +32,9 @@ import java.io.File; import java.io.IOException; import java.io.InputStream; -import java.math.BigDecimal; -import java.math.BigInteger; import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.Arrays; -import java.util.Collection; import java.util.Collections; import java.util.HashMap; import java.util.HashSet; @@ -54,11 +51,8 @@ import java.util.function.Function; import java.util.function.Supplier; -import net.imglib2.RandomAccessibleInterval; - import org.apposed.appose.Appose; import org.apposed.appose.Environment; -import org.apposed.appose.NDArray; import org.apposed.appose.Service; import org.apposed.appose.util.Environments; import org.scijava.Context; @@ -68,6 +62,9 @@ import org.scijava.code.api.Completion; import org.scijava.code.api.CompletionRequest; import org.scijava.code.api.CompletionResult; +import org.scijava.code.api.EnvironmentContext; +import org.scijava.code.api.ScriptDialect; +import org.scijava.code.api.ScriptDocument; import org.scijava.code.api.SignatureHelp; import org.scijava.code.api.SignatureHelp.Fit; import org.scijava.code.api.Signatures; @@ -75,6 +72,7 @@ import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; +import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; import org.scijava.plugins.scripting.appose.python.completion.JediClient; import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax; @@ -150,6 +148,9 @@ protected boolean removeEldestEntry( */ private static final long OVERLAY_SETTLE = 1500; + /** How scripts see their parameters (as Python values) and {@code task}. */ + private static final ApposePythonDialect DIALECT = new ApposePythonDialect(); + @Override public boolean supports(final ScriptLanguage language) { return language instanceof ApposePythonScriptLanguage; @@ -165,10 +166,11 @@ public CompletionResult complete(final CompletionRequest request) { return CompletionResult.EMPTY; } - final Map params = parameterTypes(text); - final List local = localCompletions(ctx, params); - final CompletableFuture> answer = jediCompletions(request, - ctx, params); + final JediClient client = jediClient(); + final ScriptDocument doc = document(request, client != null, false); + final List local = localCompletions(ctx, doc.parameters()); + final CompletableFuture> answer = client == null ? null + : jediCompletions(client, request, ctx, doc); if (answer == null) return merge(ctx, null, local); // Use Jedi's answer if it is (almost) ready; otherwise answer without it @@ -195,15 +197,14 @@ public void prepare(final CompletionRequest request) { // analyze the script against its environment, in the background. final JediClient client = jediClient(); if (client == null) return; - final String text = request.text(); - final String python = scriptPython(text, request.path()); - final String overlay = overlay(text, request.path(), python, true); - final String source = preamble(parameterTypes(text)) + text; - final int line = count(source, '\n') + 1; - final int column = source.length() - source.lastIndexOf('\n') - 1; - client.answer("prepare\n" + request.path() + "\n" + text, - new JediClient.Query(source, line, column, request.path(), python, 0, - overlay)); + final ScriptDocument doc = document(request, true, true); + client.answer("prepare\n" + request.path() + "\n" + request.text(), query( + doc, request.text().length(), request.path(), 0)); + } + + @Override + public ScriptDialect dialect() { + return DIALECT; } @Override @@ -220,19 +221,12 @@ public SignatureHelp signatureHelp(final CompletionRequest request) { final JediClient client = jediClient(); if (client == null) return SignatureHelp.NONE; - final String python = scriptPython(text, request.path()); - final String overlay = overlay(text, request.path(), python, false); - final String preamble = preamble(parameterTypes(text)); - final String before = preamble + text.substring(0, caret); - final int line = count(before, '\n') + 1; - final int column = before.length() - before.lastIndexOf('\n') - 1; - final JediClient.Query query = new JediClient.Query(preamble + text, line, - column, request.path(), python, 0, overlay); + final ScriptDocument doc = document(request, true, false); + final JediClient.Query query = query(doc, caret, request.path(), 0); final int active = args.size() - 1; // NB: Where Jedi cannot tell the argument, count the commas. final CompletableFuture help = client.signatures( - "signatures\n" + python + "\n" + overlay + "\n" + request.path() + - "\n" + before, query).handle((h, error) -> h == null ? null + "signatures\n" + key(doc, caret, request.path()), query).handle((h, error) -> h == null ? null : new SignatureHelp(h.signatures(), h.activeParameter() >= 0 ? h .activeParameter() : active, open)); try { @@ -348,7 +342,7 @@ private static List localCompletions( : p.getValue() + " (script parameter)").build()); } if (matches("task", prefix)) out.add(Completion.builder("task").kind( - Completion.Kind.VARIABLE).summary("appose.python_worker.Task").build()); + Completion.Kind.VARIABLE).summary(ApposePythonDialect.TASK_TYPE).build()); for (final String name : PythonSyntax.BUILTINS) { if (matches(name, prefix)) out.add(Completion.builder(name).kind( Completion.Kind.OTHER).summary("builtin").build()); @@ -365,32 +359,70 @@ private static List localCompletions( * * @return Jedi's answer, or pending answer; or null if Jedi is unavailable. */ - private CompletableFuture> jediCompletions( - final CompletionRequest request, final PythonSyntax.Context ctx, - final Map params) + private static CompletableFuture> jediCompletions( + final JediClient client, final CompletionRequest request, + final PythonSyntax.Context ctx, final ScriptDocument doc) { - final JediClient client = jediClient(); - if (client == null) return null; - final String text = request.text(); - final String python = scriptPython(text, request.path()); - final String overlay = overlay(text, request.path(), python, false); - // Ask about the first character typed, if any: later keystrokes of the // same word then reuse the answer (filtered), while it stays short. final int caret = ctx.replaceStart + Math.min(1, ctx.prefix.length()); - final String preamble = preamble(params); - final String before = preamble + text.substring(0, caret); - final int line = count(before, '\n') + 1; - final int column = before.length() - before.lastIndexOf('\n') - 1; // NB: Signatures are expensive, so ask for them only once a word is // begun, which narrows the completions down. Before that, functions are // offered by name. - final JediClient.Query query = new JediClient.Query(preamble + text, line, - column, request.path(), python, ctx.prefix.isEmpty() ? 0 : SIGNATURES, - overlay); - final String key = python + "\n" + overlay + "\n" + request.path() + - "\n" + before; - return client.answer(key, query); + final JediClient.Query query = query(doc, caret, request.path(), ctx.prefix + .isEmpty() ? 0 : SIGNATURES); + return client.answer(key(doc, caret, request.path()), query); + } + + /** + * Gets the script as plain Python, for Jedi: its parameters declared, and + * (if asked) where it runs: its environment's Python, and its overlay. + * + * @param environment Whether to find where the script runs. + * @param settled Whether the dependencies are known not to be changing. + */ + private ScriptDocument document(final CompletionRequest request, + final boolean environment, final boolean settled) + { + final String text = request.text(); + EnvironmentContext env = null; + if (environment) { + final String python = scriptPython(text, request.path()); + final String overlay = overlay(text, request.path(), python, settled); + env = new EnvironmentContext(python, overlay == null ? null + : Collections.singletonList(overlay), null); + } + final CodeCompletionService service = service( + CodeCompletionService.class); + final Map> params = service == null ? Collections + .emptyMap() : service.scriptParameters(text); + return ScriptDocument.of(text, params, DIALECT, env); + } + + /** Asks Jedi about the given offset of the script. */ + private static JediClient.Query query(final ScriptDocument doc, + final int offset, final String path, final int flags) + { + final String before = doc.text().substring(0, doc.map().toTransformed( + offset)); + final int line = count(before, '\n') + 1; + final int column = before.length() - before.lastIndexOf('\n') - 1; + final EnvironmentContext env = doc.environment(); + final String overlay = env.searchPaths().isEmpty() ? null : env + .searchPaths().get(0); + return new JediClient.Query(doc.text(), line, column, path, env + .interpreter(), flags, overlay); + } + + /** Identifies a question about the given offset, for reusing answers. */ + private static String key(final ScriptDocument doc, final int offset, + final String path) + { + final EnvironmentContext env = doc.environment(); + final String overlay = env.searchPaths().isEmpty() ? null : env + .searchPaths().get(0); + return env.interpreter() + "\n" + overlay + "\n" + path + "\n" + doc + .text().substring(0, doc.map().toTransformed(offset)); } /** @@ -459,63 +491,6 @@ private static String installOverlay(final ResidentWorkerService workers, } } - /** - * A line declaring the types of the script's parameters, for Jedi, e.g. - * {@code import numpy; image: numpy.ndarray; sigma: float}. - */ - static String preamble(final Map params) { - final StringBuilder sb = new StringBuilder( - "import appose.python_worker; task: appose.python_worker.Task"); - if (params.containsValue("numpy.ndarray")) sb.append("; import numpy"); - for (final Map.Entry p : params.entrySet()) { - if (p.getValue() == null) continue; - sb.append("; ").append(p.getKey()).append(": ").append(p.getValue()); - } - return sb.append('\n').toString(); - } - - /** - * Gets the script's parameters, with the Python types they have when the - * script runs (see {@link ApposePythonScriptEngine#marshal}), or null types - * if unknown. - */ - private Map parameterTypes(final String text) { - final Map params = new LinkedHashMap<>(); - final Context context = getContext(); - final CodeCompletionService service = context == null ? null : context - .getService(CodeCompletionService.class); - if (service == null) return params; - service.scriptParameters(text).forEach((name, type) -> params.put(name, - pythonType(type))); - return params; - } - - /** The Python type a Java value of the given type becomes in a script. */ - static String pythonType(final Class type) { - if (type == null) return null; - if (RandomAccessibleInterval.class.isAssignableFrom(type) || NDArray.class - .isAssignableFrom(type)) return "numpy.ndarray"; - if (type == boolean.class || type == Boolean.class) return "bool"; - if (type == int.class || type == long.class || type == short.class || - type == byte.class || type == Integer.class || type == Long.class || - type == Short.class || type == Byte.class || type == BigInteger.class) - { - return "int"; - } - if (type == float.class || type == double.class || type == Float.class || - type == Double.class || type == BigDecimal.class) return "float"; - if (CharSequence.class.isAssignableFrom(type) || type == char.class || - type == Character.class || type.isEnum() || type == File.class) - { - return "str"; - } - if (type.isArray() || Collection.class.isAssignableFrom(type)) { - return "list"; - } - if (Map.class.isAssignableFrom(type)) return "dict"; - return null; - } - /** * Gets the Python executable of the script's environment, if it has been * built already: in this session (e.g. by running the script), or in an diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java new file mode 100644 index 0000000..837dd92 --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -0,0 +1,101 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import java.io.File; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.List; +import java.util.Map; + +import net.imglib2.RandomAccessibleInterval; + +import org.apposed.appose.NDArray; +import org.scijava.code.api.ScriptDialect; + +/** + * How Python scripts run via Appose see their script parameters: as the + * Python values the Java values become (see + * {@code ApposePythonScriptEngine#marshal}), plus Appose's {@code task}. + * + * @author Gabriel Selzer + */ +public final class ApposePythonDialect implements ScriptDialect { + + /** The type of the {@code task} variable scripts can use. */ + public static final String TASK_TYPE = "appose.python_worker.Task"; + + @Override + public String runtimeType(final Class type) { + if (type == null) return null; + if (RandomAccessibleInterval.class.isAssignableFrom(type) || NDArray.class + .isAssignableFrom(type)) return "numpy.ndarray"; + if (type == boolean.class || type == Boolean.class) return "bool"; + if (type == int.class || type == long.class || type == short.class || + type == byte.class || type == Integer.class || type == Long.class || + type == Short.class || type == Byte.class || type == BigInteger.class) + { + return "int"; + } + if (type == float.class || type == double.class || type == Float.class || + type == Double.class || type == BigDecimal.class) return "float"; + if (CharSequence.class.isAssignableFrom(type) || type == char.class || + type == Character.class || type.isEnum() || type == File.class) + { + return "str"; + } + if (type.isArray() || Collection.class.isAssignableFrom(type)) { + return "list"; + } + if (Map.class.isAssignableFrom(type)) return "dict"; + return null; + } + + @Override + public String declare(final String name, final String type) { + return name + ": " + type; + } + + @Override + public List imports(final Collection types) { + final List imports = new ArrayList<>(); + imports.add("import appose.python_worker"); + if (types.contains("numpy.ndarray")) imports.add("import numpy"); + return imports; + } + + @Override + public List extraDeclarations() { + return Collections.singletonList(declare("task", TASK_TYPE)); + } +} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java index 56f244b..a392d3c 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java @@ -55,6 +55,7 @@ import org.scijava.code.api.Completion; import org.scijava.code.api.CompletionRequest; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; +import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; import org.scijava.plugins.scripting.appose.python.completion.JediClient; import org.scijava.script.ScriptLanguage; import org.scijava.script.ScriptService; @@ -66,6 +67,8 @@ */ public class ApposePythonCodeCompleterTest { + private static final ApposePythonDialect DIALECT = new ApposePythonDialect(); + private Context context; private ScriptLanguage language; private CodeCompletionService service; @@ -362,8 +365,8 @@ public void testJediQuery() { complete(code); // The script's parameters are typed in a preamble, so Jedi knows them. assertTrue(query[0].source.startsWith( - "import appose.python_worker; task: appose.python_worker.Task; " + - "import numpy; image: numpy.ndarray; sigma: float\n")); + "import appose.python_worker; import numpy; image: numpy.ndarray; " + + "sigma: float; task: appose.python_worker.Task\n")); // The caret is after the word's first character, shifted by the preamble. assertEquals(4, query[0].line); assertEquals("image.s".length(), query[0].column); @@ -484,14 +487,14 @@ public void testEnvironmentFromEarlierSession() throws Exception { @Test public void testPythonTypes() { - assertEquals("numpy.ndarray", ApposePythonCodeCompleter.pythonType( + assertEquals("numpy.ndarray", DIALECT.runtimeType( Img.class)); - assertEquals("float", ApposePythonCodeCompleter.pythonType(double.class)); - assertEquals("int", ApposePythonCodeCompleter.pythonType(Long.class)); - assertEquals("str", ApposePythonCodeCompleter.pythonType(String.class)); - assertEquals("bool", ApposePythonCodeCompleter.pythonType(boolean.class)); - assertEquals("list", ApposePythonCodeCompleter.pythonType(int[].class)); - assertNull(ApposePythonCodeCompleter.pythonType(Context.class)); + assertEquals("float", DIALECT.runtimeType(double.class)); + assertEquals("int", DIALECT.runtimeType(Long.class)); + assertEquals("str", DIALECT.runtimeType(String.class)); + assertEquals("bool", DIALECT.runtimeType(boolean.class)); + assertEquals("list", DIALECT.runtimeType(int[].class)); + assertNull(DIALECT.runtimeType(Context.class)); } @Test From e2b486c7c538a4630df07e829980ae704a4c53dd Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Tue, 6 Oct 2026 13:09:21 -0500 Subject: [PATCH 03/37] Complete via jedi-language-server The Jedi tasks (JediClient, jedi_complete.py) give way to jedi-language-server, run from the tooling environment via code-lsp's LspClient; PythonLspClient reads its Python signatures. Tree-sitter's completions, dependency overlays and script environments stay as they were: the overlay and the environment's Python become the server's extra path and environment. Co-Authored-By: Claude Opus 5.5 --- README.md | 26 +- pom.xml | 5 + .../python/ApposePythonCodeCompleter.java | 391 +++++------------- .../appose/python/completion/JediClient.java | 288 ------------- .../completion/JediLanguageServerSpec.java | 126 ++++++ .../python/completion/PythonLspClient.java | 146 +++++++ .../python/completion/PythonSyntax.java | 2 +- .../appose/python/completion/jedi_complete.py | 172 -------- .../python/ApposePythonCodeCompleterTest.java | 383 ++++++++--------- ...ApposePythonCompletionIntegrationTest.java | 18 +- .../completion/PythonLspClientTest.java | 75 ++++ 11 files changed, 659 insertions(+), 973 deletions(-) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediClient.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClient.java delete mode 100644 src/main/resources/org/scijava/plugins/scripting/appose/python/completion/jedi_complete.py create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClientTest.java diff --git a/README.md b/README.md index 2421f0f..b19535e 100644 --- a/README.md +++ b/README.md @@ -143,15 +143,22 @@ The Script Editor completes code in these scripts, in two layers: functions, classes, imports, parameters of the enclosing function), the script's `#@` parameters, `task`, builtins and keywords. This comes from parsing the script with [tree-sitter](https://tree-sitter.github.io/). -* **With [Jedi](https://jedi.readthedocs.io/):** members of objects and +* **With a language server, + [jedi-language-server](https://github.com/pappasam/jedi-language-server) + (built on [Jedi](https://jedi.readthedocs.io/)):** members of objects and modules, function signatures (with parameter names and types) and - documentation. Jedi runs in a small environment of its own - (`appose-python-completion`: Python, Jedi and Appose), which is built in - the background the first time you complete code, and takes a while only - then. + documentation. The server runs in a small environment of its own + (`appose-python-completion`: Python, jedi-language-server, Appose and uv), + which is built in the background the first time you complete code, and + takes a while only then. The server talks to the editor via the + [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) + (see `scijava-code-lsp`); it sees the script as plain Python, with its `#@` + parameters declared. One server runs per script environment; servers + unused for 10 minutes stop. Their error output goes to + `appose-python-completion-server.log`, next to Appose's environments. Once a script's environment is built, by running the script (in this session -or an earlier one), Jedi analyzes the script against it, so that the packages +or an earlier one), the server analyzes the script against it, so that the packages installed there are known too. Packages the script declares (in its `# /// script` block) but its environment lacks, e.g. a dependency you just added, are installed for completion only, into a separate directory, a moment @@ -162,6 +169,7 @@ Completion never builds a script's environment itself, and never disturbs its worker. Script parameters are known by the Python types they have at run time, e.g. images as NumPy arrays. -Typing never waits for Jedi: if its answer for a word is not ready within a -moment (the first answer for a package can take a few seconds), the instant -completions show, and Jedi's are used once ready. +Typing never waits for the server: if its answer for a word is not ready +within a moment (the first answer for a package can take a few seconds), the +instant completions show, and the server's are used once ready. Questions +that are no longer of interest (e.g. as you type on) are cancelled. diff --git a/pom.xml b/pom.xml index 132480f..878cc65 100644 --- a/pom.xml +++ b/pom.xml @@ -115,6 +115,11 @@ scijava-code-api ${scijava-code-api.version} + + org.scijava + scijava-code-lsp + ${scijava-code-api.version} + diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java index 56140cd..7edbb32 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java @@ -48,8 +48,6 @@ import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; -import java.util.function.Function; -import java.util.function.Supplier; import org.apposed.appose.Appose; import org.apposed.appose.Environment; @@ -66,15 +64,19 @@ import org.scijava.code.api.ScriptDialect; import org.scijava.code.api.ScriptDocument; import org.scijava.code.api.SignatureHelp; -import org.scijava.code.api.SignatureHelp.Fit; import org.scijava.code.api.Signatures; +import org.scijava.code.lsp.LspClient; +import org.scijava.code.lsp.LspServers; +import org.scijava.log.LogService; +import org.scijava.log.StderrLogService; import org.scijava.plugin.Plugin; import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; -import org.scijava.plugins.scripting.appose.python.completion.JediClient; +import org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec; +import org.scijava.plugins.scripting.appose.python.completion.PythonLspClient; import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax; import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Kind; import org.scijava.script.ScriptLanguage; @@ -86,12 +88,13 @@ *
  • {@link PythonSyntax} (tree-sitter, in process): instantly, with no * Python needed, names in scope, the script's parameters, builtins and * keywords.
    • - *
  • Jedi, in a small tooling environment of its own, built in the - * background on first use: members of objects and modules, signatures and - * documentation. Once the script's own environment is built (by running the - * script), Jedi analyzes the script against it, so its packages are known. - * Jedi's answers are used if they come within a short time budget; typing - * never waits for them.
    • + *
  • jedi-language-server (a language server built on Jedi), in a small + * tooling environment of its own, built in the background on first use: + * members of objects and modules, signatures and documentation. Once the + * script's own environment is built (by running the script), the server + * analyzes the script against it, so its packages are known. Its answers are + * used if they come within a short time budget; typing never waits for + * them.
    • * *

    * Completion never builds a script's environment, and never disturbs its @@ -103,24 +106,27 @@ @Plugin(type = CodeCompleterPlugin.class, name = "appose-python") public class ApposePythonCodeCompleter extends AbstractCodeCompleterPlugin { - /** Name of the tooling environment in which Jedi runs. */ + /** Name of the tooling environment in which the language server runs. */ static final String TOOLING_ENV = "appose-python-completion"; /** - * How long to wait for Jedi before answering without it (and updating the - * answer once Jedi's arrives), in milliseconds. + * How long to wait for the language server before answering without it + * (and updating the answer once its arrives), in milliseconds. */ private static final long BUDGET = Long.getLong( "scijava.appose.completion.budget", 25); - /** How long to wait for a completion's documentation, in milliseconds. */ - private static final long DESCRIBE_BUDGET = 1000; - /** How many completions get signatures, once a word is begun. */ private static final int SIGNATURES = 30; - /** How many signatures (overloads) each gets, at most. */ - private static final int OVERLOADS = 8; + /** How long a language server may go unused before it stops, in ms. */ + private static final long SERVER_IDLE = 10 * 60 * 1000; + + /** + * How many language servers may run at once (one per script environment + * and overlay). + */ + private static final int MAX_SERVERS = 3; /** Python executables of script environments, by script and metadata. */ private final Map pythons = new LinkedHashMap( @@ -135,10 +141,10 @@ protected boolean removeEldestEntry( } }; - /** Task scripts (see jedi_complete.py, overlay.py), by resource name. */ + /** Task scripts (see overlay.py), by resource name. */ private static final Map SCRIPTS = new HashMap<>(); - private JediClient jedi; + private LspClient lsp; private DependencyOverlays overlays; @@ -166,40 +172,55 @@ public CompletionResult complete(final CompletionRequest request) { return CompletionResult.EMPTY; } - final JediClient client = jediClient(); + final LspClient client = lspClient(); final ScriptDocument doc = document(request, client != null, false); final List local = localCompletions(ctx, doc.parameters()); - final CompletableFuture> answer = client == null ? null - : jediCompletions(client, request, ctx, doc); - if (answer == null) return merge(ctx, null, local); + if (client == null) return merge(ctx, null, local); + // Ask about the first character typed, if any: later keystrokes of the + // same word then reuse the answer (filtered), while it stays short. + // NB: Signatures are expensive, so ask for them only once a word is + // begun, which narrows the completions down. Before that, functions are + // offered by name. + final CompletableFuture> answer = client.complete(doc, + uri(request), ctx.replaceStart + Math.min(1, ctx.prefix.length()), + ctx.prefix.isEmpty() ? 0 : SIGNATURES); - // Use Jedi's answer if it is (almost) ready; otherwise answer without it - // at once, and announce the better result to follow. + // Use the server's answer if it is (almost) ready; otherwise answer + // without it at once, and announce the better result to follow. try { return merge(ctx, answer.get(BUDGET, TimeUnit.MILLISECONDS), local); } catch (final TimeoutException exc) { - return merge(ctx, null, local).withUpdate(answer.handle((jedi, - error) -> jedi == null ? null : merge(ctx, jedi, local))); + return merge(ctx, null, local).withUpdate(answer.handle((server, + error) -> server == null ? null : merge(ctx, server, local))); } catch (final InterruptedException exc) { Thread.currentThread().interrupt(); } catch (final ExecutionException | CancellationException exc) { - // NB: Jedi failed; the next request asks again. + // NB: The server failed (and said why in the log); the next request + // asks again. } return merge(ctx, null, local); } @Override public void prepare(final CompletionRequest request) { - // Warm up: start Jedi (building its environment, if needed), and have it - // analyze the script against its environment, in the background. - final JediClient client = jediClient(); + // Warm up: start the language server (building its environment, if + // needed), and have it analyze the script against its environment, in + // the background. + final LspClient client = lspClient(); if (client == null) return; - final ScriptDocument doc = document(request, true, true); - client.answer("prepare\n" + request.path() + "\n" + request.text(), query( - doc, request.text().length(), request.path(), 0)); + client.prepare(document(request, true, true), uri(request)); + } + + @Override + public void closed(final CompletionRequest request) { + final LspClient client; + synchronized (this) { + client = lsp; + } + if (client != null) client.close(uri(request)); } @Override @@ -218,17 +239,13 @@ public SignatureHelp signatureHelp(final CompletionRequest request) { if (open < 0) return SignatureHelp.NONE; final List args = Signatures.arguments(text, open, caret); if (args == null) return SignatureHelp.NONE; - final JediClient client = jediClient(); + final LspClient client = lspClient(); if (client == null) return SignatureHelp.NONE; - final ScriptDocument doc = document(request, true, false); - final JediClient.Query query = query(doc, caret, request.path(), 0); - final int active = args.size() - 1; - // NB: Where Jedi cannot tell the argument, count the commas. - final CompletableFuture help = client.signatures( - "signatures\n" + key(doc, caret, request.path()), query).handle((h, error) -> h == null ? null - : new SignatureHelp(h.signatures(), h.activeParameter() >= 0 ? h - .activeParameter() : active, open)); + // NB: Where the server cannot tell the argument, count the commas. + final CompletableFuture help = client.signatureHelp( + document(request, true, false), uri(request), caret, open, args.size() - + 1).handle((h, error) -> h); try { final SignatureHelp h = help.get(BUDGET, TimeUnit.MILLISECONDS); return h == null ? SignatureHelp.NONE : h; @@ -240,43 +257,23 @@ public SignatureHelp signatureHelp(final CompletionRequest request) { Thread.currentThread().interrupt(); } catch (final ExecutionException | CancellationException exc) { - // NB: Jedi failed. + // NB: The server failed. } return SignatureHelp.NONE; } /** - * Adapts the "signatures" task output to the shape of "completions" items - * (one item per signature), for {@link #toCompletions}. - */ - @SuppressWarnings("unchecked") - private static List signatureItems(final Object output) { - final List items = new ArrayList<>(); - if (!(output instanceof List)) return items; - for (final Object o : (List) output) { - if (!(o instanceof Map)) continue; - final Map sig = (Map) o; - final Map item = new HashMap<>(); - item.put("name", sig.get("name")); - item.put("type", "function"); - item.put("signatures", Collections.singletonList(sig)); - items.add(item); - } - return items; - } - - /** - * Merges Jedi's completions (if any) with the local ones: Jedi's first, - * since they carry types, signatures and documentation; then whatever only - * the local analysis knows. + * Merges the server's completions (if any) with the local ones: the + * server's first, since they carry types, signatures and documentation; + * then whatever only the local analysis knows. */ private static CompletionResult merge(final PythonSyntax.Context ctx, - final List jedi, final List local) + final List server, final List local) { final List out = new ArrayList<>(); final Set seen = new HashSet<>(); - if (jedi != null) { - for (final Completion c : jedi) { + if (server != null) { + for (final Completion c : server) { if (!matches(c.insertionText(), ctx.prefix)) continue; if (ctx.kind == Kind.IMPORT) { // NB: An import names things rather than calling them: one plain @@ -299,9 +296,9 @@ synchronized void setOverlays(final DependencyOverlays overlays) { this.overlays = overlays; } - /** Sets the Jedi client, e.g. to use a fake one in tests. */ - synchronized void setJediClient(final JediClient jedi) { - this.jedi = jedi; + /** Sets the language server client, e.g. to use a fake one in tests. */ + synchronized void setLspClient(final LspClient lsp) { + this.lsp = lsp; } // -- Helper methods -- @@ -355,27 +352,8 @@ private static List localCompletions( } /** - * Asks Jedi for completions. - * - * @return Jedi's answer, or pending answer; or null if Jedi is unavailable. - */ - private static CompletableFuture> jediCompletions( - final JediClient client, final CompletionRequest request, - final PythonSyntax.Context ctx, final ScriptDocument doc) - { - // Ask about the first character typed, if any: later keystrokes of the - // same word then reuse the answer (filtered), while it stays short. - final int caret = ctx.replaceStart + Math.min(1, ctx.prefix.length()); - // NB: Signatures are expensive, so ask for them only once a word is - // begun, which narrows the completions down. Before that, functions are - // offered by name. - final JediClient.Query query = query(doc, caret, request.path(), ctx.prefix - .isEmpty() ? 0 : SIGNATURES); - return client.answer(key(doc, caret, request.path()), query); - } - - /** - * Gets the script as plain Python, for Jedi: its parameters declared, and + * Gets the script as plain Python, for the server: its parameters + * declared, and * (if asked) where it runs: its environment's Python, and its overlay. * * @param environment Whether to find where the script runs. @@ -399,30 +377,9 @@ private ScriptDocument document(final CompletionRequest request, return ScriptDocument.of(text, params, DIALECT, env); } - /** Asks Jedi about the given offset of the script. */ - private static JediClient.Query query(final ScriptDocument doc, - final int offset, final String path, final int flags) - { - final String before = doc.text().substring(0, doc.map().toTransformed( - offset)); - final int line = count(before, '\n') + 1; - final int column = before.length() - before.lastIndexOf('\n') - 1; - final EnvironmentContext env = doc.environment(); - final String overlay = env.searchPaths().isEmpty() ? null : env - .searchPaths().get(0); - return new JediClient.Query(doc.text(), line, column, path, env - .interpreter(), flags, overlay); - } - - /** Identifies a question about the given offset, for reusing answers. */ - private static String key(final ScriptDocument doc, final int offset, - final String path) - { - final EnvironmentContext env = doc.environment(); - final String overlay = env.searchPaths().isEmpty() ? null : env - .searchPaths().get(0); - return env.interpreter() + "\n" + overlay + "\n" + path + "\n" + doc - .text().substring(0, doc.map().toTransformed(offset)); + /** The URI of the script, for the language server. */ + private static String uri(final CompletionRequest request) { + return LspClient.uri(request.path(), "script.py"); } /** @@ -610,91 +567,27 @@ private static String findPython(final List dirs) { return null; } - /** Gets the Jedi client, creating it on first use. */ - private synchronized JediClient jediClient() { - if (jedi != null) return jedi; + /** Gets the language server client, creating it on first use. */ + private synchronized LspClient lspClient() { + if (lsp != null) return lsp; final ResidentWorkerService workers = service( ResidentWorkerService.class); if (workers == null) return null; - final JediClient[] client = { null }; - client[0] = new JediClient(new JediClient.Backend() { - - @Override - public List complete(final String key, - final JediClient.Query query) - { - final Object output = runJedi(workers, "complete", key, query, null) - .get("completions"); - // NB: Documentation is fetched only for completions looked at. - return toCompletions(output, name -> () -> client[0].describe(key, - query, name, DESCRIBE_BUDGET)); - } - - @Override - public SignatureHelp signatures(final String key, - final JediClient.Query query) - { - final Map outputs = runJedi(workers, "signatures", key, - query, null); - final List signatures = new ArrayList<>(); - for (final Completion c : toCompletions(signatureItems(outputs.get( - "signatures")))) - { - // NB: Python's conversions are not rated yet: Jedi's order. - signatures.add(new SignatureHelp.Signature(c, Fit.UNKNOWN)); - } - final Object index = outputs.get("index"); - return new SignatureHelp(signatures, index instanceof Number - ? ((Number) index).intValue() : -1, -1); - } - - @Override - public String describe(final String key, final JediClient.Query query, - final String name) - { - final Object doc = runJedi(workers, "describe", key, query, name).get( - "docstring"); - return doc instanceof String && !((String) doc).isEmpty() ? "
    " +
-					escapeHTML((String) doc) + "
    " : null; - } - }); - jedi = client[0]; - return jedi; - } - - /** - * Runs a Jedi task (see jedi_complete.py) on the tooling environment's - * worker, building the environment first, if needed. Called on the Jedi - * client's background thread. - * - * @return The task's outputs. - */ - private static Map runJedi( - final ResidentWorkerService workers, final String op, final String key, - final JediClient.Query query, final String name) - { - final ResidentWorker worker = toolingWorker(workers); - final Map inputs = new HashMap<>(); - inputs.put("op", op); - inputs.put("key", key); - inputs.put("source", query.source); - inputs.put("line", query.line); - inputs.put("column", query.column); - inputs.put("path", query.path); - inputs.put("python", query.python); - inputs.put("max_signatures", query.signatures); - inputs.put("max_overloads", OVERLOADS); - inputs.put("name", name); - inputs.put("overlay", query.overlay); - try { - final Service.Task task = worker.task(resource( - "completion/jedi_complete.py"), inputs); - task.waitFor(); - return task.outputs; - } - catch (final Exception exc) { - throw new IllegalStateException("Jedi failed", exc); - } + LogService log = service(LogService.class); + if (log == null) log = new StderrLogService(); + final File envs = new File(Environments.apposeEnvsDir()); + final JediLanguageServerSpec spec = new JediLanguageServerSpec(() -> { + // NB: Builds the tooling environment, if needed (but starts no worker). + final String python = pythonExecutable(toolingWorker(workers) + .environment().get()); + if (python == null) throw new IOException("No Python in " + + TOOLING_ENV); + return python; + }, new File(envs, TOOLING_ENV + "-workspace"), new File(envs, TOOLING_ENV + + "-server.log")); + lsp = new PythonLspClient(new LspServers(spec, log, SERVER_IDLE, + MAX_SERVERS)); + return lsp; } /** Gets the tooling environment's worker, building it first if needed. */ @@ -707,8 +600,8 @@ private static ResidentWorker toolingWorker( } /** - * The tooling environment: Python, Jedi and Appose; and uv, to install - * packages for {@link DependencyOverlays}. + * The tooling environment: Python, Jedi, jedi-language-server and Appose; + * and uv, to install packages for {@link DependencyOverlays}. */ static String toolingConfig() { return "[workspace]\n" + // @@ -722,87 +615,11 @@ static String toolingConfig() { "\n" + // "[pypi-dependencies]\n" + // "appose = \"*\"\n" + // + "jedi-language-server = \">=0.47\"\n" + // "uv = \"*\"\n"; } - /** Converts the task's output (see jedi_complete.py) to completions. */ - static List toCompletions(final Object output) { - return toCompletions(output, null); - } - - /** - * Converts the task's output (see jedi_complete.py) to completions. - * - * @param docs Supplies documentation, by completion name, on demand; or - * null if there is none. - */ - @SuppressWarnings("unchecked") - static List toCompletions(final Object output, - final Function> docs) - { - if (!(output instanceof List)) return Collections.emptyList(); - final List out = new ArrayList<>(); - for (final Object o : (List) output) { - if (!(o instanceof Map)) continue; - final Map item = (Map) o; - final String name = (String) item.get("name"); - if (name == null) continue; - final String type = (String) item.get("type"); - final String doc = (String) item.get("docstring"); - final Supplier description = doc != null ? () -> "
    " +
-				escapeHTML(doc) + "
    " : docs != null ? docs.apply(name) : null; - final List signatures = item.get( - "signatures") instanceof List ? (List) item.get("signatures") - : Collections.emptyList(); - if (signatures.isEmpty()) { - // NB: Without a signature, even a function is offered by name only. - final Completion.Kind kind = kind(type); - out.add(Completion.builder(name).kind(kind == Completion.Kind.METHOD - ? Completion.Kind.OTHER : kind).summary((String) item.get( - "description")).lazyDescription(description).build()); - continue; - } - // One completion per signature, e.g. for each typing.overload. - for (final Object s : signatures) { - final Map sig = (Map) s; - final List params = new ArrayList<>(); - for (final Object p : (List) sig.get("params")) { - final List param = (List) p; - params.add(new Completion.Parameter((String) param.get(0), - (String) param.get(1))); - } - out.add(Completion.builder(name).kind(Completion.Kind.METHOD) - .parameters(params).returnType((String) sig.get("returns")).summary( - (String) item.get("description")).lazyDescription(description) - .build()); - } - } - return out; - } - - private static Completion.Kind kind(final String jediType) { - if (jediType == null) return Completion.Kind.OTHER; - switch (jediType) { - case "function": - return Completion.Kind.METHOD; - case "class": - return Completion.Kind.CLASS; - case "module": - return Completion.Kind.PACKAGE; - case "property": - return Completion.Kind.FIELD; - case "keyword": - return Completion.Kind.KEYWORD; - case "instance": - case "statement": - case "param": - return Completion.Kind.VARIABLE; - default: - return Completion.Kind.OTHER; - } - } - - /** Gets a task script (e.g. jedi_complete.py), by resource name. */ + /** Gets a task script (e.g. overlay.py), by resource name. */ private static String resource(final String name) throws IOException { synchronized (SCRIPTS) { final String known = SCRIPTS.get(name); @@ -836,16 +653,4 @@ private static boolean matches(final String name, final String prefix) { return name.toLowerCase(Locale.ROOT).startsWith(prefix.toLowerCase( Locale.ROOT)) && !name.equals(prefix); } - - private static int count(final String s, final char c) { - int n = 0; - for (int i = 0; i < s.length(); i++) { - if (s.charAt(i) == c) n++; - } - return n; - } - - private static String escapeHTML(final String s) { - return s.replace("&", "&").replace("<", "<").replace(">", ">"); - } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediClient.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediClient.java deleted file mode 100644 index 6098a1b..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediClient.java +++ /dev/null @@ -1,288 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.util.HashMap; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.ExecutionException; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.concurrent.TimeUnit; -import java.util.concurrent.TimeoutException; -import java.util.function.Function; - -import org.scijava.code.api.Completion; -import org.scijava.code.api.SignatureHelp; - -/** - * Asks Jedi for completions without ever making the caller wait long: the - * editor calls completers on its UI thread, on every keystroke, while Jedi's - * answers can take a while (its first for a script's environment, seconds; - * its tooling environment, minutes to build). - *

    - * Requests run one at a time, in the background, and their answers come as - * futures. Answers are cached by a key, which callers choose so that - * keystrokes within a word share one answer. A new query supersedes queued - * ones that have not started. - *

    - * - * @author Gabriel Selzer - */ -public final class JediClient { - - /** What to ask Jedi. */ - public static final class Query { - - /** The source to analyze. */ - public final String source; - - /** The caret's line (1-based) in {@link #source}. */ - public final int line; - - /** The caret's column (0-based) in {@link #source}. */ - public final int column; - - /** The script's path, or null if unsaved. */ - public final String path; - - /** - * The Python executable of the script's environment, or null to analyze - * with Jedi's own interpreter (standard library only). - */ - public final String python; - - /** - * How many completions to give signatures, which are expensive (e.g. - * those of NumPy's many overloaded functions). - */ - public final int signatures; - - /** - * A directory of packages the environment lacks (see - * {@link DependencyOverlays}), to search too; or null. - */ - public final String overlay; - - public Query(final String source, final int line, final int column, - final String path, final String python) - { - this(source, line, column, path, python, 0, null); - } - - public Query(final String source, final int line, final int column, - final String path, final String python, final int signatures, - final String overlay) - { - this.source = source; - this.line = line; - this.column = column; - this.path = path; - this.python = python; - this.signatures = signatures; - this.overlay = overlay; - } - } - - /** Does the actual work: asks Jedi, e.g. by running a task on a worker. */ - public interface Backend { - - /** Gets completions for a query, remembering them by its key. */ - List complete(String key, Query query); - - /** Gets the documentation of a completion of the query with this key. */ - String describe(String key, Query query, String name); - - /** Gets the signatures of the call around the query's caret. */ - default SignatureHelp signatures(final String key, final Query query) { - return SignatureHelp.NONE; - } - } - - /** Number of answers to remember. */ - private static final int CACHE_SIZE = 32; - - /** - * How long an answer stays good, in milliseconds: long enough for typing - * a word, short enough to notice an environment that changed meanwhile - * (e.g. was rebuilt by running the script). - */ - private static final long ANSWER_LIFETIME = 20000; - - private final Backend backend; - private final ExecutorService executor = Executors.newSingleThreadExecutor( - r -> { - final Thread t = new Thread(r, "appose-python-completion"); - t.setDaemon(true); - return t; - }); - - /** Answers (or pending answers), by key, least recently used first. */ - private final Map>> answers = - new LinkedHashMap>>(16, 0.75f, - true) - { - - @Override - protected boolean removeEldestEntry( - final Map.Entry>> eldest) - { - return size() > CACHE_SIZE; - } - }; - - private CompletableFuture> latest; - - /** When each cached answer was asked for, by key. */ - private final Map askedAt = new HashMap<>(); - - /** - * @param backend Asks Jedi; called on a background thread, one request at - * a time. - */ - public JediClient(final Backend backend) { - this.backend = backend; - } - - /** - * @param completer Computes completions for a query; there is no - * documentation. - */ - public JediClient(final Function> completer) { - this(new Backend() { - - @Override - public List complete(final String key, final Query query) { - return completer.apply(query); - } - - @Override - public String describe(final String key, final Query query, - final String name) - { - return null; - } - }); - } - - /** - * Gets Jedi's completions for a query: the answer, or a pending one. - * - * @param key Identifies the answer: queries with the same key share one. - * @param query What to ask, should the answer not be known yet. - * @return The completions (or null, if Jedi has none), once known. Fails if - * they could not be computed; the next request with the same key - * then asks again. - */ - public CompletableFuture> answer(final String key, - final Query query) - { - synchronized (answers) { - final CompletableFuture> known = answers.get(key); - final Long asked = askedAt.get(key); - if (known != null && !known.isCancelled() && !known - .isCompletedExceptionally() && asked != null && System - .currentTimeMillis() - asked < ANSWER_LIFETIME) return known; - // Supersede the previous query, unless it has started already: a - // cancelled future's task is skipped when its turn comes. - if (latest != null) latest.cancel(false); - final CompletableFuture> f = CompletableFuture - .supplyAsync(() -> backend.complete(key, query), executor); - answers.put(key, f); - askedAt.keySet().retainAll(answers.keySet()); - askedAt.put(key, System.currentTimeMillis()); - latest = f; - return f; - } - } - - /** - * Gets Jedi's completions for a query, if available within the time budget. - * - * @return The completions, or null if they are not available in time (or - * could not be computed). - */ - public List complete(final String key, final Query query, - final long budgetMillis) - { - return await(answer(key, query), budgetMillis); - } - - /** - * Gets the documentation of a completion, waiting at most the given time. - * - * @param key The key of the query that yielded the completion. - * @param query That query. - * @param name The completion's name. - * @return Its documentation, or null if none (or not available in time). - */ - public String describe(final String key, final Query query, - final String name, final long budgetMillis) - { - return await(CompletableFuture.supplyAsync(() -> backend.describe(key, - query, name), executor), budgetMillis); - } - - /** - * Gets the signatures of the call around the query's caret (see - * {@link org.scijava.code.api.CodeCompleter#signatureHelp}), once known. - */ - public CompletableFuture signatures(final String key, - final Query query) - { - return CompletableFuture.supplyAsync(() -> backend.signatures(key, query), - executor); - } - - /** Stops the background thread. */ - public void dispose() { - executor.shutdownNow(); - } - - private static T await(final CompletableFuture f, - final long budgetMillis) - { - try { - return f.get(budgetMillis, TimeUnit.MILLISECONDS); - } - catch (final TimeoutException | ExecutionException exc) { - return null; // NB: Not (yet) available. - } - catch (final java.util.concurrent.CancellationException exc) { - return null; - } - catch (final InterruptedException exc) { - Thread.currentThread().interrupt(); - return null; - } - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java new file mode 100644 index 0000000..7f5c8fc --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java @@ -0,0 +1,126 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import java.io.File; +import java.io.IOException; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.concurrent.Callable; + +import org.scijava.code.api.EnvironmentContext; +import org.scijava.code.lsp.LspConnection; +import org.scijava.code.lsp.LspServerSpec; + +/** + * Starts jedi-language-server, with the Python of a tooling environment, to + * analyze scripts against their environment's Python (if known) and extra + * search paths (e.g. a dependency overlay). + * + * @author Gabriel Selzer + */ +public class JediLanguageServerSpec implements LspServerSpec { + + private final Callable python; + private final File workspace; + private final File log; + + /** + * @param python Gets the Python executable of the environment + * jedi-language-server is installed in (building it, if needed). + * @param workspace An (empty) directory to serve as workspace root. + * @param log Where to append the server's error output. + */ + public JediLanguageServerSpec(final Callable python, + final File workspace, final File log) + { + this.python = python; + this.workspace = workspace; + this.log = log; + } + + @Override + public String languageId() { + return "python"; + } + + @Override + public LspConnection start(final EnvironmentContext env) + throws IOException + { + final String exe; + try { + exe = python.call(); + } + catch (final IOException exc) { + throw exc; + } + catch (final Exception exc) { + throw new IOException("No Python for jedi-language-server", exc); + } + if (exe == null) throw new IOException( + "No Python for jedi-language-server"); + log.getParentFile().mkdirs(); + final ProcessBuilder pb = new ProcessBuilder(exe, "-c", + "from jedi_language_server.cli import cli; cli()"); + pb.redirectError(ProcessBuilder.Redirect.appendTo(log)); + return LspConnection.of(pb.start()); + } + + @Override + public Object initializationOptions(final EnvironmentContext env) { + final Map workspaceOptions = new LinkedHashMap<>(); + // NB: Without an environment, Jedi uses the server's own Python. + if (env.interpreter() != null) { + workspaceOptions.put("environmentPath", env.interpreter()); + } + workspaceOptions.put("extraPaths", env.searchPaths()); + final Map options = new LinkedHashMap<>(); + options.put("workspace", workspaceOptions); + options.put("diagnostics", map("enable", false)); + return options; + } + + @Override + public String rootUri(final EnvironmentContext env) { + // NB: jedi-language-server applies its workspace options only with a + // root. An empty one, so that nothing irrelevant gets indexed. + workspace.mkdirs(); + return workspace.toURI().toString(); + } + + private static Map map(final String key, + final Object value) + { + final Map map = new LinkedHashMap<>(); + map.put(key, value); + return map; + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClient.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClient.java new file mode 100644 index 0000000..50a6d1f --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClient.java @@ -0,0 +1,146 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import java.util.ArrayList; +import java.util.List; + +import org.scijava.code.api.Completion; +import org.scijava.code.lsp.LspClient; +import org.scijava.code.lsp.LspServers; + +/** + * Asks a Python language server (jedi-language-server) about scripts, and + * understands its signatures, e.g. + * {@code def dumps(obj, *, skipkeys: bool=False, **kw) -> str}. + * + * @author Gabriel Selzer + */ +public class PythonLspClient extends LspClient { + + public PythonLspClient(final LspServers servers) { + super(servers); + } + + @Override + protected Completion.Builder callable(final String name, + final String signature) + { + return parse(name, signature); + } + + /** + * Parses a Python signature (as Jedi writes them): the parameters, with + * their annotations as types; and the return annotation. Returns null if it + * is not a signature. + */ + public static Completion.Builder parse(final String name, + final String signature) + { + final int open = signature.indexOf('('); + if (open < 0) return null; + final int close = closing(signature, open); + if (close < 0) return null; + final List params = new ArrayList<>(); + for (final String part : split(signature.substring(open + 1, close))) { + final String p = part.trim(); + // NB: Bare * and / only mark keyword-only and positional-only ones. + if (p.isEmpty() || p.equals("*") || p.equals("/")) continue; + final int colon = topLevel(p, ':'), eq = topLevel(p, '='); + String paramName = p, type = null; + if (colon >= 0 && (eq < 0 || colon < eq)) { + paramName = p.substring(0, colon); + type = p.substring(colon + 1, eq < 0 ? p.length() : eq).trim(); + } + else if (eq >= 0) paramName = p.substring(0, eq); + paramName = paramName.trim().replaceFirst("^\\*{1,2}", ""); + params.add(new Completion.Parameter(paramName, type == null || type + .isEmpty() ? null : type)); + } + final String rest = signature.substring(close + 1).trim(); + final String returns = rest.startsWith("->") ? rest.substring(2).trim() + : null; + return Completion.builder(name).parameters(params).returnType( + returns == null || returns.isEmpty() ? null : returns); + } + + // -- Helper methods -- + + /** The index of the bracket closing the one at {@code open}, or -1. */ + private static int closing(final String s, final int open) { + int depth = 0; + char quote = 0; + for (int i = open; i < s.length(); i++) { + final char c = s.charAt(i); + if (quote != 0) { + if (c == '\\') i++; + else if (c == quote) quote = 0; + } + else if (c == '\'' || c == '"') quote = c; + else if (c == '(' || c == '[' || c == '{') depth++; + else if (c == ')' || c == ']' || c == '}') { + if (--depth == 0) return i; + } + } + return -1; + } + + /** Splits at the commas outside brackets and strings. */ + private static List split(final String s) { + final List parts = new ArrayList<>(); + int start = 0; + while (true) { + final int comma = topLevel(s.substring(start), ','); + if (comma < 0) break; + parts.add(s.substring(start, start + comma)); + start += comma + 1; + } + parts.add(s.substring(start)); + return parts; + } + + /** The index of the first {@code target} outside brackets and strings. */ + private static int topLevel(final String s, final char target) { + int depth = 0; + char quote = 0; + for (int i = 0; i < s.length(); i++) { + final char c = s.charAt(i); + if (quote != 0) { + if (c == '\\') i++; + else if (c == quote) quote = 0; + } + else if (c == '\'' || c == '"') quote = c; + else if (c == '(' || c == '[' || c == '{') depth++; + else if (c == ')' || c == ']' || c == '}') depth--; + else if (c == target && depth == 0) return i; + } + return -1; + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java index bb71141..00691cc 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java @@ -51,7 +51,7 @@ *

    * This needs no Python, so it works instantly, before (or without) any Appose * environment. It knows nothing of types, though: members of objects come - * from Jedi (see {@link JediClient}). + * from the language server (see {@link PythonLspClient}). *

    * * @author Gabriel Selzer diff --git a/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/jedi_complete.py b/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/jedi_complete.py deleted file mode 100644 index 6ca716b..0000000 --- a/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/jedi_complete.py +++ /dev/null @@ -1,172 +0,0 @@ -### -# #%L -# Python scripting language plugin backed by Appose. -# %% -# Copyright (C) 2026 SciJava developers. -# %% -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# -# 1. Redistributions of source code must retain the above copyright notice, -# this list of conditions and the following disclaimer. -# 2. Redistributions in binary form must reproduce the above copyright notice, -# this list of conditions and the following disclaimer in the documentation -# and/or other materials provided with the distribution. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" -# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE -# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR -# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF -# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS -# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN -# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) -# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE -# POSSIBILITY OF SUCH DAMAGE. -# #L% -### -# Appose task script that asks Jedi for code completions in a script, or for -# the documentation of one of them. -# -# Runs in the completion tooling environment (Python + Jedi), and analyzes -# the script against its own environment's interpreter, if given, so that -# the packages installed there are known. -# -# Task inputs: -# -# op -- "complete" (the default), "describe" or "signatures" -# key -- identifies the query: "describe" looks up the -# completions of a "complete" with the same key -# source -- the script's source code (with a preamble typing its -# SciJava parameters) -# line, column -- the caret position (line 1-based, column 0-based) -# path -- the script's file path, or None if unsaved -# python -- the Python executable of the script's environment, or -# None to use Jedi's own interpreter -# max_signatures -- for "complete": how many completions to give -# signatures (they are expensive: give few, or none) -# max_overloads -- for "complete": how many signatures to give each -# name -- for "describe": which completion to describe -# overlay -- a directory of packages the environment lacks, which -# Jedi should search too; or None -# -# Task outputs: -# -# completions -- for "complete": list of dicts: name, type, description, -# and signatures: a list of dicts with params (list of -# [name, annotation or None]) and returns (or None) -# docstring -- for "describe": the completion's documentation, or None -# signatures -- for "signatures": the signatures of the call around the -# caret: list of dicts with name, params and returns -# index -- for "signatures": index of the argument the caret is in, -# or None if Jedi cannot tell - -import os as _sj_os -import sys as _sj_sys - - -def _sj_cache(): - # Note: Each task gets fresh globals, so keep state that is slow to - # recompute in a module that persists in the worker: Jedi's environments, - # and the latest completions (to describe them on request). - cache = _sj_sys.modules.get("_scijava_jedi_cache") - if cache is None: - import types - - cache = types.ModuleType("_scijava_jedi_cache") - cache.environments = {} - cache.completions = {} # key -> {name: jedi Completion} - _sj_sys.modules["_scijava_jedi_cache"] = cache - return cache - - -def _sj_script(cache): - import jedi - - environment = None - if python: - environment = cache.environments.get(python) - if environment is None: - try: - environment = jedi.create_environment(python, safe=False) - except Exception: # noqa: BLE001 -- e.g. a Python Jedi cannot use - environment = jedi.InterpreterEnvironment() - cache.environments[python] = environment - else: - environment = jedi.InterpreterEnvironment() - - extra = [overlay] if globals().get("overlay") else [] - project = None - if path or extra: - root = _sj_os.path.dirname(_sj_os.path.abspath(path)) if path else None - project = jedi.Project(root or _sj_os.getcwd(), added_sys_path=extra) - return jedi.Script( - source, path=path, environment=environment, project=project - ) - - -def _sj_completions(cache): - completions = _sj_script(cache).complete(line, column) - # Remember them, to describe them on request; only the latest few. - cache.completions[key] = {c.name: c for c in completions} - while len(cache.completions) > 8: - cache.completions.pop(next(iter(cache.completions))) - return completions - - -def _sj_signature(sig): - params = [] - for p in sig.params: - text = p.to_string() - annotation = None - if ":" in text: - annotation = text.split(":", 1)[1].split("=", 1)[0].strip() - params.append([p.name, annotation]) - sig_text = sig.to_string() - returns = sig_text.rsplit("->", 1)[1].strip() if "->" in sig_text else None - return {"name": sig.name, "params": params, "returns": returns} - - -def _sj_signatures(c): - return [_sj_signature(sig) for sig in c.get_signatures()[:max_overloads]] - - -def _sj_main(): - cache = _sj_cache() - if globals().get("op") == "signatures": - signatures = _sj_script(cache).get_signatures(line, column) - task.outputs["index"] = signatures[0].index if signatures else None - task.outputs["signatures"] = [ - _sj_signature(s) for s in signatures[:max_overloads] - ] - return - if globals().get("op") == "describe": - known = cache.completions.get(key) - if known is None: - # Note: E.g. the worker restarted since; ask Jedi again. - known = {c.name: c for c in _sj_completions(cache)} - c = known.get(name) - task.outputs["docstring"] = (c.docstring(raw=True) or None) if c else None - return - - results = [] - for i, c in enumerate(_sj_completions(cache)): - signatures = [] - if i < max_signatures and c.type in ("function", "class"): - try: - signatures = _sj_signatures(c) - except Exception: # noqa: BLE001,S110 -- best effort - pass - results.append( - { - "name": c.name, - "type": c.type, - "description": c.description, - "signatures": signatures, - } - ) - task.outputs["completions"] = results - - -_sj_main() diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java index a392d3c..24429aa 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java @@ -34,19 +34,19 @@ import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; +import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; -import java.util.HashMap; -import java.util.LinkedHashMap; import java.util.List; -import java.util.Map; +import java.util.concurrent.CompletableFuture; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; -import java.util.concurrent.atomic.AtomicInteger; +import java.util.function.Function; import java.util.stream.Collectors; import net.imglib2.img.Img; +import org.eclipse.lsp4j.Position; import org.junit.After; import org.junit.Before; import org.junit.Test; @@ -54,14 +54,18 @@ import org.scijava.code.api.CodeCompletionService; import org.scijava.code.api.Completion; import org.scijava.code.api.CompletionRequest; +import org.scijava.code.api.EnvironmentContext; +import org.scijava.code.api.ScriptDocument; +import org.scijava.code.api.SignatureHelp; +import org.scijava.code.lsp.LspClient; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; -import org.scijava.plugins.scripting.appose.python.completion.JediClient; +import org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec; import org.scijava.script.ScriptLanguage; import org.scijava.script.ScriptService; /** - * Tests {@link ApposePythonCodeCompleter}, with a fake Jedi. + * Tests {@link ApposePythonCodeCompleter}, with a fake language server client. * * @author Gabriel Selzer */ @@ -97,7 +101,7 @@ public void testCompleterIsDiscovered() { @Test public void testLocalCompletions() { - noJedi(); + noServer(); final String header = "#@ net.imglib2.img.Img image\n#@ double sigma\n"; final List names = complete(header + "def blur(img, s):\n result = img\n re"); @@ -112,12 +116,12 @@ public void testLocalCompletions() { } @Test - public void testJediResultsMergedFirst() { + public void testServerResultsMergedFirst() { final Completion zeros = Completion.builder("zeros").kind( Completion.Kind.METHOD).parameters(Collections.singletonList( new Completion.Parameter("shape", "_ShapeLike"))).build(); final Completion zeta = Completion.of("zeta"); - completer.setJediClient(new JediClient(q -> Arrays.asList(zeros, zeta))); + fake(q -> Arrays.asList(zeros, zeta)); final List completions = completions( "import numpy as np\nnp.zeros(3)\nnp.ze"); assertEquals("zeros", completions.get(0).insertionText()); @@ -128,13 +132,12 @@ public void testJediResultsMergedFirst() { @Test public void testImportsAreNames() { - // Jedi reports imshow as a function, once per overload. + // The server reports imshow as a function, with its signature. final Completion imshow = Completion.builder("imshow").kind( Completion.Kind.METHOD).parameters(Collections.singletonList( new Completion.Parameter("data", "Any"))).summary("def imshow") .build(); - completer.setJediClient(new JediClient(q -> Arrays.asList(imshow, - imshow))); + fake(q -> Arrays.asList(imshow, imshow)); // In an import, it is a name to import, not a call: listed once, without // a parameter list for the editor to insert. final List imported = completions("from ndv import i"); @@ -147,40 +150,23 @@ public void testImportsAreNames() { } @Test - public void testTimeBudgetAndReuse() throws InterruptedException { - final AtomicInteger calls = new AtomicInteger(); - final CountDownLatch release = new CountDownLatch(1); - final CountDownLatch done = new CountDownLatch(1); - completer.setJediClient(new JediClient(q -> { - calls.incrementAndGet(); - try { - release.await(); - } - catch (final InterruptedException exc) { - Thread.currentThread().interrupt(); - } - done.countDown(); - return Collections.singletonList(Completion.of("zeros")); - })); - // Jedi is slow: the local results come back, in time. - final String code = "import numpy as np\nnp.zeros(3)\nnp.z"; - final long start = System.nanoTime(); - final List first = complete(code); - assertTrue(System.nanoTime() - start < TimeUnit.SECONDS.toNanos(2)); - assertEquals(Collections.singletonList("zeros"), first); // used before - // Once Jedi answers, later keystrokes of the word reuse its answer. - release.countDown(); - assertTrue(done.await(5, TimeUnit.SECONDS)); - Thread.sleep(50); - assertTrue(complete(code + "e").contains("zeros")); - assertTrue(complete(code + "er").contains("zeros")); - assertEquals(1, calls.get()); + public void testSameQuestionWhileTyping() { + final FakeLsp fake = fake(q -> Collections.singletonList(Completion.of( + "zeros"))); + // Later keystrokes of a word ask about its first character, so that the + // client can reuse the answer. + final String code = "import numpy as np\nnp.z"; + complete(code); + complete(code + "e"); + complete(code + "er"); + assertEquals(3, fake.asked.size()); + for (final Asked a : fake.asked) assertEquals(code.length(), a.offset); } @Test - public void testUpdateWhenJediIsSlow() throws Exception { + public void testUpdateWhenServerIsSlow() throws Exception { final CountDownLatch release = new CountDownLatch(1); - completer.setJediClient(new JediClient(q -> { + fake(q -> { try { release.await(); } @@ -188,14 +174,16 @@ public void testUpdateWhenJediIsSlow() throws Exception { Thread.currentThread().interrupt(); } return Collections.singletonList(Completion.of("zeros")); - })); + }); // At once: what is known locally, and the announcement of more. final String code = "import numpy as np\nnp.z"; + final long start = System.nanoTime(); final org.scijava.code.api.CompletionResult first = service.complete( new CompletionRequest(code, language, null)); + assertTrue(System.nanoTime() - start < TimeUnit.SECONDS.toNanos(2)); assertTrue(first.completions().isEmpty()); assertTrue(first.update() != null); - // Then Jedi's answer, merged. + // Then the server's answer, merged. release.countDown(); final org.scijava.code.api.CompletionResult better = first.update().get(5, TimeUnit.SECONDS); @@ -203,76 +191,25 @@ public void testUpdateWhenJediIsSlow() throws Exception { assertEquals(first.replaceStart(), better.replaceStart()); } - @Test - public void testLazyDocumentation() { - final List described = new java.util.ArrayList<>(); - final String[] keys = { null, null }; - completer.setJediClient(new JediClient(new JediClient.Backend() { - - @Override - public List complete(final String key, - final JediClient.Query query) - { - keys[0] = key; - final Map zeros = new HashMap<>(); - zeros.put("name", "zeros"); - zeros.put("type", "function"); - return ApposePythonCodeCompleter.toCompletions(Collections - .singletonList(zeros), name -> () -> { - described.add(name); - keys[1] = key; - return "Return a new array of zeros."; - }); - } - - @Override - public String describe(final String key, final JediClient.Query query, - final String name) - { - return null; - } - })); - final Completion zeros = completions("import numpy\nnumpy.z").get(0); - // Not described until asked, then by the same query. - assertTrue(described.isEmpty()); - assertEquals("Return a new array of zeros.", zeros.description()); - assertEquals(Collections.singletonList("zeros"), described); - assertEquals(keys[0], keys[1]); - } - @Test public void testSignaturesOnlyOnceWordBegun() { - final List signatures = new java.util.ArrayList<>(); - completer.setJediClient(new JediClient(q -> { - signatures.add(q.signatures); - return Collections.emptyList(); - })); + final FakeLsp fake = fake(q -> Collections.emptyList()); complete("import numpy\nnumpy."); complete("import numpy\nnumpy.z"); - assertEquals(Arrays.asList(0, 30), signatures); - // A function without signature is offered by name, not as a call. - final Map zeros = new HashMap<>(); - zeros.put("name", "zeros"); - zeros.put("type", "function"); - final Completion plain = ApposePythonCodeCompleter.toCompletions( - Collections.singletonList(zeros)).get(0); - assertFalse(plain.isCallable()); + assertEquals(Arrays.asList(0, 30), fake.asked.stream().map(a -> a.resolve) + .collect(Collectors.toList())); } @Test public void testOverlayInQuery() throws Exception { // A script declaring a dependency its environment lacks. - final List> installed = new java.util.ArrayList<>(); + final List> installed = new ArrayList<>(); completer.setOverlays(new org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays( (python, dependencies, dir) -> { installed.add(dependencies); return "/overlays/blur"; }, new java.io.File("overlays"), 0, System::currentTimeMillis)); - final List queries = new java.util.ArrayList<>(); - completer.setJediClient(new JediClient(q -> { - queries.add(q); - return Collections.emptyList(); - })); + final FakeLsp fake = fake(q -> Collections.emptyList()); final String code = "# /// script\n# dependencies = [\"tifffile\"]\n" + "# ///\nimport tifffile\ntifffile.im"; // The first request notes the dependencies; once they have stayed the @@ -282,58 +219,36 @@ public void testOverlayInQuery() throws Exception { for (int i = 0; i < 100 && installed.isEmpty(); i++) Thread.sleep(20); Thread.sleep(50); complete(code + "r"); - // Once the overlay is ready, Jedi searches it too (anew: a new key). + // Once the overlay is ready, the server searches it too. assertEquals(Collections.singletonList(Collections.singletonList( "tifffile")), installed); - assertEquals("/overlays/blur", queries.get(queries.size() - 1).overlay); + assertEquals(Collections.singletonList("/overlays/blur"), last(fake).doc + .environment().searchPaths()); } @Test public void testSignatureHelp() throws Exception { - final JediClient.Query[] asked = { null }; final Completion dumps = Completion.builder("dumps").kind( Completion.Kind.METHOD).parameters(Arrays.asList( new Completion.Parameter("obj", null), new Completion.Parameter( "indent", "int"))).build(); - completer.setJediClient(new JediClient(new JediClient.Backend() { - - @Override - public List complete(final String key, - final JediClient.Query query) - { - return Collections.emptyList(); - } - - @Override - public String describe(final String key, final JediClient.Query query, - final String name) - { - return null; - } - - @Override - public org.scijava.code.api.SignatureHelp signatures(final String key, - final JediClient.Query query) - { - asked[0] = query; - // NB: Jedi cannot always tell the argument (-1). - return new org.scijava.code.api.SignatureHelp(Collections - .singletonList(new org.scijava.code.api.SignatureHelp.Signature( - dumps, org.scijava.code.api.SignatureHelp.Fit.UNKNOWN)), -1, -1); - } - })); + final FakeLsp fake = fake(q -> Collections.emptyList()); + fake.signatures = Collections.singletonList(new SignatureHelp.Signature( + dumps, SignatureHelp.Fit.UNKNOWN)); final String code = "import json\njson.dumps(obj, "; - org.scijava.code.api.SignatureHelp help = completer.signatureHelp( - new CompletionRequest(code, language, null)); + SignatureHelp help = completer.signatureHelp(new CompletionRequest(code, + language, null)); if (help.update() != null) help = help.update().get(5, TimeUnit.SECONDS); assertEquals("dumps", help.signatures().get(0).callable().insertionText()); - // The argument the caret is in, counted where Jedi cannot tell; and the - // call's opening parenthesis. + // The argument the caret is in, counted (for when the server cannot + // tell); and the call's opening parenthesis. assertEquals(1, help.activeParameter()); assertEquals(code.indexOf('('), help.callStart()); - // Jedi was asked about the caret (shifted by the preamble). - assertEquals(3, asked[0].line); - assertEquals("json.dumps(obj, ".length(), asked[0].column); + // The server was asked about the caret (shifted by the preamble). + final Asked asked = last(fake); + assertEquals(code.length(), asked.offset); + assertEquals(new Position(2, "json.dumps(obj, ".length()), position( + asked)); // No help outside calls, or in strings. assertTrue(completer.signatureHelp(new CompletionRequest("x = 1", language, null)).isEmpty()); @@ -342,41 +257,54 @@ public org.scijava.code.api.SignatureHelp signatures(final String key, } @Test - public void testPrepareWarmsUp() throws Exception { - final CountDownLatch asked = new CountDownLatch(1); - completer.setJediClient(new JediClient(q -> { - asked.countDown(); - return Collections.emptyList(); - })); + public void testPrepareAndClose() throws Exception { + final FakeLsp fake = fake(q -> Collections.emptyList()); completer.prepare(new CompletionRequest("import numpy\n", language, null)); - assertTrue(asked.await(5, TimeUnit.SECONDS)); + assertEquals(1, fake.asked.size()); + assertEquals("untitled:/script.py", fake.asked.get(0).uri); + completer.closed(new CompletionRequest("import numpy\n", language, null)); + assertEquals(Collections.singletonList("untitled:/script.py"), + fake.closed); } @Test - public void testJediQuery() { - final JediClient.Query[] query = { null }; - completer.setJediClient(new JediClient(q -> { - query[0] = q; - return Collections.emptyList(); - })); + public void testServerQuery() { + final FakeLsp fake = fake(q -> Collections.emptyList()); final String code = "#@ net.imglib2.img.Img image\n#@ double sigma\n" + "image.sha"; complete(code); - // The script's parameters are typed in a preamble, so Jedi knows them. - assertTrue(query[0].source.startsWith( + final Asked asked = last(fake); + // The script's parameters are typed in a preamble, so the server knows + // them. + assertTrue(asked.doc.text().startsWith( "import appose.python_worker; import numpy; image: numpy.ndarray; " + "sigma: float; task: appose.python_worker.Task\n")); // The caret is after the word's first character, shifted by the preamble. - assertEquals(4, query[0].line); - assertEquals("image.s".length(), query[0].column); - // No script environment has been built: Jedi uses its own interpreter. - assertNull(query[0].python); + assertEquals(new Position(3, "image.s".length()), position(asked)); + // No script environment has been built: the server uses its own Python. + assertNull(asked.doc.environment().interpreter()); + } + + @Test + public void testServerOptions() { + final JediLanguageServerSpec spec = new JediLanguageServerSpec(() -> null, + new java.io.File("workspace"), new java.io.File("server.log")); + assertEquals("python", spec.languageId()); + assertEquals("{workspace={environmentPath=/env/bin/python, " + + "extraPaths=[/overlay]}, diagnostics={enable=false}}", String.valueOf( + spec.initializationOptions(new EnvironmentContext("/env/bin/python", + Collections.singletonList("/overlay"), null)))); + // Without a script environment: the server's own. + assertEquals("{workspace={extraPaths=[]}, diagnostics={enable=false}}", + String.valueOf(spec.initializationOptions(EnvironmentContext.EMPTY))); + assertTrue(spec.rootUri(EnvironmentContext.EMPTY).startsWith("file:")); + new java.io.File("workspace").delete(); } @Test public void testScriptWorkersUntouched() { - noJedi(); + noServer(); // Completing in a script declaring an environment must not create (or // replace) a worker for it, even as its metadata is edited. final String script = "# /// script\n# dependencies = [\"appose\"]\n" + @@ -464,16 +392,16 @@ public void testEnvironmentFromEarlierSession() throws Exception { java.nio.file.Files.createDirectories(python.getParent()); java.nio.file.Files.createFile(python); - final JediClient.Query[] query = { null }; - completer.setJediClient(new JediClient(q -> { - query[0] = q; - return Collections.emptyList(); - })); + final FakeLsp fake = fake(q -> Collections.emptyList()); final String code = header + "numpy.ze"; service.complete(new CompletionRequest(code, code.length(), language, null, null, script.getPath())); - // Jedi analyzes against the environment's Python... - assertEquals(python.toFile().getAbsolutePath(), query[0].python); + // The server analyzes against the environment's Python... + final Asked asked = last(fake); + assertEquals(python.toFile().getAbsolutePath(), asked.doc.environment() + .interpreter()); + // ...of the saved script, by its file... + assertEquals(script.toURI().toString(), asked.uri); // ...without anything built or started. assertTrue(context.service(ResidentWorkerService.class).workers() .isEmpty()); @@ -487,8 +415,7 @@ public void testEnvironmentFromEarlierSession() throws Exception { @Test public void testPythonTypes() { - assertEquals("numpy.ndarray", DIALECT.runtimeType( - Img.class)); + assertEquals("numpy.ndarray", DIALECT.runtimeType(Img.class)); assertEquals("float", DIALECT.runtimeType(double.class)); assertEquals("int", DIALECT.runtimeType(Long.class)); assertEquals("str", DIALECT.runtimeType(String.class)); @@ -497,39 +424,6 @@ public void testPythonTypes() { assertNull(DIALECT.runtimeType(Context.class)); } - @Test - public void testJediOutputConversion() { - final Map sig1 = new HashMap<>(); - sig1.put("params", Arrays.asList(Arrays.asList("shape", "_ShapeLike"), - Arrays.asList("dtype", null))); - sig1.put("returns", "NDArray[float64]"); - final Map sig2 = new HashMap<>(); - sig2.put("params", Collections.singletonList(Arrays.asList("shape", - null))); - sig2.put("returns", null); - final Map zeros = new LinkedHashMap<>(); - zeros.put("name", "zeros"); - zeros.put("type", "function"); - zeros.put("description", "def zeros"); - zeros.put("docstring", "zeros(shape) -> a < b"); - zeros.put("signatures", Arrays.asList(sig1, sig2)); - final Map shape = new LinkedHashMap<>(); - shape.put("name", "shape"); - shape.put("type", "property"); - shape.put("signatures", Collections.emptyList()); - final List out = ApposePythonCodeCompleter.toCompletions( - Arrays.asList(zeros, shape)); - // One completion per overload. - assertEquals(3, out.size()); - assertEquals(Completion.Kind.METHOD, out.get(0).kind()); - assertEquals("_ShapeLike", out.get(0).parameters().get(0).type()); - assertEquals("NDArray[float64]", out.get(0).returnType()); - assertEquals(1, out.get(1).parameters().size()); - assertTrue(out.get(0).description().contains("a < b")); - assertEquals(Completion.Kind.FIELD, out.get(2).kind()); - assertFalse(out.get(2).isCallable()); - } - // -- Helper methods -- private static void delete(final java.nio.file.Path dir) throws Exception { @@ -569,9 +463,32 @@ public org.apposed.appose.Builder builder() { }; } - /** Uses a Jedi that never answers in time. */ - private void noJedi() { - completer.setJediClient(new JediClient(q -> null)); + /** Uses a server that never answers. */ + private void noServer() { + completer.setLspClient(new FakeLsp(null)); + } + + /** Uses a server answering via {@code answers}, asynchronously. */ + private FakeLsp fake(final Function> answers) { + final FakeLsp fake = new FakeLsp(answers); + completer.setLspClient(fake); + return fake; + } + + private static Asked last(final FakeLsp fake) { + return fake.asked.get(fake.asked.size() - 1); + } + + /** Where in the server's document a question was asked. */ + private static Position position(final Asked asked) { + final String text = asked.doc.text(); + final int at = asked.doc.map().toTransformed(asked.offset); + final int lineStart = text.lastIndexOf('\n', at - 1) + 1; + int line = 0; + for (int i = 0; i < lineStart; i++) { + if (text.charAt(i) == '\n') line++; + } + return new Position(line, at - lineStart); } private List completions(final String code) { @@ -583,4 +500,66 @@ private List complete(final String code) { return completions(code).stream().map(Completion::insertionText).collect( Collectors.toList()); } + + /** A question asked of the language server. */ + private static final class Asked { + + private final ScriptDocument doc; + private final String uri; + private final int offset, resolve; + + private Asked(final ScriptDocument doc, final String uri, + final int offset, final int resolve) + { + this.doc = doc; + this.uri = uri; + this.offset = offset; + this.resolve = resolve; + } + } + + /** A language server client answering from a function. */ + private static final class FakeLsp extends LspClient { + + private final Function> answers; + private final List asked = new java.util.concurrent.CopyOnWriteArrayList<>(); + private final List closed = new ArrayList<>(); + private List signatures = Collections.emptyList(); + + private FakeLsp(final Function> answers) { + super(null); + this.answers = answers; + } + + @Override + public CompletableFuture> complete( + final ScriptDocument doc, final String uri, final int offset, + final int resolve) + { + final Asked q = new Asked(doc, uri, offset, resolve); + asked.add(q); + if (answers == null) return new CompletableFuture<>(); + return CompletableFuture.supplyAsync(() -> answers.apply(q)); + } + + @Override + public CompletableFuture signatureHelp( + final ScriptDocument doc, final String uri, final int offset, + final int callStart, final int active) + { + asked.add(new Asked(doc, uri, offset, 0)); + return CompletableFuture.completedFuture(new SignatureHelp(signatures, + active, callStart)); + } + + @Override + public void prepare(final ScriptDocument doc, final String uri) { + asked.add(new Asked(doc, uri, doc.original().length(), 0)); + } + + @Override + public void close(final String uri) { + closed.add(uri); + } + } } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java index 4b0707b..5230f50 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -58,11 +58,13 @@ import org.scijava.script.ScriptService; /** - * End-to-end tests of {@link ApposePythonCodeCompleter} with a real Jedi. + * End-to-end tests of {@link ApposePythonCodeCompleter} with a real language + * server (jedi-language-server). *

    * The first completion builds the completion tooling environment (Python and - * Jedi, via pixi), and a script run builds the small uv environment of - * {@code appose-test/requirements.txt}; both require network access. + * jedi-language-server, via pixi), and a script run builds the small uv + * environment of {@code appose-test/requirements.txt}; both require network + * access. *

    * * @author Gabriel Selzer @@ -97,7 +99,7 @@ public static void tearDown() { @Test public void testStandardLibrary() throws Exception { - // Before any script environment is built, Jedi knows the standard + // Before any script environment is built, the server knows the standard // library, with signatures and documentation. final List completions = completeUntil( "import json\njson.du", null, "dumps"); @@ -123,7 +125,7 @@ public void testScriptEnvironment() throws Exception { moduleService.run(module, false, Collections.singletonMap("image", null)).get(); - // Now Jedi analyzes the script against its environment: numpy is known, + // Now the server analyzes the script against its environment: numpy is known, // and so is the type of the image parameter. completeUntil(header + "np.ze", script.getPath(), "zeros"); completeUntil(header + "image.sha", script.getPath(), "shape"); @@ -200,8 +202,8 @@ private static List complete(final String code, /** * Prints how long completions take, once warm: for another keystroke of a - * word Jedi has answered (cached); and for a new word, how long the editor - * waits (barely: it answers at once), and how long until Jedi's completions + * word the server has answered (cached); and for a new word, how long the editor + * waits (barely: it answers at once), and how long until the server's completions * arrive as an update. */ private static void report(final String code, final String path, @@ -221,7 +223,7 @@ private static void report(final String code, final String path, final long arrived = System.nanoTime() - t0; System.out.println("[completion timing] same word (cached): " + cached / 1000000 + " ms, " + n + " completions; new word: editor waits " + wait / - 1000000 + " ms, Jedi's " + update.completions().size() + + 1000000 + " ms, the server's " + update.completions().size() + " completions after " + arrived / 1000000 + " ms"); } } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClientTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClientTest.java new file mode 100644 index 0000000..fa3d0e5 --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClientTest.java @@ -0,0 +1,75 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; + +import org.junit.Test; +import org.scijava.code.api.Completion; + +/** + * Tests {@link PythonLspClient}'s reading of signatures. + * + * @author Gabriel Selzer + */ +public class PythonLspClientTest { + + @Test + public void testParse() { + Completion c = parse("dumps", + "def dumps(obj, *, skipkeys: bool=False, cls=None, **kw) -> str"); + assertEquals("[obj, bool skipkeys, cls, kw]", String.valueOf(c + .parameters())); + assertEquals("str", c.returnType()); + + // Brackets and strings in annotations and defaults; positional-only. + c = parse("zeros", "def zeros(shape: tuple[int, ...], dtype: str=\"f,8\"" + + ", /, order: Literal['C', 'F']='C') -> NDArray[float64]"); + assertEquals("[tuple[int, ...] shape, str dtype, " + + "Literal['C', 'F'] order]", String.valueOf(c.parameters())); + assertEquals("NDArray[float64]", c.returnType()); + + // Classes (their constructors), without parameters or return type. + c = parse("Path", "class Path()"); + assertEquals(0, c.parameters().size()); + assertNull(c.returnType()); + c = parse("f", "def f(*args)"); + assertEquals("args", c.parameters().get(0).name()); + + // Not a signature. + assertNull(PythonLspClient.parse("x", "instance x")); + } + + private static Completion parse(final String name, final String sig) { + return PythonLspClient.parse(name, sig).kind(Completion.Kind.METHOD) + .build(); + } +} From 3ef818c04479dc2ad81cad910255b0b84ef0f902 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Tue, 6 Oct 2026 13:19:08 -0500 Subject: [PATCH 04/37] Offer hover and diagnostics from jedi-language-server Co-Authored-By: Claude Opus 5.5 --- .../python/ApposePythonCodeCompleter.java | 42 ++++++++++++++++ .../completion/JediLanguageServerSpec.java | 7 ++- .../python/ApposePythonCodeCompleterTest.java | 49 +++++++++++++++++-- ...ApposePythonCompletionIntegrationTest.java | 39 +++++++++++++++ 4 files changed, 132 insertions(+), 5 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java index 7edbb32..ebd180f 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java @@ -60,6 +60,8 @@ import org.scijava.code.api.Completion; import org.scijava.code.api.CompletionRequest; import org.scijava.code.api.CompletionResult; +import org.scijava.code.api.Diagnostic; +import org.scijava.code.api.Hover; import org.scijava.code.api.EnvironmentContext; import org.scijava.code.api.ScriptDialect; import org.scijava.code.api.ScriptDocument; @@ -116,6 +118,9 @@ public class ApposePythonCodeCompleter extends AbstractCodeCompleterPlugin { private static final long BUDGET = Long.getLong( "scijava.appose.completion.budget", 25); + /** How long to wait for the server to find problems, in milliseconds. */ + private static final long DIAGNOSE_TIMEOUT = 30_000; + /** How many completions get signatures, once a word is begun. */ private static final int SIGNATURES = 30; @@ -228,6 +233,43 @@ public ScriptDialect dialect() { return DIALECT; } + @Override + public Hover hover(final CompletionRequest request) { + final String text = request.text(); + final int at = request.offset(); + if (PythonSyntax.analyze(text, at).kind == Kind.NONE) { + return Hover.NONE; // NB: In a comment or string. + } + final LspClient client = lspClient(); + if (client == null) return Hover.NONE; + final CompletableFuture hover = client.hover(document(request, + true, false), uri(request), at).handle((h, error) -> h); + try { + final Hover h = hover.get(BUDGET, TimeUnit.MILLISECONDS); + return h == null ? Hover.NONE : h; + } + catch (final TimeoutException exc) { + return Hover.NONE.withUpdate(hover); + } + catch (final InterruptedException exc) { + Thread.currentThread().interrupt(); + } + catch (final ExecutionException | CancellationException exc) { + // NB: The server failed. + } + return Hover.NONE; + } + + @Override + public CompletableFuture> diagnose( + final CompletionRequest request) + { + final LspClient client = lspClient(); + if (client == null) return super.diagnose(request); + return client.diagnose(document(request, true, false), uri(request), + DIAGNOSE_TIMEOUT); + } + @Override public SignatureHelp signatureHelp(final CompletionRequest request) { final String text = request.text(); diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java index 7f5c8fc..f882eb0 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java @@ -104,7 +104,12 @@ public Object initializationOptions(final EnvironmentContext env) { workspaceOptions.put("extraPaths", env.searchPaths()); final Map options = new LinkedHashMap<>(); options.put("workspace", workspaceOptions); - options.put("diagnostics", map("enable", false)); + // NB: Jedi's diagnostics are syntax errors, found as the script changes. + final Map diagnostics = map("enable", true); + diagnostics.put("didOpen", true); + diagnostics.put("didChange", true); + diagnostics.put("didSave", false); + options.put("diagnostics", diagnostics); return options; } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java index 24429aa..e067fec 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java @@ -268,6 +268,26 @@ public void testPrepareAndClose() throws Exception { fake.closed); } + @Test + public void testHoverAndDiagnostics() throws Exception { + final FakeLsp fake = fake(q -> Collections.emptyList()); + final String code = "#@ double sigma\nimport json\njson.dumps(sigma)"; + org.scijava.code.api.Hover hover = completer.hover(new CompletionRequest( + code, code.indexOf("dumps") + 1, language, null, null, null)); + if (hover.update() != null) hover = hover.update().get(5, + TimeUnit.SECONDS); + assertEquals("
    dumps
    ", hover.text()); + assertEquals(code.indexOf("dumps") + 1, last(fake).offset); + // Not in comments or strings. + assertTrue(completer.hover(new CompletionRequest("# json.dumps", 4, + language, null, null, null)).isEmpty()); + + final List problems = completer.diagnose( + new CompletionRequest(code, language, null)).get(5, TimeUnit.SECONDS); + assertEquals(1, problems.size()); + assertTrue(last(fake).doc.text().endsWith(code)); + } + @Test public void testServerQuery() { final FakeLsp fake = fake(q -> Collections.emptyList()); @@ -291,13 +311,15 @@ public void testServerOptions() { final JediLanguageServerSpec spec = new JediLanguageServerSpec(() -> null, new java.io.File("workspace"), new java.io.File("server.log")); assertEquals("python", spec.languageId()); + final String diagnostics = "diagnostics={enable=true, didOpen=true, " + + "didChange=true, didSave=false}"; assertEquals("{workspace={environmentPath=/env/bin/python, " + - "extraPaths=[/overlay]}, diagnostics={enable=false}}", String.valueOf( - spec.initializationOptions(new EnvironmentContext("/env/bin/python", + "extraPaths=[/overlay]}, " + diagnostics + "}", String.valueOf(spec + .initializationOptions(new EnvironmentContext("/env/bin/python", Collections.singletonList("/overlay"), null)))); // Without a script environment: the server's own. - assertEquals("{workspace={extraPaths=[]}, diagnostics={enable=false}}", - String.valueOf(spec.initializationOptions(EnvironmentContext.EMPTY))); + assertEquals("{workspace={extraPaths=[]}, " + diagnostics + "}", String + .valueOf(spec.initializationOptions(EnvironmentContext.EMPTY))); assertTrue(spec.rootUri(EnvironmentContext.EMPTY).startsWith("file:")); new java.io.File("workspace").delete(); } @@ -552,6 +574,25 @@ public CompletableFuture signatureHelp( active, callStart)); } + @Override + public CompletableFuture hover( + final ScriptDocument doc, final String uri, final int offset) + { + asked.add(new Asked(doc, uri, offset, 0)); + return CompletableFuture.completedFuture(new org.scijava.code.api.Hover( + "
    dumps
    ", offset, offset)); + } + + @Override + public CompletableFuture> diagnose( + final ScriptDocument doc, final String uri, final long timeout) + { + asked.add(new Asked(doc, uri, 0, 0)); + return CompletableFuture.completedFuture(Collections.singletonList( + new org.scijava.code.api.Diagnostic(0, 1, + org.scijava.code.api.Diagnostic.Severity.ERROR, "Oops", "fake"))); + } + @Override public void prepare(final ScriptDocument doc, final String uri) { asked.add(new Asked(doc, uri, doc.original().length(), 0)); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java index 5230f50..cabdec7 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -50,6 +50,8 @@ import org.scijava.code.api.Completion; import org.scijava.code.api.CompletionRequest; import org.scijava.code.api.CompletionResult; +import org.scijava.code.api.Diagnostic; +import org.scijava.code.api.Hover; import org.scijava.code.api.SignatureHelp; import org.scijava.module.ModuleService; import org.scijava.script.ScriptInfo; @@ -151,6 +153,43 @@ public void testSignatureHelp() throws Exception { assertEquals(code.indexOf('('), help.callStart()); } + @Test + public void testHover() throws Exception { + final String code = "import json\njson.dumps(1)"; + final CompletionRequest request = new CompletionRequest(code, code + .indexOf("dumps") + 2, language, null, null, null); + final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; + Hover hover = Hover.NONE; + while (hover.isEmpty() && System.currentTimeMillis() < deadline) { + hover = completer().hover(request); + if (hover.update() != null) { + final Hover better = hover.update().get(1, TimeUnit.MINUTES); + if (better != null) hover = better; + } + } + assertTrue(hover.text(), hover.text().contains("JSON")); + assertEquals(code.indexOf("dumps"), hover.start()); + assertEquals(code.indexOf("dumps") + "dumps".length(), hover.end()); + } + + @Test + public void testDiagnostics() throws Exception { + // A syntax error, on the script's third line. + final String code = "#@ double sigma\nx = sigma\ny = (1 +\n"; + final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; + List problems = Collections.emptyList(); + while (problems.isEmpty() && System.currentTimeMillis() < deadline) { + problems = completer().diagnose(new CompletionRequest(code, language, + null)).get(2, TimeUnit.MINUTES); + } + assertFalse(problems.isEmpty()); + assertTrue(problems.toString(), problems.get(0).start() >= code.indexOf( + "y = ")); + // Fixed: no more problems. + assertTrue(completer().diagnose(new CompletionRequest(code.replace("(1 +", + "(1 + 2)"), language, null)).get(1, TimeUnit.MINUTES).isEmpty()); + } + @Test public void testNewDependency() throws Exception { // A script declaring a dependency that is installed nowhere: its From 5fd45f7d96f2359b9a44389949eca6fdc4f8a0cd Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Tue, 6 Oct 2026 13:30:23 -0500 Subject: [PATCH 05/37] Offer jedi-language-server as the Python language server ApposePythonLspServerPlugin offers it via code-lsp's LspService, for this completer and others (e.g. Jython's); PythonLspClient moved to code-lsp. Co-Authored-By: Claude Opus 5.5 --- .../python/ApposePythonCodeCompleter.java | 44 +++--- .../python/ApposePythonLspServerPlugin.java} | 53 +++---- .../python/completion/PythonLspClient.java | 146 ------------------ .../python/completion/PythonSyntax.java | 2 +- 4 files changed, 42 insertions(+), 203 deletions(-) rename src/{test/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClientTest.java => main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java} (50%) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClient.java diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java index ebd180f..aedbf16 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java @@ -69,8 +69,8 @@ import org.scijava.code.api.Signatures; import org.scijava.code.lsp.LspClient; import org.scijava.code.lsp.LspServers; -import org.scijava.log.LogService; -import org.scijava.log.StderrLogService; +import org.scijava.code.lsp.LspService; +import org.scijava.code.lsp.PythonLspClient; import org.scijava.plugin.Plugin; import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; @@ -78,7 +78,6 @@ import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; import org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec; -import org.scijava.plugins.scripting.appose.python.completion.PythonLspClient; import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax; import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Kind; import org.scijava.script.ScriptLanguage; @@ -124,14 +123,6 @@ public class ApposePythonCodeCompleter extends AbstractCodeCompleterPlugin { /** How many completions get signatures, once a word is begun. */ private static final int SIGNATURES = 30; - /** How long a language server may go unused before it stops, in ms. */ - private static final long SERVER_IDLE = 10 * 60 * 1000; - - /** - * How many language servers may run at once (one per script environment - * and overlay). - */ - private static final int MAX_SERVERS = 3; /** Python executables of script environments, by script and metadata. */ private final Map pythons = new LinkedHashMap( @@ -609,17 +600,29 @@ private static String findPython(final List dirs) { return null; } - /** Gets the language server client, creating it on first use. */ + /** + * Gets the language server client, creating it on first use; for the + * Python servers of the context (see {@link ApposePythonLspServerPlugin}). + */ private synchronized LspClient lspClient() { if (lsp != null) return lsp; - final ResidentWorkerService workers = service( - ResidentWorkerService.class); - if (workers == null) return null; - LogService log = service(LogService.class); - if (log == null) log = new StderrLogService(); + final LspService service = service(LspService.class); + final LspServers servers = service == null ? null : service.servers( + "python"); + if (servers == null) return null; + lsp = new PythonLspClient(servers); + return lsp; + } + + /** + * How to start jedi-language-server: with the Python of the tooling + * environment, built first if needed (but starting no worker). + */ + static JediLanguageServerSpec serverSpec( + final ResidentWorkerService workers) + { final File envs = new File(Environments.apposeEnvsDir()); - final JediLanguageServerSpec spec = new JediLanguageServerSpec(() -> { - // NB: Builds the tooling environment, if needed (but starts no worker). + return new JediLanguageServerSpec(() -> { final String python = pythonExecutable(toolingWorker(workers) .environment().get()); if (python == null) throw new IOException("No Python in " + @@ -627,9 +630,6 @@ private synchronized LspClient lspClient() { return python; }, new File(envs, TOOLING_ENV + "-workspace"), new File(envs, TOOLING_ENV + "-server.log")); - lsp = new PythonLspClient(new LspServers(spec, log, SERVER_IDLE, - MAX_SERVERS)); - return lsp; } /** Gets the tooling environment's worker, building it first if needed. */ diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClientTest.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java similarity index 50% rename from src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClientTest.java rename to src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java index fa3d0e5..94b16fb 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClientTest.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java @@ -27,49 +27,34 @@ * #L% */ -package org.scijava.plugins.scripting.appose.python.completion; +package org.scijava.plugins.scripting.appose.python; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNull; - -import org.junit.Test; -import org.scijava.code.api.Completion; +import org.scijava.code.lsp.LspServerPlugin; +import org.scijava.code.lsp.LspServerSpec; +import org.scijava.plugin.Parameter; +import org.scijava.plugin.Plugin; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; /** - * Tests {@link PythonLspClient}'s reading of signatures. + * Offers jedi-language-server, from the completion tooling environment, as + * the Python language server: for Python scripts run via Appose, and for any + * other completer wanting one (e.g. Jython's). * * @author Gabriel Selzer */ -public class PythonLspClientTest { - - @Test - public void testParse() { - Completion c = parse("dumps", - "def dumps(obj, *, skipkeys: bool=False, cls=None, **kw) -> str"); - assertEquals("[obj, bool skipkeys, cls, kw]", String.valueOf(c - .parameters())); - assertEquals("str", c.returnType()); - - // Brackets and strings in annotations and defaults; positional-only. - c = parse("zeros", "def zeros(shape: tuple[int, ...], dtype: str=\"f,8\"" + - ", /, order: Literal['C', 'F']='C') -> NDArray[float64]"); - assertEquals("[tuple[int, ...] shape, str dtype, " + - "Literal['C', 'F'] order]", String.valueOf(c.parameters())); - assertEquals("NDArray[float64]", c.returnType()); +@Plugin(type = LspServerPlugin.class, name = "jedi-language-server") +public class ApposePythonLspServerPlugin implements LspServerPlugin { - // Classes (their constructors), without parameters or return type. - c = parse("Path", "class Path()"); - assertEquals(0, c.parameters().size()); - assertNull(c.returnType()); - c = parse("f", "def f(*args)"); - assertEquals("args", c.parameters().get(0).name()); + @Parameter + private ResidentWorkerService workers; - // Not a signature. - assertNull(PythonLspClient.parse("x", "instance x")); + @Override + public boolean supports(final String languageId) { + return "python".equals(languageId); } - private static Completion parse(final String name, final String sig) { - return PythonLspClient.parse(name, sig).kind(Completion.Kind.METHOD) - .build(); + @Override + public LspServerSpec spec() { + return ApposePythonCodeCompleter.serverSpec(workers); } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClient.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClient.java deleted file mode 100644 index 50a6d1f..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonLspClient.java +++ /dev/null @@ -1,146 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.util.ArrayList; -import java.util.List; - -import org.scijava.code.api.Completion; -import org.scijava.code.lsp.LspClient; -import org.scijava.code.lsp.LspServers; - -/** - * Asks a Python language server (jedi-language-server) about scripts, and - * understands its signatures, e.g. - * {@code def dumps(obj, *, skipkeys: bool=False, **kw) -> str}. - * - * @author Gabriel Selzer - */ -public class PythonLspClient extends LspClient { - - public PythonLspClient(final LspServers servers) { - super(servers); - } - - @Override - protected Completion.Builder callable(final String name, - final String signature) - { - return parse(name, signature); - } - - /** - * Parses a Python signature (as Jedi writes them): the parameters, with - * their annotations as types; and the return annotation. Returns null if it - * is not a signature. - */ - public static Completion.Builder parse(final String name, - final String signature) - { - final int open = signature.indexOf('('); - if (open < 0) return null; - final int close = closing(signature, open); - if (close < 0) return null; - final List params = new ArrayList<>(); - for (final String part : split(signature.substring(open + 1, close))) { - final String p = part.trim(); - // NB: Bare * and / only mark keyword-only and positional-only ones. - if (p.isEmpty() || p.equals("*") || p.equals("/")) continue; - final int colon = topLevel(p, ':'), eq = topLevel(p, '='); - String paramName = p, type = null; - if (colon >= 0 && (eq < 0 || colon < eq)) { - paramName = p.substring(0, colon); - type = p.substring(colon + 1, eq < 0 ? p.length() : eq).trim(); - } - else if (eq >= 0) paramName = p.substring(0, eq); - paramName = paramName.trim().replaceFirst("^\\*{1,2}", ""); - params.add(new Completion.Parameter(paramName, type == null || type - .isEmpty() ? null : type)); - } - final String rest = signature.substring(close + 1).trim(); - final String returns = rest.startsWith("->") ? rest.substring(2).trim() - : null; - return Completion.builder(name).parameters(params).returnType( - returns == null || returns.isEmpty() ? null : returns); - } - - // -- Helper methods -- - - /** The index of the bracket closing the one at {@code open}, or -1. */ - private static int closing(final String s, final int open) { - int depth = 0; - char quote = 0; - for (int i = open; i < s.length(); i++) { - final char c = s.charAt(i); - if (quote != 0) { - if (c == '\\') i++; - else if (c == quote) quote = 0; - } - else if (c == '\'' || c == '"') quote = c; - else if (c == '(' || c == '[' || c == '{') depth++; - else if (c == ')' || c == ']' || c == '}') { - if (--depth == 0) return i; - } - } - return -1; - } - - /** Splits at the commas outside brackets and strings. */ - private static List split(final String s) { - final List parts = new ArrayList<>(); - int start = 0; - while (true) { - final int comma = topLevel(s.substring(start), ','); - if (comma < 0) break; - parts.add(s.substring(start, start + comma)); - start += comma + 1; - } - parts.add(s.substring(start)); - return parts; - } - - /** The index of the first {@code target} outside brackets and strings. */ - private static int topLevel(final String s, final char target) { - int depth = 0; - char quote = 0; - for (int i = 0; i < s.length(); i++) { - final char c = s.charAt(i); - if (quote != 0) { - if (c == '\\') i++; - else if (c == quote) quote = 0; - } - else if (c == '\'' || c == '"') quote = c; - else if (c == '(' || c == '[' || c == '{') depth++; - else if (c == ')' || c == ']' || c == '}') depth--; - else if (c == target && depth == 0) return i; - } - return -1; - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java index 00691cc..c70d3e0 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java @@ -51,7 +51,7 @@ *

    * This needs no Python, so it works instantly, before (or without) any Appose * environment. It knows nothing of types, though: members of objects come - * from the language server (see {@link PythonLspClient}). + * from the language server (see {@link org.scijava.code.lsp.PythonLspClient}). *

    * * @author Gabriel Selzer From 1d07b044d4feaa7c1987d798c59d210ecacb1df2 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Tue, 6 Oct 2026 15:12:23 -0500 Subject: [PATCH 06/37] Fix image parameters, and orphaned servers The tooling environment gets NumPy: script parameters typed as NumPy arrays (e.g. images) were unknown to the server until the script's own environment was built, so 'image.' offered nothing from it. jedi-language-server is started via a launcher that exits once this JVM is gone: the server does not notice by itself, and on Windows then loops, failing to write to the closed pipe, flooding its log (which now starts anew beyond 5 MB). Co-Authored-By: Claude Opus 5.5 --- .../python/ApposePythonCodeCompleter.java | 7 +++-- .../completion/JediLanguageServerSpec.java | 29 +++++++++++++++++-- ...ApposePythonCompletionIntegrationTest.java | 26 +++++++++++++++++ 3 files changed, 58 insertions(+), 4 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java index aedbf16..51d693a 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java @@ -642,8 +642,8 @@ private static ResidentWorker toolingWorker( } /** - * The tooling environment: Python, Jedi, jedi-language-server and Appose; - * and uv, to install packages for {@link DependencyOverlays}. + * The tooling environment: Python, Jedi, jedi-language-server, NumPy and + * Appose; and uv, to install packages for {@link DependencyOverlays}. */ static String toolingConfig() { return "[workspace]\n" + // @@ -654,6 +654,9 @@ static String toolingConfig() { "[dependencies]\n" + // "python = \"3.12.*\"\n" + // "jedi = \">=0.19.2\"\n" + // + // NB: Script parameters are typed as NumPy arrays (see + // ApposePythonDialect): known even before a script's environment is. + "numpy = \"*\"\n" + // "\n" + // "[pypi-dependencies]\n" + // "appose = \"*\"\n" + // diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java index f882eb0..df6fde9 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java @@ -48,6 +48,30 @@ */ public class JediLanguageServerSpec implements LspServerSpec { + /** + * Runs jedi-language-server, exiting once the given process (this JVM) is + * gone. NB: The server does not notice by itself; on Windows, it then even + * loops, failing to write to the closed pipe. + */ + public static final String LAUNCHER = String.join("\n", // + "import os, sys, threading, time", // + "parent = int(sys.argv.pop(1))", // + "def watch():", // + " if os.name == 'nt':", // + " import ctypes", // + " k = ctypes.windll.kernel32", // + " h = k.OpenProcess(0x00100000, False, parent) # SYNCHRONIZE", // + " if h: k.WaitForSingleObject(h, 0xFFFFFFFF)", // + " else:", // + " while os.getppid() == parent: time.sleep(2)", // + " os._exit(0)", // + "threading.Thread(target=watch, daemon=True).start()", // + "from jedi_language_server.cli import cli", // + "cli()"); + + /** How large the error log may grow before it is started anew, in bytes. */ + private static final long MAX_LOG = 5 * 1024 * 1024; + private final Callable python; private final File workspace; private final File log; @@ -88,8 +112,9 @@ public LspConnection start(final EnvironmentContext env) if (exe == null) throw new IOException( "No Python for jedi-language-server"); log.getParentFile().mkdirs(); - final ProcessBuilder pb = new ProcessBuilder(exe, "-c", - "from jedi_language_server.cli import cli; cli()"); + if (log.length() > MAX_LOG) log.delete(); + final ProcessBuilder pb = new ProcessBuilder(exe, "-c", LAUNCHER, String + .valueOf(ProcessHandle.current().pid())); pb.redirectError(ProcessBuilder.Redirect.appendTo(log)); return LspConnection.of(pb.start()); } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java index cabdec7..78a502f 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -190,6 +190,32 @@ public void testDiagnostics() throws Exception { "(1 + 2)"), language, null)).get(1, TimeUnit.MINUTES).isEmpty()); } + @Test + public void testParameterBeforeEnvironment() throws Exception { + // An image parameter is a NumPy array: known, even though the script's + // environment is not built (here, it is not even declared). + completeUntil("#@ net.imglib2.img.Img image\nx = 1\nimage.", null, + "shape"); + } + + @Test + public void testServerOutlivesNoParent() throws Exception { + // The server, once its parent (this JVM) is gone, must exit: here, its + // parent is a process that already has. + completeUntil("import json\njson.du", null, "dumps"); + final String python = ApposePythonCodeCompleter.pythonOnDisk(new File( + org.apposed.appose.util.Environments.apposeEnvsDir(), + ApposePythonCodeCompleter.TOOLING_ENV)); + final Process gone = new ProcessBuilder(python, "-c", "pass").start(); + gone.waitFor(); + final Process server = new ProcessBuilder(python, "-c", + org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec.LAUNCHER, + String.valueOf(gone.pid())).redirectErrorStream(true).start(); + final boolean exited = server.waitFor(30, TimeUnit.SECONDS); + if (!exited) server.destroyForcibly(); + assertTrue(exited); + } + @Test public void testNewDependency() throws Exception { // A script declaring a dependency that is installed nowhere: its From d43e685fbb81bf072eda72693fdaf5ea8c2c0182 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Wed, 7 Oct 2026 11:00:14 -0500 Subject: [PATCH 07/37] Serve appose-python as language servers Completion now comes from two lsp4j language servers, which code-lsp's LanguageServerService merges: PythonSyntaxServerPlugin (tree-sitter: instant names, parameters, builtins and keywords) and JediServerPlugin (jedi-language-server, against the script's environment plus overlays, seeing the script as plain Python). ApposePythonEnvironments holds what the completer knew about environments; ApposePythonCodeCompleter, and its merge and time budget, are gone. Co-Authored-By: Claude Opus 5.5 --- README.md | 13 +- .../python/ApposePythonCodeCompleter.java | 701 ------------------ .../python/ApposePythonEnvironments.java | 398 ++++++++++ .../python/ApposePythonLspServerPlugin.java | 2 +- .../appose/python/JediServerPlugin.java | 110 +++ .../python/PythonSyntaxLanguageServer.java | 145 ++++ .../python/PythonSyntaxServerPlugin.java | 62 ++ .../appose/python/ScriptEnvironment.java | 2 +- .../python/ApposePythonCodeCompleterTest.java | 606 --------------- ...ApposePythonCompletionIntegrationTest.java | 207 ++++-- .../python/ApposePythonServersTest.java | 362 +++++++++ 11 files changed, 1213 insertions(+), 1395 deletions(-) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java delete mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java diff --git a/README.md b/README.md index b19535e..fd5525f 100644 --- a/README.md +++ b/README.md @@ -137,17 +137,20 @@ script's own line numbers. ## Code completion -The Script Editor completes code in these scripts, in two layers: +The Script Editor completes code in these scripts with two language servers, +which `scijava-code-lsp`'s `LanguageServerService` merges into one (so any +other LSP client of it gets the same): * **Instantly, with no Python needed:** names in scope (variables, functions, classes, imports, parameters of the enclosing function), the script's `#@` parameters, `task`, builtins and keywords. This comes from - parsing the script with [tree-sitter](https://tree-sitter.github.io/). + parsing the script with [tree-sitter](https://tree-sitter.github.io/) + (`PythonSyntaxServerPlugin`). * **With a language server, [jedi-language-server](https://github.com/pappasam/jedi-language-server) - (built on [Jedi](https://jedi.readthedocs.io/)):** members of objects and - modules, function signatures (with parameter names and types) and - documentation. The server runs in a small environment of its own + (built on [Jedi](https://jedi.readthedocs.io/)), via `JediServerPlugin`:** + members of objects and modules, function signatures (with parameter names + and types), documentation, hover and syntax errors. The server runs in a small environment of its own (`appose-python-completion`: Python, jedi-language-server, Appose and uv), which is built in the background the first time you complete code, and takes a while only then. The server talks to the editor via the diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java deleted file mode 100644 index 51d693a..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleter.java +++ /dev/null @@ -1,701 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python; - -import java.io.File; -import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.HashMap; -import java.util.HashSet; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Locale; -import java.util.Map; -import java.util.Set; -import java.util.concurrent.CancellationException; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.ExecutionException; -import java.util.concurrent.TimeUnit; -import java.util.concurrent.TimeoutException; - -import org.apposed.appose.Appose; -import org.apposed.appose.Environment; -import org.apposed.appose.Service; -import org.apposed.appose.util.Environments; -import org.scijava.Context; -import org.scijava.code.api.AbstractCodeCompleterPlugin; -import org.scijava.code.api.CodeCompleterPlugin; -import org.scijava.code.api.CodeCompletionService; -import org.scijava.code.api.Completion; -import org.scijava.code.api.CompletionRequest; -import org.scijava.code.api.CompletionResult; -import org.scijava.code.api.Diagnostic; -import org.scijava.code.api.Hover; -import org.scijava.code.api.EnvironmentContext; -import org.scijava.code.api.ScriptDialect; -import org.scijava.code.api.ScriptDocument; -import org.scijava.code.api.SignatureHelp; -import org.scijava.code.api.Signatures; -import org.scijava.code.lsp.LspClient; -import org.scijava.code.lsp.LspServers; -import org.scijava.code.lsp.LspService; -import org.scijava.code.lsp.PythonLspClient; -import org.scijava.plugin.Plugin; -import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; -import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; -import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; -import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; -import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; -import org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec; -import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax; -import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Kind; -import org.scijava.script.ScriptLanguage; - -/** - * Code completion for {@link ApposePythonScriptLanguage} scripts, in two - * layers: - *
      - *
    1. {@link PythonSyntax} (tree-sitter, in process): instantly, with no - * Python needed, names in scope, the script's parameters, builtins and - * keywords.
      2. - *
    2. jedi-language-server (a language server built on Jedi), in a small - * tooling environment of its own, built in the background on first use: - * members of objects and modules, signatures and documentation. Once the - * script's own environment is built (by running the script), the server - * analyzes the script against it, so its packages are known. Its answers are - * used if they come within a short time budget; typing never waits for - * them.
      3. - *
    - *

    - * Completion never builds a script's environment, and never disturbs its - * worker. - *

    - * - * @author Gabriel Selzer - */ -@Plugin(type = CodeCompleterPlugin.class, name = "appose-python") -public class ApposePythonCodeCompleter extends AbstractCodeCompleterPlugin { - - /** Name of the tooling environment in which the language server runs. */ - static final String TOOLING_ENV = "appose-python-completion"; - - /** - * How long to wait for the language server before answering without it - * (and updating the answer once its arrives), in milliseconds. - */ - private static final long BUDGET = Long.getLong( - "scijava.appose.completion.budget", 25); - - /** How long to wait for the server to find problems, in milliseconds. */ - private static final long DIAGNOSE_TIMEOUT = 30_000; - - /** How many completions get signatures, once a word is begun. */ - private static final int SIGNATURES = 30; - - - /** Python executables of script environments, by script and metadata. */ - private final Map pythons = new LinkedHashMap( - 16, 0.75f, true) - { - - @Override - protected boolean removeEldestEntry( - final Map.Entry eldest) - { - return size() > 16; - } - }; - - /** Task scripts (see overlay.py), by resource name. */ - private static final Map SCRIPTS = new HashMap<>(); - - private LspClient lsp; - - private DependencyOverlays overlays; - - /** - * How long a script's dependencies must stay unchanged before packages - * missing from its environment are installed for completion, in ms. - */ - private static final long OVERLAY_SETTLE = 1500; - - /** How scripts see their parameters (as Python values) and {@code task}. */ - private static final ApposePythonDialect DIALECT = new ApposePythonDialect(); - - @Override - public boolean supports(final ScriptLanguage language) { - return language instanceof ApposePythonScriptLanguage; - } - - @Override - public CompletionResult complete(final CompletionRequest request) { - final String text = request.text(); - final PythonSyntax.Context ctx = PythonSyntax.analyze(text, request - .offset()); - if (ctx.kind == Kind.NONE) return CompletionResult.EMPTY; - if (ctx.kind == Kind.NAME && ctx.prefix.isEmpty()) { - return CompletionResult.EMPTY; - } - - final LspClient client = lspClient(); - final ScriptDocument doc = document(request, client != null, false); - final List local = localCompletions(ctx, doc.parameters()); - if (client == null) return merge(ctx, null, local); - // Ask about the first character typed, if any: later keystrokes of the - // same word then reuse the answer (filtered), while it stays short. - // NB: Signatures are expensive, so ask for them only once a word is - // begun, which narrows the completions down. Before that, functions are - // offered by name. - final CompletableFuture> answer = client.complete(doc, - uri(request), ctx.replaceStart + Math.min(1, ctx.prefix.length()), - ctx.prefix.isEmpty() ? 0 : SIGNATURES); - - // Use the server's answer if it is (almost) ready; otherwise answer - // without it at once, and announce the better result to follow. - try { - return merge(ctx, answer.get(BUDGET, TimeUnit.MILLISECONDS), local); - } - catch (final TimeoutException exc) { - return merge(ctx, null, local).withUpdate(answer.handle((server, - error) -> server == null ? null : merge(ctx, server, local))); - } - catch (final InterruptedException exc) { - Thread.currentThread().interrupt(); - } - catch (final ExecutionException | CancellationException exc) { - // NB: The server failed (and said why in the log); the next request - // asks again. - } - return merge(ctx, null, local); - } - - @Override - public void prepare(final CompletionRequest request) { - // Warm up: start the language server (building its environment, if - // needed), and have it analyze the script against its environment, in - // the background. - final LspClient client = lspClient(); - if (client == null) return; - client.prepare(document(request, true, true), uri(request)); - } - - @Override - public void closed(final CompletionRequest request) { - final LspClient client; - synchronized (this) { - client = lsp; - } - if (client != null) client.close(uri(request)); - } - - @Override - public ScriptDialect dialect() { - return DIALECT; - } - - @Override - public Hover hover(final CompletionRequest request) { - final String text = request.text(); - final int at = request.offset(); - if (PythonSyntax.analyze(text, at).kind == Kind.NONE) { - return Hover.NONE; // NB: In a comment or string. - } - final LspClient client = lspClient(); - if (client == null) return Hover.NONE; - final CompletableFuture hover = client.hover(document(request, - true, false), uri(request), at).handle((h, error) -> h); - try { - final Hover h = hover.get(BUDGET, TimeUnit.MILLISECONDS); - return h == null ? Hover.NONE : h; - } - catch (final TimeoutException exc) { - return Hover.NONE.withUpdate(hover); - } - catch (final InterruptedException exc) { - Thread.currentThread().interrupt(); - } - catch (final ExecutionException | CancellationException exc) { - // NB: The server failed. - } - return Hover.NONE; - } - - @Override - public CompletableFuture> diagnose( - final CompletionRequest request) - { - final LspClient client = lspClient(); - if (client == null) return super.diagnose(request); - return client.diagnose(document(request, true, false), uri(request), - DIAGNOSE_TIMEOUT); - } - - @Override - public SignatureHelp signatureHelp(final CompletionRequest request) { - final String text = request.text(); - final int caret = request.offset(); - if (PythonSyntax.analyze(text, caret).kind == Kind.NONE) { - return SignatureHelp.NONE; // NB: In a comment or string. - } - final int open = Signatures.callStart(text, caret); - if (open < 0) return SignatureHelp.NONE; - final List args = Signatures.arguments(text, open, caret); - if (args == null) return SignatureHelp.NONE; - final LspClient client = lspClient(); - if (client == null) return SignatureHelp.NONE; - - // NB: Where the server cannot tell the argument, count the commas. - final CompletableFuture help = client.signatureHelp( - document(request, true, false), uri(request), caret, open, args.size() - - 1).handle((h, error) -> h); - try { - final SignatureHelp h = help.get(BUDGET, TimeUnit.MILLISECONDS); - return h == null ? SignatureHelp.NONE : h; - } - catch (final TimeoutException exc) { - return SignatureHelp.NONE.withUpdate(help); - } - catch (final InterruptedException exc) { - Thread.currentThread().interrupt(); - } - catch (final ExecutionException | CancellationException exc) { - // NB: The server failed. - } - return SignatureHelp.NONE; - } - - /** - * Merges the server's completions (if any) with the local ones: the - * server's first, since they carry types, signatures and documentation; - * then whatever only the local analysis knows. - */ - private static CompletionResult merge(final PythonSyntax.Context ctx, - final List server, final List local) - { - final List out = new ArrayList<>(); - final Set seen = new HashSet<>(); - if (server != null) { - for (final Completion c : server) { - if (!matches(c.insertionText(), ctx.prefix)) continue; - if (ctx.kind == Kind.IMPORT) { - // NB: An import names things rather than calling them: one plain - // name per function or class, without its parameters. - if (seen.add(c.insertionText())) out.add(name(c)); - continue; - } - out.add(c); - seen.add(c.insertionText()); - } - } - for (final Completion c : local) { - if (seen.add(c.insertionText())) out.add(c); - } - return new CompletionResult(out, ctx.replaceStart); - } - - /** Sets the dependency overlays, e.g. to use fake ones in tests. */ - synchronized void setOverlays(final DependencyOverlays overlays) { - this.overlays = overlays; - } - - /** Sets the language server client, e.g. to use a fake one in tests. */ - synchronized void setLspClient(final LspClient lsp) { - this.lsp = lsp; - } - - // -- Helper methods -- - - /** - * A completion inserting just the name of {@code c}: not callable, so the - * editor inserts no parameter list for it. - */ - static Completion name(final Completion c) { - final Completion.Kind kind = c.kind() == Completion.Kind.METHOD - ? Completion.Kind.OTHER : c.kind(); - return Completion.builder(c.insertionText()).kind(kind).summary(c - .summary()).lazyDescription(c::description).build(); - } - - /** Completions from the local (syntactic) analysis alone. */ - private static List localCompletions( - final PythonSyntax.Context ctx, final Map params) - { - final List out = new ArrayList<>(); - final String prefix = ctx.prefix; - if (ctx.kind == Kind.ATTRIBUTE) { - for (final String name : ctx.attributesUsed) { - if (matches(name, prefix)) out.add(Completion.builder(name).kind( - Completion.Kind.OTHER).summary("Used elsewhere in this script") - .build()); - } - return out; - } - if (ctx.kind != Kind.NAME) return out; - for (final String name : ctx.names) { - if (matches(name, prefix)) out.add(Completion.builder(name).kind( - Completion.Kind.VARIABLE).build()); - } - for (final Map.Entry p : params.entrySet()) { - if (matches(p.getKey(), prefix)) out.add(Completion.builder(p.getKey()) - .kind(Completion.Kind.VARIABLE).summary(p.getValue() == null ? "Script parameter" - : p.getValue() + " (script parameter)").build()); - } - if (matches("task", prefix)) out.add(Completion.builder("task").kind( - Completion.Kind.VARIABLE).summary(ApposePythonDialect.TASK_TYPE).build()); - for (final String name : PythonSyntax.BUILTINS) { - if (matches(name, prefix)) out.add(Completion.builder(name).kind( - Completion.Kind.OTHER).summary("builtin").build()); - } - for (final String keyword : PythonSyntax.KEYWORDS) { - if (matches(keyword, prefix)) out.add(Completion.builder(keyword).kind( - Completion.Kind.KEYWORD).build()); - } - return out; - } - - /** - * Gets the script as plain Python, for the server: its parameters - * declared, and - * (if asked) where it runs: its environment's Python, and its overlay. - * - * @param environment Whether to find where the script runs. - * @param settled Whether the dependencies are known not to be changing. - */ - private ScriptDocument document(final CompletionRequest request, - final boolean environment, final boolean settled) - { - final String text = request.text(); - EnvironmentContext env = null; - if (environment) { - final String python = scriptPython(text, request.path()); - final String overlay = overlay(text, request.path(), python, settled); - env = new EnvironmentContext(python, overlay == null ? null - : Collections.singletonList(overlay), null); - } - final CodeCompletionService service = service( - CodeCompletionService.class); - final Map> params = service == null ? Collections - .emptyMap() : service.scriptParameters(text); - return ScriptDocument.of(text, params, DIALECT, env); - } - - /** The URI of the script, for the language server. */ - private static String uri(final CompletionRequest request) { - return LspClient.uri(request.path(), "script.py"); - } - - /** - * Gets the overlay of packages the script declares but its environment - * lacks (see {@link DependencyOverlays}), if ready; starts preparing it - * otherwise. Only for environments declared inline, for now. - * - * @param settled Whether the dependencies are known not to be changing. - * @return The overlay directory, or null if none is needed or ready. - */ - private String overlay(final String text, final String path, - final String python, final boolean settled) - { - final DependencyOverlays overlays = overlays(); - if (overlays == null) return null; - final List dependencies; - final String name; - try { - final String metadata = InlineMetadata.extract(text); - if (metadata == null) return null; - dependencies = InlineMetadata.dependencies(metadata); - // NB: Unsaved scripts' environments are named after their metadata, - // which changes as dependencies are edited: share one overlay. - name = path == null ? "unsaved" : ScriptEnvironment.inlineName(metadata, - path); - } - catch (final RuntimeException exc) { - return null; // NB: E.g. half-typed metadata. - } - return overlays.overlay(name, python, dependencies, settled); - } - - /** Gets the dependency overlays, creating them on first use. */ - private synchronized DependencyOverlays overlays() { - if (overlays != null) return overlays; - final ResidentWorkerService workers = service( - ResidentWorkerService.class); - if (workers == null) return null; - overlays = new DependencyOverlays((python, dependencies, - dir) -> installOverlay(workers, python, dependencies, dir), new File( - Environments.apposeEnvsDir(), TOOLING_ENV + "-overlays"), - OVERLAY_SETTLE, System::currentTimeMillis); - return overlays; - } - - /** - * Installs an overlay by running overlay.py on the tooling environment's - * worker. Called on the overlays' background thread. - */ - private static String installOverlay(final ResidentWorkerService workers, - final String python, final List dependencies, final File dir) - { - final Map inputs = new HashMap<>(); - inputs.put("python", python); - inputs.put("dependencies", dependencies); - inputs.put("overlay", dir.getAbsolutePath()); - try { - final Service.Task task = toolingWorker(workers).task(resource( - "completion/overlay.py"), inputs); - task.waitFor(); - final Object overlay = task.outputs.get("path"); - return overlay instanceof String ? (String) overlay : null; - } - catch (final Exception exc) { - throw new IllegalStateException("Overlay installation failed", exc); - } - } - - /** - * Gets the Python executable of the script's environment, if it has been - * built already: in this session (e.g. by running the script), or in an - * earlier one. Completion never builds it. Returns null if not built. - */ - private String scriptPython(final String text, final String path) { - // NB: Called on every keystroke, so remember what was found (but not - // what was not: the environment may be built at any moment). - final String key; - try { - key = path + "\n" + ScriptEnvironment.directive(text) + "\n" + - InlineMetadata.extract(text); - } - catch (final RuntimeException exc) { - return null; // NB: E.g. half-typed metadata. - } - synchronized (pythons) { - final String known = pythons.get(key); - if (known != null && new File(known).isFile()) return known; - } - final String python = findScriptPython(text, path); - if (python != null) { - synchronized (pythons) { - pythons.put(key, python); - } - } - return python; - } - - /** As {@link #scriptPython}, without remembering. */ - private String findScriptPython(final String text, final String path) { - final ScriptEnvironment env; - try { - final Map directive = ScriptEnvironment.directive(text); - env = ScriptEnvironment.of(text, path, directive.get("env"), directive - .get("scheme")); - } - catch (final IOException | RuntimeException exc) { - return null; // NB: E.g. half-typed metadata. - } - if (env == null) return null; - // NB: A read-only lookup: worker(...) would replace the worker whenever - // the configuration changed, i.e. as the user edits it. - final ResidentWorkerService workers = service( - ResidentWorkerService.class); - final ResidentWorker worker = workers == null ? null : workers.find( - env.name, env.config()); - if (worker != null && worker.environment().isBuilt()) { - try { - final String python = pythonExecutable(worker.environment().get()); - if (python != null) return python; - } - catch (final Exception exc) { - // NB: Fall back to looking on disk. - } - } - // Built in an earlier session? Its files are where Appose keeps them. - return pythonOnDisk(new File(Environments.apposeEnvsDir(), env.name)); - } - - /** Finds the Python executable of a built environment, or null. */ - static String pythonExecutable(final Environment env) { - // NB: Look in the bin paths, and in their parents: on Windows, conda-style - // (e.g. pixi) environments keep python.exe in the environment's root, - // above the bin directory that Appose reports. - final List dirs = new ArrayList<>(); - for (final String dir : env.binPaths()) { - dirs.add(new File(dir)); - final File parent = new File(dir).getParentFile(); - if (parent != null) dirs.add(parent); - } - if (env.base() != null) { - final File base = new File(env.base()); - dirs.add(base); - dirs.add(new File(base, "Scripts")); - dirs.add(new File(base, "bin")); - } - return findPython(dirs); - } - - /** - * Finds the Python executable of an environment in the given directory, as - * Appose builds them (pixi, uv or conda), without building or starting - * anything; or null if there is none. - */ - static String pythonOnDisk(final File envDir) { - if (!envDir.isDirectory()) return null; - final List dirs = new ArrayList<>(); - // pixi: .pixi/envs/, with "default" first. - final File pixiEnvs = new File(envDir, ".pixi" + File.separator + "envs"); - final List pixi = new ArrayList<>(); - pixi.add(new File(pixiEnvs, "default")); - final File[] others = pixiEnvs.listFiles(File::isDirectory); - if (others != null) pixi.addAll(Arrays.asList(others)); - for (final File p : pixi) { - dirs.add(p); - dirs.add(new File(p, "bin")); - } - // uv (a virtual environment) and conda/mamba. - for (final File base : new File[] { envDir, new File(envDir, ".venv") }) { - dirs.add(new File(base, "Scripts")); - dirs.add(new File(base, "bin")); - dirs.add(base); - } - return findPython(dirs); - } - - private static String findPython(final List dirs) { - for (final File dir : dirs) { - for (final String exe : new String[] { "python.exe", "python", - "python3" }) - { - final File f = new File(dir, exe); - if (f.isFile()) return f.getAbsolutePath(); - } - } - return null; - } - - /** - * Gets the language server client, creating it on first use; for the - * Python servers of the context (see {@link ApposePythonLspServerPlugin}). - */ - private synchronized LspClient lspClient() { - if (lsp != null) return lsp; - final LspService service = service(LspService.class); - final LspServers servers = service == null ? null : service.servers( - "python"); - if (servers == null) return null; - lsp = new PythonLspClient(servers); - return lsp; - } - - /** - * How to start jedi-language-server: with the Python of the tooling - * environment, built first if needed (but starting no worker). - */ - static JediLanguageServerSpec serverSpec( - final ResidentWorkerService workers) - { - final File envs = new File(Environments.apposeEnvsDir()); - return new JediLanguageServerSpec(() -> { - final String python = pythonExecutable(toolingWorker(workers) - .environment().get()); - if (python == null) throw new IOException("No Python in " + - TOOLING_ENV); - return python; - }, new File(envs, TOOLING_ENV + "-workspace"), new File(envs, TOOLING_ENV + - "-server.log")); - } - - /** Gets the tooling environment's worker, building it first if needed. */ - private static ResidentWorker toolingWorker( - final ResidentWorkerService workers) - { - final String config = toolingConfig(); - return workers.worker(TOOLING_ENV, config, () -> Appose.content(config) - .scheme("pixi.toml"), Environment::python); - } - - /** - * The tooling environment: Python, Jedi, jedi-language-server, NumPy and - * Appose; and uv, to install packages for {@link DependencyOverlays}. - */ - static String toolingConfig() { - return "[workspace]\n" + // - "name = \"" + TOOLING_ENV + "\"\n" + // - "channels = [\"conda-forge\"]\n" + // - "platforms = [\"" + InlineMetadata.condaPlatform() + "\"]\n" + // - "\n" + // - "[dependencies]\n" + // - "python = \"3.12.*\"\n" + // - "jedi = \">=0.19.2\"\n" + // - // NB: Script parameters are typed as NumPy arrays (see - // ApposePythonDialect): known even before a script's environment is. - "numpy = \"*\"\n" + // - "\n" + // - "[pypi-dependencies]\n" + // - "appose = \"*\"\n" + // - "jedi-language-server = \">=0.47\"\n" + // - "uv = \"*\"\n"; - } - - /** Gets a task script (e.g. overlay.py), by resource name. */ - private static String resource(final String name) throws IOException { - synchronized (SCRIPTS) { - final String known = SCRIPTS.get(name); - if (known != null) return known; - try (final InputStream in = ApposePythonCodeCompleter.class - .getResourceAsStream(name)) - { - if (in == null) throw new IOException(name + " not found"); - final java.io.ByteArrayOutputStream out = - new java.io.ByteArrayOutputStream(); - final byte[] buf = new byte[8192]; - int n; - while ((n = in.read(buf)) > 0) out.write(buf, 0, n); - final String script = new String(out.toByteArray(), - StandardCharsets.UTF_8); - SCRIPTS.put(name, script); - return script; - } - } - } - - private S service( - final Class type) - { - final Context context = getContext(); - return context == null ? null : context.getService(type); - } - - private static boolean matches(final String name, final String prefix) { - if (name.startsWith("__") && !prefix.startsWith("_")) return false; - return name.toLowerCase(Locale.ROOT).startsWith(prefix.toLowerCase( - Locale.ROOT)) && !name.equals(prefix); - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java new file mode 100644 index 0000000..3a66417 --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java @@ -0,0 +1,398 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.WeakHashMap; + +import org.apposed.appose.Appose; +import org.apposed.appose.Environment; +import org.apposed.appose.Service; +import org.apposed.appose.util.Environments; +import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; +import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; +import org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec; + +/** + * Where Python scripts run via Appose, as their language servers need to + * know: the Python of a script's environment, if built (in this session, or an + * earlier one; never building it, nor disturbing its worker); the overlay of + * packages it declares but its environment lacks (see + * {@link DependencyOverlays}); and the tooling environment jedi-language-server + * runs in. + * + * @author Gabriel Selzer + */ +final class ApposePythonEnvironments { + + /** Name of the tooling environment in which the language server runs. */ + static final String TOOLING_ENV = "appose-python-completion"; + + /** + * How long a script's dependencies must stay unchanged before packages + * missing from its environment are installed for completion, in ms. + */ + private static final long OVERLAY_SETTLE = 1500; + + /** Task scripts (see overlay.py), by resource name. */ + private static final Map SCRIPTS = new HashMap<>(); + + /** One per resident worker service (that is, per context). */ + private static final Map ALL = + new WeakHashMap<>(); + + private final ResidentWorkerService workers; + + /** Python executables of script environments, by script and metadata. */ + private final Map pythons = new LinkedHashMap( + 16, 0.75f, true) + { + + @Override + protected boolean removeEldestEntry( + final Map.Entry eldest) + { + return size() > 16; + } + }; + + private DependencyOverlays overlays; + + private ApposePythonEnvironments(final ResidentWorkerService workers) { + this.workers = workers; + } + + /** Gets those of a context (by its resident worker service). */ + static ApposePythonEnvironments of(final ResidentWorkerService workers) { + synchronized (ALL) { + return ALL.computeIfAbsent(workers, ApposePythonEnvironments::new); + } + } + + /** + * Gets where a script runs, for its language server: its environment's + * Python (or null, if not built), and its overlay (if any, and ready). + * + * @param uri The script's URI (a file path, if saved). + * @param settled Whether the dependencies are known not to be changing + * (e.g. on opening the script). + */ + org.scijava.code.lsp.Environment environment(final String text, + final String uri, final boolean settled) + { + final String path = path(uri); + final String python = scriptPython(text, path); + final String overlay = overlay(text, path, python, settled); + return new org.scijava.code.lsp.Environment(python, overlay == null ? null + : Collections.singletonList(overlay), null, null); + } + + /** Sets the dependency overlays, e.g. to use fake ones in tests. */ + synchronized void setOverlays(final DependencyOverlays overlays) { + this.overlays = overlays; + } + + /** The file path of a {@code file:} URI, or null. */ + static String path(final String uri) { + try { + final URI u = URI.create(uri); + return "file".equals(u.getScheme()) ? new File(u).getPath() : null; + } + catch (final IllegalArgumentException exc) { + return null; + } + } + + /** + * Gets the overlay of packages the script declares but its environment + * lacks (see {@link DependencyOverlays}), if ready; starts preparing it + * otherwise. Only for environments declared inline, for now. + * + * @param settled Whether the dependencies are known not to be changing. + * @return The overlay directory, or null if none is needed or ready. + */ + String overlay(final String text, final String path, + final String python, final boolean settled) + { + final DependencyOverlays overlays = overlays(); + if (overlays == null) return null; + final List dependencies; + final String name; + try { + final String metadata = InlineMetadata.extract(text); + if (metadata == null) return null; + dependencies = InlineMetadata.dependencies(metadata); + // NB: Unsaved scripts' environments are named after their metadata, + // which changes as dependencies are edited: share one overlay. + name = path == null ? "unsaved" : ScriptEnvironment.inlineName(metadata, + path); + } + catch (final RuntimeException exc) { + return null; // NB: E.g. half-typed metadata. + } + return overlays.overlay(name, python, dependencies, settled); + } + + /** Gets the dependency overlays, creating them on first use. */ + private synchronized DependencyOverlays overlays() { + if (overlays != null) return overlays; + if (workers == null) return null; + overlays = new DependencyOverlays((python, dependencies, + dir) -> installOverlay(workers, python, dependencies, dir), new File( + Environments.apposeEnvsDir(), TOOLING_ENV + "-overlays"), + OVERLAY_SETTLE, System::currentTimeMillis); + return overlays; + } + + /** + * Installs an overlay by running overlay.py on the tooling environment's + * worker. Called on the overlays' background thread. + */ + private static String installOverlay(final ResidentWorkerService workers, + final String python, final List dependencies, final File dir) + { + final Map inputs = new HashMap<>(); + inputs.put("python", python); + inputs.put("dependencies", dependencies); + inputs.put("overlay", dir.getAbsolutePath()); + try { + final Service.Task task = toolingWorker(workers).task(resource( + "completion/overlay.py"), inputs); + task.waitFor(); + final Object overlay = task.outputs.get("path"); + return overlay instanceof String ? (String) overlay : null; + } + catch (final Exception exc) { + throw new IllegalStateException("Overlay installation failed", exc); + } + } + + /** + * Gets the Python executable of the script's environment, if it has been + * built already: in this session (e.g. by running the script), or in an + * earlier one. Completion never builds it. Returns null if not built. + */ + String scriptPython(final String text, final String path) { + // NB: Called on every keystroke, so remember what was found (but not + // what was not: the environment may be built at any moment). + final String key; + try { + key = path + "\n" + ScriptEnvironment.directive(text) + "\n" + + InlineMetadata.extract(text); + } + catch (final RuntimeException exc) { + return null; // NB: E.g. half-typed metadata. + } + synchronized (pythons) { + final String known = pythons.get(key); + if (known != null && new File(known).isFile()) return known; + } + final String python = findScriptPython(text, path); + if (python != null) { + synchronized (pythons) { + pythons.put(key, python); + } + } + return python; + } + + /** As {@link #scriptPython}, without remembering. */ + private String findScriptPython(final String text, final String path) { + final ScriptEnvironment env; + try { + final Map directive = ScriptEnvironment.directive(text); + env = ScriptEnvironment.of(text, path, directive.get("env"), directive + .get("scheme")); + } + catch (final IOException | RuntimeException exc) { + return null; // NB: E.g. half-typed metadata. + } + if (env == null) return null; + // NB: A read-only lookup: worker(...) would replace the worker whenever + // the configuration changed, i.e. as the user edits it. + final ResidentWorker worker = workers == null ? null : workers.find( + env.name, env.config()); + if (worker != null && worker.environment().isBuilt()) { + try { + final String python = pythonExecutable(worker.environment().get()); + if (python != null) return python; + } + catch (final Exception exc) { + // NB: Fall back to looking on disk. + } + } + // Built in an earlier session? Its files are where Appose keeps them. + return pythonOnDisk(new File(Environments.apposeEnvsDir(), env.name)); + } + + /** Finds the Python executable of a built environment, or null. */ + static String pythonExecutable(final Environment env) { + // NB: Look in the bin paths, and in their parents: on Windows, conda-style + // (e.g. pixi) environments keep python.exe in the environment's root, + // above the bin directory that Appose reports. + final List dirs = new ArrayList<>(); + for (final String dir : env.binPaths()) { + dirs.add(new File(dir)); + final File parent = new File(dir).getParentFile(); + if (parent != null) dirs.add(parent); + } + if (env.base() != null) { + final File base = new File(env.base()); + dirs.add(base); + dirs.add(new File(base, "Scripts")); + dirs.add(new File(base, "bin")); + } + return findPython(dirs); + } + + /** + * Finds the Python executable of an environment in the given directory, as + * Appose builds them (pixi, uv or conda), without building or starting + * anything; or null if there is none. + */ + static String pythonOnDisk(final File envDir) { + if (!envDir.isDirectory()) return null; + final List dirs = new ArrayList<>(); + // pixi: .pixi/envs/, with "default" first. + final File pixiEnvs = new File(envDir, ".pixi" + File.separator + "envs"); + final List pixi = new ArrayList<>(); + pixi.add(new File(pixiEnvs, "default")); + final File[] others = pixiEnvs.listFiles(File::isDirectory); + if (others != null) pixi.addAll(Arrays.asList(others)); + for (final File p : pixi) { + dirs.add(p); + dirs.add(new File(p, "bin")); + } + // uv (a virtual environment) and conda/mamba. + for (final File base : new File[] { envDir, new File(envDir, ".venv") }) { + dirs.add(new File(base, "Scripts")); + dirs.add(new File(base, "bin")); + dirs.add(base); + } + return findPython(dirs); + } + + private static String findPython(final List dirs) { + for (final File dir : dirs) { + for (final String exe : new String[] { "python.exe", "python", + "python3" }) + { + final File f = new File(dir, exe); + if (f.isFile()) return f.getAbsolutePath(); + } + } + return null; + } + + /** + * How to start jedi-language-server: with the Python of the tooling + * environment, built first if needed (but starting no worker). + */ + static JediLanguageServerSpec serverSpec( + final ResidentWorkerService workers) + { + final File envs = new File(Environments.apposeEnvsDir()); + return new JediLanguageServerSpec(() -> { + final String python = pythonExecutable(toolingWorker(workers) + .environment().get()); + if (python == null) throw new IOException("No Python in " + + TOOLING_ENV); + return python; + }, new File(envs, TOOLING_ENV + "-workspace"), new File(envs, TOOLING_ENV + + "-server.log")); + } + + /** Gets the tooling environment's worker, building it first if needed. */ + private static ResidentWorker toolingWorker( + final ResidentWorkerService workers) + { + final String config = toolingConfig(); + return workers.worker(TOOLING_ENV, config, () -> Appose.content(config) + .scheme("pixi.toml"), Environment::python); + } + + /** + * The tooling environment: Python, Jedi, jedi-language-server, NumPy and + * Appose; and uv, to install packages for {@link DependencyOverlays}. + */ + static String toolingConfig() { + return "[workspace]\n" + // + "name = \"" + TOOLING_ENV + "\"\n" + // + "channels = [\"conda-forge\"]\n" + // + "platforms = [\"" + InlineMetadata.condaPlatform() + "\"]\n" + // + "\n" + // + "[dependencies]\n" + // + "python = \"3.12.*\"\n" + // + "jedi = \">=0.19.2\"\n" + // + // NB: Script parameters are typed as NumPy arrays (see + // ApposePythonDialect): known even before a script's environment is. + "numpy = \"*\"\n" + // + "\n" + // + "[pypi-dependencies]\n" + // + "appose = \"*\"\n" + // + "jedi-language-server = \">=0.47\"\n" + // + "uv = \"*\"\n"; + } + + /** Gets a task script (e.g. overlay.py), by resource name. */ + private static String resource(final String name) throws IOException { + synchronized (SCRIPTS) { + final String known = SCRIPTS.get(name); + if (known != null) return known; + try (final InputStream in = ApposePythonEnvironments.class + .getResourceAsStream(name)) + { + if (in == null) throw new IOException(name + " not found"); + final java.io.ByteArrayOutputStream out = + new java.io.ByteArrayOutputStream(); + final byte[] buf = new byte[8192]; + int n; + while ((n = in.read(buf)) > 0) out.write(buf, 0, n); + final String script = new String(out.toByteArray(), + StandardCharsets.UTF_8); + SCRIPTS.put(name, script); + return script; + } + } + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java index 94b16fb..43769bf 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java @@ -55,6 +55,6 @@ public boolean supports(final String languageId) { @Override public LspServerSpec spec() { - return ApposePythonCodeCompleter.serverSpec(workers); + return ApposePythonEnvironments.serverSpec(workers); } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java new file mode 100644 index 0000000..4a5e7bc --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java @@ -0,0 +1,110 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import java.util.Collections; +import java.util.Map; + +import org.eclipse.lsp4j.TextDocumentItem; +import org.eclipse.lsp4j.services.LanguageServer; +import org.scijava.Priority; +import org.scijava.code.api.CodeCompletionService; +import org.scijava.code.api.ScriptDocument; +import org.scijava.code.lsp.Environment; +import org.scijava.code.lsp.LanguageServerPlugin; +import org.scijava.code.lsp.LanguageServerService; +import org.scijava.code.lsp.PythonCompletionServer; +import org.scijava.code.lsp.TransformingLanguageServer; +import org.scijava.plugin.Parameter; +import org.scijava.plugin.Plugin; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; +import org.scijava.script.ScriptLanguage; + +/** + * Offers jedi-language-server (see {@link ApposePythonLspServerPlugin}) for + * Python scripts run via Appose: members of objects and modules, signatures, + * documentation, hover and syntax errors. Once a script's environment is built + * (by running the script), the server analyzes the script against it, so its + * packages are known; packages the script declares but its environment lacks + * are installed for completion only (see + * {@link org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays}). + * The server sees the script as plain Python, its parameters declared with + * the types they have when it runs (e.g. images as NumPy arrays). + *

    + * Completion never builds a script's environment, and never disturbs its + * worker. + *

    + * + * @author Gabriel Selzer + */ +@Plugin(type = LanguageServerPlugin.class, name = "jedi-language-server", + priority = Priority.HIGH) +public class JediServerPlugin implements LanguageServerPlugin { + + /** How many completions get signatures, once a word is begun. */ + private static final int SIGNATURES = 30; + + @Parameter + private LanguageServerService servers; + + @Parameter + private ResidentWorkerService workers; + + @Parameter(required = false) + private CodeCompletionService completion; + + @Override + public boolean supports(final ScriptLanguage language) { + return language instanceof ApposePythonScriptLanguage && servers.launch( + "python", Environment.NONE) != null; + } + + @Override + public Environment environment(final TextDocumentItem document) { + // NB: On opening the script, its dependencies are settled. + return ApposePythonEnvironments.of(workers).environment(document.getText(), + document.getUri(), document.getVersion() <= 1); + } + + @Override + public LanguageServer create(final Environment environment) { + return new PythonCompletionServer(new TransformingLanguageServer(servers + .launch("python", environment), text -> document(text, environment)), + SIGNATURES); + } + + /** The script as plain Python, for the server. */ + ScriptDocument document(final String text, final Environment environment) { + final Map> params = completion == null ? Collections + .emptyMap() : completion.scriptParameters(text); + return ScriptDocument.of(text, params, PythonSyntaxLanguageServer.DIALECT, + environment.toContext()); + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java new file mode 100644 index 0000000..5238847 --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java @@ -0,0 +1,145 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; + +import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionItemKind; +import org.eclipse.lsp4j.CompletionList; +import org.scijava.code.api.CodeCompletionService; +import org.scijava.code.lsp.ScriptLanguageServer; +import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; +import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax; +import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Kind; + +/** + * An in-process language server for Python scripts run via Appose, from the + * syntax alone ({@link PythonSyntax}: tree-sitter): instantly, with no Python + * needed, names in scope, the script's parameters, {@code task}, builtins and + * keywords; and, after a dot, the names used after dots elsewhere in the + * script. (A Python language server, e.g. jedi-language-server, knows more; + * see {@link JediServerPlugin}.) + * + * @author Gabriel Selzer + */ +public class PythonSyntaxLanguageServer extends ScriptLanguageServer { + + static final ApposePythonDialect DIALECT = new ApposePythonDialect(); + + private final CodeCompletionService completion; + + /** @param completion Reads the scripts' parameters. */ + public PythonSyntaxLanguageServer(final CodeCompletionService completion) { + super(null); + this.completion = completion; + } + + @Override + protected CompletionList complete(final Document doc, final int caret) { + final PythonSyntax.Context ctx = PythonSyntax.analyze(doc.text(), caret); + if (ctx.kind == Kind.NONE || ctx.kind == Kind.NAME && ctx.prefix + .isEmpty()) + { + return list(doc, caret, caret, new ArrayList<>()); + } + return list(doc, ctx.replaceStart, caret, localCompletions(ctx, parameters( + doc.text()))); + } + + /** + * The script's parameters, with the Python types they have when the script + * runs (null if unknown). + */ + private Map parameters(final String text) { + final Map params = new LinkedHashMap<>(); + if (completion == null) return params; + completion.scriptParameters(text).forEach((name, type) -> params.put(name, + DIALECT.runtimeType(type))); + return params; + } + + private static CompletionItem item(final String name, + final CompletionItemKind kind, final String detail) + { + final CompletionItem item = new CompletionItem(name); + item.setKind(kind); + item.setInsertText(name); + item.setDetail(detail); + return item; + } + + /** Completions from the local (syntactic) analysis alone. */ + private static List localCompletions( + final PythonSyntax.Context ctx, final Map params) + { + final List out = new ArrayList<>(); + final String prefix = ctx.prefix; + if (ctx.kind == Kind.ATTRIBUTE) { + for (final String name : ctx.attributesUsed) { + if (matches(name, prefix)) out.add(item(name, CompletionItemKind.Text, + "Used elsewhere in this script")); + } + return out; + } + if (ctx.kind != Kind.NAME) return out; + for (final String name : ctx.names) { + if (matches(name, prefix)) out.add(item(name, + CompletionItemKind.Variable, null)); + } + for (final Map.Entry p : params.entrySet()) { + if (matches(p.getKey(), prefix)) out.add(item(p.getKey(), + CompletionItemKind.Variable, p.getValue() == null ? "Script parameter" + : p.getValue() + " (script parameter)")); + } + if (matches("task", prefix)) out.add(item("task", + CompletionItemKind.Variable, ApposePythonDialect.TASK_TYPE)); + for (final String name : PythonSyntax.BUILTINS) { + if (matches(name, prefix)) out.add(item(name, CompletionItemKind.Function, + "builtin")); + } + for (final String keyword : PythonSyntax.KEYWORDS) { + if (matches(keyword, prefix)) out.add(item(keyword, + CompletionItemKind.Keyword, null)); + } + return out; + } + + private static boolean matches(final String name, final String prefix) { + if (name.startsWith("__") && !prefix.startsWith("_")) return false; + return name.toLowerCase(Locale.ROOT).startsWith(prefix.toLowerCase( + Locale.ROOT)) && !name.equals(prefix); + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java new file mode 100644 index 0000000..6a9566c --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java @@ -0,0 +1,62 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import org.eclipse.lsp4j.services.LanguageServer; +import org.scijava.code.api.CodeCompletionService; +import org.scijava.code.lsp.Environment; +import org.scijava.code.lsp.LanguageServerPlugin; +import org.scijava.plugin.Parameter; +import org.scijava.plugin.Plugin; +import org.scijava.script.ScriptLanguage; + +/** + * Offers the syntax-only language server for Python scripts run via Appose + * (see {@link PythonSyntaxLanguageServer}): instant, and needing no Python. + * Its completions follow jedi-language-server's (see {@link JediServerPlugin}). + * + * @author Gabriel Selzer + */ +@Plugin(type = LanguageServerPlugin.class, name = "Python syntax") +public class PythonSyntaxServerPlugin implements LanguageServerPlugin { + + @Parameter(required = false) + private CodeCompletionService completion; + + @Override + public boolean supports(final ScriptLanguage language) { + return language instanceof ApposePythonScriptLanguage; + } + + @Override + public LanguageServer create(final Environment environment) { + return new PythonSyntaxLanguageServer(completion); + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java index 5c13d83..e211e8f 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java @@ -49,7 +49,7 @@ * directive. *

    * Both running a script ({@link ApposePythonScriptEngine}) and completing code - * in it ({@link ApposePythonCodeCompleter}) resolve its environment here, so + * in it ({@link JediServerPlugin}) resolve its environment here, so * that they agree on the environment's name and configuration. *

    */ diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java deleted file mode 100644 index e067fec..0000000 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCodeCompleterTest.java +++ /dev/null @@ -1,606 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertTrue; - -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.List; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.CountDownLatch; -import java.util.concurrent.TimeUnit; -import java.util.function.Function; -import java.util.stream.Collectors; - -import net.imglib2.img.Img; - -import org.eclipse.lsp4j.Position; -import org.junit.After; -import org.junit.Before; -import org.junit.Test; -import org.scijava.Context; -import org.scijava.code.api.CodeCompletionService; -import org.scijava.code.api.Completion; -import org.scijava.code.api.CompletionRequest; -import org.scijava.code.api.EnvironmentContext; -import org.scijava.code.api.ScriptDocument; -import org.scijava.code.api.SignatureHelp; -import org.scijava.code.lsp.LspClient; -import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; -import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; -import org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec; -import org.scijava.script.ScriptLanguage; -import org.scijava.script.ScriptService; - -/** - * Tests {@link ApposePythonCodeCompleter}, with a fake language server client. - * - * @author Gabriel Selzer - */ -public class ApposePythonCodeCompleterTest { - - private static final ApposePythonDialect DIALECT = new ApposePythonDialect(); - - private Context context; - private ScriptLanguage language; - private CodeCompletionService service; - private ApposePythonCodeCompleter completer; - - @Before - public void setUp() { - context = new Context(ScriptService.class, CodeCompletionService.class, - ResidentWorkerService.class); - language = context.service(ScriptService.class).getLanguageByName( - "appose-python"); - service = context.service(CodeCompletionService.class); - completer = (ApposePythonCodeCompleter) service.getCompleterPlugin( - language); - } - - @After - public void tearDown() { - context.dispose(); - } - - @Test - public void testCompleterIsDiscovered() { - assertTrue(completer != null); - } - - @Test - public void testLocalCompletions() { - noServer(); - final String header = "#@ net.imglib2.img.Img image\n#@ double sigma\n"; - final List names = complete(header + - "def blur(img, s):\n result = img\n re"); - assertTrue(names.toString(), names.contains("result")); - assertTrue(names.contains("return")); - assertTrue(complete(header + "x = si").contains("sigma")); - assertTrue(complete(header + "pri").contains("print")); - assertTrue(complete(header + "ta").contains("task")); - // Nothing on parameter lines, in strings or comments. - assertTrue(complete("#@ String na").isEmpty()); - assertTrue(complete("s = 'pri").isEmpty()); - } - - @Test - public void testServerResultsMergedFirst() { - final Completion zeros = Completion.builder("zeros").kind( - Completion.Kind.METHOD).parameters(Collections.singletonList( - new Completion.Parameter("shape", "_ShapeLike"))).build(); - final Completion zeta = Completion.of("zeta"); - fake(q -> Arrays.asList(zeros, zeta)); - final List completions = completions( - "import numpy as np\nnp.zeros(3)\nnp.ze"); - assertEquals("zeros", completions.get(0).insertionText()); - assertEquals(1, completions.get(0).parameters().size()); - assertEquals(Arrays.asList("zeros", "zeta"), completions.stream().map( - Completion::insertionText).collect(Collectors.toList())); - } - - @Test - public void testImportsAreNames() { - // The server reports imshow as a function, with its signature. - final Completion imshow = Completion.builder("imshow").kind( - Completion.Kind.METHOD).parameters(Collections.singletonList( - new Completion.Parameter("data", "Any"))).summary("def imshow") - .build(); - fake(q -> Arrays.asList(imshow, imshow)); - // In an import, it is a name to import, not a call: listed once, without - // a parameter list for the editor to insert. - final List imported = completions("from ndv import i"); - assertEquals(1, imported.size()); - assertEquals("imshow", imported.get(0).insertionText()); - assertFalse(imported.get(0).isCallable()); - assertEquals("def imshow", imported.get(0).summary()); - // Elsewhere, it is still a call. - assertTrue(completions("import ndv\nndv.i").get(0).isCallable()); - } - - @Test - public void testSameQuestionWhileTyping() { - final FakeLsp fake = fake(q -> Collections.singletonList(Completion.of( - "zeros"))); - // Later keystrokes of a word ask about its first character, so that the - // client can reuse the answer. - final String code = "import numpy as np\nnp.z"; - complete(code); - complete(code + "e"); - complete(code + "er"); - assertEquals(3, fake.asked.size()); - for (final Asked a : fake.asked) assertEquals(code.length(), a.offset); - } - - @Test - public void testUpdateWhenServerIsSlow() throws Exception { - final CountDownLatch release = new CountDownLatch(1); - fake(q -> { - try { - release.await(); - } - catch (final InterruptedException exc) { - Thread.currentThread().interrupt(); - } - return Collections.singletonList(Completion.of("zeros")); - }); - // At once: what is known locally, and the announcement of more. - final String code = "import numpy as np\nnp.z"; - final long start = System.nanoTime(); - final org.scijava.code.api.CompletionResult first = service.complete( - new CompletionRequest(code, language, null)); - assertTrue(System.nanoTime() - start < TimeUnit.SECONDS.toNanos(2)); - assertTrue(first.completions().isEmpty()); - assertTrue(first.update() != null); - // Then the server's answer, merged. - release.countDown(); - final org.scijava.code.api.CompletionResult better = first.update().get(5, - TimeUnit.SECONDS); - assertEquals("zeros", better.completions().get(0).insertionText()); - assertEquals(first.replaceStart(), better.replaceStart()); - } - - @Test - public void testSignaturesOnlyOnceWordBegun() { - final FakeLsp fake = fake(q -> Collections.emptyList()); - complete("import numpy\nnumpy."); - complete("import numpy\nnumpy.z"); - assertEquals(Arrays.asList(0, 30), fake.asked.stream().map(a -> a.resolve) - .collect(Collectors.toList())); - } - - @Test - public void testOverlayInQuery() throws Exception { - // A script declaring a dependency its environment lacks. - final List> installed = new ArrayList<>(); - completer.setOverlays(new org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays( - (python, dependencies, dir) -> { - installed.add(dependencies); - return "/overlays/blur"; - }, new java.io.File("overlays"), 0, System::currentTimeMillis)); - final FakeLsp fake = fake(q -> Collections.emptyList()); - final String code = "# /// script\n# dependencies = [\"tifffile\"]\n" + - "# ///\nimport tifffile\ntifffile.im"; - // The first request notes the dependencies; once they have stayed the - // same (here, for 0 ms), the next one starts installing the overlay. - complete(code); - complete(code); - for (int i = 0; i < 100 && installed.isEmpty(); i++) Thread.sleep(20); - Thread.sleep(50); - complete(code + "r"); - // Once the overlay is ready, the server searches it too. - assertEquals(Collections.singletonList(Collections.singletonList( - "tifffile")), installed); - assertEquals(Collections.singletonList("/overlays/blur"), last(fake).doc - .environment().searchPaths()); - } - - @Test - public void testSignatureHelp() throws Exception { - final Completion dumps = Completion.builder("dumps").kind( - Completion.Kind.METHOD).parameters(Arrays.asList( - new Completion.Parameter("obj", null), new Completion.Parameter( - "indent", "int"))).build(); - final FakeLsp fake = fake(q -> Collections.emptyList()); - fake.signatures = Collections.singletonList(new SignatureHelp.Signature( - dumps, SignatureHelp.Fit.UNKNOWN)); - final String code = "import json\njson.dumps(obj, "; - SignatureHelp help = completer.signatureHelp(new CompletionRequest(code, - language, null)); - if (help.update() != null) help = help.update().get(5, TimeUnit.SECONDS); - assertEquals("dumps", help.signatures().get(0).callable().insertionText()); - // The argument the caret is in, counted (for when the server cannot - // tell); and the call's opening parenthesis. - assertEquals(1, help.activeParameter()); - assertEquals(code.indexOf('('), help.callStart()); - // The server was asked about the caret (shifted by the preamble). - final Asked asked = last(fake); - assertEquals(code.length(), asked.offset); - assertEquals(new Position(2, "json.dumps(obj, ".length()), position( - asked)); - // No help outside calls, or in strings. - assertTrue(completer.signatureHelp(new CompletionRequest("x = 1", - language, null)).isEmpty()); - assertTrue(completer.signatureHelp(new CompletionRequest("s = 'f(1, ", - language, null)).isEmpty()); - } - - @Test - public void testPrepareAndClose() throws Exception { - final FakeLsp fake = fake(q -> Collections.emptyList()); - completer.prepare(new CompletionRequest("import numpy\n", language, - null)); - assertEquals(1, fake.asked.size()); - assertEquals("untitled:/script.py", fake.asked.get(0).uri); - completer.closed(new CompletionRequest("import numpy\n", language, null)); - assertEquals(Collections.singletonList("untitled:/script.py"), - fake.closed); - } - - @Test - public void testHoverAndDiagnostics() throws Exception { - final FakeLsp fake = fake(q -> Collections.emptyList()); - final String code = "#@ double sigma\nimport json\njson.dumps(sigma)"; - org.scijava.code.api.Hover hover = completer.hover(new CompletionRequest( - code, code.indexOf("dumps") + 1, language, null, null, null)); - if (hover.update() != null) hover = hover.update().get(5, - TimeUnit.SECONDS); - assertEquals("
    dumps
    ", hover.text()); - assertEquals(code.indexOf("dumps") + 1, last(fake).offset); - // Not in comments or strings. - assertTrue(completer.hover(new CompletionRequest("# json.dumps", 4, - language, null, null, null)).isEmpty()); - - final List problems = completer.diagnose( - new CompletionRequest(code, language, null)).get(5, TimeUnit.SECONDS); - assertEquals(1, problems.size()); - assertTrue(last(fake).doc.text().endsWith(code)); - } - - @Test - public void testServerQuery() { - final FakeLsp fake = fake(q -> Collections.emptyList()); - final String code = "#@ net.imglib2.img.Img image\n#@ double sigma\n" + - "image.sha"; - complete(code); - final Asked asked = last(fake); - // The script's parameters are typed in a preamble, so the server knows - // them. - assertTrue(asked.doc.text().startsWith( - "import appose.python_worker; import numpy; image: numpy.ndarray; " + - "sigma: float; task: appose.python_worker.Task\n")); - // The caret is after the word's first character, shifted by the preamble. - assertEquals(new Position(3, "image.s".length()), position(asked)); - // No script environment has been built: the server uses its own Python. - assertNull(asked.doc.environment().interpreter()); - } - - @Test - public void testServerOptions() { - final JediLanguageServerSpec spec = new JediLanguageServerSpec(() -> null, - new java.io.File("workspace"), new java.io.File("server.log")); - assertEquals("python", spec.languageId()); - final String diagnostics = "diagnostics={enable=true, didOpen=true, " + - "didChange=true, didSave=false}"; - assertEquals("{workspace={environmentPath=/env/bin/python, " + - "extraPaths=[/overlay]}, " + diagnostics + "}", String.valueOf(spec - .initializationOptions(new EnvironmentContext("/env/bin/python", - Collections.singletonList("/overlay"), null)))); - // Without a script environment: the server's own. - assertEquals("{workspace={extraPaths=[]}, " + diagnostics + "}", String - .valueOf(spec.initializationOptions(EnvironmentContext.EMPTY))); - assertTrue(spec.rootUri(EnvironmentContext.EMPTY).startsWith("file:")); - new java.io.File("workspace").delete(); - } - - @Test - public void testScriptWorkersUntouched() { - noServer(); - // Completing in a script declaring an environment must not create (or - // replace) a worker for it, even as its metadata is edited. - final String script = "# /// script\n# dependencies = [\"appose\"]\n" + - "# ///\nimport os\nos.pa"; - complete(script); - complete(script.replace("appose", "appose, numpy")); - assertTrue(context.service(ResidentWorkerService.class).workers() - .isEmpty()); - } - - @Test - public void testPythonExecutable() throws Exception { - final java.nio.file.Path tmp = java.nio.file.Files.createTempDirectory( - "appose-envs"); - try { - // pixi on Windows: python.exe is in the environment's root, above the - // bin directory Appose reports (which may not even exist). - final java.nio.file.Path pixiEnv = tmp.resolve( - "pixi/.pixi/envs/default"); - java.nio.file.Files.createDirectories(pixiEnv); - final java.nio.file.Path pixiPython = java.nio.file.Files.createFile( - pixiEnv.resolve("python.exe")); - assertEquals(pixiPython.toFile().getAbsolutePath(), - ApposePythonCodeCompleter.pythonExecutable(env(tmp.resolve("pixi"), - pixiEnv.resolve("bin")))); - // uv on Windows: python.exe is in the reported Scripts directory. - final java.nio.file.Path scripts = tmp.resolve("uv/Scripts"); - java.nio.file.Files.createDirectories(scripts); - final java.nio.file.Path uvPython = java.nio.file.Files.createFile( - scripts.resolve("python.exe")); - assertEquals(uvPython.toFile().getAbsolutePath(), - ApposePythonCodeCompleter.pythonExecutable(env(tmp.resolve("uv"), - scripts))); - // No Python at all. - assertNull(ApposePythonCodeCompleter.pythonExecutable(env(tmp.resolve( - "none"), tmp.resolve("none/bin")))); - } - finally { - delete(tmp); - } - } - - @Test - public void testPythonOnDisk() throws Exception { - final java.nio.file.Path tmp = java.nio.file.Files.createTempDirectory( - "appose-envs"); - try { - for (final String layout : new String[] { - ".pixi/envs/default/python.exe", // pixi, Windows - ".pixi/envs/default/bin/python", // pixi, Linux & macOS - "Scripts/python.exe", // uv, Windows - "bin/python" }) // uv or conda, Linux & macOS - { - final java.nio.file.Path env = tmp.resolve("env" + layout.hashCode()); - final java.nio.file.Path python = env.resolve(layout); - java.nio.file.Files.createDirectories(python.getParent()); - java.nio.file.Files.createFile(python); - assertEquals(layout, python.toFile().getAbsolutePath(), - ApposePythonCodeCompleter.pythonOnDisk(env.toFile())); - } - assertNull(ApposePythonCodeCompleter.pythonOnDisk(tmp.resolve("missing") - .toFile())); - } - finally { - delete(tmp); - } - } - - @Test - public void testEnvironmentFromEarlierSession() throws Exception { - // A script whose environment was built in an earlier session: it is on - // disk, but no worker for it exists in this one. - final java.nio.file.Path tmp = java.nio.file.Files.createTempDirectory( - "appose-envs"); - final String old = System.getProperty("appose.envs-dir"); - System.setProperty("appose.envs-dir", tmp.toString()); - try { - final java.io.File script = tmp.resolve("blur.py").toFile(); - final String header = "# /// script\n# dependencies = [\"numpy\"]\n" + - "# ///\nimport numpy\n"; - final String envName = ScriptEnvironment.of(header, script.getPath(), - null, null).name; - final java.nio.file.Path python = tmp.resolve(envName).resolve( - ".pixi/envs/default/python.exe"); - java.nio.file.Files.createDirectories(python.getParent()); - java.nio.file.Files.createFile(python); - - final FakeLsp fake = fake(q -> Collections.emptyList()); - final String code = header + "numpy.ze"; - service.complete(new CompletionRequest(code, code.length(), language, - null, null, script.getPath())); - // The server analyzes against the environment's Python... - final Asked asked = last(fake); - assertEquals(python.toFile().getAbsolutePath(), asked.doc.environment() - .interpreter()); - // ...of the saved script, by its file... - assertEquals(script.toURI().toString(), asked.uri); - // ...without anything built or started. - assertTrue(context.service(ResidentWorkerService.class).workers() - .isEmpty()); - } - finally { - if (old == null) System.clearProperty("appose.envs-dir"); - else System.setProperty("appose.envs-dir", old); - delete(tmp); - } - } - - @Test - public void testPythonTypes() { - assertEquals("numpy.ndarray", DIALECT.runtimeType(Img.class)); - assertEquals("float", DIALECT.runtimeType(double.class)); - assertEquals("int", DIALECT.runtimeType(Long.class)); - assertEquals("str", DIALECT.runtimeType(String.class)); - assertEquals("bool", DIALECT.runtimeType(boolean.class)); - assertEquals("list", DIALECT.runtimeType(int[].class)); - assertNull(DIALECT.runtimeType(Context.class)); - } - - // -- Helper methods -- - - private static void delete(final java.nio.file.Path dir) throws Exception { - try (java.util.stream.Stream paths = - java.nio.file.Files.walk(dir)) - { - paths.sorted(java.util.Comparator.reverseOrder()).forEach(p -> p - .toFile().delete()); - } - } - - /** A built environment with the given base and bin directory. */ - private static org.apposed.appose.Environment env( - final java.nio.file.Path base, final java.nio.file.Path bin) - { - return new org.apposed.appose.Environment() { - - @Override - public String base() { - return base.toString(); - } - - @Override - public List binPaths() { - return Collections.singletonList(bin.toString()); - } - - @Override - public List launchArgs() { - return Collections.emptyList(); - } - - @Override - public org.apposed.appose.Builder builder() { - return null; - } - }; - } - - /** Uses a server that never answers. */ - private void noServer() { - completer.setLspClient(new FakeLsp(null)); - } - - /** Uses a server answering via {@code answers}, asynchronously. */ - private FakeLsp fake(final Function> answers) { - final FakeLsp fake = new FakeLsp(answers); - completer.setLspClient(fake); - return fake; - } - - private static Asked last(final FakeLsp fake) { - return fake.asked.get(fake.asked.size() - 1); - } - - /** Where in the server's document a question was asked. */ - private static Position position(final Asked asked) { - final String text = asked.doc.text(); - final int at = asked.doc.map().toTransformed(asked.offset); - final int lineStart = text.lastIndexOf('\n', at - 1) + 1; - int line = 0; - for (int i = 0; i < lineStart; i++) { - if (text.charAt(i) == '\n') line++; - } - return new Position(line, at - lineStart); - } - - private List completions(final String code) { - return service.complete(new CompletionRequest(code, language, null)) - .completions(); - } - - private List complete(final String code) { - return completions(code).stream().map(Completion::insertionText).collect( - Collectors.toList()); - } - - /** A question asked of the language server. */ - private static final class Asked { - - private final ScriptDocument doc; - private final String uri; - private final int offset, resolve; - - private Asked(final ScriptDocument doc, final String uri, - final int offset, final int resolve) - { - this.doc = doc; - this.uri = uri; - this.offset = offset; - this.resolve = resolve; - } - } - - /** A language server client answering from a function. */ - private static final class FakeLsp extends LspClient { - - private final Function> answers; - private final List asked = new java.util.concurrent.CopyOnWriteArrayList<>(); - private final List closed = new ArrayList<>(); - private List signatures = Collections.emptyList(); - - private FakeLsp(final Function> answers) { - super(null); - this.answers = answers; - } - - @Override - public CompletableFuture> complete( - final ScriptDocument doc, final String uri, final int offset, - final int resolve) - { - final Asked q = new Asked(doc, uri, offset, resolve); - asked.add(q); - if (answers == null) return new CompletableFuture<>(); - return CompletableFuture.supplyAsync(() -> answers.apply(q)); - } - - @Override - public CompletableFuture signatureHelp( - final ScriptDocument doc, final String uri, final int offset, - final int callStart, final int active) - { - asked.add(new Asked(doc, uri, offset, 0)); - return CompletableFuture.completedFuture(new SignatureHelp(signatures, - active, callStart)); - } - - @Override - public CompletableFuture hover( - final ScriptDocument doc, final String uri, final int offset) - { - asked.add(new Asked(doc, uri, offset, 0)); - return CompletableFuture.completedFuture(new org.scijava.code.api.Hover( - "
    dumps
    ", offset, offset)); - } - - @Override - public CompletableFuture> diagnose( - final ScriptDocument doc, final String uri, final long timeout) - { - asked.add(new Asked(doc, uri, 0, 0)); - return CompletableFuture.completedFuture(Collections.singletonList( - new org.scijava.code.api.Diagnostic(0, 1, - org.scijava.code.api.Diagnostic.Severity.ERROR, "Oops", "fake"))); - } - - @Override - public void prepare(final ScriptDocument doc, final String uri) { - asked.add(new Asked(doc, uri, doc.original().length(), 0)); - } - - @Override - public void close(final String uri) { - closed.add(uri); - } - } -} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java index 78a502f..9cfd558 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -39,20 +39,33 @@ import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.util.Collections; +import java.util.HashMap; import java.util.List; +import java.util.Map; import java.util.concurrent.TimeUnit; +import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionList; +import org.eclipse.lsp4j.CompletionParams; +import org.eclipse.lsp4j.Diagnostic; +import org.eclipse.lsp4j.DidChangeTextDocumentParams; +import org.eclipse.lsp4j.DidOpenTextDocumentParams; +import org.eclipse.lsp4j.Hover; +import org.eclipse.lsp4j.HoverParams; +import org.eclipse.lsp4j.SignatureHelp; +import org.eclipse.lsp4j.SignatureHelpParams; +import org.eclipse.lsp4j.TextDocumentContentChangeEvent; +import org.eclipse.lsp4j.TextDocumentIdentifier; +import org.eclipse.lsp4j.TextDocumentItem; +import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import org.scijava.Context; -import org.scijava.code.api.CodeCompletionService; -import org.scijava.code.api.Completion; -import org.scijava.code.api.CompletionRequest; -import org.scijava.code.api.CompletionResult; -import org.scijava.code.api.Diagnostic; -import org.scijava.code.api.Hover; -import org.scijava.code.api.SignatureHelp; +import org.scijava.code.lsp.LanguageServerService; +import org.scijava.code.lsp.LspClient; +import org.scijava.code.lsp.MergedLanguageServer; +import org.scijava.code.lsp.UpdatingCompletionList; import org.scijava.module.ModuleService; import org.scijava.script.ScriptInfo; import org.scijava.script.ScriptLanguage; @@ -60,8 +73,9 @@ import org.scijava.script.ScriptService; /** - * End-to-end tests of {@link ApposePythonCodeCompleter} with a real language - * server (jedi-language-server). + * End-to-end tests of the language servers of Python scripts run via Appose + * (see {@link JediServerPlugin}), with a real jedi-language-server, through + * the {@link LanguageServerService}, as editors use it. *

    * The first completion builds the completion tooling environment (Python and * jedi-language-server, via pixi), and a script run builds the small uv @@ -81,7 +95,7 @@ public class ApposePythonCompletionIntegrationTest { private static Context context; private static ScriptLanguage language; - private static CodeCompletionService service; + private static MergedLanguageServer server; private static File scriptDir; @BeforeClass @@ -89,7 +103,7 @@ public static void setUp() throws URISyntaxException { context = new Context(); language = context.service(ScriptService.class).getLanguageByName( "appose-python"); - service = context.service(CodeCompletionService.class); + server = context.service(LanguageServerService.class).server(language); scriptDir = new File(ApposePythonCompletionIntegrationTest.class .getResource("/appose-test/requirements.txt").toURI()).getParentFile(); } @@ -103,13 +117,15 @@ public static void tearDown() { public void testStandardLibrary() throws Exception { // Before any script environment is built, the server knows the standard // library, with signatures and documentation. - final List completions = completeUntil( + final List completions = completeUntil( "import json\njson.du", null, "dumps"); - final Completion dumps = completions.stream().filter(c -> c - .insertionText().equals("dumps")).findFirst().get(); - assertTrue(dumps.isCallable()); - assertFalse(dumps.parameters().isEmpty()); - assertTrue(dumps.description() != null); + final CompletionItem dumps = completions.stream().filter(c -> c.getLabel() + .equals("dumps") && c.getLabelDetails() != null).findFirst().get(); + assertTrue(dumps.getLabelDetails().getDetail(), dumps.getLabelDetails() + .getDetail().startsWith("(Any obj, bool skipkeys")); + assertTrue(dumps.getDocumentation() != null || server + .resolveCompletionItem(dumps).get(1, TimeUnit.MINUTES) + .getDocumentation() != null); report("import json\njson.l", null, "loads", "o"); } @@ -127,8 +143,8 @@ public void testScriptEnvironment() throws Exception { moduleService.run(module, false, Collections.singletonMap("image", null)).get(); - // Now the server analyzes the script against its environment: numpy is known, - // and so is the type of the image parameter. + // Now the server analyzes the script against its environment: numpy is + // known, and so is the type of the image parameter. completeUntil(header + "np.ze", script.getPath(), "zeros"); completeUntil(header + "image.sha", script.getPath(), "shape"); report(header + "np.random.r", script.getPath(), "randn", "a"); @@ -137,57 +153,57 @@ public void testScriptEnvironment() throws Exception { @Test public void testSignatureHelp() throws Exception { final String code = "import json\njson.dumps(obj, "; + final String uri = sync(code, null); final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; - SignatureHelp help = SignatureHelp.NONE; - while (help.isEmpty() && System.currentTimeMillis() < deadline) { - help = completer().signatureHelp(new CompletionRequest(code, language, - null)); - if (help.update() != null) { - final SignatureHelp better = help.update().get(1, TimeUnit.MINUTES); - if (better != null) help = better; - } + SignatureHelp help = null; + while (help == null && System.currentTimeMillis() < deadline) { + help = server.signatureHelp(new SignatureHelpParams( + new TextDocumentIdentifier(uri), LspClient.position(code, code + .length()))).get(1, TimeUnit.MINUTES); } - assertEquals("dumps", help.signatures().get(0).callable().insertionText()); - assertFalse(help.signatures().get(0).callable().parameters().isEmpty()); - assertEquals(1, help.activeParameter()); - assertEquals(code.indexOf('('), help.callStart()); + assertTrue(help.getSignatures().get(0).getLabel().contains("dumps(")); + assertFalse(help.getSignatures().get(0).getParameters().isEmpty()); + assertEquals(Integer.valueOf(1), help.getActiveParameter()); } @Test public void testHover() throws Exception { final String code = "import json\njson.dumps(1)"; - final CompletionRequest request = new CompletionRequest(code, code - .indexOf("dumps") + 2, language, null, null, null); + final String uri = sync(code, null); final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; - Hover hover = Hover.NONE; - while (hover.isEmpty() && System.currentTimeMillis() < deadline) { - hover = completer().hover(request); - if (hover.update() != null) { - final Hover better = hover.update().get(1, TimeUnit.MINUTES); - if (better != null) hover = better; - } + Hover hover = null; + while (hover == null && System.currentTimeMillis() < deadline) { + hover = server.hover(new HoverParams(new TextDocumentIdentifier(uri), + LspClient.position(code, code.indexOf("dumps") + 2))).get(1, + TimeUnit.MINUTES); } - assertTrue(hover.text(), hover.text().contains("JSON")); - assertEquals(code.indexOf("dumps"), hover.start()); - assertEquals(code.indexOf("dumps") + "dumps".length(), hover.end()); + assertTrue(String.valueOf(hover), hover.getContents().getRight().getValue() + .contains("JSON")); + // The word's range, in the script. + assertEquals(LspClient.position(code, code.indexOf("dumps")), hover + .getRange().getStart()); + assertEquals(LspClient.position(code, code.indexOf("dumps") + 5), hover + .getRange().getEnd()); } @Test public void testDiagnostics() throws Exception { // A syntax error, on the script's third line. final String code = "#@ double sigma\nx = sigma\ny = (1 +\n"; + final LanguageServerService servers = context.service( + LanguageServerService.class); final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; List problems = Collections.emptyList(); while (problems.isEmpty() && System.currentTimeMillis() < deadline) { - problems = completer().diagnose(new CompletionRequest(code, language, - null)).get(2, TimeUnit.MINUTES); + problems = servers.diagnose(language, "untitled:/diagnose.py", code, + 120_000); } assertFalse(problems.isEmpty()); - assertTrue(problems.toString(), problems.get(0).start() >= code.indexOf( - "y = ")); + assertTrue(problems.toString(), problems.get(0).getRange().getStart() + .getLine() >= 2); // Fixed: no more problems. - assertTrue(completer().diagnose(new CompletionRequest(code.replace("(1 +", - "(1 + 2)"), language, null)).get(1, TimeUnit.MINUTES).isEmpty()); + assertTrue(servers.diagnose(language, "untitled:/diagnose.py", code + .replace("(1 +", "(1 + 2)"), 120_000).isEmpty()); } @Test @@ -203,16 +219,16 @@ public void testServerOutlivesNoParent() throws Exception { // The server, once its parent (this JVM) is gone, must exit: here, its // parent is a process that already has. completeUntil("import json\njson.du", null, "dumps"); - final String python = ApposePythonCodeCompleter.pythonOnDisk(new File( + final String python = ApposePythonEnvironments.pythonOnDisk(new File( org.apposed.appose.util.Environments.apposeEnvsDir(), - ApposePythonCodeCompleter.TOOLING_ENV)); + ApposePythonEnvironments.TOOLING_ENV)); final Process gone = new ProcessBuilder(python, "-c", "pass").start(); gone.waitFor(); - final Process server = new ProcessBuilder(python, "-c", + final Process process = new ProcessBuilder(python, "-c", org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec.LAUNCHER, String.valueOf(gone.pid())).redirectErrorStream(true).start(); - final boolean exited = server.waitFor(30, TimeUnit.SECONDS); - if (!exited) server.destroyForcibly(); + final boolean exited = process.waitFor(30, TimeUnit.SECONDS); + if (!exited) process.destroyForcibly(); assertTrue(exited); } @@ -227,68 +243,97 @@ public void testNewDependency() throws Exception { StandardCharsets.UTF_8)); // Opening it prepares completion, including the dependency's overlay. final String code = header + "tifffile.imr"; - completer().prepare(new CompletionRequest(code, code.length(), language, - null, null, script.getPath())); + sync(code, script.getPath()); completeUntil(code, script.getPath(), "imread"); } // -- Helper methods -- - private static ApposePythonCodeCompleter completer() { - return (ApposePythonCodeCompleter) service.getCompleterPlugin(language); + /** The open documents' versions, by URI. */ + private static final Map versions = new HashMap<>(); + + /** Opens a script on the server, or changes it; returns its URI. */ + private static String sync(final String code, final String path) { + final String uri = path == null ? "untitled:/integration.py" : new File( + path).toURI().toString(); + final Integer version = versions.get(uri); + if (version == null) { + versions.put(uri, 1); + server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(uri, + "appose-python", 1, code))); + } + else { + versions.put(uri, version + 1); + server.didChange(new DidChangeTextDocumentParams( + new VersionedTextDocumentIdentifier(uri, version + 1), Collections + .singletonList(new TextDocumentContentChangeEvent(code)))); + } + return uri; } /** Completes until the expected completion appears (environments build). */ - private static List completeUntil(final String code, - final String path, final String expected) + private static List completeUntil(final String code, + final String path, final String expected) throws Exception { final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; while (System.currentTimeMillis() < deadline) { - final List completions = complete(code, path); - if (completions.stream().anyMatch(c -> c.insertionText().equals( - expected))) return completions; - try { - Thread.sleep(1000); - } - catch (final InterruptedException exc) { - throw new RuntimeException(exc); + final List completions = complete(code, path, true); + if (completions.stream().anyMatch(c -> c.getLabel().equals(expected))) { + return completions; } + Thread.sleep(1000); } fail("No '" + expected + "' completion for: " + code); return null; } - private static List complete(final String code, - final String path) + /** + * Completes at the end of the code. + * + * @param settled Whether to wait for the slower servers' completions. + */ + private static List complete(final String code, + final String path, final boolean settled) throws Exception { - return service.complete(new CompletionRequest(code, code.length(), - language, null, null, path)).completions(); + final String uri = sync(code, path); + CompletionList list = server.completion(new CompletionParams( + new TextDocumentIdentifier(uri), LspClient.position(code, code.length()))) + .get(1, TimeUnit.MINUTES).getRight(); + if (settled && list instanceof UpdatingCompletionList) { + final CompletionList better = ((UpdatingCompletionList) list).update() + .get(1, TimeUnit.MINUTES); + if (better != null) list = better; + } + return list.getItems(); } /** * Prints how long completions take, once warm: for another keystroke of a - * word the server has answered (cached); and for a new word, how long the editor - * waits (barely: it answers at once), and how long until the server's completions - * arrive as an update. + * word the server has answered (cached); and for a new word, how long the + * editor waits (barely: it answers at once), and how long until the + * server's completions arrive as an update. */ private static void report(final String code, final String path, final String expected, final String more) throws Exception { completeUntil(code, path, expected); long t0 = System.nanoTime(); - final int n = complete(code + more, path).size(); + final int n = complete(code + more, path, true).size(); final long cached = System.nanoTime() - t0; final String other = code.substring(0, code.lastIndexOf('.') + 1) + "s"; + final String uri = sync(other, path); t0 = System.nanoTime(); - final CompletionResult result = service.complete(new CompletionRequest( - other, other.length(), language, null, null, path)); + final CompletionList first = server.completion(new CompletionParams( + new TextDocumentIdentifier(uri), LspClient.position(other, other + .length()))).get(1, TimeUnit.MINUTES).getRight(); final long wait = System.nanoTime() - t0; - final CompletionResult update = result.update() == null ? result : result - .update().get(1, TimeUnit.MINUTES); + final CompletionList update = first instanceof UpdatingCompletionList + ? ((UpdatingCompletionList) first).update().get(1, TimeUnit.MINUTES) + : first; final long arrived = System.nanoTime() - t0; System.out.println("[completion timing] same word (cached): " + cached / 1000000 + " ms, " + n + " completions; new word: editor waits " + wait / - 1000000 + " ms, the server's " + update.completions().size() + + 1000000 + " ms, all servers' " + update.getItems().size() + " completions after " + arrived / 1000000 + " ms"); } } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java new file mode 100644 index 0000000..1cf5c3a --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java @@ -0,0 +1,362 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; + +import net.imglib2.img.Img; + +import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionParams; +import org.eclipse.lsp4j.DidOpenTextDocumentParams; +import org.eclipse.lsp4j.TextDocumentIdentifier; +import org.eclipse.lsp4j.TextDocumentItem; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.scijava.Context; +import org.scijava.code.api.CodeCompletionService; +import org.scijava.code.lsp.Environment; +import org.scijava.code.lsp.LanguageServerPlugin; +import org.scijava.code.lsp.LanguageServerService; +import org.scijava.code.lsp.LspClient; +import org.scijava.plugin.PluginService; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; +import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; +import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; +import org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec; +import org.scijava.script.ScriptLanguage; +import org.scijava.script.ScriptService; + +/** + * Tests the language servers of Python scripts run via Appose, without + * starting any server process: the syntax-only server + * ({@link PythonSyntaxLanguageServer}), and what {@link JediServerPlugin} + * tells jedi-language-server (the script as plain Python, and where it runs). + * + * @author Gabriel Selzer + */ +public class ApposePythonServersTest { + + private static final ApposePythonDialect DIALECT = new ApposePythonDialect(); + + private Context context; + private ScriptLanguage language; + private JediServerPlugin jedi; + private ApposePythonEnvironments environments; + + @Before + public void setUp() { + context = new Context(ScriptService.class, CodeCompletionService.class, + ResidentWorkerService.class, LanguageServerService.class); + language = context.service(ScriptService.class).getLanguageByName( + "appose-python"); + jedi = context.service(PluginService.class).createInstancesOfType( + LanguageServerPlugin.class).stream().filter( + p -> p instanceof JediServerPlugin).map(p -> (JediServerPlugin) p) + .findFirst().get(); + environments = ApposePythonEnvironments.of(context.service( + ResidentWorkerService.class)); + } + + @After + public void tearDown() { + context.dispose(); + } + + @Test + public void testServersAreOffered() { + assertTrue(context.service(LanguageServerService.class).supports( + language)); + // NB: Here, appose-python offers the Python server itself. + assertTrue(jedi.supports(language)); + } + + @Test + public void testLocalCompletions() { + final String header = "#@ net.imglib2.img.Img image\n#@ double sigma\n"; + final List names = complete(header + + "def blur(img, s):\n result = img\n re"); + assertTrue(names.toString(), names.contains("result")); + assertTrue(names.contains("return")); + assertTrue(complete(header + "x = si").contains("sigma")); + assertTrue(complete(header + "pri").contains("print")); + assertTrue(complete(header + "ta").contains("task")); + // Nothing on parameter lines, in strings or comments. + assertTrue(complete("#@ String na").isEmpty()); + assertTrue(complete("s = 'pri").isEmpty()); + } + + @Test + public void testScriptAsPython() { + final String code = "#@ net.imglib2.img.Img image\n#@ double sigma\n" + + "image.sha"; + // The script's parameters are typed in a preamble, so the server knows + // them. + assertTrue(jedi.document(code, Environment.NONE).text().startsWith( + "import appose.python_worker; import numpy; image: numpy.ndarray; " + + "sigma: float; task: appose.python_worker.Task\n")); + } + + @Test + public void testEnvironmentWithoutOne() { + // No script environment has been built: the server's own Python. + final Environment env = jedi.environment(new TextDocumentItem( + "untitled:/a.py", "appose-python", 2, "import os\nos.pa")); + assertNull(env.interpreter()); + assertTrue(env.searchPaths().isEmpty()); + } + + @Test + public void testOverlay() throws Exception { + // A script declaring a dependency its environment lacks. + final List> installed = new ArrayList<>(); + environments.setOverlays(new DependencyOverlays((python, dependencies, + dir) -> { + installed.add(dependencies); + return "/overlays/blur"; + }, new File("overlays"), 0, System::currentTimeMillis)); + final String code = "# /// script\n# dependencies = [\"tifffile\"]\n" + + "# ///\nimport tifffile\ntifffile.im"; + final TextDocumentItem doc = new TextDocumentItem("untitled:/a.py", + "appose-python", 1, code); + // On opening the script, its dependencies are settled: the overlay is + // installed; once ready, the server searches it. + jedi.environment(doc); + for (int i = 0; i < 100 && installed.isEmpty(); i++) Thread.sleep(20); + Thread.sleep(50); + assertEquals(Collections.singletonList(Collections.singletonList( + "tifffile")), installed); + assertEquals(Collections.singletonList("/overlays/blur"), jedi.environment( + doc).searchPaths()); + } + + @Test + public void testScriptWorkersUntouched() { + // Finding where a script runs must not create (or replace) a worker for + // it, even as its metadata is edited. + final String script = "# /// script\n# dependencies = [\"appose\"]\n" + + "# ///\nimport os\nos.pa"; + jedi.environment(new TextDocumentItem("untitled:/a.py", "appose-python", 2, + script)); + jedi.environment(new TextDocumentItem("untitled:/a.py", "appose-python", 3, + script.replace("appose", "appose, numpy"))); + assertTrue(context.service(ResidentWorkerService.class).workers() + .isEmpty()); + } + + @Test + public void testEnvironmentFromEarlierSession() throws Exception { + // A script whose environment was built in an earlier session: it is on + // disk, but no worker for it exists in this one. + final java.nio.file.Path tmp = java.nio.file.Files.createTempDirectory( + "appose-envs"); + final String old = System.getProperty("appose.envs-dir"); + System.setProperty("appose.envs-dir", tmp.toString()); + try { + final File script = tmp.resolve("blur.py").toFile(); + final String header = "# /// script\n# dependencies = [\"numpy\"]\n" + + "# ///\nimport numpy\n"; + final String envName = ScriptEnvironment.of(header, script.getPath(), + null, null).name; + final java.nio.file.Path python = tmp.resolve(envName).resolve( + ".pixi/envs/default/python.exe"); + java.nio.file.Files.createDirectories(python.getParent()); + java.nio.file.Files.createFile(python); + + final Environment env = jedi.environment(new TextDocumentItem(script + .toURI().toString(), "appose-python", 2, header + "numpy.ze")); + // The server analyzes against the environment's Python... + assertEquals(python.toFile().getAbsolutePath(), env.interpreter()); + // ...without anything built or started. + assertTrue(context.service(ResidentWorkerService.class).workers() + .isEmpty()); + } + finally { + if (old == null) System.clearProperty("appose.envs-dir"); + else System.setProperty("appose.envs-dir", old); + delete(tmp); + } + } + + @Test + public void testPythonExecutable() throws Exception { + final java.nio.file.Path tmp = java.nio.file.Files.createTempDirectory( + "appose-envs"); + try { + // pixi on Windows: python.exe is in the environment's root, above the + // bin directory Appose reports (which may not even exist). + final java.nio.file.Path pixiEnv = tmp.resolve( + "pixi/.pixi/envs/default"); + java.nio.file.Files.createDirectories(pixiEnv); + final java.nio.file.Path pixiPython = java.nio.file.Files.createFile( + pixiEnv.resolve("python.exe")); + assertEquals(pixiPython.toFile().getAbsolutePath(), + ApposePythonEnvironments.pythonExecutable(env(tmp.resolve("pixi"), + pixiEnv.resolve("bin")))); + // uv on Windows: python.exe is in the reported Scripts directory. + final java.nio.file.Path scripts = tmp.resolve("uv/Scripts"); + java.nio.file.Files.createDirectories(scripts); + final java.nio.file.Path uvPython = java.nio.file.Files.createFile( + scripts.resolve("python.exe")); + assertEquals(uvPython.toFile().getAbsolutePath(), + ApposePythonEnvironments.pythonExecutable(env(tmp.resolve("uv"), + scripts))); + // No Python at all. + assertNull(ApposePythonEnvironments.pythonExecutable(env(tmp.resolve( + "none"), tmp.resolve("none/bin")))); + } + finally { + delete(tmp); + } + } + + @Test + public void testPythonOnDisk() throws Exception { + final java.nio.file.Path tmp = java.nio.file.Files.createTempDirectory( + "appose-envs"); + try { + for (final String layout : new String[] { + ".pixi/envs/default/python.exe", // pixi, Windows + ".pixi/envs/default/bin/python", // pixi, Linux & macOS + "Scripts/python.exe", // uv, Windows + "bin/python" }) // uv or conda, Linux & macOS + { + final java.nio.file.Path env = tmp.resolve("env" + layout.hashCode()); + final java.nio.file.Path python = env.resolve(layout); + java.nio.file.Files.createDirectories(python.getParent()); + java.nio.file.Files.createFile(python); + assertEquals(layout, python.toFile().getAbsolutePath(), + ApposePythonEnvironments.pythonOnDisk(env.toFile())); + } + assertNull(ApposePythonEnvironments.pythonOnDisk(tmp.resolve("missing") + .toFile())); + } + finally { + delete(tmp); + } + } + + @Test + public void testPythonTypes() { + assertEquals("numpy.ndarray", DIALECT.runtimeType(Img.class)); + assertEquals("float", DIALECT.runtimeType(double.class)); + assertEquals("int", DIALECT.runtimeType(Long.class)); + assertEquals("str", DIALECT.runtimeType(String.class)); + assertEquals("bool", DIALECT.runtimeType(boolean.class)); + assertEquals("list", DIALECT.runtimeType(int[].class)); + assertNull(DIALECT.runtimeType(Context.class)); + } + + @Test + public void testServerOptions() { + final JediLanguageServerSpec spec = new JediLanguageServerSpec(() -> null, + new File("workspace"), new File("server.log")); + assertEquals("python", spec.languageId()); + final String diagnostics = "diagnostics={enable=true, didOpen=true, " + + "didChange=true, didSave=false}"; + assertEquals("{workspace={environmentPath=/env/bin/python, " + + "extraPaths=[/overlay]}, " + diagnostics + "}", String.valueOf(spec + .initializationOptions(new org.scijava.code.api.EnvironmentContext( + "/env/bin/python", Collections.singletonList("/overlay"), null)))); + // Without a script environment: the server's own. + assertEquals("{workspace={extraPaths=[]}, " + diagnostics + "}", String + .valueOf(spec.initializationOptions( + org.scijava.code.api.EnvironmentContext.EMPTY))); + assertTrue(spec.rootUri(org.scijava.code.api.EnvironmentContext.EMPTY) + .startsWith("file:")); + new File("workspace").delete(); + } + + // -- Helper methods -- + + /** Completes with the syntax-only server. */ + private List complete(final String code) { + final PythonSyntaxLanguageServer server = new PythonSyntaxLanguageServer( + context.service(CodeCompletionService.class)); + server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem( + "untitled:/a.py", "appose-python", 1, code))); + try { + return server.completion(new CompletionParams(new TextDocumentIdentifier( + "untitled:/a.py"), LspClient.position(code, code.length()))).get(10, + TimeUnit.SECONDS).getRight().getItems().stream().map( + CompletionItem::getLabel).collect(Collectors.toList()); + } + catch (final Exception exc) { + throw new RuntimeException(exc); + } + } + + private static void delete(final java.nio.file.Path dir) throws Exception { + try (java.util.stream.Stream paths = + java.nio.file.Files.walk(dir)) + { + paths.sorted(java.util.Comparator.reverseOrder()).forEach(p -> p + .toFile().delete()); + } + } + + /** A built environment with the given base and bin directory. */ + private static org.apposed.appose.Environment env( + final java.nio.file.Path base, final java.nio.file.Path bin) + { + return new org.apposed.appose.Environment() { + + @Override + public String base() { + return base.toString(); + } + + @Override + public List binPaths() { + return Collections.singletonList(bin.toString()); + } + + @Override + public List launchArgs() { + return Collections.emptyList(); + } + + @Override + public org.apposed.appose.Builder builder() { + return null; + } + }; + } +} From 5ecc6a0e68c810fbe07ccce3d2220178b3e6a6db Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Wed, 7 Oct 2026 13:22:11 -0500 Subject: [PATCH 08/37] Move off scijava-code-api, now removed Its remaining types (ScriptDocument, ScriptDialect, ClassIndex, Signatures, Environment) come from scijava-code-lsp, and script parameters from the LanguageServerService. Co-Authored-By: Claude Opus 5.5 --- pom.xml | 9 ++------- .../appose/python/JediServerPlugin.java | 13 +++++-------- .../python/PythonSyntaxLanguageServer.java | 14 +++++++------- .../python/PythonSyntaxServerPlugin.java | 6 +++--- .../completion/ApposePythonDialect.java | 2 +- .../completion/JediLanguageServerSpec.java | 8 ++++---- ...ApposePythonCompletionIntegrationTest.java | 14 +++++++------- .../python/ApposePythonServersTest.java | 19 +++++++++---------- 8 files changed, 38 insertions(+), 47 deletions(-) diff --git a/pom.xml b/pom.xml index 878cc65..ca78e97 100644 --- a/pom.xml +++ b/pom.xml @@ -95,7 +95,7 @@ 2.101.0 - 0.1.0-SNAPSHOT + 0.1.0-SNAPSHOT 0.26.6 0.25.0 @@ -110,15 +110,10 @@ org.scijava script-editor - - org.scijava - scijava-code-api - ${scijava-code-api.version} - org.scijava scijava-code-lsp - ${scijava-code-api.version} + ${scijava-code-lsp.version} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java index 4a5e7bc..2fdbfd0 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java @@ -35,11 +35,10 @@ import org.eclipse.lsp4j.TextDocumentItem; import org.eclipse.lsp4j.services.LanguageServer; import org.scijava.Priority; -import org.scijava.code.api.CodeCompletionService; -import org.scijava.code.api.ScriptDocument; +import org.scijava.code.lsp.LanguageServerService; +import org.scijava.code.lsp.ScriptDocument; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; -import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.PythonCompletionServer; import org.scijava.code.lsp.TransformingLanguageServer; import org.scijava.plugin.Parameter; @@ -77,8 +76,6 @@ public class JediServerPlugin implements LanguageServerPlugin { @Parameter private ResidentWorkerService workers; - @Parameter(required = false) - private CodeCompletionService completion; @Override public boolean supports(final ScriptLanguage language) { @@ -102,9 +99,9 @@ public LanguageServer create(final Environment environment) { /** The script as plain Python, for the server. */ ScriptDocument document(final String text, final Environment environment) { - final Map> params = completion == null ? Collections - .emptyMap() : completion.scriptParameters(text); + final Map> params = servers == null ? Collections + .emptyMap() : servers.scriptParameters(text); return ScriptDocument.of(text, params, PythonSyntaxLanguageServer.DIALECT, - environment.toContext()); + environment); } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java index 5238847..75a0bbc 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java @@ -39,7 +39,7 @@ import org.eclipse.lsp4j.CompletionItem; import org.eclipse.lsp4j.CompletionItemKind; import org.eclipse.lsp4j.CompletionList; -import org.scijava.code.api.CodeCompletionService; +import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.ScriptLanguageServer; import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax; @@ -59,12 +59,12 @@ public class PythonSyntaxLanguageServer extends ScriptLanguageServer { static final ApposePythonDialect DIALECT = new ApposePythonDialect(); - private final CodeCompletionService completion; + private final LanguageServerService servers; - /** @param completion Reads the scripts' parameters. */ - public PythonSyntaxLanguageServer(final CodeCompletionService completion) { + /** @param servers Reads the scripts' parameters. */ + public PythonSyntaxLanguageServer(final LanguageServerService servers) { super(null); - this.completion = completion; + this.servers = servers; } @Override @@ -85,8 +85,8 @@ protected CompletionList complete(final Document doc, final int caret) { */ private Map parameters(final String text) { final Map params = new LinkedHashMap<>(); - if (completion == null) return params; - completion.scriptParameters(text).forEach((name, type) -> params.put(name, + if (servers == null) return params; + servers.scriptParameters(text).forEach((name, type) -> params.put(name, DIALECT.runtimeType(type))); return params; } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java index 6a9566c..2cc7f15 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java @@ -30,7 +30,7 @@ package org.scijava.plugins.scripting.appose.python; import org.eclipse.lsp4j.services.LanguageServer; -import org.scijava.code.api.CodeCompletionService; +import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; import org.scijava.plugin.Parameter; @@ -48,7 +48,7 @@ public class PythonSyntaxServerPlugin implements LanguageServerPlugin { @Parameter(required = false) - private CodeCompletionService completion; + private LanguageServerService servers; @Override public boolean supports(final ScriptLanguage language) { @@ -57,6 +57,6 @@ public boolean supports(final ScriptLanguage language) { @Override public LanguageServer create(final Environment environment) { - return new PythonSyntaxLanguageServer(completion); + return new PythonSyntaxLanguageServer(servers); } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index 837dd92..65a58ca 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -41,7 +41,7 @@ import net.imglib2.RandomAccessibleInterval; import org.apposed.appose.NDArray; -import org.scijava.code.api.ScriptDialect; +import org.scijava.code.lsp.ScriptDialect; /** * How Python scripts run via Appose see their script parameters: as the diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java index df6fde9..94f3173 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java @@ -35,7 +35,7 @@ import java.util.Map; import java.util.concurrent.Callable; -import org.scijava.code.api.EnvironmentContext; +import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LspConnection; import org.scijava.code.lsp.LspServerSpec; @@ -96,7 +96,7 @@ public String languageId() { } @Override - public LspConnection start(final EnvironmentContext env) + public LspConnection start(final Environment env) throws IOException { final String exe; @@ -120,7 +120,7 @@ public LspConnection start(final EnvironmentContext env) } @Override - public Object initializationOptions(final EnvironmentContext env) { + public Object initializationOptions(final Environment env) { final Map workspaceOptions = new LinkedHashMap<>(); // NB: Without an environment, Jedi uses the server's own Python. if (env.interpreter() != null) { @@ -139,7 +139,7 @@ public Object initializationOptions(final EnvironmentContext env) { } @Override - public String rootUri(final EnvironmentContext env) { + public String rootUri(final Environment env) { // NB: jedi-language-server applies its workspace options only with a // root. An empty one, so that nothing irrelevant gets indexed. workspace.mkdirs(); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java index 9cfd558..fb884f7 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -63,7 +63,7 @@ import org.junit.Test; import org.scijava.Context; import org.scijava.code.lsp.LanguageServerService; -import org.scijava.code.lsp.LspClient; +import org.scijava.code.lsp.Positions; import org.scijava.code.lsp.MergedLanguageServer; import org.scijava.code.lsp.UpdatingCompletionList; import org.scijava.module.ModuleService; @@ -158,7 +158,7 @@ public void testSignatureHelp() throws Exception { SignatureHelp help = null; while (help == null && System.currentTimeMillis() < deadline) { help = server.signatureHelp(new SignatureHelpParams( - new TextDocumentIdentifier(uri), LspClient.position(code, code + new TextDocumentIdentifier(uri), Positions.position(code, code .length()))).get(1, TimeUnit.MINUTES); } assertTrue(help.getSignatures().get(0).getLabel().contains("dumps(")); @@ -174,15 +174,15 @@ public void testHover() throws Exception { Hover hover = null; while (hover == null && System.currentTimeMillis() < deadline) { hover = server.hover(new HoverParams(new TextDocumentIdentifier(uri), - LspClient.position(code, code.indexOf("dumps") + 2))).get(1, + Positions.position(code, code.indexOf("dumps") + 2))).get(1, TimeUnit.MINUTES); } assertTrue(String.valueOf(hover), hover.getContents().getRight().getValue() .contains("JSON")); // The word's range, in the script. - assertEquals(LspClient.position(code, code.indexOf("dumps")), hover + assertEquals(Positions.position(code, code.indexOf("dumps")), hover .getRange().getStart()); - assertEquals(LspClient.position(code, code.indexOf("dumps") + 5), hover + assertEquals(Positions.position(code, code.indexOf("dumps") + 5), hover .getRange().getEnd()); } @@ -297,7 +297,7 @@ private static List complete(final String code, { final String uri = sync(code, path); CompletionList list = server.completion(new CompletionParams( - new TextDocumentIdentifier(uri), LspClient.position(code, code.length()))) + new TextDocumentIdentifier(uri), Positions.position(code, code.length()))) .get(1, TimeUnit.MINUTES).getRight(); if (settled && list instanceof UpdatingCompletionList) { final CompletionList better = ((UpdatingCompletionList) list).update() @@ -324,7 +324,7 @@ private static void report(final String code, final String path, final String uri = sync(other, path); t0 = System.nanoTime(); final CompletionList first = server.completion(new CompletionParams( - new TextDocumentIdentifier(uri), LspClient.position(other, other + new TextDocumentIdentifier(uri), Positions.position(other, other .length()))).get(1, TimeUnit.MINUTES).getRight(); final long wait = System.nanoTime() - t0; final CompletionList update = first instanceof UpdatingCompletionList diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java index 1cf5c3a..4854e8a 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java @@ -51,11 +51,10 @@ import org.junit.Before; import org.junit.Test; import org.scijava.Context; -import org.scijava.code.api.CodeCompletionService; +import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; -import org.scijava.code.lsp.LanguageServerService; -import org.scijava.code.lsp.LspClient; +import org.scijava.code.lsp.Positions; import org.scijava.plugin.PluginService; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; @@ -83,7 +82,7 @@ public class ApposePythonServersTest { @Before public void setUp() { - context = new Context(ScriptService.class, CodeCompletionService.class, + context = new Context(ScriptService.class, LanguageServerService.class, ResidentWorkerService.class, LanguageServerService.class); language = context.service(ScriptService.class).getLanguageByName( "appose-python"); @@ -293,13 +292,13 @@ public void testServerOptions() { "didChange=true, didSave=false}"; assertEquals("{workspace={environmentPath=/env/bin/python, " + "extraPaths=[/overlay]}, " + diagnostics + "}", String.valueOf(spec - .initializationOptions(new org.scijava.code.api.EnvironmentContext( - "/env/bin/python", Collections.singletonList("/overlay"), null)))); + .initializationOptions(new Environment( + "/env/bin/python", Collections.singletonList("/overlay"), null, null)))); // Without a script environment: the server's own. assertEquals("{workspace={extraPaths=[]}, " + diagnostics + "}", String .valueOf(spec.initializationOptions( - org.scijava.code.api.EnvironmentContext.EMPTY))); - assertTrue(spec.rootUri(org.scijava.code.api.EnvironmentContext.EMPTY) + Environment.NONE))); + assertTrue(spec.rootUri(Environment.NONE) .startsWith("file:")); new File("workspace").delete(); } @@ -309,12 +308,12 @@ public void testServerOptions() { /** Completes with the syntax-only server. */ private List complete(final String code) { final PythonSyntaxLanguageServer server = new PythonSyntaxLanguageServer( - context.service(CodeCompletionService.class)); + context.service(LanguageServerService.class)); server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem( "untitled:/a.py", "appose-python", 1, code))); try { return server.completion(new CompletionParams(new TextDocumentIdentifier( - "untitled:/a.py"), LspClient.position(code, code.length()))).get(10, + "untitled:/a.py"), Positions.position(code, code.length()))).get(10, TimeUnit.SECONDS).getRight().getItems().stream().map( CompletionItem::getLabel).collect(Collectors.toList()); } From a6fa19987a8bb530d16f1981c4a7d835222e8b07 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Wed, 7 Oct 2026 15:56:37 -0500 Subject: [PATCH 09/37] Tell language servers when script environments change A script's environment can become available without the script changing: when a script run builds it, or when packages it declares are installed for completion. ResidentWorkerService now notifies listeners of finished builds, and DependencyOverlays of ready overlays; the jedi plugin passes both on to LanguageServerService.environmentsChanged(), so open scripts move to the servers of where they run now. Tests ask through script sessions. Co-Authored-By: Claude Opus 5.5 --- .../python/ApposePythonEnvironments.java | 26 ++++++ .../appose/python/JediServerPlugin.java | 8 +- .../_internal/ResidentWorkerService.java | 22 +++++ .../python/completion/DependencyOverlays.java | 20 ++++- ...ApposePythonCompletionIntegrationTest.java | 83 +++++++------------ .../python/ApposePythonServersTest.java | 21 ++--- 6 files changed, 110 insertions(+), 70 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java index 3a66417..cb72757 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java @@ -49,6 +49,7 @@ import org.apposed.appose.util.Environments; import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; +import org.scijava.plugins.scripting.appose.python._internal.BuildListener; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; import org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec; @@ -98,8 +99,19 @@ protected boolean removeEldestEntry( private DependencyOverlays overlays; + /** Told when where scripts run may have changed. */ + private volatile Runnable onChange = () -> {}; + private ApposePythonEnvironments(final ResidentWorkerService workers) { this.workers = workers; + // NB: A script run builds its environment: its Python is known then. + if (workers != null) workers.addBuildListener(new BuildListener() { + + @Override + public void buildFinished(final String envName, final Throwable error) { + if (error == null) changed(); + } + }); } /** Gets those of a context (by its resident worker service). */ @@ -127,9 +139,22 @@ org.scijava.code.lsp.Environment environment(final String text, : Collections.singletonList(overlay), null, null); } + /** + * Sets what to do when where scripts run may have changed without them + * changing: when an environment is built, or an overlay is ready. + */ + void onChange(final Runnable onChange) { + this.onChange = onChange == null ? () -> {} : onChange; + } + + private void changed() { + onChange.run(); + } + /** Sets the dependency overlays, e.g. to use fake ones in tests. */ synchronized void setOverlays(final DependencyOverlays overlays) { this.overlays = overlays; + if (overlays != null) overlays.setOnReady(this::changed); } /** The file path of a {@code file:} URI, or null. */ @@ -181,6 +206,7 @@ private synchronized DependencyOverlays overlays() { dir) -> installOverlay(workers, python, dependencies, dir), new File( Environments.apposeEnvsDir(), TOOLING_ENV + "-overlays"), OVERLAY_SETTLE, System::currentTimeMillis); + overlays.setOnReady(this::changed); return overlays; } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java index 2fdbfd0..5dc0a8a 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java @@ -85,9 +85,13 @@ public boolean supports(final ScriptLanguage language) { @Override public Environment environment(final TextDocumentItem document) { + final ApposePythonEnvironments environments = ApposePythonEnvironments.of( + workers); + // NB: Scripts move as their environments are built, or overlays ready. + environments.onChange(servers::environmentsChanged); // NB: On opening the script, its dependencies are settled. - return ApposePythonEnvironments.of(workers).environment(document.getText(), - document.getUri(), document.getVersion() <= 1); + return environments.environment(document.getText(), document.getUri(), + document.getVersion() <= 1); } @Override diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java index d3572ca..f1134b9 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java @@ -33,6 +33,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.concurrent.CopyOnWriteArrayList; import java.util.function.Function; import java.util.function.Supplier; @@ -75,6 +76,10 @@ public class ResidentWorkerService extends AbstractService implements /** Environments by name, each with its configuration and worker. */ private final Map entries = new HashMap<>(); + /** Told about every environment's builds. */ + private final List buildListeners = + new CopyOnWriteArrayList<>(); + /** * Gets the resident worker for the named environment, creating the * environment and worker if needed. @@ -102,6 +107,15 @@ public ResidentWorker worker(final String name, final String config, final LazyEnvironment env = new LazyEnvironment(name, builder.get()); env.addBuildListener(SciJavaTasks.buildListener(taskService, statusService, log)); + env.addBuildListener(new BuildListener() { + + @Override + public void buildFinished(final String envName, final Throwable error) { + for (final BuildListener l : buildListeners) { + l.buildFinished(envName, error); + } + } + }); worker = new ResidentWorker(env, launcher); entries.put(name, new Entry(config, worker)); } @@ -109,6 +123,14 @@ public ResidentWorker worker(final String name, final String config, return worker; } + /** + * Adds a listener told when any environment's build ends (e.g. so that + * code completion can use it). + */ + public void addBuildListener(final BuildListener listener) { + buildListeners.add(listener); + } + /** * Gets the resident worker for the named environment, if there is one with * the given configuration. Unlike {@link #worker}, this never creates, diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java index b5c6ecc..52b803d 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java @@ -106,6 +106,9 @@ public interface Installer { /** The dependencies last seen for each environment, and since when. */ private final Map seen = new HashMap<>(); + /** Told when an overlay is ready. */ + private volatile Runnable onReady = () -> {}; + /** * @param installer Does the installing, on a background thread. * @param root Directory holding the overlays (one per environment). @@ -122,6 +125,14 @@ public DependencyOverlays(final Installer installer, final File root, this.clock = clock; } + /** + * Sets what to do when an overlay becomes ready (on the installing + * thread), e.g. tell language servers to look again. + */ + public void setOnReady(final Runnable onReady) { + this.onReady = onReady == null ? () -> {} : onReady; + } + /** * Gets the overlay providing the packages the environment lacks, if it is * ready; starts preparing it once the dependencies have settled. Never @@ -154,8 +165,13 @@ public String overlay(final String envName, final String python, } else if (!settled && now - last.since < settleMillis) return null; final File overlay = new File(root, envName); - installs.put(key, CompletableFuture.supplyAsync(() -> installer.install( - python, dependencies, overlay), executor)); + final CompletableFuture started = CompletableFuture.supplyAsync( + () -> installer.install(python, dependencies, overlay), executor); + installs.put(key, started); + // NB: Once done, so that asking again gets the overlay. + started.thenAccept(dir -> { + if (dir != null) onReady.run(); + }); return null; } } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java index fb884f7..72ad1da 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -46,25 +46,16 @@ import org.eclipse.lsp4j.CompletionItem; import org.eclipse.lsp4j.CompletionList; -import org.eclipse.lsp4j.CompletionParams; import org.eclipse.lsp4j.Diagnostic; -import org.eclipse.lsp4j.DidChangeTextDocumentParams; -import org.eclipse.lsp4j.DidOpenTextDocumentParams; import org.eclipse.lsp4j.Hover; -import org.eclipse.lsp4j.HoverParams; import org.eclipse.lsp4j.SignatureHelp; -import org.eclipse.lsp4j.SignatureHelpParams; -import org.eclipse.lsp4j.TextDocumentContentChangeEvent; -import org.eclipse.lsp4j.TextDocumentIdentifier; -import org.eclipse.lsp4j.TextDocumentItem; -import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import org.scijava.Context; import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.Positions; -import org.scijava.code.lsp.MergedLanguageServer; +import org.scijava.code.lsp.ScriptSession; import org.scijava.code.lsp.UpdatingCompletionList; import org.scijava.module.ModuleService; import org.scijava.script.ScriptInfo; @@ -95,7 +86,7 @@ public class ApposePythonCompletionIntegrationTest { private static Context context; private static ScriptLanguage language; - private static MergedLanguageServer server; + private static LanguageServerService servers; private static File scriptDir; @BeforeClass @@ -103,13 +94,14 @@ public static void setUp() throws URISyntaxException { context = new Context(); language = context.service(ScriptService.class).getLanguageByName( "appose-python"); - server = context.service(LanguageServerService.class).server(language); + servers = context.service(LanguageServerService.class); scriptDir = new File(ApposePythonCompletionIntegrationTest.class .getResource("/appose-test/requirements.txt").toURI()).getParentFile(); } @AfterClass public static void tearDown() { + sessions.values().forEach(ScriptSession::close); if (context != null) context.dispose(); } @@ -123,8 +115,8 @@ public void testStandardLibrary() throws Exception { .equals("dumps") && c.getLabelDetails() != null).findFirst().get(); assertTrue(dumps.getLabelDetails().getDetail(), dumps.getLabelDetails() .getDetail().startsWith("(Any obj, bool skipkeys")); - assertTrue(dumps.getDocumentation() != null || server - .resolveCompletionItem(dumps).get(1, TimeUnit.MINUTES) + assertTrue(dumps.getDocumentation() != null || session(null).resolve( + dumps).get(1, TimeUnit.MINUTES) .getDocumentation() != null); report("import json\njson.l", null, "loads", "o"); } @@ -153,13 +145,12 @@ public void testScriptEnvironment() throws Exception { @Test public void testSignatureHelp() throws Exception { final String code = "import json\njson.dumps(obj, "; - final String uri = sync(code, null); + final ScriptSession session = session(null); final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; SignatureHelp help = null; while (help == null && System.currentTimeMillis() < deadline) { - help = server.signatureHelp(new SignatureHelpParams( - new TextDocumentIdentifier(uri), Positions.position(code, code - .length()))).get(1, TimeUnit.MINUTES); + help = session.signatureHelp(code, code.length()).get(1, + TimeUnit.MINUTES); } assertTrue(help.getSignatures().get(0).getLabel().contains("dumps(")); assertFalse(help.getSignatures().get(0).getParameters().isEmpty()); @@ -169,13 +160,12 @@ public void testSignatureHelp() throws Exception { @Test public void testHover() throws Exception { final String code = "import json\njson.dumps(1)"; - final String uri = sync(code, null); + final ScriptSession session = session(null); final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; Hover hover = null; while (hover == null && System.currentTimeMillis() < deadline) { - hover = server.hover(new HoverParams(new TextDocumentIdentifier(uri), - Positions.position(code, code.indexOf("dumps") + 2))).get(1, - TimeUnit.MINUTES); + hover = session.hover(code, code.indexOf("dumps") + 2).get(1, + TimeUnit.MINUTES); } assertTrue(String.valueOf(hover), hover.getContents().getRight().getValue() .contains("JSON")); @@ -190,19 +180,17 @@ public void testHover() throws Exception { public void testDiagnostics() throws Exception { // A syntax error, on the script's third line. final String code = "#@ double sigma\nx = sigma\ny = (1 +\n"; - final LanguageServerService servers = context.service( - LanguageServerService.class); final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; List problems = Collections.emptyList(); while (problems.isEmpty() && System.currentTimeMillis() < deadline) { - problems = servers.diagnose(language, "untitled:/diagnose.py", code, + problems = servers.diagnose(language, code, 120_000); } assertFalse(problems.isEmpty()); assertTrue(problems.toString(), problems.get(0).getRange().getStart() .getLine() >= 2); // Fixed: no more problems. - assertTrue(servers.diagnose(language, "untitled:/diagnose.py", code + assertTrue(servers.diagnose(language, code .replace("(1 +", "(1 + 2)"), 120_000).isEmpty()); } @@ -249,26 +237,18 @@ public void testNewDependency() throws Exception { // -- Helper methods -- - /** The open documents' versions, by URI. */ - private static final Map versions = new HashMap<>(); + /** The scripts' sessions, by path ("" for an untitled one). */ + private static final Map sessions = new HashMap<>(); - /** Opens a script on the server, or changes it; returns its URI. */ - private static String sync(final String code, final String path) { - final String uri = path == null ? "untitled:/integration.py" : new File( - path).toURI().toString(); - final Integer version = versions.get(uri); - if (version == null) { - versions.put(uri, 1); - server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(uri, - "appose-python", 1, code))); - } - else { - versions.put(uri, version + 1); - server.didChange(new DidChangeTextDocumentParams( - new VersionedTextDocumentIdentifier(uri, version + 1), Collections - .singletonList(new TextDocumentContentChangeEvent(code)))); - } - return uri; + /** The session about a script (null path: untitled). */ + private static ScriptSession session(final String path) { + return sessions.computeIfAbsent(path == null ? "" : path, p -> servers + .session(language).setFile(path == null ? null : new File(path))); + } + + /** Tells the servers a script's text. */ + private static ScriptSession sync(final String code, final String path) { + return session(path).update(code); } /** Completes until the expected completion appears (environments build). */ @@ -295,10 +275,8 @@ private static List completeUntil(final String code, private static List complete(final String code, final String path, final boolean settled) throws Exception { - final String uri = sync(code, path); - CompletionList list = server.completion(new CompletionParams( - new TextDocumentIdentifier(uri), Positions.position(code, code.length()))) - .get(1, TimeUnit.MINUTES).getRight(); + CompletionList list = session(path).completion(code, code.length()) + .get(1, TimeUnit.MINUTES); if (settled && list instanceof UpdatingCompletionList) { final CompletionList better = ((UpdatingCompletionList) list).update() .get(1, TimeUnit.MINUTES); @@ -321,11 +299,10 @@ private static void report(final String code, final String path, final int n = complete(code + more, path, true).size(); final long cached = System.nanoTime() - t0; final String other = code.substring(0, code.lastIndexOf('.') + 1) + "s"; - final String uri = sync(other, path); + final ScriptSession session = session(path); t0 = System.nanoTime(); - final CompletionList first = server.completion(new CompletionParams( - new TextDocumentIdentifier(uri), Positions.position(other, other - .length()))).get(1, TimeUnit.MINUTES).getRight(); + final CompletionList first = session.completion(other, other.length()) + .get(1, TimeUnit.MINUTES); final long wait = System.nanoTime() - t0; final CompletionList update = first instanceof UpdatingCompletionList ? ((UpdatingCompletionList) first).update().get(1, TimeUnit.MINUTES) diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java index 4854e8a..1c1d623 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java @@ -43,9 +43,6 @@ import net.imglib2.img.Img; import org.eclipse.lsp4j.CompletionItem; -import org.eclipse.lsp4j.CompletionParams; -import org.eclipse.lsp4j.DidOpenTextDocumentParams; -import org.eclipse.lsp4j.TextDocumentIdentifier; import org.eclipse.lsp4j.TextDocumentItem; import org.junit.After; import org.junit.Before; @@ -54,7 +51,7 @@ import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; -import org.scijava.code.lsp.Positions; +import org.scijava.code.lsp.ScriptSession; import org.scijava.plugin.PluginService; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; @@ -307,15 +304,13 @@ public void testServerOptions() { /** Completes with the syntax-only server. */ private List complete(final String code) { - final PythonSyntaxLanguageServer server = new PythonSyntaxLanguageServer( - context.service(LanguageServerService.class)); - server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem( - "untitled:/a.py", "appose-python", 1, code))); - try { - return server.completion(new CompletionParams(new TextDocumentIdentifier( - "untitled:/a.py"), Positions.position(code, code.length()))).get(10, - TimeUnit.SECONDS).getRight().getItems().stream().map( - CompletionItem::getLabel).collect(Collectors.toList()); + try (ScriptSession session = ScriptSession.owning( + new PythonSyntaxLanguageServer(context.service( + LanguageServerService.class)), language)) + { + return session.completion(code, code.length()).get(10, + TimeUnit.SECONDS).getItems().stream().map(CompletionItem::getLabel) + .collect(Collectors.toList()); } catch (final Exception exc) { throw new RuntimeException(exc); From 4142f8feacacd9d80bafb6393429e9542ff2f861 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Thu, 8 Oct 2026 13:10:13 -0500 Subject: [PATCH 10/37] Take over the Python completion front from code-lsp PythonCompletionServer (with DelegatingLanguageServer) completes Java members on objects jedi cannot see into. It is specific to Python, so it lives here now, with the one plugin using it. Co-Authored-By: Claude Opus 5.5 --- .../appose/python/JediServerPlugin.java | 2 +- .../completion/DelegatingLanguageServer.java | 168 ++++++++ .../completion/PythonCompletionServer.java | 368 ++++++++++++++++++ .../PythonCompletionServerTest.java | 294 ++++++++++++++ 4 files changed, 831 insertions(+), 1 deletion(-) create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/DelegatingLanguageServer.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java index 5dc0a8a..0cd9641 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java @@ -39,7 +39,7 @@ import org.scijava.code.lsp.ScriptDocument; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; -import org.scijava.code.lsp.PythonCompletionServer; +import org.scijava.plugins.scripting.appose.python.completion.PythonCompletionServer; import org.scijava.code.lsp.TransformingLanguageServer; import org.scijava.plugin.Parameter; import org.scijava.plugin.Plugin; diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DelegatingLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DelegatingLanguageServer.java new file mode 100644 index 0000000..4fae53d --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DelegatingLanguageServer.java @@ -0,0 +1,168 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import java.util.List; +import java.util.concurrent.CompletableFuture; + +import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionList; +import org.eclipse.lsp4j.CompletionParams; +import org.eclipse.lsp4j.DidChangeConfigurationParams; +import org.eclipse.lsp4j.DidChangeTextDocumentParams; +import org.eclipse.lsp4j.DidChangeWatchedFilesParams; +import org.eclipse.lsp4j.DidCloseTextDocumentParams; +import org.eclipse.lsp4j.DidOpenTextDocumentParams; +import org.eclipse.lsp4j.DidSaveTextDocumentParams; +import org.eclipse.lsp4j.Hover; +import org.eclipse.lsp4j.HoverParams; +import org.eclipse.lsp4j.InitializeParams; +import org.eclipse.lsp4j.InitializeResult; +import org.eclipse.lsp4j.InitializedParams; +import org.eclipse.lsp4j.SignatureHelp; +import org.eclipse.lsp4j.SignatureHelpParams; +import org.eclipse.lsp4j.jsonrpc.messages.Either; +import org.eclipse.lsp4j.services.LanguageClient; +import org.eclipse.lsp4j.services.LanguageClientAware; +import org.eclipse.lsp4j.services.LanguageServer; +import org.eclipse.lsp4j.services.TextDocumentService; +import org.eclipse.lsp4j.services.WorkspaceService; + +/** + * A language server passing everything on to another: a base for decorators + * changing some of its answers. + * + * @author Gabriel Selzer + */ +public class DelegatingLanguageServer implements LanguageServer, + TextDocumentService, WorkspaceService, LanguageClientAware +{ + + protected final LanguageServer inner; + + public DelegatingLanguageServer(final LanguageServer inner) { + this.inner = inner; + } + + @Override + public void connect(final LanguageClient client) { + if (inner instanceof LanguageClientAware) { + ((LanguageClientAware) inner).connect(client); + } + } + + @Override + public CompletableFuture initialize( + final InitializeParams params) + { + return inner.initialize(params); + } + + @Override + public void initialized(final InitializedParams params) { + inner.initialized(params); + } + + @Override + public CompletableFuture shutdown() { + return inner.shutdown(); + } + + @Override + public void exit() { + inner.exit(); + } + + @Override + public TextDocumentService getTextDocumentService() { + return this; + } + + @Override + public WorkspaceService getWorkspaceService() { + return this; + } + + @Override + public void didOpen(final DidOpenTextDocumentParams params) { + inner.getTextDocumentService().didOpen(params); + } + + @Override + public void didChange(final DidChangeTextDocumentParams params) { + inner.getTextDocumentService().didChange(params); + } + + @Override + public void didClose(final DidCloseTextDocumentParams params) { + inner.getTextDocumentService().didClose(params); + } + + @Override + public void didSave(final DidSaveTextDocumentParams params) { + inner.getTextDocumentService().didSave(params); + } + + @Override + public CompletableFuture, CompletionList>> + completion(final CompletionParams params) + { + return inner.getTextDocumentService().completion(params); + } + + @Override + public CompletableFuture resolveCompletionItem( + final CompletionItem item) + { + return inner.getTextDocumentService().resolveCompletionItem(item); + } + + @Override + public CompletableFuture signatureHelp( + final SignatureHelpParams params) + { + return inner.getTextDocumentService().signatureHelp(params); + } + + @Override + public CompletableFuture hover(final HoverParams params) { + return inner.getTextDocumentService().hover(params); + } + + @Override + public void didChangeConfiguration(final DidChangeConfigurationParams p) { + inner.getWorkspaceService().didChangeConfiguration(p); + } + + @Override + public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) { + inner.getWorkspaceService().didChangeWatchedFiles(p); + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java new file mode 100644 index 0000000..b60032b --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java @@ -0,0 +1,368 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import java.util.ArrayList; +import java.util.Comparator; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.function.LongSupplier; +import java.util.regex.Pattern; + +import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionItemKind; +import org.eclipse.lsp4j.CompletionList; +import org.eclipse.lsp4j.CompletionParams; +import org.eclipse.lsp4j.DidChangeTextDocumentParams; +import org.eclipse.lsp4j.DidCloseTextDocumentParams; +import org.eclipse.lsp4j.DidOpenTextDocumentParams; +import org.eclipse.lsp4j.jsonrpc.messages.Either; +import org.eclipse.lsp4j.services.LanguageServer; +import org.scijava.code.lsp.Callable; +import org.scijava.code.lsp.Positions; + +/** + * Makes a Python language server's completions (e.g. jedi-language-server's) + * work well in an editor completing as the user types: + *
      + *
    • Asks about a word's first character only, and reuses the answer + * (filtered) as the word is typed on, for a while.
      • + *
    • Cancels a question still pending once another is asked about the same + * document.
      • + *
    • Once a word is begun, resolves the first completions, and makes those + * with signatures (e.g. {@code def dumps(obj, *, skipkeys=False) -> str}) + * {@link Callable}s: snippets with their parameters and types. (Not in + * imports: an import names things, rather than calling them.)
      • + *
    + * + * @author Gabriel Selzer + */ +public class PythonCompletionServer extends DelegatingLanguageServer { + + /** How long an answer may be reused, in milliseconds. */ + private static final long ANSWER_LIFETIME = 20_000; + + /** An import statement being typed. */ + private static final Pattern IMPORT = Pattern.compile( + "^\\s*(from|import)\\s"); + + private final int signatures; + private final LongSupplier clock; + + /** The open documents' texts, by URI. */ + private final Map texts = new ConcurrentHashMap<>(); + + /** Answers, by question; least recently used first. */ + private final Map answers = new LinkedHashMap( + 16, 0.75f, true) + { + + @Override + protected boolean removeEldestEntry(final Map.Entry e) { + return size() > 32; + } + }; + + /** The latest question about each document, by URI. */ + private final Map pending = new ConcurrentHashMap<>(); + + /** + * @param inner The Python language server. + * @param signatures How many completions to resolve for their signatures, + * once a word is begun. + */ + public PythonCompletionServer(final LanguageServer inner, + final int signatures) + { + this(inner, signatures, System::currentTimeMillis); + } + + PythonCompletionServer(final LanguageServer inner, final int signatures, + final LongSupplier clock) + { + super(inner); + this.signatures = signatures; + this.clock = clock; + } + + @Override + public void didOpen(final DidOpenTextDocumentParams params) { + texts.put(params.getTextDocument().getUri(), params.getTextDocument() + .getText()); + super.didOpen(params); + } + + @Override + public void didChange(final DidChangeTextDocumentParams params) { + if (!params.getContentChanges().isEmpty()) { + texts.put(params.getTextDocument().getUri(), params.getContentChanges() + .get(params.getContentChanges().size() - 1).getText()); + } + super.didChange(params); + } + + @Override + public void didClose(final DidCloseTextDocumentParams params) { + texts.remove(params.getTextDocument().getUri()); + pending.remove(params.getTextDocument().getUri()); + super.didClose(params); + } + + @Override + public CompletableFuture, CompletionList>> + completion(final CompletionParams params) + { + final String uri = params.getTextDocument().getUri(); + final String text = texts.get(uri); + if (text == null) return super.completion(params); + final int caret = Math.min(Positions.offset(text, params.getPosition()), + text.length()); + int start = caret; + while (start > 0 && Character.isJavaIdentifierPart(text.charAt(start - + 1))) start--; + final String word = text.substring(start, caret); + // Ask about the first character typed, if any: later keystrokes of the + // same word then reuse the answer (filtered). + final int ask = start + Math.min(1, word.length()); + final String line = text.substring(text.lastIndexOf('\n', start - 1) + 1, + start); + final boolean importing = IMPORT.matcher(line).find(); + final String key = uri + "\n" + text.substring(0, ask); + final long now = clock.getAsLong(); + Answer answer; + synchronized (answers) { + answer = answers.get(key); + if (answer != null && (now - answer.time > ANSWER_LIFETIME || + answer.future.isCompletedExceptionally())) + { + answer = null; + } + } + if (answer == null) { + final CompletionParams p = new CompletionParams(params.getTextDocument(), + Positions.position(text, ask), params.getContext()); + final CompletableFuture, CompletionList>> raw = + super.completion(p); + final int n = word.isEmpty() || importing ? 0 : signatures; + answer = new Answer(raw, raw.thenCompose(list -> resolve(list, n)), now); + synchronized (answers) { + answers.put(key, answer); + } + final Answer previous = pending.put(uri, answer); + if (previous != null && previous != answer && !previous.future + .isDone()) + { + // NB: Cancelling the request tells the server ($/cancelRequest). + previous.future.cancel(true); + previous.raw.cancel(true); + } + } + return answer.future.thenApply(list -> Either.forRight(filter(list, + word))); + } + + /** Resolves the first completions, making those with signatures callables. */ + private CompletableFuture resolve( + final Either, CompletionList> answer, final int n) + { + final List items = new ArrayList<>(answer == null + ? new ArrayList<>() : answer.isRight() ? answer.getRight().getItems() + : answer.getLeft()); + items.sort(Comparator.comparing(i -> i.getSortText() != null ? i + .getSortText() : i.getLabel())); + final int count = Math.min(n, items.size()); + final List> resolved = new ArrayList<>(); + for (int i = 0; i < count; i++) { + // NB: Unresolved, should resolving fail. + resolved.add(inner.getTextDocumentService().resolveCompletionItem(items + .get(i)).handle((r, error) -> r)); + } + return CompletableFuture.allOf(resolved.toArray(new CompletableFuture[0])) + .thenApply(x -> { + for (int i = 0; i < count; i++) { + final CompletionItem r = resolved.get(i).join(); + final CompletionItem callable = r == null ? null : callable(items + .get(i), r); + if (callable != null) items.set(i, callable); + } + return new CompletionList(false, items); + }); + } + + /** + * A completion as a callable, from its resolved signature (its detail); + * or null if it has none. + */ + private static CompletionItem callable(final CompletionItem item, + final CompletionItem resolved) + { + final CompletionItemKind kind = item.getKind(); + if (kind != CompletionItemKind.Function && kind != CompletionItemKind.Method && + kind != CompletionItemKind.Class && kind != CompletionItemKind.Constructor) + { + return null; + } + final String name = item.getInsertText() != null ? item.getInsertText() + : item.getLabel(); + if (resolved.getDetail() == null) return null; + final Callable parsed = parse(name, resolved.getDetail()); + if (parsed == null) return null; + final CompletionItem out = new Callable(name, parsed.params(), parsed + .returnType()).kind(CompletionItemKind.Method).detail(resolved + .getDetail()).toItem(); + out.setSortText(item.getSortText()); + out.setDocumentation(resolved.getDocumentation()); + out.setData(item.getData()); + return out; + } + + /** The completions of the word typed so far. */ + private static CompletionList filter(final CompletionList list, + final String word) + { + final String w = word.toLowerCase(Locale.ROOT); + final List out = new ArrayList<>(); + for (final CompletionItem item : list.getItems()) { + final String name = item.getFilterText() != null ? item.getFilterText() + : item.getLabel(); + if (name.startsWith("_") && !word.startsWith("_")) continue; + if (!name.toLowerCase(Locale.ROOT).startsWith(w) || name.equals(word)) { + continue; + } + out.add(item); + } + return new CompletionList(false, out); + } + + private static final class Answer { + + private final CompletableFuture raw; + private final CompletableFuture future; + private final long time; + + private Answer(final CompletableFuture raw, + final CompletableFuture future, final long time) + { + this.raw = raw; + this.future = future; + this.time = time; + } + } + + /** + * Parses a Python signature (as Jedi writes them): the parameters, with + * their annotations as types; and the return annotation. Returns null if it + * is not a signature. + */ + static Callable parse(final String name, + final String signature) + { + final int open = signature.indexOf('('); + if (open < 0) return null; + final int close = closing(signature, open); + if (close < 0) return null; + final List params = new ArrayList<>(); + for (final String part : split(signature.substring(open + 1, close))) { + final String p = part.trim(); + // NB: Bare * and / only mark keyword-only and positional-only ones. + if (p.isEmpty() || p.equals("*") || p.equals("/")) continue; + final int colon = topLevel(p, ':'), eq = topLevel(p, '='); + String paramName = p, type = null; + if (colon >= 0 && (eq < 0 || colon < eq)) { + paramName = p.substring(0, colon); + type = p.substring(colon + 1, eq < 0 ? p.length() : eq).trim(); + } + else if (eq >= 0) paramName = p.substring(0, eq); + paramName = paramName.trim().replaceFirst("^\\*{1,2}", ""); + params.add(new Callable.Param(paramName, type == null || type + .isEmpty() ? null : type)); + } + final String rest = signature.substring(close + 1).trim(); + final String returns = rest.startsWith("->") ? rest.substring(2).trim() + : null; + return new Callable(name, params, returns == null || returns.isEmpty() + ? null : returns); + } + + /** The index of the bracket closing the one at {@code open}, or -1. */ + private static int closing(final String s, final int open) { + int depth = 0; + char quote = 0; + for (int i = open; i < s.length(); i++) { + final char c = s.charAt(i); + if (quote != 0) { + if (c == '\\') i++; + else if (c == quote) quote = 0; + } + else if (c == '\'' || c == '"') quote = c; + else if (c == '(' || c == '[' || c == '{') depth++; + else if (c == ')' || c == ']' || c == '}') { + if (--depth == 0) return i; + } + } + return -1; + } + + /** Splits at the commas outside brackets and strings. */ + private static List split(final String s) { + final List parts = new ArrayList<>(); + int start = 0; + while (true) { + final int comma = topLevel(s.substring(start), ','); + if (comma < 0) break; + parts.add(s.substring(start, start + comma)); + start += comma + 1; + } + parts.add(s.substring(start)); + return parts; + } + + /** The index of the first {@code target} outside brackets and strings. */ + private static int topLevel(final String s, final char target) { + int depth = 0; + char quote = 0; + for (int i = 0; i < s.length(); i++) { + final char c = s.charAt(i); + if (quote != 0) { + if (c == '\\') i++; + else if (c == quote) quote = 0; + } + else if (c == '\'' || c == '"') quote = c; + else if (c == '(' || c == '[' || c == '{') depth++; + else if (c == ')' || c == ']' || c == '}') depth--; + else if (c == target && depth == 0) return i; + } + return -1; + } +} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java new file mode 100644 index 0000000..3758bbb --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java @@ -0,0 +1,294 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertThrows; +import static org.junit.Assert.assertTrue; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; + +import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionItemKind; +import org.eclipse.lsp4j.CompletionList; +import org.eclipse.lsp4j.CompletionParams; +import org.eclipse.lsp4j.DidChangeConfigurationParams; +import org.eclipse.lsp4j.DidChangeTextDocumentParams; +import org.eclipse.lsp4j.DidChangeWatchedFilesParams; +import org.eclipse.lsp4j.DidCloseTextDocumentParams; +import org.eclipse.lsp4j.DidOpenTextDocumentParams; +import org.eclipse.lsp4j.DidSaveTextDocumentParams; +import org.eclipse.lsp4j.InitializeParams; +import org.eclipse.lsp4j.InitializeResult; +import org.eclipse.lsp4j.Position; +import org.eclipse.lsp4j.ServerCapabilities; +import org.eclipse.lsp4j.TextDocumentContentChangeEvent; +import org.eclipse.lsp4j.TextDocumentIdentifier; +import org.eclipse.lsp4j.TextDocumentItem; +import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; +import org.eclipse.lsp4j.jsonrpc.messages.Either; +import org.eclipse.lsp4j.services.LanguageServer; +import org.eclipse.lsp4j.services.TextDocumentService; +import org.eclipse.lsp4j.services.WorkspaceService; +import org.junit.Test; +import org.scijava.code.lsp.Callable; +import org.scijava.code.lsp.Positions; + +/** + * Tests {@link PythonCompletionServer}, around a fake Python server. + * + * @author Gabriel Selzer + */ +public class PythonCompletionServerTest { + + private static final String URI = "untitled:/a.py"; + + /** Knows json's dumps (a function), dump, decoder (a module), __doc__. */ + private static final class Jedi implements LanguageServer, + TextDocumentService, WorkspaceService + { + + final List asked = new CopyOnWriteArrayList<>(); + final List resolved = new CopyOnWriteArrayList<>(); + CompletableFuture hold; + final List> held = new CopyOnWriteArrayList<>(); + + @Override + public CompletableFuture initialize( + final InitializeParams params) + { + return CompletableFuture.completedFuture(new InitializeResult( + new ServerCapabilities())); + } + + @Override + public CompletableFuture shutdown() { + return CompletableFuture.completedFuture(null); + } + + @Override + public void exit() {} + + @Override + public TextDocumentService getTextDocumentService() { + return this; + } + + @Override + public WorkspaceService getWorkspaceService() { + return this; + } + + @Override + public void didOpen(final DidOpenTextDocumentParams p) {} + + @Override + public void didChange(final DidChangeTextDocumentParams p) {} + + @Override + public void didClose(final DidCloseTextDocumentParams p) {} + + @Override + public void didSave(final DidSaveTextDocumentParams p) {} + + @Override + public void didChangeConfiguration(final DidChangeConfigurationParams p) {} + + @Override + public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) {} + + @Override + public CompletableFuture, CompletionList>> + completion(final CompletionParams p) + { + asked.add(p.getPosition()); + final List items = new ArrayList<>(); + int i = 0; + for (final String name : new String[] { "dumps", "dump", "decoder", + "__doc__" }) + { + final CompletionItem item = new CompletionItem(name); + item.setKind(name.equals("decoder") ? CompletionItemKind.Module + : CompletionItemKind.Function); + item.setInsertText(name); + item.setSortText("z" + i++); + items.add(item); + } + final Either, CompletionList> answer = Either + .forRight(new CompletionList(items)); + if (hold == null) return CompletableFuture.completedFuture(answer); + final CompletableFuture, CompletionList>> f = + new CompletableFuture<>(); + held.add(f); + hold.thenRun(() -> f.complete(answer)); + return f; + } + + @Override + public CompletableFuture resolveCompletionItem( + final CompletionItem item) + { + resolved.add(item.getLabel()); + item.setDetail(item.getLabel().equals("decoder") ? "module decoder" + : "def " + item.getLabel() + "(obj, *, indent: int=None) -> str"); + return CompletableFuture.completedFuture(item); + } + } + + @Test + public void testAskOnceAndFilter() throws Exception { + final Jedi jedi = new Jedi(); + final PythonCompletionServer server = new PythonCompletionServer(jedi, 30); + open(server, "import json\njson.du"); + List items = complete(server, "import json\njson.du"); + // Asked about the word's first character. + assertEquals(new Position(1, 6), jedi.asked.get(0)); + // The word's completions, without dunders. + assertEquals(Arrays.asList("dumps", "dump"), labels(items)); + // Once a word is begun: callables, from the resolved signatures. + final CompletionItem dumps = items.get(0); + assertEquals("(obj, int indent)", dumps.getLabelDetails().getDetail()); + assertEquals("str", dumps.getLabelDetails().getDescription()); + assertEquals("dumps(${1:obj}, ${2:indent})", dumps.getInsertText()); + + // Typing on: the same answer, filtered; not asked again. + change(server, "import json\njson.dum", 2); + items = complete(server, "import json\njson.dum"); + assertEquals(Arrays.asList("dumps", "dump"), labels(items)); + // (The word typed in full is not offered again.) + change(server, "import json\njson.dumps", 3); + assertTrue(complete(server, "import json\njson.dumps").isEmpty()); + assertEquals(1, jedi.asked.size()); + + // Nothing typed: names only (nothing resolved), dunders left out. + change(server, "import json\njson.", 4); + final int resolved = jedi.resolved.size(); + items = complete(server, "import json\njson."); + assertEquals(Arrays.asList("dumps", "dump", "decoder"), labels(items)); + assertNull(items.get(0).getLabelDetails()); + assertEquals(resolved, jedi.resolved.size()); + } + + @Test + public void testImportsAreNames() throws Exception { + final Jedi jedi = new Jedi(); + final PythonCompletionServer server = new PythonCompletionServer(jedi, 30); + open(server, "from json import du"); + final List items = complete(server, "from json import du"); + assertEquals(Arrays.asList("dumps", "dump"), labels(items)); + assertNull(items.get(0).getLabelDetails()); + } + + @Test + public void testSuperseded() throws Exception { + final Jedi jedi = new Jedi(); + jedi.hold = new CompletableFuture<>(); + final PythonCompletionServer server = new PythonCompletionServer(jedi, 0); + open(server, "x.a"); + final CompletableFuture, CompletionList>> slow = + server.completion(params("x.a")); + change(server, "x.b", 2); + final CompletableFuture, CompletionList>> next = + server.completion(params("x.b")); + // The first question is cancelled, also on the server. + assertThrows(Exception.class, () -> slow.get(10, TimeUnit.SECONDS)); + assertTrue(jedi.held.get(0).isCancelled()); + jedi.hold.complete(null); + assertTrue(next.get(10, TimeUnit.SECONDS).getRight().getItems().isEmpty()); + } + + // -- Helper methods -- + + private static CompletionParams params(final String text) { + return new CompletionParams(new TextDocumentIdentifier(URI), Positions + .position(text, text.length())); + } + + private static List complete( + final PythonCompletionServer server, final String text) throws Exception + { + return server.completion(params(text)).get(10, TimeUnit.SECONDS) + .getRight().getItems(); + } + + private static void open(final PythonCompletionServer server, + final String text) + { + server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(URI, + "python", 1, text))); + } + + private static void change(final PythonCompletionServer server, + final String text, final int version) + { + server.didChange(new DidChangeTextDocumentParams( + new VersionedTextDocumentIdentifier(URI, version), Collections + .singletonList(new TextDocumentContentChangeEvent(text)))); + } + + private static List labels(final List items) { + return items.stream().map(CompletionItem::getLabel).collect(Collectors + .toList()); + } + + @Test + public void testParse() { + Callable c = PythonCompletionServer.parse("dumps", + "def dumps(obj, *, skipkeys: bool=False, cls=None, **kw) -> str"); + assertEquals("[obj, bool skipkeys, cls, kw]", String.valueOf(c + .params())); + assertEquals("str", c.returnType()); + + // Brackets and strings in annotations and defaults; positional-only. + c = PythonCompletionServer.parse("zeros", "def zeros(shape: tuple[int, ...], dtype: str=\"f,8\"" + + ", /, order: Literal['C', 'F']='C') -> NDArray[float64]"); + assertEquals("[tuple[int, ...] shape, str dtype, " + + "Literal['C', 'F'] order]", String.valueOf(c.params())); + assertEquals("NDArray[float64]", c.returnType()); + + // Classes (their constructors), without parameters or return type. + c = PythonCompletionServer.parse("Path", "class Path()"); + assertEquals(0, c.params().size()); + assertNull(c.returnType()); + c = PythonCompletionServer.parse("f", "def f(*args)"); + assertEquals("args", c.params().get(0).name()); + + // Not a signature. + assertNull(PythonCompletionServer.parse("x", "instance x")); + } +} From 4b4e8bcf3b874eec4e1efe180383d842e119d875 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Thu, 8 Oct 2026 13:10:14 -0500 Subject: [PATCH 11/37] Follow code-lsp's packages by purpose Co-Authored-By: Claude Opus 5.5 --- .../appose/python/ApposePythonLspServerPlugin.java | 4 ++-- .../plugins/scripting/appose/python/JediServerPlugin.java | 8 ++++---- .../appose/python/PythonSyntaxLanguageServer.java | 2 +- .../appose/python/completion/ApposePythonDialect.java | 2 +- .../appose/python/completion/JediLanguageServerSpec.java | 4 ++-- .../appose/python/completion/PythonCompletionServer.java | 2 +- .../python/completion/PythonCompletionServerTest.java | 2 +- 7 files changed, 12 insertions(+), 12 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java index 43769bf..2ded0ec 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java @@ -29,8 +29,8 @@ package org.scijava.plugins.scripting.appose.python; -import org.scijava.code.lsp.LspServerPlugin; -import org.scijava.code.lsp.LspServerSpec; +import org.scijava.code.lsp.launch.LspServerPlugin; +import org.scijava.code.lsp.launch.LspServerSpec; import org.scijava.plugin.Parameter; import org.scijava.plugin.Plugin; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java index 0cd9641..c5c1595 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java @@ -35,15 +35,15 @@ import org.eclipse.lsp4j.TextDocumentItem; import org.eclipse.lsp4j.services.LanguageServer; import org.scijava.Priority; -import org.scijava.code.lsp.LanguageServerService; -import org.scijava.code.lsp.ScriptDocument; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; -import org.scijava.plugins.scripting.appose.python.completion.PythonCompletionServer; -import org.scijava.code.lsp.TransformingLanguageServer; +import org.scijava.code.lsp.LanguageServerService; +import org.scijava.code.lsp.script.ScriptDocument; +import org.scijava.code.lsp.script.TransformingLanguageServer; import org.scijava.plugin.Parameter; import org.scijava.plugin.Plugin; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; +import org.scijava.plugins.scripting.appose.python.completion.PythonCompletionServer; import org.scijava.script.ScriptLanguage; /** diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java index 75a0bbc..8d74e1d 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java @@ -40,7 +40,7 @@ import org.eclipse.lsp4j.CompletionItemKind; import org.eclipse.lsp4j.CompletionList; import org.scijava.code.lsp.LanguageServerService; -import org.scijava.code.lsp.ScriptLanguageServer; +import org.scijava.code.lsp.jvm.ScriptLanguageServer; import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax; import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Kind; diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index 65a58ca..fd0c2d5 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -41,7 +41,7 @@ import net.imglib2.RandomAccessibleInterval; import org.apposed.appose.NDArray; -import org.scijava.code.lsp.ScriptDialect; +import org.scijava.code.lsp.script.ScriptDialect; /** * How Python scripts run via Appose see their script parameters: as the diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java index 94f3173..dc1fd44 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java @@ -36,8 +36,8 @@ import java.util.concurrent.Callable; import org.scijava.code.lsp.Environment; -import org.scijava.code.lsp.LspConnection; -import org.scijava.code.lsp.LspServerSpec; +import org.scijava.code.lsp.launch.LspConnection; +import org.scijava.code.lsp.launch.LspServerSpec; /** * Starts jedi-language-server, with the Python of a tooling environment, to diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java index b60032b..0b2604c 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java @@ -49,7 +49,7 @@ import org.eclipse.lsp4j.DidOpenTextDocumentParams; import org.eclipse.lsp4j.jsonrpc.messages.Either; import org.eclipse.lsp4j.services.LanguageServer; -import org.scijava.code.lsp.Callable; +import org.scijava.code.lsp.jvm.Callable; import org.scijava.code.lsp.Positions; /** diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java index 3758bbb..ecfac2a 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java @@ -66,7 +66,7 @@ import org.eclipse.lsp4j.services.TextDocumentService; import org.eclipse.lsp4j.services.WorkspaceService; import org.junit.Test; -import org.scijava.code.lsp.Callable; +import org.scijava.code.lsp.jvm.Callable; import org.scijava.code.lsp.Positions; /** From 42c0286b1722b8c604dec7ead596c41ed3035bd0 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Thu, 8 Oct 2026 13:52:06 -0500 Subject: [PATCH 12/37] Set script files as suppliers in tests ScriptSession.setFile(File) is gone from code-lsp. Co-Authored-By: Claude Opus 5.5 --- .../appose/python/ApposePythonCompletionIntegrationTest.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java index 72ad1da..cfd257e 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -243,7 +243,7 @@ public void testNewDependency() throws Exception { /** The session about a script (null path: untitled). */ private static ScriptSession session(final String path) { return sessions.computeIfAbsent(path == null ? "" : path, p -> servers - .session(language).setFile(path == null ? null : new File(path))); + .session(language).setFile(() -> path == null ? null : new File(path))); } /** Tells the servers a script's text. */ From c1be7b6be9ad0c6b9cc7ef730cbb98888dfc3361 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Thu, 8 Oct 2026 14:27:47 -0500 Subject: [PATCH 13/37] Take completions as one answer in tests code-lsp's merged completions now come once all servers have answered. Co-Authored-By: Claude Opus 5.5 --- ...ApposePythonCompletionIntegrationTest.java | 40 +++++-------------- 1 file changed, 11 insertions(+), 29 deletions(-) diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java index cfd257e..7ecceb4 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -56,7 +56,6 @@ import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.Positions; import org.scijava.code.lsp.ScriptSession; -import org.scijava.code.lsp.UpdatingCompletionList; import org.scijava.module.ModuleService; import org.scijava.script.ScriptInfo; import org.scijava.script.ScriptLanguage; @@ -257,7 +256,7 @@ private static List completeUntil(final String code, { final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; while (System.currentTimeMillis() < deadline) { - final List completions = complete(code, path, true); + final List completions = complete(code, path); if (completions.stream().anyMatch(c -> c.getLabel().equals(expected))) { return completions; } @@ -267,50 +266,33 @@ private static List completeUntil(final String code, return null; } - /** - * Completes at the end of the code. - * - * @param settled Whether to wait for the slower servers' completions. - */ + /** Completes at the end of the code: all servers' completions. */ private static List complete(final String code, - final String path, final boolean settled) throws Exception + final String path) throws Exception { - CompletionList list = session(path).completion(code, code.length()) - .get(1, TimeUnit.MINUTES); - if (settled && list instanceof UpdatingCompletionList) { - final CompletionList better = ((UpdatingCompletionList) list).update() - .get(1, TimeUnit.MINUTES); - if (better != null) list = better; - } - return list.getItems(); + return session(path).completion(code, code.length()).get(1, + TimeUnit.MINUTES).getItems(); } /** * Prints how long completions take, once warm: for another keystroke of a - * word the server has answered (cached); and for a new word, how long the - * editor waits (barely: it answers at once), and how long until the - * server's completions arrive as an update. + * word the server has answered (cached); and for a new word. */ private static void report(final String code, final String path, final String expected, final String more) throws Exception { completeUntil(code, path, expected); long t0 = System.nanoTime(); - final int n = complete(code + more, path, true).size(); + final int n = complete(code + more, path).size(); final long cached = System.nanoTime() - t0; final String other = code.substring(0, code.lastIndexOf('.') + 1) + "s"; final ScriptSession session = session(path); t0 = System.nanoTime(); - final CompletionList first = session.completion(other, other.length()) - .get(1, TimeUnit.MINUTES); - final long wait = System.nanoTime() - t0; - final CompletionList update = first instanceof UpdatingCompletionList - ? ((UpdatingCompletionList) first).update().get(1, TimeUnit.MINUTES) - : first; + final CompletionList all = session.completion(other, other.length()).get( + 1, TimeUnit.MINUTES); final long arrived = System.nanoTime() - t0; System.out.println("[completion timing] same word (cached): " + cached / - 1000000 + " ms, " + n + " completions; new word: editor waits " + wait / - 1000000 + " ms, all servers' " + update.getItems().size() + - " completions after " + arrived / 1000000 + " ms"); + 1000000 + " ms, " + n + " completions; new word: " + all.getItems() + .size() + " completions after " + arrived / 1000000 + " ms"); } } From 0fc15fcb1fb191704998a5e2df3dba0e8511ccc3 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Thu, 8 Oct 2026 17:18:54 -0500 Subject: [PATCH 14/37] Combine jedi and the syntax-only server in one server code-lsp no longer merges several plugins' servers, nor moves scripts between them. ApposePythonServerPlugin now offers one server, ApposePythonLanguageServer: jedi's completions first, then the syntax server's for other names; jedi's signatures, hover and problems, else the syntax server's. It moves each script to a jedi of where it runs as that changes (as its metadata is edited, or its environment is built). Co-Authored-By: Claude Opus 5.5 --- README.md | 12 +- .../python/ApposePythonEnvironments.java | 15 +- .../python/ApposePythonLanguageServer.java | 521 ++++++++++++++++++ ...gin.java => ApposePythonServerPlugin.java} | 60 +- .../python/PythonSyntaxLanguageServer.java | 2 +- .../python/PythonSyntaxServerPlugin.java | 62 --- .../appose/python/ScriptEnvironment.java | 2 +- ...ApposePythonCompletionIntegrationTest.java | 2 +- .../ApposePythonLanguageServerTest.java | 368 +++++++++++++ .../python/ApposePythonServersTest.java | 25 +- 10 files changed, 958 insertions(+), 111 deletions(-) create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServer.java rename src/main/java/org/scijava/plugins/scripting/appose/python/{JediServerPlugin.java => ApposePythonServerPlugin.java} (64%) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java diff --git a/README.md b/README.md index fd5525f..bb6975d 100644 --- a/README.md +++ b/README.md @@ -137,18 +137,20 @@ script's own line numbers. ## Code completion -The Script Editor completes code in these scripts with two language servers, -which `scijava-code-lsp`'s `LanguageServerService` merges into one (so any -other LSP client of it gets the same): +The Script Editor completes code in these scripts with one language server +(`ApposePythonLanguageServer`, offered to `scijava-code-lsp`'s +`LanguageServerService` by `ApposePythonServerPlugin`, so any other LSP client +of it gets the same), combining two sources: jedi's answers first, then the +syntax-only ones for names jedi has not completed. * **Instantly, with no Python needed:** names in scope (variables, functions, classes, imports, parameters of the enclosing function), the script's `#@` parameters, `task`, builtins and keywords. This comes from parsing the script with [tree-sitter](https://tree-sitter.github.io/) - (`PythonSyntaxServerPlugin`). + (`PythonSyntaxLanguageServer`). * **With a language server, [jedi-language-server](https://github.com/pappasam/jedi-language-server) - (built on [Jedi](https://jedi.readthedocs.io/)), via `JediServerPlugin`:** + (built on [Jedi](https://jedi.readthedocs.io/)):** members of objects and modules, function signatures (with parameter names and types), documentation, hover and syntax errors. The server runs in a small environment of its own (`appose-python-completion`: Python, jedi-language-server, Appose and uv), diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java index cb72757..915459c 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java @@ -42,6 +42,7 @@ import java.util.List; import java.util.Map; import java.util.WeakHashMap; +import java.util.concurrent.CopyOnWriteArrayList; import org.apposed.appose.Appose; import org.apposed.appose.Environment; @@ -100,7 +101,7 @@ protected boolean removeEldestEntry( private DependencyOverlays overlays; /** Told when where scripts run may have changed. */ - private volatile Runnable onChange = () -> {}; + private final List listeners = new CopyOnWriteArrayList<>(); private ApposePythonEnvironments(final ResidentWorkerService workers) { this.workers = workers; @@ -140,15 +141,19 @@ org.scijava.code.lsp.Environment environment(final String text, } /** - * Sets what to do when where scripts run may have changed without them + * Adds what to do when where scripts run may have changed without them * changing: when an environment is built, or an overlay is ready. */ - void onChange(final Runnable onChange) { - this.onChange = onChange == null ? () -> {} : onChange; + void addListener(final Runnable listener) { + listeners.add(listener); + } + + void removeListener(final Runnable listener) { + listeners.remove(listener); } private void changed() { - onChange.run(); + for (final Runnable listener : listeners) listener.run(); } /** Sets the dependency overlays, e.g. to use fake ones in tests. */ diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServer.java new file mode 100644 index 0000000..74bfe77 --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServer.java @@ -0,0 +1,521 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.Comparator; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Function; + +import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionList; +import org.eclipse.lsp4j.CompletionOptions; +import org.eclipse.lsp4j.CompletionParams; +import org.eclipse.lsp4j.DidChangeConfigurationParams; +import org.eclipse.lsp4j.DidChangeTextDocumentParams; +import org.eclipse.lsp4j.DidChangeWatchedFilesParams; +import org.eclipse.lsp4j.DidCloseTextDocumentParams; +import org.eclipse.lsp4j.DidOpenTextDocumentParams; +import org.eclipse.lsp4j.DidSaveTextDocumentParams; +import org.eclipse.lsp4j.Hover; +import org.eclipse.lsp4j.HoverParams; +import org.eclipse.lsp4j.InitializeParams; +import org.eclipse.lsp4j.InitializeResult; +import org.eclipse.lsp4j.InitializedParams; +import org.eclipse.lsp4j.MessageActionItem; +import org.eclipse.lsp4j.MessageParams; +import org.eclipse.lsp4j.PublishDiagnosticsParams; +import org.eclipse.lsp4j.ServerCapabilities; +import org.eclipse.lsp4j.ShowMessageRequestParams; +import org.eclipse.lsp4j.SignatureHelp; +import org.eclipse.lsp4j.SignatureHelpOptions; +import org.eclipse.lsp4j.SignatureHelpParams; +import org.eclipse.lsp4j.TextDocumentContentChangeEvent; +import org.eclipse.lsp4j.TextDocumentIdentifier; +import org.eclipse.lsp4j.TextDocumentItem; +import org.eclipse.lsp4j.TextDocumentSyncKind; +import org.eclipse.lsp4j.jsonrpc.messages.Either; +import org.eclipse.lsp4j.services.LanguageClient; +import org.eclipse.lsp4j.services.LanguageClientAware; +import org.eclipse.lsp4j.services.LanguageServer; +import org.eclipse.lsp4j.services.TextDocumentService; +import org.eclipse.lsp4j.services.WorkspaceService; +import org.scijava.code.lsp.Environment; + +/** + * The language server of Python scripts run via Appose: the syntax-only + * server (see {@link PythonSyntaxLanguageServer}), instant and needing no + * Python; and, if offered, jedi-language-server, knowing more. + *
      + *
    • Completions: jedi's, then the syntax server's that complete other + * names. (Without jedi, the syntax server's.)
      • + *
    • Signature help and hover: jedi's, if any; otherwise the syntax + * server's.
      • + *
    • Problems: jedi's (the syntax server finds none).
      • + *
    + *

    + * Each script is analyzed by a jedi of where it runs: its environment's + * Python, once built, and an overlay of the packages it declares but its + * environment lacks (see {@link ApposePythonEnvironments}). As that changes + * (as the script's metadata is edited, or its environment is built), the + * script moves to another jedi. + *

    + * + * @author Gabriel Selzer + */ +public class ApposePythonLanguageServer implements LanguageServer, + TextDocumentService, WorkspaceService, LanguageClientAware +{ + + private final LanguageServer syntax; + private final Function environment; + private final Function jedi; + private final ApposePythonEnvironments environments; + private final Runnable relocateAll = this::relocateAll; + + /** The open documents, by URI. */ + private final Map docs = new ConcurrentHashMap<>(); + + private volatile LanguageClient client; + + /** + * @param syntax The syntax-only server. + * @param environment Gets where a script runs. + * @param jedi Makes a jedi for scripts running in an environment; or null, + * if jedi is not offered. + * @param environments Tells when where scripts run may have changed without + * them changing (e.g. as environments are built); or null. + */ + ApposePythonLanguageServer(final LanguageServer syntax, + final Function environment, + final Function jedi, + final ApposePythonEnvironments environments) + { + this.syntax = syntax; + this.environment = environment; + this.jedi = jedi; + this.environments = environments; + if (syntax instanceof LanguageClientAware) { + ((LanguageClientAware) syntax).connect(new Forwarder(null)); + } + if (jedi != null && environments != null) { + environments.addListener(relocateAll); + } + } + + @Override + public void connect(final LanguageClient c) { + client = c; + } + + // -- LanguageServer -- + + @Override + public CompletableFuture initialize( + final InitializeParams params) + { + syntax.initialize(params).thenRun(() -> syntax.initialized( + new InitializedParams())); + final ServerCapabilities capabilities = new ServerCapabilities(); + capabilities.setTextDocumentSync(TextDocumentSyncKind.Full); + capabilities.setCompletionProvider(new CompletionOptions(true, + Collections.singletonList("."))); + capabilities.setSignatureHelpProvider(new SignatureHelpOptions(Collections + .singletonList("("))); + capabilities.setHoverProvider(true); + return CompletableFuture.completedFuture(new InitializeResult( + capabilities)); + } + + @Override + public CompletableFuture shutdown() { + if (environments != null) environments.removeListener(relocateAll); + for (final String uri : new ArrayList<>(docs.keySet())) { + didClose(new DidCloseTextDocumentParams(new TextDocumentIdentifier( + uri))); + } + return syntax.shutdown(); + } + + @Override + public void exit() { + syntax.exit(); + } + + @Override + public TextDocumentService getTextDocumentService() { + return this; + } + + @Override + public WorkspaceService getWorkspaceService() { + return this; + } + + // -- TextDocumentService: documents -- + + @Override + public void didOpen(final DidOpenTextDocumentParams params) { + syntax.getTextDocumentService().didOpen(params); + if (jedi == null) return; + final Doc doc = new Doc(params.getTextDocument()); + docs.put(doc.item.getUri(), doc); + synchronized (doc) { + doc.environment = environment.apply(doc.item); + doc.jedi = start(doc.environment); + doc.jedi.getTextDocumentService().didOpen(params); + } + } + + @Override + public void didChange(final DidChangeTextDocumentParams params) { + syntax.getTextDocumentService().didChange(params); + final Doc doc = docs.get(params.getTextDocument().getUri()); + final List changes = params + .getContentChanges(); + if (doc == null || changes.isEmpty()) return; + synchronized (doc) { + // NB: Full sync: the last change is the whole text. + doc.item = new TextDocumentItem(doc.item.getUri(), doc.item + .getLanguageId(), params.getTextDocument().getVersion(), changes.get( + changes.size() - 1).getText()); + if (!relocate(doc)) doc.jedi.getTextDocumentService().didChange(params); + } + } + + @Override + public void didClose(final DidCloseTextDocumentParams params) { + syntax.getTextDocumentService().didClose(params); + final Doc doc = docs.remove(params.getTextDocument().getUri()); + if (doc == null) return; + synchronized (doc) { + stop(doc, params.getTextDocument().getUri()); + } + } + + @Override + public void didSave(final DidSaveTextDocumentParams params) { + // NB: Changes come with didChange. + } + + // -- TextDocumentService: requests -- + + @Override + public CompletableFuture, CompletionList>> + completion(final CompletionParams params) + { + final CompletableFuture own = orNull(syntax + .getTextDocumentService().completion(params)).thenApply( + ApposePythonLanguageServer::list); + final LanguageServer j = jedi(params.getTextDocument().getUri()); + if (j == null) return own.thenApply(Either::forRight); + return orNull(j.getTextDocumentService().completion(params)).thenApply( + ApposePythonLanguageServer::list).thenCombine(own, (theirs, + mine) -> Either.forRight(merge(theirs, mine, params.getTextDocument() + .getUri()))); + } + + @Override + public CompletableFuture resolveCompletionItem( + final CompletionItem item) + { + if (!(item.getData() instanceof Source)) { + // NB: Not merged (e.g. without jedi): the syntax server's. + return syntax.getTextDocumentService().resolveCompletionItem(item); + } + final Source from = (Source) item.getData(); + final LanguageServer server = from.jedi ? jedi(from.uri) : syntax; + final String sortText = item.getSortText(); + item.setData(from.data); + item.setSortText(from.sortText); + if (server == null) { + item.setData(from); + item.setSortText(sortText); + return CompletableFuture.completedFuture(item); + } + return orNull(server.getTextDocumentService().resolveCompletionItem(item)) + .thenApply(resolved -> { + final CompletionItem out = resolved != null ? resolved : item; + out.setData(from); + out.setSortText(sortText); + return out; + }); + } + + @Override + public CompletableFuture signatureHelp( + final SignatureHelpParams params) + { + return first(params.getTextDocument().getUri(), s -> s.signatureHelp( + params), h -> h != null && h.getSignatures() != null && !h + .getSignatures().isEmpty()); + } + + @Override + public CompletableFuture hover(final HoverParams params) { + return first(params.getTextDocument().getUri(), s -> s.hover(params), + h -> h != null && h.getContents() != null); + } + + // -- WorkspaceService -- + + @Override + public void didChangeConfiguration(final DidChangeConfigurationParams p) { + // NB: Nothing to configure. + } + + @Override + public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) { + // NB: Nothing watched. + } + + // -- Helper methods -- + + /** The jedi analyzing a document, or null if none. */ + private LanguageServer jedi(final String uri) { + final Doc doc = docs.get(uri); + if (doc == null) return null; + synchronized (doc) { + return doc.jedi; + } + } + + /** Starts a jedi for scripts running in an environment. */ + private LanguageServer start(final Environment env) { + final LanguageServer server = jedi.apply(env); + if (server instanceof LanguageClientAware) { + ((LanguageClientAware) server).connect(new Forwarder(server)); + } + server.initialize(new InitializeParams()).thenRun(() -> server + .initialized(new InitializedParams())); + return server; + } + + /** Closes a document on its jedi, and lets that jedi go. */ + private static void stop(final Doc doc, final String uri) { + doc.jedi.getTextDocumentService().didClose(new DidCloseTextDocumentParams( + new TextDocumentIdentifier(uri))); + doc.jedi.shutdown(); + } + + /** + * Moves a document to a jedi of where its script runs now, if that changed. + * + * @return Whether it moved (and the new jedi has its current text). + */ + private boolean relocate(final Doc doc) { + final Environment env = environment.apply(doc.item); + if (env.equals(doc.environment)) return false; + // TODO: Settle how often this may happen while dependencies are edited + // (debounce?). + stop(doc, doc.item.getUri()); + doc.environment = env; + doc.jedi = start(env); + doc.jedi.getTextDocumentService().didOpen(new DidOpenTextDocumentParams( + new TextDocumentItem(doc.item.getUri(), doc.item.getLanguageId(), doc.item + .getVersion(), doc.item.getText()))); + return true; + } + + /** + * Moves each open document to a jedi of where its script runs now, if that + * changed without the script changing (e.g. its environment was built). + */ + void relocateAll() { + for (final Doc doc : docs.values()) { + synchronized (doc) { + relocate(doc); + } + } + } + + /** Asks jedi, then (if its answer is not accepted) the syntax server. */ + private CompletableFuture first(final String uri, + final Function> request, + final java.util.function.Predicate accepted) + { + final LanguageServer j = jedi(uri); + final CompletableFuture theirs = j == null ? CompletableFuture + .completedFuture(null) : orNull(request.apply(j + .getTextDocumentService())); + return theirs.thenCompose(t -> accepted.test(t) ? CompletableFuture + .completedFuture(t) : orNull(request.apply(syntax + .getTextDocumentService()))); + } + + /** + * Jedi's completions, then the syntax server's whose names jedi has not + * completed, each in its own order. Each remembers whose it is, for + * resolving. + */ + private static CompletionList merge(final CompletionList theirs, + final CompletionList own, final String uri) + { + final List out = new ArrayList<>(); + final Set names = new HashSet<>(); + int i = 0; + for (final CompletionItem item : sorted(theirs)) { + names.add(name(item)); + mark(item, true, uri, i++); + out.add(item); + } + for (final CompletionItem item : sorted(own)) { + if (names.contains(name(item))) continue; + mark(item, false, uri, i++); + out.add(item); + } + final boolean incomplete = theirs != null && theirs.isIncomplete() || + own != null && own.isIncomplete(); + return new CompletionList(incomplete, out); + } + + private static void mark(final CompletionItem item, final boolean fromJedi, + final String uri, final int position) + { + item.setData(new Source(fromJedi, uri, item.getData(), item + .getSortText())); + item.setSortText(String.format("%d%06d", fromJedi ? 0 : 1, position)); + } + + private static List sorted(final CompletionList list) { + final List items = new ArrayList<>(); + if (list != null && list.getItems() != null) items.addAll(list.getItems()); + items.sort(Comparator.comparing(c -> c.getSortText() != null ? c + .getSortText() : c.getLabel())); + return items; + } + + /** What a completion completes: its label, without any parameters. */ + private static String name(final CompletionItem item) { + final String label = item.getLabel(); + final int paren = label.indexOf('('); + return (paren < 0 ? label : label.substring(0, paren)).trim(); + } + + private static CompletionList list( + final Either, CompletionList> answer) + { + if (answer == null) return null; + return answer.isRight() ? answer.getRight() : new CompletionList(answer + .getLeft()); + } + + /** The answer; or null, if the server failed (e.g. it stopped). */ + private static CompletableFuture orNull( + final CompletableFuture answer) + { + return answer == null ? CompletableFuture.completedFuture(null) : answer + .exceptionally(error -> null); + } + + /** An open document, and the jedi analyzing it. */ + private static final class Doc { + + private TextDocumentItem item; + private Environment environment; + private LanguageServer jedi; + + private Doc(final TextDocumentItem item) { + this.item = item; + } + } + + /** Remembers whose a merged completion is. */ + private static final class Source { + + private final boolean jedi; + private final String uri; + private final Object data; + private final String sortText; + + private Source(final boolean jedi, final String uri, final Object data, + final String sortText) + { + this.jedi = jedi; + this.uri = uri; + this.data = data; + this.sortText = sortText; + } + } + + /** + * Passes problems on to the client: jedi's, from the jedi analyzing their + * document; or, without jedi, the syntax server's (none). + */ + private final class Forwarder implements LanguageClient { + + /** The jedi, or null for the syntax server. */ + private final LanguageServer source; + + private Forwarder(final LanguageServer source) { + this.source = source; + } + + @Override + public void publishDiagnostics(final PublishDiagnosticsParams p) { + final LanguageClient c = client; + if (c == null) return; + if (source == null) { + if (jedi == null) c.publishDiagnostics(p); + return; + } + // NB: Only from the jedi analyzing the document now. + if (jedi(p.getUri()) == source) c.publishDiagnostics(p); + } + + @Override + public void telemetryEvent(final Object object) {} + + @Override + public void showMessage(final MessageParams m) { + final LanguageClient c = client; + if (c != null) c.showMessage(m); + } + + @Override + public CompletableFuture showMessageRequest( + final ShowMessageRequestParams r) + { + return CompletableFuture.completedFuture(null); + } + + @Override + public void logMessage(final MessageParams m) { + final LanguageClient c = client; + if (c != null) c.logMessage(m); + } + } +} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java similarity index 64% rename from src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java rename to src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java index c5c1595..d5ab8cf 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java @@ -47,15 +47,17 @@ import org.scijava.script.ScriptLanguage; /** - * Offers jedi-language-server (see {@link ApposePythonLspServerPlugin}) for - * Python scripts run via Appose: members of objects and modules, signatures, - * documentation, hover and syntax errors. Once a script's environment is built - * (by running the script), the server analyzes the script against it, so its - * packages are known; packages the script declares but its environment lacks - * are installed for completion only (see + * Offers the language server of Python scripts run via Appose (see + * {@link ApposePythonLanguageServer}): the syntax-only server, and, if offered + * (see {@link ApposePythonLspServerPlugin}), jedi-language-server: members of + * objects and modules, signatures, documentation, hover and syntax errors. + * Once a script's environment is built (by running the script), jedi + * analyzes the script against it, so its packages are known; packages the + * script declares but its environment lacks are installed for completion only + * (see * {@link org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays}). - * The server sees the script as plain Python, its parameters declared with - * the types they have when it runs (e.g. images as NumPy arrays). + * Jedi sees the script as plain Python, its parameters declared with the + * types they have when it runs (e.g. images as NumPy arrays). *

    * Completion never builds a script's environment, and never disturbs its * worker. @@ -63,9 +65,9 @@ * * @author Gabriel Selzer */ -@Plugin(type = LanguageServerPlugin.class, name = "jedi-language-server", +@Plugin(type = LanguageServerPlugin.class, name = "appose-python", priority = Priority.HIGH) -public class JediServerPlugin implements LanguageServerPlugin { +public class ApposePythonServerPlugin implements LanguageServerPlugin { /** How many completions get signatures, once a word is begun. */ private static final int SIGNATURES = 30; @@ -76,32 +78,42 @@ public class JediServerPlugin implements LanguageServerPlugin { @Parameter private ResidentWorkerService workers; - @Override public boolean supports(final ScriptLanguage language) { - return language instanceof ApposePythonScriptLanguage && servers.launch( - "python", Environment.NONE) != null; + return language instanceof ApposePythonScriptLanguage; } @Override - public Environment environment(final TextDocumentItem document) { - final ApposePythonEnvironments environments = ApposePythonEnvironments.of( - workers); - // NB: Scripts move as their environments are built, or overlays ready. - environments.onChange(servers::environmentsChanged); - // NB: On opening the script, its dependencies are settled. - return environments.environment(document.getText(), document.getUri(), - document.getVersion() <= 1); + public LanguageServer create(final Environment environment) { + final boolean jedi = jediOffered(); + return new ApposePythonLanguageServer(new PythonSyntaxLanguageServer( + servers), this::environment, jedi ? this::jedi : null, + ApposePythonEnvironments.of(workers)); } - @Override - public LanguageServer create(final Environment environment) { + /** Whether jedi-language-server is offered. */ + boolean jediOffered() { + return servers.launch("python", Environment.NONE) != null; + } + + /** + * Gets where a script runs, for jedi: its environment's Python (once built), + * and an overlay of the packages it declares but its environment lacks. + * On opening the script, its dependencies are settled. + */ + Environment environment(final TextDocumentItem document) { + return ApposePythonEnvironments.of(workers).environment(document + .getText(), document.getUri(), document.getVersion() <= 1); + } + + /** Makes a jedi for scripts running in an environment. */ + private LanguageServer jedi(final Environment environment) { return new PythonCompletionServer(new TransformingLanguageServer(servers .launch("python", environment), text -> document(text, environment)), SIGNATURES); } - /** The script as plain Python, for the server. */ + /** The script as plain Python, for jedi. */ ScriptDocument document(final String text, final Environment environment) { final Map> params = servers == null ? Collections .emptyMap() : servers.scriptParameters(text); diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java index 8d74e1d..bda354f 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java @@ -51,7 +51,7 @@ * needed, names in scope, the script's parameters, {@code task}, builtins and * keywords; and, after a dot, the names used after dots elsewhere in the * script. (A Python language server, e.g. jedi-language-server, knows more; - * see {@link JediServerPlugin}.) + * see {@link ApposePythonLanguageServer}.) * * @author Gabriel Selzer */ diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java deleted file mode 100644 index 2cc7f15..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxServerPlugin.java +++ /dev/null @@ -1,62 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python; - -import org.eclipse.lsp4j.services.LanguageServer; -import org.scijava.code.lsp.LanguageServerService; -import org.scijava.code.lsp.Environment; -import org.scijava.code.lsp.LanguageServerPlugin; -import org.scijava.plugin.Parameter; -import org.scijava.plugin.Plugin; -import org.scijava.script.ScriptLanguage; - -/** - * Offers the syntax-only language server for Python scripts run via Appose - * (see {@link PythonSyntaxLanguageServer}): instant, and needing no Python. - * Its completions follow jedi-language-server's (see {@link JediServerPlugin}). - * - * @author Gabriel Selzer - */ -@Plugin(type = LanguageServerPlugin.class, name = "Python syntax") -public class PythonSyntaxServerPlugin implements LanguageServerPlugin { - - @Parameter(required = false) - private LanguageServerService servers; - - @Override - public boolean supports(final ScriptLanguage language) { - return language instanceof ApposePythonScriptLanguage; - } - - @Override - public LanguageServer create(final Environment environment) { - return new PythonSyntaxLanguageServer(servers); - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java index e211e8f..11e75a5 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java @@ -49,7 +49,7 @@ * directive. *

    * Both running a script ({@link ApposePythonScriptEngine}) and completing code - * in it ({@link JediServerPlugin}) resolve its environment here, so + * in it ({@link ApposePythonServerPlugin}) resolve its environment here, so * that they agree on the environment's name and configuration. *

    */ diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java index 7ecceb4..cbb2899 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -64,7 +64,7 @@ /** * End-to-end tests of the language servers of Python scripts run via Appose - * (see {@link JediServerPlugin}), with a real jedi-language-server, through + * (see {@link ApposePythonServerPlugin}), with a real jedi-language-server, through * the {@link LanguageServerService}, as editors use it. *

    * The first completion builds the completion tooling environment (Python and diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java new file mode 100644 index 0000000..245a77e --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java @@ -0,0 +1,368 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python; + +import static org.junit.Assert.assertEquals; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.TimeUnit; +import java.util.function.Function; +import java.util.stream.Collectors; + +import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionList; +import org.eclipse.lsp4j.CompletionParams; +import org.eclipse.lsp4j.Diagnostic; +import org.eclipse.lsp4j.DidChangeConfigurationParams; +import org.eclipse.lsp4j.DidChangeTextDocumentParams; +import org.eclipse.lsp4j.DidChangeWatchedFilesParams; +import org.eclipse.lsp4j.DidCloseTextDocumentParams; +import org.eclipse.lsp4j.DidOpenTextDocumentParams; +import org.eclipse.lsp4j.DidSaveTextDocumentParams; +import org.eclipse.lsp4j.InitializeParams; +import org.eclipse.lsp4j.InitializeResult; +import org.eclipse.lsp4j.MessageActionItem; +import org.eclipse.lsp4j.MessageParams; +import org.eclipse.lsp4j.Position; +import org.eclipse.lsp4j.PublishDiagnosticsParams; +import org.eclipse.lsp4j.Range; +import org.eclipse.lsp4j.ServerCapabilities; +import org.eclipse.lsp4j.ShowMessageRequestParams; +import org.eclipse.lsp4j.TextDocumentContentChangeEvent; +import org.eclipse.lsp4j.TextDocumentIdentifier; +import org.eclipse.lsp4j.TextDocumentItem; +import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; +import org.eclipse.lsp4j.jsonrpc.messages.Either; +import org.eclipse.lsp4j.services.LanguageClient; +import org.eclipse.lsp4j.services.LanguageClientAware; +import org.eclipse.lsp4j.services.LanguageServer; +import org.eclipse.lsp4j.services.TextDocumentService; +import org.eclipse.lsp4j.services.WorkspaceService; +import org.junit.Before; +import org.junit.Test; +import org.scijava.code.lsp.Environment; + +/** + * Tests {@link ApposePythonLanguageServer}, with fake servers: how it combines + * jedi and the syntax-only server, and moves scripts between jedis as where + * they run changes. + * + * @author Gabriel Selzer + */ +public class ApposePythonLanguageServerTest { + + private static final String URI = "untitled:/a.py"; + + /** The jedis started, in order. */ + private final List jedis = new CopyOnWriteArrayList<>(); + private final List published = + new CopyOnWriteArrayList<>(); + + /** Where scripts run when their text does not say (e.g. once built). */ + private volatile String fallback = "A"; + + private Fake syntax; + private ApposePythonLanguageServer server; + + @Before + public void setUp() { + syntax = new Fake("syntax", "max", "min"); + server = server(env -> { + final Fake jedi = new Fake("jedi:" + env.interpreter(), "max", "map"); + jedis.add(jedi); + return jedi; + }); + } + + @Test + public void testRelocation() { + open("x = 1"); + assertEquals(1, jedis.size()); + assertEquals(Collections.singletonList("open 1 x = 1"), jedis.get(0).events); + + // Its text says it runs elsewhere: it moves to a jedi there. + change("#env=B\nx = 1", 2); + assertEquals(Arrays.asList("open 1 x = 1", "close", "shutdown"), jedis.get( + 0).events); + assertEquals("jedi:B", jedis.get(1).name); + assertEquals(Collections.singletonList("open 2 #env=B\nx = 1"), jedis.get( + 1).events); + // Otherwise, changes go to the same jedi. + change("#env=B\nx = 2", 3); + assertEquals("change 3 #env=B\nx = 2", jedis.get(1).events.get(1)); + assertEquals(2, jedis.size()); + + // Where it runs changes without it changing (e.g. its environment was + // built): nothing moves until told so. + change("y = 1", 4); + assertEquals("jedi:A", jedis.get(2).name); + fallback = "C"; + assertEquals(3, jedis.size()); + server.relocateAll(); + assertEquals(Arrays.asList("open 4 y = 1", "close", "shutdown"), jedis.get( + 2).events); + assertEquals("jedi:C", jedis.get(3).name); + assertEquals(Collections.singletonList("open 4 y = 1"), jedis.get(3).events); + + // Closed: on the syntax server and its jedi, which is let go. + server.didClose(new DidCloseTextDocumentParams(new TextDocumentIdentifier( + URI))); + assertEquals(Arrays.asList("open 4 y = 1", "close", "shutdown"), jedis.get( + 3).events); + assertEquals("close", syntax.events.get(syntax.events.size() - 1)); + } + + @Test + public void testCompletions() throws Exception { + open("ma"); + final CompletionList list = complete(); + // Jedi's, then the syntax server's for names jedi did not complete; each + // in its own order. + assertEquals(Arrays.asList("map", "max", "min"), labels(list)); + + // Resolved by the server each came from, with its own data. + final CompletionItem min = list.getItems().get(2); + final Object marker = min.getData(); + final CompletionItem resolved = server.resolveCompletionItem(min).get(10, + TimeUnit.SECONDS); + assertEquals("syntax resolved min (data syntax)", resolved.getDetail()); + assertEquals(marker, resolved.getData()); + assertEquals("jedi:A resolved map (data jedi:A)", server + .resolveCompletionItem(list.getItems().get(0)).get(10, TimeUnit.SECONDS) + .getDetail()); + } + + @Test + public void testDiagnostics() { + open("x = 1"); + final Fake a = jedis.get(0); + // The syntax server's are not passed on (jedi's are the problems). + syntax.publish(URI, 1, "syntax"); + a.publish(URI, 1, "from A"); + assertEquals(Collections.singletonList("from A"), messages()); + + // Moved: the old jedi's are out of date. + change("#env=B\nx = 1", 2); + a.publish(URI, 1, "stale"); + jedis.get(1).publish(URI, 2, "from B"); + assertEquals(Arrays.asList("from A", "from B"), messages()); + } + + @Test + public void testWithoutJedi() throws Exception { + server = server(null); + open("ma"); + // The syntax server's answers, as they are. + final CompletionList list = complete(); + assertEquals(Arrays.asList("max", "min"), labels(list)); + assertEquals("syntax", list.getItems().get(0).getData()); + assertEquals("syntax resolved max (data syntax)", server + .resolveCompletionItem(list.getItems().get(0)).get(10, TimeUnit.SECONDS) + .getDetail()); + syntax.publish(URI, 1, "syntax"); + assertEquals(Collections.singletonList("syntax"), messages()); + } + + // -- Helper methods -- + + private ApposePythonLanguageServer server( + final Function jedi) + { + final ApposePythonLanguageServer s = new ApposePythonLanguageServer(syntax, + doc -> new Environment(doc.getText().contains("#env=B") ? "B" + : fallback, null, null, null), jedi, null); + s.connect(new Client()); + s.initialize(new InitializeParams()); + return s; + } + + private void open(final String text) { + server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(URI, + "appose-python", 1, text))); + } + + private void change(final String text, final int version) { + server.didChange(new DidChangeTextDocumentParams( + new VersionedTextDocumentIdentifier(URI, version), Collections + .singletonList(new TextDocumentContentChangeEvent(text)))); + } + + private CompletionList complete() throws Exception { + return server.completion(new CompletionParams(new TextDocumentIdentifier( + URI), new Position(0, 2))).get(10, TimeUnit.SECONDS).getRight(); + } + + private static List labels(final CompletionList list) { + return list.getItems().stream().map(CompletionItem::getLabel).collect( + Collectors.toList()); + } + + private List messages() { + final List out = new ArrayList<>(); + for (final PublishDiagnosticsParams p : published) { + for (final Diagnostic d : p.getDiagnostics()) { + out.add(d.getMessage().getLeft()); + } + } + return out; + } + + private final class Client implements LanguageClient { + + @Override + public void publishDiagnostics(final PublishDiagnosticsParams p) { + published.add(p); + } + + @Override + public void telemetryEvent(final Object object) {} + + @Override + public void showMessage(final MessageParams m) {} + + @Override + public CompletableFuture showMessageRequest( + final ShowMessageRequestParams r) + { + return CompletableFuture.completedFuture(null); + } + + @Override + public void logMessage(final MessageParams m) {} + } + + /** A fake server, recording what it is told, completing given labels. */ + private static final class Fake implements LanguageServer, + TextDocumentService, WorkspaceService, LanguageClientAware + { + + private final String name; + private final List labels; + private final List events = new CopyOnWriteArrayList<>(); + private LanguageClient client; + + private Fake(final String name, final String... labels) { + this.name = name; + this.labels = Arrays.asList(labels); + } + + void publish(final String uri, final int version, final String message) { + client.publishDiagnostics(new PublishDiagnosticsParams(uri, + Collections.singletonList(new Diagnostic(new Range(new Position(0, 0), + new Position(0, 1)), message)), version)); + } + + @Override + public void connect(final LanguageClient c) { + client = c; + } + + @Override + public CompletableFuture initialize( + final InitializeParams params) + { + return CompletableFuture.completedFuture(new InitializeResult( + new ServerCapabilities())); + } + + @Override + public CompletableFuture shutdown() { + events.add("shutdown"); + return CompletableFuture.completedFuture(null); + } + + @Override + public void exit() {} + + @Override + public TextDocumentService getTextDocumentService() { + return this; + } + + @Override + public WorkspaceService getWorkspaceService() { + return this; + } + + @Override + public void didOpen(final DidOpenTextDocumentParams p) { + final TextDocumentItem d = p.getTextDocument(); + events.add("open " + d.getVersion() + " " + d.getText()); + } + + @Override + public void didChange(final DidChangeTextDocumentParams p) { + events.add("change " + p.getTextDocument().getVersion() + " " + p + .getContentChanges().get(0).getText()); + } + + @Override + public void didClose(final DidCloseTextDocumentParams p) { + events.add("close"); + } + + @Override + public void didSave(final DidSaveTextDocumentParams p) {} + + @Override + public CompletableFuture, CompletionList>> + completion(final CompletionParams p) + { + final List items = new ArrayList<>(); + for (final String label : labels) { + final CompletionItem item = new CompletionItem(label); + item.setData(name); + items.add(item); + } + return CompletableFuture.completedFuture(Either.forRight( + new CompletionList(items))); + } + + @Override + public CompletableFuture resolveCompletionItem( + final CompletionItem item) + { + item.setDetail(name + " resolved " + item.getLabel() + " (data " + item + .getData() + ")"); + return CompletableFuture.completedFuture(item); + } + + @Override + public void didChangeConfiguration(final DidChangeConfigurationParams p) {} + + @Override + public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) {} + } +} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java index 1c1d623..495ccd5 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java @@ -63,7 +63,7 @@ /** * Tests the language servers of Python scripts run via Appose, without * starting any server process: the syntax-only server - * ({@link PythonSyntaxLanguageServer}), and what {@link JediServerPlugin} + * ({@link PythonSyntaxLanguageServer}), and what {@link ApposePythonServerPlugin} * tells jedi-language-server (the script as plain Python, and where it runs). * * @author Gabriel Selzer @@ -74,7 +74,7 @@ public class ApposePythonServersTest { private Context context; private ScriptLanguage language; - private JediServerPlugin jedi; + private ApposePythonServerPlugin plugin; private ApposePythonEnvironments environments; @Before @@ -83,9 +83,10 @@ public void setUp() { ResidentWorkerService.class, LanguageServerService.class); language = context.service(ScriptService.class).getLanguageByName( "appose-python"); - jedi = context.service(PluginService.class).createInstancesOfType( + plugin = context.service(PluginService.class).createInstancesOfType( LanguageServerPlugin.class).stream().filter( - p -> p instanceof JediServerPlugin).map(p -> (JediServerPlugin) p) + p -> p instanceof ApposePythonServerPlugin).map( + p -> (ApposePythonServerPlugin) p) .findFirst().get(); environments = ApposePythonEnvironments.of(context.service( ResidentWorkerService.class)); @@ -101,7 +102,7 @@ public void testServersAreOffered() { assertTrue(context.service(LanguageServerService.class).supports( language)); // NB: Here, appose-python offers the Python server itself. - assertTrue(jedi.supports(language)); + assertTrue(plugin.jediOffered()); } @Test @@ -125,7 +126,7 @@ public void testScriptAsPython() { "image.sha"; // The script's parameters are typed in a preamble, so the server knows // them. - assertTrue(jedi.document(code, Environment.NONE).text().startsWith( + assertTrue(plugin.document(code, Environment.NONE).text().startsWith( "import appose.python_worker; import numpy; image: numpy.ndarray; " + "sigma: float; task: appose.python_worker.Task\n")); } @@ -133,7 +134,7 @@ public void testScriptAsPython() { @Test public void testEnvironmentWithoutOne() { // No script environment has been built: the server's own Python. - final Environment env = jedi.environment(new TextDocumentItem( + final Environment env = plugin.environment(new TextDocumentItem( "untitled:/a.py", "appose-python", 2, "import os\nos.pa")); assertNull(env.interpreter()); assertTrue(env.searchPaths().isEmpty()); @@ -154,12 +155,12 @@ public void testOverlay() throws Exception { "appose-python", 1, code); // On opening the script, its dependencies are settled: the overlay is // installed; once ready, the server searches it. - jedi.environment(doc); + plugin.environment(doc); for (int i = 0; i < 100 && installed.isEmpty(); i++) Thread.sleep(20); Thread.sleep(50); assertEquals(Collections.singletonList(Collections.singletonList( "tifffile")), installed); - assertEquals(Collections.singletonList("/overlays/blur"), jedi.environment( + assertEquals(Collections.singletonList("/overlays/blur"), plugin.environment( doc).searchPaths()); } @@ -169,9 +170,9 @@ public void testScriptWorkersUntouched() { // it, even as its metadata is edited. final String script = "# /// script\n# dependencies = [\"appose\"]\n" + "# ///\nimport os\nos.pa"; - jedi.environment(new TextDocumentItem("untitled:/a.py", "appose-python", 2, + plugin.environment(new TextDocumentItem("untitled:/a.py", "appose-python", 2, script)); - jedi.environment(new TextDocumentItem("untitled:/a.py", "appose-python", 3, + plugin.environment(new TextDocumentItem("untitled:/a.py", "appose-python", 3, script.replace("appose", "appose, numpy"))); assertTrue(context.service(ResidentWorkerService.class).workers() .isEmpty()); @@ -196,7 +197,7 @@ public void testEnvironmentFromEarlierSession() throws Exception { java.nio.file.Files.createDirectories(python.getParent()); java.nio.file.Files.createFile(python); - final Environment env = jedi.environment(new TextDocumentItem(script + final Environment env = plugin.environment(new TextDocumentItem(script .toURI().toString(), "appose-python", 2, header + "numpy.ze")); // The server analyzes against the environment's Python... assertEquals(python.toFile().getAbsolutePath(), env.interpreter()); From dabdd814ee8ba92c44b645ce6592dc1c7e3f19fb Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 9 Oct 2026 10:28:04 -0500 Subject: [PATCH 15/37] Unify Environment's search paths and classpath Both are ordered places imports are resolved from; each server reads them as its language does. Nothing read the classpath. Co-Authored-By: Claude Opus 5.5 --- .../scripting/appose/python/ApposePythonEnvironments.java | 2 +- .../appose/python/completion/JediLanguageServerSpec.java | 2 +- .../appose/python/ApposePythonLanguageServerTest.java | 2 +- .../scripting/appose/python/ApposePythonServersTest.java | 6 +++--- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java index 915459c..c3568d2 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java @@ -137,7 +137,7 @@ org.scijava.code.lsp.Environment environment(final String text, final String python = scriptPython(text, path); final String overlay = overlay(text, path, python, settled); return new org.scijava.code.lsp.Environment(python, overlay == null ? null - : Collections.singletonList(overlay), null, null); + : Collections.singletonList(overlay), null); } /** diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java index dc1fd44..f45ca2f 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java @@ -126,7 +126,7 @@ public Object initializationOptions(final Environment env) { if (env.interpreter() != null) { workspaceOptions.put("environmentPath", env.interpreter()); } - workspaceOptions.put("extraPaths", env.searchPaths()); + workspaceOptions.put("extraPaths", env.paths()); final Map options = new LinkedHashMap<>(); options.put("workspace", workspaceOptions); // NB: Jedi's diagnostics are syntax errors, found as the script changes. diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java index 245a77e..90e22a2 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java @@ -202,7 +202,7 @@ private ApposePythonLanguageServer server( { final ApposePythonLanguageServer s = new ApposePythonLanguageServer(syntax, doc -> new Environment(doc.getText().contains("#env=B") ? "B" - : fallback, null, null, null), jedi, null); + : fallback, null, null), jedi, null); s.connect(new Client()); s.initialize(new InitializeParams()); return s; diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java index 495ccd5..3146bbb 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java @@ -137,7 +137,7 @@ public void testEnvironmentWithoutOne() { final Environment env = plugin.environment(new TextDocumentItem( "untitled:/a.py", "appose-python", 2, "import os\nos.pa")); assertNull(env.interpreter()); - assertTrue(env.searchPaths().isEmpty()); + assertTrue(env.paths().isEmpty()); } @Test @@ -161,7 +161,7 @@ public void testOverlay() throws Exception { assertEquals(Collections.singletonList(Collections.singletonList( "tifffile")), installed); assertEquals(Collections.singletonList("/overlays/blur"), plugin.environment( - doc).searchPaths()); + doc).paths()); } @Test @@ -291,7 +291,7 @@ public void testServerOptions() { assertEquals("{workspace={environmentPath=/env/bin/python, " + "extraPaths=[/overlay]}, " + diagnostics + "}", String.valueOf(spec .initializationOptions(new Environment( - "/env/bin/python", Collections.singletonList("/overlay"), null, null)))); + "/env/bin/python", Collections.singletonList("/overlay"), null)))); // Without a script environment: the server's own. assertEquals("{workspace={extraPaths=[]}, " + diagnostics + "}", String .valueOf(spec.initializationOptions( From 105b7686722701813f1a65ca52f34e5599c3898a Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 9 Oct 2026 13:23:22 -0500 Subject: [PATCH 16/37] Serve plain Python: sessions declare the parameters Jedi and the syntax-only server now read the text the session makes in ApposePythonDialect. The syntax server no longer reads #@ lines: it finds the parameters and task among the names declared, with the types they are annotated with. Co-Authored-By: Claude Opus 5.5 --- .../python/ApposePythonServerPlugin.java | 33 ++++----- .../python/PythonSyntaxLanguageServer.java | 42 ++---------- .../python/completion/PythonSyntax.java | 68 ++++++++++++++----- .../python/ApposePythonServersTest.java | 32 +++++++-- 4 files changed, 96 insertions(+), 79 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java index d5ab8cf..ee4c65b 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java @@ -29,17 +29,13 @@ package org.scijava.plugins.scripting.appose.python; -import java.util.Collections; -import java.util.Map; - import org.eclipse.lsp4j.TextDocumentItem; import org.eclipse.lsp4j.services.LanguageServer; import org.scijava.Priority; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; import org.scijava.code.lsp.LanguageServerService; -import org.scijava.code.lsp.script.ScriptDocument; -import org.scijava.code.lsp.script.TransformingLanguageServer; +import org.scijava.code.lsp.script.ScriptDialect; import org.scijava.plugin.Parameter; import org.scijava.plugin.Plugin; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; @@ -56,8 +52,9 @@ * script declares but its environment lacks are installed for completion only * (see * {@link org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays}). - * Jedi sees the script as plain Python, its parameters declared with the - * types they have when it runs (e.g. images as NumPy arrays). + * Both see the script as plain Python, its parameters declared with the + * types they have when it runs (e.g. images as NumPy arrays; see + * {@link #dialect()}). *

    * Completion never builds a script's environment, and never disturbs its * worker. @@ -86,9 +83,14 @@ public boolean supports(final ScriptLanguage language) { @Override public LanguageServer create(final Environment environment) { final boolean jedi = jediOffered(); - return new ApposePythonLanguageServer(new PythonSyntaxLanguageServer( - servers), this::environment, jedi ? this::jedi : null, - ApposePythonEnvironments.of(workers)); + return new ApposePythonLanguageServer(new PythonSyntaxLanguageServer(), + this::environment, jedi ? this::jedi : null, ApposePythonEnvironments.of( + workers)); + } + + @Override + public ScriptDialect dialect() { + return PythonSyntaxLanguageServer.DIALECT; } /** Whether jedi-language-server is offered. */ @@ -108,16 +110,7 @@ Environment environment(final TextDocumentItem document) { /** Makes a jedi for scripts running in an environment. */ private LanguageServer jedi(final Environment environment) { - return new PythonCompletionServer(new TransformingLanguageServer(servers - .launch("python", environment), text -> document(text, environment)), + return new PythonCompletionServer(servers.launch("python", environment), SIGNATURES); } - - /** The script as plain Python, for jedi. */ - ScriptDocument document(final String text, final Environment environment) { - final Map> params = servers == null ? Collections - .emptyMap() : servers.scriptParameters(text); - return ScriptDocument.of(text, params, PythonSyntaxLanguageServer.DIALECT, - environment); - } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java index bda354f..1b9483b 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java @@ -30,16 +30,12 @@ package org.scijava.plugins.scripting.appose.python; import java.util.ArrayList; -import java.util.Collections; -import java.util.LinkedHashMap; import java.util.List; import java.util.Locale; -import java.util.Map; import org.eclipse.lsp4j.CompletionItem; import org.eclipse.lsp4j.CompletionItemKind; import org.eclipse.lsp4j.CompletionList; -import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.jvm.ScriptLanguageServer; import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax; @@ -48,9 +44,9 @@ /** * An in-process language server for Python scripts run via Appose, from the * syntax alone ({@link PythonSyntax}: tree-sitter): instantly, with no Python - * needed, names in scope, the script's parameters, {@code task}, builtins and - * keywords; and, after a dot, the names used after dots elsewhere in the - * script. (A Python language server, e.g. jedi-language-server, knows more; + * needed, names in scope (including the script's parameters and {@code task}, + * declared as {@link #DIALECT} says), builtins and keywords; and, after a dot, + * the names used after dots elsewhere in the script. (A Python language server, e.g. jedi-language-server, knows more; * see {@link ApposePythonLanguageServer}.) * * @author Gabriel Selzer @@ -59,12 +55,8 @@ public class PythonSyntaxLanguageServer extends ScriptLanguageServer { static final ApposePythonDialect DIALECT = new ApposePythonDialect(); - private final LanguageServerService servers; - - /** @param servers Reads the scripts' parameters. */ - public PythonSyntaxLanguageServer(final LanguageServerService servers) { + public PythonSyntaxLanguageServer() { super(null); - this.servers = servers; } @Override @@ -75,20 +67,7 @@ protected CompletionList complete(final Document doc, final int caret) { { return list(doc, caret, caret, new ArrayList<>()); } - return list(doc, ctx.replaceStart, caret, localCompletions(ctx, parameters( - doc.text()))); - } - - /** - * The script's parameters, with the Python types they have when the script - * runs (null if unknown). - */ - private Map parameters(final String text) { - final Map params = new LinkedHashMap<>(); - if (servers == null) return params; - servers.scriptParameters(text).forEach((name, type) -> params.put(name, - DIALECT.runtimeType(type))); - return params; + return list(doc, ctx.replaceStart, caret, localCompletions(ctx)); } private static CompletionItem item(final String name, @@ -103,7 +82,7 @@ private static CompletionItem item(final String name, /** Completions from the local (syntactic) analysis alone. */ private static List localCompletions( - final PythonSyntax.Context ctx, final Map params) + final PythonSyntax.Context ctx) { final List out = new ArrayList<>(); final String prefix = ctx.prefix; @@ -117,15 +96,8 @@ private static List localCompletions( if (ctx.kind != Kind.NAME) return out; for (final String name : ctx.names) { if (matches(name, prefix)) out.add(item(name, - CompletionItemKind.Variable, null)); - } - for (final Map.Entry p : params.entrySet()) { - if (matches(p.getKey(), prefix)) out.add(item(p.getKey(), - CompletionItemKind.Variable, p.getValue() == null ? "Script parameter" - : p.getValue() + " (script parameter)")); + CompletionItemKind.Variable, ctx.types.get(name))); } - if (matches("task", prefix)) out.add(item("task", - CompletionItemKind.Variable, ApposePythonDialect.TASK_TYPE)); for (final String name : PythonSyntax.BUILTINS) { if (matches(name, prefix)) out.add(item(name, CompletionItemKind.Function, "builtin")); diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java index c70d3e0..14b184b 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java @@ -34,8 +34,10 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.Deque; +import java.util.HashMap; import java.util.LinkedHashSet; import java.util.List; +import java.util.Map; import java.util.Set; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -91,18 +93,25 @@ public static final class Context { /** Names in scope at the caret, innermost first. */ public final Set names; + /** + * The types names in scope are annotated with, as written (e.g. + * {@code numpy.ndarray} for {@code image: numpy.ndarray}), by name. + */ + public final Map types; + /** For {@link Kind#ATTRIBUTE}: attributes used on the same receiver. */ public final Set attributesUsed; private Context(final Kind kind, final String prefix, final int replaceStart, final String receiver, final Set names, - final Set attributesUsed) + final Map types, final Set attributesUsed) { this.kind = kind; this.prefix = prefix; this.replaceStart = replaceStart; this.receiver = receiver; this.names = names; + this.types = types; this.attributesUsed = attributesUsed; } } @@ -177,18 +186,20 @@ else if (before.trim().endsWith(".") && !before.trim().endsWith("..")) { // Parse the script up to the caret, with the partial identifier replaced // by a placeholder, and any open brackets closed: a clean tree there. final String source = textBefore + CARET + closers; - final Set names = kind == Kind.NAME ? namesInScope(source, - textBefore.length()) : new LinkedHashSet<>(); + final Set names = new LinkedHashSet<>(); + final Map types = new HashMap<>(); + if (kind == Kind.NAME) namesInScope(source, textBefore.length(), names, + types); final Set used = kind == Kind.ATTRIBUTE && receiver != null ? attributesUsed(text, receiver, start) : new LinkedHashSet<>(); - return new Context(kind, prefix, start, receiver, names, used); + return new Context(kind, prefix, start, receiver, names, types, used); } // -- Helper methods -- private static Context none(final String prefix, final int start) { return new Context(Kind.NONE, prefix, start, null, new LinkedHashSet<>(), - new LinkedHashSet<>()); + new HashMap<>(), new LinkedHashSet<>()); } /** @@ -272,14 +283,16 @@ private static Set attributesUsed(final String text, return used; } - /** Names in scope at the placeholder, which starts at {@code caret}. */ - private static Set namesInScope(final String source, - final int caret) + /** + * Finds the names in scope at the placeholder, which starts at + * {@code caret}, and the types they are annotated with. + */ + private static void namesInScope(final String source, final int caret, + final Set names, final Map types) { final byte[] bytes = source.getBytes(StandardCharsets.UTF_8); final int caretByte = source.substring(0, caret).getBytes( StandardCharsets.UTF_8).length; - final Set names = new LinkedHashSet<>(); final TSParser parser = new TSParser(); try { parser.setLanguage(new TreeSitterPython()); @@ -305,13 +318,13 @@ private static Set namesInScope(final String source, if (type.endsWith("comprehension") || type.equals( "generator_expression")) { - collectBindings(node, bytes, names, true); + collectBindings(node, bytes, names, types, true); } else if (body != null && !body.isNull() && !type.equals("lambda")) { - collectBindings(body, bytes, names, false); + collectBindings(body, bytes, names, types, false); } } - collectBindings(root, bytes, names, false); + collectBindings(root, bytes, names, types, false); } catch (final RuntimeException | LinkageError exc) { // NB: Native parser unavailable or failed: no names, then. @@ -320,15 +333,16 @@ else if (body != null && !body.isNull() && !type.equals("lambda")) { parser.close(); } names.remove(CARET); - return names; } /** * Collects the names bound by statements within {@code node}, without - * descending into nested scopes (whose own names are bound here, though). + * descending into nested scopes (whose own names are bound here, though); + * and the types annotated names are declared with. */ private static void collectBindings(final TSNode node, final byte[] bytes, - final Set names, final boolean comprehension) + final Set names, final Map types, + final boolean comprehension) { final int n = node.getNamedChildCount(); for (int i = 0; i < n; i++) { @@ -347,6 +361,9 @@ private static void collectBindings(final TSNode node, final byte[] bytes, if (!comprehension) continue; // NB: A nested scope. break; case "assignment": + addType(child, bytes, types); + addTargets(child.getChildByFieldName("left"), bytes, names); + break; case "augmented_assignment": case "for_statement": case "for_in_clause": @@ -366,10 +383,21 @@ private static void collectBindings(final TSNode node, final byte[] bytes, continue; default: } - collectBindings(child, bytes, names, comprehension); + collectBindings(child, bytes, names, types, comprehension); } } + /** Adds the type of an annotated assignment, e.g. {@code x: int = 1}. */ + private static void addType(final TSNode assignment, final byte[] bytes, + final Map types) + { + final TSNode left = assignment.getChildByFieldName("left"); + final TSNode type = assignment.getChildByFieldName("type"); + if (left == null || left.isNull() || !left.getType().equals( + "identifier") || type == null || type.isNull()) return; + types.putIfAbsent(text(left, bytes), text(type, bytes)); + } + /** Adds the names an import statement binds. */ private static void addImported(final TSNode statement, final byte[] bytes, final Set names, final boolean plainImport) @@ -428,8 +456,12 @@ private static void addIdentifier(final TSNode node, final byte[] bytes, { if (node == null || node.isNull() || !node.getType().equals( "identifier")) return; - names.add(new String(bytes, node.getStartByte(), node.getEndByte() - node - .getStartByte(), StandardCharsets.UTF_8)); + names.add(text(node, bytes)); + } + + private static String text(final TSNode node, final byte[] bytes) { + return new String(bytes, node.getStartByte(), node.getEndByte() - node + .getStartByte(), StandardCharsets.UTF_8); } private static boolean isIdentifierPart(final char c) { diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java index 3146bbb..7b6e0e8 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java @@ -52,6 +52,7 @@ import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; import org.scijava.code.lsp.ScriptSession; +import org.scijava.code.lsp.script.ScriptDocument; import org.scijava.plugin.PluginService; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; @@ -115,6 +116,10 @@ public void testLocalCompletions() { assertTrue(complete(header + "x = si").contains("sigma")); assertTrue(complete(header + "pri").contains("print")); assertTrue(complete(header + "ta").contains("task")); + // Declared with the types they have when the script runs. + assertEquals("float", detail(header + "x = si", "sigma")); + assertEquals("numpy.ndarray", detail(header + "x = ima", "image")); + assertEquals("appose.python_worker.Task", detail(header + "ta", "task")); // Nothing on parameter lines, in strings or comments. assertTrue(complete("#@ String na").isEmpty()); assertTrue(complete("s = 'pri").isEmpty()); @@ -124,9 +129,9 @@ public void testLocalCompletions() { public void testScriptAsPython() { final String code = "#@ net.imglib2.img.Img image\n#@ double sigma\n" + "image.sha"; - // The script's parameters are typed in a preamble, so the server knows + // The script's parameters are typed in a preamble, so the servers know // them. - assertTrue(plugin.document(code, Environment.NONE).text().startsWith( + assertTrue(document(code).text().startsWith( "import appose.python_worker; import numpy; image: numpy.ndarray; " + "sigma: float; task: appose.python_worker.Task\n")); } @@ -303,15 +308,30 @@ public void testServerOptions() { // -- Helper methods -- + /** The script as plain Python, as the plugin's servers see it. */ + private ScriptDocument document(final String code) { + return ScriptDocument.of(code, context.service(LanguageServerService.class) + .scriptParameters(code), plugin.dialect()); + } + /** Completes with the syntax-only server. */ private List complete(final String code) { + return items(code).stream().map(CompletionItem::getLabel).collect( + Collectors.toList()); + } + + /** The detail of a completion with the syntax-only server, or null. */ + private String detail(final String code, final String label) { + return items(code).stream().filter(i -> i.getLabel().equals(label)) + .findFirst().map(CompletionItem::getDetail).orElse(null); + } + + private List items(final String code) { try (ScriptSession session = ScriptSession.owning( - new PythonSyntaxLanguageServer(context.service( - LanguageServerService.class)), language)) + new PythonSyntaxLanguageServer(), language, this::document)) { return session.completion(code, code.length()).get(10, - TimeUnit.SECONDS).getItems().stream().map(CompletionItem::getLabel) - .collect(Collectors.toList()); + TimeUnit.SECONDS).getItems(); } catch (final Exception exc) { throw new RuntimeException(exc); From d7528038b6623af0f02d6a01bfd34a65c9ef96ac Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 9 Oct 2026 13:28:29 -0500 Subject: [PATCH 17/37] Name what the declarations import, so it is not completed E.g. a package imported only to declare a parameter's type is no longer offered (see ScriptDialect#imported). Co-Authored-By: Claude Opus 5.5 --- .../appose/python/completion/ApposePythonDialect.java | 7 +++++++ .../scripting/appose/python/ApposePythonServersTest.java | 5 +++++ 2 files changed, 12 insertions(+) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index fd0c2d5..cb9b551 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -94,6 +94,13 @@ public List imports(final Collection types) { return imports; } + @Override + public String imported(final String statement) { + // NB: "import a.b" binds a. + return statement.startsWith("import ") ? statement.substring(7).split( + "[.\\s]")[0] : null; + } + @Override public List extraDeclarations() { return Collections.singletonList(declare("task", TASK_TYPE)); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java index 7b6e0e8..13ccdae 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java @@ -30,6 +30,7 @@ package org.scijava.plugins.scripting.appose.python; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; @@ -120,6 +121,10 @@ public void testLocalCompletions() { assertEquals("float", detail(header + "x = si", "sigma")); assertEquals("numpy.ndarray", detail(header + "x = ima", "image")); assertEquals("appose.python_worker.Task", detail(header + "ta", "task")); + // Modules imported only to declare those: not completed. + assertFalse(complete(header + "nu").contains("numpy")); + assertFalse(complete(header + "ap").contains("appose")); + assertTrue(complete(header + "import numpy\nnu").contains("numpy")); // Nothing on parameter lines, in strings or comments. assertTrue(complete("#@ String na").isEmpty()); assertTrue(complete("s = 'pri").isEmpty()); From 35543a2027bef5b61a03926d54f6afcaffffc3c7 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 9 Oct 2026 15:50:51 -0500 Subject: [PATCH 18/37] Follow ScriptDialect and ScriptDocument into code-lsp's core package Co-Authored-By: Claude Opus 5.5 --- .../scripting/appose/python/ApposePythonServerPlugin.java | 2 +- .../scripting/appose/python/completion/ApposePythonDialect.java | 2 +- .../scripting/appose/python/ApposePythonServersTest.java | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java index ee4c65b..2486f7b 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java @@ -35,7 +35,7 @@ import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; import org.scijava.code.lsp.LanguageServerService; -import org.scijava.code.lsp.script.ScriptDialect; +import org.scijava.code.lsp.ScriptDialect; import org.scijava.plugin.Parameter; import org.scijava.plugin.Plugin; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index cb9b551..b512c54 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -41,7 +41,7 @@ import net.imglib2.RandomAccessibleInterval; import org.apposed.appose.NDArray; -import org.scijava.code.lsp.script.ScriptDialect; +import org.scijava.code.lsp.ScriptDialect; /** * How Python scripts run via Appose see their script parameters: as the diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java index 13ccdae..e719589 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java @@ -53,7 +53,7 @@ import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; import org.scijava.code.lsp.ScriptSession; -import org.scijava.code.lsp.script.ScriptDocument; +import org.scijava.code.lsp.ScriptDocument; import org.scijava.plugin.PluginService; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; From 54899bce907237aae17a1f4355f89e55399198ef Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Mon, 12 Oct 2026 13:09:01 -0500 Subject: [PATCH 19/37] Offer jedi-language-server as a ServerProcessPlugin code-lsp now runs server processes from ServerProcessPlugins, replacing its launch package. JediLanguageServerSpec becomes JediServerProcessPlugin, a plugin that starts jedi-language-server in the tooling environment; ApposePythonLspServerPlugin, which offered the spec, is gone, and ApposePythonEnvironments gives just the tooling Python. Co-Authored-By: Claude Opus 5.5 --- .../python/ApposePythonEnvironments.java | 23 +++---- .../python/ApposePythonLspServerPlugin.java | 60 ------------------- .../python/ApposePythonServerPlugin.java | 2 +- ...Spec.java => JediServerProcessPlugin.java} | 40 +++++++++---- .../python/completion/PythonSyntax.java | 3 +- ...ApposePythonCompletionIntegrationTest.java | 2 +- .../python/ApposePythonServersTest.java | 3 +- 7 files changed, 43 insertions(+), 90 deletions(-) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java rename src/main/java/org/scijava/plugins/scripting/appose/python/{completion/JediLanguageServerSpec.java => JediServerProcessPlugin.java} (78%) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java index c3568d2..bc6fef3 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java @@ -53,7 +53,6 @@ import org.scijava.plugins.scripting.appose.python._internal.BuildListener; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; -import org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec; /** * Where Python scripts run via Appose, as their language servers need to @@ -356,21 +355,17 @@ private static String findPython(final List dirs) { } /** - * How to start jedi-language-server: with the Python of the tooling - * environment, built first if needed (but starting no worker). + * Gets the Python of the tooling environment, in which jedi-language-server + * runs (see {@link JediServerProcessPlugin}): built first if needed (but + * starting no worker). */ - static JediLanguageServerSpec serverSpec( - final ResidentWorkerService workers) + static String toolingPython(final ResidentWorkerService workers) + throws Exception { - final File envs = new File(Environments.apposeEnvsDir()); - return new JediLanguageServerSpec(() -> { - final String python = pythonExecutable(toolingWorker(workers) - .environment().get()); - if (python == null) throw new IOException("No Python in " + - TOOLING_ENV); - return python; - }, new File(envs, TOOLING_ENV + "-workspace"), new File(envs, TOOLING_ENV + - "-server.log")); + final String python = pythonExecutable(toolingWorker(workers).environment() + .get()); + if (python == null) throw new IOException("No Python in " + TOOLING_ENV); + return python; } /** Gets the tooling environment's worker, building it first if needed. */ diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java deleted file mode 100644 index 2ded0ec..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLspServerPlugin.java +++ /dev/null @@ -1,60 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python; - -import org.scijava.code.lsp.launch.LspServerPlugin; -import org.scijava.code.lsp.launch.LspServerSpec; -import org.scijava.plugin.Parameter; -import org.scijava.plugin.Plugin; -import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; - -/** - * Offers jedi-language-server, from the completion tooling environment, as - * the Python language server: for Python scripts run via Appose, and for any - * other completer wanting one (e.g. Jython's). - * - * @author Gabriel Selzer - */ -@Plugin(type = LspServerPlugin.class, name = "jedi-language-server") -public class ApposePythonLspServerPlugin implements LspServerPlugin { - - @Parameter - private ResidentWorkerService workers; - - @Override - public boolean supports(final String languageId) { - return "python".equals(languageId); - } - - @Override - public LspServerSpec spec() { - return ApposePythonEnvironments.serverSpec(workers); - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java index 2486f7b..6cad129 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java @@ -45,7 +45,7 @@ /** * Offers the language server of Python scripts run via Appose (see * {@link ApposePythonLanguageServer}): the syntax-only server, and, if offered - * (see {@link ApposePythonLspServerPlugin}), jedi-language-server: members of + * (see {@link JediServerProcessPlugin}), jedi-language-server: members of * objects and modules, signatures, documentation, hover and syntax errors. * Once a script's environment is built (by running the script), jedi * analyzes the script against it, so its packages are known; packages the diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerProcessPlugin.java similarity index 78% rename from src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java rename to src/main/java/org/scijava/plugins/scripting/appose/python/JediServerProcessPlugin.java index f45ca2f..299e8fa 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediLanguageServerSpec.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerProcessPlugin.java @@ -27,7 +27,7 @@ * #L% */ -package org.scijava.plugins.scripting.appose.python.completion; +package org.scijava.plugins.scripting.appose.python; import java.io.File; import java.io.IOException; @@ -35,18 +35,24 @@ import java.util.Map; import java.util.concurrent.Callable; +import org.apposed.appose.util.Environments; import org.scijava.code.lsp.Environment; -import org.scijava.code.lsp.launch.LspConnection; -import org.scijava.code.lsp.launch.LspServerSpec; +import org.scijava.code.lsp.ServerProcessPlugin; +import org.scijava.plugin.Parameter; +import org.scijava.plugin.Plugin; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; /** - * Starts jedi-language-server, with the Python of a tooling environment, to - * analyze scripts against their environment's Python (if known) and extra - * search paths (e.g. a dependency overlay). + * Offers jedi-language-server, for {@code python}: run with the Python of a + * tooling environment (see {@link ApposePythonEnvironments}), to analyze + * scripts against their environment's Python (if known) and extra search + * paths (e.g. a dependency overlay). Python scripts run via Appose use it, + * and so do Jython's. * * @author Gabriel Selzer */ -public class JediLanguageServerSpec implements LspServerSpec { +@Plugin(type = ServerProcessPlugin.class, name = "jedi-language-server") +public class JediServerProcessPlugin implements ServerProcessPlugin { /** * Runs jedi-language-server, exiting once the given process (this JVM) is @@ -72,18 +78,30 @@ public class JediLanguageServerSpec implements LspServerSpec { /** How large the error log may grow before it is started anew, in bytes. */ private static final long MAX_LOG = 5 * 1024 * 1024; + @Parameter + private ResidentWorkerService workers; + private final Callable python; private final File workspace; private final File log; + /** Runs the server in the tooling environment, building it if needed. */ + public JediServerProcessPlugin() { + final File envs = new File(Environments.apposeEnvsDir()); + python = () -> ApposePythonEnvironments.toolingPython(workers); + workspace = new File(envs, ApposePythonEnvironments.TOOLING_ENV + + "-workspace"); + log = new File(envs, ApposePythonEnvironments.TOOLING_ENV + "-server.log"); + } + /** * @param python Gets the Python executable of the environment * jedi-language-server is installed in (building it, if needed). * @param workspace An (empty) directory to serve as workspace root. * @param log Where to append the server's error output. */ - public JediLanguageServerSpec(final Callable python, - final File workspace, final File log) + JediServerProcessPlugin(final Callable python, final File workspace, + final File log) { this.python = python; this.workspace = workspace; @@ -96,7 +114,7 @@ public String languageId() { } @Override - public LspConnection start(final Environment env) + public Process start(final Environment env) throws IOException { final String exe; @@ -116,7 +134,7 @@ public LspConnection start(final Environment env) final ProcessBuilder pb = new ProcessBuilder(exe, "-c", LAUNCHER, String .valueOf(ProcessHandle.current().pid())); pb.redirectError(ProcessBuilder.Redirect.appendTo(log)); - return LspConnection.of(pb.start()); + return pb.start(); } @Override diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java index 14b184b..e79cbbd 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java @@ -53,7 +53,8 @@ *

    * This needs no Python, so it works instantly, before (or without) any Appose * environment. It knows nothing of types, though: members of objects come - * from the language server (see {@link org.scijava.code.lsp.PythonLspClient}). + * from jedi-language-server (see + * {@link org.scijava.plugins.scripting.appose.python.ApposePythonLanguageServer}). *

    * * @author Gabriel Selzer diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java index cbb2899..e029099 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java @@ -212,7 +212,7 @@ public void testServerOutlivesNoParent() throws Exception { final Process gone = new ProcessBuilder(python, "-c", "pass").start(); gone.waitFor(); final Process process = new ProcessBuilder(python, "-c", - org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec.LAUNCHER, + JediServerProcessPlugin.LAUNCHER, String.valueOf(gone.pid())).redirectErrorStream(true).start(); final boolean exited = process.waitFor(30, TimeUnit.SECONDS); if (!exited) process.destroyForcibly(); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java index e719589..ecd5cea 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java @@ -58,7 +58,6 @@ import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; -import org.scijava.plugins.scripting.appose.python.completion.JediLanguageServerSpec; import org.scijava.script.ScriptLanguage; import org.scijava.script.ScriptService; @@ -293,7 +292,7 @@ public void testPythonTypes() { @Test public void testServerOptions() { - final JediLanguageServerSpec spec = new JediLanguageServerSpec(() -> null, + final JediServerProcessPlugin spec = new JediServerProcessPlugin(() -> null, new File("workspace"), new File("server.log")); assertEquals("python", spec.languageId()); final String diagnostics = "diagnostics={enable=true, didOpen=true, " + From 2b09eb242df7cb453eda5526dec187fb577fbcbe Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Mon, 12 Oct 2026 13:10:03 -0500 Subject: [PATCH 20/37] Gather the completion code into the completion package The language server plugin, its server, the environments it analyzes scripts against, jedi-language-server's plugin and the syntax-only server join the rest of the completion code (and their tests, the rest of its tests). The python package keeps what runs scripts. ScriptEnvironment, which both resolve scripts' environments with, is public now; so is what completion uses of it. Co-Authored-By: Claude Opus 5.5 --- .../appose/python/ScriptEnvironment.java | 23 +++++++++++-------- .../ApposePythonEnvironments.java | 6 ++--- .../ApposePythonLanguageServer.java | 2 +- .../ApposePythonServerPlugin.java | 4 ++-- .../JediServerProcessPlugin.java | 2 +- .../python/completion/PythonSyntax.java | 3 +-- .../PythonSyntaxLanguageServer.java | 4 +--- ...ApposePythonCompletionIntegrationTest.java | 2 +- .../ApposePythonLanguageServerTest.java | 2 +- .../ApposePythonServersTest.java | 5 ++-- 10 files changed, 27 insertions(+), 26 deletions(-) rename src/main/java/org/scijava/plugins/scripting/appose/python/{ => completion}/ApposePythonEnvironments.java (98%) rename src/main/java/org/scijava/plugins/scripting/appose/python/{ => completion}/ApposePythonLanguageServer.java (99%) rename src/main/java/org/scijava/plugins/scripting/appose/python/{ => completion}/ApposePythonServerPlugin.java (96%) rename src/main/java/org/scijava/plugins/scripting/appose/python/{ => completion}/JediServerProcessPlugin.java (98%) rename src/main/java/org/scijava/plugins/scripting/appose/python/{ => completion}/PythonSyntaxLanguageServer.java (95%) rename src/test/java/org/scijava/plugins/scripting/appose/python/{ => completion}/ApposePythonCompletionIntegrationTest.java (99%) rename src/test/java/org/scijava/plugins/scripting/appose/python/{ => completion}/ApposePythonLanguageServerTest.java (99%) rename src/test/java/org/scijava/plugins/scripting/appose/python/{ => completion}/ApposePythonServersTest.java (98%) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java index 11e75a5..c620bdb 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java @@ -49,11 +49,13 @@ * directive. *

    * Both running a script ({@link ApposePythonScriptEngine}) and completing code - * in it ({@link ApposePythonServerPlugin}) resolve its environment here, so - * that they agree on the environment's name and configuration. + * in it ({@link + * org.scijava.plugins.scripting.appose.python.completion.ApposePythonServerPlugin}) + * resolve its environment here, so that they agree on the environment's name + * and configuration. *

    */ -final class ScriptEnvironment { +public final class ScriptEnvironment { /** The {@code #@script(...)} directive. */ private static final Pattern SCRIPT_DIRECTIVE = Pattern.compile( @@ -64,7 +66,7 @@ final class ScriptEnvironment { "(\\w+)\\s*=\\s*(?:\"([^\"]*)\"|'([^']*)'|([^,\\s)]+))"); /** Name of the environment. */ - final String name; + public final String name; /** The environment's configuration, e.g. the contents of its file. */ final String content; @@ -94,8 +96,9 @@ private ScriptEnvironment(final String name, final String content, * malformed, given twice, or naming a missing file. * @throws IOException If the environment file cannot be read. */ - static ScriptEnvironment of(final String script, final String scriptPath, - final String envRef, final String scheme) throws IOException + public static ScriptEnvironment of(final String script, + final String scriptPath, final String envRef, final String scheme) + throws IOException { final String metadata = InlineMetadata.extract(script); if (metadata != null) { @@ -134,7 +137,9 @@ static ScriptEnvironment of(final String script, final String scriptPath, * {@link InlineMetadata#extract}). * @param scriptPath The script's file path, or null if unsaved. */ - static String inlineName(final String metadata, final String scriptPath) { + public static String inlineName(final String metadata, + final String scriptPath) + { // Note: Name the environment after the script, as for environment // files, so that editing the metadata updates the environment and // replaces its worker, rather than leaving the old one running. @@ -147,7 +152,7 @@ static String inlineName(final String metadata, final String scriptPath) { * Gets the attributes of a script's {@code #@script(...)} directive, e.g. * {@code env} and {@code scheme}, without fully parsing its parameters. */ - static Map directive(final String script) { + public static Map directive(final String script) { final Map attrs = new HashMap<>(); final Matcher m = SCRIPT_DIRECTIVE.matcher(script); if (!m.find()) return attrs; @@ -165,7 +170,7 @@ static Map directive(final String script) { * org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService} * keeps the environment: changes to it replace the environment's worker. */ - String config() { + public String config() { return scheme + "\n" + content; } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java similarity index 98% rename from src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java rename to src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java index bc6fef3..2f83048 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java @@ -27,7 +27,7 @@ * #L% */ -package org.scijava.plugins.scripting.appose.python; +package org.scijava.plugins.scripting.appose.python.completion; import java.io.File; import java.io.IOException; @@ -48,11 +48,11 @@ import org.apposed.appose.Environment; import org.apposed.appose.Service; import org.apposed.appose.util.Environments; +import org.scijava.plugins.scripting.appose.python.ScriptEnvironment; import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; import org.scijava.plugins.scripting.appose.python._internal.BuildListener; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; -import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; /** * Where Python scripts run via Appose, as their language servers need to @@ -227,7 +227,7 @@ private static String installOverlay(final ResidentWorkerService workers, inputs.put("overlay", dir.getAbsolutePath()); try { final Service.Task task = toolingWorker(workers).task(resource( - "completion/overlay.py"), inputs); + "overlay.py"), inputs); task.waitFor(); final Object overlay = task.outputs.get("path"); return overlay instanceof String ? (String) overlay : null; diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java similarity index 99% rename from src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServer.java rename to src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java index 74bfe77..f37d0c1 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java @@ -27,7 +27,7 @@ * #L% */ -package org.scijava.plugins.scripting.appose.python; +package org.scijava.plugins.scripting.appose.python.completion; import java.util.ArrayList; import java.util.Collections; diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java similarity index 96% rename from src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java rename to src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java index 6cad129..7e0ed83 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java @@ -27,7 +27,7 @@ * #L% */ -package org.scijava.plugins.scripting.appose.python; +package org.scijava.plugins.scripting.appose.python.completion; import org.eclipse.lsp4j.TextDocumentItem; import org.eclipse.lsp4j.services.LanguageServer; @@ -38,8 +38,8 @@ import org.scijava.code.lsp.ScriptDialect; import org.scijava.plugin.Parameter; import org.scijava.plugin.Plugin; +import org.scijava.plugins.scripting.appose.python.ApposePythonScriptLanguage; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; -import org.scijava.plugins.scripting.appose.python.completion.PythonCompletionServer; import org.scijava.script.ScriptLanguage; /** diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerProcessPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediServerProcessPlugin.java similarity index 98% rename from src/main/java/org/scijava/plugins/scripting/appose/python/JediServerProcessPlugin.java rename to src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediServerProcessPlugin.java index 299e8fa..910ce9b 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/JediServerProcessPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediServerProcessPlugin.java @@ -27,7 +27,7 @@ * #L% */ -package org.scijava.plugins.scripting.appose.python; +package org.scijava.plugins.scripting.appose.python.completion; import java.io.File; import java.io.IOException; diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java index e79cbbd..b151659 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java @@ -53,8 +53,7 @@ *

    * This needs no Python, so it works instantly, before (or without) any Appose * environment. It knows nothing of types, though: members of objects come - * from jedi-language-server (see - * {@link org.scijava.plugins.scripting.appose.python.ApposePythonLanguageServer}). + * from jedi-language-server (see {@link ApposePythonLanguageServer}). *

    * * @author Gabriel Selzer diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java similarity index 95% rename from src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java rename to src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java index 1b9483b..09d2f86 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/PythonSyntaxLanguageServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java @@ -27,7 +27,7 @@ * #L% */ -package org.scijava.plugins.scripting.appose.python; +package org.scijava.plugins.scripting.appose.python.completion; import java.util.ArrayList; import java.util.List; @@ -37,8 +37,6 @@ import org.eclipse.lsp4j.CompletionItemKind; import org.eclipse.lsp4j.CompletionList; import org.scijava.code.lsp.jvm.ScriptLanguageServer; -import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; -import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax; import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Kind; /** diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java similarity index 99% rename from src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java rename to src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java index e029099..1a36054 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java @@ -27,7 +27,7 @@ * #L% */ -package org.scijava.plugins.scripting.appose.python; +package org.scijava.plugins.scripting.appose.python.completion; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java similarity index 99% rename from src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java rename to src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java index 90e22a2..89715b6 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonLanguageServerTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java @@ -27,7 +27,7 @@ * #L% */ -package org.scijava.plugins.scripting.appose.python; +package org.scijava.plugins.scripting.appose.python.completion; import static org.junit.Assert.assertEquals; diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java similarity index 98% rename from src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java rename to src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java index ecd5cea..6562ce9 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java @@ -27,7 +27,7 @@ * #L% */ -package org.scijava.plugins.scripting.appose.python; +package org.scijava.plugins.scripting.appose.python.completion; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; @@ -55,9 +55,8 @@ import org.scijava.code.lsp.ScriptSession; import org.scijava.code.lsp.ScriptDocument; import org.scijava.plugin.PluginService; +import org.scijava.plugins.scripting.appose.python.ScriptEnvironment; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; -import org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect; -import org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays; import org.scijava.script.ScriptLanguage; import org.scijava.script.ScriptService; From 342671d019adb67611f507a6619d529eeebd2c5e Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Mon, 12 Oct 2026 13:41:06 -0500 Subject: [PATCH 21/37] Fold PythonSyntax into PythonSyntaxLanguageServer The syntax-only server was its analysis's only user, so the two are one class now: the analysis (analyze, Context, Kind) is package-private, for the tests. The dialect is the plugin's, so ApposePythonServerPlugin keeps it. Co-Authored-By: Claude Opus 5.5 --- .../completion/ApposePythonServerPlugin.java | 7 +- .../python/completion/PythonSyntax.java | 475 ------------------ .../PythonSyntaxLanguageServer.java | 454 ++++++++++++++++- ...va => PythonSyntaxLanguageServerTest.java} | 42 +- 4 files changed, 464 insertions(+), 514 deletions(-) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java rename src/test/java/org/scijava/plugins/scripting/appose/python/completion/{PythonSyntaxTest.java => PythonSyntaxLanguageServerTest.java} (74%) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java index 7e0ed83..3ae5edb 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java @@ -50,8 +50,7 @@ * Once a script's environment is built (by running the script), jedi * analyzes the script against it, so its packages are known; packages the * script declares but its environment lacks are installed for completion only - * (see - * {@link org.scijava.plugins.scripting.appose.python.completion.DependencyOverlays}). + * (see {@link DependencyOverlays}). * Both see the script as plain Python, its parameters declared with the * types they have when it runs (e.g. images as NumPy arrays; see * {@link #dialect()}). @@ -69,6 +68,8 @@ public class ApposePythonServerPlugin implements LanguageServerPlugin { /** How many completions get signatures, once a word is begun. */ private static final int SIGNATURES = 30; + private static final ApposePythonDialect DIALECT = new ApposePythonDialect(); + @Parameter private LanguageServerService servers; @@ -90,7 +91,7 @@ public LanguageServer create(final Environment environment) { @Override public ScriptDialect dialect() { - return PythonSyntaxLanguageServer.DIALECT; + return DIALECT; } /** Whether jedi-language-server is offered. */ diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java deleted file mode 100644 index b151659..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntax.java +++ /dev/null @@ -1,475 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.nio.charset.StandardCharsets; -import java.util.ArrayDeque; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Deque; -import java.util.HashMap; -import java.util.LinkedHashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.regex.Matcher; -import java.util.regex.Pattern; - -import org.treesitter.TSNode; -import org.treesitter.TSParser; -import org.treesitter.TSTree; -import org.treesitter.TreeSitterPython; - -/** - * Syntactic analysis of a Python script at the caret, using tree-sitter: what - * kind of completion the caret calls for, and which names are in scope there. - *

    - * This needs no Python, so it works instantly, before (or without) any Appose - * environment. It knows nothing of types, though: members of objects come - * from jedi-language-server (see {@link ApposePythonLanguageServer}). - *

    - * - * @author Gabriel Selzer - */ -public final class PythonSyntax { - - /** Identifier marking the caret position in the parsed source. */ - static final String CARET = "__scijava_caret__"; - - /** What the caret calls for. */ - public enum Kind { - /** Nothing: the caret is in a comment, string or directive. */ - NONE, - /** A name, e.g. a variable, function or keyword. */ - NAME, - /** An attribute of the expression before a dot. */ - ATTRIBUTE, - /** A module or name in an import statement. */ - IMPORT - } - - /** The completion context at the caret. */ - public static final class Context { - - /** What the caret calls for. */ - public final Kind kind; - - /** The partial identifier being typed (possibly empty). */ - public final String prefix; - - /** Offset where the partial identifier starts. */ - public final int replaceStart; - - /** For {@link Kind#ATTRIBUTE}: the source of the expression, or null. */ - public final String receiver; - - /** Names in scope at the caret, innermost first. */ - public final Set names; - - /** - * The types names in scope are annotated with, as written (e.g. - * {@code numpy.ndarray} for {@code image: numpy.ndarray}), by name. - */ - public final Map types; - - /** For {@link Kind#ATTRIBUTE}: attributes used on the same receiver. */ - public final Set attributesUsed; - - private Context(final Kind kind, final String prefix, - final int replaceStart, final String receiver, final Set names, - final Map types, final Set attributesUsed) - { - this.kind = kind; - this.prefix = prefix; - this.replaceStart = replaceStart; - this.receiver = receiver; - this.names = names; - this.types = types; - this.attributesUsed = attributesUsed; - } - } - - /** Python keywords, including soft keywords. */ - public static final List KEYWORDS = Arrays.asList("False", "None", - "True", "and", "as", "assert", "async", "await", "break", "case", "class", - "continue", "def", "del", "elif", "else", "except", "finally", "for", - "from", "global", "if", "import", "in", "is", "lambda", "match", - "nonlocal", "not", "or", "pass", "raise", "return", "try", "while", - "with", "yield"); - - /** Python's built-in functions, types and constants. */ - public static final List BUILTINS = Arrays.asList("abs", "aiter", - "all", "anext", "any", "ascii", "bin", "bool", "breakpoint", "bytearray", - "bytes", "callable", "chr", "classmethod", "compile", "complex", - "delattr", "dict", "dir", "divmod", "enumerate", "eval", "exec", - "filter", "float", "format", "frozenset", "getattr", "globals", - "hasattr", "hash", "help", "hex", "id", "input", "int", "isinstance", - "issubclass", "iter", "len", "list", "locals", "map", "max", - "memoryview", "min", "next", "object", "oct", "open", "ord", "pow", - "print", "property", "range", "repr", "reversed", "round", "set", - "setattr", "slice", "sorted", "staticmethod", "str", "sum", "super", - "tuple", "type", "vars", "zip", "__name__", "__file__", "Ellipsis", - "NotImplemented", "Exception", "BaseException", "ValueError", - "TypeError", "KeyError", "IndexError", "RuntimeError", "AttributeError", - "ImportError", "OSError", "FileNotFoundError", "StopIteration", - "NotImplementedError", "ZeroDivisionError"); - - private static final Pattern IMPORT_LINE = Pattern.compile( - "^\\s*(from\\s+\\S*(\\s+import\\b.*)?|import\\s.*)$"); - - /** Node types that open a new scope. */ - private static final Set SCOPES = new LinkedHashSet<>(Arrays.asList( - "function_definition", "class_definition", "lambda", "list_comprehension", - "set_comprehension", "dictionary_comprehension", "generator_expression")); - - private PythonSyntax() { - // NB: Prevent instantiation of utility class. - } - - /** Analyzes the given script at the given caret offset. */ - public static Context analyze(final String text, final int caret) { - final int lineStart = text.lastIndexOf('\n', caret - 1) + 1; - final String line = text.substring(lineStart, caret); - int start = caret; - while (start > lineStart && isIdentifierPart(text.charAt(start - 1))) { - start--; - } - final String prefix = text.substring(start, caret); - final String before = text.substring(lineStart, start); - final String textBefore = text.substring(0, start); - final String closers = closers(textBefore); - if (closers == null || line.trim().startsWith("#")) return none(prefix, - start); - if (!prefix.isEmpty() && Character.isDigit(prefix.charAt(0))) { - return none(prefix, start); - } - - final Kind kind; - String receiver = null; - if (IMPORT_LINE.matcher(before + prefix).matches()) kind = Kind.IMPORT; - else if (before.trim().endsWith(".") && !before.trim().endsWith("..")) { - receiver = receiver(before.substring(0, before.lastIndexOf('.'))); - if (receiver != null && receiver.matches("\\d+")) { - return none(prefix, start); // a number, e.g. "1." - } - kind = Kind.ATTRIBUTE; - } - else kind = Kind.NAME; - - // Parse the script up to the caret, with the partial identifier replaced - // by a placeholder, and any open brackets closed: a clean tree there. - final String source = textBefore + CARET + closers; - final Set names = new LinkedHashSet<>(); - final Map types = new HashMap<>(); - if (kind == Kind.NAME) namesInScope(source, textBefore.length(), names, - types); - final Set used = kind == Kind.ATTRIBUTE && receiver != null - ? attributesUsed(text, receiver, start) : new LinkedHashSet<>(); - return new Context(kind, prefix, start, receiver, names, types, used); - } - - // -- Helper methods -- - - private static Context none(final String prefix, final int start) { - return new Context(Kind.NONE, prefix, start, null, new LinkedHashSet<>(), - new HashMap<>(), new LinkedHashSet<>()); - } - - /** - * Gets the closing brackets for those left open at the end of - * {@code text}, or null if the text ends inside a string or comment. - */ - static String closers(final String text) { - final Deque open = new ArrayDeque<>(); - int i = 0; - final int n = text.length(); - while (i < n) { - final char c = text.charAt(i); - if (c == '#') { - final int end = text.indexOf('\n', i); - if (end < 0) return null; - i = end + 1; - } - else if (c == '\'' || c == '"') { - final boolean triple = text.startsWith(String.valueOf(c) + c + c, i); - final String quote = triple ? String.valueOf(c) + c + c : String - .valueOf(c); - int j = i + quote.length(); - while (true) { - if (j >= n) return null; - final char d = text.charAt(j); - if (d == '\\') j += 2; - else if (!triple && d == '\n') return null; // unterminated line - else if (text.startsWith(quote, j)) break; - else j++; - } - i = j + quote.length(); - } - else { - if (c == '(') open.push(')'); - else if (c == '[') open.push(']'); - else if (c == '{') open.push('}'); - else if ((c == ')' || c == ']' || c == '}') && !open.isEmpty() && open - .peek() == c) open.pop(); - i++; - } - } - final StringBuilder sb = new StringBuilder(); - for (final char c : open) sb.append(c); - return sb.toString(); - } - - /** - * Gets the source of the expression ending the given text, e.g. - * {@code np.random} in {@code x = np.random}, or null if there is none. - */ - static String receiver(final String text) { - int i = text.length(); - while (i > 0 && Character.isWhitespace(text.charAt(i - 1))) i--; - final int end = i; - int depth = 0; - while (i > 0) { - final char c = text.charAt(i - 1); - if (c == ')' || c == ']' || c == '}') depth++; - else if (c == '(' || c == '[' || c == '{') { - if (depth == 0) break; - depth--; - } - else if (depth == 0 && !isIdentifierPart(c) && c != '.') break; - i--; - } - final String receiver = text.substring(i, end).trim(); - return receiver.isEmpty() ? null : receiver; - } - - /** Attributes used on {@code receiver} anywhere in the script. */ - private static Set attributesUsed(final String text, - final String receiver, final int exclude) - { - final Set used = new LinkedHashSet<>(); - // NB: Only spaces around the dot: "a.\nprint" is not an attribute. - final Matcher m = Pattern.compile("(? names, final Map types) - { - final byte[] bytes = source.getBytes(StandardCharsets.UTF_8); - final int caretByte = source.substring(0, caret).getBytes( - StandardCharsets.UTF_8).length; - final TSParser parser = new TSParser(); - try { - parser.setLanguage(new TreeSitterPython()); - final TSTree tree = parser.parseString(null, source); - final TSNode root = tree.getRootNode(); - TSNode node = root.getNamedDescendantForByteRange(caretByte, caretByte + - CARET.length()); - // Walk outward through the enclosing scopes, innermost first. - boolean inFunction = false; - for (; node != null && !node.isNull(); node = node.getParent()) { - final String type = node.getType(); - if (!SCOPES.contains(type)) continue; - // NB: Class bodies are not visible from their methods. - if (type.equals("class_definition") && inFunction) continue; - if (type.equals("function_definition") || type.equals("lambda")) { - inFunction = true; - } - final TSNode params = node.getChildByFieldName("parameters"); - if (params != null && !params.isNull()) { - addTargets(params, bytes, names); - } - final TSNode body = node.getChildByFieldName("body"); - if (type.endsWith("comprehension") || type.equals( - "generator_expression")) - { - collectBindings(node, bytes, names, types, true); - } - else if (body != null && !body.isNull() && !type.equals("lambda")) { - collectBindings(body, bytes, names, types, false); - } - } - collectBindings(root, bytes, names, types, false); - } - catch (final RuntimeException | LinkageError exc) { - // NB: Native parser unavailable or failed: no names, then. - } - finally { - parser.close(); - } - names.remove(CARET); - } - - /** - * Collects the names bound by statements within {@code node}, without - * descending into nested scopes (whose own names are bound here, though); - * and the types annotated names are declared with. - */ - private static void collectBindings(final TSNode node, final byte[] bytes, - final Set names, final Map types, - final boolean comprehension) - { - final int n = node.getNamedChildCount(); - for (int i = 0; i < n; i++) { - final TSNode child = node.getNamedChild(i); - final String type = child.getType(); - switch (type) { - case "function_definition": - case "class_definition": - addIdentifier(child.getChildByFieldName("name"), bytes, names); - continue; // NB: A nested scope. - case "lambda": - case "list_comprehension": - case "set_comprehension": - case "dictionary_comprehension": - case "generator_expression": - if (!comprehension) continue; // NB: A nested scope. - break; - case "assignment": - addType(child, bytes, types); - addTargets(child.getChildByFieldName("left"), bytes, names); - break; - case "augmented_assignment": - case "for_statement": - case "for_in_clause": - addTargets(child.getChildByFieldName("left"), bytes, names); - break; - case "named_expression": - addIdentifier(child.getChildByFieldName("name"), bytes, names); - break; - case "as_pattern_target": - addTargets(child, bytes, names); - break; - case "import_statement": - addImported(child, bytes, names, true); - continue; - case "import_from_statement": - addImported(child, bytes, names, false); - continue; - default: - } - collectBindings(child, bytes, names, types, comprehension); - } - } - - /** Adds the type of an annotated assignment, e.g. {@code x: int = 1}. */ - private static void addType(final TSNode assignment, final byte[] bytes, - final Map types) - { - final TSNode left = assignment.getChildByFieldName("left"); - final TSNode type = assignment.getChildByFieldName("type"); - if (left == null || left.isNull() || !left.getType().equals( - "identifier") || type == null || type.isNull()) return; - types.putIfAbsent(text(left, bytes), text(type, bytes)); - } - - /** Adds the names an import statement binds. */ - private static void addImported(final TSNode statement, final byte[] bytes, - final Set names, final boolean plainImport) - { - final int n = statement.getNamedChildCount(); - for (int i = 0; i < n; i++) { - if (!"name".equals(statement.getFieldNameForNamedChild(i))) continue; - final TSNode child = statement.getNamedChild(i); - if (child.getType().equals("aliased_import")) { - addIdentifier(child.getChildByFieldName("alias"), bytes, names); - } - else if (child.getType().equals("dotted_name")) { - // "import a.b" binds a; "from x import y" binds y. - final int count = child.getNamedChildCount(); - if (count > 0) addIdentifier(child.getNamedChild(plainImport ? 0 - : count - 1), bytes, names); - } - } - } - - /** - * Adds the identifiers bound by an assignment target or parameter list, - * e.g. {@code a, (b, c)} or {@code x, y: int = 1, *args, **kwargs}, but not - * attributes or subscripts. - */ - private static void addTargets(final TSNode node, final byte[] bytes, - final Set names) - { - if (node == null || node.isNull()) return; - switch (node.getType()) { - case "identifier": - addIdentifier(node, bytes, names); - return; - case "attribute": - case "subscript": - case "type": - return; - case "default_parameter": - case "typed_default_parameter": - addIdentifier(node.getChildByFieldName("name"), bytes, names); - return; - case "typed_parameter": - // NB: Its name is its first child (e.g. x in "x: int"). - if (node.getNamedChildCount() > 0) addTargets(node.getNamedChild(0), - bytes, names); - return; - default: - for (int i = 0; i < node.getNamedChildCount(); i++) { - addTargets(node.getNamedChild(i), bytes, names); - } - } - } - - private static void addIdentifier(final TSNode node, final byte[] bytes, - final Set names) - { - if (node == null || node.isNull() || !node.getType().equals( - "identifier")) return; - names.add(text(node, bytes)); - } - - private static String text(final TSNode node, final byte[] bytes) { - return new String(bytes, node.getStartByte(), node.getEndByte() - node - .getStartByte(), StandardCharsets.UTF_8); - } - - private static boolean isIdentifierPart(final char c) { - return Character.isLetterOrDigit(c) || c == '_'; - } - - /** For tests: the names found by the given parse, as a list. */ - static List names(final String text, final int caret) { - return new ArrayList<>(analyze(text, caret).names); - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java index 09d2f86..c20f8c8 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java @@ -6,13 +6,13 @@ * %% * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: - * + * * 1. Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. - * + * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE @@ -29,29 +29,133 @@ package org.scijava.plugins.scripting.appose.python.completion; +import java.nio.charset.StandardCharsets; +import java.util.ArrayDeque; import java.util.ArrayList; +import java.util.Arrays; +import java.util.Deque; +import java.util.HashMap; +import java.util.LinkedHashSet; import java.util.List; import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.regex.Matcher; +import java.util.regex.Pattern; import org.eclipse.lsp4j.CompletionItem; import org.eclipse.lsp4j.CompletionItemKind; import org.eclipse.lsp4j.CompletionList; import org.scijava.code.lsp.jvm.ScriptLanguageServer; -import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Kind; +import org.treesitter.TSNode; +import org.treesitter.TSParser; +import org.treesitter.TSTree; +import org.treesitter.TreeSitterPython; /** * An in-process language server for Python scripts run via Appose, from the - * syntax alone ({@link PythonSyntax}: tree-sitter): instantly, with no Python - * needed, names in scope (including the script's parameters and {@code task}, - * declared as {@link #DIALECT} says), builtins and keywords; and, after a dot, - * the names used after dots elsewhere in the script. (A Python language server, e.g. jedi-language-server, knows more; - * see {@link ApposePythonLanguageServer}.) + * syntax alone: tree-sitter parses the script at the caret, so it answers + * instantly, with no Python needed. It completes names in scope (including + * the script's parameters and {@code task}, as the session declares them; see + * {@link ApposePythonDialect}), builtins and keywords; and, after a dot, the + * names used after dots elsewhere in the script. + *

    + * It knows nothing of types, though: a Python language server, e.g. + * jedi-language-server, knows more (see {@link ApposePythonLanguageServer}). + *

    * * @author Gabriel Selzer */ public class PythonSyntaxLanguageServer extends ScriptLanguageServer { - static final ApposePythonDialect DIALECT = new ApposePythonDialect(); + /** Identifier marking the caret position in the parsed source. */ + static final String CARET = "__scijava_caret__"; + + /** Python keywords, including soft keywords. */ + private static final List KEYWORDS = Arrays.asList("False", "None", + "True", "and", "as", "assert", "async", "await", "break", "case", "class", + "continue", "def", "del", "elif", "else", "except", "finally", "for", + "from", "global", "if", "import", "in", "is", "lambda", "match", + "nonlocal", "not", "or", "pass", "raise", "return", "try", "while", + "with", "yield"); + + /** Python's built-in functions, types and constants. */ + private static final List BUILTINS = Arrays.asList("abs", "aiter", + "all", "anext", "any", "ascii", "bin", "bool", "breakpoint", "bytearray", + "bytes", "callable", "chr", "classmethod", "compile", "complex", + "delattr", "dict", "dir", "divmod", "enumerate", "eval", "exec", + "filter", "float", "format", "frozenset", "getattr", "globals", + "hasattr", "hash", "help", "hex", "id", "input", "int", "isinstance", + "issubclass", "iter", "len", "list", "locals", "map", "max", + "memoryview", "min", "next", "object", "oct", "open", "ord", "pow", + "print", "property", "range", "repr", "reversed", "round", "set", + "setattr", "slice", "sorted", "staticmethod", "str", "sum", "super", + "tuple", "type", "vars", "zip", "__name__", "__file__", "Ellipsis", + "NotImplemented", "Exception", "BaseException", "ValueError", + "TypeError", "KeyError", "IndexError", "RuntimeError", "AttributeError", + "ImportError", "OSError", "FileNotFoundError", "StopIteration", + "NotImplementedError", "ZeroDivisionError"); + + private static final Pattern IMPORT_LINE = Pattern.compile( + "^\\s*(from\\s+\\S*(\\s+import\\b.*)?|import\\s.*)$"); + + /** Node types that open a new scope. */ + private static final Set SCOPES = new LinkedHashSet<>(Arrays.asList( + "function_definition", "class_definition", "lambda", "list_comprehension", + "set_comprehension", "dictionary_comprehension", "generator_expression")); + + /** What the caret calls for. */ + enum Kind { + /** Nothing: the caret is in a comment, string or directive. */ + NONE, + /** A name, e.g. a variable, function or keyword. */ + NAME, + /** An attribute of the expression before a dot. */ + ATTRIBUTE, + /** A module or name in an import statement. */ + IMPORT + } + + /** The completion context at the caret, as {@link #analyze} finds it. */ + static final class Context { + + /** What the caret calls for. */ + final Kind kind; + + /** The partial identifier being typed (possibly empty). */ + final String prefix; + + /** Offset where the partial identifier starts. */ + final int replaceStart; + + /** For {@link Kind#ATTRIBUTE}: the source of the expression, or null. */ + final String receiver; + + /** Names in scope at the caret, innermost first. */ + final Set names; + + /** + * The types names in scope are annotated with, as written (e.g. + * {@code numpy.ndarray} for {@code image: numpy.ndarray}), by name. + */ + final Map types; + + /** For {@link Kind#ATTRIBUTE}: attributes used on the same receiver. */ + final Set attributesUsed; + + private Context(final Kind kind, final String prefix, + final int replaceStart, final String receiver, final Set names, + final Map types, final Set attributesUsed) + { + this.kind = kind; + this.prefix = prefix; + this.replaceStart = replaceStart; + this.receiver = receiver; + this.names = names; + this.types = types; + this.attributesUsed = attributesUsed; + } + } public PythonSyntaxLanguageServer() { super(null); @@ -59,7 +163,7 @@ public PythonSyntaxLanguageServer() { @Override protected CompletionList complete(final Document doc, final int caret) { - final PythonSyntax.Context ctx = PythonSyntax.analyze(doc.text(), caret); + final Context ctx = analyze(doc.text(), caret); if (ctx.kind == Kind.NONE || ctx.kind == Kind.NAME && ctx.prefix .isEmpty()) { @@ -68,6 +172,55 @@ protected CompletionList complete(final Document doc, final int caret) { return list(doc, ctx.replaceStart, caret, localCompletions(ctx)); } + /** Analyzes the given script at the given caret offset. */ + static Context analyze(final String text, final int caret) { + final int lineStart = text.lastIndexOf('\n', caret - 1) + 1; + final String line = text.substring(lineStart, caret); + int start = caret; + while (start > lineStart && isIdentifierPart(text.charAt(start - 1))) { + start--; + } + final String prefix = text.substring(start, caret); + final String before = text.substring(lineStart, start); + final String textBefore = text.substring(0, start); + final String closers = closers(textBefore); + if (closers == null || line.trim().startsWith("#")) return none(prefix, + start); + if (!prefix.isEmpty() && Character.isDigit(prefix.charAt(0))) { + return none(prefix, start); + } + + final Kind kind; + String receiver = null; + if (IMPORT_LINE.matcher(before + prefix).matches()) kind = Kind.IMPORT; + else if (before.trim().endsWith(".") && !before.trim().endsWith("..")) { + receiver = receiver(before.substring(0, before.lastIndexOf('.'))); + if (receiver != null && receiver.matches("\\d+")) { + return none(prefix, start); // a number, e.g. "1." + } + kind = Kind.ATTRIBUTE; + } + else kind = Kind.NAME; + + // Parse the script up to the caret, with the partial identifier replaced + // by a placeholder, and any open brackets closed: a clean tree there. + final String source = textBefore + CARET + closers; + final Set names = new LinkedHashSet<>(); + final Map types = new HashMap<>(); + if (kind == Kind.NAME) namesInScope(source, textBefore.length(), names, + types); + final Set used = kind == Kind.ATTRIBUTE && receiver != null + ? attributesUsed(text, receiver, start) : new LinkedHashSet<>(); + return new Context(kind, prefix, start, receiver, names, types, used); + } + + /** For tests: the names found by the given parse, as a list. */ + static List names(final String text, final int caret) { + return new ArrayList<>(analyze(text, caret).names); + } + + // -- Helper methods: completions -- + private static CompletionItem item(final String name, final CompletionItemKind kind, final String detail) { @@ -79,9 +232,7 @@ private static CompletionItem item(final String name, } /** Completions from the local (syntactic) analysis alone. */ - private static List localCompletions( - final PythonSyntax.Context ctx) - { + private static List localCompletions(final Context ctx) { final List out = new ArrayList<>(); final String prefix = ctx.prefix; if (ctx.kind == Kind.ATTRIBUTE) { @@ -96,11 +247,11 @@ private static List localCompletions( if (matches(name, prefix)) out.add(item(name, CompletionItemKind.Variable, ctx.types.get(name))); } - for (final String name : PythonSyntax.BUILTINS) { + for (final String name : BUILTINS) { if (matches(name, prefix)) out.add(item(name, CompletionItemKind.Function, "builtin")); } - for (final String keyword : PythonSyntax.KEYWORDS) { + for (final String keyword : KEYWORDS) { if (matches(keyword, prefix)) out.add(item(keyword, CompletionItemKind.Keyword, null)); } @@ -112,4 +263,277 @@ private static boolean matches(final String name, final String prefix) { return name.toLowerCase(Locale.ROOT).startsWith(prefix.toLowerCase( Locale.ROOT)) && !name.equals(prefix); } + + // -- Helper methods: analysis -- + + private static Context none(final String prefix, final int start) { + return new Context(Kind.NONE, prefix, start, null, new LinkedHashSet<>(), + new HashMap<>(), new LinkedHashSet<>()); + } + + /** + * Gets the closing brackets for those left open at the end of + * {@code text}, or null if the text ends inside a string or comment. + */ + static String closers(final String text) { + final Deque open = new ArrayDeque<>(); + int i = 0; + final int n = text.length(); + while (i < n) { + final char c = text.charAt(i); + if (c == '#') { + final int end = text.indexOf('\n', i); + if (end < 0) return null; + i = end + 1; + } + else if (c == '\'' || c == '"') { + final boolean triple = text.startsWith(String.valueOf(c) + c + c, i); + final String quote = triple ? String.valueOf(c) + c + c : String + .valueOf(c); + int j = i + quote.length(); + while (true) { + if (j >= n) return null; + final char d = text.charAt(j); + if (d == '\\') j += 2; + else if (!triple && d == '\n') return null; // unterminated line + else if (text.startsWith(quote, j)) break; + else j++; + } + i = j + quote.length(); + } + else { + if (c == '(') open.push(')'); + else if (c == '[') open.push(']'); + else if (c == '{') open.push('}'); + else if ((c == ')' || c == ']' || c == '}') && !open.isEmpty() && open + .peek() == c) open.pop(); + i++; + } + } + final StringBuilder sb = new StringBuilder(); + for (final char c : open) sb.append(c); + return sb.toString(); + } + + /** + * Gets the source of the expression ending the given text, e.g. + * {@code np.random} in {@code x = np.random}, or null if there is none. + */ + static String receiver(final String text) { + int i = text.length(); + while (i > 0 && Character.isWhitespace(text.charAt(i - 1))) i--; + final int end = i; + int depth = 0; + while (i > 0) { + final char c = text.charAt(i - 1); + if (c == ')' || c == ']' || c == '}') depth++; + else if (c == '(' || c == '[' || c == '{') { + if (depth == 0) break; + depth--; + } + else if (depth == 0 && !isIdentifierPart(c) && c != '.') break; + i--; + } + final String receiver = text.substring(i, end).trim(); + return receiver.isEmpty() ? null : receiver; + } + + /** Attributes used on {@code receiver} anywhere in the script. */ + private static Set attributesUsed(final String text, + final String receiver, final int exclude) + { + final Set used = new LinkedHashSet<>(); + // NB: Only spaces around the dot: "a.\nprint" is not an attribute. + final Matcher m = Pattern.compile("(? names, final Map types) + { + final byte[] bytes = source.getBytes(StandardCharsets.UTF_8); + final int caretByte = source.substring(0, caret).getBytes( + StandardCharsets.UTF_8).length; + final TSParser parser = new TSParser(); + try { + parser.setLanguage(new TreeSitterPython()); + final TSTree tree = parser.parseString(null, source); + final TSNode root = tree.getRootNode(); + TSNode node = root.getNamedDescendantForByteRange(caretByte, caretByte + + CARET.length()); + // Walk outward through the enclosing scopes, innermost first. + boolean inFunction = false; + for (; node != null && !node.isNull(); node = node.getParent()) { + final String type = node.getType(); + if (!SCOPES.contains(type)) continue; + // NB: Class bodies are not visible from their methods. + if (type.equals("class_definition") && inFunction) continue; + if (type.equals("function_definition") || type.equals("lambda")) { + inFunction = true; + } + final TSNode params = node.getChildByFieldName("parameters"); + if (params != null && !params.isNull()) { + addTargets(params, bytes, names); + } + final TSNode body = node.getChildByFieldName("body"); + if (type.endsWith("comprehension") || type.equals( + "generator_expression")) + { + collectBindings(node, bytes, names, types, true); + } + else if (body != null && !body.isNull() && !type.equals("lambda")) { + collectBindings(body, bytes, names, types, false); + } + } + collectBindings(root, bytes, names, types, false); + } + catch (final RuntimeException | LinkageError exc) { + // NB: Native parser unavailable or failed: no names, then. + } + finally { + parser.close(); + } + names.remove(CARET); + } + + /** + * Collects the names bound by statements within {@code node}, without + * descending into nested scopes (whose own names are bound here, though); + * and the types annotated names are declared with. + */ + private static void collectBindings(final TSNode node, final byte[] bytes, + final Set names, final Map types, + final boolean comprehension) + { + final int n = node.getNamedChildCount(); + for (int i = 0; i < n; i++) { + final TSNode child = node.getNamedChild(i); + final String type = child.getType(); + switch (type) { + case "function_definition": + case "class_definition": + addIdentifier(child.getChildByFieldName("name"), bytes, names); + continue; // NB: A nested scope. + case "lambda": + case "list_comprehension": + case "set_comprehension": + case "dictionary_comprehension": + case "generator_expression": + if (!comprehension) continue; // NB: A nested scope. + break; + case "assignment": + addType(child, bytes, types); + addTargets(child.getChildByFieldName("left"), bytes, names); + break; + case "augmented_assignment": + case "for_statement": + case "for_in_clause": + addTargets(child.getChildByFieldName("left"), bytes, names); + break; + case "named_expression": + addIdentifier(child.getChildByFieldName("name"), bytes, names); + break; + case "as_pattern_target": + addTargets(child, bytes, names); + break; + case "import_statement": + addImported(child, bytes, names, true); + continue; + case "import_from_statement": + addImported(child, bytes, names, false); + continue; + default: + } + collectBindings(child, bytes, names, types, comprehension); + } + } + + /** Adds the type of an annotated assignment, e.g. {@code x: int = 1}. */ + private static void addType(final TSNode assignment, final byte[] bytes, + final Map types) + { + final TSNode left = assignment.getChildByFieldName("left"); + final TSNode type = assignment.getChildByFieldName("type"); + if (left == null || left.isNull() || !left.getType().equals( + "identifier") || type == null || type.isNull()) return; + types.putIfAbsent(text(left, bytes), text(type, bytes)); + } + + /** Adds the names an import statement binds. */ + private static void addImported(final TSNode statement, final byte[] bytes, + final Set names, final boolean plainImport) + { + final int n = statement.getNamedChildCount(); + for (int i = 0; i < n; i++) { + if (!"name".equals(statement.getFieldNameForNamedChild(i))) continue; + final TSNode child = statement.getNamedChild(i); + if (child.getType().equals("aliased_import")) { + addIdentifier(child.getChildByFieldName("alias"), bytes, names); + } + else if (child.getType().equals("dotted_name")) { + // "import a.b" binds a; "from x import y" binds y. + final int count = child.getNamedChildCount(); + if (count > 0) addIdentifier(child.getNamedChild(plainImport ? 0 + : count - 1), bytes, names); + } + } + } + + /** + * Adds the identifiers bound by an assignment target or parameter list, + * e.g. {@code a, (b, c)} or {@code x, y: int = 1, *args, **kwargs}, but not + * attributes or subscripts. + */ + private static void addTargets(final TSNode node, final byte[] bytes, + final Set names) + { + if (node == null || node.isNull()) return; + switch (node.getType()) { + case "identifier": + addIdentifier(node, bytes, names); + return; + case "attribute": + case "subscript": + case "type": + return; + case "default_parameter": + case "typed_default_parameter": + addIdentifier(node.getChildByFieldName("name"), bytes, names); + return; + case "typed_parameter": + // NB: Its name is its first child (e.g. x in "x: int"). + if (node.getNamedChildCount() > 0) addTargets(node.getNamedChild(0), + bytes, names); + return; + default: + for (int i = 0; i < node.getNamedChildCount(); i++) { + addTargets(node.getNamedChild(i), bytes, names); + } + } + } + + private static void addIdentifier(final TSNode node, final byte[] bytes, + final Set names) + { + if (node == null || node.isNull() || !node.getType().equals( + "identifier")) return; + names.add(text(node, bytes)); + } + + private static String text(final TSNode node, final byte[] bytes) { + return new String(bytes, node.getStartByte(), node.getEndByte() - node + .getStartByte(), StandardCharsets.UTF_8); + } + + private static boolean isIdentifierPart(final char c) { + return Character.isLetterOrDigit(c) || c == '_'; + } } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServerTest.java similarity index 74% rename from src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxTest.java rename to src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServerTest.java index d368538..05eafd6 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServerTest.java @@ -36,15 +36,15 @@ import java.util.List; import org.junit.Test; -import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Context; -import org.scijava.plugins.scripting.appose.python.completion.PythonSyntax.Kind; +import org.scijava.plugins.scripting.appose.python.completion.PythonSyntaxLanguageServer.Context; +import org.scijava.plugins.scripting.appose.python.completion.PythonSyntaxLanguageServer.Kind; /** - * Tests {@link PythonSyntax}. + * Tests how {@link PythonSyntaxLanguageServer} analyzes scripts. * * @author Gabriel Selzer */ -public class PythonSyntaxTest { +public class PythonSyntaxLanguageServerTest { @Test public void testNamesInScope() { @@ -54,7 +54,7 @@ public void testNamesInScope() { "def f(a, b=2, *args, c: int = 3, **kw):\n" + // " loc = 3\n" + // " lo"; - final List names = PythonSyntax.names(code, code.length()); + final List names = PythonSyntaxLanguageServer.names(code, code.length()); for (final String name : new String[] { "loc", "a", "b", "args", "c", "kw", "f", "x", "np", "os", "p", "sep" }) { @@ -62,52 +62,52 @@ public void testNamesInScope() { } // Innermost scope first. assertTrue(names.indexOf("loc") < names.indexOf("x")); - assertFalse(names.contains(PythonSyntax.CARET)); + assertFalse(names.contains(PythonSyntaxLanguageServer.CARET)); } @Test public void testScopeRules() { // Class bodies are not visible from their methods. final String cls = "class C:\n attr = 1\n def m(self):\n at"; - final List inMethod = PythonSyntax.names(cls, cls.length()); + final List inMethod = PythonSyntaxLanguageServer.names(cls, cls.length()); assertTrue(inMethod.contains("self")); assertTrue(inMethod.contains("C")); assertFalse(inMethod.contains("attr")); // Functions' locals are not visible outside them. final String fn = "def g():\n inner = 1\nin"; - assertFalse(PythonSyntax.names(fn, fn.length()).contains("inner")); + assertFalse(PythonSyntaxLanguageServer.names(fn, fn.length()).contains("inner")); // Comprehension, with, except and walrus targets. final String comp = "[val for val in range(3) if va"; - assertTrue(PythonSyntax.names(comp, comp.length()).contains("val")); + assertTrue(PythonSyntaxLanguageServer.names(comp, comp.length()).contains("val")); final String blocks = "with open(f) as fh:\n try:\n pass\n" + " except E as err:\n if (n := 2):\n e"; - final List names = PythonSyntax.names(blocks, blocks.length()); + final List names = PythonSyntaxLanguageServer.names(blocks, blocks.length()); assertTrue(names.toString(), names.contains("fh") && names.contains( "err") && names.contains("n")); // Unclosed brackets before the caret. final String open = "def h(alpha):\n print(1, [2, al"; - assertTrue(PythonSyntax.names(open, open.length()).contains("alpha")); + assertTrue(PythonSyntaxLanguageServer.names(open, open.length()).contains("alpha")); } @Test public void testAttributeContext() { final String code = "import numpy as np\nnp.random.ra"; - final Context ctx = PythonSyntax.analyze(code, code.length()); + final Context ctx = PythonSyntaxLanguageServer.analyze(code, code.length()); assertEquals(Kind.ATTRIBUTE, ctx.kind); assertEquals("np.random", ctx.receiver); assertEquals("ra", ctx.prefix); assertEquals(code.length() - 2, ctx.replaceStart); - assertEquals("foo(x)[0]", PythonSyntax.analyze("y = foo(x)[0].", 14) + assertEquals("foo(x)[0]", PythonSyntaxLanguageServer.analyze("y = foo(x)[0].", 14) .receiver); // Attributes used elsewhere on the same receiver. final String img = "img.shape\nimg.dtype\nmyimg.size\nimg.sh"; - final Context used = PythonSyntax.analyze(img, img.length()); + final Context used = PythonSyntaxLanguageServer.analyze(img, img.length()); assertTrue(used.attributesUsed.contains("shape")); assertTrue(used.attributesUsed.contains("dtype")); assertFalse(used.attributesUsed.contains("size")); // A dot ending a line is not followed by an attribute on the next one. final String nextLine = "a = numpy.ndarray(3)\na.\nprint(a.shape)\n"; - final Context dot = PythonSyntax.analyze(nextLine, nextLine.indexOf( + final Context dot = PythonSyntaxLanguageServer.analyze(nextLine, nextLine.indexOf( "a.\n") + 2); assertTrue(dot.attributesUsed.contains("shape")); assertFalse(dot.attributesUsed.toString(), dot.attributesUsed.contains( @@ -120,26 +120,26 @@ public void testNoCompletion() { "x = 1 # np.", "d = \"\"\"doc\nmore np.", "#@ String na", "x = 1.", "f\"{x}.y" }) { - assertEquals(code, Kind.NONE, PythonSyntax.analyze(code, code.length()) + assertEquals(code, Kind.NONE, PythonSyntaxLanguageServer.analyze(code, code.length()) .kind); } // But after a closed string, completion works again. final String closed = "s = 'a.b'\nx = s.up"; - assertEquals(Kind.ATTRIBUTE, PythonSyntax.analyze(closed, closed.length()) + assertEquals(Kind.ATTRIBUTE, PythonSyntaxLanguageServer.analyze(closed, closed.length()) .kind); } @Test public void testImports() { - assertEquals(Kind.IMPORT, PythonSyntax.analyze("import nu", 9).kind); - assertEquals(Kind.IMPORT, PythonSyntax.analyze("from scipy.nd", 13).kind); - assertEquals(Kind.IMPORT, PythonSyntax.analyze( + assertEquals(Kind.IMPORT, PythonSyntaxLanguageServer.analyze("import nu", 9).kind); + assertEquals(Kind.IMPORT, PythonSyntaxLanguageServer.analyze("from scipy.nd", 13).kind); + assertEquals(Kind.IMPORT, PythonSyntaxLanguageServer.analyze( "from scipy import ndim", 22).kind); } @Test public void testUnicode() { final String code = "s = '\u00e9t\u00e9'\nvaleur = 1\nval"; - assertTrue(PythonSyntax.names(code, code.length()).contains("valeur")); + assertTrue(PythonSyntaxLanguageServer.names(code, code.length()).contains("valeur")); } } From fa75940f997d15f1a9e76802247cf5e299742334 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Mon, 12 Oct 2026 13:49:26 -0500 Subject: [PATCH 22/37] Fold DelegatingLanguageServer into PythonCompletionServer PythonCompletionServer was the only server built on it, so it passes on what it does not change itself. Co-Authored-By: Claude Opus 5.5 --- .../completion/DelegatingLanguageServer.java | 168 ------------------ .../completion/PythonCompletionServer.java | 120 ++++++++++++- 2 files changed, 113 insertions(+), 175 deletions(-) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/DelegatingLanguageServer.java diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DelegatingLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DelegatingLanguageServer.java deleted file mode 100644 index 4fae53d..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DelegatingLanguageServer.java +++ /dev/null @@ -1,168 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.util.List; -import java.util.concurrent.CompletableFuture; - -import org.eclipse.lsp4j.CompletionItem; -import org.eclipse.lsp4j.CompletionList; -import org.eclipse.lsp4j.CompletionParams; -import org.eclipse.lsp4j.DidChangeConfigurationParams; -import org.eclipse.lsp4j.DidChangeTextDocumentParams; -import org.eclipse.lsp4j.DidChangeWatchedFilesParams; -import org.eclipse.lsp4j.DidCloseTextDocumentParams; -import org.eclipse.lsp4j.DidOpenTextDocumentParams; -import org.eclipse.lsp4j.DidSaveTextDocumentParams; -import org.eclipse.lsp4j.Hover; -import org.eclipse.lsp4j.HoverParams; -import org.eclipse.lsp4j.InitializeParams; -import org.eclipse.lsp4j.InitializeResult; -import org.eclipse.lsp4j.InitializedParams; -import org.eclipse.lsp4j.SignatureHelp; -import org.eclipse.lsp4j.SignatureHelpParams; -import org.eclipse.lsp4j.jsonrpc.messages.Either; -import org.eclipse.lsp4j.services.LanguageClient; -import org.eclipse.lsp4j.services.LanguageClientAware; -import org.eclipse.lsp4j.services.LanguageServer; -import org.eclipse.lsp4j.services.TextDocumentService; -import org.eclipse.lsp4j.services.WorkspaceService; - -/** - * A language server passing everything on to another: a base for decorators - * changing some of its answers. - * - * @author Gabriel Selzer - */ -public class DelegatingLanguageServer implements LanguageServer, - TextDocumentService, WorkspaceService, LanguageClientAware -{ - - protected final LanguageServer inner; - - public DelegatingLanguageServer(final LanguageServer inner) { - this.inner = inner; - } - - @Override - public void connect(final LanguageClient client) { - if (inner instanceof LanguageClientAware) { - ((LanguageClientAware) inner).connect(client); - } - } - - @Override - public CompletableFuture initialize( - final InitializeParams params) - { - return inner.initialize(params); - } - - @Override - public void initialized(final InitializedParams params) { - inner.initialized(params); - } - - @Override - public CompletableFuture shutdown() { - return inner.shutdown(); - } - - @Override - public void exit() { - inner.exit(); - } - - @Override - public TextDocumentService getTextDocumentService() { - return this; - } - - @Override - public WorkspaceService getWorkspaceService() { - return this; - } - - @Override - public void didOpen(final DidOpenTextDocumentParams params) { - inner.getTextDocumentService().didOpen(params); - } - - @Override - public void didChange(final DidChangeTextDocumentParams params) { - inner.getTextDocumentService().didChange(params); - } - - @Override - public void didClose(final DidCloseTextDocumentParams params) { - inner.getTextDocumentService().didClose(params); - } - - @Override - public void didSave(final DidSaveTextDocumentParams params) { - inner.getTextDocumentService().didSave(params); - } - - @Override - public CompletableFuture, CompletionList>> - completion(final CompletionParams params) - { - return inner.getTextDocumentService().completion(params); - } - - @Override - public CompletableFuture resolveCompletionItem( - final CompletionItem item) - { - return inner.getTextDocumentService().resolveCompletionItem(item); - } - - @Override - public CompletableFuture signatureHelp( - final SignatureHelpParams params) - { - return inner.getTextDocumentService().signatureHelp(params); - } - - @Override - public CompletableFuture hover(final HoverParams params) { - return inner.getTextDocumentService().hover(params); - } - - @Override - public void didChangeConfiguration(final DidChangeConfigurationParams p) { - inner.getWorkspaceService().didChangeConfiguration(p); - } - - @Override - public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) { - inner.getWorkspaceService().didChangeWatchedFiles(p); - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java index 0b2604c..2990bc9 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java @@ -44,11 +44,25 @@ import org.eclipse.lsp4j.CompletionItemKind; import org.eclipse.lsp4j.CompletionList; import org.eclipse.lsp4j.CompletionParams; +import org.eclipse.lsp4j.DidChangeConfigurationParams; import org.eclipse.lsp4j.DidChangeTextDocumentParams; +import org.eclipse.lsp4j.DidChangeWatchedFilesParams; import org.eclipse.lsp4j.DidCloseTextDocumentParams; import org.eclipse.lsp4j.DidOpenTextDocumentParams; +import org.eclipse.lsp4j.DidSaveTextDocumentParams; +import org.eclipse.lsp4j.Hover; +import org.eclipse.lsp4j.HoverParams; +import org.eclipse.lsp4j.InitializeParams; +import org.eclipse.lsp4j.InitializeResult; +import org.eclipse.lsp4j.InitializedParams; +import org.eclipse.lsp4j.SignatureHelp; +import org.eclipse.lsp4j.SignatureHelpParams; import org.eclipse.lsp4j.jsonrpc.messages.Either; +import org.eclipse.lsp4j.services.LanguageClient; +import org.eclipse.lsp4j.services.LanguageClientAware; import org.eclipse.lsp4j.services.LanguageServer; +import org.eclipse.lsp4j.services.TextDocumentService; +import org.eclipse.lsp4j.services.WorkspaceService; import org.scijava.code.lsp.jvm.Callable; import org.scijava.code.lsp.Positions; @@ -65,10 +79,15 @@ * {@link Callable}s: snippets with their parameters and types. (Not in * imports: an import names things, rather than calling them.) * + *

    + * Everything else is passed on to the server as is. + *

    * * @author Gabriel Selzer */ -public class PythonCompletionServer extends DelegatingLanguageServer { +public class PythonCompletionServer implements LanguageServer, + TextDocumentService, WorkspaceService, LanguageClientAware +{ /** How long an answer may be reused, in milliseconds. */ private static final long ANSWER_LIFETIME = 20_000; @@ -77,6 +96,9 @@ public class PythonCompletionServer extends DelegatingLanguageServer { private static final Pattern IMPORT = Pattern.compile( "^\\s*(from|import)\\s"); + /** The Python language server. */ + private final LanguageServer inner; + private final int signatures; private final LongSupplier clock; @@ -111,16 +133,61 @@ public PythonCompletionServer(final LanguageServer inner, PythonCompletionServer(final LanguageServer inner, final int signatures, final LongSupplier clock) { - super(inner); + this.inner = inner; this.signatures = signatures; this.clock = clock; } + // -- LanguageClientAware -- + + @Override + public void connect(final LanguageClient client) { + if (inner instanceof LanguageClientAware) { + ((LanguageClientAware) inner).connect(client); + } + } + + // -- LanguageServer -- + + @Override + public CompletableFuture initialize( + final InitializeParams params) + { + return inner.initialize(params); + } + + @Override + public void initialized(final InitializedParams params) { + inner.initialized(params); + } + + @Override + public CompletableFuture shutdown() { + return inner.shutdown(); + } + + @Override + public void exit() { + inner.exit(); + } + + @Override + public TextDocumentService getTextDocumentService() { + return this; + } + + @Override + public WorkspaceService getWorkspaceService() { + return this; + } + + // -- TextDocumentService -- + @Override public void didOpen(final DidOpenTextDocumentParams params) { texts.put(params.getTextDocument().getUri(), params.getTextDocument() .getText()); - super.didOpen(params); + inner.getTextDocumentService().didOpen(params); } @Override @@ -129,14 +196,14 @@ public void didChange(final DidChangeTextDocumentParams params) { texts.put(params.getTextDocument().getUri(), params.getContentChanges() .get(params.getContentChanges().size() - 1).getText()); } - super.didChange(params); + inner.getTextDocumentService().didChange(params); } @Override public void didClose(final DidCloseTextDocumentParams params) { texts.remove(params.getTextDocument().getUri()); pending.remove(params.getTextDocument().getUri()); - super.didClose(params); + inner.getTextDocumentService().didClose(params); } @Override @@ -145,7 +212,8 @@ public void didClose(final DidCloseTextDocumentParams params) { { final String uri = params.getTextDocument().getUri(); final String text = texts.get(uri); - if (text == null) return super.completion(params); + if (text == null) return inner.getTextDocumentService().completion( + params); final int caret = Math.min(Positions.offset(text, params.getPosition()), text.length()); int start = caret; @@ -173,7 +241,7 @@ public void didClose(final DidCloseTextDocumentParams params) { final CompletionParams p = new CompletionParams(params.getTextDocument(), Positions.position(text, ask), params.getContext()); final CompletableFuture, CompletionList>> raw = - super.completion(p); + inner.getTextDocumentService().completion(p); final int n = word.isEmpty() || importing ? 0 : signatures; answer = new Answer(raw, raw.thenCompose(list -> resolve(list, n)), now); synchronized (answers) { @@ -192,6 +260,44 @@ public void didClose(final DidCloseTextDocumentParams params) { word))); } + @Override + public void didSave(final DidSaveTextDocumentParams params) { + inner.getTextDocumentService().didSave(params); + } + + @Override + public CompletableFuture resolveCompletionItem( + final CompletionItem item) + { + return inner.getTextDocumentService().resolveCompletionItem(item); + } + + @Override + public CompletableFuture signatureHelp( + final SignatureHelpParams params) + { + return inner.getTextDocumentService().signatureHelp(params); + } + + @Override + public CompletableFuture hover(final HoverParams params) { + return inner.getTextDocumentService().hover(params); + } + + // -- WorkspaceService -- + + @Override + public void didChangeConfiguration(final DidChangeConfigurationParams p) { + inner.getWorkspaceService().didChangeConfiguration(p); + } + + @Override + public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) { + inner.getWorkspaceService().didChangeWatchedFiles(p); + } + + // -- Helper methods -- + /** Resolves the first completions, making those with signatures callables. */ private CompletableFuture resolve( final Either, CompletionList> answer, final int n) From 9552b91cf76fda4d84d9282471b6eb00ba418d2d Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Mon, 12 Oct 2026 14:40:11 -0500 Subject: [PATCH 23/37] Complete Appose Python with jedi alone The syntax-only server (tree-sitter) answered only when jedi had failed, or with names jedi had not completed: completions waited for jedi anyway. Gone with it are the merging of the two servers' answers, the fallbacks from jedi to it, and the tree-sitter dependencies. Without jedi, the plugin no longer serves Appose Python at all. Co-Authored-By: Claude Opus 5.5 --- README.md | 45 +- pom.xml | 13 - .../ApposePythonLanguageServer.java | 182 ++---- .../completion/ApposePythonServerPlugin.java | 28 +- .../PythonSyntaxLanguageServer.java | 539 ------------------ .../ApposePythonLanguageServerTest.java | 56 +- .../completion/ApposePythonServersTest.java | 59 +- .../PythonSyntaxLanguageServerTest.java | 145 ----- 8 files changed, 98 insertions(+), 969 deletions(-) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java delete mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServerTest.java diff --git a/README.md b/README.md index bb6975d..d9919fd 100644 --- a/README.md +++ b/README.md @@ -137,30 +137,25 @@ script's own line numbers. ## Code completion -The Script Editor completes code in these scripts with one language server +The Script Editor completes code in these scripts with a language server (`ApposePythonLanguageServer`, offered to `scijava-code-lsp`'s `LanguageServerService` by `ApposePythonServerPlugin`, so any other LSP client -of it gets the same), combining two sources: jedi's answers first, then the -syntax-only ones for names jedi has not completed. - -* **Instantly, with no Python needed:** names in scope (variables, - functions, classes, imports, parameters of the enclosing function), the - script's `#@` parameters, `task`, builtins and keywords. This comes from - parsing the script with [tree-sitter](https://tree-sitter.github.io/) - (`PythonSyntaxLanguageServer`). -* **With a language server, - [jedi-language-server](https://github.com/pappasam/jedi-language-server) - (built on [Jedi](https://jedi.readthedocs.io/)):** - members of objects and modules, function signatures (with parameter names - and types), documentation, hover and syntax errors. The server runs in a small environment of its own - (`appose-python-completion`: Python, jedi-language-server, Appose and uv), - which is built in the background the first time you complete code, and - takes a while only then. The server talks to the editor via the - [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) - (see `scijava-code-lsp`); it sees the script as plain Python, with its `#@` - parameters declared. One server runs per script environment; servers - unused for 10 minutes stop. Their error output goes to - `appose-python-completion-server.log`, next to Appose's environments. +of it gets the same): [jedi-language-server](https://github.com/pappasam/jedi-language-server) +(built on [Jedi](https://jedi.readthedocs.io/)). It completes names in scope, +the script's `#@` parameters, `task`, builtins and keywords, and members of +objects and modules; and it gives function signatures (with parameter names +and types), documentation, hover and syntax errors. + +The server runs in a small environment of its own +(`appose-python-completion`: Python, jedi-language-server, Appose and uv), +which is built in the background the first time you complete code, and +takes a while only then; until it is, there are no completions. The server +talks to the editor via the +[Language Server Protocol](https://microsoft.github.io/language-server-protocol/) +(see `scijava-code-lsp`); it sees the script as plain Python, with its `#@` +parameters declared. One server runs per script environment; servers +unused for 10 minutes stop. Their error output goes to +`appose-python-completion-server.log`, next to Appose's environments. Once a script's environment is built, by running the script (in this session or an earlier one), the server analyzes the script against it, so that the packages @@ -175,6 +170,6 @@ its worker. Script parameters are known by the Python types they have at run time, e.g. images as NumPy arrays. Typing never waits for the server: if its answer for a word is not ready -within a moment (the first answer for a package can take a few seconds), the -instant completions show, and the server's are used once ready. Questions -that are no longer of interest (e.g. as you type on) are cancelled. +within a moment (the first answer for a package can take a few seconds), it +is shown once ready. Questions that are no longer of interest (e.g. as you +type on) are cancelled. diff --git a/pom.xml b/pom.xml index ca78e97..8d7098d 100644 --- a/pom.xml +++ b/pom.xml @@ -96,8 +96,6 @@ 2.101.0 0.1.0-SNAPSHOT - 0.26.6 - 0.25.0 @@ -117,17 +115,6 @@ - - - io.github.bonede - tree-sitter - ${tree-sitter.version} - - - io.github.bonede - tree-sitter-python - ${tree-sitter-python.version} - com.fasterxml.jackson.core jackson-databind diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java index f37d0c1..ef98348 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java @@ -6,13 +6,13 @@ * %% * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: - * + * * 1. Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. - * + * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE @@ -31,11 +31,8 @@ import java.util.ArrayList; import java.util.Collections; -import java.util.Comparator; -import java.util.HashSet; import java.util.List; import java.util.Map; -import java.util.Set; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; import java.util.function.Function; @@ -76,22 +73,15 @@ import org.scijava.code.lsp.Environment; /** - * The language server of Python scripts run via Appose: the syntax-only - * server (see {@link PythonSyntaxLanguageServer}), instant and needing no - * Python; and, if offered, jedi-language-server, knowing more. - *
      - *
    • Completions: jedi's, then the syntax server's that complete other - * names. (Without jedi, the syntax server's.)
      • - *
    • Signature help and hover: jedi's, if any; otherwise the syntax - * server's.
      • - *
    • Problems: jedi's (the syntax server finds none).
      • - *
    + * The language server of Python scripts run via Appose: jedi-language-server, + * analyzing each script against where it runs (its environment's Python, once + * built, and an overlay of the packages it declares but its environment lacks; + * see {@link ApposePythonEnvironments}). *

    - * Each script is analyzed by a jedi of where it runs: its environment's - * Python, once built, and an overlay of the packages it declares but its - * environment lacks (see {@link ApposePythonEnvironments}). As that changes - * (as the script's metadata is edited, or its environment is built), the - * script moves to another jedi. + * Jedi learns where scripts run only when it starts, so each script is + * analyzed by a jedi of its own environment. As that changes (as the script's + * metadata is edited, or its environment is built), the script moves to + * another jedi. *

    * * @author Gabriel Selzer @@ -100,7 +90,6 @@ public class ApposePythonLanguageServer implements LanguageServer, TextDocumentService, WorkspaceService, LanguageClientAware { - private final LanguageServer syntax; private final Function environment; private final Function jedi; private final ApposePythonEnvironments environments; @@ -112,28 +101,20 @@ public class ApposePythonLanguageServer implements LanguageServer, private volatile LanguageClient client; /** - * @param syntax The syntax-only server. * @param environment Gets where a script runs. - * @param jedi Makes a jedi for scripts running in an environment; or null, - * if jedi is not offered. + * @param jedi Makes a jedi for scripts running in an environment. * @param environments Tells when where scripts run may have changed without * them changing (e.g. as environments are built); or null. */ - ApposePythonLanguageServer(final LanguageServer syntax, + ApposePythonLanguageServer( final Function environment, final Function jedi, final ApposePythonEnvironments environments) { - this.syntax = syntax; this.environment = environment; this.jedi = jedi; this.environments = environments; - if (syntax instanceof LanguageClientAware) { - ((LanguageClientAware) syntax).connect(new Forwarder(null)); - } - if (jedi != null && environments != null) { - environments.addListener(relocateAll); - } + if (environments != null) environments.addListener(relocateAll); } @Override @@ -147,8 +128,6 @@ public void connect(final LanguageClient c) { public CompletableFuture initialize( final InitializeParams params) { - syntax.initialize(params).thenRun(() -> syntax.initialized( - new InitializedParams())); final ServerCapabilities capabilities = new ServerCapabilities(); capabilities.setTextDocumentSync(TextDocumentSyncKind.Full); capabilities.setCompletionProvider(new CompletionOptions(true, @@ -167,12 +146,12 @@ public CompletableFuture shutdown() { didClose(new DidCloseTextDocumentParams(new TextDocumentIdentifier( uri))); } - return syntax.shutdown(); + return CompletableFuture.completedFuture(null); } @Override public void exit() { - syntax.exit(); + // NB: The jedis are let go as their documents close. } @Override @@ -189,8 +168,6 @@ public WorkspaceService getWorkspaceService() { @Override public void didOpen(final DidOpenTextDocumentParams params) { - syntax.getTextDocumentService().didOpen(params); - if (jedi == null) return; final Doc doc = new Doc(params.getTextDocument()); docs.put(doc.item.getUri(), doc); synchronized (doc) { @@ -202,7 +179,6 @@ public void didOpen(final DidOpenTextDocumentParams params) { @Override public void didChange(final DidChangeTextDocumentParams params) { - syntax.getTextDocumentService().didChange(params); final Doc doc = docs.get(params.getTextDocument().getUri()); final List changes = params .getContentChanges(); @@ -218,7 +194,6 @@ public void didChange(final DidChangeTextDocumentParams params) { @Override public void didClose(final DidCloseTextDocumentParams params) { - syntax.getTextDocumentService().didClose(params); final Doc doc = docs.remove(params.getTextDocument().getUri()); if (doc == null) return; synchronized (doc) { @@ -237,15 +212,12 @@ public void didSave(final DidSaveTextDocumentParams params) { public CompletableFuture, CompletionList>> completion(final CompletionParams params) { - final CompletableFuture own = orNull(syntax - .getTextDocumentService().completion(params)).thenApply( - ApposePythonLanguageServer::list); - final LanguageServer j = jedi(params.getTextDocument().getUri()); - if (j == null) return own.thenApply(Either::forRight); + final String uri = params.getTextDocument().getUri(); + final LanguageServer j = jedi(uri); + if (j == null) return CompletableFuture.completedFuture(Either.forRight( + new CompletionList())); return orNull(j.getTextDocumentService().completion(params)).thenApply( - ApposePythonLanguageServer::list).thenCombine(own, (theirs, - mine) -> Either.forRight(merge(theirs, mine, params.getTextDocument() - .getUri()))); + answer -> Either.forRight(mark(list(answer), uri))); } @Override @@ -253,24 +225,16 @@ public CompletableFuture resolveCompletionItem( final CompletionItem item) { if (!(item.getData() instanceof Source)) { - // NB: Not merged (e.g. without jedi): the syntax server's. - return syntax.getTextDocumentService().resolveCompletionItem(item); + return CompletableFuture.completedFuture(item); } final Source from = (Source) item.getData(); - final LanguageServer server = from.jedi ? jedi(from.uri) : syntax; - final String sortText = item.getSortText(); + final LanguageServer j = jedi(from.uri); + if (j == null) return CompletableFuture.completedFuture(item); item.setData(from.data); - item.setSortText(from.sortText); - if (server == null) { - item.setData(from); - item.setSortText(sortText); - return CompletableFuture.completedFuture(item); - } - return orNull(server.getTextDocumentService().resolveCompletionItem(item)) + return orNull(j.getTextDocumentService().resolveCompletionItem(item)) .thenApply(resolved -> { final CompletionItem out = resolved != null ? resolved : item; out.setData(from); - out.setSortText(sortText); return out; }); } @@ -279,15 +243,16 @@ public CompletableFuture resolveCompletionItem( public CompletableFuture signatureHelp( final SignatureHelpParams params) { - return first(params.getTextDocument().getUri(), s -> s.signatureHelp( - params), h -> h != null && h.getSignatures() != null && !h - .getSignatures().isEmpty()); + final LanguageServer j = jedi(params.getTextDocument().getUri()); + return j == null ? CompletableFuture.completedFuture(null) : orNull(j + .getTextDocumentService().signatureHelp(params)); } @Override public CompletableFuture hover(final HoverParams params) { - return first(params.getTextDocument().getUri(), s -> s.hover(params), - h -> h != null && h.getContents() != null); + final LanguageServer j = jedi(params.getTextDocument().getUri()); + return j == null ? CompletableFuture.completedFuture(null) : orNull(j + .getTextDocumentService().hover(params)); } // -- WorkspaceService -- @@ -362,67 +327,18 @@ void relocateAll() { } } - /** Asks jedi, then (if its answer is not accepted) the syntax server. */ - private CompletableFuture first(final String uri, - final Function> request, - final java.util.function.Predicate accepted) - { - final LanguageServer j = jedi(uri); - final CompletableFuture theirs = j == null ? CompletableFuture - .completedFuture(null) : orNull(request.apply(j - .getTextDocumentService())); - return theirs.thenCompose(t -> accepted.test(t) ? CompletableFuture - .completedFuture(t) : orNull(request.apply(syntax - .getTextDocumentService()))); - } - /** - * Jedi's completions, then the syntax server's whose names jedi has not - * completed, each in its own order. Each remembers whose it is, for - * resolving. + * Marks each completion with its document, so that it is resolved by the + * jedi analyzing it. */ - private static CompletionList merge(final CompletionList theirs, - final CompletionList own, final String uri) + private static CompletionList mark(final CompletionList list, + final String uri) { - final List out = new ArrayList<>(); - final Set names = new HashSet<>(); - int i = 0; - for (final CompletionItem item : sorted(theirs)) { - names.add(name(item)); - mark(item, true, uri, i++); - out.add(item); - } - for (final CompletionItem item : sorted(own)) { - if (names.contains(name(item))) continue; - mark(item, false, uri, i++); - out.add(item); + if (list == null) return new CompletionList(); + for (final CompletionItem item : list.getItems()) { + item.setData(new Source(uri, item.getData())); } - final boolean incomplete = theirs != null && theirs.isIncomplete() || - own != null && own.isIncomplete(); - return new CompletionList(incomplete, out); - } - - private static void mark(final CompletionItem item, final boolean fromJedi, - final String uri, final int position) - { - item.setData(new Source(fromJedi, uri, item.getData(), item - .getSortText())); - item.setSortText(String.format("%d%06d", fromJedi ? 0 : 1, position)); - } - - private static List sorted(final CompletionList list) { - final List items = new ArrayList<>(); - if (list != null && list.getItems() != null) items.addAll(list.getItems()); - items.sort(Comparator.comparing(c -> c.getSortText() != null ? c - .getSortText() : c.getLabel())); - return items; - } - - /** What a completion completes: its label, without any parameters. */ - private static String name(final CompletionItem item) { - final String label = item.getLabel(); - final int paren = label.indexOf('('); - return (paren < 0 ? label : label.substring(0, paren)).trim(); + return list; } private static CompletionList list( @@ -453,31 +369,24 @@ private Doc(final TextDocumentItem item) { } } - /** Remembers whose a merged completion is. */ + /** Remembers which document a completion is of, for resolving. */ private static final class Source { - private final boolean jedi; private final String uri; private final Object data; - private final String sortText; - private Source(final boolean jedi, final String uri, final Object data, - final String sortText) - { - this.jedi = jedi; + private Source(final String uri, final Object data) { this.uri = uri; this.data = data; - this.sortText = sortText; } } /** - * Passes problems on to the client: jedi's, from the jedi analyzing their - * document; or, without jedi, the syntax server's (none). + * Passes the problems the jedi analyzing their document finds on to the + * client; and its messages. */ private final class Forwarder implements LanguageClient { - /** The jedi, or null for the syntax server. */ private final LanguageServer source; private Forwarder(final LanguageServer source) { @@ -487,13 +396,8 @@ private Forwarder(final LanguageServer source) { @Override public void publishDiagnostics(final PublishDiagnosticsParams p) { final LanguageClient c = client; - if (c == null) return; - if (source == null) { - if (jedi == null) c.publishDiagnostics(p); - return; - } // NB: Only from the jedi analyzing the document now. - if (jedi(p.getUri()) == source) c.publishDiagnostics(p); + if (c != null && jedi(p.getUri()) == source) c.publishDiagnostics(p); } @Override diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java index 3ae5edb..7afe6fc 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java @@ -44,16 +44,15 @@ /** * Offers the language server of Python scripts run via Appose (see - * {@link ApposePythonLanguageServer}): the syntax-only server, and, if offered - * (see {@link JediServerProcessPlugin}), jedi-language-server: members of - * objects and modules, signatures, documentation, hover and syntax errors. - * Once a script's environment is built (by running the script), jedi - * analyzes the script against it, so its packages are known; packages the - * script declares but its environment lacks are installed for completion only - * (see {@link DependencyOverlays}). - * Both see the script as plain Python, its parameters declared with the - * types they have when it runs (e.g. images as NumPy arrays; see - * {@link #dialect()}). + * {@link ApposePythonLanguageServer}): jedi-language-server (see + * {@link JediServerProcessPlugin}), for members of objects and modules, + * signatures, documentation, hover and syntax errors. Once a script's + * environment is built (by running the script), jedi analyzes the script + * against it, so its packages are known; packages the script declares but its + * environment lacks are installed for completion only (see + * {@link DependencyOverlays}). Jedi sees the script as plain Python, its + * parameters declared with the types they have when it runs (e.g. images as + * NumPy arrays; see {@link #dialect()}). *

    * Completion never builds a script's environment, and never disturbs its * worker. @@ -78,15 +77,14 @@ public class ApposePythonServerPlugin implements LanguageServerPlugin { @Override public boolean supports(final ScriptLanguage language) { - return language instanceof ApposePythonScriptLanguage; + // NB: Without jedi, there is nothing to serve. + return language instanceof ApposePythonScriptLanguage && jediOffered(); } @Override public LanguageServer create(final Environment environment) { - final boolean jedi = jediOffered(); - return new ApposePythonLanguageServer(new PythonSyntaxLanguageServer(), - this::environment, jedi ? this::jedi : null, ApposePythonEnvironments.of( - workers)); + return new ApposePythonLanguageServer(this::environment, this::jedi, + ApposePythonEnvironments.of(workers)); } @Override diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java deleted file mode 100644 index c20f8c8..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServer.java +++ /dev/null @@ -1,539 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.nio.charset.StandardCharsets; -import java.util.ArrayDeque; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Deque; -import java.util.HashMap; -import java.util.LinkedHashSet; -import java.util.List; -import java.util.Locale; -import java.util.Map; -import java.util.Set; -import java.util.regex.Matcher; -import java.util.regex.Pattern; - -import org.eclipse.lsp4j.CompletionItem; -import org.eclipse.lsp4j.CompletionItemKind; -import org.eclipse.lsp4j.CompletionList; -import org.scijava.code.lsp.jvm.ScriptLanguageServer; -import org.treesitter.TSNode; -import org.treesitter.TSParser; -import org.treesitter.TSTree; -import org.treesitter.TreeSitterPython; - -/** - * An in-process language server for Python scripts run via Appose, from the - * syntax alone: tree-sitter parses the script at the caret, so it answers - * instantly, with no Python needed. It completes names in scope (including - * the script's parameters and {@code task}, as the session declares them; see - * {@link ApposePythonDialect}), builtins and keywords; and, after a dot, the - * names used after dots elsewhere in the script. - *

    - * It knows nothing of types, though: a Python language server, e.g. - * jedi-language-server, knows more (see {@link ApposePythonLanguageServer}). - *

    - * - * @author Gabriel Selzer - */ -public class PythonSyntaxLanguageServer extends ScriptLanguageServer { - - /** Identifier marking the caret position in the parsed source. */ - static final String CARET = "__scijava_caret__"; - - /** Python keywords, including soft keywords. */ - private static final List KEYWORDS = Arrays.asList("False", "None", - "True", "and", "as", "assert", "async", "await", "break", "case", "class", - "continue", "def", "del", "elif", "else", "except", "finally", "for", - "from", "global", "if", "import", "in", "is", "lambda", "match", - "nonlocal", "not", "or", "pass", "raise", "return", "try", "while", - "with", "yield"); - - /** Python's built-in functions, types and constants. */ - private static final List BUILTINS = Arrays.asList("abs", "aiter", - "all", "anext", "any", "ascii", "bin", "bool", "breakpoint", "bytearray", - "bytes", "callable", "chr", "classmethod", "compile", "complex", - "delattr", "dict", "dir", "divmod", "enumerate", "eval", "exec", - "filter", "float", "format", "frozenset", "getattr", "globals", - "hasattr", "hash", "help", "hex", "id", "input", "int", "isinstance", - "issubclass", "iter", "len", "list", "locals", "map", "max", - "memoryview", "min", "next", "object", "oct", "open", "ord", "pow", - "print", "property", "range", "repr", "reversed", "round", "set", - "setattr", "slice", "sorted", "staticmethod", "str", "sum", "super", - "tuple", "type", "vars", "zip", "__name__", "__file__", "Ellipsis", - "NotImplemented", "Exception", "BaseException", "ValueError", - "TypeError", "KeyError", "IndexError", "RuntimeError", "AttributeError", - "ImportError", "OSError", "FileNotFoundError", "StopIteration", - "NotImplementedError", "ZeroDivisionError"); - - private static final Pattern IMPORT_LINE = Pattern.compile( - "^\\s*(from\\s+\\S*(\\s+import\\b.*)?|import\\s.*)$"); - - /** Node types that open a new scope. */ - private static final Set SCOPES = new LinkedHashSet<>(Arrays.asList( - "function_definition", "class_definition", "lambda", "list_comprehension", - "set_comprehension", "dictionary_comprehension", "generator_expression")); - - /** What the caret calls for. */ - enum Kind { - /** Nothing: the caret is in a comment, string or directive. */ - NONE, - /** A name, e.g. a variable, function or keyword. */ - NAME, - /** An attribute of the expression before a dot. */ - ATTRIBUTE, - /** A module or name in an import statement. */ - IMPORT - } - - /** The completion context at the caret, as {@link #analyze} finds it. */ - static final class Context { - - /** What the caret calls for. */ - final Kind kind; - - /** The partial identifier being typed (possibly empty). */ - final String prefix; - - /** Offset where the partial identifier starts. */ - final int replaceStart; - - /** For {@link Kind#ATTRIBUTE}: the source of the expression, or null. */ - final String receiver; - - /** Names in scope at the caret, innermost first. */ - final Set names; - - /** - * The types names in scope are annotated with, as written (e.g. - * {@code numpy.ndarray} for {@code image: numpy.ndarray}), by name. - */ - final Map types; - - /** For {@link Kind#ATTRIBUTE}: attributes used on the same receiver. */ - final Set attributesUsed; - - private Context(final Kind kind, final String prefix, - final int replaceStart, final String receiver, final Set names, - final Map types, final Set attributesUsed) - { - this.kind = kind; - this.prefix = prefix; - this.replaceStart = replaceStart; - this.receiver = receiver; - this.names = names; - this.types = types; - this.attributesUsed = attributesUsed; - } - } - - public PythonSyntaxLanguageServer() { - super(null); - } - - @Override - protected CompletionList complete(final Document doc, final int caret) { - final Context ctx = analyze(doc.text(), caret); - if (ctx.kind == Kind.NONE || ctx.kind == Kind.NAME && ctx.prefix - .isEmpty()) - { - return list(doc, caret, caret, new ArrayList<>()); - } - return list(doc, ctx.replaceStart, caret, localCompletions(ctx)); - } - - /** Analyzes the given script at the given caret offset. */ - static Context analyze(final String text, final int caret) { - final int lineStart = text.lastIndexOf('\n', caret - 1) + 1; - final String line = text.substring(lineStart, caret); - int start = caret; - while (start > lineStart && isIdentifierPart(text.charAt(start - 1))) { - start--; - } - final String prefix = text.substring(start, caret); - final String before = text.substring(lineStart, start); - final String textBefore = text.substring(0, start); - final String closers = closers(textBefore); - if (closers == null || line.trim().startsWith("#")) return none(prefix, - start); - if (!prefix.isEmpty() && Character.isDigit(prefix.charAt(0))) { - return none(prefix, start); - } - - final Kind kind; - String receiver = null; - if (IMPORT_LINE.matcher(before + prefix).matches()) kind = Kind.IMPORT; - else if (before.trim().endsWith(".") && !before.trim().endsWith("..")) { - receiver = receiver(before.substring(0, before.lastIndexOf('.'))); - if (receiver != null && receiver.matches("\\d+")) { - return none(prefix, start); // a number, e.g. "1." - } - kind = Kind.ATTRIBUTE; - } - else kind = Kind.NAME; - - // Parse the script up to the caret, with the partial identifier replaced - // by a placeholder, and any open brackets closed: a clean tree there. - final String source = textBefore + CARET + closers; - final Set names = new LinkedHashSet<>(); - final Map types = new HashMap<>(); - if (kind == Kind.NAME) namesInScope(source, textBefore.length(), names, - types); - final Set used = kind == Kind.ATTRIBUTE && receiver != null - ? attributesUsed(text, receiver, start) : new LinkedHashSet<>(); - return new Context(kind, prefix, start, receiver, names, types, used); - } - - /** For tests: the names found by the given parse, as a list. */ - static List names(final String text, final int caret) { - return new ArrayList<>(analyze(text, caret).names); - } - - // -- Helper methods: completions -- - - private static CompletionItem item(final String name, - final CompletionItemKind kind, final String detail) - { - final CompletionItem item = new CompletionItem(name); - item.setKind(kind); - item.setInsertText(name); - item.setDetail(detail); - return item; - } - - /** Completions from the local (syntactic) analysis alone. */ - private static List localCompletions(final Context ctx) { - final List out = new ArrayList<>(); - final String prefix = ctx.prefix; - if (ctx.kind == Kind.ATTRIBUTE) { - for (final String name : ctx.attributesUsed) { - if (matches(name, prefix)) out.add(item(name, CompletionItemKind.Text, - "Used elsewhere in this script")); - } - return out; - } - if (ctx.kind != Kind.NAME) return out; - for (final String name : ctx.names) { - if (matches(name, prefix)) out.add(item(name, - CompletionItemKind.Variable, ctx.types.get(name))); - } - for (final String name : BUILTINS) { - if (matches(name, prefix)) out.add(item(name, CompletionItemKind.Function, - "builtin")); - } - for (final String keyword : KEYWORDS) { - if (matches(keyword, prefix)) out.add(item(keyword, - CompletionItemKind.Keyword, null)); - } - return out; - } - - private static boolean matches(final String name, final String prefix) { - if (name.startsWith("__") && !prefix.startsWith("_")) return false; - return name.toLowerCase(Locale.ROOT).startsWith(prefix.toLowerCase( - Locale.ROOT)) && !name.equals(prefix); - } - - // -- Helper methods: analysis -- - - private static Context none(final String prefix, final int start) { - return new Context(Kind.NONE, prefix, start, null, new LinkedHashSet<>(), - new HashMap<>(), new LinkedHashSet<>()); - } - - /** - * Gets the closing brackets for those left open at the end of - * {@code text}, or null if the text ends inside a string or comment. - */ - static String closers(final String text) { - final Deque open = new ArrayDeque<>(); - int i = 0; - final int n = text.length(); - while (i < n) { - final char c = text.charAt(i); - if (c == '#') { - final int end = text.indexOf('\n', i); - if (end < 0) return null; - i = end + 1; - } - else if (c == '\'' || c == '"') { - final boolean triple = text.startsWith(String.valueOf(c) + c + c, i); - final String quote = triple ? String.valueOf(c) + c + c : String - .valueOf(c); - int j = i + quote.length(); - while (true) { - if (j >= n) return null; - final char d = text.charAt(j); - if (d == '\\') j += 2; - else if (!triple && d == '\n') return null; // unterminated line - else if (text.startsWith(quote, j)) break; - else j++; - } - i = j + quote.length(); - } - else { - if (c == '(') open.push(')'); - else if (c == '[') open.push(']'); - else if (c == '{') open.push('}'); - else if ((c == ')' || c == ']' || c == '}') && !open.isEmpty() && open - .peek() == c) open.pop(); - i++; - } - } - final StringBuilder sb = new StringBuilder(); - for (final char c : open) sb.append(c); - return sb.toString(); - } - - /** - * Gets the source of the expression ending the given text, e.g. - * {@code np.random} in {@code x = np.random}, or null if there is none. - */ - static String receiver(final String text) { - int i = text.length(); - while (i > 0 && Character.isWhitespace(text.charAt(i - 1))) i--; - final int end = i; - int depth = 0; - while (i > 0) { - final char c = text.charAt(i - 1); - if (c == ')' || c == ']' || c == '}') depth++; - else if (c == '(' || c == '[' || c == '{') { - if (depth == 0) break; - depth--; - } - else if (depth == 0 && !isIdentifierPart(c) && c != '.') break; - i--; - } - final String receiver = text.substring(i, end).trim(); - return receiver.isEmpty() ? null : receiver; - } - - /** Attributes used on {@code receiver} anywhere in the script. */ - private static Set attributesUsed(final String text, - final String receiver, final int exclude) - { - final Set used = new LinkedHashSet<>(); - // NB: Only spaces around the dot: "a.\nprint" is not an attribute. - final Matcher m = Pattern.compile("(? names, final Map types) - { - final byte[] bytes = source.getBytes(StandardCharsets.UTF_8); - final int caretByte = source.substring(0, caret).getBytes( - StandardCharsets.UTF_8).length; - final TSParser parser = new TSParser(); - try { - parser.setLanguage(new TreeSitterPython()); - final TSTree tree = parser.parseString(null, source); - final TSNode root = tree.getRootNode(); - TSNode node = root.getNamedDescendantForByteRange(caretByte, caretByte + - CARET.length()); - // Walk outward through the enclosing scopes, innermost first. - boolean inFunction = false; - for (; node != null && !node.isNull(); node = node.getParent()) { - final String type = node.getType(); - if (!SCOPES.contains(type)) continue; - // NB: Class bodies are not visible from their methods. - if (type.equals("class_definition") && inFunction) continue; - if (type.equals("function_definition") || type.equals("lambda")) { - inFunction = true; - } - final TSNode params = node.getChildByFieldName("parameters"); - if (params != null && !params.isNull()) { - addTargets(params, bytes, names); - } - final TSNode body = node.getChildByFieldName("body"); - if (type.endsWith("comprehension") || type.equals( - "generator_expression")) - { - collectBindings(node, bytes, names, types, true); - } - else if (body != null && !body.isNull() && !type.equals("lambda")) { - collectBindings(body, bytes, names, types, false); - } - } - collectBindings(root, bytes, names, types, false); - } - catch (final RuntimeException | LinkageError exc) { - // NB: Native parser unavailable or failed: no names, then. - } - finally { - parser.close(); - } - names.remove(CARET); - } - - /** - * Collects the names bound by statements within {@code node}, without - * descending into nested scopes (whose own names are bound here, though); - * and the types annotated names are declared with. - */ - private static void collectBindings(final TSNode node, final byte[] bytes, - final Set names, final Map types, - final boolean comprehension) - { - final int n = node.getNamedChildCount(); - for (int i = 0; i < n; i++) { - final TSNode child = node.getNamedChild(i); - final String type = child.getType(); - switch (type) { - case "function_definition": - case "class_definition": - addIdentifier(child.getChildByFieldName("name"), bytes, names); - continue; // NB: A nested scope. - case "lambda": - case "list_comprehension": - case "set_comprehension": - case "dictionary_comprehension": - case "generator_expression": - if (!comprehension) continue; // NB: A nested scope. - break; - case "assignment": - addType(child, bytes, types); - addTargets(child.getChildByFieldName("left"), bytes, names); - break; - case "augmented_assignment": - case "for_statement": - case "for_in_clause": - addTargets(child.getChildByFieldName("left"), bytes, names); - break; - case "named_expression": - addIdentifier(child.getChildByFieldName("name"), bytes, names); - break; - case "as_pattern_target": - addTargets(child, bytes, names); - break; - case "import_statement": - addImported(child, bytes, names, true); - continue; - case "import_from_statement": - addImported(child, bytes, names, false); - continue; - default: - } - collectBindings(child, bytes, names, types, comprehension); - } - } - - /** Adds the type of an annotated assignment, e.g. {@code x: int = 1}. */ - private static void addType(final TSNode assignment, final byte[] bytes, - final Map types) - { - final TSNode left = assignment.getChildByFieldName("left"); - final TSNode type = assignment.getChildByFieldName("type"); - if (left == null || left.isNull() || !left.getType().equals( - "identifier") || type == null || type.isNull()) return; - types.putIfAbsent(text(left, bytes), text(type, bytes)); - } - - /** Adds the names an import statement binds. */ - private static void addImported(final TSNode statement, final byte[] bytes, - final Set names, final boolean plainImport) - { - final int n = statement.getNamedChildCount(); - for (int i = 0; i < n; i++) { - if (!"name".equals(statement.getFieldNameForNamedChild(i))) continue; - final TSNode child = statement.getNamedChild(i); - if (child.getType().equals("aliased_import")) { - addIdentifier(child.getChildByFieldName("alias"), bytes, names); - } - else if (child.getType().equals("dotted_name")) { - // "import a.b" binds a; "from x import y" binds y. - final int count = child.getNamedChildCount(); - if (count > 0) addIdentifier(child.getNamedChild(plainImport ? 0 - : count - 1), bytes, names); - } - } - } - - /** - * Adds the identifiers bound by an assignment target or parameter list, - * e.g. {@code a, (b, c)} or {@code x, y: int = 1, *args, **kwargs}, but not - * attributes or subscripts. - */ - private static void addTargets(final TSNode node, final byte[] bytes, - final Set names) - { - if (node == null || node.isNull()) return; - switch (node.getType()) { - case "identifier": - addIdentifier(node, bytes, names); - return; - case "attribute": - case "subscript": - case "type": - return; - case "default_parameter": - case "typed_default_parameter": - addIdentifier(node.getChildByFieldName("name"), bytes, names); - return; - case "typed_parameter": - // NB: Its name is its first child (e.g. x in "x: int"). - if (node.getNamedChildCount() > 0) addTargets(node.getNamedChild(0), - bytes, names); - return; - default: - for (int i = 0; i < node.getNamedChildCount(); i++) { - addTargets(node.getNamedChild(i), bytes, names); - } - } - } - - private static void addIdentifier(final TSNode node, final byte[] bytes, - final Set names) - { - if (node == null || node.isNull() || !node.getType().equals( - "identifier")) return; - names.add(text(node, bytes)); - } - - private static String text(final TSNode node, final byte[] bytes) { - return new String(bytes, node.getStartByte(), node.getEndByte() - node - .getStartByte(), StandardCharsets.UTF_8); - } - - private static boolean isIdentifierPart(final char c) { - return Character.isLetterOrDigit(c) || c == '_'; - } -} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java index 89715b6..b61f1df 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java @@ -75,9 +75,9 @@ import org.scijava.code.lsp.Environment; /** - * Tests {@link ApposePythonLanguageServer}, with fake servers: how it combines - * jedi and the syntax-only server, and moves scripts between jedis as where - * they run changes. + * Tests {@link ApposePythonLanguageServer}, with fake jedis: how it routes + * each script to the jedi of where it runs, and moves scripts between jedis + * as that changes. * * @author Gabriel Selzer */ @@ -93,12 +93,10 @@ public class ApposePythonLanguageServerTest { /** Where scripts run when their text does not say (e.g. once built). */ private volatile String fallback = "A"; - private Fake syntax; private ApposePythonLanguageServer server; @Before public void setUp() { - syntax = new Fake("syntax", "max", "min"); server = server(env -> { final Fake jedi = new Fake("jedi:" + env.interpreter(), "max", "map"); jedis.add(jedi); @@ -136,40 +134,39 @@ public void testRelocation() { assertEquals("jedi:C", jedis.get(3).name); assertEquals(Collections.singletonList("open 4 y = 1"), jedis.get(3).events); - // Closed: on the syntax server and its jedi, which is let go. + // Closed: on its jedi, which is let go. server.didClose(new DidCloseTextDocumentParams(new TextDocumentIdentifier( URI))); assertEquals(Arrays.asList("open 4 y = 1", "close", "shutdown"), jedis.get( 3).events); - assertEquals("close", syntax.events.get(syntax.events.size() - 1)); } @Test public void testCompletions() throws Exception { open("ma"); final CompletionList list = complete(); - // Jedi's, then the syntax server's for names jedi did not complete; each - // in its own order. - assertEquals(Arrays.asList("map", "max", "min"), labels(list)); - - // Resolved by the server each came from, with its own data. - final CompletionItem min = list.getItems().get(2); - final Object marker = min.getData(); - final CompletionItem resolved = server.resolveCompletionItem(min).get(10, + // Jedi's, as they are. + assertEquals(Arrays.asList("max", "map"), labels(list)); + + // Resolved by the jedi analyzing the script, with its own data. + final CompletionItem max = list.getItems().get(0); + final Object marker = max.getData(); + final CompletionItem resolved = server.resolveCompletionItem(max).get(10, TimeUnit.SECONDS); - assertEquals("syntax resolved min (data syntax)", resolved.getDetail()); + assertEquals("jedi:A resolved max (data jedi:A)", resolved.getDetail()); assertEquals(marker, resolved.getData()); - assertEquals("jedi:A resolved map (data jedi:A)", server - .resolveCompletionItem(list.getItems().get(0)).get(10, TimeUnit.SECONDS) - .getDetail()); + + // Moved: resolved by the jedi analyzing it now. + change("#env=B\nma", 2); + assertEquals("jedi:B resolved max (data jedi:B)", server + .resolveCompletionItem(complete().getItems().get(0)).get(10, + TimeUnit.SECONDS).getDetail()); } @Test public void testDiagnostics() { open("x = 1"); final Fake a = jedis.get(0); - // The syntax server's are not passed on (jedi's are the problems). - syntax.publish(URI, 1, "syntax"); a.publish(URI, 1, "from A"); assertEquals(Collections.singletonList("from A"), messages()); @@ -180,27 +177,12 @@ public void testDiagnostics() { assertEquals(Arrays.asList("from A", "from B"), messages()); } - @Test - public void testWithoutJedi() throws Exception { - server = server(null); - open("ma"); - // The syntax server's answers, as they are. - final CompletionList list = complete(); - assertEquals(Arrays.asList("max", "min"), labels(list)); - assertEquals("syntax", list.getItems().get(0).getData()); - assertEquals("syntax resolved max (data syntax)", server - .resolveCompletionItem(list.getItems().get(0)).get(10, TimeUnit.SECONDS) - .getDetail()); - syntax.publish(URI, 1, "syntax"); - assertEquals(Collections.singletonList("syntax"), messages()); - } - // -- Helper methods -- private ApposePythonLanguageServer server( final Function jedi) { - final ApposePythonLanguageServer s = new ApposePythonLanguageServer(syntax, + final ApposePythonLanguageServer s = new ApposePythonLanguageServer( doc -> new Environment(doc.getText().contains("#env=B") ? "B" : fallback, null, null), jedi, null); s.connect(new Client()); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java index 6562ce9..118ced8 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java @@ -30,7 +30,6 @@ package org.scijava.plugins.scripting.appose.python.completion; import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; @@ -38,12 +37,9 @@ import java.util.ArrayList; import java.util.Collections; import java.util.List; -import java.util.concurrent.TimeUnit; -import java.util.stream.Collectors; import net.imglib2.img.Img; -import org.eclipse.lsp4j.CompletionItem; import org.eclipse.lsp4j.TextDocumentItem; import org.junit.After; import org.junit.Before; @@ -52,7 +48,6 @@ import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.LanguageServerPlugin; -import org.scijava.code.lsp.ScriptSession; import org.scijava.code.lsp.ScriptDocument; import org.scijava.plugin.PluginService; import org.scijava.plugins.scripting.appose.python.ScriptEnvironment; @@ -61,10 +56,9 @@ import org.scijava.script.ScriptService; /** - * Tests the language servers of Python scripts run via Appose, without - * starting any server process: the syntax-only server - * ({@link PythonSyntaxLanguageServer}), and what {@link ApposePythonServerPlugin} - * tells jedi-language-server (the script as plain Python, and where it runs). + * Tests what {@link ApposePythonServerPlugin} tells jedi-language-server about + * Python scripts run via Appose (the script as plain Python, and where it + * runs), without starting any server process. * * @author Gabriel Selzer */ @@ -105,29 +99,6 @@ public void testServersAreOffered() { assertTrue(plugin.jediOffered()); } - @Test - public void testLocalCompletions() { - final String header = "#@ net.imglib2.img.Img image\n#@ double sigma\n"; - final List names = complete(header + - "def blur(img, s):\n result = img\n re"); - assertTrue(names.toString(), names.contains("result")); - assertTrue(names.contains("return")); - assertTrue(complete(header + "x = si").contains("sigma")); - assertTrue(complete(header + "pri").contains("print")); - assertTrue(complete(header + "ta").contains("task")); - // Declared with the types they have when the script runs. - assertEquals("float", detail(header + "x = si", "sigma")); - assertEquals("numpy.ndarray", detail(header + "x = ima", "image")); - assertEquals("appose.python_worker.Task", detail(header + "ta", "task")); - // Modules imported only to declare those: not completed. - assertFalse(complete(header + "nu").contains("numpy")); - assertFalse(complete(header + "ap").contains("appose")); - assertTrue(complete(header + "import numpy\nnu").contains("numpy")); - // Nothing on parameter lines, in strings or comments. - assertTrue(complete("#@ String na").isEmpty()); - assertTrue(complete("s = 'pri").isEmpty()); - } - @Test public void testScriptAsPython() { final String code = "#@ net.imglib2.img.Img image\n#@ double sigma\n" + @@ -317,30 +288,6 @@ private ScriptDocument document(final String code) { .scriptParameters(code), plugin.dialect()); } - /** Completes with the syntax-only server. */ - private List complete(final String code) { - return items(code).stream().map(CompletionItem::getLabel).collect( - Collectors.toList()); - } - - /** The detail of a completion with the syntax-only server, or null. */ - private String detail(final String code, final String label) { - return items(code).stream().filter(i -> i.getLabel().equals(label)) - .findFirst().map(CompletionItem::getDetail).orElse(null); - } - - private List items(final String code) { - try (ScriptSession session = ScriptSession.owning( - new PythonSyntaxLanguageServer(), language, this::document)) - { - return session.completion(code, code.length()).get(10, - TimeUnit.SECONDS).getItems(); - } - catch (final Exception exc) { - throw new RuntimeException(exc); - } - } - private static void delete(final java.nio.file.Path dir) throws Exception { try (java.util.stream.Stream paths = java.nio.file.Files.walk(dir)) diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServerTest.java deleted file mode 100644 index 05eafd6..0000000 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonSyntaxLanguageServerTest.java +++ /dev/null @@ -1,145 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; - -import java.util.List; - -import org.junit.Test; -import org.scijava.plugins.scripting.appose.python.completion.PythonSyntaxLanguageServer.Context; -import org.scijava.plugins.scripting.appose.python.completion.PythonSyntaxLanguageServer.Kind; - -/** - * Tests how {@link PythonSyntaxLanguageServer} analyzes scripts. - * - * @author Gabriel Selzer - */ -public class PythonSyntaxLanguageServerTest { - - @Test - public void testNamesInScope() { - final String code = "import numpy as np, os.path\n" + // - "from os import path as p, sep\n" + // - "x = 1\n" + // - "def f(a, b=2, *args, c: int = 3, **kw):\n" + // - " loc = 3\n" + // - " lo"; - final List names = PythonSyntaxLanguageServer.names(code, code.length()); - for (final String name : new String[] { "loc", "a", "b", "args", "c", "kw", - "f", "x", "np", "os", "p", "sep" }) - { - assertTrue(name + " in " + names, names.contains(name)); - } - // Innermost scope first. - assertTrue(names.indexOf("loc") < names.indexOf("x")); - assertFalse(names.contains(PythonSyntaxLanguageServer.CARET)); - } - - @Test - public void testScopeRules() { - // Class bodies are not visible from their methods. - final String cls = "class C:\n attr = 1\n def m(self):\n at"; - final List inMethod = PythonSyntaxLanguageServer.names(cls, cls.length()); - assertTrue(inMethod.contains("self")); - assertTrue(inMethod.contains("C")); - assertFalse(inMethod.contains("attr")); - // Functions' locals are not visible outside them. - final String fn = "def g():\n inner = 1\nin"; - assertFalse(PythonSyntaxLanguageServer.names(fn, fn.length()).contains("inner")); - // Comprehension, with, except and walrus targets. - final String comp = "[val for val in range(3) if va"; - assertTrue(PythonSyntaxLanguageServer.names(comp, comp.length()).contains("val")); - final String blocks = "with open(f) as fh:\n try:\n pass\n" + - " except E as err:\n if (n := 2):\n e"; - final List names = PythonSyntaxLanguageServer.names(blocks, blocks.length()); - assertTrue(names.toString(), names.contains("fh") && names.contains( - "err") && names.contains("n")); - // Unclosed brackets before the caret. - final String open = "def h(alpha):\n print(1, [2, al"; - assertTrue(PythonSyntaxLanguageServer.names(open, open.length()).contains("alpha")); - } - - @Test - public void testAttributeContext() { - final String code = "import numpy as np\nnp.random.ra"; - final Context ctx = PythonSyntaxLanguageServer.analyze(code, code.length()); - assertEquals(Kind.ATTRIBUTE, ctx.kind); - assertEquals("np.random", ctx.receiver); - assertEquals("ra", ctx.prefix); - assertEquals(code.length() - 2, ctx.replaceStart); - assertEquals("foo(x)[0]", PythonSyntaxLanguageServer.analyze("y = foo(x)[0].", 14) - .receiver); - // Attributes used elsewhere on the same receiver. - final String img = "img.shape\nimg.dtype\nmyimg.size\nimg.sh"; - final Context used = PythonSyntaxLanguageServer.analyze(img, img.length()); - assertTrue(used.attributesUsed.contains("shape")); - assertTrue(used.attributesUsed.contains("dtype")); - assertFalse(used.attributesUsed.contains("size")); - // A dot ending a line is not followed by an attribute on the next one. - final String nextLine = "a = numpy.ndarray(3)\na.\nprint(a.shape)\n"; - final Context dot = PythonSyntaxLanguageServer.analyze(nextLine, nextLine.indexOf( - "a.\n") + 2); - assertTrue(dot.attributesUsed.contains("shape")); - assertFalse(dot.attributesUsed.toString(), dot.attributesUsed.contains( - "print")); - } - - @Test - public void testNoCompletion() { - for (final String code : new String[] { "s = 'np.ra", "# np.ra", - "x = 1 # np.", "d = \"\"\"doc\nmore np.", "#@ String na", "x = 1.", - "f\"{x}.y" }) - { - assertEquals(code, Kind.NONE, PythonSyntaxLanguageServer.analyze(code, code.length()) - .kind); - } - // But after a closed string, completion works again. - final String closed = "s = 'a.b'\nx = s.up"; - assertEquals(Kind.ATTRIBUTE, PythonSyntaxLanguageServer.analyze(closed, closed.length()) - .kind); - } - - @Test - public void testImports() { - assertEquals(Kind.IMPORT, PythonSyntaxLanguageServer.analyze("import nu", 9).kind); - assertEquals(Kind.IMPORT, PythonSyntaxLanguageServer.analyze("from scipy.nd", 13).kind); - assertEquals(Kind.IMPORT, PythonSyntaxLanguageServer.analyze( - "from scipy import ndim", 22).kind); - } - - @Test - public void testUnicode() { - final String code = "s = '\u00e9t\u00e9'\nvaleur = 1\nval"; - assertTrue(PythonSyntaxLanguageServer.names(code, code.length()).contains("valeur")); - } -} From 4cbc7e3b3dfbaf9314f565ae9c8cb5df8c6cf5f7 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Tue, 13 Oct 2026 15:57:37 -0500 Subject: [PATCH 24/37] Choose each script's environment as it is analyzed ApposePythonEnvironments now gives the environment of each script as code-lsp asks, before each request, so the script moves to another jedi process once its environment is built or its overlay is ready. This replaces the router that relocated scripts between servers, and the listeners that told it to; PythonCompletionServer becomes ApposePythonLanguageServer in its place. Empty answers are no longer reused, as Jedi may know more once asked again (e.g. after a new dependency's overlay is ready). Everything but the two plugins is now package-private. Co-Authored-By: Claude Opus 5.5 --- README.md | 6 +- .../completion/ApposePythonDialect.java | 8 +- .../completion/ApposePythonEnvironments.java | 69 +-- .../ApposePythonLanguageServer.java | 541 ++++++++++-------- .../completion/ApposePythonServerPlugin.java | 37 +- .../python/completion/DependencyOverlays.java | 29 +- .../completion/PythonCompletionServer.java | 474 --------------- .../ApposePythonLanguageServerTest.java | 400 ++++++------- .../completion/ApposePythonServersTest.java | 18 +- .../PythonCompletionServerTest.java | 294 ---------- 10 files changed, 532 insertions(+), 1344 deletions(-) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java delete mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java diff --git a/README.md b/README.md index d9919fd..4377e5c 100644 --- a/README.md +++ b/README.md @@ -138,9 +138,9 @@ script's own line numbers. ## Code completion The Script Editor completes code in these scripts with a language server -(`ApposePythonLanguageServer`, offered to `scijava-code-lsp`'s -`LanguageServerService` by `ApposePythonServerPlugin`, so any other LSP client -of it gets the same): [jedi-language-server](https://github.com/pappasam/jedi-language-server) +(offered to `scijava-code-lsp`'s `LanguageServerService` by +`ApposePythonServerPlugin`, so any other LSP client of it gets the same): +[jedi-language-server](https://github.com/pappasam/jedi-language-server) (built on [Jedi](https://jedi.readthedocs.io/)). It completes names in scope, the script's `#@` parameters, `task`, builtins and keywords, and members of objects and modules; and it gives function signatures (with parameter names diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index b512c54..e758e90 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -44,16 +44,14 @@ import org.scijava.code.lsp.ScriptDialect; /** - * How Python scripts run via Appose see their script parameters: as the - * Python values the Java values become (see - * {@code ApposePythonScriptEngine#marshal}), plus Appose's {@code task}. + * Describes how to turn Appose python scripts into plain python code. * * @author Gabriel Selzer */ -public final class ApposePythonDialect implements ScriptDialect { +final class ApposePythonDialect implements ScriptDialect { /** The type of the {@code task} variable scripts can use. */ - public static final String TASK_TYPE = "appose.python_worker.Task"; + private static final String TASK_TYPE = "appose.python_worker.Task"; @Override public String runtimeType(final Class type) { diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java index 2f83048..990ca8c 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java @@ -42,16 +42,15 @@ import java.util.List; import java.util.Map; import java.util.WeakHashMap; -import java.util.concurrent.CopyOnWriteArrayList; import org.apposed.appose.Appose; import org.apposed.appose.Environment; import org.apposed.appose.Service; import org.apposed.appose.util.Environments; +import org.eclipse.lsp4j.TextDocumentItem; import org.scijava.plugins.scripting.appose.python.ScriptEnvironment; import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; -import org.scijava.plugins.scripting.appose.python._internal.BuildListener; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; /** @@ -99,19 +98,8 @@ protected boolean removeEldestEntry( private DependencyOverlays overlays; - /** Told when where scripts run may have changed. */ - private final List listeners = new CopyOnWriteArrayList<>(); - private ApposePythonEnvironments(final ResidentWorkerService workers) { this.workers = workers; - // NB: A script run builds its environment: its Python is known then. - if (workers != null) workers.addBuildListener(new BuildListener() { - - @Override - public void buildFinished(final String envName, final Throwable error) { - if (error == null) changed(); - } - }); } /** Gets those of a context (by its resident worker service). */ @@ -122,43 +110,43 @@ static ApposePythonEnvironments of(final ResidentWorkerService workers) { } /** - * Gets where a script runs, for its language server: its environment's - * Python (or null, if not built), and its overlay (if any, and ready). + * Derives an {@link org.scijava.code.lsp.Environment} for an Appose script. + *

    + * The {@code python} executable is found using the script's text and file + * path. For convenience, packages declared in the script but not yet + * installed (i.e. between the time the dependency is added to the script + * and the time when it is run) are provided to the language server in the + * overlay. + *

    * - * @param uri The script's URI (a file path, if saved). - * @param settled Whether the dependencies are known not to be changing - * (e.g. on opening the script). + * @param script the script + * @return an {@link org.scijava.code.lsp.Environment} where: + *
      + *
    • the interpreter is the Python executable of the script's + * declared environment, if that has already been built, + * or {@code null} otherwise;
      • + *
    • the paths hold the overlay, once installed; none if no package + * is missing, or until it is installed (which starts once the + * script's dependencies stop changing, or at once if the script was + * just opened).
      • + *
    */ - org.scijava.code.lsp.Environment environment(final String text, - final String uri, final boolean settled) - { - final String path = path(uri); + org.scijava.code.lsp.Environment environment(final TextDocumentItem script) { + final String text = script.getText(); + final String path = path(script.getUri()); final String python = scriptPython(text, path); + final boolean settled = script.getVersion() <= 1; + // NB: An alternative design could have all the script's dependencies + // in a constant overlay, but that would have the effect of installing them + // twice - once for the overlay and once for the script's environment. final String overlay = overlay(text, path, python, settled); return new org.scijava.code.lsp.Environment(python, overlay == null ? null : Collections.singletonList(overlay), null); } - /** - * Adds what to do when where scripts run may have changed without them - * changing: when an environment is built, or an overlay is ready. - */ - void addListener(final Runnable listener) { - listeners.add(listener); - } - - void removeListener(final Runnable listener) { - listeners.remove(listener); - } - - private void changed() { - for (final Runnable listener : listeners) listener.run(); - } - /** Sets the dependency overlays, e.g. to use fake ones in tests. */ synchronized void setOverlays(final DependencyOverlays overlays) { this.overlays = overlays; - if (overlays != null) overlays.setOnReady(this::changed); } /** The file path of a {@code file:} URI, or null. */ @@ -210,7 +198,6 @@ private synchronized DependencyOverlays overlays() { dir) -> installOverlay(workers, python, dependencies, dir), new File( Environments.apposeEnvsDir(), TOOLING_ENV + "-overlays"), OVERLAY_SETTLE, System::currentTimeMillis); - overlays.setOnReady(this::changed); return overlays; } @@ -243,7 +230,7 @@ private static String installOverlay(final ResidentWorkerService workers, * earlier one. Completion never builds it. Returns null if not built. */ String scriptPython(final String text, final String path) { - // NB: Called on every keystroke, so remember what was found (but not + // NB: Called before each request, so remember what was found (but not // what was not: the environment may be built at any moment). final String key; try { diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java index ef98348..91d4a44 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java @@ -6,13 +6,13 @@ * %% * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: - * + * * 1. Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. - * + * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE @@ -30,16 +30,19 @@ package org.scijava.plugins.scripting.appose.python.completion; import java.util.ArrayList; -import java.util.Collections; +import java.util.Comparator; +import java.util.LinkedHashMap; import java.util.List; +import java.util.Locale; import java.util.Map; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; -import java.util.function.Function; +import java.util.function.LongSupplier; +import java.util.regex.Pattern; import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionItemKind; import org.eclipse.lsp4j.CompletionList; -import org.eclipse.lsp4j.CompletionOptions; import org.eclipse.lsp4j.CompletionParams; import org.eclipse.lsp4j.DidChangeConfigurationParams; import org.eclipse.lsp4j.DidChangeTextDocumentParams; @@ -52,74 +55,98 @@ import org.eclipse.lsp4j.InitializeParams; import org.eclipse.lsp4j.InitializeResult; import org.eclipse.lsp4j.InitializedParams; -import org.eclipse.lsp4j.MessageActionItem; -import org.eclipse.lsp4j.MessageParams; -import org.eclipse.lsp4j.PublishDiagnosticsParams; -import org.eclipse.lsp4j.ServerCapabilities; -import org.eclipse.lsp4j.ShowMessageRequestParams; import org.eclipse.lsp4j.SignatureHelp; -import org.eclipse.lsp4j.SignatureHelpOptions; import org.eclipse.lsp4j.SignatureHelpParams; -import org.eclipse.lsp4j.TextDocumentContentChangeEvent; -import org.eclipse.lsp4j.TextDocumentIdentifier; -import org.eclipse.lsp4j.TextDocumentItem; -import org.eclipse.lsp4j.TextDocumentSyncKind; import org.eclipse.lsp4j.jsonrpc.messages.Either; import org.eclipse.lsp4j.services.LanguageClient; import org.eclipse.lsp4j.services.LanguageClientAware; import org.eclipse.lsp4j.services.LanguageServer; import org.eclipse.lsp4j.services.TextDocumentService; import org.eclipse.lsp4j.services.WorkspaceService; -import org.scijava.code.lsp.Environment; +import org.scijava.code.lsp.jvm.Callable; +import org.scijava.code.lsp.Positions; /** - * The language server of Python scripts run via Appose: jedi-language-server, - * analyzing each script against where it runs (its environment's Python, once - * built, and an overlay of the packages it declares but its environment lacks; - * see {@link ApposePythonEnvironments}). + * The language server of Python scripts run via Appose (see + * {@link ApposePythonServerPlugin}): a Python language server, i.e. + * jedi-language-server, whose completions are made to work well in an editor + * completing as the user types: + *
      + *
    • Asks about a word's first character only, and reuses the answer + * (filtered) as the word is typed on, for a while.
      • + *
    • Cancels a question still pending once another is asked about the same + * document.
      • + *
    • Once a word is begun, resolves the first completions, and makes those + * with signatures (e.g. {@code def dumps(obj, *, skipkeys=False) -> str}) + * {@link Callable}s: snippets with their parameters and types. (Not in + * imports: an import names things, rather than calling them.)
      • + *
    *

    - * Jedi learns where scripts run only when it starts, so each script is - * analyzed by a jedi of its own environment. As that changes (as the script's - * metadata is edited, or its environment is built), the script moves to - * another jedi. + * Everything else is passed on to the server as is. *

    * * @author Gabriel Selzer */ -public class ApposePythonLanguageServer implements LanguageServer, +class ApposePythonLanguageServer implements LanguageServer, TextDocumentService, WorkspaceService, LanguageClientAware { - private final Function environment; - private final Function jedi; - private final ApposePythonEnvironments environments; - private final Runnable relocateAll = this::relocateAll; + /** How long an answer may be reused, in milliseconds. */ + private static final long ANSWER_LIFETIME = 20_000; + + /** An import statement being typed. */ + private static final Pattern IMPORT = Pattern.compile( + "^\\s*(from|import)\\s"); - /** The open documents, by URI. */ - private final Map docs = new ConcurrentHashMap<>(); + /** The Python language server. */ + private final LanguageServer inner; - private volatile LanguageClient client; + private final int signatures; + private final LongSupplier clock; + + /** The open documents' texts, by URI. */ + private final Map texts = new ConcurrentHashMap<>(); + + /** Answers, by question; least recently used first. */ + private final Map answers = new LinkedHashMap( + 16, 0.75f, true) + { + + @Override + protected boolean removeEldestEntry(final Map.Entry e) { + return size() > 32; + } + }; + + /** The latest question about each document, by URI. */ + private final Map pending = new ConcurrentHashMap<>(); /** - * @param environment Gets where a script runs. - * @param jedi Makes a jedi for scripts running in an environment. - * @param environments Tells when where scripts run may have changed without - * them changing (e.g. as environments are built); or null. + * @param inner The Python language server. + * @param signatures How many completions to resolve for their signatures, + * once a word is begun. */ - ApposePythonLanguageServer( - final Function environment, - final Function jedi, - final ApposePythonEnvironments environments) + ApposePythonLanguageServer(final LanguageServer inner, + final int signatures) { - this.environment = environment; - this.jedi = jedi; - this.environments = environments; - if (environments != null) environments.addListener(relocateAll); + this(inner, signatures, System::currentTimeMillis); } + ApposePythonLanguageServer(final LanguageServer inner, final int signatures, + final LongSupplier clock) + { + this.inner = inner; + this.signatures = signatures; + this.clock = clock; + } + + // -- LanguageClientAware -- + @Override - public void connect(final LanguageClient c) { - client = c; + public void connect(final LanguageClient client) { + if (inner instanceof LanguageClientAware) { + ((LanguageClientAware) inner).connect(client); + } } // -- LanguageServer -- @@ -128,30 +155,22 @@ public void connect(final LanguageClient c) { public CompletableFuture initialize( final InitializeParams params) { - final ServerCapabilities capabilities = new ServerCapabilities(); - capabilities.setTextDocumentSync(TextDocumentSyncKind.Full); - capabilities.setCompletionProvider(new CompletionOptions(true, - Collections.singletonList("."))); - capabilities.setSignatureHelpProvider(new SignatureHelpOptions(Collections - .singletonList("("))); - capabilities.setHoverProvider(true); - return CompletableFuture.completedFuture(new InitializeResult( - capabilities)); + return inner.initialize(params); + } + + @Override + public void initialized(final InitializedParams params) { + inner.initialized(params); } @Override public CompletableFuture shutdown() { - if (environments != null) environments.removeListener(relocateAll); - for (final String uri : new ArrayList<>(docs.keySet())) { - didClose(new DidCloseTextDocumentParams(new TextDocumentIdentifier( - uri))); - } - return CompletableFuture.completedFuture(null); + return inner.shutdown(); } @Override public void exit() { - // NB: The jedis are let go as their documents close. + inner.exit(); } @Override @@ -164,262 +183,302 @@ public WorkspaceService getWorkspaceService() { return this; } - // -- TextDocumentService: documents -- + // -- TextDocumentService -- @Override public void didOpen(final DidOpenTextDocumentParams params) { - final Doc doc = new Doc(params.getTextDocument()); - docs.put(doc.item.getUri(), doc); - synchronized (doc) { - doc.environment = environment.apply(doc.item); - doc.jedi = start(doc.environment); - doc.jedi.getTextDocumentService().didOpen(params); - } + texts.put(params.getTextDocument().getUri(), params.getTextDocument() + .getText()); + inner.getTextDocumentService().didOpen(params); } @Override public void didChange(final DidChangeTextDocumentParams params) { - final Doc doc = docs.get(params.getTextDocument().getUri()); - final List changes = params - .getContentChanges(); - if (doc == null || changes.isEmpty()) return; - synchronized (doc) { - // NB: Full sync: the last change is the whole text. - doc.item = new TextDocumentItem(doc.item.getUri(), doc.item - .getLanguageId(), params.getTextDocument().getVersion(), changes.get( - changes.size() - 1).getText()); - if (!relocate(doc)) doc.jedi.getTextDocumentService().didChange(params); + if (!params.getContentChanges().isEmpty()) { + texts.put(params.getTextDocument().getUri(), params.getContentChanges() + .get(params.getContentChanges().size() - 1).getText()); } + inner.getTextDocumentService().didChange(params); } @Override public void didClose(final DidCloseTextDocumentParams params) { - final Doc doc = docs.remove(params.getTextDocument().getUri()); - if (doc == null) return; - synchronized (doc) { - stop(doc, params.getTextDocument().getUri()); - } + texts.remove(params.getTextDocument().getUri()); + pending.remove(params.getTextDocument().getUri()); + inner.getTextDocumentService().didClose(params); } - @Override - public void didSave(final DidSaveTextDocumentParams params) { - // NB: Changes come with didChange. - } - - // -- TextDocumentService: requests -- - @Override public CompletableFuture, CompletionList>> completion(final CompletionParams params) { final String uri = params.getTextDocument().getUri(); - final LanguageServer j = jedi(uri); - if (j == null) return CompletableFuture.completedFuture(Either.forRight( - new CompletionList())); - return orNull(j.getTextDocumentService().completion(params)).thenApply( - answer -> Either.forRight(mark(list(answer), uri))); + final String text = texts.get(uri); + if (text == null) return inner.getTextDocumentService().completion( + params); + final int caret = Math.min(Positions.offset(text, params.getPosition()), + text.length()); + int start = caret; + while (start > 0 && Character.isJavaIdentifierPart(text.charAt(start - + 1))) start--; + final String word = text.substring(start, caret); + // Ask about the first character typed, if any: later keystrokes of the + // same word then reuse the answer (filtered). + final int ask = start + Math.min(1, word.length()); + final String line = text.substring(text.lastIndexOf('\n', start - 1) + 1, + start); + final boolean importing = IMPORT.matcher(line).find(); + final String key = uri + "\n" + text.substring(0, ask); + final long now = clock.getAsLong(); + Answer answer; + synchronized (answers) { + answer = answers.get(key); + // NB: Not an empty answer either: the server may know more by now (e.g. + // once the script's packages are, or its environment is built). + if (answer != null && (now - answer.time > ANSWER_LIFETIME || + answer.future.isCompletedExceptionally() || answer.isEmpty())) + { + answer = null; + } + } + if (answer == null) { + final CompletionParams p = new CompletionParams(params.getTextDocument(), + Positions.position(text, ask), params.getContext()); + final CompletableFuture, CompletionList>> raw = + inner.getTextDocumentService().completion(p); + final int n = word.isEmpty() || importing ? 0 : signatures; + answer = new Answer(raw, raw.thenCompose(list -> resolve(list, n)), now); + synchronized (answers) { + answers.put(key, answer); + } + final Answer previous = pending.put(uri, answer); + if (previous != null && previous != answer && !previous.future + .isDone()) + { + // NB: Cancelling the request tells the server ($/cancelRequest). + previous.future.cancel(true); + previous.raw.cancel(true); + } + } + return answer.future.thenApply(list -> Either.forRight(filter(list, + word))); + } + + @Override + public void didSave(final DidSaveTextDocumentParams params) { + inner.getTextDocumentService().didSave(params); } @Override public CompletableFuture resolveCompletionItem( final CompletionItem item) { - if (!(item.getData() instanceof Source)) { - return CompletableFuture.completedFuture(item); - } - final Source from = (Source) item.getData(); - final LanguageServer j = jedi(from.uri); - if (j == null) return CompletableFuture.completedFuture(item); - item.setData(from.data); - return orNull(j.getTextDocumentService().resolveCompletionItem(item)) - .thenApply(resolved -> { - final CompletionItem out = resolved != null ? resolved : item; - out.setData(from); - return out; - }); + return inner.getTextDocumentService().resolveCompletionItem(item); } @Override public CompletableFuture signatureHelp( final SignatureHelpParams params) { - final LanguageServer j = jedi(params.getTextDocument().getUri()); - return j == null ? CompletableFuture.completedFuture(null) : orNull(j - .getTextDocumentService().signatureHelp(params)); + return inner.getTextDocumentService().signatureHelp(params); } @Override public CompletableFuture hover(final HoverParams params) { - final LanguageServer j = jedi(params.getTextDocument().getUri()); - return j == null ? CompletableFuture.completedFuture(null) : orNull(j - .getTextDocumentService().hover(params)); + return inner.getTextDocumentService().hover(params); } // -- WorkspaceService -- @Override public void didChangeConfiguration(final DidChangeConfigurationParams p) { - // NB: Nothing to configure. + inner.getWorkspaceService().didChangeConfiguration(p); } @Override public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) { - // NB: Nothing watched. + inner.getWorkspaceService().didChangeWatchedFiles(p); } // -- Helper methods -- - /** The jedi analyzing a document, or null if none. */ - private LanguageServer jedi(final String uri) { - final Doc doc = docs.get(uri); - if (doc == null) return null; - synchronized (doc) { - return doc.jedi; - } - } - - /** Starts a jedi for scripts running in an environment. */ - private LanguageServer start(final Environment env) { - final LanguageServer server = jedi.apply(env); - if (server instanceof LanguageClientAware) { - ((LanguageClientAware) server).connect(new Forwarder(server)); + /** Resolves the first completions, making those with signatures callables. */ + private CompletableFuture resolve( + final Either, CompletionList> answer, final int n) + { + final List items = new ArrayList<>(answer == null + ? new ArrayList<>() : answer.isRight() ? answer.getRight().getItems() + : answer.getLeft()); + items.sort(Comparator.comparing(i -> i.getSortText() != null ? i + .getSortText() : i.getLabel())); + final int count = Math.min(n, items.size()); + final List> resolved = new ArrayList<>(); + for (int i = 0; i < count; i++) { + // NB: Unresolved, should resolving fail. + resolved.add(inner.getTextDocumentService().resolveCompletionItem(items + .get(i)).handle((r, error) -> r)); } - server.initialize(new InitializeParams()).thenRun(() -> server - .initialized(new InitializedParams())); - return server; - } - - /** Closes a document on its jedi, and lets that jedi go. */ - private static void stop(final Doc doc, final String uri) { - doc.jedi.getTextDocumentService().didClose(new DidCloseTextDocumentParams( - new TextDocumentIdentifier(uri))); - doc.jedi.shutdown(); - } - - /** - * Moves a document to a jedi of where its script runs now, if that changed. - * - * @return Whether it moved (and the new jedi has its current text). - */ - private boolean relocate(final Doc doc) { - final Environment env = environment.apply(doc.item); - if (env.equals(doc.environment)) return false; - // TODO: Settle how often this may happen while dependencies are edited - // (debounce?). - stop(doc, doc.item.getUri()); - doc.environment = env; - doc.jedi = start(env); - doc.jedi.getTextDocumentService().didOpen(new DidOpenTextDocumentParams( - new TextDocumentItem(doc.item.getUri(), doc.item.getLanguageId(), doc.item - .getVersion(), doc.item.getText()))); - return true; + return CompletableFuture.allOf(resolved.toArray(new CompletableFuture[0])) + .thenApply(x -> { + for (int i = 0; i < count; i++) { + final CompletionItem r = resolved.get(i).join(); + final CompletionItem callable = r == null ? null : callable(items + .get(i), r); + if (callable != null) items.set(i, callable); + } + return new CompletionList(false, items); + }); } /** - * Moves each open document to a jedi of where its script runs now, if that - * changed without the script changing (e.g. its environment was built). + * A completion as a callable, from its resolved signature (its detail); + * or null if it has none. */ - void relocateAll() { - for (final Doc doc : docs.values()) { - synchronized (doc) { - relocate(doc); - } + private static CompletionItem callable(final CompletionItem item, + final CompletionItem resolved) + { + final CompletionItemKind kind = item.getKind(); + if (kind != CompletionItemKind.Function && kind != CompletionItemKind.Method && + kind != CompletionItemKind.Class && kind != CompletionItemKind.Constructor) + { + return null; } + final String name = item.getInsertText() != null ? item.getInsertText() + : item.getLabel(); + if (resolved.getDetail() == null) return null; + final Callable parsed = parse(name, resolved.getDetail()); + if (parsed == null) return null; + final CompletionItem out = new Callable(name, parsed.params(), parsed + .returnType()).kind(CompletionItemKind.Method).detail(resolved + .getDetail()).toItem(); + out.setSortText(item.getSortText()); + out.setDocumentation(resolved.getDocumentation()); + out.setData(item.getData()); + return out; } - /** - * Marks each completion with its document, so that it is resolved by the - * jedi analyzing it. - */ - private static CompletionList mark(final CompletionList list, - final String uri) + /** The completions of the word typed so far. */ + private static CompletionList filter(final CompletionList list, + final String word) { - if (list == null) return new CompletionList(); + final String w = word.toLowerCase(Locale.ROOT); + final List out = new ArrayList<>(); for (final CompletionItem item : list.getItems()) { - item.setData(new Source(uri, item.getData())); + final String name = item.getFilterText() != null ? item.getFilterText() + : item.getLabel(); + if (name.startsWith("_") && !word.startsWith("_")) continue; + if (!name.toLowerCase(Locale.ROOT).startsWith(w) || name.equals(word)) { + continue; + } + out.add(item); } - return list; - } - - private static CompletionList list( - final Either, CompletionList> answer) - { - if (answer == null) return null; - return answer.isRight() ? answer.getRight() : new CompletionList(answer - .getLeft()); - } - - /** The answer; or null, if the server failed (e.g. it stopped). */ - private static CompletableFuture orNull( - final CompletableFuture answer) - { - return answer == null ? CompletableFuture.completedFuture(null) : answer - .exceptionally(error -> null); + return new CompletionList(false, out); } - /** An open document, and the jedi analyzing it. */ - private static final class Doc { + private static final class Answer { - private TextDocumentItem item; - private Environment environment; - private LanguageServer jedi; + private final CompletableFuture raw; + private final CompletableFuture future; + private final long time; - private Doc(final TextDocumentItem item) { - this.item = item; + private Answer(final CompletableFuture raw, + final CompletableFuture future, final long time) + { + this.raw = raw; + this.future = future; + this.time = time; } - } - - /** Remembers which document a completion is of, for resolving. */ - private static final class Source { - private final String uri; - private final Object data; - - private Source(final String uri, final Object data) { - this.uri = uri; - this.data = data; + /** Whether the server answered, with no completions. */ + private boolean isEmpty() { + final CompletionList list = future.getNow(null); + return list != null && list.getItems().isEmpty(); } } /** - * Passes the problems the jedi analyzing their document finds on to the - * client; and its messages. + * Parses a Python signature (as Jedi writes them): the parameters, with + * their annotations as types; and the return annotation. Returns null if it + * is not a signature. */ - private final class Forwarder implements LanguageClient { - - private final LanguageServer source; - - private Forwarder(final LanguageServer source) { - this.source = source; - } - - @Override - public void publishDiagnostics(final PublishDiagnosticsParams p) { - final LanguageClient c = client; - // NB: Only from the jedi analyzing the document now. - if (c != null && jedi(p.getUri()) == source) c.publishDiagnostics(p); + static Callable parse(final String name, + final String signature) + { + final int open = signature.indexOf('('); + if (open < 0) return null; + final int close = closing(signature, open); + if (close < 0) return null; + final List params = new ArrayList<>(); + for (final String part : split(signature.substring(open + 1, close))) { + final String p = part.trim(); + // NB: Bare * and / only mark keyword-only and positional-only ones. + if (p.isEmpty() || p.equals("*") || p.equals("/")) continue; + final int colon = topLevel(p, ':'), eq = topLevel(p, '='); + String paramName = p, type = null; + if (colon >= 0 && (eq < 0 || colon < eq)) { + paramName = p.substring(0, colon); + type = p.substring(colon + 1, eq < 0 ? p.length() : eq).trim(); + } + else if (eq >= 0) paramName = p.substring(0, eq); + paramName = paramName.trim().replaceFirst("^\\*{1,2}", ""); + params.add(new Callable.Param(paramName, type == null || type + .isEmpty() ? null : type)); } + final String rest = signature.substring(close + 1).trim(); + final String returns = rest.startsWith("->") ? rest.substring(2).trim() + : null; + return new Callable(name, params, returns == null || returns.isEmpty() + ? null : returns); + } - @Override - public void telemetryEvent(final Object object) {} - - @Override - public void showMessage(final MessageParams m) { - final LanguageClient c = client; - if (c != null) c.showMessage(m); + /** The index of the bracket closing the one at {@code open}, or -1. */ + private static int closing(final String s, final int open) { + int depth = 0; + char quote = 0; + for (int i = open; i < s.length(); i++) { + final char c = s.charAt(i); + if (quote != 0) { + if (c == '\\') i++; + else if (c == quote) quote = 0; + } + else if (c == '\'' || c == '"') quote = c; + else if (c == '(' || c == '[' || c == '{') depth++; + else if (c == ')' || c == ']' || c == '}') { + if (--depth == 0) return i; + } } + return -1; + } - @Override - public CompletableFuture showMessageRequest( - final ShowMessageRequestParams r) - { - return CompletableFuture.completedFuture(null); + /** Splits at the commas outside brackets and strings. */ + private static List split(final String s) { + final List parts = new ArrayList<>(); + int start = 0; + while (true) { + final int comma = topLevel(s.substring(start), ','); + if (comma < 0) break; + parts.add(s.substring(start, start + comma)); + start += comma + 1; } + parts.add(s.substring(start)); + return parts; + } - @Override - public void logMessage(final MessageParams m) { - final LanguageClient c = client; - if (c != null) c.logMessage(m); + /** The index of the first {@code target} outside brackets and strings. */ + private static int topLevel(final String s, final char target) { + int depth = 0; + char quote = 0; + for (int i = 0; i < s.length(); i++) { + final char c = s.charAt(i); + if (quote != 0) { + if (c == '\\') i++; + else if (c == quote) quote = 0; + } + else if (c == '\'' || c == '"') quote = c; + else if (c == '(' || c == '[' || c == '{') depth++; + else if (c == ')' || c == ']' || c == '}') depth--; + else if (c == target && depth == 0) return i; } + return -1; } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java index 7afe6fc..a13f0c9 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java @@ -29,7 +29,6 @@ package org.scijava.plugins.scripting.appose.python.completion; -import org.eclipse.lsp4j.TextDocumentItem; import org.eclipse.lsp4j.services.LanguageServer; import org.scijava.Priority; import org.scijava.code.lsp.Environment; @@ -43,20 +42,8 @@ import org.scijava.script.ScriptLanguage; /** - * Offers the language server of Python scripts run via Appose (see - * {@link ApposePythonLanguageServer}): jedi-language-server (see - * {@link JediServerProcessPlugin}), for members of objects and modules, - * signatures, documentation, hover and syntax errors. Once a script's - * environment is built (by running the script), jedi analyzes the script - * against it, so its packages are known; packages the script declares but its - * environment lacks are installed for completion only (see - * {@link DependencyOverlays}). Jedi sees the script as plain Python, its - * parameters declared with the types they have when it runs (e.g. images as - * NumPy arrays; see {@link #dialect()}). - *

    - * Completion never builds a script's environment, and never disturbs its - * worker. - *

    + * A {@link LanguageServerPlugin} for Appose-Python scripts, backed by + * jedi-language-server. * * @author Gabriel Selzer */ @@ -83,8 +70,8 @@ public boolean supports(final ScriptLanguage language) { @Override public LanguageServer create(final Environment environment) { - return new ApposePythonLanguageServer(this::environment, this::jedi, - ApposePythonEnvironments.of(workers)); + return new ApposePythonLanguageServer(servers.launch("python", + ApposePythonEnvironments.of(workers)::environment), SIGNATURES); } @Override @@ -96,20 +83,4 @@ public ScriptDialect dialect() { boolean jediOffered() { return servers.launch("python", Environment.NONE) != null; } - - /** - * Gets where a script runs, for jedi: its environment's Python (once built), - * and an overlay of the packages it declares but its environment lacks. - * On opening the script, its dependencies are settled. - */ - Environment environment(final TextDocumentItem document) { - return ApposePythonEnvironments.of(workers).environment(document - .getText(), document.getUri(), document.getVersion() <= 1); - } - - /** Makes a jedi for scripts running in an environment. */ - private LanguageServer jedi(final Environment environment) { - return new PythonCompletionServer(servers.launch("python", environment), - SIGNATURES); - } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java index 52b803d..8e8b131 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java @@ -69,11 +69,11 @@ * * @author Gabriel Selzer */ -public final class DependencyOverlays { +final class DependencyOverlays { /** Installs the missing packages of a script's environment. */ @FunctionalInterface - public interface Installer { + interface Installer { /** * Installs the packages among {@code dependencies} that the environment @@ -106,9 +106,6 @@ public interface Installer { /** The dependencies last seen for each environment, and since when. */ private final Map seen = new HashMap<>(); - /** Told when an overlay is ready. */ - private volatile Runnable onReady = () -> {}; - /** * @param installer Does the installing, on a background thread. * @param root Directory holding the overlays (one per environment). @@ -116,7 +113,7 @@ public interface Installer { * their missing packages are installed. * @param clock The current time, in milliseconds. */ - public DependencyOverlays(final Installer installer, final File root, + DependencyOverlays(final Installer installer, final File root, final long settleMillis, final LongSupplier clock) { this.installer = installer; @@ -125,14 +122,6 @@ public DependencyOverlays(final Installer installer, final File root, this.clock = clock; } - /** - * Sets what to do when an overlay becomes ready (on the installing - * thread), e.g. tell language servers to look again. - */ - public void setOnReady(final Runnable onReady) { - this.onReady = onReady == null ? () -> {} : onReady; - } - /** * Gets the overlay providing the packages the environment lacks, if it is * ready; starts preparing it once the dependencies have settled. Never @@ -145,7 +134,7 @@ public void setOnReady(final Runnable onReady) { * (e.g. the script was just opened), so need no time to settle. * @return The overlay directory, or null if none is needed or ready. */ - public String overlay(final String envName, final String python, + String overlay(final String envName, final String python, final List dependencies, final boolean settled) { if (dependencies == null || dependencies.isEmpty()) return null; @@ -167,20 +156,12 @@ public String overlay(final String envName, final String python, final File overlay = new File(root, envName); final CompletableFuture started = CompletableFuture.supplyAsync( () -> installer.install(python, dependencies, overlay), executor); + // NB: Once done, asking again gets the overlay. installs.put(key, started); - // NB: Once done, so that asking again gets the overlay. - started.thenAccept(dir -> { - if (dir != null) onReady.run(); - }); return null; } } - /** Stops the background thread. */ - public void dispose() { - executor.shutdownNow(); - } - private static final class Seen { private final String key; diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java deleted file mode 100644 index 2990bc9..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServer.java +++ /dev/null @@ -1,474 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.util.ArrayList; -import java.util.Comparator; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Locale; -import java.util.Map; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.ConcurrentHashMap; -import java.util.function.LongSupplier; -import java.util.regex.Pattern; - -import org.eclipse.lsp4j.CompletionItem; -import org.eclipse.lsp4j.CompletionItemKind; -import org.eclipse.lsp4j.CompletionList; -import org.eclipse.lsp4j.CompletionParams; -import org.eclipse.lsp4j.DidChangeConfigurationParams; -import org.eclipse.lsp4j.DidChangeTextDocumentParams; -import org.eclipse.lsp4j.DidChangeWatchedFilesParams; -import org.eclipse.lsp4j.DidCloseTextDocumentParams; -import org.eclipse.lsp4j.DidOpenTextDocumentParams; -import org.eclipse.lsp4j.DidSaveTextDocumentParams; -import org.eclipse.lsp4j.Hover; -import org.eclipse.lsp4j.HoverParams; -import org.eclipse.lsp4j.InitializeParams; -import org.eclipse.lsp4j.InitializeResult; -import org.eclipse.lsp4j.InitializedParams; -import org.eclipse.lsp4j.SignatureHelp; -import org.eclipse.lsp4j.SignatureHelpParams; -import org.eclipse.lsp4j.jsonrpc.messages.Either; -import org.eclipse.lsp4j.services.LanguageClient; -import org.eclipse.lsp4j.services.LanguageClientAware; -import org.eclipse.lsp4j.services.LanguageServer; -import org.eclipse.lsp4j.services.TextDocumentService; -import org.eclipse.lsp4j.services.WorkspaceService; -import org.scijava.code.lsp.jvm.Callable; -import org.scijava.code.lsp.Positions; - -/** - * Makes a Python language server's completions (e.g. jedi-language-server's) - * work well in an editor completing as the user types: - *
      - *
    • Asks about a word's first character only, and reuses the answer - * (filtered) as the word is typed on, for a while.
      • - *
    • Cancels a question still pending once another is asked about the same - * document.
      • - *
    • Once a word is begun, resolves the first completions, and makes those - * with signatures (e.g. {@code def dumps(obj, *, skipkeys=False) -> str}) - * {@link Callable}s: snippets with their parameters and types. (Not in - * imports: an import names things, rather than calling them.)
      • - *
    - *

    - * Everything else is passed on to the server as is. - *

    - * - * @author Gabriel Selzer - */ -public class PythonCompletionServer implements LanguageServer, - TextDocumentService, WorkspaceService, LanguageClientAware -{ - - /** How long an answer may be reused, in milliseconds. */ - private static final long ANSWER_LIFETIME = 20_000; - - /** An import statement being typed. */ - private static final Pattern IMPORT = Pattern.compile( - "^\\s*(from|import)\\s"); - - /** The Python language server. */ - private final LanguageServer inner; - - private final int signatures; - private final LongSupplier clock; - - /** The open documents' texts, by URI. */ - private final Map texts = new ConcurrentHashMap<>(); - - /** Answers, by question; least recently used first. */ - private final Map answers = new LinkedHashMap( - 16, 0.75f, true) - { - - @Override - protected boolean removeEldestEntry(final Map.Entry e) { - return size() > 32; - } - }; - - /** The latest question about each document, by URI. */ - private final Map pending = new ConcurrentHashMap<>(); - - /** - * @param inner The Python language server. - * @param signatures How many completions to resolve for their signatures, - * once a word is begun. - */ - public PythonCompletionServer(final LanguageServer inner, - final int signatures) - { - this(inner, signatures, System::currentTimeMillis); - } - - PythonCompletionServer(final LanguageServer inner, final int signatures, - final LongSupplier clock) - { - this.inner = inner; - this.signatures = signatures; - this.clock = clock; - } - - // -- LanguageClientAware -- - - @Override - public void connect(final LanguageClient client) { - if (inner instanceof LanguageClientAware) { - ((LanguageClientAware) inner).connect(client); - } - } - - // -- LanguageServer -- - - @Override - public CompletableFuture initialize( - final InitializeParams params) - { - return inner.initialize(params); - } - - @Override - public void initialized(final InitializedParams params) { - inner.initialized(params); - } - - @Override - public CompletableFuture shutdown() { - return inner.shutdown(); - } - - @Override - public void exit() { - inner.exit(); - } - - @Override - public TextDocumentService getTextDocumentService() { - return this; - } - - @Override - public WorkspaceService getWorkspaceService() { - return this; - } - - // -- TextDocumentService -- - - @Override - public void didOpen(final DidOpenTextDocumentParams params) { - texts.put(params.getTextDocument().getUri(), params.getTextDocument() - .getText()); - inner.getTextDocumentService().didOpen(params); - } - - @Override - public void didChange(final DidChangeTextDocumentParams params) { - if (!params.getContentChanges().isEmpty()) { - texts.put(params.getTextDocument().getUri(), params.getContentChanges() - .get(params.getContentChanges().size() - 1).getText()); - } - inner.getTextDocumentService().didChange(params); - } - - @Override - public void didClose(final DidCloseTextDocumentParams params) { - texts.remove(params.getTextDocument().getUri()); - pending.remove(params.getTextDocument().getUri()); - inner.getTextDocumentService().didClose(params); - } - - @Override - public CompletableFuture, CompletionList>> - completion(final CompletionParams params) - { - final String uri = params.getTextDocument().getUri(); - final String text = texts.get(uri); - if (text == null) return inner.getTextDocumentService().completion( - params); - final int caret = Math.min(Positions.offset(text, params.getPosition()), - text.length()); - int start = caret; - while (start > 0 && Character.isJavaIdentifierPart(text.charAt(start - - 1))) start--; - final String word = text.substring(start, caret); - // Ask about the first character typed, if any: later keystrokes of the - // same word then reuse the answer (filtered). - final int ask = start + Math.min(1, word.length()); - final String line = text.substring(text.lastIndexOf('\n', start - 1) + 1, - start); - final boolean importing = IMPORT.matcher(line).find(); - final String key = uri + "\n" + text.substring(0, ask); - final long now = clock.getAsLong(); - Answer answer; - synchronized (answers) { - answer = answers.get(key); - if (answer != null && (now - answer.time > ANSWER_LIFETIME || - answer.future.isCompletedExceptionally())) - { - answer = null; - } - } - if (answer == null) { - final CompletionParams p = new CompletionParams(params.getTextDocument(), - Positions.position(text, ask), params.getContext()); - final CompletableFuture, CompletionList>> raw = - inner.getTextDocumentService().completion(p); - final int n = word.isEmpty() || importing ? 0 : signatures; - answer = new Answer(raw, raw.thenCompose(list -> resolve(list, n)), now); - synchronized (answers) { - answers.put(key, answer); - } - final Answer previous = pending.put(uri, answer); - if (previous != null && previous != answer && !previous.future - .isDone()) - { - // NB: Cancelling the request tells the server ($/cancelRequest). - previous.future.cancel(true); - previous.raw.cancel(true); - } - } - return answer.future.thenApply(list -> Either.forRight(filter(list, - word))); - } - - @Override - public void didSave(final DidSaveTextDocumentParams params) { - inner.getTextDocumentService().didSave(params); - } - - @Override - public CompletableFuture resolveCompletionItem( - final CompletionItem item) - { - return inner.getTextDocumentService().resolveCompletionItem(item); - } - - @Override - public CompletableFuture signatureHelp( - final SignatureHelpParams params) - { - return inner.getTextDocumentService().signatureHelp(params); - } - - @Override - public CompletableFuture hover(final HoverParams params) { - return inner.getTextDocumentService().hover(params); - } - - // -- WorkspaceService -- - - @Override - public void didChangeConfiguration(final DidChangeConfigurationParams p) { - inner.getWorkspaceService().didChangeConfiguration(p); - } - - @Override - public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) { - inner.getWorkspaceService().didChangeWatchedFiles(p); - } - - // -- Helper methods -- - - /** Resolves the first completions, making those with signatures callables. */ - private CompletableFuture resolve( - final Either, CompletionList> answer, final int n) - { - final List items = new ArrayList<>(answer == null - ? new ArrayList<>() : answer.isRight() ? answer.getRight().getItems() - : answer.getLeft()); - items.sort(Comparator.comparing(i -> i.getSortText() != null ? i - .getSortText() : i.getLabel())); - final int count = Math.min(n, items.size()); - final List> resolved = new ArrayList<>(); - for (int i = 0; i < count; i++) { - // NB: Unresolved, should resolving fail. - resolved.add(inner.getTextDocumentService().resolveCompletionItem(items - .get(i)).handle((r, error) -> r)); - } - return CompletableFuture.allOf(resolved.toArray(new CompletableFuture[0])) - .thenApply(x -> { - for (int i = 0; i < count; i++) { - final CompletionItem r = resolved.get(i).join(); - final CompletionItem callable = r == null ? null : callable(items - .get(i), r); - if (callable != null) items.set(i, callable); - } - return new CompletionList(false, items); - }); - } - - /** - * A completion as a callable, from its resolved signature (its detail); - * or null if it has none. - */ - private static CompletionItem callable(final CompletionItem item, - final CompletionItem resolved) - { - final CompletionItemKind kind = item.getKind(); - if (kind != CompletionItemKind.Function && kind != CompletionItemKind.Method && - kind != CompletionItemKind.Class && kind != CompletionItemKind.Constructor) - { - return null; - } - final String name = item.getInsertText() != null ? item.getInsertText() - : item.getLabel(); - if (resolved.getDetail() == null) return null; - final Callable parsed = parse(name, resolved.getDetail()); - if (parsed == null) return null; - final CompletionItem out = new Callable(name, parsed.params(), parsed - .returnType()).kind(CompletionItemKind.Method).detail(resolved - .getDetail()).toItem(); - out.setSortText(item.getSortText()); - out.setDocumentation(resolved.getDocumentation()); - out.setData(item.getData()); - return out; - } - - /** The completions of the word typed so far. */ - private static CompletionList filter(final CompletionList list, - final String word) - { - final String w = word.toLowerCase(Locale.ROOT); - final List out = new ArrayList<>(); - for (final CompletionItem item : list.getItems()) { - final String name = item.getFilterText() != null ? item.getFilterText() - : item.getLabel(); - if (name.startsWith("_") && !word.startsWith("_")) continue; - if (!name.toLowerCase(Locale.ROOT).startsWith(w) || name.equals(word)) { - continue; - } - out.add(item); - } - return new CompletionList(false, out); - } - - private static final class Answer { - - private final CompletableFuture raw; - private final CompletableFuture future; - private final long time; - - private Answer(final CompletableFuture raw, - final CompletableFuture future, final long time) - { - this.raw = raw; - this.future = future; - this.time = time; - } - } - - /** - * Parses a Python signature (as Jedi writes them): the parameters, with - * their annotations as types; and the return annotation. Returns null if it - * is not a signature. - */ - static Callable parse(final String name, - final String signature) - { - final int open = signature.indexOf('('); - if (open < 0) return null; - final int close = closing(signature, open); - if (close < 0) return null; - final List params = new ArrayList<>(); - for (final String part : split(signature.substring(open + 1, close))) { - final String p = part.trim(); - // NB: Bare * and / only mark keyword-only and positional-only ones. - if (p.isEmpty() || p.equals("*") || p.equals("/")) continue; - final int colon = topLevel(p, ':'), eq = topLevel(p, '='); - String paramName = p, type = null; - if (colon >= 0 && (eq < 0 || colon < eq)) { - paramName = p.substring(0, colon); - type = p.substring(colon + 1, eq < 0 ? p.length() : eq).trim(); - } - else if (eq >= 0) paramName = p.substring(0, eq); - paramName = paramName.trim().replaceFirst("^\\*{1,2}", ""); - params.add(new Callable.Param(paramName, type == null || type - .isEmpty() ? null : type)); - } - final String rest = signature.substring(close + 1).trim(); - final String returns = rest.startsWith("->") ? rest.substring(2).trim() - : null; - return new Callable(name, params, returns == null || returns.isEmpty() - ? null : returns); - } - - /** The index of the bracket closing the one at {@code open}, or -1. */ - private static int closing(final String s, final int open) { - int depth = 0; - char quote = 0; - for (int i = open; i < s.length(); i++) { - final char c = s.charAt(i); - if (quote != 0) { - if (c == '\\') i++; - else if (c == quote) quote = 0; - } - else if (c == '\'' || c == '"') quote = c; - else if (c == '(' || c == '[' || c == '{') depth++; - else if (c == ')' || c == ']' || c == '}') { - if (--depth == 0) return i; - } - } - return -1; - } - - /** Splits at the commas outside brackets and strings. */ - private static List split(final String s) { - final List parts = new ArrayList<>(); - int start = 0; - while (true) { - final int comma = topLevel(s.substring(start), ','); - if (comma < 0) break; - parts.add(s.substring(start, start + comma)); - start += comma + 1; - } - parts.add(s.substring(start)); - return parts; - } - - /** The index of the first {@code target} outside brackets and strings. */ - private static int topLevel(final String s, final char target) { - int depth = 0; - char quote = 0; - for (int i = 0; i < s.length(); i++) { - final char c = s.charAt(i); - if (quote != 0) { - if (c == '\\') i++; - else if (c == quote) quote = 0; - } - else if (c == '\'' || c == '"') quote = c; - else if (c == '(' || c == '[' || c == '{') depth++; - else if (c == ')' || c == ']' || c == '}') depth--; - else if (c == target && depth == 0) return i; - } - return -1; - } -} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java index b61f1df..75e6d11 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java @@ -30,6 +30,9 @@ package org.scijava.plugins.scripting.appose.python.completion; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertThrows; +import static org.junit.Assert.assertTrue; import java.util.ArrayList; import java.util.Arrays; @@ -38,13 +41,12 @@ import java.util.concurrent.CompletableFuture; import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.TimeUnit; -import java.util.function.Function; import java.util.stream.Collectors; import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionItemKind; import org.eclipse.lsp4j.CompletionList; import org.eclipse.lsp4j.CompletionParams; -import org.eclipse.lsp4j.Diagnostic; import org.eclipse.lsp4j.DidChangeConfigurationParams; import org.eclipse.lsp4j.DidChangeTextDocumentParams; import org.eclipse.lsp4j.DidChangeWatchedFilesParams; @@ -53,31 +55,22 @@ import org.eclipse.lsp4j.DidSaveTextDocumentParams; import org.eclipse.lsp4j.InitializeParams; import org.eclipse.lsp4j.InitializeResult; -import org.eclipse.lsp4j.MessageActionItem; -import org.eclipse.lsp4j.MessageParams; import org.eclipse.lsp4j.Position; -import org.eclipse.lsp4j.PublishDiagnosticsParams; -import org.eclipse.lsp4j.Range; import org.eclipse.lsp4j.ServerCapabilities; -import org.eclipse.lsp4j.ShowMessageRequestParams; import org.eclipse.lsp4j.TextDocumentContentChangeEvent; import org.eclipse.lsp4j.TextDocumentIdentifier; import org.eclipse.lsp4j.TextDocumentItem; import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; import org.eclipse.lsp4j.jsonrpc.messages.Either; -import org.eclipse.lsp4j.services.LanguageClient; -import org.eclipse.lsp4j.services.LanguageClientAware; import org.eclipse.lsp4j.services.LanguageServer; import org.eclipse.lsp4j.services.TextDocumentService; import org.eclipse.lsp4j.services.WorkspaceService; -import org.junit.Before; import org.junit.Test; -import org.scijava.code.lsp.Environment; +import org.scijava.code.lsp.jvm.Callable; +import org.scijava.code.lsp.Positions; /** - * Tests {@link ApposePythonLanguageServer}, with fake jedis: how it routes - * each script to the jedi of where it runs, and moves scripts between jedis - * as that changes. + * Tests {@link ApposePythonLanguageServer}, around a fake Python server. * * @author Gabriel Selzer */ @@ -85,191 +78,17 @@ public class ApposePythonLanguageServerTest { private static final String URI = "untitled:/a.py"; - /** The jedis started, in order. */ - private final List jedis = new CopyOnWriteArrayList<>(); - private final List published = - new CopyOnWriteArrayList<>(); - - /** Where scripts run when their text does not say (e.g. once built). */ - private volatile String fallback = "A"; - - private ApposePythonLanguageServer server; - - @Before - public void setUp() { - server = server(env -> { - final Fake jedi = new Fake("jedi:" + env.interpreter(), "max", "map"); - jedis.add(jedi); - return jedi; - }); - } - - @Test - public void testRelocation() { - open("x = 1"); - assertEquals(1, jedis.size()); - assertEquals(Collections.singletonList("open 1 x = 1"), jedis.get(0).events); - - // Its text says it runs elsewhere: it moves to a jedi there. - change("#env=B\nx = 1", 2); - assertEquals(Arrays.asList("open 1 x = 1", "close", "shutdown"), jedis.get( - 0).events); - assertEquals("jedi:B", jedis.get(1).name); - assertEquals(Collections.singletonList("open 2 #env=B\nx = 1"), jedis.get( - 1).events); - // Otherwise, changes go to the same jedi. - change("#env=B\nx = 2", 3); - assertEquals("change 3 #env=B\nx = 2", jedis.get(1).events.get(1)); - assertEquals(2, jedis.size()); - - // Where it runs changes without it changing (e.g. its environment was - // built): nothing moves until told so. - change("y = 1", 4); - assertEquals("jedi:A", jedis.get(2).name); - fallback = "C"; - assertEquals(3, jedis.size()); - server.relocateAll(); - assertEquals(Arrays.asList("open 4 y = 1", "close", "shutdown"), jedis.get( - 2).events); - assertEquals("jedi:C", jedis.get(3).name); - assertEquals(Collections.singletonList("open 4 y = 1"), jedis.get(3).events); - - // Closed: on its jedi, which is let go. - server.didClose(new DidCloseTextDocumentParams(new TextDocumentIdentifier( - URI))); - assertEquals(Arrays.asList("open 4 y = 1", "close", "shutdown"), jedis.get( - 3).events); - } - - @Test - public void testCompletions() throws Exception { - open("ma"); - final CompletionList list = complete(); - // Jedi's, as they are. - assertEquals(Arrays.asList("max", "map"), labels(list)); - - // Resolved by the jedi analyzing the script, with its own data. - final CompletionItem max = list.getItems().get(0); - final Object marker = max.getData(); - final CompletionItem resolved = server.resolveCompletionItem(max).get(10, - TimeUnit.SECONDS); - assertEquals("jedi:A resolved max (data jedi:A)", resolved.getDetail()); - assertEquals(marker, resolved.getData()); - - // Moved: resolved by the jedi analyzing it now. - change("#env=B\nma", 2); - assertEquals("jedi:B resolved max (data jedi:B)", server - .resolveCompletionItem(complete().getItems().get(0)).get(10, - TimeUnit.SECONDS).getDetail()); - } - - @Test - public void testDiagnostics() { - open("x = 1"); - final Fake a = jedis.get(0); - a.publish(URI, 1, "from A"); - assertEquals(Collections.singletonList("from A"), messages()); - - // Moved: the old jedi's are out of date. - change("#env=B\nx = 1", 2); - a.publish(URI, 1, "stale"); - jedis.get(1).publish(URI, 2, "from B"); - assertEquals(Arrays.asList("from A", "from B"), messages()); - } - - // -- Helper methods -- - - private ApposePythonLanguageServer server( - final Function jedi) + /** Knows json's dumps (a function), dump, decoder (a module), __doc__. */ + private static final class Jedi implements LanguageServer, + TextDocumentService, WorkspaceService { - final ApposePythonLanguageServer s = new ApposePythonLanguageServer( - doc -> new Environment(doc.getText().contains("#env=B") ? "B" - : fallback, null, null), jedi, null); - s.connect(new Client()); - s.initialize(new InitializeParams()); - return s; - } - - private void open(final String text) { - server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(URI, - "appose-python", 1, text))); - } - - private void change(final String text, final int version) { - server.didChange(new DidChangeTextDocumentParams( - new VersionedTextDocumentIdentifier(URI, version), Collections - .singletonList(new TextDocumentContentChangeEvent(text)))); - } - - private CompletionList complete() throws Exception { - return server.completion(new CompletionParams(new TextDocumentIdentifier( - URI), new Position(0, 2))).get(10, TimeUnit.SECONDS).getRight(); - } - - private static List labels(final CompletionList list) { - return list.getItems().stream().map(CompletionItem::getLabel).collect( - Collectors.toList()); - } - - private List messages() { - final List out = new ArrayList<>(); - for (final PublishDiagnosticsParams p : published) { - for (final Diagnostic d : p.getDiagnostics()) { - out.add(d.getMessage().getLeft()); - } - } - return out; - } - - private final class Client implements LanguageClient { - - @Override - public void publishDiagnostics(final PublishDiagnosticsParams p) { - published.add(p); - } - - @Override - public void telemetryEvent(final Object object) {} - @Override - public void showMessage(final MessageParams m) {} - - @Override - public CompletableFuture showMessageRequest( - final ShowMessageRequestParams r) - { - return CompletableFuture.completedFuture(null); - } - - @Override - public void logMessage(final MessageParams m) {} - } - - /** A fake server, recording what it is told, completing given labels. */ - private static final class Fake implements LanguageServer, - TextDocumentService, WorkspaceService, LanguageClientAware - { - - private final String name; - private final List labels; - private final List events = new CopyOnWriteArrayList<>(); - private LanguageClient client; - - private Fake(final String name, final String... labels) { - this.name = name; - this.labels = Arrays.asList(labels); - } - - void publish(final String uri, final int version, final String message) { - client.publishDiagnostics(new PublishDiagnosticsParams(uri, - Collections.singletonList(new Diagnostic(new Range(new Position(0, 0), - new Position(0, 1)), message)), version)); - } - - @Override - public void connect(final LanguageClient c) { - client = c; - } + final List asked = new CopyOnWriteArrayList<>(); + final List resolved = new CopyOnWriteArrayList<>(); + CompletableFuture hold; + final List> held = new CopyOnWriteArrayList<>(); + /** Whether it knows nothing yet (e.g. while json is installed). */ + volatile boolean ignorant; @Override public CompletableFuture initialize( @@ -281,7 +100,6 @@ public CompletableFuture initialize( @Override public CompletableFuture shutdown() { - events.add("shutdown"); return CompletableFuture.completedFuture(null); } @@ -299,52 +117,194 @@ public WorkspaceService getWorkspaceService() { } @Override - public void didOpen(final DidOpenTextDocumentParams p) { - final TextDocumentItem d = p.getTextDocument(); - events.add("open " + d.getVersion() + " " + d.getText()); - } + public void didOpen(final DidOpenTextDocumentParams p) {} @Override - public void didChange(final DidChangeTextDocumentParams p) { - events.add("change " + p.getTextDocument().getVersion() + " " + p - .getContentChanges().get(0).getText()); - } + public void didChange(final DidChangeTextDocumentParams p) {} @Override - public void didClose(final DidCloseTextDocumentParams p) { - events.add("close"); - } + public void didClose(final DidCloseTextDocumentParams p) {} @Override public void didSave(final DidSaveTextDocumentParams p) {} + @Override + public void didChangeConfiguration(final DidChangeConfigurationParams p) {} + + @Override + public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) {} + @Override public CompletableFuture, CompletionList>> completion(final CompletionParams p) { + asked.add(p.getPosition()); final List items = new ArrayList<>(); - for (final String label : labels) { - final CompletionItem item = new CompletionItem(label); - item.setData(name); + int i = 0; + for (final String name : ignorant ? new String[0] : new String[] { + "dumps", "dump", "decoder", "__doc__" }) + { + final CompletionItem item = new CompletionItem(name); + item.setKind(name.equals("decoder") ? CompletionItemKind.Module + : CompletionItemKind.Function); + item.setInsertText(name); + item.setSortText("z" + i++); items.add(item); } - return CompletableFuture.completedFuture(Either.forRight( - new CompletionList(items))); + final Either, CompletionList> answer = Either + .forRight(new CompletionList(items)); + if (hold == null) return CompletableFuture.completedFuture(answer); + final CompletableFuture, CompletionList>> f = + new CompletableFuture<>(); + held.add(f); + hold.thenRun(() -> f.complete(answer)); + return f; } @Override public CompletableFuture resolveCompletionItem( final CompletionItem item) { - item.setDetail(name + " resolved " + item.getLabel() + " (data " + item - .getData() + ")"); + resolved.add(item.getLabel()); + item.setDetail(item.getLabel().equals("decoder") ? "module decoder" + : "def " + item.getLabel() + "(obj, *, indent: int=None) -> str"); return CompletableFuture.completedFuture(item); } + } - @Override - public void didChangeConfiguration(final DidChangeConfigurationParams p) {} + @Test + public void testAskOnceAndFilter() throws Exception { + final Jedi jedi = new Jedi(); + final ApposePythonLanguageServer server = new ApposePythonLanguageServer(jedi, 30); + open(server, "import json\njson.du"); + List items = complete(server, "import json\njson.du"); + // Asked about the word's first character. + assertEquals(new Position(1, 6), jedi.asked.get(0)); + // The word's completions, without dunders. + assertEquals(Arrays.asList("dumps", "dump"), labels(items)); + // Once a word is begun: callables, from the resolved signatures. + final CompletionItem dumps = items.get(0); + assertEquals("(obj, int indent)", dumps.getLabelDetails().getDetail()); + assertEquals("str", dumps.getLabelDetails().getDescription()); + assertEquals("dumps(${1:obj}, ${2:indent})", dumps.getInsertText()); + + // Typing on: the same answer, filtered; not asked again. + change(server, "import json\njson.dum", 2); + items = complete(server, "import json\njson.dum"); + assertEquals(Arrays.asList("dumps", "dump"), labels(items)); + // (The word typed in full is not offered again.) + change(server, "import json\njson.dumps", 3); + assertTrue(complete(server, "import json\njson.dumps").isEmpty()); + assertEquals(1, jedi.asked.size()); + + // Nothing typed: names only (nothing resolved), dunders left out. + change(server, "import json\njson.", 4); + final int resolved = jedi.resolved.size(); + items = complete(server, "import json\njson."); + assertEquals(Arrays.asList("dumps", "dump", "decoder"), labels(items)); + assertNull(items.get(0).getLabelDetails()); + assertEquals(resolved, jedi.resolved.size()); + } - @Override - public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) {} + @Test + public void testEmptyAnswersAskedAgain() throws Exception { + final Jedi jedi = new Jedi(); + jedi.ignorant = true; + final ApposePythonLanguageServer server = new ApposePythonLanguageServer(jedi, 0); + open(server, "import json\njson.du"); + assertTrue(complete(server, "import json\njson.du").isEmpty()); + // It knows more now: asked again, rather than the empty answer reused. + jedi.ignorant = false; + assertEquals(Arrays.asList("dumps", "dump"), labels(complete(server, + "import json\njson.du"))); + assertEquals(2, jedi.asked.size()); + } + + @Test + public void testImportsAreNames() throws Exception { + final Jedi jedi = new Jedi(); + final ApposePythonLanguageServer server = new ApposePythonLanguageServer(jedi, 30); + open(server, "from json import du"); + final List items = complete(server, "from json import du"); + assertEquals(Arrays.asList("dumps", "dump"), labels(items)); + assertNull(items.get(0).getLabelDetails()); + } + + @Test + public void testSuperseded() throws Exception { + final Jedi jedi = new Jedi(); + jedi.hold = new CompletableFuture<>(); + final ApposePythonLanguageServer server = new ApposePythonLanguageServer(jedi, 0); + open(server, "x.a"); + final CompletableFuture, CompletionList>> slow = + server.completion(params("x.a")); + change(server, "x.b", 2); + final CompletableFuture, CompletionList>> next = + server.completion(params("x.b")); + // The first question is cancelled, also on the server. + assertThrows(Exception.class, () -> slow.get(10, TimeUnit.SECONDS)); + assertTrue(jedi.held.get(0).isCancelled()); + jedi.hold.complete(null); + assertTrue(next.get(10, TimeUnit.SECONDS).getRight().getItems().isEmpty()); + } + + // -- Helper methods -- + + private static CompletionParams params(final String text) { + return new CompletionParams(new TextDocumentIdentifier(URI), Positions + .position(text, text.length())); + } + + private static List complete( + final ApposePythonLanguageServer server, final String text) throws Exception + { + return server.completion(params(text)).get(10, TimeUnit.SECONDS) + .getRight().getItems(); + } + + private static void open(final ApposePythonLanguageServer server, + final String text) + { + server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(URI, + "python", 1, text))); + } + + private static void change(final ApposePythonLanguageServer server, + final String text, final int version) + { + server.didChange(new DidChangeTextDocumentParams( + new VersionedTextDocumentIdentifier(URI, version), Collections + .singletonList(new TextDocumentContentChangeEvent(text)))); + } + + private static List labels(final List items) { + return items.stream().map(CompletionItem::getLabel).collect(Collectors + .toList()); + } + + @Test + public void testParse() { + Callable c = ApposePythonLanguageServer.parse("dumps", + "def dumps(obj, *, skipkeys: bool=False, cls=None, **kw) -> str"); + assertEquals("[obj, bool skipkeys, cls, kw]", String.valueOf(c + .params())); + assertEquals("str", c.returnType()); + + // Brackets and strings in annotations and defaults; positional-only. + c = ApposePythonLanguageServer.parse("zeros", "def zeros(shape: tuple[int, ...], dtype: str=\"f,8\"" + + ", /, order: Literal['C', 'F']='C') -> NDArray[float64]"); + assertEquals("[tuple[int, ...] shape, str dtype, " + + "Literal['C', 'F'] order]", String.valueOf(c.params())); + assertEquals("NDArray[float64]", c.returnType()); + + // Classes (their constructors), without parameters or return type. + c = ApposePythonLanguageServer.parse("Path", "class Path()"); + assertEquals(0, c.params().size()); + assertNull(c.returnType()); + c = ApposePythonLanguageServer.parse("f", "def f(*args)"); + assertEquals("args", c.params().get(0).name()); + + // Not a signature. + assertNull(ApposePythonLanguageServer.parse("x", "instance x")); } } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java index 118ced8..b797e82 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java @@ -113,7 +113,7 @@ public void testScriptAsPython() { @Test public void testEnvironmentWithoutOne() { // No script environment has been built: the server's own Python. - final Environment env = plugin.environment(new TextDocumentItem( + final Environment env = environments.environment(new TextDocumentItem( "untitled:/a.py", "appose-python", 2, "import os\nos.pa")); assertNull(env.interpreter()); assertTrue(env.paths().isEmpty()); @@ -134,13 +134,13 @@ public void testOverlay() throws Exception { "appose-python", 1, code); // On opening the script, its dependencies are settled: the overlay is // installed; once ready, the server searches it. - plugin.environment(doc); + environments.environment(doc); for (int i = 0; i < 100 && installed.isEmpty(); i++) Thread.sleep(20); Thread.sleep(50); assertEquals(Collections.singletonList(Collections.singletonList( "tifffile")), installed); - assertEquals(Collections.singletonList("/overlays/blur"), plugin.environment( - doc).paths()); + assertEquals(Collections.singletonList("/overlays/blur"), environments + .environment(doc).paths()); } @Test @@ -149,10 +149,10 @@ public void testScriptWorkersUntouched() { // it, even as its metadata is edited. final String script = "# /// script\n# dependencies = [\"appose\"]\n" + "# ///\nimport os\nos.pa"; - plugin.environment(new TextDocumentItem("untitled:/a.py", "appose-python", 2, - script)); - plugin.environment(new TextDocumentItem("untitled:/a.py", "appose-python", 3, - script.replace("appose", "appose, numpy"))); + environments.environment(new TextDocumentItem("untitled:/a.py", + "appose-python", 2, script)); + environments.environment(new TextDocumentItem("untitled:/a.py", + "appose-python", 3, script.replace("appose", "appose, numpy"))); assertTrue(context.service(ResidentWorkerService.class).workers() .isEmpty()); } @@ -176,7 +176,7 @@ public void testEnvironmentFromEarlierSession() throws Exception { java.nio.file.Files.createDirectories(python.getParent()); java.nio.file.Files.createFile(python); - final Environment env = plugin.environment(new TextDocumentItem(script + final Environment env = environments.environment(new TextDocumentItem(script .toURI().toString(), "appose-python", 2, header + "numpy.ze")); // The server analyzes against the environment's Python... assertEquals(python.toFile().getAbsolutePath(), env.interpreter()); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java deleted file mode 100644 index ecfac2a..0000000 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/PythonCompletionServerTest.java +++ /dev/null @@ -1,294 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertThrows; -import static org.junit.Assert.assertTrue; - -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.List; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.CopyOnWriteArrayList; -import java.util.concurrent.TimeUnit; -import java.util.stream.Collectors; - -import org.eclipse.lsp4j.CompletionItem; -import org.eclipse.lsp4j.CompletionItemKind; -import org.eclipse.lsp4j.CompletionList; -import org.eclipse.lsp4j.CompletionParams; -import org.eclipse.lsp4j.DidChangeConfigurationParams; -import org.eclipse.lsp4j.DidChangeTextDocumentParams; -import org.eclipse.lsp4j.DidChangeWatchedFilesParams; -import org.eclipse.lsp4j.DidCloseTextDocumentParams; -import org.eclipse.lsp4j.DidOpenTextDocumentParams; -import org.eclipse.lsp4j.DidSaveTextDocumentParams; -import org.eclipse.lsp4j.InitializeParams; -import org.eclipse.lsp4j.InitializeResult; -import org.eclipse.lsp4j.Position; -import org.eclipse.lsp4j.ServerCapabilities; -import org.eclipse.lsp4j.TextDocumentContentChangeEvent; -import org.eclipse.lsp4j.TextDocumentIdentifier; -import org.eclipse.lsp4j.TextDocumentItem; -import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; -import org.eclipse.lsp4j.jsonrpc.messages.Either; -import org.eclipse.lsp4j.services.LanguageServer; -import org.eclipse.lsp4j.services.TextDocumentService; -import org.eclipse.lsp4j.services.WorkspaceService; -import org.junit.Test; -import org.scijava.code.lsp.jvm.Callable; -import org.scijava.code.lsp.Positions; - -/** - * Tests {@link PythonCompletionServer}, around a fake Python server. - * - * @author Gabriel Selzer - */ -public class PythonCompletionServerTest { - - private static final String URI = "untitled:/a.py"; - - /** Knows json's dumps (a function), dump, decoder (a module), __doc__. */ - private static final class Jedi implements LanguageServer, - TextDocumentService, WorkspaceService - { - - final List asked = new CopyOnWriteArrayList<>(); - final List resolved = new CopyOnWriteArrayList<>(); - CompletableFuture hold; - final List> held = new CopyOnWriteArrayList<>(); - - @Override - public CompletableFuture initialize( - final InitializeParams params) - { - return CompletableFuture.completedFuture(new InitializeResult( - new ServerCapabilities())); - } - - @Override - public CompletableFuture shutdown() { - return CompletableFuture.completedFuture(null); - } - - @Override - public void exit() {} - - @Override - public TextDocumentService getTextDocumentService() { - return this; - } - - @Override - public WorkspaceService getWorkspaceService() { - return this; - } - - @Override - public void didOpen(final DidOpenTextDocumentParams p) {} - - @Override - public void didChange(final DidChangeTextDocumentParams p) {} - - @Override - public void didClose(final DidCloseTextDocumentParams p) {} - - @Override - public void didSave(final DidSaveTextDocumentParams p) {} - - @Override - public void didChangeConfiguration(final DidChangeConfigurationParams p) {} - - @Override - public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) {} - - @Override - public CompletableFuture, CompletionList>> - completion(final CompletionParams p) - { - asked.add(p.getPosition()); - final List items = new ArrayList<>(); - int i = 0; - for (final String name : new String[] { "dumps", "dump", "decoder", - "__doc__" }) - { - final CompletionItem item = new CompletionItem(name); - item.setKind(name.equals("decoder") ? CompletionItemKind.Module - : CompletionItemKind.Function); - item.setInsertText(name); - item.setSortText("z" + i++); - items.add(item); - } - final Either, CompletionList> answer = Either - .forRight(new CompletionList(items)); - if (hold == null) return CompletableFuture.completedFuture(answer); - final CompletableFuture, CompletionList>> f = - new CompletableFuture<>(); - held.add(f); - hold.thenRun(() -> f.complete(answer)); - return f; - } - - @Override - public CompletableFuture resolveCompletionItem( - final CompletionItem item) - { - resolved.add(item.getLabel()); - item.setDetail(item.getLabel().equals("decoder") ? "module decoder" - : "def " + item.getLabel() + "(obj, *, indent: int=None) -> str"); - return CompletableFuture.completedFuture(item); - } - } - - @Test - public void testAskOnceAndFilter() throws Exception { - final Jedi jedi = new Jedi(); - final PythonCompletionServer server = new PythonCompletionServer(jedi, 30); - open(server, "import json\njson.du"); - List items = complete(server, "import json\njson.du"); - // Asked about the word's first character. - assertEquals(new Position(1, 6), jedi.asked.get(0)); - // The word's completions, without dunders. - assertEquals(Arrays.asList("dumps", "dump"), labels(items)); - // Once a word is begun: callables, from the resolved signatures. - final CompletionItem dumps = items.get(0); - assertEquals("(obj, int indent)", dumps.getLabelDetails().getDetail()); - assertEquals("str", dumps.getLabelDetails().getDescription()); - assertEquals("dumps(${1:obj}, ${2:indent})", dumps.getInsertText()); - - // Typing on: the same answer, filtered; not asked again. - change(server, "import json\njson.dum", 2); - items = complete(server, "import json\njson.dum"); - assertEquals(Arrays.asList("dumps", "dump"), labels(items)); - // (The word typed in full is not offered again.) - change(server, "import json\njson.dumps", 3); - assertTrue(complete(server, "import json\njson.dumps").isEmpty()); - assertEquals(1, jedi.asked.size()); - - // Nothing typed: names only (nothing resolved), dunders left out. - change(server, "import json\njson.", 4); - final int resolved = jedi.resolved.size(); - items = complete(server, "import json\njson."); - assertEquals(Arrays.asList("dumps", "dump", "decoder"), labels(items)); - assertNull(items.get(0).getLabelDetails()); - assertEquals(resolved, jedi.resolved.size()); - } - - @Test - public void testImportsAreNames() throws Exception { - final Jedi jedi = new Jedi(); - final PythonCompletionServer server = new PythonCompletionServer(jedi, 30); - open(server, "from json import du"); - final List items = complete(server, "from json import du"); - assertEquals(Arrays.asList("dumps", "dump"), labels(items)); - assertNull(items.get(0).getLabelDetails()); - } - - @Test - public void testSuperseded() throws Exception { - final Jedi jedi = new Jedi(); - jedi.hold = new CompletableFuture<>(); - final PythonCompletionServer server = new PythonCompletionServer(jedi, 0); - open(server, "x.a"); - final CompletableFuture, CompletionList>> slow = - server.completion(params("x.a")); - change(server, "x.b", 2); - final CompletableFuture, CompletionList>> next = - server.completion(params("x.b")); - // The first question is cancelled, also on the server. - assertThrows(Exception.class, () -> slow.get(10, TimeUnit.SECONDS)); - assertTrue(jedi.held.get(0).isCancelled()); - jedi.hold.complete(null); - assertTrue(next.get(10, TimeUnit.SECONDS).getRight().getItems().isEmpty()); - } - - // -- Helper methods -- - - private static CompletionParams params(final String text) { - return new CompletionParams(new TextDocumentIdentifier(URI), Positions - .position(text, text.length())); - } - - private static List complete( - final PythonCompletionServer server, final String text) throws Exception - { - return server.completion(params(text)).get(10, TimeUnit.SECONDS) - .getRight().getItems(); - } - - private static void open(final PythonCompletionServer server, - final String text) - { - server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(URI, - "python", 1, text))); - } - - private static void change(final PythonCompletionServer server, - final String text, final int version) - { - server.didChange(new DidChangeTextDocumentParams( - new VersionedTextDocumentIdentifier(URI, version), Collections - .singletonList(new TextDocumentContentChangeEvent(text)))); - } - - private static List labels(final List items) { - return items.stream().map(CompletionItem::getLabel).collect(Collectors - .toList()); - } - - @Test - public void testParse() { - Callable c = PythonCompletionServer.parse("dumps", - "def dumps(obj, *, skipkeys: bool=False, cls=None, **kw) -> str"); - assertEquals("[obj, bool skipkeys, cls, kw]", String.valueOf(c - .params())); - assertEquals("str", c.returnType()); - - // Brackets and strings in annotations and defaults; positional-only. - c = PythonCompletionServer.parse("zeros", "def zeros(shape: tuple[int, ...], dtype: str=\"f,8\"" + - ", /, order: Literal['C', 'F']='C') -> NDArray[float64]"); - assertEquals("[tuple[int, ...] shape, str dtype, " + - "Literal['C', 'F'] order]", String.valueOf(c.params())); - assertEquals("NDArray[float64]", c.returnType()); - - // Classes (their constructors), without parameters or return type. - c = PythonCompletionServer.parse("Path", "class Path()"); - assertEquals(0, c.params().size()); - assertNull(c.returnType()); - c = PythonCompletionServer.parse("f", "def f(*args)"); - assertEquals("args", c.params().get(0).name()); - - // Not a signature. - assertNull(PythonCompletionServer.parse("x", "instance x")); - } -} From cd8b72307eb5f7b408989dd85c58bcdf8c0e55ed Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Thu, 15 Oct 2026 17:57:38 -0500 Subject: [PATCH 25/37] Get jedi from code-lsp-jedi; route scripts between its servers jedi-language-server is now offered by scijava-code-lsp-jedi (a runtime dependency), so JediServerProcessPlugin goes; the tooling environment here only installs dependency overlays. code-lsp's servers now analyze scripts in one environment each. Since a script's environment is in its text, and changes as it is edited, the new RoutingLanguageServer keeps a Python server per environment and moves each script to the one where it runs now, as asked before each request (and closes servers left without scripts). Also follow code-lsp's API: plugins say which LSP language they serve (appose-python), and ask whether python is served rather than catching its absence. Co-Authored-By: Claude Opus 5.5 --- README.md | 9 +- pom.xml | 8 + .../completion/ApposePythonEnvironments.java | 32 +- .../completion/ApposePythonServerPlugin.java | 23 +- .../completion/JediServerProcessPlugin.java | 174 ------- .../completion/RoutingLanguageServer.java | 443 ++++++++++++++++++ ...ApposePythonCompletionIntegrationTest.java | 39 +- .../completion/ApposePythonServersTest.java | 20 - .../completion/RoutingLanguageServerTest.java | 304 ++++++++++++ 9 files changed, 799 insertions(+), 253 deletions(-) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediServerProcessPlugin.java create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServer.java create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServerTest.java diff --git a/README.md b/README.md index 4377e5c..4d26f3d 100644 --- a/README.md +++ b/README.md @@ -146,8 +146,8 @@ the script's `#@` parameters, `task`, builtins and keywords, and members of objects and modules; and it gives function signatures (with parameter names and types), documentation, hover and syntax errors. -The server runs in a small environment of its own -(`appose-python-completion`: Python, jedi-language-server, Appose and uv), +The server, offered by `scijava-code-lsp-jedi`, runs in a small environment +of its own (`code-lsp-jedi`: Python, Jedi, jedi-language-server and NumPy), which is built in the background the first time you complete code, and takes a while only then; until it is, there are no completions. The server talks to the editor via the @@ -155,7 +155,7 @@ talks to the editor via the (see `scijava-code-lsp`); it sees the script as plain Python, with its `#@` parameters declared. One server runs per script environment; servers unused for 10 minutes stop. Their error output goes to -`appose-python-completion-server.log`, next to Appose's environments. +`code-lsp-jedi-server.log`, next to Appose's environments. Once a script's environment is built, by running the script (in this session or an earlier one), the server analyzes the script against it, so that the packages @@ -163,7 +163,8 @@ installed there are known too. Packages the script declares (in its `# /// script` block) but its environment lacks, e.g. a dependency you just added, are installed for completion only, into a separate directory, a moment after you stop editing the dependencies; so they are known before you run the -script. This covers PyPI packages; conda packages (`[tool.pixi.dependencies]`) +script. (They are installed with uv, from another small environment, +`appose-python-completion`.) This covers PyPI packages; conda packages (`[tool.pixi.dependencies]`) and environments declared in files are known once the script has run. Completion never builds a script's environment itself, and never disturbs its worker. Script parameters are known by the Python types they have at run diff --git a/pom.xml b/pom.xml index 8d7098d..06fa379 100644 --- a/pom.xml +++ b/pom.xml @@ -96,6 +96,7 @@ 2.101.0 0.1.0-SNAPSHOT + 0.1.0-SNAPSHOT @@ -113,6 +114,13 @@ scijava-code-lsp ${scijava-code-lsp.version} + + org.scijava + scijava-code-lsp-jedi + ${scijava-code-lsp-jedi.version} + + runtime + diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java index 990ca8c..306d8c7 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java @@ -58,14 +58,14 @@ * know: the Python of a script's environment, if built (in this session, or an * earlier one; never building it, nor disturbing its worker); the overlay of * packages it declares but its environment lacks (see - * {@link DependencyOverlays}); and the tooling environment jedi-language-server - * runs in. + * {@link DependencyOverlays}), and the tooling environment overlays are + * installed from. * * @author Gabriel Selzer */ final class ApposePythonEnvironments { - /** Name of the tooling environment in which the language server runs. */ + /** Name of the tooling environment that installs overlays. */ static final String TOOLING_ENV = "appose-python-completion"; /** @@ -341,20 +341,6 @@ private static String findPython(final List dirs) { return null; } - /** - * Gets the Python of the tooling environment, in which jedi-language-server - * runs (see {@link JediServerProcessPlugin}): built first if needed (but - * starting no worker). - */ - static String toolingPython(final ResidentWorkerService workers) - throws Exception - { - final String python = pythonExecutable(toolingWorker(workers).environment() - .get()); - if (python == null) throw new IOException("No Python in " + TOOLING_ENV); - return python; - } - /** Gets the tooling environment's worker, building it first if needed. */ private static ResidentWorker toolingWorker( final ResidentWorkerService workers) @@ -365,8 +351,8 @@ private static ResidentWorker toolingWorker( } /** - * The tooling environment: Python, Jedi, jedi-language-server, NumPy and - * Appose; and uv, to install packages for {@link DependencyOverlays}. + * The tooling environment: Python and Appose, to run overlay.py; and uv, to + * install packages for {@link DependencyOverlays}. */ static String toolingConfig() { return "[workspace]\n" + // @@ -375,15 +361,13 @@ static String toolingConfig() { "platforms = [\"" + InlineMetadata.condaPlatform() + "\"]\n" + // "\n" + // "[dependencies]\n" + // + // NB: The overlays of scripts whose environment is not built are + // installed for this Python, for jedi-language-server's Python (see + // code-lsp-jedi) to find: the versions must match. "python = \"3.12.*\"\n" + // - "jedi = \">=0.19.2\"\n" + // - // NB: Script parameters are typed as NumPy arrays (see - // ApposePythonDialect): known even before a script's environment is. - "numpy = \"*\"\n" + // "\n" + // "[pypi-dependencies]\n" + // "appose = \"*\"\n" + // - "jedi-language-server = \">=0.47\"\n" + // "uv = \"*\"\n"; } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java index a13f0c9..c896e70 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java @@ -62,6 +62,13 @@ public class ApposePythonServerPlugin implements LanguageServerPlugin { @Parameter private ResidentWorkerService workers; + @Override + public boolean supports(final String languageId) { + // NB: Not "python", which this builds on. Without jedi, there is + // nothing to serve. + return "appose-python".equals(languageId) && jediOffered(); + } + @Override public boolean supports(final ScriptLanguage language) { // NB: Without jedi, there is nothing to serve. @@ -69,9 +76,15 @@ public boolean supports(final ScriptLanguage language) { } @Override - public LanguageServer create(final Environment environment) { - return new ApposePythonLanguageServer(servers.launch("python", - ApposePythonEnvironments.of(workers)::environment), SIGNATURES); + public LanguageServer create(final String languageId, + final Environment environment) + { + // NB: Where each script runs is as its text says (see + // ApposePythonEnvironments), whatever the caller thinks; and may change + // as it is edited, moving it to another Python server. + return new ApposePythonLanguageServer(new RoutingLanguageServer( + env -> servers.launch("python", env), ApposePythonEnvironments.of( + workers)::environment), SIGNATURES); } @Override @@ -79,8 +92,8 @@ public ScriptDialect dialect() { return DIALECT; } - /** Whether jedi-language-server is offered. */ + /** Whether jedi-language-server (or another Python server) is offered. */ boolean jediOffered() { - return servers.launch("python", Environment.NONE) != null; + return servers.supports("python"); } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediServerProcessPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediServerProcessPlugin.java deleted file mode 100644 index 910ce9b..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/JediServerProcessPlugin.java +++ /dev/null @@ -1,174 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.io.File; -import java.io.IOException; -import java.util.LinkedHashMap; -import java.util.Map; -import java.util.concurrent.Callable; - -import org.apposed.appose.util.Environments; -import org.scijava.code.lsp.Environment; -import org.scijava.code.lsp.ServerProcessPlugin; -import org.scijava.plugin.Parameter; -import org.scijava.plugin.Plugin; -import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; - -/** - * Offers jedi-language-server, for {@code python}: run with the Python of a - * tooling environment (see {@link ApposePythonEnvironments}), to analyze - * scripts against their environment's Python (if known) and extra search - * paths (e.g. a dependency overlay). Python scripts run via Appose use it, - * and so do Jython's. - * - * @author Gabriel Selzer - */ -@Plugin(type = ServerProcessPlugin.class, name = "jedi-language-server") -public class JediServerProcessPlugin implements ServerProcessPlugin { - - /** - * Runs jedi-language-server, exiting once the given process (this JVM) is - * gone. NB: The server does not notice by itself; on Windows, it then even - * loops, failing to write to the closed pipe. - */ - public static final String LAUNCHER = String.join("\n", // - "import os, sys, threading, time", // - "parent = int(sys.argv.pop(1))", // - "def watch():", // - " if os.name == 'nt':", // - " import ctypes", // - " k = ctypes.windll.kernel32", // - " h = k.OpenProcess(0x00100000, False, parent) # SYNCHRONIZE", // - " if h: k.WaitForSingleObject(h, 0xFFFFFFFF)", // - " else:", // - " while os.getppid() == parent: time.sleep(2)", // - " os._exit(0)", // - "threading.Thread(target=watch, daemon=True).start()", // - "from jedi_language_server.cli import cli", // - "cli()"); - - /** How large the error log may grow before it is started anew, in bytes. */ - private static final long MAX_LOG = 5 * 1024 * 1024; - - @Parameter - private ResidentWorkerService workers; - - private final Callable python; - private final File workspace; - private final File log; - - /** Runs the server in the tooling environment, building it if needed. */ - public JediServerProcessPlugin() { - final File envs = new File(Environments.apposeEnvsDir()); - python = () -> ApposePythonEnvironments.toolingPython(workers); - workspace = new File(envs, ApposePythonEnvironments.TOOLING_ENV + - "-workspace"); - log = new File(envs, ApposePythonEnvironments.TOOLING_ENV + "-server.log"); - } - - /** - * @param python Gets the Python executable of the environment - * jedi-language-server is installed in (building it, if needed). - * @param workspace An (empty) directory to serve as workspace root. - * @param log Where to append the server's error output. - */ - JediServerProcessPlugin(final Callable python, final File workspace, - final File log) - { - this.python = python; - this.workspace = workspace; - this.log = log; - } - - @Override - public String languageId() { - return "python"; - } - - @Override - public Process start(final Environment env) - throws IOException - { - final String exe; - try { - exe = python.call(); - } - catch (final IOException exc) { - throw exc; - } - catch (final Exception exc) { - throw new IOException("No Python for jedi-language-server", exc); - } - if (exe == null) throw new IOException( - "No Python for jedi-language-server"); - log.getParentFile().mkdirs(); - if (log.length() > MAX_LOG) log.delete(); - final ProcessBuilder pb = new ProcessBuilder(exe, "-c", LAUNCHER, String - .valueOf(ProcessHandle.current().pid())); - pb.redirectError(ProcessBuilder.Redirect.appendTo(log)); - return pb.start(); - } - - @Override - public Object initializationOptions(final Environment env) { - final Map workspaceOptions = new LinkedHashMap<>(); - // NB: Without an environment, Jedi uses the server's own Python. - if (env.interpreter() != null) { - workspaceOptions.put("environmentPath", env.interpreter()); - } - workspaceOptions.put("extraPaths", env.paths()); - final Map options = new LinkedHashMap<>(); - options.put("workspace", workspaceOptions); - // NB: Jedi's diagnostics are syntax errors, found as the script changes. - final Map diagnostics = map("enable", true); - diagnostics.put("didOpen", true); - diagnostics.put("didChange", true); - diagnostics.put("didSave", false); - options.put("diagnostics", diagnostics); - return options; - } - - @Override - public String rootUri(final Environment env) { - // NB: jedi-language-server applies its workspace options only with a - // root. An empty one, so that nothing irrelevant gets indexed. - workspace.mkdirs(); - return workspace.toURI().toString(); - } - - private static Map map(final String key, - final Object value) - { - final Map map = new LinkedHashMap<>(); - map.put(key, value); - return map; - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServer.java new file mode 100644 index 0000000..92c7d62 --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServer.java @@ -0,0 +1,443 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.function.Function; + +import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionList; +import org.eclipse.lsp4j.CompletionParams; +import org.eclipse.lsp4j.DidChangeConfigurationParams; +import org.eclipse.lsp4j.DidChangeTextDocumentParams; +import org.eclipse.lsp4j.DidChangeWatchedFilesParams; +import org.eclipse.lsp4j.DidCloseTextDocumentParams; +import org.eclipse.lsp4j.DidOpenTextDocumentParams; +import org.eclipse.lsp4j.DidSaveTextDocumentParams; +import org.eclipse.lsp4j.Hover; +import org.eclipse.lsp4j.HoverParams; +import org.eclipse.lsp4j.InitializeParams; +import org.eclipse.lsp4j.InitializeResult; +import org.eclipse.lsp4j.InitializedParams; +import org.eclipse.lsp4j.MessageActionItem; +import org.eclipse.lsp4j.MessageParams; +import org.eclipse.lsp4j.PublishDiagnosticsParams; +import org.eclipse.lsp4j.ServerCapabilities; +import org.eclipse.lsp4j.ShowMessageRequestParams; +import org.eclipse.lsp4j.SignatureHelp; +import org.eclipse.lsp4j.SignatureHelpParams; +import org.eclipse.lsp4j.TextDocumentContentChangeEvent; +import org.eclipse.lsp4j.TextDocumentIdentifier; +import org.eclipse.lsp4j.TextDocumentItem; +import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; +import org.eclipse.lsp4j.jsonrpc.messages.Either; +import org.eclipse.lsp4j.services.LanguageClient; +import org.eclipse.lsp4j.services.LanguageClientAware; +import org.eclipse.lsp4j.services.LanguageServer; +import org.eclipse.lsp4j.services.TextDocumentService; +import org.eclipse.lsp4j.services.WorkspaceService; +import org.scijava.code.lsp.Environment; + +/** + * A language server for scripts whose text says where they run (which may + * change as they are edited): sends each document to a server of where its + * script runs now, as asked just before each request about it, and moves it + * to another as that changes. + *

    + * Where scripts run is asked off the callers' threads (e.g. an editor's), as + * that may look at files; one document at a time, in order, so that each + * server is told of a document's changes before it is asked about it. + *

    + * + * @author Gabriel Selzer + */ +class RoutingLanguageServer implements LanguageServer, TextDocumentService, + WorkspaceService, LanguageClientAware +{ + + private static final ExecutorService WHERE = Executors + .newSingleThreadExecutor(r -> { + final Thread t = new Thread(r, "appose-python-environments"); + t.setDaemon(true); + return t; + }); + + private final Function launch; + private final Function environment; + + /** The servers, by where the scripts they analyze run. */ + private final Map servers = + new ConcurrentHashMap<>(); + + /** The open documents, by URI. */ + private final Map docs = new ConcurrentHashMap<>(); + + private volatile LanguageClient client; + + /** + * @param launch Gets a new server, for scripts running in an environment. + * @param environment Where the script of a document runs now. + */ + RoutingLanguageServer(final Function launch, + final Function environment) + { + this.launch = launch; + this.environment = environment; + } + + // -- LanguageClientAware -- + + @Override + public void connect(final LanguageClient c) { + client = c; + } + + // -- LanguageServer -- + + @Override + public CompletableFuture initialize( + final InitializeParams params) + { + // NB: Each server is initialized as it is launched. + return CompletableFuture.completedFuture(new InitializeResult( + new ServerCapabilities())); + } + + @Override + public void initialized(final InitializedParams params) {} + + @Override + public CompletableFuture shutdown() { + final List> all = new ArrayList<>(); + for (final LanguageServer s : servers.values()) all.add(s.shutdown()); + servers.clear(); + docs.clear(); + return CompletableFuture.allOf(all.toArray(new CompletableFuture[0])) + .thenApply(x -> null); + } + + @Override + public void exit() { + for (final LanguageServer s : servers.values()) s.exit(); + } + + @Override + public TextDocumentService getTextDocumentService() { + return this; + } + + @Override + public WorkspaceService getWorkspaceService() { + return this; + } + + // -- TextDocumentService -- + + @Override + public void didOpen(final DidOpenTextDocumentParams params) { + final TextDocumentItem item = params.getTextDocument(); + final Doc doc = new Doc(item); + docs.put(item.getUri(), doc); + WHERE.execute(() -> route(doc)); + } + + @Override + public void didChange(final DidChangeTextDocumentParams params) { + final Doc doc = docs.get(params.getTextDocument().getUri()); + if (doc == null || params.getContentChanges().isEmpty()) return; + doc.change(params.getContentChanges().get(params.getContentChanges() + .size() - 1).getText(), params.getTextDocument().getVersion()); + WHERE.execute(() -> route(doc)); + } + + @Override + public void didClose(final DidCloseTextDocumentParams params) { + final Doc doc = docs.remove(params.getTextDocument().getUri()); + if (doc != null) WHERE.execute(() -> leave(doc)); + } + + @Override + public void didSave(final DidSaveTextDocumentParams params) { + // NB: Changes come with didChange. + } + + @Override + public CompletableFuture, CompletionList>> + completion(final CompletionParams params) + { + return request(params.getTextDocument().getUri(), (s, documents) -> documents + .completion(params).thenApply(answer -> mark(answer, s))); + } + + @Override + public CompletableFuture resolveCompletionItem( + final CompletionItem item) + { + // NB: Resolved by the server that completed it. + if (!(item.getData() instanceof Origin)) { + return CompletableFuture.completedFuture(item); + } + final Origin origin = (Origin) item.getData(); + item.setData(origin.data); + return origin.server.getTextDocumentService().resolveCompletionItem(item) + .handle((resolved, error) -> { + final CompletionItem out = resolved != null ? resolved : item; + out.setData(origin); + return out; + }); + } + + @Override + public CompletableFuture signatureHelp( + final SignatureHelpParams params) + { + return request(params.getTextDocument().getUri(), (s, + documents) -> documents.signatureHelp(params)); + } + + @Override + public CompletableFuture hover(final HoverParams params) { + return request(params.getTextDocument().getUri(), (s, + documents) -> documents.hover(params)); + } + + // -- WorkspaceService -- + + @Override + public void didChangeConfiguration(final DidChangeConfigurationParams p) {} + + @Override + public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) {} + + // -- Helper methods -- + + /** + * Asks the server of where a document's script runs now (once it is up to + * date there). NB: Cancelling the answer cancels the request. + */ + private CompletableFuture request(final String uri, + final Request request) + { + final Doc doc = docs.get(uri); + if (doc == null) return CompletableFuture.completedFuture(null); + final CompletableFuture answer = new CompletableFuture<>(); + CompletableFuture.supplyAsync(() -> route(doc), WHERE).thenAccept(s -> { + if (answer.isDone()) return; + final CompletableFuture raw = request.ask(s, s + .getTextDocumentService()); + answer.whenComplete((r, error) -> { + if (answer.isCancelled()) raw.cancel(true); + }); + raw.whenComplete((r, error) -> { + if (error != null) answer.completeExceptionally(error); + else answer.complete(r); + }); + }).exceptionally(error -> { + answer.completeExceptionally(error); + return null; + }); + return answer; + } + + /** + * Brings a document up to date on the server of where its script runs now: + * if that changed, closes it on the server of where it ran, and opens it on + * this one. Called on {@link #WHERE} only. + * + * @return The server. + */ + private LanguageServer route(final Doc doc) { + final TextDocumentItem item = doc.item(); + final LanguageServer target = server(environment.apply(item)); + if (doc.server != target) { + leave(doc); + target.getTextDocumentService().didOpen(new DidOpenTextDocumentParams( + item)); + doc.server = target; + } + else if (doc.sent != item.getVersion()) { + target.getTextDocumentService().didChange(new DidChangeTextDocumentParams( + new VersionedTextDocumentIdentifier(item.getUri(), item.getVersion()), + Collections.singletonList(new TextDocumentContentChangeEvent(item + .getText())))); + } + doc.sent = item.getVersion(); + return target; + } + + /** + * Closes a document on its server; and shuts that down, if it has no other + * documents. Called on {@link #WHERE} only. + */ + private void leave(final Doc doc) { + final LanguageServer s = doc.server; + if (s == null) return; + doc.server = null; + s.getTextDocumentService().didClose(new DidCloseTextDocumentParams( + new TextDocumentIdentifier(doc.uri))); + for (final Doc other : docs.values()) { + if (other.server == s) return; + } + servers.values().remove(s); + s.shutdown(); + } + + /** The server of an environment, launching it if needed. */ + private LanguageServer server(final Environment env) { + return servers.computeIfAbsent(env, e -> { + final LanguageServer s = launch.apply(e); + if (s instanceof LanguageClientAware) { + ((LanguageClientAware) s).connect(new Forwarding(s)); + } + s.initialize(new InitializeParams()).thenRun(() -> s.initialized( + new InitializedParams())); + return s; + }); + } + + /** Marks each completion with the server that made it, for resolving. */ + private static Either, CompletionList> mark( + final Either, CompletionList> answer, + final LanguageServer s) + { + if (answer == null) return null; + final List items = answer.isRight() ? answer.getRight() + .getItems() : answer.getLeft(); + if (items != null) { + for (final CompletionItem item : items) { + item.setData(new Origin(s, item.getData())); + } + } + return answer; + } + + /** Asks a server something. */ + private interface Request { + + CompletableFuture ask(LanguageServer server, + TextDocumentService documents); + } + + /** An open document. */ + private static final class Doc { + + private final String uri; + private final String languageId; + private String text; + private int version; + /** The server it is open on (set on {@link #WHERE} only). */ + private volatile LanguageServer server; + /** The version its server last got (on {@link #WHERE} only). */ + private int sent = -1; + + private Doc(final TextDocumentItem item) { + uri = item.getUri(); + languageId = item.getLanguageId(); + text = item.getText(); + version = item.getVersion(); + } + + private synchronized void change(final String newText, + final int newVersion) + { + text = newText; + version = newVersion; + } + + private synchronized TextDocumentItem item() { + return new TextDocumentItem(uri, languageId, version, text); + } + } + + /** The server a completion was made by, and the data it came with. */ + private static final class Origin { + + private final LanguageServer server; + private final Object data; + + private Origin(final LanguageServer server, final Object data) { + this.server = server; + this.data = data; + } + } + + /** + * Passes on what a server tells the client: its problems with a document + * only while the document is open on it. + */ + private final class Forwarding implements LanguageClient { + + private final LanguageServer server; + + private Forwarding(final LanguageServer server) { + this.server = server; + } + + @Override + public void publishDiagnostics(final PublishDiagnosticsParams p) { + final LanguageClient c = client; + final Doc doc = docs.get(p.getUri()); + if (c != null && doc != null && doc.server == server) { + c.publishDiagnostics(p); + } + } + + @Override + public void telemetryEvent(final Object object) {} + + @Override + public void showMessage(final MessageParams m) { + final LanguageClient c = client; + if (c != null) c.showMessage(m); + } + + @Override + public CompletableFuture showMessageRequest( + final ShowMessageRequestParams r) + { + final LanguageClient c = client; + return c != null ? c.showMessageRequest(r) : CompletableFuture + .completedFuture(null); + } + + @Override + public void logMessage(final MessageParams m) { + final LanguageClient c = client; + if (c != null) c.logMessage(m); + } + } +} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java index 1a36054..306ec51 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java @@ -67,10 +67,10 @@ * (see {@link ApposePythonServerPlugin}), with a real jedi-language-server, through * the {@link LanguageServerService}, as editors use it. *

    - * The first completion builds the completion tooling environment (Python and - * jedi-language-server, via pixi), and a script run builds the small uv - * environment of {@code appose-test/requirements.txt}; both require network - * access. + * The first completion builds jedi-language-server's environment (see + * code-lsp-jedi), a new dependency the tooling environment that installs + * overlays, and a script run the small uv environment of + * {@code appose-test/requirements.txt}; all require network access. *

    * * @author Gabriel Selzer @@ -182,15 +182,13 @@ public void testDiagnostics() throws Exception { final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; List problems = Collections.emptyList(); while (problems.isEmpty() && System.currentTimeMillis() < deadline) { - problems = servers.diagnose(language, code, - 120_000); + problems = problems(code); } assertFalse(problems.isEmpty()); assertTrue(problems.toString(), problems.get(0).getRange().getStart() .getLine() >= 2); // Fixed: no more problems. - assertTrue(servers.diagnose(language, code - .replace("(1 +", "(1 + 2)"), 120_000).isEmpty()); + assertTrue(problems(code.replace("(1 +", "(1 + 2)")).isEmpty()); } @Test @@ -201,24 +199,6 @@ public void testParameterBeforeEnvironment() throws Exception { "shape"); } - @Test - public void testServerOutlivesNoParent() throws Exception { - // The server, once its parent (this JVM) is gone, must exit: here, its - // parent is a process that already has. - completeUntil("import json\njson.du", null, "dumps"); - final String python = ApposePythonEnvironments.pythonOnDisk(new File( - org.apposed.appose.util.Environments.apposeEnvsDir(), - ApposePythonEnvironments.TOOLING_ENV)); - final Process gone = new ProcessBuilder(python, "-c", "pass").start(); - gone.waitFor(); - final Process process = new ProcessBuilder(python, "-c", - JediServerProcessPlugin.LAUNCHER, - String.valueOf(gone.pid())).redirectErrorStream(true).start(); - final boolean exited = process.waitFor(30, TimeUnit.SECONDS); - if (!exited) process.destroyForcibly(); - assertTrue(exited); - } - @Test public void testNewDependency() throws Exception { // A script declaring a dependency that is installed nowhere: its @@ -245,6 +225,13 @@ private static ScriptSession session(final String path) { .session(language).setFile(() -> path == null ? null : new File(path))); } + /** The problems found in a script, in a session of its own. */ + private static List problems(final String code) { + try (ScriptSession session = servers.session(language)) { + return session.update(code).diagnostics(120_000); + } + } + /** Tells the servers a script's text. */ private static ScriptSession sync(final String code, final String path) { return session(path).update(code); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java index b797e82..9401a85 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java @@ -260,26 +260,6 @@ public void testPythonTypes() { assertNull(DIALECT.runtimeType(Context.class)); } - @Test - public void testServerOptions() { - final JediServerProcessPlugin spec = new JediServerProcessPlugin(() -> null, - new File("workspace"), new File("server.log")); - assertEquals("python", spec.languageId()); - final String diagnostics = "diagnostics={enable=true, didOpen=true, " + - "didChange=true, didSave=false}"; - assertEquals("{workspace={environmentPath=/env/bin/python, " + - "extraPaths=[/overlay]}, " + diagnostics + "}", String.valueOf(spec - .initializationOptions(new Environment( - "/env/bin/python", Collections.singletonList("/overlay"), null)))); - // Without a script environment: the server's own. - assertEquals("{workspace={extraPaths=[]}, " + diagnostics + "}", String - .valueOf(spec.initializationOptions( - Environment.NONE))); - assertTrue(spec.rootUri(Environment.NONE) - .startsWith("file:")); - new File("workspace").delete(); - } - // -- Helper methods -- /** The script as plain Python, as the plugin's servers see it. */ diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServerTest.java new file mode 100644 index 0000000..0c0e745 --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServerTest.java @@ -0,0 +1,304 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.util.Collections; +import java.util.List; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.TimeUnit; +import java.util.function.BooleanSupplier; + +import org.eclipse.lsp4j.CompletionItem; +import org.eclipse.lsp4j.CompletionList; +import org.eclipse.lsp4j.CompletionParams; +import org.eclipse.lsp4j.Diagnostic; +import org.eclipse.lsp4j.DidChangeConfigurationParams; +import org.eclipse.lsp4j.DidChangeTextDocumentParams; +import org.eclipse.lsp4j.DidChangeWatchedFilesParams; +import org.eclipse.lsp4j.DidCloseTextDocumentParams; +import org.eclipse.lsp4j.DidOpenTextDocumentParams; +import org.eclipse.lsp4j.DidSaveTextDocumentParams; +import org.eclipse.lsp4j.InitializeParams; +import org.eclipse.lsp4j.InitializeResult; +import org.eclipse.lsp4j.MessageActionItem; +import org.eclipse.lsp4j.MessageParams; +import org.eclipse.lsp4j.Position; +import org.eclipse.lsp4j.PublishDiagnosticsParams; +import org.eclipse.lsp4j.Range; +import org.eclipse.lsp4j.ServerCapabilities; +import org.eclipse.lsp4j.ShowMessageRequestParams; +import org.eclipse.lsp4j.TextDocumentContentChangeEvent; +import org.eclipse.lsp4j.TextDocumentIdentifier; +import org.eclipse.lsp4j.TextDocumentItem; +import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; +import org.eclipse.lsp4j.jsonrpc.messages.Either; +import org.eclipse.lsp4j.services.LanguageClient; +import org.eclipse.lsp4j.services.LanguageClientAware; +import org.eclipse.lsp4j.services.LanguageServer; +import org.eclipse.lsp4j.services.TextDocumentService; +import org.eclipse.lsp4j.services.WorkspaceService; +import org.junit.Test; +import org.scijava.code.lsp.Environment; + +/** + * Tests {@link RoutingLanguageServer}, around fake servers. + * + * @author Gabriel Selzer + */ +public class RoutingLanguageServerTest { + + private static final String URI = "untitled:/m.py"; + + private static final Environment A = new Environment("/a/python", null, + null); + + private static final Environment B = new Environment("/b/python", null, + null); + + /** The servers launched, in order. */ + private final List launched = new CopyOnWriteArrayList<>(); + + /** The problems the client heard of. */ + private final List heard = + new CopyOnWriteArrayList<>(); + + @Test + public void testMoves() throws Exception { + // Each document is analyzed by the server of where its script runs, as + // its text says here. + final RoutingLanguageServer server = new RoutingLanguageServer(env -> { + final Fake f = new Fake(env); + launched.add(f); + return f; + }, doc -> doc.getText().startsWith("#B") ? B : A); + server.connect(client()); + server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(URI, + "python", 1, "x = 1"))); + complete(server); + assertEquals(1, launched.size()); + final Fake a = launched.get(0); + assertEquals(A, a.environment); + assertEquals(Collections.singletonList("open x = 1"), a.events.subList(0, + 1)); + + // Changed, but still running there: the server is told of the change. + change(server, 2, "x = 2"); + complete(server); + assertEquals(1, launched.size()); + assertTrue(a.events.contains("change x = 2")); + + // It runs elsewhere now: the next request goes there, and the server of + // where it ran lets it go (and, with nothing left to analyze, stops). + change(server, 3, "#B\nx = 1"); + final List items = complete(server); + assertEquals(2, launched.size()); + final Fake b = launched.get(1); + assertEquals(B, b.environment); + assertTrue(b.events.contains("open #B\nx = 1")); + assertTrue(a.events.contains("close")); + assertTrue(a.events.contains("shutdown")); + + // Completions are resolved by the server that made them. + server.resolveCompletionItem(items.get(0)).get(10, TimeUnit.SECONDS); + assertTrue(b.events.contains("resolve dumps")); + assertFalse(a.events.contains("resolve dumps")); + + // Problems: heard only from the server the document is open on. + a.publish(); + b.publish(); + waitFor(() -> !heard.isEmpty()); + assertEquals(1, heard.size()); + assertEquals("/b/python", heard.get(0).getDiagnostics().get(0) + .getMessage().getLeft()); + + // Closed: let go, and the server stops. + server.didClose(new DidCloseTextDocumentParams(new TextDocumentIdentifier( + URI))); + waitFor(() -> b.events.contains("shutdown")); + assertTrue(b.events.contains("close")); + } + + // -- Helper methods -- + + private static List complete(final RoutingLanguageServer s) + throws Exception + { + return s.completion(new CompletionParams(new TextDocumentIdentifier(URI), + new Position(0, 0))).get(10, TimeUnit.SECONDS).getRight().getItems(); + } + + private static void change(final RoutingLanguageServer s, final int version, + final String text) + { + s.didChange(new DidChangeTextDocumentParams( + new VersionedTextDocumentIdentifier(URI, version), Collections + .singletonList(new TextDocumentContentChangeEvent(text)))); + } + + private static void waitFor(final BooleanSupplier condition) + throws InterruptedException + { + final long deadline = System.currentTimeMillis() + 10_000; + while (!condition.getAsBoolean()) { + if (System.currentTimeMillis() > deadline) { + throw new AssertionError("Timed out"); + } + Thread.sleep(10); + } + } + + private LanguageClient client() { + return new LanguageClient() { + + @Override + public void publishDiagnostics(final PublishDiagnosticsParams p) { + heard.add(p); + } + + @Override + public void telemetryEvent(final Object object) {} + + @Override + public void showMessage(final MessageParams m) {} + + @Override + public CompletableFuture showMessageRequest( + final ShowMessageRequestParams r) + { + return CompletableFuture.completedFuture(null); + } + + @Override + public void logMessage(final MessageParams m) {} + }; + } + + /** A server of an environment, recording what it is told. */ + private static final class Fake implements LanguageServer, + TextDocumentService, WorkspaceService, LanguageClientAware + { + + private final Environment environment; + private final List events = new CopyOnWriteArrayList<>(); + private LanguageClient client; + + private Fake(final Environment environment) { + this.environment = environment; + } + + /** Reports a problem, naming where it runs. */ + private void publish() { + client.publishDiagnostics(new PublishDiagnosticsParams(URI, Collections + .singletonList(new Diagnostic(new Range(new Position(0, 0), + new Position(0, 1)), environment.interpreter())))); + } + + @Override + public void connect(final LanguageClient c) { + client = c; + } + + @Override + public CompletableFuture initialize( + final InitializeParams params) + { + return CompletableFuture.completedFuture(new InitializeResult( + new ServerCapabilities())); + } + + @Override + public CompletableFuture shutdown() { + events.add("shutdown"); + return CompletableFuture.completedFuture(null); + } + + @Override + public void exit() {} + + @Override + public TextDocumentService getTextDocumentService() { + return this; + } + + @Override + public WorkspaceService getWorkspaceService() { + return this; + } + + @Override + public void didOpen(final DidOpenTextDocumentParams p) { + events.add("open " + p.getTextDocument().getText()); + } + + @Override + public void didChange(final DidChangeTextDocumentParams p) { + events.add("change " + p.getContentChanges().get(0).getText()); + } + + @Override + public void didClose(final DidCloseTextDocumentParams p) { + events.add("close"); + } + + @Override + public void didSave(final DidSaveTextDocumentParams p) {} + + @Override + public CompletableFuture, CompletionList>> + completion(final CompletionParams p) + { + events.add("complete"); + final CompletionItem item = new CompletionItem("dumps"); + item.setData("dumps"); + return CompletableFuture.completedFuture(Either.forRight( + new CompletionList(false, new CopyOnWriteArrayList<>(Collections + .singletonList(item))))); + } + + @Override + public CompletableFuture resolveCompletionItem( + final CompletionItem item) + { + events.add("resolve " + item.getData()); + return CompletableFuture.completedFuture(item); + } + + @Override + public void didChangeConfiguration(final DidChangeConfigurationParams p) {} + + @Override + public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) {} + } +} From 3a784e572ffda2368631e8d16be5144a6db2fd78 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 16 Oct 2026 00:26:51 -0500 Subject: [PATCH 26/37] Serve the script language by its name, as code-lsp now asks code-lsp's sessions ask for the LSP language named as the script's language, in lower case, so the plugin no longer declares the script language it serves separately. Co-Authored-By: Claude Opus 5.5 --- .../python/completion/ApposePythonServerPlugin.java | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java index c896e70..653c4ab 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java @@ -37,9 +37,7 @@ import org.scijava.code.lsp.ScriptDialect; import org.scijava.plugin.Parameter; import org.scijava.plugin.Plugin; -import org.scijava.plugins.scripting.appose.python.ApposePythonScriptLanguage; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; -import org.scijava.script.ScriptLanguage; /** * A {@link LanguageServerPlugin} for Appose-Python scripts, backed by @@ -64,17 +62,11 @@ public class ApposePythonServerPlugin implements LanguageServerPlugin { @Override public boolean supports(final String languageId) { - // NB: Not "python", which this builds on. Without jedi, there is - // nothing to serve. + // NB: Appose-Python scripts too, by their language's name. Not "python", + // which this builds on. Without jedi, there is nothing to serve. return "appose-python".equals(languageId) && jediOffered(); } - @Override - public boolean supports(final ScriptLanguage language) { - // NB: Without jedi, there is nothing to serve. - return language instanceof ApposePythonScriptLanguage && jediOffered(); - } - @Override public LanguageServer create(final String languageId, final Environment environment) From 856b76f3c172a3b90a0bea2d5c0a42339226a47c Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 16 Oct 2026 10:26:21 -0500 Subject: [PATCH 27/37] Offer only an Appose-Python dialect: completion moves to code-lsp-jedi ApposePythonDialect is now a ScriptDialect plugin: it shows scripts as plain Python and says which built Appose environment each runs in. The tuned server, the overlays and the routing moved to code-lsp-jedi and code-lsp. Co-Authored-By: Claude Opus 5.5 --- README.md | 21 +- .../appose/python/ScriptEnvironment.java | 2 +- .../completion/ApposePythonDialect.java | 42 +- .../completion/ApposePythonEnvironments.java | 216 +------- .../ApposePythonLanguageServer.java | 484 ------------------ .../completion/ApposePythonServerPlugin.java | 91 ---- .../python/completion/DependencyOverlays.java | 175 ------- .../completion/RoutingLanguageServer.java | 443 ---------------- .../appose/python/completion/overlay.py | 127 ----- ...ApposePythonCompletionIntegrationTest.java | 5 +- ...Test.java => ApposePythonDialectTest.java} | 66 +-- .../ApposePythonLanguageServerTest.java | 310 ----------- .../completion/DependencyOverlaysTest.java | 109 ---- .../completion/RoutingLanguageServerTest.java | 304 ----------- 14 files changed, 88 insertions(+), 2307 deletions(-) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServer.java delete mode 100644 src/main/resources/org/scijava/plugins/scripting/appose/python/completion/overlay.py rename src/test/java/org/scijava/plugins/scripting/appose/python/completion/{ApposePythonServersTest.java => ApposePythonDialectTest.java} (78%) delete mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java delete mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java delete mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServerTest.java diff --git a/README.md b/README.md index 4d26f3d..83da5bd 100644 --- a/README.md +++ b/README.md @@ -137,19 +137,20 @@ script's own line numbers. ## Code completion -The Script Editor completes code in these scripts with a language server -(offered to `scijava-code-lsp`'s `LanguageServerService` by -`ApposePythonServerPlugin`, so any other LSP client of it gets the same): +The Script Editor completes code in these scripts with a Python language +server, `scijava-code-lsp-jedi`'s: [jedi-language-server](https://github.com/pappasam/jedi-language-server) -(built on [Jedi](https://jedi.readthedocs.io/)). It completes names in scope, +(built on [Jedi](https://jedi.readthedocs.io/)). This plugin's +`ApposePythonDialect` shows it each script as plain Python, and says where +the script runs (so any other client of `scijava-code-lsp`'s +`LanguageServerService` gets the same). It completes names in scope, the script's `#@` parameters, `task`, builtins and keywords, and members of objects and modules; and it gives function signatures (with parameter names and types), documentation, hover and syntax errors. -The server, offered by `scijava-code-lsp-jedi`, runs in a small environment -of its own (`code-lsp-jedi`: Python, Jedi, jedi-language-server and NumPy), -which is built in the background the first time you complete code, and -takes a while only then; until it is, there are no completions. The server +The server runs in a small environment of its own (`code-lsp-jedi`: Python, +Jedi, jedi-language-server, NumPy and uv), which is built in the background +the first time you complete code, and takes a while only then; until it is, there are no completions. The server talks to the editor via the [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) (see `scijava-code-lsp`); it sees the script as plain Python, with its `#@` @@ -163,8 +164,8 @@ installed there are known too. Packages the script declares (in its `# /// script` block) but its environment lacks, e.g. a dependency you just added, are installed for completion only, into a separate directory, a moment after you stop editing the dependencies; so they are known before you run the -script. (They are installed with uv, from another small environment, -`appose-python-completion`.) This covers PyPI packages; conda packages (`[tool.pixi.dependencies]`) +script. (`scijava-code-lsp-jedi` does this, for any Python script.) This +covers PyPI packages; conda packages (`[tool.pixi.dependencies]`) and environments declared in files are known once the script has run. Completion never builds a script's environment itself, and never disturbs its worker. Script parameters are known by the Python types they have at run diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java index c620bdb..6442b2f 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java @@ -50,7 +50,7 @@ *

    * Both running a script ({@link ApposePythonScriptEngine}) and completing code * in it ({@link - * org.scijava.plugins.scripting.appose.python.completion.ApposePythonServerPlugin}) + * org.scijava.plugins.scripting.appose.python.completion.ApposePythonDialect}) * resolve its environment here, so that they agree on the environment's name * and configuration. *

    diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index e758e90..4220fde 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -41,18 +41,48 @@ import net.imglib2.RandomAccessibleInterval; import org.apposed.appose.NDArray; +import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.ScriptDialect; +import org.scijava.plugin.Parameter; +import org.scijava.plugin.Plugin; +import org.scijava.plugins.scripting.appose.python.ApposePythonScriptLanguage; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; +import org.scijava.script.ScriptLanguage; /** - * Describes how to turn Appose python scripts into plain python code. + * Describes how to turn Appose python scripts into plain python code, and + * where they run (see {@link ApposePythonEnvironments}). * * @author Gabriel Selzer */ -final class ApposePythonDialect implements ScriptDialect { +@Plugin(type = ScriptDialect.class, name = "Appose-Python") +public class ApposePythonDialect implements ScriptDialect { /** The type of the {@code task} variable scripts can use. */ private static final String TASK_TYPE = "appose.python_worker.Task"; + @Parameter(required = false) + private ResidentWorkerService workers; + + private ApposePythonEnvironments environments; + + @Override + public boolean supports(final ScriptLanguage language) { + return language instanceof ApposePythonScriptLanguage; + } + + @Override + public String languageId() { + return "python"; + } + + @Override + public Environment environment(final String text, final File file) { + // NB: As its #@script line, or its inline metadata, says. + return environments().environment(text, file == null ? null : file + .getPath()); + } + @Override public String runtimeType(final Class type) { if (type == null) return null; @@ -103,4 +133,12 @@ public String imported(final String statement) { public List extraDeclarations() { return Collections.singletonList(declare("task", TASK_TYPE)); } + + /** Gets where scripts run, creating it on first use. */ + private synchronized ApposePythonEnvironments environments() { + if (environments == null) { + environments = new ApposePythonEnvironments(workers); + } + return environments; + } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java index 306d8c7..3453f83 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java @@ -31,23 +31,14 @@ import java.io.File; import java.io.IOException; -import java.io.InputStream; -import java.net.URI; -import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.Arrays; -import java.util.Collections; -import java.util.HashMap; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; -import java.util.WeakHashMap; -import org.apposed.appose.Appose; import org.apposed.appose.Environment; -import org.apposed.appose.Service; import org.apposed.appose.util.Environments; -import org.eclipse.lsp4j.TextDocumentItem; import org.scijava.plugins.scripting.appose.python.ScriptEnvironment; import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; @@ -56,31 +47,15 @@ /** * Where Python scripts run via Appose, as their language servers need to * know: the Python of a script's environment, if built (in this session, or an - * earlier one; never building it, nor disturbing its worker); the overlay of - * packages it declares but its environment lacks (see - * {@link DependencyOverlays}), and the tooling environment overlays are - * installed from. + * earlier one; never building it, nor disturbing its worker). (Packages a + * script declares, but its environment lacks, are known to the servers too; + * see scijava-code-lsp-jedi.) * * @author Gabriel Selzer */ final class ApposePythonEnvironments { - /** Name of the tooling environment that installs overlays. */ - static final String TOOLING_ENV = "appose-python-completion"; - - /** - * How long a script's dependencies must stay unchanged before packages - * missing from its environment are installed for completion, in ms. - */ - private static final long OVERLAY_SETTLE = 1500; - - /** Task scripts (see overlay.py), by resource name. */ - private static final Map SCRIPTS = new HashMap<>(); - - /** One per resident worker service (that is, per context). */ - private static final Map ALL = - new WeakHashMap<>(); - + /** The resident workers, to find built environments; or null if none. */ private final ResidentWorkerService workers; /** Python executables of script environments, by script and metadata. */ @@ -96,132 +71,25 @@ protected boolean removeEldestEntry( } }; - private DependencyOverlays overlays; - - private ApposePythonEnvironments(final ResidentWorkerService workers) { + /** @param workers The resident workers, or null if none. */ + ApposePythonEnvironments(final ResidentWorkerService workers) { this.workers = workers; } - /** Gets those of a context (by its resident worker service). */ - static ApposePythonEnvironments of(final ResidentWorkerService workers) { - synchronized (ALL) { - return ALL.computeIfAbsent(workers, ApposePythonEnvironments::new); - } - } - - /** - * Derives an {@link org.scijava.code.lsp.Environment} for an Appose script. - *

    - * The {@code python} executable is found using the script's text and file - * path. For convenience, packages declared in the script but not yet - * installed (i.e. between the time the dependency is added to the script - * and the time when it is run) are provided to the language server in the - * overlay. - *

    - * - * @param script the script - * @return an {@link org.scijava.code.lsp.Environment} where: - *
      - *
    • the interpreter is the Python executable of the script's - * declared environment, if that has already been built, - * or {@code null} otherwise;
      • - *
    • the paths hold the overlay, once installed; none if no package - * is missing, or until it is installed (which starts once the - * script's dependencies stop changing, or at once if the script was - * just opened).
      • - *
    - */ - org.scijava.code.lsp.Environment environment(final TextDocumentItem script) { - final String text = script.getText(); - final String path = path(script.getUri()); - final String python = scriptPython(text, path); - final boolean settled = script.getVersion() <= 1; - // NB: An alternative design could have all the script's dependencies - // in a constant overlay, but that would have the effect of installing them - // twice - once for the overlay and once for the script's environment. - final String overlay = overlay(text, path, python, settled); - return new org.scijava.code.lsp.Environment(python, overlay == null ? null - : Collections.singletonList(overlay), null); - } - - /** Sets the dependency overlays, e.g. to use fake ones in tests. */ - synchronized void setOverlays(final DependencyOverlays overlays) { - this.overlays = overlays; - } - - /** The file path of a {@code file:} URI, or null. */ - static String path(final String uri) { - try { - final URI u = URI.create(uri); - return "file".equals(u.getScheme()) ? new File(u).getPath() : null; - } - catch (final IllegalArgumentException exc) { - return null; - } - } - /** - * Gets the overlay of packages the script declares but its environment - * lacks (see {@link DependencyOverlays}), if ready; starts preparing it - * otherwise. Only for environments declared inline, for now. + * Gets where an Appose script runs: the Python executable of its declared + * environment, if that has already been built; or none (the + * servers' own Python), otherwise. * - * @param settled Whether the dependencies are known not to be changing. - * @return The overlay directory, or null if none is needed or ready. + * @param text The script's text. + * @param path The script's file path, or null if it has none. */ - String overlay(final String text, final String path, - final String python, final boolean settled) + org.scijava.code.lsp.Environment environment(final String text, + final String path) { - final DependencyOverlays overlays = overlays(); - if (overlays == null) return null; - final List dependencies; - final String name; - try { - final String metadata = InlineMetadata.extract(text); - if (metadata == null) return null; - dependencies = InlineMetadata.dependencies(metadata); - // NB: Unsaved scripts' environments are named after their metadata, - // which changes as dependencies are edited: share one overlay. - name = path == null ? "unsaved" : ScriptEnvironment.inlineName(metadata, - path); - } - catch (final RuntimeException exc) { - return null; // NB: E.g. half-typed metadata. - } - return overlays.overlay(name, python, dependencies, settled); - } - - /** Gets the dependency overlays, creating them on first use. */ - private synchronized DependencyOverlays overlays() { - if (overlays != null) return overlays; - if (workers == null) return null; - overlays = new DependencyOverlays((python, dependencies, - dir) -> installOverlay(workers, python, dependencies, dir), new File( - Environments.apposeEnvsDir(), TOOLING_ENV + "-overlays"), - OVERLAY_SETTLE, System::currentTimeMillis); - return overlays; - } - - /** - * Installs an overlay by running overlay.py on the tooling environment's - * worker. Called on the overlays' background thread. - */ - private static String installOverlay(final ResidentWorkerService workers, - final String python, final List dependencies, final File dir) - { - final Map inputs = new HashMap<>(); - inputs.put("python", python); - inputs.put("dependencies", dependencies); - inputs.put("overlay", dir.getAbsolutePath()); - try { - final Service.Task task = toolingWorker(workers).task(resource( - "overlay.py"), inputs); - task.waitFor(); - final Object overlay = task.outputs.get("path"); - return overlay instanceof String ? (String) overlay : null; - } - catch (final Exception exc) { - throw new IllegalStateException("Overlay installation failed", exc); - } + final String python = scriptPython(text, path); + return python == null ? org.scijava.code.lsp.Environment.NONE + : new org.scijava.code.lsp.Environment(python, null, null); } /** @@ -340,56 +208,4 @@ private static String findPython(final List dirs) { } return null; } - - /** Gets the tooling environment's worker, building it first if needed. */ - private static ResidentWorker toolingWorker( - final ResidentWorkerService workers) - { - final String config = toolingConfig(); - return workers.worker(TOOLING_ENV, config, () -> Appose.content(config) - .scheme("pixi.toml"), Environment::python); - } - - /** - * The tooling environment: Python and Appose, to run overlay.py; and uv, to - * install packages for {@link DependencyOverlays}. - */ - static String toolingConfig() { - return "[workspace]\n" + // - "name = \"" + TOOLING_ENV + "\"\n" + // - "channels = [\"conda-forge\"]\n" + // - "platforms = [\"" + InlineMetadata.condaPlatform() + "\"]\n" + // - "\n" + // - "[dependencies]\n" + // - // NB: The overlays of scripts whose environment is not built are - // installed for this Python, for jedi-language-server's Python (see - // code-lsp-jedi) to find: the versions must match. - "python = \"3.12.*\"\n" + // - "\n" + // - "[pypi-dependencies]\n" + // - "appose = \"*\"\n" + // - "uv = \"*\"\n"; - } - - /** Gets a task script (e.g. overlay.py), by resource name. */ - private static String resource(final String name) throws IOException { - synchronized (SCRIPTS) { - final String known = SCRIPTS.get(name); - if (known != null) return known; - try (final InputStream in = ApposePythonEnvironments.class - .getResourceAsStream(name)) - { - if (in == null) throw new IOException(name + " not found"); - final java.io.ByteArrayOutputStream out = - new java.io.ByteArrayOutputStream(); - final byte[] buf = new byte[8192]; - int n; - while ((n = in.read(buf)) > 0) out.write(buf, 0, n); - final String script = new String(out.toByteArray(), - StandardCharsets.UTF_8); - SCRIPTS.put(name, script); - return script; - } - } - } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java deleted file mode 100644 index 91d4a44..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServer.java +++ /dev/null @@ -1,484 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.util.ArrayList; -import java.util.Comparator; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Locale; -import java.util.Map; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.ConcurrentHashMap; -import java.util.function.LongSupplier; -import java.util.regex.Pattern; - -import org.eclipse.lsp4j.CompletionItem; -import org.eclipse.lsp4j.CompletionItemKind; -import org.eclipse.lsp4j.CompletionList; -import org.eclipse.lsp4j.CompletionParams; -import org.eclipse.lsp4j.DidChangeConfigurationParams; -import org.eclipse.lsp4j.DidChangeTextDocumentParams; -import org.eclipse.lsp4j.DidChangeWatchedFilesParams; -import org.eclipse.lsp4j.DidCloseTextDocumentParams; -import org.eclipse.lsp4j.DidOpenTextDocumentParams; -import org.eclipse.lsp4j.DidSaveTextDocumentParams; -import org.eclipse.lsp4j.Hover; -import org.eclipse.lsp4j.HoverParams; -import org.eclipse.lsp4j.InitializeParams; -import org.eclipse.lsp4j.InitializeResult; -import org.eclipse.lsp4j.InitializedParams; -import org.eclipse.lsp4j.SignatureHelp; -import org.eclipse.lsp4j.SignatureHelpParams; -import org.eclipse.lsp4j.jsonrpc.messages.Either; -import org.eclipse.lsp4j.services.LanguageClient; -import org.eclipse.lsp4j.services.LanguageClientAware; -import org.eclipse.lsp4j.services.LanguageServer; -import org.eclipse.lsp4j.services.TextDocumentService; -import org.eclipse.lsp4j.services.WorkspaceService; -import org.scijava.code.lsp.jvm.Callable; -import org.scijava.code.lsp.Positions; - -/** - * The language server of Python scripts run via Appose (see - * {@link ApposePythonServerPlugin}): a Python language server, i.e. - * jedi-language-server, whose completions are made to work well in an editor - * completing as the user types: - *
      - *
    • Asks about a word's first character only, and reuses the answer - * (filtered) as the word is typed on, for a while.
      • - *
    • Cancels a question still pending once another is asked about the same - * document.
      • - *
    • Once a word is begun, resolves the first completions, and makes those - * with signatures (e.g. {@code def dumps(obj, *, skipkeys=False) -> str}) - * {@link Callable}s: snippets with their parameters and types. (Not in - * imports: an import names things, rather than calling them.)
      • - *
    - *

    - * Everything else is passed on to the server as is. - *

    - * - * @author Gabriel Selzer - */ -class ApposePythonLanguageServer implements LanguageServer, - TextDocumentService, WorkspaceService, LanguageClientAware -{ - - /** How long an answer may be reused, in milliseconds. */ - private static final long ANSWER_LIFETIME = 20_000; - - /** An import statement being typed. */ - private static final Pattern IMPORT = Pattern.compile( - "^\\s*(from|import)\\s"); - - /** The Python language server. */ - private final LanguageServer inner; - - private final int signatures; - private final LongSupplier clock; - - /** The open documents' texts, by URI. */ - private final Map texts = new ConcurrentHashMap<>(); - - /** Answers, by question; least recently used first. */ - private final Map answers = new LinkedHashMap( - 16, 0.75f, true) - { - - @Override - protected boolean removeEldestEntry(final Map.Entry e) { - return size() > 32; - } - }; - - /** The latest question about each document, by URI. */ - private final Map pending = new ConcurrentHashMap<>(); - - /** - * @param inner The Python language server. - * @param signatures How many completions to resolve for their signatures, - * once a word is begun. - */ - ApposePythonLanguageServer(final LanguageServer inner, - final int signatures) - { - this(inner, signatures, System::currentTimeMillis); - } - - ApposePythonLanguageServer(final LanguageServer inner, final int signatures, - final LongSupplier clock) - { - this.inner = inner; - this.signatures = signatures; - this.clock = clock; - } - - // -- LanguageClientAware -- - - @Override - public void connect(final LanguageClient client) { - if (inner instanceof LanguageClientAware) { - ((LanguageClientAware) inner).connect(client); - } - } - - // -- LanguageServer -- - - @Override - public CompletableFuture initialize( - final InitializeParams params) - { - return inner.initialize(params); - } - - @Override - public void initialized(final InitializedParams params) { - inner.initialized(params); - } - - @Override - public CompletableFuture shutdown() { - return inner.shutdown(); - } - - @Override - public void exit() { - inner.exit(); - } - - @Override - public TextDocumentService getTextDocumentService() { - return this; - } - - @Override - public WorkspaceService getWorkspaceService() { - return this; - } - - // -- TextDocumentService -- - - @Override - public void didOpen(final DidOpenTextDocumentParams params) { - texts.put(params.getTextDocument().getUri(), params.getTextDocument() - .getText()); - inner.getTextDocumentService().didOpen(params); - } - - @Override - public void didChange(final DidChangeTextDocumentParams params) { - if (!params.getContentChanges().isEmpty()) { - texts.put(params.getTextDocument().getUri(), params.getContentChanges() - .get(params.getContentChanges().size() - 1).getText()); - } - inner.getTextDocumentService().didChange(params); - } - - @Override - public void didClose(final DidCloseTextDocumentParams params) { - texts.remove(params.getTextDocument().getUri()); - pending.remove(params.getTextDocument().getUri()); - inner.getTextDocumentService().didClose(params); - } - - @Override - public CompletableFuture, CompletionList>> - completion(final CompletionParams params) - { - final String uri = params.getTextDocument().getUri(); - final String text = texts.get(uri); - if (text == null) return inner.getTextDocumentService().completion( - params); - final int caret = Math.min(Positions.offset(text, params.getPosition()), - text.length()); - int start = caret; - while (start > 0 && Character.isJavaIdentifierPart(text.charAt(start - - 1))) start--; - final String word = text.substring(start, caret); - // Ask about the first character typed, if any: later keystrokes of the - // same word then reuse the answer (filtered). - final int ask = start + Math.min(1, word.length()); - final String line = text.substring(text.lastIndexOf('\n', start - 1) + 1, - start); - final boolean importing = IMPORT.matcher(line).find(); - final String key = uri + "\n" + text.substring(0, ask); - final long now = clock.getAsLong(); - Answer answer; - synchronized (answers) { - answer = answers.get(key); - // NB: Not an empty answer either: the server may know more by now (e.g. - // once the script's packages are, or its environment is built). - if (answer != null && (now - answer.time > ANSWER_LIFETIME || - answer.future.isCompletedExceptionally() || answer.isEmpty())) - { - answer = null; - } - } - if (answer == null) { - final CompletionParams p = new CompletionParams(params.getTextDocument(), - Positions.position(text, ask), params.getContext()); - final CompletableFuture, CompletionList>> raw = - inner.getTextDocumentService().completion(p); - final int n = word.isEmpty() || importing ? 0 : signatures; - answer = new Answer(raw, raw.thenCompose(list -> resolve(list, n)), now); - synchronized (answers) { - answers.put(key, answer); - } - final Answer previous = pending.put(uri, answer); - if (previous != null && previous != answer && !previous.future - .isDone()) - { - // NB: Cancelling the request tells the server ($/cancelRequest). - previous.future.cancel(true); - previous.raw.cancel(true); - } - } - return answer.future.thenApply(list -> Either.forRight(filter(list, - word))); - } - - @Override - public void didSave(final DidSaveTextDocumentParams params) { - inner.getTextDocumentService().didSave(params); - } - - @Override - public CompletableFuture resolveCompletionItem( - final CompletionItem item) - { - return inner.getTextDocumentService().resolveCompletionItem(item); - } - - @Override - public CompletableFuture signatureHelp( - final SignatureHelpParams params) - { - return inner.getTextDocumentService().signatureHelp(params); - } - - @Override - public CompletableFuture hover(final HoverParams params) { - return inner.getTextDocumentService().hover(params); - } - - // -- WorkspaceService -- - - @Override - public void didChangeConfiguration(final DidChangeConfigurationParams p) { - inner.getWorkspaceService().didChangeConfiguration(p); - } - - @Override - public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) { - inner.getWorkspaceService().didChangeWatchedFiles(p); - } - - // -- Helper methods -- - - /** Resolves the first completions, making those with signatures callables. */ - private CompletableFuture resolve( - final Either, CompletionList> answer, final int n) - { - final List items = new ArrayList<>(answer == null - ? new ArrayList<>() : answer.isRight() ? answer.getRight().getItems() - : answer.getLeft()); - items.sort(Comparator.comparing(i -> i.getSortText() != null ? i - .getSortText() : i.getLabel())); - final int count = Math.min(n, items.size()); - final List> resolved = new ArrayList<>(); - for (int i = 0; i < count; i++) { - // NB: Unresolved, should resolving fail. - resolved.add(inner.getTextDocumentService().resolveCompletionItem(items - .get(i)).handle((r, error) -> r)); - } - return CompletableFuture.allOf(resolved.toArray(new CompletableFuture[0])) - .thenApply(x -> { - for (int i = 0; i < count; i++) { - final CompletionItem r = resolved.get(i).join(); - final CompletionItem callable = r == null ? null : callable(items - .get(i), r); - if (callable != null) items.set(i, callable); - } - return new CompletionList(false, items); - }); - } - - /** - * A completion as a callable, from its resolved signature (its detail); - * or null if it has none. - */ - private static CompletionItem callable(final CompletionItem item, - final CompletionItem resolved) - { - final CompletionItemKind kind = item.getKind(); - if (kind != CompletionItemKind.Function && kind != CompletionItemKind.Method && - kind != CompletionItemKind.Class && kind != CompletionItemKind.Constructor) - { - return null; - } - final String name = item.getInsertText() != null ? item.getInsertText() - : item.getLabel(); - if (resolved.getDetail() == null) return null; - final Callable parsed = parse(name, resolved.getDetail()); - if (parsed == null) return null; - final CompletionItem out = new Callable(name, parsed.params(), parsed - .returnType()).kind(CompletionItemKind.Method).detail(resolved - .getDetail()).toItem(); - out.setSortText(item.getSortText()); - out.setDocumentation(resolved.getDocumentation()); - out.setData(item.getData()); - return out; - } - - /** The completions of the word typed so far. */ - private static CompletionList filter(final CompletionList list, - final String word) - { - final String w = word.toLowerCase(Locale.ROOT); - final List out = new ArrayList<>(); - for (final CompletionItem item : list.getItems()) { - final String name = item.getFilterText() != null ? item.getFilterText() - : item.getLabel(); - if (name.startsWith("_") && !word.startsWith("_")) continue; - if (!name.toLowerCase(Locale.ROOT).startsWith(w) || name.equals(word)) { - continue; - } - out.add(item); - } - return new CompletionList(false, out); - } - - private static final class Answer { - - private final CompletableFuture raw; - private final CompletableFuture future; - private final long time; - - private Answer(final CompletableFuture raw, - final CompletableFuture future, final long time) - { - this.raw = raw; - this.future = future; - this.time = time; - } - - /** Whether the server answered, with no completions. */ - private boolean isEmpty() { - final CompletionList list = future.getNow(null); - return list != null && list.getItems().isEmpty(); - } - } - - /** - * Parses a Python signature (as Jedi writes them): the parameters, with - * their annotations as types; and the return annotation. Returns null if it - * is not a signature. - */ - static Callable parse(final String name, - final String signature) - { - final int open = signature.indexOf('('); - if (open < 0) return null; - final int close = closing(signature, open); - if (close < 0) return null; - final List params = new ArrayList<>(); - for (final String part : split(signature.substring(open + 1, close))) { - final String p = part.trim(); - // NB: Bare * and / only mark keyword-only and positional-only ones. - if (p.isEmpty() || p.equals("*") || p.equals("/")) continue; - final int colon = topLevel(p, ':'), eq = topLevel(p, '='); - String paramName = p, type = null; - if (colon >= 0 && (eq < 0 || colon < eq)) { - paramName = p.substring(0, colon); - type = p.substring(colon + 1, eq < 0 ? p.length() : eq).trim(); - } - else if (eq >= 0) paramName = p.substring(0, eq); - paramName = paramName.trim().replaceFirst("^\\*{1,2}", ""); - params.add(new Callable.Param(paramName, type == null || type - .isEmpty() ? null : type)); - } - final String rest = signature.substring(close + 1).trim(); - final String returns = rest.startsWith("->") ? rest.substring(2).trim() - : null; - return new Callable(name, params, returns == null || returns.isEmpty() - ? null : returns); - } - - /** The index of the bracket closing the one at {@code open}, or -1. */ - private static int closing(final String s, final int open) { - int depth = 0; - char quote = 0; - for (int i = open; i < s.length(); i++) { - final char c = s.charAt(i); - if (quote != 0) { - if (c == '\\') i++; - else if (c == quote) quote = 0; - } - else if (c == '\'' || c == '"') quote = c; - else if (c == '(' || c == '[' || c == '{') depth++; - else if (c == ')' || c == ']' || c == '}') { - if (--depth == 0) return i; - } - } - return -1; - } - - /** Splits at the commas outside brackets and strings. */ - private static List split(final String s) { - final List parts = new ArrayList<>(); - int start = 0; - while (true) { - final int comma = topLevel(s.substring(start), ','); - if (comma < 0) break; - parts.add(s.substring(start, start + comma)); - start += comma + 1; - } - parts.add(s.substring(start)); - return parts; - } - - /** The index of the first {@code target} outside brackets and strings. */ - private static int topLevel(final String s, final char target) { - int depth = 0; - char quote = 0; - for (int i = 0; i < s.length(); i++) { - final char c = s.charAt(i); - if (quote != 0) { - if (c == '\\') i++; - else if (c == quote) quote = 0; - } - else if (c == '\'' || c == '"') quote = c; - else if (c == '(' || c == '[' || c == '{') depth++; - else if (c == ')' || c == ']' || c == '}') depth--; - else if (c == target && depth == 0) return i; - } - return -1; - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java deleted file mode 100644 index 653c4ab..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServerPlugin.java +++ /dev/null @@ -1,91 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import org.eclipse.lsp4j.services.LanguageServer; -import org.scijava.Priority; -import org.scijava.code.lsp.Environment; -import org.scijava.code.lsp.LanguageServerPlugin; -import org.scijava.code.lsp.LanguageServerService; -import org.scijava.code.lsp.ScriptDialect; -import org.scijava.plugin.Parameter; -import org.scijava.plugin.Plugin; -import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; - -/** - * A {@link LanguageServerPlugin} for Appose-Python scripts, backed by - * jedi-language-server. - * - * @author Gabriel Selzer - */ -@Plugin(type = LanguageServerPlugin.class, name = "appose-python", - priority = Priority.HIGH) -public class ApposePythonServerPlugin implements LanguageServerPlugin { - - /** How many completions get signatures, once a word is begun. */ - private static final int SIGNATURES = 30; - - private static final ApposePythonDialect DIALECT = new ApposePythonDialect(); - - @Parameter - private LanguageServerService servers; - - @Parameter - private ResidentWorkerService workers; - - @Override - public boolean supports(final String languageId) { - // NB: Appose-Python scripts too, by their language's name. Not "python", - // which this builds on. Without jedi, there is nothing to serve. - return "appose-python".equals(languageId) && jediOffered(); - } - - @Override - public LanguageServer create(final String languageId, - final Environment environment) - { - // NB: Where each script runs is as its text says (see - // ApposePythonEnvironments), whatever the caller thinks; and may change - // as it is edited, moving it to another Python server. - return new ApposePythonLanguageServer(new RoutingLanguageServer( - env -> servers.launch("python", env), ApposePythonEnvironments.of( - workers)::environment), SIGNATURES); - } - - @Override - public ScriptDialect dialect() { - return DIALECT; - } - - /** Whether jedi-language-server (or another Python server) is offered. */ - boolean jediOffered() { - return servers.supports("python"); - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java deleted file mode 100644 index 8e8b131..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java +++ /dev/null @@ -1,175 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.io.File; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.function.LongSupplier; - -/** - * Makes packages a script declares, but its environment lacks, known to code - * completion before the environment is rebuilt (by running the script): e.g. - * a dependency just added to the script's {@code # /// script} block. - *

    - * The missing packages are installed, in the background, into a separate - * directory per script environment (an "overlay"), which Jedi adds to its - * search path. The environment itself is never touched, so this never - * disturbs its resident worker. Installs start only once the dependencies - * have stopped changing for a moment, so that half-typed names do not cause - * installs; and only if the missing packages differ from the overlay's. - *

    - *

    - * Alternatives considered, which may be worth revisiting: - *

    - *
      - *
    • Rebuilding the script's environment as its dependencies - * change. Jedi would then see exactly what will run, but builds can take - * minutes and gigabytes, half-finished edits would trigger builds, and Appose - * rebuilds environments in place, which can interfere with (or, by changing - * its configuration, release) a resident worker using it.
      • - *
    • Installing nothing, but hinting that new dependencies are - * known only once the script has run. Cheap and honest, but the user must - * run the script first. It may still be worthwhile for dependencies an - * overlay cannot provide: conda packages (e.g. - * {@code [tool.pixi.dependencies]}), and environments declared in files via - * {@code #@script(env=...)}, which overlays do not handle yet.
      • - *
    - * - * @author Gabriel Selzer - */ -final class DependencyOverlays { - - /** Installs the missing packages of a script's environment. */ - @FunctionalInterface - interface Installer { - - /** - * Installs the packages among {@code dependencies} that the environment - * with the given Python lacks into the given overlay directory. - * - * @param python The environment's Python executable, or null if the - * environment has not been built (all packages are missing). - * @param dependencies The script's dependencies, as PEP 508 strings. - * @param overlay The directory to install into. - * @return The overlay directory, or null if nothing is missing. - */ - String install(String python, List dependencies, File overlay); - } - - private final Installer installer; - private final File root; - private final long settleMillis; - private final LongSupplier clock; - private final ExecutorService executor = Executors.newSingleThreadExecutor( - r -> { - final Thread t = new Thread(r, "appose-python-overlays"); - t.setDaemon(true); - return t; - }); - - /** Installs, by environment name, Python and dependencies. */ - private final Map> installs = - new HashMap<>(); - - /** The dependencies last seen for each environment, and since when. */ - private final Map seen = new HashMap<>(); - - /** - * @param installer Does the installing, on a background thread. - * @param root Directory holding the overlays (one per environment). - * @param settleMillis How long dependencies must stay unchanged before - * their missing packages are installed. - * @param clock The current time, in milliseconds. - */ - DependencyOverlays(final Installer installer, final File root, - final long settleMillis, final LongSupplier clock) - { - this.installer = installer; - this.root = root; - this.settleMillis = settleMillis; - this.clock = clock; - } - - /** - * Gets the overlay providing the packages the environment lacks, if it is - * ready; starts preparing it once the dependencies have settled. Never - * blocks. - * - * @param envName The environment's name, which names its overlay. - * @param python The environment's Python executable, or null if not built. - * @param dependencies The script's dependencies (PEP 508 strings). - * @param settled Whether the dependencies are known not to be changing - * (e.g. the script was just opened), so need no time to settle. - * @return The overlay directory, or null if none is needed or ready. - */ - String overlay(final String envName, final String python, - final List dependencies, final boolean settled) - { - if (dependencies == null || dependencies.isEmpty()) return null; - final String key = envName + "\n" + python + "\n" + dependencies; - synchronized (installs) { - final CompletableFuture install = installs.get(key); - if (install != null) { - return install.isDone() && !install.isCompletedExceptionally() - ? install.join() : null; - } - // Wait for the dependencies to settle, e.g. a name to be typed fully. - final long now = clock.getAsLong(); - final Seen last = seen.get(envName); - if (last == null || !last.key.equals(key)) { - seen.put(envName, new Seen(key, now)); - if (!settled) return null; - } - else if (!settled && now - last.since < settleMillis) return null; - final File overlay = new File(root, envName); - final CompletableFuture started = CompletableFuture.supplyAsync( - () -> installer.install(python, dependencies, overlay), executor); - // NB: Once done, asking again gets the overlay. - installs.put(key, started); - return null; - } - } - - private static final class Seen { - - private final String key; - private final long since; - - private Seen(final String key, final long since) { - this.key = key; - this.since = since; - } - } -} diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServer.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServer.java deleted file mode 100644 index 92c7d62..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServer.java +++ /dev/null @@ -1,443 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.util.ArrayList; -import java.util.Collections; -import java.util.List; -import java.util.Map; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.function.Function; - -import org.eclipse.lsp4j.CompletionItem; -import org.eclipse.lsp4j.CompletionList; -import org.eclipse.lsp4j.CompletionParams; -import org.eclipse.lsp4j.DidChangeConfigurationParams; -import org.eclipse.lsp4j.DidChangeTextDocumentParams; -import org.eclipse.lsp4j.DidChangeWatchedFilesParams; -import org.eclipse.lsp4j.DidCloseTextDocumentParams; -import org.eclipse.lsp4j.DidOpenTextDocumentParams; -import org.eclipse.lsp4j.DidSaveTextDocumentParams; -import org.eclipse.lsp4j.Hover; -import org.eclipse.lsp4j.HoverParams; -import org.eclipse.lsp4j.InitializeParams; -import org.eclipse.lsp4j.InitializeResult; -import org.eclipse.lsp4j.InitializedParams; -import org.eclipse.lsp4j.MessageActionItem; -import org.eclipse.lsp4j.MessageParams; -import org.eclipse.lsp4j.PublishDiagnosticsParams; -import org.eclipse.lsp4j.ServerCapabilities; -import org.eclipse.lsp4j.ShowMessageRequestParams; -import org.eclipse.lsp4j.SignatureHelp; -import org.eclipse.lsp4j.SignatureHelpParams; -import org.eclipse.lsp4j.TextDocumentContentChangeEvent; -import org.eclipse.lsp4j.TextDocumentIdentifier; -import org.eclipse.lsp4j.TextDocumentItem; -import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; -import org.eclipse.lsp4j.jsonrpc.messages.Either; -import org.eclipse.lsp4j.services.LanguageClient; -import org.eclipse.lsp4j.services.LanguageClientAware; -import org.eclipse.lsp4j.services.LanguageServer; -import org.eclipse.lsp4j.services.TextDocumentService; -import org.eclipse.lsp4j.services.WorkspaceService; -import org.scijava.code.lsp.Environment; - -/** - * A language server for scripts whose text says where they run (which may - * change as they are edited): sends each document to a server of where its - * script runs now, as asked just before each request about it, and moves it - * to another as that changes. - *

    - * Where scripts run is asked off the callers' threads (e.g. an editor's), as - * that may look at files; one document at a time, in order, so that each - * server is told of a document's changes before it is asked about it. - *

    - * - * @author Gabriel Selzer - */ -class RoutingLanguageServer implements LanguageServer, TextDocumentService, - WorkspaceService, LanguageClientAware -{ - - private static final ExecutorService WHERE = Executors - .newSingleThreadExecutor(r -> { - final Thread t = new Thread(r, "appose-python-environments"); - t.setDaemon(true); - return t; - }); - - private final Function launch; - private final Function environment; - - /** The servers, by where the scripts they analyze run. */ - private final Map servers = - new ConcurrentHashMap<>(); - - /** The open documents, by URI. */ - private final Map docs = new ConcurrentHashMap<>(); - - private volatile LanguageClient client; - - /** - * @param launch Gets a new server, for scripts running in an environment. - * @param environment Where the script of a document runs now. - */ - RoutingLanguageServer(final Function launch, - final Function environment) - { - this.launch = launch; - this.environment = environment; - } - - // -- LanguageClientAware -- - - @Override - public void connect(final LanguageClient c) { - client = c; - } - - // -- LanguageServer -- - - @Override - public CompletableFuture initialize( - final InitializeParams params) - { - // NB: Each server is initialized as it is launched. - return CompletableFuture.completedFuture(new InitializeResult( - new ServerCapabilities())); - } - - @Override - public void initialized(final InitializedParams params) {} - - @Override - public CompletableFuture shutdown() { - final List> all = new ArrayList<>(); - for (final LanguageServer s : servers.values()) all.add(s.shutdown()); - servers.clear(); - docs.clear(); - return CompletableFuture.allOf(all.toArray(new CompletableFuture[0])) - .thenApply(x -> null); - } - - @Override - public void exit() { - for (final LanguageServer s : servers.values()) s.exit(); - } - - @Override - public TextDocumentService getTextDocumentService() { - return this; - } - - @Override - public WorkspaceService getWorkspaceService() { - return this; - } - - // -- TextDocumentService -- - - @Override - public void didOpen(final DidOpenTextDocumentParams params) { - final TextDocumentItem item = params.getTextDocument(); - final Doc doc = new Doc(item); - docs.put(item.getUri(), doc); - WHERE.execute(() -> route(doc)); - } - - @Override - public void didChange(final DidChangeTextDocumentParams params) { - final Doc doc = docs.get(params.getTextDocument().getUri()); - if (doc == null || params.getContentChanges().isEmpty()) return; - doc.change(params.getContentChanges().get(params.getContentChanges() - .size() - 1).getText(), params.getTextDocument().getVersion()); - WHERE.execute(() -> route(doc)); - } - - @Override - public void didClose(final DidCloseTextDocumentParams params) { - final Doc doc = docs.remove(params.getTextDocument().getUri()); - if (doc != null) WHERE.execute(() -> leave(doc)); - } - - @Override - public void didSave(final DidSaveTextDocumentParams params) { - // NB: Changes come with didChange. - } - - @Override - public CompletableFuture, CompletionList>> - completion(final CompletionParams params) - { - return request(params.getTextDocument().getUri(), (s, documents) -> documents - .completion(params).thenApply(answer -> mark(answer, s))); - } - - @Override - public CompletableFuture resolveCompletionItem( - final CompletionItem item) - { - // NB: Resolved by the server that completed it. - if (!(item.getData() instanceof Origin)) { - return CompletableFuture.completedFuture(item); - } - final Origin origin = (Origin) item.getData(); - item.setData(origin.data); - return origin.server.getTextDocumentService().resolveCompletionItem(item) - .handle((resolved, error) -> { - final CompletionItem out = resolved != null ? resolved : item; - out.setData(origin); - return out; - }); - } - - @Override - public CompletableFuture signatureHelp( - final SignatureHelpParams params) - { - return request(params.getTextDocument().getUri(), (s, - documents) -> documents.signatureHelp(params)); - } - - @Override - public CompletableFuture hover(final HoverParams params) { - return request(params.getTextDocument().getUri(), (s, - documents) -> documents.hover(params)); - } - - // -- WorkspaceService -- - - @Override - public void didChangeConfiguration(final DidChangeConfigurationParams p) {} - - @Override - public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) {} - - // -- Helper methods -- - - /** - * Asks the server of where a document's script runs now (once it is up to - * date there). NB: Cancelling the answer cancels the request. - */ - private CompletableFuture request(final String uri, - final Request request) - { - final Doc doc = docs.get(uri); - if (doc == null) return CompletableFuture.completedFuture(null); - final CompletableFuture answer = new CompletableFuture<>(); - CompletableFuture.supplyAsync(() -> route(doc), WHERE).thenAccept(s -> { - if (answer.isDone()) return; - final CompletableFuture raw = request.ask(s, s - .getTextDocumentService()); - answer.whenComplete((r, error) -> { - if (answer.isCancelled()) raw.cancel(true); - }); - raw.whenComplete((r, error) -> { - if (error != null) answer.completeExceptionally(error); - else answer.complete(r); - }); - }).exceptionally(error -> { - answer.completeExceptionally(error); - return null; - }); - return answer; - } - - /** - * Brings a document up to date on the server of where its script runs now: - * if that changed, closes it on the server of where it ran, and opens it on - * this one. Called on {@link #WHERE} only. - * - * @return The server. - */ - private LanguageServer route(final Doc doc) { - final TextDocumentItem item = doc.item(); - final LanguageServer target = server(environment.apply(item)); - if (doc.server != target) { - leave(doc); - target.getTextDocumentService().didOpen(new DidOpenTextDocumentParams( - item)); - doc.server = target; - } - else if (doc.sent != item.getVersion()) { - target.getTextDocumentService().didChange(new DidChangeTextDocumentParams( - new VersionedTextDocumentIdentifier(item.getUri(), item.getVersion()), - Collections.singletonList(new TextDocumentContentChangeEvent(item - .getText())))); - } - doc.sent = item.getVersion(); - return target; - } - - /** - * Closes a document on its server; and shuts that down, if it has no other - * documents. Called on {@link #WHERE} only. - */ - private void leave(final Doc doc) { - final LanguageServer s = doc.server; - if (s == null) return; - doc.server = null; - s.getTextDocumentService().didClose(new DidCloseTextDocumentParams( - new TextDocumentIdentifier(doc.uri))); - for (final Doc other : docs.values()) { - if (other.server == s) return; - } - servers.values().remove(s); - s.shutdown(); - } - - /** The server of an environment, launching it if needed. */ - private LanguageServer server(final Environment env) { - return servers.computeIfAbsent(env, e -> { - final LanguageServer s = launch.apply(e); - if (s instanceof LanguageClientAware) { - ((LanguageClientAware) s).connect(new Forwarding(s)); - } - s.initialize(new InitializeParams()).thenRun(() -> s.initialized( - new InitializedParams())); - return s; - }); - } - - /** Marks each completion with the server that made it, for resolving. */ - private static Either, CompletionList> mark( - final Either, CompletionList> answer, - final LanguageServer s) - { - if (answer == null) return null; - final List items = answer.isRight() ? answer.getRight() - .getItems() : answer.getLeft(); - if (items != null) { - for (final CompletionItem item : items) { - item.setData(new Origin(s, item.getData())); - } - } - return answer; - } - - /** Asks a server something. */ - private interface Request { - - CompletableFuture ask(LanguageServer server, - TextDocumentService documents); - } - - /** An open document. */ - private static final class Doc { - - private final String uri; - private final String languageId; - private String text; - private int version; - /** The server it is open on (set on {@link #WHERE} only). */ - private volatile LanguageServer server; - /** The version its server last got (on {@link #WHERE} only). */ - private int sent = -1; - - private Doc(final TextDocumentItem item) { - uri = item.getUri(); - languageId = item.getLanguageId(); - text = item.getText(); - version = item.getVersion(); - } - - private synchronized void change(final String newText, - final int newVersion) - { - text = newText; - version = newVersion; - } - - private synchronized TextDocumentItem item() { - return new TextDocumentItem(uri, languageId, version, text); - } - } - - /** The server a completion was made by, and the data it came with. */ - private static final class Origin { - - private final LanguageServer server; - private final Object data; - - private Origin(final LanguageServer server, final Object data) { - this.server = server; - this.data = data; - } - } - - /** - * Passes on what a server tells the client: its problems with a document - * only while the document is open on it. - */ - private final class Forwarding implements LanguageClient { - - private final LanguageServer server; - - private Forwarding(final LanguageServer server) { - this.server = server; - } - - @Override - public void publishDiagnostics(final PublishDiagnosticsParams p) { - final LanguageClient c = client; - final Doc doc = docs.get(p.getUri()); - if (c != null && doc != null && doc.server == server) { - c.publishDiagnostics(p); - } - } - - @Override - public void telemetryEvent(final Object object) {} - - @Override - public void showMessage(final MessageParams m) { - final LanguageClient c = client; - if (c != null) c.showMessage(m); - } - - @Override - public CompletableFuture showMessageRequest( - final ShowMessageRequestParams r) - { - final LanguageClient c = client; - return c != null ? c.showMessageRequest(r) : CompletableFuture - .completedFuture(null); - } - - @Override - public void logMessage(final MessageParams m) { - final LanguageClient c = client; - if (c != null) c.logMessage(m); - } - } -} diff --git a/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/overlay.py b/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/overlay.py deleted file mode 100644 index 133f37e..0000000 --- a/src/main/resources/org/scijava/plugins/scripting/appose/python/completion/overlay.py +++ /dev/null @@ -1,127 +0,0 @@ -### -# #%L -# Python scripting language plugin backed by Appose. -# %% -# Copyright (C) 2026 SciJava developers. -# %% -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# -# 1. Redistributions of source code must retain the above copyright notice, -# this list of conditions and the following disclaimer. -# 2. Redistributions in binary form must reproduce the above copyright notice, -# this list of conditions and the following disclaimer in the documentation -# and/or other materials provided with the distribution. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" -# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE -# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR -# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF -# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS -# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN -# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) -# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE -# POSSIBILITY OF SUCH DAMAGE. -# #L% -### -# Appose task script that installs, for code completion only, the packages a -# script declares but its environment lacks (e.g. ones just added to its -# dependencies), so that Jedi knows them before the environment is rebuilt. -# -# Runs in the completion tooling environment, which includes uv. Installs -# into a separate directory (an "overlay", which Jedi adds to its search -# path), never into the script's environment itself; and only the missing -# packages themselves (--no-deps): their dependencies usually are installed. -# -# Task inputs: -# -# python -- the Python executable of the script's environment, or -# None if it has not been built (then all dependencies are -# missing, installed for the tooling environment's Python) -# dependencies -- the script's dependencies (PEP 508 requirement strings) -# overlay -- the directory to install into -# -# Task outputs: -# -# path -- the overlay directory, or None if nothing is missing -# installed -- the requirements installed into it - -import json as _sj_json -import os as _sj_os -import re as _sj_re -import shutil as _sj_shutil -import subprocess as _sj_subprocess -import sys as _sj_sys - -_SJ_INSTALLED = ( - "import importlib.metadata as m, json, re\n" - "print(json.dumps(sorted({re.sub(r'[-_.]+', '-', (d.metadata['Name'] or '')" - ".lower()) for d in m.distributions()})))" -) - - -def _sj_name(requirement): - """The normalized distribution name of a PEP 508 requirement string.""" - m = _sj_re.match(r"\s*([A-Za-z0-9][A-Za-z0-9._-]*)", requirement) - return _sj_re.sub(r"[-_.]+", "-", m.group(1).lower()) if m else None - - -def _sj_overlay(): - target = python or _sj_sys.executable - installed = set() - if python: - result = _sj_subprocess.run( - [python, "-c", _SJ_INSTALLED], - capture_output=True, - text=True, - timeout=120, - ) - if result.returncode == 0: - installed = set(_sj_json.loads(result.stdout)) - missing = [ - r for r in dependencies if _sj_name(r) and _sj_name(r) not in installed - ] - - task.outputs["installed"] = missing - if not missing: - _sj_shutil.rmtree(overlay, ignore_errors=True) - task.outputs["path"] = None - return - - marker = _sj_os.path.join(overlay, ".scijava-overlay.json") - wanted = {"python": target, "requirements": sorted(missing)} - try: - with open(marker, encoding="utf-8") as f: - if _sj_json.load(f) == wanted: - task.outputs["path"] = overlay # Already up to date. - return - except (OSError, ValueError): - pass - - from uv import find_uv_bin - - # Install next to the overlay, then swap it in, so that Jedi never sees a - # half-installed overlay. - staging = overlay + ".new" - _sj_shutil.rmtree(staging, ignore_errors=True) - result = _sj_subprocess.run( - [find_uv_bin(), "pip", "install", "--target", staging, "--no-deps", - "--python", target, *missing], - capture_output=True, - text=True, - timeout=600, - ) - if result.returncode != 0: - _sj_shutil.rmtree(staging, ignore_errors=True) - raise RuntimeError("uv pip install failed:\n" + result.stderr) - with open(_sj_os.path.join(staging, ".scijava-overlay.json"), "w", - encoding="utf-8") as f: - _sj_json.dump(wanted, f) - _sj_shutil.rmtree(overlay, ignore_errors=True) - _sj_os.replace(staging, overlay) - task.outputs["path"] = overlay - - -_sj_overlay() diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java index 306ec51..84fb1cf 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java @@ -64,8 +64,9 @@ /** * End-to-end tests of the language servers of Python scripts run via Appose - * (see {@link ApposePythonServerPlugin}), with a real jedi-language-server, through - * the {@link LanguageServerService}, as editors use it. + * (see {@link ApposePythonDialect}, and scijava-code-lsp-jedi), with a real + * jedi-language-server, through the {@link LanguageServerService}, as editors + * use it. *

    * The first completion builds jedi-language-server's environment (see * code-lsp-jedi), a new dependency the tooling environment that installs diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java similarity index 78% rename from src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java rename to src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java index 9401a85..7d17c25 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonServersTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java @@ -34,20 +34,18 @@ import static org.junit.Assert.assertTrue; import java.io.File; -import java.util.ArrayList; import java.util.Collections; import java.util.List; import net.imglib2.img.Img; -import org.eclipse.lsp4j.TextDocumentItem; import org.junit.After; import org.junit.Before; import org.junit.Test; import org.scijava.Context; import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.Environment; -import org.scijava.code.lsp.LanguageServerPlugin; +import org.scijava.code.lsp.ScriptDialect; import org.scijava.code.lsp.ScriptDocument; import org.scijava.plugin.PluginService; import org.scijava.plugins.scripting.appose.python.ScriptEnvironment; @@ -56,20 +54,19 @@ import org.scijava.script.ScriptService; /** - * Tests what {@link ApposePythonServerPlugin} tells jedi-language-server about + * Tests what {@link ApposePythonDialect} tells Python language servers about * Python scripts run via Appose (the script as plain Python, and where it * runs), without starting any server process. * * @author Gabriel Selzer */ -public class ApposePythonServersTest { +public class ApposePythonDialectTest { private static final ApposePythonDialect DIALECT = new ApposePythonDialect(); private Context context; private ScriptLanguage language; - private ApposePythonServerPlugin plugin; - private ApposePythonEnvironments environments; + private ApposePythonDialect dialect; @Before public void setUp() { @@ -77,13 +74,11 @@ public void setUp() { ResidentWorkerService.class, LanguageServerService.class); language = context.service(ScriptService.class).getLanguageByName( "appose-python"); - plugin = context.service(PluginService.class).createInstancesOfType( - LanguageServerPlugin.class).stream().filter( - p -> p instanceof ApposePythonServerPlugin).map( - p -> (ApposePythonServerPlugin) p) + dialect = context.service(PluginService.class).createInstancesOfType( + ScriptDialect.class).stream().filter( + p -> p instanceof ApposePythonDialect).map( + p -> (ApposePythonDialect) p) .findFirst().get(); - environments = ApposePythonEnvironments.of(context.service( - ResidentWorkerService.class)); } @After @@ -93,10 +88,11 @@ public void tearDown() { @Test public void testServersAreOffered() { + // Its scripts are plain Python, which scijava-code-lsp-jedi serves. + assertTrue(dialect.supports(language)); + assertEquals("python", dialect.languageId()); assertTrue(context.service(LanguageServerService.class).supports( language)); - // NB: Here, appose-python offers the Python server itself. - assertTrue(plugin.jediOffered()); } @Test @@ -113,46 +109,19 @@ public void testScriptAsPython() { @Test public void testEnvironmentWithoutOne() { // No script environment has been built: the server's own Python. - final Environment env = environments.environment(new TextDocumentItem( - "untitled:/a.py", "appose-python", 2, "import os\nos.pa")); + final Environment env = dialect.environment("import os\nos.pa", null); assertNull(env.interpreter()); assertTrue(env.paths().isEmpty()); } - @Test - public void testOverlay() throws Exception { - // A script declaring a dependency its environment lacks. - final List> installed = new ArrayList<>(); - environments.setOverlays(new DependencyOverlays((python, dependencies, - dir) -> { - installed.add(dependencies); - return "/overlays/blur"; - }, new File("overlays"), 0, System::currentTimeMillis)); - final String code = "# /// script\n# dependencies = [\"tifffile\"]\n" + - "# ///\nimport tifffile\ntifffile.im"; - final TextDocumentItem doc = new TextDocumentItem("untitled:/a.py", - "appose-python", 1, code); - // On opening the script, its dependencies are settled: the overlay is - // installed; once ready, the server searches it. - environments.environment(doc); - for (int i = 0; i < 100 && installed.isEmpty(); i++) Thread.sleep(20); - Thread.sleep(50); - assertEquals(Collections.singletonList(Collections.singletonList( - "tifffile")), installed); - assertEquals(Collections.singletonList("/overlays/blur"), environments - .environment(doc).paths()); - } - @Test public void testScriptWorkersUntouched() { // Finding where a script runs must not create (or replace) a worker for // it, even as its metadata is edited. final String script = "# /// script\n# dependencies = [\"appose\"]\n" + "# ///\nimport os\nos.pa"; - environments.environment(new TextDocumentItem("untitled:/a.py", - "appose-python", 2, script)); - environments.environment(new TextDocumentItem("untitled:/a.py", - "appose-python", 3, script.replace("appose", "appose, numpy"))); + dialect.environment(script, null); + dialect.environment(script.replace("appose", "appose, numpy"), null); assertTrue(context.service(ResidentWorkerService.class).workers() .isEmpty()); } @@ -176,8 +145,7 @@ public void testEnvironmentFromEarlierSession() throws Exception { java.nio.file.Files.createDirectories(python.getParent()); java.nio.file.Files.createFile(python); - final Environment env = environments.environment(new TextDocumentItem(script - .toURI().toString(), "appose-python", 2, header + "numpy.ze")); + final Environment env = dialect.environment(header + "numpy.ze", script); // The server analyzes against the environment's Python... assertEquals(python.toFile().getAbsolutePath(), env.interpreter()); // ...without anything built or started. @@ -262,10 +230,10 @@ public void testPythonTypes() { // -- Helper methods -- - /** The script as plain Python, as the plugin's servers see it. */ + /** The script as plain Python, as its servers see it. */ private ScriptDocument document(final String code) { return ScriptDocument.of(code, context.service(LanguageServerService.class) - .scriptParameters(code), plugin.dialect()); + .scriptParameters(code), dialect); } private static void delete(final java.nio.file.Path dir) throws Exception { diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java deleted file mode 100644 index 75e6d11..0000000 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonLanguageServerTest.java +++ /dev/null @@ -1,310 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertThrows; -import static org.junit.Assert.assertTrue; - -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.List; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.CopyOnWriteArrayList; -import java.util.concurrent.TimeUnit; -import java.util.stream.Collectors; - -import org.eclipse.lsp4j.CompletionItem; -import org.eclipse.lsp4j.CompletionItemKind; -import org.eclipse.lsp4j.CompletionList; -import org.eclipse.lsp4j.CompletionParams; -import org.eclipse.lsp4j.DidChangeConfigurationParams; -import org.eclipse.lsp4j.DidChangeTextDocumentParams; -import org.eclipse.lsp4j.DidChangeWatchedFilesParams; -import org.eclipse.lsp4j.DidCloseTextDocumentParams; -import org.eclipse.lsp4j.DidOpenTextDocumentParams; -import org.eclipse.lsp4j.DidSaveTextDocumentParams; -import org.eclipse.lsp4j.InitializeParams; -import org.eclipse.lsp4j.InitializeResult; -import org.eclipse.lsp4j.Position; -import org.eclipse.lsp4j.ServerCapabilities; -import org.eclipse.lsp4j.TextDocumentContentChangeEvent; -import org.eclipse.lsp4j.TextDocumentIdentifier; -import org.eclipse.lsp4j.TextDocumentItem; -import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; -import org.eclipse.lsp4j.jsonrpc.messages.Either; -import org.eclipse.lsp4j.services.LanguageServer; -import org.eclipse.lsp4j.services.TextDocumentService; -import org.eclipse.lsp4j.services.WorkspaceService; -import org.junit.Test; -import org.scijava.code.lsp.jvm.Callable; -import org.scijava.code.lsp.Positions; - -/** - * Tests {@link ApposePythonLanguageServer}, around a fake Python server. - * - * @author Gabriel Selzer - */ -public class ApposePythonLanguageServerTest { - - private static final String URI = "untitled:/a.py"; - - /** Knows json's dumps (a function), dump, decoder (a module), __doc__. */ - private static final class Jedi implements LanguageServer, - TextDocumentService, WorkspaceService - { - - final List asked = new CopyOnWriteArrayList<>(); - final List resolved = new CopyOnWriteArrayList<>(); - CompletableFuture hold; - final List> held = new CopyOnWriteArrayList<>(); - /** Whether it knows nothing yet (e.g. while json is installed). */ - volatile boolean ignorant; - - @Override - public CompletableFuture initialize( - final InitializeParams params) - { - return CompletableFuture.completedFuture(new InitializeResult( - new ServerCapabilities())); - } - - @Override - public CompletableFuture shutdown() { - return CompletableFuture.completedFuture(null); - } - - @Override - public void exit() {} - - @Override - public TextDocumentService getTextDocumentService() { - return this; - } - - @Override - public WorkspaceService getWorkspaceService() { - return this; - } - - @Override - public void didOpen(final DidOpenTextDocumentParams p) {} - - @Override - public void didChange(final DidChangeTextDocumentParams p) {} - - @Override - public void didClose(final DidCloseTextDocumentParams p) {} - - @Override - public void didSave(final DidSaveTextDocumentParams p) {} - - @Override - public void didChangeConfiguration(final DidChangeConfigurationParams p) {} - - @Override - public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) {} - - @Override - public CompletableFuture, CompletionList>> - completion(final CompletionParams p) - { - asked.add(p.getPosition()); - final List items = new ArrayList<>(); - int i = 0; - for (final String name : ignorant ? new String[0] : new String[] { - "dumps", "dump", "decoder", "__doc__" }) - { - final CompletionItem item = new CompletionItem(name); - item.setKind(name.equals("decoder") ? CompletionItemKind.Module - : CompletionItemKind.Function); - item.setInsertText(name); - item.setSortText("z" + i++); - items.add(item); - } - final Either, CompletionList> answer = Either - .forRight(new CompletionList(items)); - if (hold == null) return CompletableFuture.completedFuture(answer); - final CompletableFuture, CompletionList>> f = - new CompletableFuture<>(); - held.add(f); - hold.thenRun(() -> f.complete(answer)); - return f; - } - - @Override - public CompletableFuture resolveCompletionItem( - final CompletionItem item) - { - resolved.add(item.getLabel()); - item.setDetail(item.getLabel().equals("decoder") ? "module decoder" - : "def " + item.getLabel() + "(obj, *, indent: int=None) -> str"); - return CompletableFuture.completedFuture(item); - } - } - - @Test - public void testAskOnceAndFilter() throws Exception { - final Jedi jedi = new Jedi(); - final ApposePythonLanguageServer server = new ApposePythonLanguageServer(jedi, 30); - open(server, "import json\njson.du"); - List items = complete(server, "import json\njson.du"); - // Asked about the word's first character. - assertEquals(new Position(1, 6), jedi.asked.get(0)); - // The word's completions, without dunders. - assertEquals(Arrays.asList("dumps", "dump"), labels(items)); - // Once a word is begun: callables, from the resolved signatures. - final CompletionItem dumps = items.get(0); - assertEquals("(obj, int indent)", dumps.getLabelDetails().getDetail()); - assertEquals("str", dumps.getLabelDetails().getDescription()); - assertEquals("dumps(${1:obj}, ${2:indent})", dumps.getInsertText()); - - // Typing on: the same answer, filtered; not asked again. - change(server, "import json\njson.dum", 2); - items = complete(server, "import json\njson.dum"); - assertEquals(Arrays.asList("dumps", "dump"), labels(items)); - // (The word typed in full is not offered again.) - change(server, "import json\njson.dumps", 3); - assertTrue(complete(server, "import json\njson.dumps").isEmpty()); - assertEquals(1, jedi.asked.size()); - - // Nothing typed: names only (nothing resolved), dunders left out. - change(server, "import json\njson.", 4); - final int resolved = jedi.resolved.size(); - items = complete(server, "import json\njson."); - assertEquals(Arrays.asList("dumps", "dump", "decoder"), labels(items)); - assertNull(items.get(0).getLabelDetails()); - assertEquals(resolved, jedi.resolved.size()); - } - - @Test - public void testEmptyAnswersAskedAgain() throws Exception { - final Jedi jedi = new Jedi(); - jedi.ignorant = true; - final ApposePythonLanguageServer server = new ApposePythonLanguageServer(jedi, 0); - open(server, "import json\njson.du"); - assertTrue(complete(server, "import json\njson.du").isEmpty()); - // It knows more now: asked again, rather than the empty answer reused. - jedi.ignorant = false; - assertEquals(Arrays.asList("dumps", "dump"), labels(complete(server, - "import json\njson.du"))); - assertEquals(2, jedi.asked.size()); - } - - @Test - public void testImportsAreNames() throws Exception { - final Jedi jedi = new Jedi(); - final ApposePythonLanguageServer server = new ApposePythonLanguageServer(jedi, 30); - open(server, "from json import du"); - final List items = complete(server, "from json import du"); - assertEquals(Arrays.asList("dumps", "dump"), labels(items)); - assertNull(items.get(0).getLabelDetails()); - } - - @Test - public void testSuperseded() throws Exception { - final Jedi jedi = new Jedi(); - jedi.hold = new CompletableFuture<>(); - final ApposePythonLanguageServer server = new ApposePythonLanguageServer(jedi, 0); - open(server, "x.a"); - final CompletableFuture, CompletionList>> slow = - server.completion(params("x.a")); - change(server, "x.b", 2); - final CompletableFuture, CompletionList>> next = - server.completion(params("x.b")); - // The first question is cancelled, also on the server. - assertThrows(Exception.class, () -> slow.get(10, TimeUnit.SECONDS)); - assertTrue(jedi.held.get(0).isCancelled()); - jedi.hold.complete(null); - assertTrue(next.get(10, TimeUnit.SECONDS).getRight().getItems().isEmpty()); - } - - // -- Helper methods -- - - private static CompletionParams params(final String text) { - return new CompletionParams(new TextDocumentIdentifier(URI), Positions - .position(text, text.length())); - } - - private static List complete( - final ApposePythonLanguageServer server, final String text) throws Exception - { - return server.completion(params(text)).get(10, TimeUnit.SECONDS) - .getRight().getItems(); - } - - private static void open(final ApposePythonLanguageServer server, - final String text) - { - server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(URI, - "python", 1, text))); - } - - private static void change(final ApposePythonLanguageServer server, - final String text, final int version) - { - server.didChange(new DidChangeTextDocumentParams( - new VersionedTextDocumentIdentifier(URI, version), Collections - .singletonList(new TextDocumentContentChangeEvent(text)))); - } - - private static List labels(final List items) { - return items.stream().map(CompletionItem::getLabel).collect(Collectors - .toList()); - } - - @Test - public void testParse() { - Callable c = ApposePythonLanguageServer.parse("dumps", - "def dumps(obj, *, skipkeys: bool=False, cls=None, **kw) -> str"); - assertEquals("[obj, bool skipkeys, cls, kw]", String.valueOf(c - .params())); - assertEquals("str", c.returnType()); - - // Brackets and strings in annotations and defaults; positional-only. - c = ApposePythonLanguageServer.parse("zeros", "def zeros(shape: tuple[int, ...], dtype: str=\"f,8\"" + - ", /, order: Literal['C', 'F']='C') -> NDArray[float64]"); - assertEquals("[tuple[int, ...] shape, str dtype, " + - "Literal['C', 'F'] order]", String.valueOf(c.params())); - assertEquals("NDArray[float64]", c.returnType()); - - // Classes (their constructors), without parameters or return type. - c = ApposePythonLanguageServer.parse("Path", "class Path()"); - assertEquals(0, c.params().size()); - assertNull(c.returnType()); - c = ApposePythonLanguageServer.parse("f", "def f(*args)"); - assertEquals("args", c.params().get(0).name()); - - // Not a signature. - assertNull(ApposePythonLanguageServer.parse("x", "instance x")); - } -} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java deleted file mode 100644 index 3cc49fd..0000000 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java +++ /dev/null @@ -1,109 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNull; - -import java.io.File; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.List; -import java.util.concurrent.atomic.AtomicLong; - -import org.junit.Test; - -/** - * Tests {@link DependencyOverlays}. - * - * @author Gabriel Selzer - */ -public class DependencyOverlaysTest { - - private final AtomicLong now = new AtomicLong(); - private final List> installs = new ArrayList<>(); - - private final DependencyOverlays overlays = new DependencyOverlays((python, - dependencies, dir) -> { - synchronized (installs) { - installs.add(dependencies); - } - return dir.getPath(); - }, new File("overlays"), 1500, now::get); - - @Test - public void testWaitsForDependenciesToSettle() throws Exception { - final List deps = Arrays.asList("numpy", "scikit-ima"); - // Still being typed: nothing is installed... - assertNull(overlays.overlay("blur", null, deps, false)); - now.addAndGet(500); - assertNull(overlays.overlay("blur", null, deps, false)); - final List typed = Arrays.asList("numpy", "scikit-image"); - assertNull(overlays.overlay("blur", null, typed, false)); - now.addAndGet(1000); - assertNull(overlays.overlay("blur", null, typed, false)); - Thread.sleep(100); - assertEquals(Collections.emptyList(), installs); - // ...until the dependencies have stayed the same for a while. - now.addAndGet(1000); - assertNull(overlays.overlay("blur", null, typed, false)); - assertEquals(new File("overlays", "blur").getPath(), awaitOverlay("blur", - null, typed)); - assertEquals(Collections.singletonList(typed), installs); - } - - @Test - public void testSettledAndReused() throws Exception { - final List deps = Collections.singletonList("tifffile"); - // Known not to be changing (e.g. just opened): installed at once. - assertNull(overlays.overlay("read", "python", deps, true)); - final String overlay = awaitOverlay("read", "python", deps); - assertEquals(new File("overlays", "read").getPath(), overlay); - // Asked again: the same overlay, not installed again. - assertEquals(overlay, overlays.overlay("read", "python", deps, false)); - assertEquals(1, installs.size()); - // Nothing declared: no overlay. - assertNull(overlays.overlay("read", "python", Collections.emptyList(), - true)); - } - - /** Waits for the overlay to be ready. */ - private String awaitOverlay(final String env, final String python, - final List deps) throws InterruptedException - { - for (int i = 0; i < 100; i++) { - final String overlay = overlays.overlay(env, python, deps, false); - if (overlay != null) return overlay; - Thread.sleep(20); - } - return null; - } -} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServerTest.java deleted file mode 100644 index 0c0e745..0000000 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/RoutingLanguageServerTest.java +++ /dev/null @@ -1,304 +0,0 @@ -/* - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; - -import java.util.Collections; -import java.util.List; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.CopyOnWriteArrayList; -import java.util.concurrent.TimeUnit; -import java.util.function.BooleanSupplier; - -import org.eclipse.lsp4j.CompletionItem; -import org.eclipse.lsp4j.CompletionList; -import org.eclipse.lsp4j.CompletionParams; -import org.eclipse.lsp4j.Diagnostic; -import org.eclipse.lsp4j.DidChangeConfigurationParams; -import org.eclipse.lsp4j.DidChangeTextDocumentParams; -import org.eclipse.lsp4j.DidChangeWatchedFilesParams; -import org.eclipse.lsp4j.DidCloseTextDocumentParams; -import org.eclipse.lsp4j.DidOpenTextDocumentParams; -import org.eclipse.lsp4j.DidSaveTextDocumentParams; -import org.eclipse.lsp4j.InitializeParams; -import org.eclipse.lsp4j.InitializeResult; -import org.eclipse.lsp4j.MessageActionItem; -import org.eclipse.lsp4j.MessageParams; -import org.eclipse.lsp4j.Position; -import org.eclipse.lsp4j.PublishDiagnosticsParams; -import org.eclipse.lsp4j.Range; -import org.eclipse.lsp4j.ServerCapabilities; -import org.eclipse.lsp4j.ShowMessageRequestParams; -import org.eclipse.lsp4j.TextDocumentContentChangeEvent; -import org.eclipse.lsp4j.TextDocumentIdentifier; -import org.eclipse.lsp4j.TextDocumentItem; -import org.eclipse.lsp4j.VersionedTextDocumentIdentifier; -import org.eclipse.lsp4j.jsonrpc.messages.Either; -import org.eclipse.lsp4j.services.LanguageClient; -import org.eclipse.lsp4j.services.LanguageClientAware; -import org.eclipse.lsp4j.services.LanguageServer; -import org.eclipse.lsp4j.services.TextDocumentService; -import org.eclipse.lsp4j.services.WorkspaceService; -import org.junit.Test; -import org.scijava.code.lsp.Environment; - -/** - * Tests {@link RoutingLanguageServer}, around fake servers. - * - * @author Gabriel Selzer - */ -public class RoutingLanguageServerTest { - - private static final String URI = "untitled:/m.py"; - - private static final Environment A = new Environment("/a/python", null, - null); - - private static final Environment B = new Environment("/b/python", null, - null); - - /** The servers launched, in order. */ - private final List launched = new CopyOnWriteArrayList<>(); - - /** The problems the client heard of. */ - private final List heard = - new CopyOnWriteArrayList<>(); - - @Test - public void testMoves() throws Exception { - // Each document is analyzed by the server of where its script runs, as - // its text says here. - final RoutingLanguageServer server = new RoutingLanguageServer(env -> { - final Fake f = new Fake(env); - launched.add(f); - return f; - }, doc -> doc.getText().startsWith("#B") ? B : A); - server.connect(client()); - server.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(URI, - "python", 1, "x = 1"))); - complete(server); - assertEquals(1, launched.size()); - final Fake a = launched.get(0); - assertEquals(A, a.environment); - assertEquals(Collections.singletonList("open x = 1"), a.events.subList(0, - 1)); - - // Changed, but still running there: the server is told of the change. - change(server, 2, "x = 2"); - complete(server); - assertEquals(1, launched.size()); - assertTrue(a.events.contains("change x = 2")); - - // It runs elsewhere now: the next request goes there, and the server of - // where it ran lets it go (and, with nothing left to analyze, stops). - change(server, 3, "#B\nx = 1"); - final List items = complete(server); - assertEquals(2, launched.size()); - final Fake b = launched.get(1); - assertEquals(B, b.environment); - assertTrue(b.events.contains("open #B\nx = 1")); - assertTrue(a.events.contains("close")); - assertTrue(a.events.contains("shutdown")); - - // Completions are resolved by the server that made them. - server.resolveCompletionItem(items.get(0)).get(10, TimeUnit.SECONDS); - assertTrue(b.events.contains("resolve dumps")); - assertFalse(a.events.contains("resolve dumps")); - - // Problems: heard only from the server the document is open on. - a.publish(); - b.publish(); - waitFor(() -> !heard.isEmpty()); - assertEquals(1, heard.size()); - assertEquals("/b/python", heard.get(0).getDiagnostics().get(0) - .getMessage().getLeft()); - - // Closed: let go, and the server stops. - server.didClose(new DidCloseTextDocumentParams(new TextDocumentIdentifier( - URI))); - waitFor(() -> b.events.contains("shutdown")); - assertTrue(b.events.contains("close")); - } - - // -- Helper methods -- - - private static List complete(final RoutingLanguageServer s) - throws Exception - { - return s.completion(new CompletionParams(new TextDocumentIdentifier(URI), - new Position(0, 0))).get(10, TimeUnit.SECONDS).getRight().getItems(); - } - - private static void change(final RoutingLanguageServer s, final int version, - final String text) - { - s.didChange(new DidChangeTextDocumentParams( - new VersionedTextDocumentIdentifier(URI, version), Collections - .singletonList(new TextDocumentContentChangeEvent(text)))); - } - - private static void waitFor(final BooleanSupplier condition) - throws InterruptedException - { - final long deadline = System.currentTimeMillis() + 10_000; - while (!condition.getAsBoolean()) { - if (System.currentTimeMillis() > deadline) { - throw new AssertionError("Timed out"); - } - Thread.sleep(10); - } - } - - private LanguageClient client() { - return new LanguageClient() { - - @Override - public void publishDiagnostics(final PublishDiagnosticsParams p) { - heard.add(p); - } - - @Override - public void telemetryEvent(final Object object) {} - - @Override - public void showMessage(final MessageParams m) {} - - @Override - public CompletableFuture showMessageRequest( - final ShowMessageRequestParams r) - { - return CompletableFuture.completedFuture(null); - } - - @Override - public void logMessage(final MessageParams m) {} - }; - } - - /** A server of an environment, recording what it is told. */ - private static final class Fake implements LanguageServer, - TextDocumentService, WorkspaceService, LanguageClientAware - { - - private final Environment environment; - private final List events = new CopyOnWriteArrayList<>(); - private LanguageClient client; - - private Fake(final Environment environment) { - this.environment = environment; - } - - /** Reports a problem, naming where it runs. */ - private void publish() { - client.publishDiagnostics(new PublishDiagnosticsParams(URI, Collections - .singletonList(new Diagnostic(new Range(new Position(0, 0), - new Position(0, 1)), environment.interpreter())))); - } - - @Override - public void connect(final LanguageClient c) { - client = c; - } - - @Override - public CompletableFuture initialize( - final InitializeParams params) - { - return CompletableFuture.completedFuture(new InitializeResult( - new ServerCapabilities())); - } - - @Override - public CompletableFuture shutdown() { - events.add("shutdown"); - return CompletableFuture.completedFuture(null); - } - - @Override - public void exit() {} - - @Override - public TextDocumentService getTextDocumentService() { - return this; - } - - @Override - public WorkspaceService getWorkspaceService() { - return this; - } - - @Override - public void didOpen(final DidOpenTextDocumentParams p) { - events.add("open " + p.getTextDocument().getText()); - } - - @Override - public void didChange(final DidChangeTextDocumentParams p) { - events.add("change " + p.getContentChanges().get(0).getText()); - } - - @Override - public void didClose(final DidCloseTextDocumentParams p) { - events.add("close"); - } - - @Override - public void didSave(final DidSaveTextDocumentParams p) {} - - @Override - public CompletableFuture, CompletionList>> - completion(final CompletionParams p) - { - events.add("complete"); - final CompletionItem item = new CompletionItem("dumps"); - item.setData("dumps"); - return CompletableFuture.completedFuture(Either.forRight( - new CompletionList(false, new CopyOnWriteArrayList<>(Collections - .singletonList(item))))); - } - - @Override - public CompletableFuture resolveCompletionItem( - final CompletionItem item) - { - events.add("resolve " + item.getData()); - return CompletableFuture.completedFuture(item); - } - - @Override - public void didChangeConfiguration(final DidChangeConfigurationParams p) {} - - @Override - public void didChangeWatchedFiles(final DidChangeWatchedFilesParams p) {} - } -} From 27e57f7afe05dc61f2099cf34b1e221fadcba336 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 16 Oct 2026 12:15:25 -0500 Subject: [PATCH 28/37] Test only Appose specifics: plain Python's tests move to code-lsp-jedi The completion integration test keeps script environments, image parameters and problems after #@ lines. Co-Authored-By: Claude Opus 5.5 --- ...ApposePythonCompletionIntegrationTest.java | 117 ++---------------- 1 file changed, 10 insertions(+), 107 deletions(-) diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java index 84fb1cf..c45740b 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java @@ -29,7 +29,6 @@ package org.scijava.plugins.scripting.appose.python.completion; -import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -45,16 +44,12 @@ import java.util.concurrent.TimeUnit; import org.eclipse.lsp4j.CompletionItem; -import org.eclipse.lsp4j.CompletionList; import org.eclipse.lsp4j.Diagnostic; -import org.eclipse.lsp4j.Hover; -import org.eclipse.lsp4j.SignatureHelp; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import org.scijava.Context; import org.scijava.code.lsp.LanguageServerService; -import org.scijava.code.lsp.Positions; import org.scijava.code.lsp.ScriptSession; import org.scijava.module.ModuleService; import org.scijava.script.ScriptInfo; @@ -63,15 +58,15 @@ import org.scijava.script.ScriptService; /** - * End-to-end tests of the language servers of Python scripts run via Appose - * (see {@link ApposePythonDialect}, and scijava-code-lsp-jedi), with a real - * jedi-language-server, through the {@link LanguageServerService}, as editors - * use it. + * End-to-end tests of completing Python scripts run via Appose (see + * {@link ApposePythonDialect}), with a real jedi-language-server (from + * scijava-code-lsp-jedi, whose own tests test it on plain Python), through the + * {@link LanguageServerService}, as editors use it: what is particular to + * these scripts. *

    - * The first completion builds jedi-language-server's environment (see - * code-lsp-jedi), a new dependency the tooling environment that installs - * overlays, and a script run the small uv environment of - * {@code appose-test/requirements.txt}; all require network access. + * The first completion builds jedi-language-server's environment, and a + * script run the small uv environment of {@code appose-test/requirements.txt}; + * both require network access. *

    * * @author Gabriel Selzer @@ -105,22 +100,6 @@ public static void tearDown() { if (context != null) context.dispose(); } - @Test - public void testStandardLibrary() throws Exception { - // Before any script environment is built, the server knows the standard - // library, with signatures and documentation. - final List completions = completeUntil( - "import json\njson.du", null, "dumps"); - final CompletionItem dumps = completions.stream().filter(c -> c.getLabel() - .equals("dumps") && c.getLabelDetails() != null).findFirst().get(); - assertTrue(dumps.getLabelDetails().getDetail(), dumps.getLabelDetails() - .getDetail().startsWith("(Any obj, bool skipkeys")); - assertTrue(dumps.getDocumentation() != null || session(null).resolve( - dumps).get(1, TimeUnit.MINUTES) - .getDocumentation() != null); - report("import json\njson.l", null, "loads", "o"); - } - @Test public void testScriptEnvironment() throws Exception { // Run a script, building its environment (which includes numpy). @@ -139,46 +118,12 @@ public void testScriptEnvironment() throws Exception { // known, and so is the type of the image parameter. completeUntil(header + "np.ze", script.getPath(), "zeros"); completeUntil(header + "image.sha", script.getPath(), "shape"); - report(header + "np.random.r", script.getPath(), "randn", "a"); - } - - @Test - public void testSignatureHelp() throws Exception { - final String code = "import json\njson.dumps(obj, "; - final ScriptSession session = session(null); - final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; - SignatureHelp help = null; - while (help == null && System.currentTimeMillis() < deadline) { - help = session.signatureHelp(code, code.length()).get(1, - TimeUnit.MINUTES); - } - assertTrue(help.getSignatures().get(0).getLabel().contains("dumps(")); - assertFalse(help.getSignatures().get(0).getParameters().isEmpty()); - assertEquals(Integer.valueOf(1), help.getActiveParameter()); - } - - @Test - public void testHover() throws Exception { - final String code = "import json\njson.dumps(1)"; - final ScriptSession session = session(null); - final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; - Hover hover = null; - while (hover == null && System.currentTimeMillis() < deadline) { - hover = session.hover(code, code.indexOf("dumps") + 2).get(1, - TimeUnit.MINUTES); - } - assertTrue(String.valueOf(hover), hover.getContents().getRight().getValue() - .contains("JSON")); - // The word's range, in the script. - assertEquals(Positions.position(code, code.indexOf("dumps")), hover - .getRange().getStart()); - assertEquals(Positions.position(code, code.indexOf("dumps") + 5), hover - .getRange().getEnd()); } @Test public void testDiagnostics() throws Exception { - // A syntax error, on the script's third line. + // A syntax error, on the script's third line: after its parameter + // lines. final String code = "#@ double sigma\nx = sigma\ny = (1 +\n"; final long deadline = System.currentTimeMillis() + BUILD_TIMEOUT; List problems = Collections.emptyList(); @@ -200,21 +145,6 @@ public void testParameterBeforeEnvironment() throws Exception { "shape"); } - @Test - public void testNewDependency() throws Exception { - // A script declaring a dependency that is installed nowhere: its - // environment has not even been built. - final File script = new File(scriptDir, "overlaid.py"); - final String header = "#!appose-python\n# /// script\n" + - "# dependencies = [\"tifffile\"]\n# ///\nimport tifffile\n"; - Files.write(script.toPath(), (header + "x = 1\n").getBytes( - StandardCharsets.UTF_8)); - // Opening it prepares completion, including the dependency's overlay. - final String code = header + "tifffile.imr"; - sync(code, script.getPath()); - completeUntil(code, script.getPath(), "imread"); - } - // -- Helper methods -- /** The scripts' sessions, by path ("" for an untitled one). */ @@ -233,11 +163,6 @@ private static List problems(final String code) { } } - /** Tells the servers a script's text. */ - private static ScriptSession sync(final String code, final String path) { - return session(path).update(code); - } - /** Completes until the expected completion appears (environments build). */ private static List completeUntil(final String code, final String path, final String expected) throws Exception @@ -261,26 +186,4 @@ private static List complete(final String code, return session(path).completion(code, code.length()).get(1, TimeUnit.MINUTES).getItems(); } - - /** - * Prints how long completions take, once warm: for another keystroke of a - * word the server has answered (cached); and for a new word. - */ - private static void report(final String code, final String path, - final String expected, final String more) throws Exception - { - completeUntil(code, path, expected); - long t0 = System.nanoTime(); - final int n = complete(code + more, path).size(); - final long cached = System.nanoTime() - t0; - final String other = code.substring(0, code.lastIndexOf('.') + 1) + "s"; - final ScriptSession session = session(path); - t0 = System.nanoTime(); - final CompletionList all = session.completion(other, other.length()).get( - 1, TimeUnit.MINUTES); - final long arrived = System.nanoTime() - t0; - System.out.println("[completion timing] same word (cached): " + cached / - 1000000 + " ms, " + n + " completions; new word: " + all.getItems() - .size() + " completions after " + arrived / 1000000 + " ms"); - } } From e631dcf99cd9636aff3e64e22265cc93128aae41 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 16 Oct 2026 14:27:09 -0500 Subject: [PATCH 29/37] Follow scijava-code-lsp's rename of runtimeType to parameterType Parameter types are never null, so stop checking for null. Co-Authored-By: Claude Opus 5.5 --- .../python/completion/ApposePythonDialect.java | 3 +-- .../python/completion/ApposePythonDialectTest.java | 14 +++++++------- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index 4220fde..750f6d8 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -84,8 +84,7 @@ public Environment environment(final String text, final File file) { } @Override - public String runtimeType(final Class type) { - if (type == null) return null; + public String parameterType(final Class type) { if (RandomAccessibleInterval.class.isAssignableFrom(type) || NDArray.class .isAssignableFrom(type)) return "numpy.ndarray"; if (type == boolean.class || type == Boolean.class) return "bool"; diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java index 7d17c25..40edd0c 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java @@ -219,13 +219,13 @@ public void testPythonOnDisk() throws Exception { @Test public void testPythonTypes() { - assertEquals("numpy.ndarray", DIALECT.runtimeType(Img.class)); - assertEquals("float", DIALECT.runtimeType(double.class)); - assertEquals("int", DIALECT.runtimeType(Long.class)); - assertEquals("str", DIALECT.runtimeType(String.class)); - assertEquals("bool", DIALECT.runtimeType(boolean.class)); - assertEquals("list", DIALECT.runtimeType(int[].class)); - assertNull(DIALECT.runtimeType(Context.class)); + assertEquals("numpy.ndarray", DIALECT.parameterType(Img.class)); + assertEquals("float", DIALECT.parameterType(double.class)); + assertEquals("int", DIALECT.parameterType(Long.class)); + assertEquals("str", DIALECT.parameterType(String.class)); + assertEquals("bool", DIALECT.parameterType(boolean.class)); + assertEquals("list", DIALECT.parameterType(int[].class)); + assertNull(DIALECT.parameterType(Context.class)); } // -- Helper methods -- From 048fae7e5f62263268f46092c7c565d2e349899d Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 16 Oct 2026 17:26:04 -0500 Subject: [PATCH 30/37] Follow scijava-code-lsp's ScriptDialect.header The dialect writes its imports, its parameters' declarations and the task's in one header line, and names what it imports only for them, to hide from completion. Its tests now test each of its methods. Co-Authored-By: Claude Opus 5.5 --- .../completion/ApposePythonDialect.java | 65 ++++++++------- .../completion/ApposePythonDialectTest.java | 83 ++++++++++--------- 2 files changed, 81 insertions(+), 67 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index 750f6d8..f0d584b 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -34,7 +34,6 @@ import java.math.BigInteger; import java.util.ArrayList; import java.util.Collection; -import java.util.Collections; import java.util.List; import java.util.Map; @@ -58,6 +57,9 @@ @Plugin(type = ScriptDialect.class, name = "Appose-Python") public class ApposePythonDialect implements ScriptDialect { + /** The type of images, as Appose passes them. */ + private static final String NDARRAY = "numpy.ndarray"; + /** The type of the {@code task} variable scripts can use. */ private static final String TASK_TYPE = "appose.python_worker.Task"; @@ -84,9 +86,41 @@ public Environment environment(final String text, final File file) { } @Override - public String parameterType(final Class type) { + public Header header(final Map> parameters) { + final List code = new ArrayList<>(); + final List hidden = new ArrayList<>(); + // Each Appose-Python script has access to its task + code.add("import appose.python_worker"); + hidden.add("appose"); + // And numpy is imported for the server whenever an image parameter is + // provided (the script itself does not get it) + if (parameters.values().stream().map(ApposePythonDialect::parameterType) + .anyMatch(NDARRAY::equals)) + { + code.add("import numpy"); + hidden.add("numpy"); + } + // Declare the types of each parameter for type reasoning + parameters.forEach((name, type) -> { + final String t = parameterType(type); + // NB: Values Appose cannot pass are not declared. + if (t != null) code.add(name + ": " + t); + }); + // Declare the type of task for type reasoning + code.add("task: " + TASK_TYPE); + // Combine into a Header + return new Header(String.join("; ", code), hidden); + } + + /** + * @param type the Java type (of a SciJava Script parameter) declared in the + * script + * @return the Python type of the object given to the script by Appose, or + * null if Appose cannot pass it + */ + static String parameterType(final Class type) { if (RandomAccessibleInterval.class.isAssignableFrom(type) || NDArray.class - .isAssignableFrom(type)) return "numpy.ndarray"; + .isAssignableFrom(type)) return NDARRAY; if (type == boolean.class || type == Boolean.class) return "bool"; if (type == int.class || type == long.class || type == short.class || type == byte.class || type == Integer.class || type == Long.class || @@ -108,31 +142,6 @@ public String parameterType(final Class type) { return null; } - @Override - public String declare(final String name, final String type) { - return name + ": " + type; - } - - @Override - public List imports(final Collection types) { - final List imports = new ArrayList<>(); - imports.add("import appose.python_worker"); - if (types.contains("numpy.ndarray")) imports.add("import numpy"); - return imports; - } - - @Override - public String imported(final String statement) { - // NB: "import a.b" binds a. - return statement.startsWith("import ") ? statement.substring(7).split( - "[.\\s]")[0] : null; - } - - @Override - public List extraDeclarations() { - return Collections.singletonList(declare("task", TASK_TYPE)); - } - /** Gets where scripts run, creating it on first use. */ private synchronized ApposePythonEnvironments environments() { if (environments == null) { diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java index 40edd0c..8528e4b 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java @@ -34,8 +34,11 @@ import static org.junit.Assert.assertTrue; import java.io.File; +import java.util.Arrays; import java.util.Collections; +import java.util.LinkedHashMap; import java.util.List; +import java.util.Map; import net.imglib2.img.Img; @@ -43,10 +46,9 @@ import org.junit.Before; import org.junit.Test; import org.scijava.Context; -import org.scijava.code.lsp.LanguageServerService; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.ScriptDialect; -import org.scijava.code.lsp.ScriptDocument; +import org.scijava.code.lsp.ScriptDialect.Header; import org.scijava.plugin.PluginService; import org.scijava.plugins.scripting.appose.python.ScriptEnvironment; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; @@ -54,24 +56,19 @@ import org.scijava.script.ScriptService; /** - * Tests what {@link ApposePythonDialect} tells Python language servers about - * Python scripts run via Appose (the script as plain Python, and where it - * runs), without starting any server process. + * Tests {@link ApposePythonDialect}. * * @author Gabriel Selzer */ public class ApposePythonDialectTest { - private static final ApposePythonDialect DIALECT = new ApposePythonDialect(); - private Context context; private ScriptLanguage language; private ApposePythonDialect dialect; @Before public void setUp() { - context = new Context(ScriptService.class, LanguageServerService.class, - ResidentWorkerService.class, LanguageServerService.class); + context = new Context(ScriptService.class, ResidentWorkerService.class); language = context.service(ScriptService.class).getLanguageByName( "appose-python"); dialect = context.service(PluginService.class).createInstancesOfType( @@ -87,23 +84,48 @@ public void tearDown() { } @Test - public void testServersAreOffered() { - // Its scripts are plain Python, which scijava-code-lsp-jedi serves. + public void testScriptsLanguageAdherence() { + // Ensure the dialect supports the Appose Python script language assertTrue(dialect.supports(language)); - assertEquals("python", dialect.languageId()); - assertTrue(context.service(LanguageServerService.class).supports( - language)); } @Test - public void testScriptAsPython() { - final String code = "#@ net.imglib2.img.Img image\n#@ double sigma\n" + - "image.sha"; - // The script's parameters are typed in a preamble, so the servers know - // them. - assertTrue(document(code).text().startsWith( - "import appose.python_worker; import numpy; image: numpy.ndarray; " + - "sigma: float; task: appose.python_worker.Task\n")); + public void testParameterType() { + // Script parameters have the types Appose converts them to + assertEquals("numpy.ndarray", ApposePythonDialect.parameterType( + Img.class)); + assertEquals("float", ApposePythonDialect.parameterType(double.class)); + assertEquals("int", ApposePythonDialect.parameterType(Long.class)); + assertEquals("str", ApposePythonDialect.parameterType(String.class)); + assertEquals("bool", ApposePythonDialect.parameterType(boolean.class)); + assertEquals("list", ApposePythonDialect.parameterType(int[].class)); + assertEquals("dict", ApposePythonDialect.parameterType(Map.class)); + // Types Appose cannot convert are not declared + assertNull(ApposePythonDialect.parameterType(Context.class)); + } + + @Test + public void testHeader() { + // SciJava Script parameters should be declared in Python code with the + // types Appose converts them to, e.g. "image: numpy.ndarray"; and + // scripts can use Appose's task variable + final Map> parameters = new LinkedHashMap<>(); + parameters.put("image", Img.class); + parameters.put("sigma", double.class); + parameters.put("context", Context.class); + Header header = dialect.header(parameters); + assertEquals("import appose.python_worker; import numpy; " + + "image: numpy.ndarray; sigma: float; " + + "task: appose.python_worker.Task", header.code()); + // The imports' names are the header's alone + assertEquals(Arrays.asList("appose", "numpy"), Arrays.asList(header + .hiddenNames().toArray())); + // numpy is imported only when a parameter needs it + header = dialect.header(Collections.singletonMap("sigma", + double.class)); + assertEquals("import appose.python_worker; sigma: float; " + + "task: appose.python_worker.Task", header.code()); + assertEquals(Collections.singleton("appose"), header.hiddenNames()); } @Test @@ -217,25 +239,8 @@ public void testPythonOnDisk() throws Exception { } } - @Test - public void testPythonTypes() { - assertEquals("numpy.ndarray", DIALECT.parameterType(Img.class)); - assertEquals("float", DIALECT.parameterType(double.class)); - assertEquals("int", DIALECT.parameterType(Long.class)); - assertEquals("str", DIALECT.parameterType(String.class)); - assertEquals("bool", DIALECT.parameterType(boolean.class)); - assertEquals("list", DIALECT.parameterType(int[].class)); - assertNull(DIALECT.parameterType(Context.class)); - } - // -- Helper methods -- - /** The script as plain Python, as its servers see it. */ - private ScriptDocument document(final String code) { - return ScriptDocument.of(code, context.service(LanguageServerService.class) - .scriptParameters(code), dialect); - } - private static void delete(final java.nio.file.Path dir) throws Exception { try (java.util.stream.Stream paths = java.nio.file.Files.walk(dir)) From 9664a7c28accf5bc915c6d8337edeb87293e5549 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 16 Oct 2026 17:29:47 -0500 Subject: [PATCH 31/37] Need scijava-code-lsp-jedi only to test The dialect names its server only by language id: applications choose their servers, as scripting-groovy now has them do. Co-Authored-By: Claude Opus 5.5 --- pom.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pom.xml b/pom.xml index 06fa379..147d3ac 100644 --- a/pom.xml +++ b/pom.xml @@ -118,8 +118,8 @@ org.scijava scijava-code-lsp-jedi ${scijava-code-lsp-jedi.version} - - runtime + + test From f47ea10324ed504f03832bb006e6a70c92a1c3bb Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Mon, 19 Oct 2026 15:45:41 -0500 Subject: [PATCH 32/37] Follow scijava-code-lsp: an Optional file, and a non-null live state Co-Authored-By: Claude Opus 5.5 --- .../python/completion/ApposePythonDialect.java | 9 ++++++--- .../python/completion/ApposePythonEnvironments.java | 2 +- .../python/completion/ApposePythonDialectTest.java | 12 ++++++++---- 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index f0d584b..024ac34 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -36,6 +36,7 @@ import java.util.Collection; import java.util.List; import java.util.Map; +import java.util.Optional; import net.imglib2.RandomAccessibleInterval; @@ -79,10 +80,12 @@ public String languageId() { } @Override - public Environment environment(final String text, final File file) { + public Environment environment(final String text, + final Optional file) + { // NB: As its #@script line, or its inline metadata, says. - return environments().environment(text, file == null ? null : file - .getPath()); + return environments().environment(text, file.map(File::getPath).orElse( + null)); } @Override diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java index 3453f83..13b5629 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java @@ -89,7 +89,7 @@ org.scijava.code.lsp.Environment environment(final String text, { final String python = scriptPython(text, path); return python == null ? org.scijava.code.lsp.Environment.NONE - : new org.scijava.code.lsp.Environment(python, null, null); + : new org.scijava.code.lsp.Environment(python, null); } /** diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java index 8528e4b..034beba 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java @@ -39,6 +39,7 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.Optional; import net.imglib2.img.Img; @@ -131,7 +132,8 @@ public void testHeader() { @Test public void testEnvironmentWithoutOne() { // No script environment has been built: the server's own Python. - final Environment env = dialect.environment("import os\nos.pa", null); + final Environment env = dialect.environment("import os\nos.pa", Optional + .empty()); assertNull(env.interpreter()); assertTrue(env.paths().isEmpty()); } @@ -142,8 +144,9 @@ public void testScriptWorkersUntouched() { // it, even as its metadata is edited. final String script = "# /// script\n# dependencies = [\"appose\"]\n" + "# ///\nimport os\nos.pa"; - dialect.environment(script, null); - dialect.environment(script.replace("appose", "appose, numpy"), null); + dialect.environment(script, Optional.empty()); + dialect.environment(script.replace("appose", "appose, numpy"), Optional + .empty()); assertTrue(context.service(ResidentWorkerService.class).workers() .isEmpty()); } @@ -167,7 +170,8 @@ public void testEnvironmentFromEarlierSession() throws Exception { java.nio.file.Files.createDirectories(python.getParent()); java.nio.file.Files.createFile(python); - final Environment env = dialect.environment(header + "numpy.ze", script); + final Environment env = dialect.environment(header + "numpy.ze", + Optional.of(script)); // The server analyzes against the environment's Python... assertEquals(python.toFile().getAbsolutePath(), env.interpreter()); // ...without anything built or started. From b9d633effe3702cbe24b77cb2673884c0b563758 Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Tue, 20 Oct 2026 09:50:46 -0500 Subject: [PATCH 33/37] Analyze scripts with the packages they declare, as the dialect says The overlays of packages a script declares, but its environment lacks, move here from scijava-code-lsp-jedi, whose servers now analyze scripts in exactly the environment they are given: ApposePythonDialect includes them in the environment it reports. They are installed with Appose's uv. Also follows scijava-code-lsp: sessions are made with the service. Co-Authored-By: Claude Opus 5.5 --- .../completion/ApposePythonDialect.java | 34 +- .../python/completion/DependencyOverlays.java | 357 ++++++++++++++++++ ...ApposePythonCompletionIntegrationTest.java | 22 +- .../completion/DependencyOverlaysTest.java | 168 +++++++++ 4 files changed, 571 insertions(+), 10 deletions(-) create mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index 024ac34..36323fd 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -41,6 +41,7 @@ import net.imglib2.RandomAccessibleInterval; import org.apposed.appose.NDArray; +import org.apposed.appose.util.Environments; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.ScriptDialect; import org.scijava.plugin.Parameter; @@ -51,7 +52,8 @@ /** * Describes how to turn Appose python scripts into plain python code, and - * where they run (see {@link ApposePythonEnvironments}). + * where they run (see {@link ApposePythonEnvironments}), searching the + * packages they declare but that lacks, too (see {@link DependencyOverlays}). * * @author Gabriel Selzer */ @@ -69,6 +71,14 @@ public class ApposePythonDialect implements ScriptDialect { private ApposePythonEnvironments environments; + /** + * How long a script's dependencies must stay unchanged before packages + * missing from its environment are installed for completion, in ms. + */ + private static final long OVERLAY_SETTLE = 1500; + + private static DependencyOverlays overlays; + @Override public boolean supports(final ScriptLanguage language) { return language instanceof ApposePythonScriptLanguage; @@ -83,9 +93,12 @@ public String languageId() { public Environment environment(final String text, final Optional file) { - // NB: As its #@script line, or its inline metadata, says. - return environments().environment(text, file.map(File::getPath).orElse( - null)); + // NB: As its #@script line, or its inline metadata, says; plus the + // packages it declares but that environment lacks (installed for + // completion only, see DependencyOverlays). + final String path = file.map(File::getPath).orElse(null); + return overlays().environment(text, path, environments().environment( + text, path)); } @Override @@ -145,6 +158,19 @@ static String parameterType(final Class type) { return null; } + /** + * Gets the overlays of packages scripts declare, but their environments + * lack: one set, for all dialect instances (sessions). + */ + private static synchronized DependencyOverlays overlays() { + if (overlays == null) { + overlays = new DependencyOverlays(DependencyOverlays::install, new File( + Environments.apposeEnvsDir(), "appose-python-completion-overlays"), + OVERLAY_SETTLE, System::currentTimeMillis); + } + return overlays; + } + /** Gets where scripts run, creating it on first use. */ private synchronized ApposePythonEnvironments environments() { if (environments == null) { diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java new file mode 100644 index 0000000..af037a0 --- /dev/null +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java @@ -0,0 +1,357 @@ +/*- + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Comparator; +import java.util.LinkedHashMap; +import java.util.TreeSet; +import java.util.concurrent.TimeUnit; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.util.stream.Stream; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.function.LongSupplier; + +import com.fasterxml.jackson.databind.ObjectMapper; + +import org.apposed.appose.tool.Uv; +import org.scijava.code.lsp.Environment; +import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; + +/** + * Makes packages a script declares (in its PEP 723 {@code # /// script} + * block), but its environment lacks, known to code completion before the + * environment is (re)built: e.g. a dependency just added to the script. + *

    + * The missing packages are installed, in the background, into a separate + * directory per script (an "overlay"), which the environment a script is + * analyzed in then searches too (see {@link ApposePythonDialect#environment}). + * The environment itself is never touched, so this never disturbs whatever + * runs in it (e.g. an Appose worker). Installs start only once the + * dependencies have stopped changing for a moment, so that half-typed names + * do not cause installs; and only if the missing packages differ from the + * overlay's. + *

    + *

    + * Alternatives considered, which may be worth revisiting: + *

    + *
      + *
    • Rebuilding the script's environment as its dependencies + * change. Jedi would then see exactly what will run, but builds can take + * minutes and gigabytes, half-finished edits would trigger builds, and e.g. Appose + * rebuilds environments in place, which can interfere with (or, by changing + * its configuration, release) a resident worker using it.
      • + *
    • Installing nothing, but hinting that new dependencies are + * known only once the script has run. Cheap and honest, but the user must + * run the script first. It may still be worthwhile for dependencies an + * overlay cannot provide: conda packages (e.g. + * {@code [tool.pixi.dependencies]}), and environments declared in files via + * {@code #@script(env=...)} in Appose-Python scripts, which overlays do not + * handle.
      • + *
    + * + * @author Gabriel Selzer + */ +final class DependencyOverlays { + + /** Installs the missing packages of a script's environment. */ + @FunctionalInterface + interface Installer { + + /** + * Installs the packages among {@code dependencies} that the environment + * with the given Python lacks into the given overlay directory. + * + * @param python The environment's Python executable, or null if the + * environment has not been built (all packages are missing). + * @param dependencies The script's dependencies, as PEP 508 strings. + * @param overlay The directory to install into. + * @return The overlay directory, or null if nothing is missing. + */ + String install(String python, List dependencies, File overlay); + } + + private final Installer installer; + private final File root; + private final long settleMillis; + private final LongSupplier clock; + private final ExecutorService executor = Executors.newSingleThreadExecutor( + r -> { + final Thread t = new Thread(r, "appose-python-overlays"); + t.setDaemon(true); + return t; + }); + + /** Installs, by environment name, Python and dependencies. */ + private final Map> installs = + new HashMap<>(); + + /** The dependencies last seen for each environment, and since when. */ + private final Map seen = new HashMap<>(); + + /** The scripts asked about so far, by overlay name. */ + private final Set known = new HashSet<>(); + + /** + * @param installer Does the installing, on a background thread. + * @param root Directory holding the overlays (one per script). + * @param settleMillis How long dependencies must stay unchanged before + * their missing packages are installed. + * @param clock The current time, in milliseconds. + */ + DependencyOverlays(final Installer installer, final File root, + final long settleMillis, final LongSupplier clock) + { + this.installer = installer; + this.root = root; + this.settleMillis = settleMillis; + this.clock = clock; + } + + /** + * Gets the environment to analyze a script in: the given one, searching + * the overlay of the packages the script declares but the environment lacks + * too, once ready (see {@link #overlay}). Never blocks. + * + * @param text The script. + * @param path The script's file path, or null if it has none. + * @param env Where the script runs. + */ + Environment environment(final String text, final String path, + final Environment env) + { + final String name = name(path); + // NB: The first time a script is asked about (e.g. just opened), its + // dependencies need no time to settle. + final boolean settled; + synchronized (installs) { + settled = known.add(name); + } + final String overlay; + try { + final String metadata = InlineMetadata.extract(text); + if (metadata == null) return env; + overlay = overlay(name, env.interpreter(), InlineMetadata.dependencies( + metadata), settled); + } + catch (final RuntimeException exc) { + return env; // NB: E.g. half-typed metadata. + } + if (overlay == null) return env; + final List paths = new ArrayList<>(env.paths()); + paths.add(overlay); + return new Environment(env.interpreter(), paths, env.live()); + } + + /** + * Gets the overlay providing the packages the environment lacks, if it is + * ready; starts preparing it once the dependencies have settled. Never + * blocks. + * + * @param envName The overlay's name (e.g. of its script). + * @param python The environment's Python executable, or null if not built. + * @param dependencies The script's dependencies (PEP 508 strings). + * @param settled Whether the dependencies are known not to be changing + * (e.g. the script was just opened), so need no time to settle. + * @return The overlay directory, or null if none is needed or ready. + */ + String overlay(final String envName, final String python, + final List dependencies, final boolean settled) + { + if (dependencies == null || dependencies.isEmpty()) return null; + final String key = envName + "\n" + python + "\n" + dependencies; + synchronized (installs) { + final CompletableFuture install = installs.get(key); + if (install != null) { + return install.isDone() && !install.isCompletedExceptionally() + ? install.join() : null; + } + // Wait for the dependencies to settle, e.g. a name to be typed fully. + final long now = clock.getAsLong(); + final Seen last = seen.get(envName); + if (last == null || !last.key.equals(key)) { + seen.put(envName, new Seen(key, now)); + if (!settled) return null; + } + else if (!settled && now - last.since < settleMillis) return null; + final File overlay = new File(root, envName); + final CompletableFuture started = CompletableFuture.supplyAsync( + () -> installer.install(python, dependencies, overlay), executor); + // NB: Once done, asking again gets the overlay. + installs.put(key, started); + return null; + } + } + + /** + * The name of a script's overlay: after its file, if any; otherwise + * {@code unsaved}, as untitled scripts share one. + * + * @param path The script's file path, or null if it has none. + */ + static String name(final String path) { + if (path == null) return "unsaved"; + final File f = new File(path).getAbsoluteFile(); + return f.getName().replaceAll("[^A-Za-z0-9_.-]", "_") + "-" + Integer + .toHexString(f.getPath().hashCode()); + } + + /** + * Installs the packages the environment with the given Python lacks into an + * overlay directory, with uv (as Appose has it): {@link Installer} as used + * outside tests. + */ + static String install(final String python, final List dependencies, + final File overlay) + { + try { + final Set installed = python == null ? new HashSet<>() + : installed(python); + final List missing = new ArrayList<>(); + for (final String d : dependencies) { + final String name = distribution(d); + if (name != null && !installed.contains(name)) missing.add(d); + } + if (missing.isEmpty()) { + delete(overlay.toPath()); + return null; + } + // NB: Up to date already, if installed for this Python and these + // requirements. + final Map wanted = new LinkedHashMap<>(); + wanted.put("python", python); + wanted.put("requirements", new ArrayList<>(new TreeSet<>(missing))); + final ObjectMapper json = new ObjectMapper(); + final File marker = new File(overlay, MARKER); + if (marker.isFile() && json.readTree(marker).equals(json.valueToTree( + wanted))) return overlay.getPath(); + + // NB: Installed next to the overlay, then swapped in, so that it is + // never seen half-installed. + final Path staging = new File(overlay.getPath() + ".new").toPath(); + delete(staging); + final Uv uv = new Uv(); + if (!uv.isInstalled()) uv.install(); + final List args = new ArrayList<>(Arrays.asList("pip", + "install", "--target", staging.toString(), "--no-deps")); + if (python != null) args.addAll(Arrays.asList("--python", python)); + args.addAll(missing); + uv.exec(args.toArray(new String[0])); + json.writeValue(staging.resolve(MARKER).toFile(), wanted); + delete(overlay.toPath()); + Files.move(staging, overlay.toPath(), StandardCopyOption.ATOMIC_MOVE); + return overlay.getPath(); + } + catch (final IOException | InterruptedException | RuntimeException exc) { + throw new IllegalStateException("Overlay installation failed", exc); + } + } + + /** Marks an overlay with what was installed into it, for which Python. */ + private static final String MARKER = ".scijava-overlay.json"; + + /** A requirement's distribution name, e.g. {@code scikit-image}. */ + private static final Pattern REQUIREMENT = Pattern.compile( + "\\s*([A-Za-z0-9][A-Za-z0-9._-]*).*", Pattern.DOTALL); + + /** Lists the distributions installed in an environment, as JSON. */ + private static final String INSTALLED = "import importlib.metadata as m, " + + "json, re\n" + "print(json.dumps(sorted({re.sub(r'[-_.]+', '-', " + + "(d.metadata['Name'] or '').lower()) for d in m.distributions()})))"; + + /** + * The normalized distribution name of a PEP 508 requirement, or null if + * none. + */ + static String distribution(final String requirement) { + final Matcher m = REQUIREMENT.matcher(requirement); + return m.matches() ? m.group(1).toLowerCase().replaceAll("[-_.]+", "-") + : null; + } + + /** The distributions installed in the environment of a Python. */ + private static Set installed(final String python) + throws IOException, InterruptedException + { + final Process p = new ProcessBuilder(python, "-c", INSTALLED) + .redirectErrorStream(false).start(); + final ByteArrayOutputStream out = new ByteArrayOutputStream(); + try (final InputStream in = p.getInputStream()) { + final byte[] buf = new byte[8192]; + int n; + while ((n = in.read(buf)) > 0) out.write(buf, 0, n); + } + if (!p.waitFor(2, TimeUnit.MINUTES) || p.exitValue() != 0) { + p.destroyForcibly(); + return new HashSet<>(); // NB: As if nothing were installed. + } + final Set names = new HashSet<>(); + new ObjectMapper().readTree(new String(out.toByteArray(), + StandardCharsets.UTF_8)).forEach(n -> names.add(n.asText())); + return names; + } + + /** Deletes a directory, if any. */ + private static void delete(final Path dir) throws IOException { + if (!Files.exists(dir)) return; + try (final Stream paths = Files.walk(dir)) { + paths.sorted(Comparator.reverseOrder()).forEach(f -> f.toFile() + .delete()); + } + } + + private static final class Seen { + + private final String key; + private final long since; + + private Seen(final String key, final long since) { + this.key = key; + this.since = since; + } + } +} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java index c45740b..b63a63e 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java @@ -64,9 +64,9 @@ * {@link LanguageServerService}, as editors use it: what is particular to * these scripts. *

    - * The first completion builds jedi-language-server's environment, and a - * script run the small uv environment of {@code appose-test/requirements.txt}; - * both require network access. + * The first completion builds jedi-language-server's environment, a script + * run the small uv environment of {@code appose-test/requirements.txt}, and a + * new dependency installs an overlay; all require network access. *

    * * @author Gabriel Selzer @@ -145,6 +145,15 @@ public void testParameterBeforeEnvironment() throws Exception { "shape"); } + @Test + public void testNewDependency() throws Exception { + // A script declaring a dependency that is installed nowhere: installed + // for completion only (in an overlay), once the dependencies settle. + final String header = "# /// script\n# dependencies = [\"tifffile\"]\n" + + "# ///\nimport tifffile\n"; + completeUntil(header + "tifffile.imr", null, "imread"); + } + // -- Helper methods -- /** The scripts' sessions, by path ("" for an untitled one). */ @@ -152,13 +161,14 @@ public void testParameterBeforeEnvironment() throws Exception { /** The session about a script (null path: untitled). */ private static ScriptSession session(final String path) { - return sessions.computeIfAbsent(path == null ? "" : path, p -> servers - .session(language).setFile(() -> path == null ? null : new File(path))); + return sessions.computeIfAbsent(path == null ? "" : path, + p -> new ScriptSession(servers, language).setFile(() -> path == null + ? null : new File(path))); } /** The problems found in a script, in a session of its own. */ private static List problems(final String code) { - try (ScriptSession session = servers.session(language)) { + try (ScriptSession session = new ScriptSession(servers, language)) { return session.update(code).diagnostics(120_000); } } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java new file mode 100644 index 0000000..4fa7b15 --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java @@ -0,0 +1,168 @@ +/*- + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.atomic.AtomicLong; + +import org.junit.Test; +import org.scijava.code.lsp.Environment; + +/** + * Tests {@link DependencyOverlays}. + * + * @author Gabriel Selzer + */ +public class DependencyOverlaysTest { + + private final AtomicLong now = new AtomicLong(); + private final List> installs = new ArrayList<>(); + + private final DependencyOverlays overlays = new DependencyOverlays((python, + dependencies, dir) -> { + synchronized (installs) { + installs.add(dependencies); + } + return dir.getPath(); + }, new File("overlays"), 1500, now::get); + + @Test + public void testWaitsForDependenciesToSettle() throws Exception { + final List deps = Arrays.asList("numpy", "scikit-ima"); + // Still being typed: nothing is installed... + assertNull(overlays.overlay("blur", null, deps, false)); + now.addAndGet(500); + assertNull(overlays.overlay("blur", null, deps, false)); + final List typed = Arrays.asList("numpy", "scikit-image"); + assertNull(overlays.overlay("blur", null, typed, false)); + now.addAndGet(1000); + assertNull(overlays.overlay("blur", null, typed, false)); + Thread.sleep(100); + assertEquals(Collections.emptyList(), installs); + // ...until the dependencies have stayed the same for a while. + now.addAndGet(1000); + assertNull(overlays.overlay("blur", null, typed, false)); + assertEquals(new File("overlays", "blur").getPath(), awaitOverlay("blur", + null, typed)); + assertEquals(Collections.singletonList(typed), installs); + } + + @Test + public void testSettledAndReused() throws Exception { + final List deps = Collections.singletonList("tifffile"); + // Known not to be changing (e.g. just opened): installed at once. + assertNull(overlays.overlay("read", "python", deps, true)); + final String overlay = awaitOverlay("read", "python", deps); + assertEquals(new File("overlays", "read").getPath(), overlay); + // Asked again: the same overlay, not installed again. + assertEquals(overlay, overlays.overlay("read", "python", deps, false)); + assertEquals(1, installs.size()); + // Nothing declared: no overlay. + assertNull(overlays.overlay("read", "python", Collections.emptyList(), + true)); + } + + @Test + public void testEnvironment() throws Exception { + final Environment env = new Environment("/env/python", Collections + .singletonList("/stubs")); + // A script declaring a dependency, asked about for the first time (e.g. + // just opened): its dependencies are settled, so the overlay is + // installed; once ready, it is searched too. + final String script = "# /// script\n# dependencies = [\"tifffile\"]\n" + + "# ///\nimport tifffile\ntifffile.im"; + assertSame(env, overlays.environment(script, null, env)); + Environment overlaid = env; + for (int i = 0; i < 100 && overlaid == env; i++) { + Thread.sleep(20); + overlaid = overlays.environment(script, null, env); + } + assertEquals(Collections.singletonList(Collections.singletonList( + "tifffile")), installs); + assertEquals("/env/python", overlaid.interpreter()); + assertEquals(Arrays.asList("/stubs", new File("overlays", "unsaved") + .getPath()), overlaid.paths()); + // Nothing declared, or half-typed: as it is. + final String b = new File("b.py").getPath(); + assertSame(env, overlays.environment("import os", b, env)); + assertSame(env, overlays.environment( + "# /// script\n# dependencies = [\"tiff\n# ///\n", b, env)); + // A script typed from scratch: its dependencies, being typed, settle + // first. + assertSame(env, overlays.environment("# /// script\n" + + "# dependencies = [\"n\"]\n# ///\n", b, env)); + Thread.sleep(100); + assertEquals(1, installs.size()); + } + + @Test + public void testNames() { + // Untitled scripts share one overlay; files get one each. + assertEquals("unsaved", DependencyOverlays.name(null)); + final String name = DependencyOverlays.name(new File("a", "my blur.py") + .getPath()); + assertTrue(name.startsWith("my_blur.py-")); + assertNotEquals(name, DependencyOverlays.name(new File("b", "my blur.py") + .getPath())); + } + + @Test + public void testDistributions() { + // As pip normalizes them (PEP 503), whatever their versions. + assertEquals("scikit-image", DependencyOverlays.distribution( + "scikit_image>=0.22")); + assertEquals("numpy", DependencyOverlays.distribution("NumPy")); + assertEquals("zope-interface", DependencyOverlays.distribution( + " zope.interface ; python_version > '3'")); + assertNull(DependencyOverlays.distribution(">=1")); + } + + /** Waits for the overlay to be ready. */ + private String awaitOverlay(final String env, final String python, + final List deps) throws InterruptedException + { + for (int i = 0; i < 100; i++) { + final String overlay = overlays.overlay(env, python, deps, false); + if (overlay != null) return overlay; + Thread.sleep(20); + } + return null; + } +} From cd073e6c315178075693c49158cc945a4c311dcf Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Tue, 20 Oct 2026 15:32:48 -0500 Subject: [PATCH 34/37] Prebuild script environments for completion, not overlays Completion now builds the environment a script declares, in the background, once its configuration has stayed the same for a while (ApposePythonEnvironments): waiting for a script running in it first, and replacing its worker, as the script's next run would. Completion then analyzes the script there, conda packages and environment files included; and the next run need not build it. So the dependency overlays are gone. Also, an environment is no longer rebuilt in place while a worker of its earlier configuration may still be running: ResidentWorker.release() says when its process has ended, which LazyEnvironment waits for; and files of an environment being built are not used. Workers are made in one place (ScriptEnvironment.worker). FIXME: unsaved scripts are named differently by completion and by runs from the script editor, so a run builds another environment. Co-Authored-By: Claude Opus 5.5 --- .../python/ApposePythonScriptEngine.java | 3 +- .../appose/python/ScriptEnvironment.java | 14 + .../python/_internal/LazyEnvironment.java | 47 +++ .../python/_internal/ResidentWorker.java | 29 +- .../_internal/ResidentWorkerService.java | 31 +- .../completion/ApposePythonDialect.java | 36 +- .../completion/ApposePythonEnvironments.java | 159 +++++++- .../python/completion/DependencyOverlays.java | 357 ------------------ .../python/_internal/ResidentWorkerTest.java | 49 +++ ...ApposePythonCompletionIntegrationTest.java | 8 +- .../completion/ApposePythonDialectTest.java | 5 +- .../ApposePythonEnvironmentsTest.java | 96 +++++ .../completion/DependencyOverlaysTest.java | 168 --------- 13 files changed, 430 insertions(+), 572 deletions(-) delete mode 100644 src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java create mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironmentsTest.java delete mode 100644 src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngine.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngine.java index e5fb180..4905008 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngine.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ApposePythonScriptEngine.java @@ -455,8 +455,7 @@ private ResidentWorker worker(final ScriptInfo info, final String script) throw scriptException(e.getMessage(), e); } if (env == null) throw noEnvironment(); - return workerService.worker(env.name, env.config(), env::builder, - ApposePythonScriptEngine::launch); + return env.worker(workerService); } /** Creates the service of a worker in the given (built) environment. */ diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java index 6442b2f..dbb5b1d 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/ScriptEnvironment.java @@ -41,6 +41,8 @@ import org.apposed.appose.Appose; import org.apposed.appose.Builder; import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; +import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; /** * The Appose environment a script declares: either inline, via a PEP 723 @@ -174,6 +176,18 @@ public String config() { return scheme + "\n" + content; } + /** + * Gets the resident worker of this environment (see + * {@link ResidentWorkerService#worker}), as scripts run it: replacing the + * environment's worker of an earlier configuration, if any. The + * environment is built by the worker's first task, or when asked (see + * {@link ResidentWorker#environment()}). + */ + public ResidentWorker worker(final ResidentWorkerService workers) { + return workers.worker(name, config(), this::builder, + ApposePythonScriptEngine::launch); + } + /** Creates a builder for the environment. */ Builder builder() { final Builder builder = Appose.content(content); diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/LazyEnvironment.java b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/LazyEnvironment.java index c3f00a5..d7470cb 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/LazyEnvironment.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/LazyEnvironment.java @@ -31,7 +31,11 @@ import java.util.List; import java.util.Map; +import java.util.Set; +import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; import java.util.concurrent.CopyOnWriteArrayList; import org.apposed.appose.BuildException; @@ -45,6 +49,8 @@ * Builds of environments with the same name are serialized, even across * instances, because they share a directory on disk: e.g. when the * environment's configuration changes while an older build is still running. + * For the same reason, a build may wait for a worker of the environment's + * earlier configuration to end, so as not to rebuild beneath it. *

    */ public class LazyEnvironment { @@ -53,8 +59,20 @@ public class LazyEnvironment { private static final Map BUILD_LOCKS = new ConcurrentHashMap<>(); + /** The names of the environments being built now. */ + private static final Set BUILDING = ConcurrentHashMap.newKeySet(); + + /** + * Whether the named environment is being built now: if so, its files are + * not to be used yet (e.g. its Python may exist before its packages do). + */ + public static boolean isBuilding(final String name) { + return BUILDING.contains(name); + } + private final String name; private final Builder builder; + private final Future before; private final List listeners = new CopyOnWriteArrayList<>(); private volatile Environment env; @@ -65,7 +83,21 @@ public class LazyEnvironment { * @param builder Builder for the environment; built on first use. */ public LazyEnvironment(final String name, final Builder builder) { + this(name, builder, CompletableFuture.completedFuture(null)); + } + + /** + * @param name Name of the environment, as reported to listeners, and as + * given to the builder. + * @param builder Builder for the environment; built on first use. + * @param before What to wait for before building: e.g. the end of a worker + * still using the environment's earlier configuration. + */ + public LazyEnvironment(final String name, final Builder builder, + final Future before) + { this.name = name; + this.before = before; // Note: Subscribe once, here, and dispatch to the current listeners, // because builders accumulate subscribers with every call. this.builder = builder.name(name) // @@ -96,7 +128,19 @@ public Environment get() throws BuildException { if (env != null) return env; synchronized (BUILD_LOCKS.computeIfAbsent(name, k -> new Object())) { if (env != null) return env; + try { + before.get(); + } + catch (final InterruptedException exc) { + Thread.currentThread().interrupt(); + throw new BuildException("Interrupted waiting to build " + name, + exc); + } + catch (final ExecutionException exc) { + // NB: Whatever it was could not be waited for: build anyway. + } listeners.forEach(l -> l.buildStarted(name)); + BUILDING.add(name); try { env = builder.build(); } @@ -104,6 +148,9 @@ public Environment get() throws BuildException { listeners.forEach(l -> l.buildFinished(name, exc)); throw exc; } + finally { + BUILDING.remove(name); + } listeners.forEach(l -> l.buildFinished(name, null)); return env; } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorker.java b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorker.java index b8ae9e3..7ef0839 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorker.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorker.java @@ -32,6 +32,9 @@ import java.io.IOException; import java.io.UncheckedIOException; import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; import java.util.concurrent.locks.Lock; import java.util.concurrent.locks.ReentrantLock; import java.util.function.Consumer; @@ -61,6 +64,14 @@ */ public class ResidentWorker implements AutoCloseable { + /** Where released workers are waited for, until their processes end. */ + private static final ExecutorService EXITS = Executors.newCachedThreadPool( + r -> { + final Thread t = new Thread(r, "appose-worker-exit"); + t.setDaemon(true); + return t; + }); + private final LazyEnvironment environment; private final Function launcher; private final Lock exclusive = new ReentrantLock(); @@ -141,11 +152,23 @@ public synchronized boolean isAlive() { /** * Ends the worker process gently, letting pending tasks finish, and * releasing everything it holds. The environment stays built. + * + * @return When the process has ended (at once, if none was running): e.g. + * so that its environment is not rebuilt while it still uses it. */ - public synchronized void release() { - if (service == null) return; - service.close(); + public synchronized CompletableFuture release() { + if (service == null) return CompletableFuture.completedFuture(null); + final Service s = service; + s.close(); service = null; + return CompletableFuture.runAsync(() -> { + try { + s.waitFor(); + } + catch (final InterruptedException exc) { + Thread.currentThread().interrupt(); + } + }, EXITS); } /** Ends the worker process immediately, interrupting any pending tasks. */ diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java index f1134b9..ddb11dc 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerService.java @@ -33,6 +33,8 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.Optional; +import java.util.concurrent.CompletableFuture; import java.util.concurrent.CopyOnWriteArrayList; import java.util.function.Function; import java.util.function.Supplier; @@ -76,6 +78,9 @@ public class ResidentWorkerService extends AbstractService implements /** Environments by name, each with its configuration and worker. */ private final Map entries = new HashMap<>(); + /** Whether the context was disposed: no more workers are made. */ + private boolean disposed; + /** Told about every environment's builds. */ private final List buildListeners = new CopyOnWriteArrayList<>(); @@ -100,11 +105,18 @@ public ResidentWorker worker(final String name, final String config, { final ResidentWorker stale; final ResidentWorker worker; + // NB: The environment is rebuilt in place: only once the stale worker, + // still using it, has ended. + final CompletableFuture staleEnded = new CompletableFuture<>(); synchronized (entries) { + if (disposed) { + throw new IllegalStateException("The context was disposed"); + } final Entry entry = entries.get(name); if (entry != null && entry.config.equals(config)) return entry.worker; stale = entry == null ? null : entry.worker; - final LazyEnvironment env = new LazyEnvironment(name, builder.get()); + final LazyEnvironment env = new LazyEnvironment(name, builder.get(), + staleEnded); env.addBuildListener(SciJavaTasks.buildListener(taskService, statusService, log)); env.addBuildListener(new BuildListener() { @@ -119,7 +131,9 @@ public void buildFinished(final String envName, final Throwable error) { worker = new ResidentWorker(env, launcher); entries.put(name, new Entry(config, worker)); } - if (stale != null) stale.release(); + if (stale == null) staleEnded.complete(null); + else stale.release().whenComplete((v, exc) -> staleEnded.complete( + null)); return worker; } @@ -148,6 +162,18 @@ public ResidentWorker find(final String name, final String config) { } } + /** + * Gets the resident worker of the named environment, whatever its + * configuration: e.g. to wait, via its {@link ResidentWorker#exclusive()} + * lock, for a script it runs before replacing it. + */ + public Optional current(final String name) { + synchronized (entries) { + final Entry entry = entries.get(name); + return entry == null ? Optional.empty() : Optional.of(entry.worker); + } + } + /** Gets the current resident workers. */ public List workers() { final List list = new ArrayList<>(); @@ -172,6 +198,7 @@ public void dispose() { releaseAll(); synchronized (entries) { entries.clear(); + disposed = true; } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index 36323fd..33b88c8 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -41,7 +41,6 @@ import net.imglib2.RandomAccessibleInterval; import org.apposed.appose.NDArray; -import org.apposed.appose.util.Environments; import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.ScriptDialect; import org.scijava.plugin.Parameter; @@ -51,9 +50,8 @@ import org.scijava.script.ScriptLanguage; /** - * Describes how to turn Appose python scripts into plain python code, and - * where they run (see {@link ApposePythonEnvironments}), searching the - * packages they declare but that lacks, too (see {@link DependencyOverlays}). + * Describes how to parse Appose python scripts into plain python code and + * report the environment where they run * * @author Gabriel Selzer */ @@ -71,14 +69,6 @@ public class ApposePythonDialect implements ScriptDialect { private ApposePythonEnvironments environments; - /** - * How long a script's dependencies must stay unchanged before packages - * missing from its environment are installed for completion, in ms. - */ - private static final long OVERLAY_SETTLE = 1500; - - private static DependencyOverlays overlays; - @Override public boolean supports(final ScriptLanguage language) { return language instanceof ApposePythonScriptLanguage; @@ -93,12 +83,9 @@ public String languageId() { public Environment environment(final String text, final Optional file) { - // NB: As its #@script line, or its inline metadata, says; plus the - // packages it declares but that environment lacks (installed for - // completion only, see DependencyOverlays). - final String path = file.map(File::getPath).orElse(null); - return overlays().environment(text, path, environments().environment( - text, path)); + // NB: As its #@script line, or its inline metadata, says. + return environments().environment(text, file.map(File::getPath).orElse( + null)); } @Override @@ -158,19 +145,6 @@ static String parameterType(final Class type) { return null; } - /** - * Gets the overlays of packages scripts declare, but their environments - * lack: one set, for all dialect instances (sessions). - */ - private static synchronized DependencyOverlays overlays() { - if (overlays == null) { - overlays = new DependencyOverlays(DependencyOverlays::install, new File( - Environments.apposeEnvsDir(), "appose-python-completion-overlays"), - OVERLAY_SETTLE, System::currentTimeMillis); - } - return overlays; - } - /** Gets where scripts run, creating it on first use. */ private synchronized ApposePythonEnvironments environments() { if (environments == null) { diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java index 13b5629..d4f1c3c 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java @@ -33,31 +33,79 @@ import java.io.IOException; import java.util.ArrayList; import java.util.Arrays; +import java.util.HashMap; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.Optional; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.locks.Lock; +import java.util.function.Consumer; import org.apposed.appose.Environment; import org.apposed.appose.util.Environments; import org.scijava.plugins.scripting.appose.python.ScriptEnvironment; import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; +import org.scijava.plugins.scripting.appose.python._internal.LazyEnvironment; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorker; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; /** * Where Python scripts run via Appose, as their language servers need to * know: the Python of a script's environment, if built (in this session, or an - * earlier one; never building it, nor disturbing its worker). (Packages a - * script declares, but its environment lacks, are known to the servers too; - * see scijava-code-lsp-jedi.) + * earlier one). + *

    + * An environment not built yet (e.g. as the script's dependencies are + * edited) is built in the background, once its configuration has stayed the + * same for a while, so that completion soon knows what the script will run + * with. A script running in the environment is waited for first: its + * environment is not rebuilt beneath it. Building replaces the environment's + * resident worker (of its earlier configuration), as the script's next run + * would. + *

    * * @author Gabriel Selzer */ final class ApposePythonEnvironments { + /** + * How long an environment's configuration must stay the same before it is + * built, in milliseconds. + */ + static final long SETTLE = 5000; + + /** Where settling environments are waited for. */ + private static final ScheduledExecutorService TIMER = Executors + .newSingleThreadScheduledExecutor(daemon("appose-python-settle")); + + /** Where environments are built: one at a time. */ + private static final ExecutorService BUILDS = Executors + .newSingleThreadExecutor(daemon("appose-python-prebuild")); + /** The resident workers, to find built environments; or null if none. */ private final ResidentWorkerService workers; + /** How long a configuration must stay the same before it is built. */ + private final long settle; + + /** Builds an environment; or null, to build none. */ + private final Consumer build; + + /** Scripts waiting to settle, by path ("" if unsaved): what, and when. */ + private final Map settling = new HashMap<>(); + + /** + * The configuration last built (or failing to) for each script, by path + * ("" if unsaved): not built again, unless the script changes to another. + */ + private final Map tried = new HashMap<>(); + /** Python executables of script environments, by script and metadata. */ private final Map pythons = new LinkedHashMap( 16, 0.75f, true) @@ -71,9 +119,25 @@ protected boolean removeEldestEntry( } }; - /** @param workers The resident workers, or null if none. */ + /** @param workers The resident workers, or null if none (none built). */ ApposePythonEnvironments(final ResidentWorkerService workers) { + this(workers, SETTLE, null); + } + + /** + * @param workers The resident workers, or null if none. + * @param settle How long a configuration must stay the same before it is + * built, in milliseconds. + * @param build Builds an environment (e.g. a fake one, in tests); or null, + * to build them with the workers (if any). + */ + ApposePythonEnvironments(final ResidentWorkerService workers, + final long settle, final Consumer build) + { this.workers = workers; + this.settle = settle; + this.build = build != null ? build : workers != null ? this::build + : null; } /** @@ -123,6 +187,13 @@ String scriptPython(final String text, final String path) { /** As {@link #scriptPython}, without remembering. */ private String findScriptPython(final String text, final String path) { + // FIXME: For an unsaved script, this path is null, so its environment is + // named after its metadata (see ScriptEnvironment#inlineName). But the + // script editor runs it with its tab's name as its path (e.g. New_.py), + // so a run names it after that: the environment built here is not the + // one the run uses, which builds another. Completion and runs should + // agree on unsaved scripts' paths: e.g. the engine taking a path that is + // no file as none, or the editor giving sessions its tab's name too. final ScriptEnvironment env; try { final Map directive = ScriptEnvironment.directive(text); @@ -146,10 +217,70 @@ private String findScriptPython(final String text, final String path) { // NB: Fall back to looking on disk. } } + else prepare(path, env); + // NB: Being built (e.g. above): not usable yet, though its Python may + // already exist. + if (LazyEnvironment.isBuilding(env.name)) return null; // Built in an earlier session? Its files are where Appose keeps them. return pythonOnDisk(new File(Environments.apposeEnvsDir(), env.name)); } + /** + * Builds an environment once its configuration has stayed the same for a + * while (see {@link #settle}), unless it was tried already. + */ + private void prepare(final String path, final ScriptEnvironment env) { + // NB: By script, not by environment: an unsaved script's environment + // is named after its configuration, so changes to it with each edit. + final String script = path == null ? "" : path; + if (build == null) return; + final String key = env.name + "\n" + env.config(); + synchronized (settling) { + if (key.equals(tried.get(script))) return; + final Settling last = settling.get(script); + if (last != null) { + if (last.key.equals(key)) return; // NB: Already waiting. + last.timer.cancel(false); + } + settling.put(script, new Settling(key, TIMER.schedule(() -> { + synchronized (settling) { + settling.remove(script); + if (key.equals(tried.put(script, key))) return; + } + CompletableFuture.runAsync(() -> build.accept(env), BUILDS); + }, settle, TimeUnit.MILLISECONDS))); + } + } + + /** + * Builds an environment, with the resident workers: waiting for a script + * running in it first, then replacing its worker, as the script's next run + * would. + */ + private void build(final ScriptEnvironment env) { + try { + final Optional running = workers.current(env.name).map( + ResidentWorker::exclusive); + final ResidentWorker worker; + if (running.isPresent()) running.get().lockInterruptibly(); + try { + worker = env.worker(workers); + } + finally { + running.ifPresent(Lock::unlock); + } + // NB: Shown as a task, as all builds are (see ResidentWorkerService). + worker.environment().get(); + } + catch (final InterruptedException exc) { + Thread.currentThread().interrupt(); + } + catch (final Exception exc) { + // NB: E.g. the context was disposed, or the build failed (which its + // task shows): completion goes on without it. + } + } + /** Finds the Python executable of a built environment, or null. */ static String pythonExecutable(final Environment env) { // NB: Look in the bin paths, and in their parents: on Windows, conda-style @@ -208,4 +339,24 @@ private static String findPython(final List dirs) { } return null; } + + private static ThreadFactory daemon(final String name) { + return r -> { + final Thread t = new Thread(r, name); + t.setDaemon(true); + return t; + }; + } + + /** An environment's configuration, waiting to settle. */ + private static final class Settling { + + private final String key; + private final ScheduledFuture timer; + + private Settling(final String key, final ScheduledFuture timer) { + this.key = key; + this.timer = timer; + } + } } diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java deleted file mode 100644 index af037a0..0000000 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlays.java +++ /dev/null @@ -1,357 +0,0 @@ -/*- - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import java.io.ByteArrayOutputStream; -import java.io.File; -import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.StandardCopyOption; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Comparator; -import java.util.LinkedHashMap; -import java.util.TreeSet; -import java.util.concurrent.TimeUnit; -import java.util.regex.Matcher; -import java.util.regex.Pattern; -import java.util.stream.Stream; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.function.LongSupplier; - -import com.fasterxml.jackson.databind.ObjectMapper; - -import org.apposed.appose.tool.Uv; -import org.scijava.code.lsp.Environment; -import org.scijava.plugins.scripting.appose.python._internal.InlineMetadata; - -/** - * Makes packages a script declares (in its PEP 723 {@code # /// script} - * block), but its environment lacks, known to code completion before the - * environment is (re)built: e.g. a dependency just added to the script. - *

    - * The missing packages are installed, in the background, into a separate - * directory per script (an "overlay"), which the environment a script is - * analyzed in then searches too (see {@link ApposePythonDialect#environment}). - * The environment itself is never touched, so this never disturbs whatever - * runs in it (e.g. an Appose worker). Installs start only once the - * dependencies have stopped changing for a moment, so that half-typed names - * do not cause installs; and only if the missing packages differ from the - * overlay's. - *

    - *

    - * Alternatives considered, which may be worth revisiting: - *

    - *
      - *
    • Rebuilding the script's environment as its dependencies - * change. Jedi would then see exactly what will run, but builds can take - * minutes and gigabytes, half-finished edits would trigger builds, and e.g. Appose - * rebuilds environments in place, which can interfere with (or, by changing - * its configuration, release) a resident worker using it.
      • - *
    • Installing nothing, but hinting that new dependencies are - * known only once the script has run. Cheap and honest, but the user must - * run the script first. It may still be worthwhile for dependencies an - * overlay cannot provide: conda packages (e.g. - * {@code [tool.pixi.dependencies]}), and environments declared in files via - * {@code #@script(env=...)} in Appose-Python scripts, which overlays do not - * handle.
      • - *
    - * - * @author Gabriel Selzer - */ -final class DependencyOverlays { - - /** Installs the missing packages of a script's environment. */ - @FunctionalInterface - interface Installer { - - /** - * Installs the packages among {@code dependencies} that the environment - * with the given Python lacks into the given overlay directory. - * - * @param python The environment's Python executable, or null if the - * environment has not been built (all packages are missing). - * @param dependencies The script's dependencies, as PEP 508 strings. - * @param overlay The directory to install into. - * @return The overlay directory, or null if nothing is missing. - */ - String install(String python, List dependencies, File overlay); - } - - private final Installer installer; - private final File root; - private final long settleMillis; - private final LongSupplier clock; - private final ExecutorService executor = Executors.newSingleThreadExecutor( - r -> { - final Thread t = new Thread(r, "appose-python-overlays"); - t.setDaemon(true); - return t; - }); - - /** Installs, by environment name, Python and dependencies. */ - private final Map> installs = - new HashMap<>(); - - /** The dependencies last seen for each environment, and since when. */ - private final Map seen = new HashMap<>(); - - /** The scripts asked about so far, by overlay name. */ - private final Set known = new HashSet<>(); - - /** - * @param installer Does the installing, on a background thread. - * @param root Directory holding the overlays (one per script). - * @param settleMillis How long dependencies must stay unchanged before - * their missing packages are installed. - * @param clock The current time, in milliseconds. - */ - DependencyOverlays(final Installer installer, final File root, - final long settleMillis, final LongSupplier clock) - { - this.installer = installer; - this.root = root; - this.settleMillis = settleMillis; - this.clock = clock; - } - - /** - * Gets the environment to analyze a script in: the given one, searching - * the overlay of the packages the script declares but the environment lacks - * too, once ready (see {@link #overlay}). Never blocks. - * - * @param text The script. - * @param path The script's file path, or null if it has none. - * @param env Where the script runs. - */ - Environment environment(final String text, final String path, - final Environment env) - { - final String name = name(path); - // NB: The first time a script is asked about (e.g. just opened), its - // dependencies need no time to settle. - final boolean settled; - synchronized (installs) { - settled = known.add(name); - } - final String overlay; - try { - final String metadata = InlineMetadata.extract(text); - if (metadata == null) return env; - overlay = overlay(name, env.interpreter(), InlineMetadata.dependencies( - metadata), settled); - } - catch (final RuntimeException exc) { - return env; // NB: E.g. half-typed metadata. - } - if (overlay == null) return env; - final List paths = new ArrayList<>(env.paths()); - paths.add(overlay); - return new Environment(env.interpreter(), paths, env.live()); - } - - /** - * Gets the overlay providing the packages the environment lacks, if it is - * ready; starts preparing it once the dependencies have settled. Never - * blocks. - * - * @param envName The overlay's name (e.g. of its script). - * @param python The environment's Python executable, or null if not built. - * @param dependencies The script's dependencies (PEP 508 strings). - * @param settled Whether the dependencies are known not to be changing - * (e.g. the script was just opened), so need no time to settle. - * @return The overlay directory, or null if none is needed or ready. - */ - String overlay(final String envName, final String python, - final List dependencies, final boolean settled) - { - if (dependencies == null || dependencies.isEmpty()) return null; - final String key = envName + "\n" + python + "\n" + dependencies; - synchronized (installs) { - final CompletableFuture install = installs.get(key); - if (install != null) { - return install.isDone() && !install.isCompletedExceptionally() - ? install.join() : null; - } - // Wait for the dependencies to settle, e.g. a name to be typed fully. - final long now = clock.getAsLong(); - final Seen last = seen.get(envName); - if (last == null || !last.key.equals(key)) { - seen.put(envName, new Seen(key, now)); - if (!settled) return null; - } - else if (!settled && now - last.since < settleMillis) return null; - final File overlay = new File(root, envName); - final CompletableFuture started = CompletableFuture.supplyAsync( - () -> installer.install(python, dependencies, overlay), executor); - // NB: Once done, asking again gets the overlay. - installs.put(key, started); - return null; - } - } - - /** - * The name of a script's overlay: after its file, if any; otherwise - * {@code unsaved}, as untitled scripts share one. - * - * @param path The script's file path, or null if it has none. - */ - static String name(final String path) { - if (path == null) return "unsaved"; - final File f = new File(path).getAbsoluteFile(); - return f.getName().replaceAll("[^A-Za-z0-9_.-]", "_") + "-" + Integer - .toHexString(f.getPath().hashCode()); - } - - /** - * Installs the packages the environment with the given Python lacks into an - * overlay directory, with uv (as Appose has it): {@link Installer} as used - * outside tests. - */ - static String install(final String python, final List dependencies, - final File overlay) - { - try { - final Set installed = python == null ? new HashSet<>() - : installed(python); - final List missing = new ArrayList<>(); - for (final String d : dependencies) { - final String name = distribution(d); - if (name != null && !installed.contains(name)) missing.add(d); - } - if (missing.isEmpty()) { - delete(overlay.toPath()); - return null; - } - // NB: Up to date already, if installed for this Python and these - // requirements. - final Map wanted = new LinkedHashMap<>(); - wanted.put("python", python); - wanted.put("requirements", new ArrayList<>(new TreeSet<>(missing))); - final ObjectMapper json = new ObjectMapper(); - final File marker = new File(overlay, MARKER); - if (marker.isFile() && json.readTree(marker).equals(json.valueToTree( - wanted))) return overlay.getPath(); - - // NB: Installed next to the overlay, then swapped in, so that it is - // never seen half-installed. - final Path staging = new File(overlay.getPath() + ".new").toPath(); - delete(staging); - final Uv uv = new Uv(); - if (!uv.isInstalled()) uv.install(); - final List args = new ArrayList<>(Arrays.asList("pip", - "install", "--target", staging.toString(), "--no-deps")); - if (python != null) args.addAll(Arrays.asList("--python", python)); - args.addAll(missing); - uv.exec(args.toArray(new String[0])); - json.writeValue(staging.resolve(MARKER).toFile(), wanted); - delete(overlay.toPath()); - Files.move(staging, overlay.toPath(), StandardCopyOption.ATOMIC_MOVE); - return overlay.getPath(); - } - catch (final IOException | InterruptedException | RuntimeException exc) { - throw new IllegalStateException("Overlay installation failed", exc); - } - } - - /** Marks an overlay with what was installed into it, for which Python. */ - private static final String MARKER = ".scijava-overlay.json"; - - /** A requirement's distribution name, e.g. {@code scikit-image}. */ - private static final Pattern REQUIREMENT = Pattern.compile( - "\\s*([A-Za-z0-9][A-Za-z0-9._-]*).*", Pattern.DOTALL); - - /** Lists the distributions installed in an environment, as JSON. */ - private static final String INSTALLED = "import importlib.metadata as m, " + - "json, re\n" + "print(json.dumps(sorted({re.sub(r'[-_.]+', '-', " + - "(d.metadata['Name'] or '').lower()) for d in m.distributions()})))"; - - /** - * The normalized distribution name of a PEP 508 requirement, or null if - * none. - */ - static String distribution(final String requirement) { - final Matcher m = REQUIREMENT.matcher(requirement); - return m.matches() ? m.group(1).toLowerCase().replaceAll("[-_.]+", "-") - : null; - } - - /** The distributions installed in the environment of a Python. */ - private static Set installed(final String python) - throws IOException, InterruptedException - { - final Process p = new ProcessBuilder(python, "-c", INSTALLED) - .redirectErrorStream(false).start(); - final ByteArrayOutputStream out = new ByteArrayOutputStream(); - try (final InputStream in = p.getInputStream()) { - final byte[] buf = new byte[8192]; - int n; - while ((n = in.read(buf)) > 0) out.write(buf, 0, n); - } - if (!p.waitFor(2, TimeUnit.MINUTES) || p.exitValue() != 0) { - p.destroyForcibly(); - return new HashSet<>(); // NB: As if nothing were installed. - } - final Set names = new HashSet<>(); - new ObjectMapper().readTree(new String(out.toByteArray(), - StandardCharsets.UTF_8)).forEach(n -> names.add(n.asText())); - return names; - } - - /** Deletes a directory, if any. */ - private static void delete(final Path dir) throws IOException { - if (!Files.exists(dir)) return; - try (final Stream paths = Files.walk(dir)) { - paths.sorted(Comparator.reverseOrder()).forEach(f -> f.toFile() - .delete()); - } - } - - private static final class Seen { - - private final String key; - private final long since; - - private Seen(final String key, final long since) { - this.key = key; - this.since = since; - } - } -} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerTest.java index aa44951..7459344 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/_internal/ResidentWorkerTest.java @@ -32,14 +32,18 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotEquals; import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; import java.io.File; +import java.lang.reflect.Proxy; import java.nio.charset.StandardCharsets; import java.nio.file.Files; +import java.util.concurrent.CompletableFuture; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; import org.apposed.appose.Appose; +import org.apposed.appose.BuildException; import org.apposed.appose.Builder; import org.junit.Test; @@ -93,11 +97,56 @@ public void buildFinished(final String envName, final Throwable error) { } } + @Test + public void testBuildWaitsForEarlierWorker() throws Exception { + // An environment whose earlier worker is still ending: not rebuilt + // beneath it, but once it has ended. + final CompletableFuture ended = new CompletableFuture<>(); + final AtomicInteger builds = new AtomicInteger(); + final LazyEnvironment env = new LazyEnvironment("test", fakeBuilder( + builds), ended); + final Thread building = new Thread(() -> { + try { + env.get(); + } + catch (final BuildException exc) { + throw new RuntimeException(exc); + } + }); + building.start(); + Thread.sleep(200); + assertEquals(0, builds.get()); + ended.complete(null); + building.join(10_000); + assertEquals(1, builds.get()); + } + + @Test + public void testReleaseEnds() { + // A worker never started: ended at once. + final ResidentWorker idle = new ResidentWorker(new LazyEnvironment( + "test", fakeBuilder(new AtomicInteger())), e -> e.python()); + assertTrue(idle.release().isDone()); + } + private static int pid(final ResidentWorker worker) throws Exception { return ((Number) worker.task(PID, null).waitFor().outputs.get("pid")) .intValue(); } + /** A builder counting its builds, building nothing. */ + private static Builder fakeBuilder(final AtomicInteger builds) { + return (Builder) Proxy.newProxyInstance(Builder.class + .getClassLoader(), new Class[] { Builder.class }, (proxy, method, + args) -> { + if (method.getName().equals("build")) { + builds.incrementAndGet(); + return null; + } + return method.getReturnType().isInstance(proxy) ? proxy : null; + }); + } + private static LazyEnvironment environment() throws Exception { final File file = new File(ResidentWorkerTest.class.getResource( "/appose-test/requirements.txt").toURI()); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java index b63a63e..c92ea21 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java @@ -66,7 +66,8 @@ *

    * The first completion builds jedi-language-server's environment, a script * run the small uv environment of {@code appose-test/requirements.txt}, and a - * new dependency installs an overlay; all require network access. + * script declaring a new dependency its own environment; all require network + * access. *

    * * @author Gabriel Selzer @@ -147,8 +148,9 @@ public void testParameterBeforeEnvironment() throws Exception { @Test public void testNewDependency() throws Exception { - // A script declaring a dependency that is installed nowhere: installed - // for completion only (in an overlay), once the dependencies settle. + // A script declaring a dependency that is installed nowhere: its + // environment is built, in the background, once its dependencies have + // settled; then its servers analyze the script there. final String header = "# /// script\n# dependencies = [\"tifffile\"]\n" + "# ///\nimport tifffile\n"; completeUntil(header + "tifffile.imr", null, "imread"); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java index 034beba..84ada75 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java @@ -140,8 +140,9 @@ public void testEnvironmentWithoutOne() { @Test public void testScriptWorkersUntouched() { - // Finding where a script runs must not create (or replace) a worker for - // it, even as its metadata is edited. + // Finding where a script runs does not create (or replace) a worker for + // it at once, even as its metadata is edited: only once the metadata + // has settled (see ApposePythonEnvironmentsTest). final String script = "# /// script\n# dependencies = [\"appose\"]\n" + "# ///\nimport os\nos.pa"; dialect.environment(script, Optional.empty()); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironmentsTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironmentsTest.java new file mode 100644 index 0000000..2679158 --- /dev/null +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironmentsTest.java @@ -0,0 +1,96 @@ +/* + * #%L + * Python scripting language plugin backed by Appose. + * %% + * Copyright (C) 2026 SciJava developers. + * %% + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * #L% + */ + +package org.scijava.plugins.scripting.appose.python.completion; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; + +import org.junit.Test; + +/** + * Tests that {@link ApposePythonEnvironments} builds the environments scripts + * declare once they settle (with a fake build: nothing is built). + * + * @author Gabriel Selzer + */ +public class ApposePythonEnvironmentsTest { + + /** The configurations built, in order. */ + private final List built = new CopyOnWriteArrayList<>(); + + private final ApposePythonEnvironments environments = + new ApposePythonEnvironments(null, 200, env -> built.add(env.config())); + + @Test + public void testBuildsOnceSettled() throws Exception { + // Being typed: not built... + where(script("nu")); + Thread.sleep(100); + where(script("numpy")); + Thread.sleep(100); + assertTrue(built.isEmpty()); + // ...until it has stayed the same for a while: then once. + Thread.sleep(400); + assertEquals(1, built.size()); + assertTrue(built.get(0), built.get(0).contains("numpy")); + where(script("numpy")); + Thread.sleep(400); + assertEquals(1, built.size()); + // Changed, then changed back: built again. + where(script("scipy")); + Thread.sleep(400); + where(script("numpy")); + Thread.sleep(400); + assertEquals(3, built.size()); + } + + @Test + public void testNothingDeclared() throws Exception { + where("import os\nos.pa"); + Thread.sleep(400); + assertTrue(built.isEmpty()); + } + + // -- Helper methods -- + + /** Asks where an unsaved script runs. */ + private void where(final String script) { + environments.environment(script, null); + } + + /** An unsaved script declaring a dependency, in its inline metadata. */ + private static String script(final String dependency) { + return "# /// script\n# dependencies = [\"" + dependency + "\"]\n" + + "# ///\nimport os\n"; + } +} diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java deleted file mode 100644 index 4fa7b15..0000000 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/DependencyOverlaysTest.java +++ /dev/null @@ -1,168 +0,0 @@ -/*- - * #%L - * Python scripting language plugin backed by Appose. - * %% - * Copyright (C) 2026 SciJava developers. - * %% - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * - * 1. Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright notice, - * this list of conditions and the following disclaimer in the documentation - * and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * #L% - */ - -package org.scijava.plugins.scripting.appose.python.completion; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNotEquals; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertSame; -import static org.junit.Assert.assertTrue; - -import java.io.File; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.List; -import java.util.concurrent.atomic.AtomicLong; - -import org.junit.Test; -import org.scijava.code.lsp.Environment; - -/** - * Tests {@link DependencyOverlays}. - * - * @author Gabriel Selzer - */ -public class DependencyOverlaysTest { - - private final AtomicLong now = new AtomicLong(); - private final List> installs = new ArrayList<>(); - - private final DependencyOverlays overlays = new DependencyOverlays((python, - dependencies, dir) -> { - synchronized (installs) { - installs.add(dependencies); - } - return dir.getPath(); - }, new File("overlays"), 1500, now::get); - - @Test - public void testWaitsForDependenciesToSettle() throws Exception { - final List deps = Arrays.asList("numpy", "scikit-ima"); - // Still being typed: nothing is installed... - assertNull(overlays.overlay("blur", null, deps, false)); - now.addAndGet(500); - assertNull(overlays.overlay("blur", null, deps, false)); - final List typed = Arrays.asList("numpy", "scikit-image"); - assertNull(overlays.overlay("blur", null, typed, false)); - now.addAndGet(1000); - assertNull(overlays.overlay("blur", null, typed, false)); - Thread.sleep(100); - assertEquals(Collections.emptyList(), installs); - // ...until the dependencies have stayed the same for a while. - now.addAndGet(1000); - assertNull(overlays.overlay("blur", null, typed, false)); - assertEquals(new File("overlays", "blur").getPath(), awaitOverlay("blur", - null, typed)); - assertEquals(Collections.singletonList(typed), installs); - } - - @Test - public void testSettledAndReused() throws Exception { - final List deps = Collections.singletonList("tifffile"); - // Known not to be changing (e.g. just opened): installed at once. - assertNull(overlays.overlay("read", "python", deps, true)); - final String overlay = awaitOverlay("read", "python", deps); - assertEquals(new File("overlays", "read").getPath(), overlay); - // Asked again: the same overlay, not installed again. - assertEquals(overlay, overlays.overlay("read", "python", deps, false)); - assertEquals(1, installs.size()); - // Nothing declared: no overlay. - assertNull(overlays.overlay("read", "python", Collections.emptyList(), - true)); - } - - @Test - public void testEnvironment() throws Exception { - final Environment env = new Environment("/env/python", Collections - .singletonList("/stubs")); - // A script declaring a dependency, asked about for the first time (e.g. - // just opened): its dependencies are settled, so the overlay is - // installed; once ready, it is searched too. - final String script = "# /// script\n# dependencies = [\"tifffile\"]\n" + - "# ///\nimport tifffile\ntifffile.im"; - assertSame(env, overlays.environment(script, null, env)); - Environment overlaid = env; - for (int i = 0; i < 100 && overlaid == env; i++) { - Thread.sleep(20); - overlaid = overlays.environment(script, null, env); - } - assertEquals(Collections.singletonList(Collections.singletonList( - "tifffile")), installs); - assertEquals("/env/python", overlaid.interpreter()); - assertEquals(Arrays.asList("/stubs", new File("overlays", "unsaved") - .getPath()), overlaid.paths()); - // Nothing declared, or half-typed: as it is. - final String b = new File("b.py").getPath(); - assertSame(env, overlays.environment("import os", b, env)); - assertSame(env, overlays.environment( - "# /// script\n# dependencies = [\"tiff\n# ///\n", b, env)); - // A script typed from scratch: its dependencies, being typed, settle - // first. - assertSame(env, overlays.environment("# /// script\n" + - "# dependencies = [\"n\"]\n# ///\n", b, env)); - Thread.sleep(100); - assertEquals(1, installs.size()); - } - - @Test - public void testNames() { - // Untitled scripts share one overlay; files get one each. - assertEquals("unsaved", DependencyOverlays.name(null)); - final String name = DependencyOverlays.name(new File("a", "my blur.py") - .getPath()); - assertTrue(name.startsWith("my_blur.py-")); - assertNotEquals(name, DependencyOverlays.name(new File("b", "my blur.py") - .getPath())); - } - - @Test - public void testDistributions() { - // As pip normalizes them (PEP 503), whatever their versions. - assertEquals("scikit-image", DependencyOverlays.distribution( - "scikit_image>=0.22")); - assertEquals("numpy", DependencyOverlays.distribution("NumPy")); - assertEquals("zope-interface", DependencyOverlays.distribution( - " zope.interface ; python_version > '3'")); - assertNull(DependencyOverlays.distribution(">=1")); - } - - /** Waits for the overlay to be ready. */ - private String awaitOverlay(final String env, final String python, - final List deps) throws InterruptedException - { - for (int i = 0; i < 100; i++) { - final String overlay = overlays.overlay(env, python, deps, false); - if (overlay != null) return overlay; - Thread.sleep(20); - } - return null; - } -} From 5cd99d0f1cf6a9faa9ba1d1f56e768f8ad16df7f Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Wed, 21 Oct 2026 11:27:15 -0500 Subject: [PATCH 35/37] Follow scijava-code-lsp's sessions and environments Co-Authored-By: Claude Opus 5.5 --- .../completion/ApposePythonEnvironments.java | 5 +++-- .../ApposePythonCompletionIntegrationTest.java | 18 ++++++++++++++---- .../completion/ApposePythonDialectTest.java | 5 +++-- 3 files changed, 20 insertions(+), 8 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java index d4f1c3c..7ac0a69 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonEnvironments.java @@ -33,6 +33,7 @@ import java.io.IOException; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.HashMap; import java.util.LinkedHashMap; import java.util.List; @@ -152,8 +153,8 @@ org.scijava.code.lsp.Environment environment(final String text, final String path) { final String python = scriptPython(text, path); - return python == null ? org.scijava.code.lsp.Environment.NONE - : new org.scijava.code.lsp.Environment(python, null); + return python == null ? new org.scijava.code.lsp.Environment() + : new org.scijava.code.lsp.Environment().withInterpreter(python); } /** diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java index c92ea21..fb49732 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonCompletionIntegrationTest.java @@ -41,6 +41,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.concurrent.CompletableFuture; import java.util.concurrent.TimeUnit; import org.eclipse.lsp4j.CompletionItem; @@ -164,14 +165,23 @@ public void testNewDependency() throws Exception { /** The session about a script (null path: untitled). */ private static ScriptSession session(final String path) { return sessions.computeIfAbsent(path == null ? "" : path, - p -> new ScriptSession(servers, language).setFile(() -> path == null - ? null : new File(path))); + p -> { + final ScriptSession session = new ScriptSession(servers, language); + session.setFile(() -> path == null ? null : new File(path)); + return session; + }); } /** The problems found in a script, in a session of its own. */ - private static List problems(final String code) { + private static List problems(final String code) + throws Exception + { + final CompletableFuture> found = + new CompletableFuture<>(); try (ScriptSession session = new ScriptSession(servers, language)) { - return session.update(code).diagnostics(120_000); + session.setDiagnosticsListener(found::complete); + session.update(code); + return found.get(120, TimeUnit.SECONDS); } } diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java index 84ada75..215de78 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java @@ -134,7 +134,7 @@ public void testEnvironmentWithoutOne() { // No script environment has been built: the server's own Python. final Environment env = dialect.environment("import os\nos.pa", Optional .empty()); - assertNull(env.interpreter()); + assertEquals(Optional.empty(), env.interpreter()); assertTrue(env.paths().isEmpty()); } @@ -174,7 +174,8 @@ public void testEnvironmentFromEarlierSession() throws Exception { final Environment env = dialect.environment(header + "numpy.ze", Optional.of(script)); // The server analyzes against the environment's Python... - assertEquals(python.toFile().getAbsolutePath(), env.interpreter()); + assertEquals(Optional.of(python.toFile().getAbsolutePath()), env + .interpreter()); // ...without anything built or started. assertTrue(context.service(ResidentWorkerService.class).workers() .isEmpty()); From 3d971b4a5b44b9fa0ee0c1feff65d068a5d5e5ea Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 23 Oct 2026 12:48:38 -0500 Subject: [PATCH 36/37] Follow scijava-code-lsp's ScriptDialect.header Variables Co-Authored-By: Claude Opus 5.5 --- .../python/completion/ApposePythonDialect.java | 8 ++++---- .../python/completion/ApposePythonDialectTest.java | 13 +++++++------ 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java index 33b88c8..eeab636 100644 --- a/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java +++ b/src/main/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialect.java @@ -89,7 +89,7 @@ public Environment environment(final String text, } @Override - public Header header(final Map> parameters) { + public Header header(final Map variables) { final List code = new ArrayList<>(); final List hidden = new ArrayList<>(); // Each Appose-Python script has access to its task @@ -97,15 +97,15 @@ public Header header(final Map> parameters) { hidden.add("appose"); // And numpy is imported for the server whenever an image parameter is // provided (the script itself does not get it) - if (parameters.values().stream().map(ApposePythonDialect::parameterType) + if (variables.values().stream().map(v -> parameterType(v.type())) .anyMatch(NDARRAY::equals)) { code.add("import numpy"); hidden.add("numpy"); } // Declare the types of each parameter for type reasoning - parameters.forEach((name, type) -> { - final String t = parameterType(type); + variables.forEach((name, v) -> { + final String t = parameterType(v.type()); // NB: Values Appose cannot pass are not declared. if (t != null) code.add(name + ": " + t); }); diff --git a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java index 215de78..343663c 100644 --- a/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java +++ b/src/test/java/org/scijava/plugins/scripting/appose/python/completion/ApposePythonDialectTest.java @@ -50,6 +50,7 @@ import org.scijava.code.lsp.Environment; import org.scijava.code.lsp.ScriptDialect; import org.scijava.code.lsp.ScriptDialect.Header; +import org.scijava.code.lsp.ScriptDialect.Variable; import org.scijava.plugin.PluginService; import org.scijava.plugins.scripting.appose.python.ScriptEnvironment; import org.scijava.plugins.scripting.appose.python._internal.ResidentWorkerService; @@ -110,10 +111,10 @@ public void testHeader() { // SciJava Script parameters should be declared in Python code with the // types Appose converts them to, e.g. "image: numpy.ndarray"; and // scripts can use Appose's task variable - final Map> parameters = new LinkedHashMap<>(); - parameters.put("image", Img.class); - parameters.put("sigma", double.class); - parameters.put("context", Context.class); + final Map parameters = new LinkedHashMap<>(); + parameters.put("image", Variable.ofType(Img.class)); + parameters.put("sigma", Variable.ofType(double.class)); + parameters.put("context", Variable.ofType(Context.class)); Header header = dialect.header(parameters); assertEquals("import appose.python_worker; import numpy; " + "image: numpy.ndarray; sigma: float; " + @@ -122,8 +123,8 @@ public void testHeader() { assertEquals(Arrays.asList("appose", "numpy"), Arrays.asList(header .hiddenNames().toArray())); // numpy is imported only when a parameter needs it - header = dialect.header(Collections.singletonMap("sigma", - double.class)); + header = dialect.header(Collections.singletonMap("sigma", Variable + .ofType(double.class))); assertEquals("import appose.python_worker; sigma: float; " + "task: appose.python_worker.Task", header.code()); assertEquals(Collections.singleton("appose"), header.hiddenNames()); From 7723894ed3c401f8e15a2803baa06ddf5fec8a3c Mon Sep 17 00:00:00 2001 From: Gabriel Selzer Date: Fri, 23 Oct 2026 16:26:45 -0500 Subject: [PATCH 37/37] POM: say when scijava-common 2.101.0 needn't be pinned Co-Authored-By: Claude Opus 5.5 --- pom.xml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 147d3ac..47cd289 100644 --- a/pom.xml +++ b/pom.xml @@ -92,7 +92,8 @@ 0.12.0 0.9.0 - + 2.101.0 0.1.0-SNAPSHOT